ctheodoris

/

Geneformer

like 219

Hi i am having some trouble loading fine_tuned_models/gf-12L-95M-i4096_MTLCellClassifier_CELLxGENE_240522

Here is example code

from geneformer.perturber_utils import load_model
load_model(
    "CellClassifier",  2,
    "./Geneformer/fine_tuned_models/gf-12L-95M-i4096_MTLCellClassifier_CELLxGENE_240522",
    "eval"
)

This produces a warning

Some weights of BertForSequenceClassification were not initialized from the model checkpoint at ./Geneformer/fine_tuned_models/gf-12L-95M-i4096_MTLCellClassifier_CELLxGENE_240522 and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

I also tried "MTLCellClassifier" for the "model_type" but get a different warning

Some weights of the model checkpoint at ./Geneformer/fine_tuned_models/gf-12L-95M-i4096_MTLCellClassifier_CELLxGENE_240522 were not used when initializing BertForMaskedLM: ['bert.pooler.dense.bias', 'bert.pooler.dense.weight', 'classification_heads.0.bias', 'classification_heads.0.weight', 'classification_heads.1.bias', 'classification_heads.1.weight', 'classification_heads.2.bias', 'classification_heads.2.weight', 'classification_heads.3.bias', 'classification_heads.3.weight', 'classification_heads.4.bias', 'classification_heads.4.weight']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMaskedLM were not initialized from the model checkpoint at ./Geneformer/fine_tuned_models/gf-12L-95M-i4096_MTLCellClassifier_CELLxGENE_240522 and are newly initialized: ['cls.predictions.bias', 'cls.predictions.decoder.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Is this right?