Adding Custom Torch or Huggingface Models
Hello, I am trying to add some additional models to the setup here out of curiosity. I've noticed that you need to add a params.pt file, a config.json file, and update model_surgery.py in order to successfully add a model. I assume the config.json file is the same as the file you would find on most model's HF repos but I am not sure where the params.pt file is supposed to come from. Is this just supposed to be the model's last checkpoint?
Also, in model_surgery.py there is a method called get_final_norm. Is this just supposed to be the last time a layer norm was applied? What if your model does not end with a torch.nn.LayerNorm?
I'm very sorry for the delayed response. In case you are still interest and for others, the config.jsonand params.pt are the checkpoints for the trained tuned lens. They are simply the results of saving a lens to disk. To train a lens follow this tutorial. config.jsonand params.pt are saved to the path specified by --output.