|
---
|
|
language: en
|
|
license: cc-by-sa-4.0
|
|
tags:
|
|
- embeddings
|
|
---
|
|
|
|
# Acquiring Bidirectionality via Large and Small Language Models
|
|
|
|
This model is LLM for obtaining token-level representations as proposed in our COLING 2025 paper "[Acquiring Bidirectionality via Large and Small Language Models](https://arxiv.org/abs/2408.09640)."
|
|
Using token representation from bidirectional language models (LMs) such as BERT is still a widely used approach for token-classification tasks.
|
|
Even though there exist much larger unidirectional LMs such as Llama-2, they are rarely used to replace the token representation of bidirectional LMs.
|
|
We propose to newly train a small backward LM and concatenate its representations to those of an existing LM for downstream tasks.
|
|
|
|
This model is the "small backward LM" and it needs to be combined with another forward LM such as [GPT-2](https://huggingface.co/openai-community/gpt2).
|
|
Please refer to our [official repository](https://github.com/hitachi-nlp/backward-llm) to use this model.
|
|
This particular model uses GPT2 vocabulary for its training.
|
|
|
