Upload README.md with huggingface_hub

README.md

@@ -1,199 +1,155 @@
 ---
-library_name: transformers
-tags: []
+library_name: tokenizers
+tags: ['kl3m', 'kl3m-003', 'alea', 'legal', 'financial']
+date: 2024-03-15
 ---
 
-# Model Card for Model ID
+# kl3m-003-64k tokenizer
 
-<!-- Provide a quick summary of what the model is/does. -->
+The `kl3m-001-32k` tokenizer is a domain-specific tokenizer trained on ~1.5T tokens of financial and legal text from sources that contain primarily English, German, Spanish, and French language.
 
+This tokenizer was used for the second generation of KL3M embedding and generative models, including
+`kl3m-3.7B`, `kl3m-7B`, `kl3m-embedding-003`, and `kl3m-embedding-004`.
 
+Please see `kl3m-001-32k` for the first iteration of our research on domain-specific tokenization.
 
 ## Model Details
 
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
+### Summary
 
-[More Information Needed]
+- **Vocabulary**: 65,536
+- **Tokenizer type:** BPE
+- **Special token support:** Both causal and masked language modeling
+- **Language(s) (NLP):** Primarily English, Spanish, German, French, with a small percentage of other EU languages.
+- **Developed by:** Originally by [273 Ventures LLC](https://273ventures.com), donated to [ALEA Institute](https://aleainstitute.ai).
+- **License:** [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/)
 
-## Training Details
 
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+### Model Description
 
-[More Information Needed]
+The `kl3m-003-64k` tokenizer is a domain-specific tokenizer trained on ~1.5T tokens of financial and legal text from primarily-English sources.
 
-### Results
+This tokenizer is notable for a number of reasons:
 
-[More Information Needed]
+#### Domain Specific
 
-#### Summary
+As part of our research on more efficient SLM training for the legal and financial domain, we
+trained a domain-specific tokenizer on a large corpus of financial and legal text. This tokenizer
+has not, for example, seen any common general pretrain sources like Wikipedia or Common Crawl.
 
+#### Large Added Token Set
 
+As part of our research on efficient and reliable extraction and generation, we inserted
+a large numer of deterministic "whole" tokens into the tokenizer, such as HTML tags
+like `<span`, common Markdown elements like `#` and `##`, and legal enumerations like `(a)`.
 
-## Model Examination [optional]
+**Note that the kl3m-003-64k tokenizer has added a number of additional citation formats that were not included
+in the kl3m-001-32k tokenizer.**  These were primarily sourced from empirical data and
+the [Free Law Project's reporters-db](https://raw.githubusercontent.com/freelawproject/reporters-db/main/reporters_db/data/),
+which were added to the tokenizer to improve model behavior related to legal citations.
 
-<!-- Relevant interpretability work for the model goes here -->
+See the `get_custom_tokens` method in `kl3m_embeddings/training/kl3m_003/train_tokenizer.py` for
+more details:
 
-[More Information Needed]
+```python
+def get_custom_tokens(
+    include_whitespace: bool = True,
+    include_markdown: bool = True,
+    include_html: bool = True,
+    include_json: bool = True,
+    include_xml: bool = True,
+    include_years: bool = True,
+    include_citations: bool = True,
+    lowercase: bool = False,
+) -> list[str]:
+```
 
-## Environmental Impact
+#### Space Preservation
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+Unlike `kl3m-001-32k`, we *do not* retain the space character as a token.  This was done after adding additional legal 
+citation tokens to the vocabulary, which reduced the number of issues related to space tokenization in legal text.  This
+means that the `kl3m-003-64k` tokenizer uses substantially fewer tokens than `kl3m-001-32k` for most text.
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+#### Special Tokens for both Embedding and Generative Models
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+For both training and inference efficiency, we intended this tokenizer vocabulary to be
+usable for both embedding and generative models. As such, we included special tokens
+suitable for both causal and masked language modeling tasks.
 
-## Technical Specifications [optional]
+* `<|start|>`: `0`
+* `<|end|>`: `1`
+* `<|pad|>`: `2`
+* `<|unk|>`: `3`
+* `<|sep|>`: `4`
+* `<|cls|>`: `5`
+* `<|mask|>`: `6`
 
-### Model Architecture and Objective
+We also added a number of chat and instruction tokens that were not included in `kl3m-001-32k`, including:
 
-[More Information Needed]
+* `<|system|>`: `7`
+* `</|system|>`: `8`
+* `<|user|>`: `9`
+* `</|user|>`: `10` 
+* `<|instruction|>`: `11`
+* `</|instruction|>`: `12` 
 
-### Compute Infrastructure
+### Replication
 
-[More Information Needed]
+The entire data collection and preprocesing pipeline is being made available, along with
+training data, as part of the [ALEA Institute](https://aleainstitute.ai) [KL3M project](https://aleainstitute.ai/work/kl3m/).
 
-#### Hardware
+The source code to used to train the tokenizer is available on GitHub at:
+[https://github.com/alea-institute/kl3m-embedding-research](https://github.com/alea-institute/kl3m-embedding-research)
 
-[More Information Needed]
+The data pipeline will be available on GitHub and S3 in the near future.
 
-#### Software
+## Uses
 
-[More Information Needed]
+This tokenizer is intended to be used for English, Spanish, German, or French language text in professional contexts
+such as legal and financial documents.
 
-## Citation [optional]
+### Recommendations
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+In general, the `kl3m-003-64k` tokenizer is recommended over the original `kl3m-001-32k` tokenizer.
 
-**BibTeX:**
+```text
+Original text:  The Comptroller of the Currency shall have the same authority with respect to functions transferred to
+ the Comptroller of the Currency under the Enhancing Financial Institution Safety and Soundness Act of 2010 as was
+ vested in the Director of the Office of Thrift Supervision on the transfer date, as defined in section 311 of that
+ Act [12 U.S.C. 5411].
 
-[More Information Needed]
+kl3m-001-32k
+--------------------
+Size:  147
+Tokens:  ['The', ' ', 'Comp', 'troller', ' ', 'of', ' ', 'the', ' ', 'C', 'urrency', ' ', 'shall', ' ', 'have', ' ', 'the', ' ', 'same', ' ', 'authority', ' ', 'with', ' ', 'respect', ' ', 'to', ' ', 'fun', 'ctions', ' ', 'transferred', ' ', 'to', '\n', ' ', 'the', ' ', 'Comp', 'troller', ' ', 'of', ' ', 'the', ' ', 'C', 'urrency', ' ', 'under', ' ', 'the', ' ', 'En', 'ha', 'ncing', ' ', 'Financial', ' ', 'Institution', ' ', 'Sa', 'fe', 'ty', ' ', 'a', 'n', 'd', ' ', 'S', 'ound', 'ness', ' ', 'Act', ' ', 'of', ' ', '2010', ' ', 'as', ' ', 'was', '\n', ' ', 'vested', ' ', 'i', 'n', ' ', 'the', ' ', 'Director', ' ', 'of', ' ', 'the', ' ', 'Office', ' ', 'of', ' ', 'Th', 'rift', ' ', 'Superv', 'ision', ' ', 'o', 'n', ' ', 'the', ' ', 'transfer', ' ', 'date', ',', ' ', 'as', ' ', 'defined', ' ', 'i', 'n', ' ', 'section', ' ', '311', ' ', 'of', ' ', 'that', '\n', ' ', 'Act', ' ', '[', '12', ' ', 'U', '.', 'S', '.', 'C', '.', ' ', '54', '11', '].']
+IDs:  [815, 31673, 3546, 14529, 31673, 269, 31673, 441, 31673, 41, 9646, 31673, 5516, 31673, 4130, 31673, 441, 31673, 8685, 31673, 14765, 31673, 1946, 31673, 12500, 31673, 265, 31673, 12122, 1935, 31673, 12677, 31673, 265, 31674, 31673, 441, 31673, 3546, 14529, 31673, 269, 31673, 441, 31673, 41, 9646, 31673, 2823, 31673, 441, 31673, 1871, 288, 2655, 31673, 20796, 31673, 29543, 31673, 4778, 362, 1004, 31673, 71, 84, 74, 31673, 57, 1098, 1647, 31673, 8494, 31673, 269, 31673, 3629, 31673, 310, 31673, 3182, 31674, 31673, 9761, 31673, 79, 84, 31673, 441, 31673, 21209, 31673, 269, 31673, 441, 31673, 8827, 31673, 269, 31673, 788, 11004, 31673, 28799, 873, 31673, 85, 84, 31673, 441, 31673, 12790, 31673, 2726, 18, 31673, 310, 31673, 10212, 31673, 79, 84, 31673, 3517, 31673, 15340, 31673, 269, 31673, 1704, 31674, 31673, 8494, 31673, 65, 534, 31673, 59, 20, 57, 20, 41, 20, 31673, 2195, 572, 5582]
 
-**APA:**
+kl3m-003-64k
+--------------------
+Size:  70
+Tokens:  ['The', 'ĠComptroller', 'Ġof', 'Ġthe', 'ĠCurrency', 'Ġshall', 'Ġhave', 'Ġthe', 'Ġsame', 'Ġauthority', 'Ġwith', 'Ġrespect', 'Ġto', 'Ġfunctions', 'Ġtransferred', 'Ġto', 'Ċ', 'Ġthe', 'ĠComptroller', 'Ġof', 'Ġthe', 'ĠCurrency', 'Ġunder', 'Ġthe', 'ĠEnh', 'ancing', 'ĠFinancial', 'ĠInstitution', 'ĠSafety', 'Ġand', 'Ġ', 'Sound', 'ness', 'ĠAct', 'Ġof', 'Ġ2010', 'Ġas', 'Ġwas', 'Ċ', 'Ġvested', 'Ġin', 'Ġthe', 'ĠDirector', 'Ġof', 'Ġthe', 'ĠOffice', 'Ġof', 'ĠThrift', 'ĠSupervision', 'Ġon', 'Ġthe', 'Ġtransfer', 'Ġdate', ',', 'Ġas', 'Ġdefined', 'Ġin', 'Ġsection', 'Ġ311', 'Ġof', 'Ġthat', 'Ċ', 'ĠAct', 'Ġ[', '12', 'Ġ', 'U.S.C.', 'Ġ54', '11', '].']
+IDs:  [671, 13273, 295, 281, 25922, 735, 704, 281, 1913, 2451, 440, 1894, 312, 5860, 7264, 312, 211, 281, 13273, 295, 281, 25922, 621, 281, 18926, 4406, 3195, 24448, 5617, 310, 233, 63589, 2130, 854, 295, 1611, 398, 725, 211, 11978, 300, 281, 2827, 295, 281, 1767, 295, 44029, 37141, 395, 281, 3696, 1548, 24, 398, 3011, 300, 782, 6590, 295, 407, 211, 854, 1327, 524, 233, 63761, 3789, 547, 8578]
 
-[More Information Needed]
+```
 
-## Glossary [optional]
+## How to Get Started with the Model
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+Use the code below to get started with the model.
 
-[More Information Needed]
+```
+from tokenizers import Tokenizer
 
-## More Information [optional]
+tokenizer = Tokenizer.from_pretrained('alea-institute/kl3m-003-64k')
+```
 
-[More Information Needed]
+## Citation
 
-## Model Card Authors [optional]
+Tokenizer and dataset publications are pending.
 
-[More Information Needed]
+## Contact
 
-## Model Card Contact
+For any questions, please contact [ALEA Institute](https://aleainstitute.ai) at [[email protected]](mailto:[email protected]) or
+create an issue on this repository or [GitHub](https://github.com/alea-institute/kl3m-embedding-research).
 
-[More Information Needed]
+![logo](https://aleainstitute.ai/images/alea-logo-ascii-1x1.png)