Update README.md

README.md

@@ -1,199 +1,168 @@
 ---
-library_name: transformers
-tags: []
+license: mit
+language:
+- multilingual
+tags:
+- nlp
+base_model: HuggingFaceTB/SmolLM2-1.7B
+pipeline_tag: text-generation
+inference: true
 ---
 
-# Model Card for Model ID
+# NuExtract-v1.5 by NuMind 🔥
 
-<!-- Provide a quick summary of what the model is/does. -->
+NuExtract-1.5-smol is a fine-tuning of Hugging Face's [SmolLM2-1.7B](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B), intended for structured information extraction. It uses the same training data as [NuExtract-1.5](https://huggingface.co/numind/NuExtract-1.5) and supports multiple languages, while being less than half the size (1.7B vs 3.8B).
 
+To use the model, provide an input text and a JSON template describing the information you need to extract.
 
+Note: This model is trained to prioritize pure extraction, so in most cases all text generated by the model is present as is in the original text.
 
-## Model Details
+Check out the [blog post](https://numind.ai/blog/nuextract-1-5---multilingual-infinite-context-still-small-and-better-than-gpt-4o).
 
-### Model Description
+Try the 3.8B model here: [Playground](https://huggingface.co/spaces/numind/NuExtract-v1.5)
 
-<!-- Provide a longer summary of what this model is. -->
+We also provide a tiny (0.5B) version which is based on Qwen2.5-0.5B: [NuExtract-tiny-v1.5](https://huggingface.co/numind/NuExtract-tiny-v1.5)
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+## Benchmark
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+Zero-shot performance (English):
 
-### Model Sources [optional]
+<p align="left">
+<img src="english_bench.png" style="height: auto;">
+</p>
 
-<!-- Provide the basic links for the model. -->
+Zero-shot performance (Multilingual):
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+<p align="left">
+<img src="multilingual_bench.png" style="height: auto;">
+</p>
 
-## Uses
+## Usage
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+To use the model:
 
-### Direct Use
+```python
+import json
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+def predict_NuExtract(model, tokenizer, texts, template, batch_size=1, max_length=10_000, max_new_tokens=4_000):
+    template = json.dumps(json.loads(template), indent=4)
+    prompts = [f"""<|input|>\n### Template:\n{template}\n### Text:\n{text}\n\n<|output|>""" for text in texts]
+    
+    outputs = []
+    with torch.no_grad():
+        for i in range(0, len(prompts), batch_size):
+            batch_prompts = prompts[i:i+batch_size]
+            batch_encodings = tokenizer(batch_prompts, return_tensors="pt", truncation=True, padding=True, max_length=max_length).to(model.device)
 
-[More Information Needed]
+            pred_ids = model.generate(**batch_encodings, max_new_tokens=max_new_tokens)
+            outputs += tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
 
-### Downstream Use [optional]
+    return [output.split("<|output|>")[1] for output in outputs]
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+model_name = "numind/NuExtract-v1.5"
+device = "cuda"
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16, trust_remote_code=True).to(device).eval()
+tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
 
-[More Information Needed]
+text = """We introduce Mistral 7B, a 7–billion-parameter language model engineered for
+superior performance and efficiency. Mistral 7B outperforms the best open 13B
+model (Llama 2) across all evaluated benchmarks, and the best released 34B
+model (Llama 1) in reasoning, mathematics, and code generation. Our model
+leverages grouped-query attention (GQA) for faster inference, coupled with sliding
+window attention (SWA) to effectively handle sequences of arbitrary length with a
+reduced inference cost. We also provide a model fine-tuned to follow instructions,
+Mistral 7B – Instruct, that surpasses Llama 2 13B – chat model both on human and
+automated benchmarks. Our models are released under the Apache 2.0 license.
+Code: <https://github.com/mistralai/mistral-src>
+Webpage: <https://mistral.ai/news/announcing-mistral-7b/>"""
 
-### Out-of-Scope Use
+template = """{
+    "Model": {
+        "Name": "",
+        "Number of parameters": "",
+        "Number of max token": "",
+        "Architecture": []
+    },
+    "Usage": {
+        "Use case": [],
+        "Licence": ""
+    }
+}"""
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+prediction = predict_NuExtract(model, tokenizer, [text], template)[0]
+print(prediction)
 
-[More Information Needed]
+```
 
-## Bias, Risks, and Limitations
+Sliding window prompting:
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+```python
+import json
 
-[More Information Needed]
+MAX_INPUT_SIZE = 20_000
+MAX_NEW_TOKENS = 6000
 
-### Recommendations
+def clean_json_text(text):
+    text = text.strip()
+    text = text.replace("\#", "#").replace("\&", "&")
+    return text
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+def predict_chunk(text, template, current, model, tokenizer):
+    current = clean_json_text(current)
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
-
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+    input_llm =  f"<|input|>\n### Template:\n{template}\n### Current:\n{current}\n### Text:\n{text}\n\n<|output|>" + "{"
+    input_ids = tokenizer(input_llm, return_tensors="pt", truncation=True, max_length=MAX_INPUT_SIZE).to("cuda")
+    output = tokenizer.decode(model.generate(**input_ids, max_new_tokens=MAX_NEW_TOKENS)[0], skip_special_tokens=True)
+
+    return clean_json_text(output.split("<|output|>")[1])
+
+def split_document(document, window_size, overlap):
+    tokens = tokenizer.tokenize(document)
+    print(f"\tLength of document: {len(tokens)} tokens")
+
+    chunks = []
+    if len(tokens) > window_size:
+        for i in range(0, len(tokens), window_size-overlap):
+            print(f"\t{i} to {i + len(tokens[i:i + window_size])}")
+            chunk = tokenizer.convert_tokens_to_string(tokens[i:i + window_size])
+            chunks.append(chunk)
+
+            if i + len(tokens[i:i + window_size]) >= len(tokens):
+                break
+    else:
+        chunks.append(document)
+    print(f"\tSplit into {len(chunks)} chunks")
+
+    return chunks
+
+def handle_broken_output(pred, prev):
+    try:
+        if all([(v in ["", []]) for v in json.loads(pred).values()]):
+            # if empty json, return previous
+            pred = prev
+    except:
+        # if broken json, return previous
+        pred = prev
+
+    return pred
+
+def sliding_window_prediction(text, template, model, tokenizer, window_size=4000, overlap=128):
+    # split text into chunks of n tokens
+    tokens = tokenizer.tokenize(text)
+    chunks = split_document(text, window_size, overlap)
+
+    # iterate over text chunks
+    prev = template
+    for i, chunk in enumerate(chunks):
+        print(f"Processing chunk {i}...")
+        pred = predict_chunk(chunk, template, prev, model, tokenizer)
+
+        # handle broken output
+        pred = handle_broken_output(pred, prev)
+            
+        # iterate
+        prev = pred
+
+    return pred
+```