README.md

---
license: apache-2.0
datasets:
- briefai/LongShort-Dataset
language:
- en
pipeline_tag: text-generation
tags:
- pytorch
- mistral
- Gen-AI
- Finance
- KPI Extraction
---
# LongShort-Mistral-7B

### Model Description

LongShort-Mistral-7B is a large language model fine-tuned on earnings call documents to extract financial KPIs from the earnings call documents. It is based on the Mistral-7B Instruct Architecture.
- Model creator: [Brief AI](https://huggingface.co/briefai)
- Original model: [Mistral-7B-Instruct](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)
  
### Dataset Description
- Data Source: Factiva
- Data Description: 28K+ Earnings Call Documents
- Data Scope: 1K+ public companies
- Fine Tuning Data: Collection of 60K+ samples.

## Prompt template: LongShort-Mistral-7B

```
[INST]Given the context, answer the question.

### Question:
Extract all the finance-based performance indicators and evaluation metrics.

### Context:
{context}

### Answer:
[/INST]

```
  
## Basics
*This section provides information about the model type, version, license, funders, release date, developers, and contact information.*
*It is useful for anyone who wants to reference the model.*

  
**Developed by:**  [Brief AI Team](https://huggingface.co/briefai)
    
**Model Type:** Transformer-based Large Language Model

**Version:** 1.0.0

**Languages:** English

**License:** Apache 2.0

**Release Date Estimate:** Wednesday, 29.November.2023

**Send Questions to:** [email protected]

**Cite as:** Brief AI LongShort Language Model

**Funded by:**  UChicago Data Science Institute

**Mentored by:**  Nick Kadochnikov

## Technical Specifications
*This section includes details about the model objective and architecture, and the compute infrastructure.*
*It is useful for people interested in model development.*

Please see [the LongShort training README](https://github.com/brief-ai-uchicago/LongShort-Dataset) for full details on replicating training.

### Model Architecture and Objective

* Modified from Mistral-7B-Instruct

**Objective:** Financial KPI extraction from earnings call documents.
    
### Hardware and Software - Compute Infrastructure

* 4 NVIDIA L4 GPUs & 48 vCPUs

* Environment: PyTorch (pytorch-2.0 w/ CUDA-11.8; see [Github link](https://github.com/pytorch/pytorch))

* CPU: GCP G2 Standard 48 (Platform: Intel Cascade Lake) (Accelerator Optimized)

* CPU memory: 192GB RAM

* GPU memory: 30GB per GPU

## Training
*This section provides information about the training.*
*It is useful for people who want to learn more about the model inputs and training footprint.*

The following bits and bytes quantization config was used during training:

* quant_method: bitsandbytes
* load_in_8bit: False
* load_in_4bit: True
* llm_int8_threshold: 6.0
* llm_int8_skip_modules: None
* llm_int8_enable_fp32_cpu_offload: False
* llm_int8_has_fp16_weight: False
* bnb_4bit_quant_type: nf4
* bnb_4bit_use_double_quant: True
* bnb_4bit_compute_dtype: float16
  
Framework versions
* PEFT 0.4.0
    

### Training Data
*This section provides a high-level overview of the training data. It is relevant for anyone who wants to know the basics of what the model is learning.*

Details for the dataset can be found in [LongShort Dataset](https://github.com/brief-ai-uchicago/LongShort-Dataset)

Training data includes:

-   5000 Earnings Call Documents
    
## How to use

This model can be easily used and deployed using HuggingFace's ecosystem. This needs `transformers` and `accelerate` installed. The model can be downloaded as follows:

[LongShort-Mistral-7B](https://huggingface.co/briefai/LongShort-Mistral-7B)

## Intended Use

This model is being created in order to enable public research on large language models (LLMs). LLMs are intended to be used for language generation or as a pre-trained base model that can be further fine-tuned for specific tasks. The use cases below are not exhaustive.

### Direct Use

-   Text generation

-   Exploring characteristics of language generated by a language model

    -   Examples: Cloze tests, counterfactuals, generations with reframings

### Downstream Use

-   Tasks that leverage language models include: Information Extraction, Question Answering, Summarization


#### Out-of-scope Uses

Using the model in [high-stakes](#high-stakes) settings is out of scope for this model.  The model is not designed for [critical decisions](#critical-decisions) nor uses with any material consequences on an individual's livelihood or wellbeing. The model outputs content that appears factual but may not be correct.  

Out-of-scope Uses Include:

-   Usage for evaluating or scoring individuals, such as for employment, education, or credit

-   Applying the model for critical automatic decisions, generating factual content, creating reliable summaries, or generating predictions that must be correct

#### Misuse

Intentionally using the model for harm, violating [human rights](#human-rights), or other kinds of malicious activities, is a misuse of this model. This includes:

-   Spam generation

-   Disinformation and influence operations

-   Disparagement and defamation

-   Harassment and abuse
  
-   [Deception](#deception)

-   Unconsented impersonation and imitation

-   Unconsented surveillance 

-   Generating content without attribution to the model, as specified in the [RAIL License, Use Restrictions](https://huggingface.co/spaces/bigscience/license)

## Intended Users

### Direct Users

-   General Public

-   Researchers

-   Students

-   Educators

-   Engineers/developers

-   Non-commercial entities

-   Financial Industry

# Risks and Limitations
*This section identifies foreseeable harms and misunderstandings.*

Model may:

-   Overrepresent some viewpoints and underrepresent others

-   Contain stereotypes
  
-   Contain [personal information](#personal-data-and-information)

-   Generate:

    -   Hateful, abusive, or violent language

    -   Discriminatory or prejudicial language

    -   Content that may not be appropriate for all settings, including sexual content

-   Make errors, including producing incorrect information as if it were factual

-   Generate irrelevant or repetitive outputs

-   Induce users into attributing human traits to it, such as sentience or consciousness


# Evaluation
*This section describes the evaluation protocols and provides the results.*

Result: LongShort-Llama-2-13B gives 43.4% accuracy on a validation set of 10% of the original training dataset.



**Train-time Evaluation:**

Final checkpoint after 300 epochs:

- Training Loss: 1.228



# Recommendations
*This section provides information on warnings and potential mitigations.*

-   Indirect users should be made aware when the content they're working with is created by the LLM.

-   Users should be aware of [Risks and Limitations](#risks-and-limitations), and include an appropriate age disclaimer or blocking interface as necessary.

-   Users of the model should provide mechanisms for those affected to provide feedback, such as an email address for comments.

# Model Card Authors
Vishal Parameshwaran, Garima Sohi, Jose Gerala, Sanchit Narayan Kumar