File size: 9,726 Bytes
bd4968f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 |
---
language:
- en
pipeline_tag: text-generation
tags:
- llama-3.1
- finance
- economics
- math
- reasoning
- finetuning
license: other
library_name: transformers
---
<font size="1">Model v1.0</font>
## Llama-3.1-Hawkish-8B v1
Model has been further finetuned on a set of newly generated 50m high quality tokens related to Financial topics covering topics such as Economics, Fixed Income, Equities, Corporate Financing, Derivatives and Portfolio Management. Data was gathered from publicly available sources and went through several stages of curation into instruction data from the initial amount of 250m+ tokens. To aid in mitigating forgetting information from the original finetune, the data was mixed with instruction sets on the topics of Coding, General Knowledge, NLP and Conversational Dialogue.
The model has shown to improve over a number of benchmarks over the original model, notably in Math and Economics. This model represents the first time a 8B model has been able to convincingly get a passing score on the CFA Level 1 exam, requiring a typical 300 hours of studying, indicating a significant improvement in Financial Knowledge.
## CFA Level 1 Mock Exam Results
If you work in the financial and investment sectors, you will know about the CFA - their exam is known to be **“the world's toughest”**, requiring typically a total of over 1000 hours to study all 3 levels. Below is a comparison of different models on a sample Level 1 CFA Mock Exam, showing Llama Hawkish outperforming much larger models on the exam. The same prompt was used for all models, results are all 0-shot CoT. Sample mock exam with comparison to other models shown below.
<table>
<thead>
<tr>
<th>CFA Level 1</th>
<th>GPT-4o-mini (%)</th>
<th>Llama Hawkish 8B (%)</th>
<th>Meta-Llama Instruct 8B (%)</th>
<th>Meta-Llama Instruct 70B (%)</th>
<th>Palmyra Fin 70B (%)</th>
</tr>
</thead>
<tbody>
<tr>
<td>Ethical and Professional Standards</td>
<td><b>77.77</b></td>
<td><b>77.77</b></td>
<td>55.55</td>
<td>66.6</td>
<td>61.11</td>
</tr>
<tr>
<td>Quantitative Methods</td>
<td>71.43</td>
<td>71.4</td>
<td>64.28</td>
<td><b>85.71</b></td>
<td>71.4</td>
</tr>
<tr>
<td>Economics</td>
<td>66.66</td>
<td><b>75</b></td>
<td>58</td>
<td>58.33</td>
<td>41.66</td>
</tr>
<tr>
<td>Financial Reporting</td>
<td>79.2</td>
<td><b>87.5</b></td>
<td>66.6</td>
<td>70.83</td>
<td>50</td>
</tr>
<tr>
<td>Corporate Finance</td>
<td><b>80</b></td>
<td>60</td>
<td>50</td>
<td><b>80</b></td>
<td>50</td>
</tr>
<tr>
<td>Equity Investments</td>
<td>50</td>
<td>50</td>
<td>41.6</td>
<td><b>66.6</b></td>
<td>41.66</td>
</tr>
<tr>
<td>Fixed Income</td>
<td><b>78.57</b></td>
<td>50</td>
<td>28.57</td>
<td>50</td>
<td>42.85</td>
</tr>
<tr>
<td>Derivatives</td>
<td>50</td>
<td><b>66.7</b></td>
<td>33.3</td>
<td>33.3</td>
<td>50</td>
</tr>
<tr>
<td>Alternative Investments</td>
<td><b>100</b></td>
<td><b>100</b></td>
<td>75</td>
<td><b>100</b></td>
<td>75</td>
</tr>
<tr>
<td>Portfolio Management</td>
<td>83.3</td>
<td>83.3</td>
<td>50</td>
<td><b>100</b></td>
<td>83.3</td>
</tr>
<tr>
<td><b>Weighted Average</b></td>
<td><b>73.49</b></td>
<td><b>71.43</b></td>
<td>52.7672</td>
<td><b>69.86</b></td>
<td>54.77</td>
<td>-</td>
</tr>
<tr>
<td><b>Result</b></td>
<td><b>PASS</b></td>
<td><b>PASS</b></td>
<td><b>FAIL</b></td>
<td><b>PASS</b></td>
<td><b>FAIL</b></td>
</tr>
</tbody>
</table>
The mock exams are all of varying difficulty and pass rates can be anywhere from 64% to 72% for different Level 1 Mock Exams, with the average being around 67% which is above the 12 year average MPS of 65% for all CFA years. (https://300hours.com/cfa-passing-score/)
Some other Frontier Models were tested on CFA Level 1 Mock Exam, their scores shown below for reference, due to this being a 8B model it is not deemed feasible to outperform these however they leave benchmarks for larger versions of Hawkish or others researching financial data mixes in fine tuning.
<table border="1" cellpadding="8" cellspacing="0"> <thead> <tr> <th>Model</th> <th>Performance (%)</th> </tr> </thead> <tbody> <tr> <td>Qwen2.5 - 70B</td> <td>81.66%</td> </tr> <tr> <td>Nemotron - 70B</td> <td>79.16%</td> </tr> <tr> <td>GPT-4o</td> <td><strong>85.83%</strong></td> </tr> <tr> <td>GPT-4o-mini</td> <td>73.49%</td> </tr> <tr> <td>Gemini 1.5 Pro</td> <td>75%</td> </tr> <tr> <td>Llama 405B Instruct</td> <td>77.5%</td> </tr> <tr> <td>DeepSeekChat v2.5</td> <td>66.6%</td> </tr> </tbody> </table>
### Model Usage Example
```python
from vllm import LLM, SamplingParams
# Load the LLM using vLLM
llm = LLM(model="mukaj/Llama-3.1-Hawkish-8B", dtype="auto")
tokenizer = llm.get_tokenizer()
messages = [
{"role": "system", "content": "You are a Financial Analyst. Reason step by step before answering."},
{"role": "user", "content": """Given that an equal-weighted index and a market-capitalization-weighted index consist of the same securities, underperformance by small-cap stocks will most likely result in the market-capitalization-weighted index exhibiting what price returns compared to the equal weighted index?""" }
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# Sampling configuration for vLLM
sampling_params = SamplingParams(temperature=0.2, max_tokens=512)
# Generate response using vLLM
generation = llm.generate(prompt, sampling_params)
# Decode response
generated_response = generation[0].outputs[0].text
print(generated_response)
```
#### Output
=============
To analyze the impact of underperformance by small-cap stocks on the price returns of an equal-weighted index versus a market-capitalization-weighted index, we need to consider how each index is constructed and how their weights affect their performance.
1. **Index Construction**:
- **Equal-Weighted Index**: Each security in the index has the same weight, regardless of its market capitalization. This means that all stocks contribute equally to the index's performance.
- **Market-Capitalization-Weighted Index**: Each security's weight in the index is proportional to its market capitalization. Larger companies have a greater influence on the index's performance.
2. **Impact of Underperformance**:
- If small-cap stocks underperform, it means that these stocks are not performing as well as they were expected to. In an equal-weighted index, the underperformance of small-cap stocks will still be reflected in the index's performance, but since all stocks have equal weights, the overall impact will be more pronounced.
- In a market-capitalization-weighted index, the underperformance of small-cap stocks will have a more muted effect on the index's performance because larger stocks (which typically have higher market caps) will still dominate the index's returns. The influence of small-cap stocks, even if they are underperforming, is less significant due to their lower weight.
3. **Price Returns Comparison**:
- As a result of the underperformance of small-cap stocks, the equal-weighted index is likely to show a more significant decline in price returns compared to the market-capitalization-weighted index. This is because the equal-weighted index is more sensitive to the performance of all stocks, including the underperforming small-cap stocks.
- Conversely, the market-capitalization-weighted index will likely exhibit less severe price declines because the larger stocks will continue to drive the index's performance, mitigating the impact of the underperformance of smaller stocks.
In summary, underperformance by small-cap stocks will most likely result in the market-capitalization-weighted index exhibiting **higher price returns** compared to the equal-weighted index. The equal-weighted index will likely show a more pronounced decline in price returns due to the equal weighting of all stocks, including the underperforming small-cap stocks.
================
## Disclaimer & Intended Uses
### Model & License
This research model is derived from Meta's LLaMA 3.1 architecture and is subject to LLaMA 3.1 community license terms, with additional restrictions below. While based on LLaMA 3.1, this is an independent research implementation focused on studying financial data influences in training. Users acknowledge this is experimental research, use at own risk, and accept full responsibility for any implementation or application.
### Permitted Use
- Academic and research purposes only
- No production environments or real-world applications
- No financial decision-making or advisory use
### Liability & Responsibility
The creators of this model:
- Accept no responsibility for any use of the model
- Provide no warranties or guarantees
- Make no claims about accuracy or reliability
### Intellectual Property & Attribution
- All findings and opinions are solely those of the authors
- Not endorsed by or affiliated with Meta, CFA Institute or any institutions
- All trademarks belong to respective owners
The creators reserve the right to modify these terms at any time. |