Upload 12 files

README.md

@@ -1,12 +1,110 @@
----
-title: FinGPT Forecaster
-emoji: 🦀
-colorFrom: green
-colorTo: yellow
-sdk: gradio
-sdk_version: 4.12.0
-app_file: app.py
-pinned: false
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+![title](figs/title.png)
+
+## What is FinGPT-Forecaster?
+- FinGPT-Forecaster takes market news and optional basic financials related to the specified company from the past few weeks as input and responds with the company's **positive developments** and **potential concerns**. Then it gives out a **prediction** of stock price movement for the coming week and its **analysis** summary.
+- FinGPT-Forecaster is finetuned on Llama-2-7b-chat-hf with LoRA on the past year's DOW30 market data. But also has shown great generalization ability on other ticker symbols.
+- FinGPT-Forecaster is an easy-to-deploy junior robo-advisor, a milestone towards our goal.
+
+## Try out the demo!
+
+Try our demo at <https://huggingface.co/spaces/FinGPT/FinGPT-Forecaster>
+
+![demo_interface](figs/interface.png)
+
+Enter the following inputs:
+
+1) ticker symbol (e.g. AAPL, MSFT, NVDA)
+2) the day from which you want the prediction to happen (yyyy-mm-dd)
+3) the number of past weeks where market news are retrieved
+4) whether to add latest basic financials as additional information
+
+Then, click Submit！You'll get a response like this
+
+![demo_response](figs/response.png)
+
+This is just a demo showing what this model is capable of. Results inferred from randomly chosen news can be strongly biased.
+For more detailed and customized usage, scroll down and continue your reading.
+
+## Deploy FinGPT-Forecaster
+
+We have released our FinGPT-Forecaster trained on DOW30 market data from 2022-12-30 to 2023-9-1 on HuggingFace: [fingpt-forecaster_dow30_llama2-7b_lora](https://huggingface.co/FinGPT/fingpt-forecaster_dow30_llama2-7b_lora)
+
+We have most of the key requirements in `requirements.txt`. Before you start, do `pip install -r requirements.txt`. Then you can refer to `demo.ipynb` for our deployment and evaluation script.
+
+First let's load the model:
+
+```
+from datasets import load_dataset
+from transformers import AutoTokenizer, AutoModelForCausalLM
+from peft import PeftModel
+
+
+base_model = AutoModelForCausalLM.from_pretrained(
+    'meta-llama/Llama-2-7b-chat-hf',
+    trust_remote_code=True,
+    device_map="auto",
+    torch_dtype=torch.float16,   # optional if you have enough VRAM
+)
+tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-chat-hf')
+
+model = PeftModel.from_pretrained(base_model, 'FinGPT/fingpt-forecaster_dow30_llama2-7b_lora')
+model = model.eval()
+```
+
+Then you are ready to go, prepare your prompt with news & stock price movements in llama format (which we'll mention in the next section), and generate your own forecasting results!
+```
+B_INST, E_INST = "[INST]", "[/INST]"
+B_SYS, E_SYS = "<<SYS>>\n", "\n<</SYS>>\n\n"
+
+prompt = B_INST + B_SYS + {SYSTEM_PROMPT} + E_SYS + {YOUR_PROMPT} + E_INST
+inputs = tokenizer(
+    prompt, return_tensors='pt'
+)
+inputs = {key: value.to(model.device) for key, value in inputs.items()}
+        
+res = model.generate(
+    **inputs, max_length=4096, do_sample=True,
+    eos_token_id=tokenizer.eos_token_id,
+    use_cache=True
+)
+output = tokenizer.decode(res[0], skip_special_tokens=True)
+answer = re.sub(r'.*\[/INST\]\s*', '', output, flags=re.DOTALL) # don't forget to import re
+```
+
+## Data Preparation
+Company profile & Market news & Basic financials & Stock prices are retrieved using **yfinance & finnhub**.
+
+Prompts used are organized as below:
+
+```
+SYSTEM_PROMPT = "You are a seasoned stock market analyst. Your task is to list the positive developments and potential concerns for companies based on relevant news and basic financials from the past weeks, then provide an analysis and prediction for the companies' stock price movement for the upcoming week. Your answer format should be as follows:\n\n[Positive Developments]:\n1. ...\n\n[Potential Concerns]:\n1. ...\n\n[Prediction & Analysis]:\n...\n"
+
+prompt = """
+[Company Introduction]:
+
+{name} is a leading entity in the {finnhubIndustry} sector. Incorporated and publicly traded since {ipo}, the company has established its reputation as one of the key players in the market. As of today, {name} has a market capitalization of {marketCapitalization:.2f} in {currency}, with {shareOutstanding:.2f} shares outstanding. {name} operates primarily in the {country}, trading under the ticker {ticker} on the {exchange}. As a dominant force in the {finnhubIndustry} space, the company continues to innovate and drive progress within the industry.
+
+From {startDate} to {endDate}, {name}'s stock price {increase/decrease} from {startPrice} to {endPrice}. Company news during this period are listed below:
+
+[Headline]: ...
+[Summary]: ...
+
+[Headline]: ...
+[Summary]: ...
+
+Some recent basic financials of {name}, reported at {date}, are presented below:
+
+[Basic Financials]:
+{attr1}: {value1}
+{attr2}: {value2}
+...
+
+Based on all the information before {curday}, let's first analyze the positive developments and potential concerns for {symbol}. Come up with 2-4 most important factors respectively and keep them concise. Most factors should be inferred from company-related news. Then make your prediction of the {symbol} stock price movement for next week ({period}). Provide a summary analysis to support your prediction.
+
+"""
+```
+## Train your own FinGPT-Forecaster
+
+
+
+**Disclaimer: Nothing herein is financial advice, and NOT a recommendation to trade real money. Please use common sense and always first consult a professional before trading or investing.**

app.py

@@ -0,0 +1,320 @@
+import os
+import re
+import time
+import json
+import random
+import finnhub
+import torch
+import gradio as gr
+import pandas as pd
+import yfinance as yf
+from pynvml import *
+from peft import PeftModel
+from collections import defaultdict
+from datetime import date, datetime, timedelta
+from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
+
+
+access_token = os.environ["HF_TOKEN"]
+finnhub_client = finnhub.Client(api_key=os.environ["FINNHUB_API_KEY"])
+
+base_model = AutoModelForCausalLM.from_pretrained(
+    'meta-llama/Llama-2-7b-chat-hf',
+    token=access_token,
+    trust_remote_code=True, 
+    device_map="auto",
+    torch_dtype=torch.float16,
+    offload_folder="offload/"
+)
+model = PeftModel.from_pretrained(
+    base_model,
+    'FinGPT/fingpt-forecaster_dow30_llama2-7b_lora',
+    offload_folder="offload/"
+)
+model = model.eval()
+
+tokenizer = AutoTokenizer.from_pretrained(
+    'meta-llama/Llama-2-7b-chat-hf',
+    token=access_token
+)
+
+streamer = TextStreamer(tokenizer)
+
+B_INST, E_INST = "[INST]", "[/INST]"
+B_SYS, E_SYS = "<<SYS>>\n", "\n<</SYS>>\n\n"
+
+SYSTEM_PROMPT = "You are a seasoned stock market analyst. Your task is to list the positive developments and potential concerns for companies based on relevant news and basic financials from the past weeks, then provide an analysis and prediction for the companies' stock price movement for the upcoming week. " \
+    "Your answer format should be as follows:\n\n[Positive Developments]:\n1. ...\n\n[Potential Concerns]:\n1. ...\n\n[Prediction & Analysis]\nPrediction: ...\nAnalysis: ..."
+
+
+def print_gpu_utilization():
+    
+    nvmlInit()
+    handle = nvmlDeviceGetHandleByIndex(0)
+    info = nvmlDeviceGetMemoryInfo(handle)
+    print(f"GPU memory occupied: {info.used//1024**2} MB.")
+
+
+def get_curday():
+    
+    return date.today().strftime("%Y-%m-%d")
+
+
+def n_weeks_before(date_string, n):
+    
+    date = datetime.strptime(date_string, "%Y-%m-%d") - timedelta(days=7*n)
+
+    return date.strftime("%Y-%m-%d")
+
+
+def get_stock_data(stock_symbol, steps):
+
+    stock_data = yf.download(stock_symbol, steps[0], steps[-1])
+    if len(stock_data) == 0:
+        raise gr.Error(f"Failed to download stock price data for symbol {stock_symbol} from yfinance!")
+    
+#     print(stock_data)
+    
+    dates, prices = [], []
+    available_dates = stock_data.index.format()
+    
+    for date in steps[:-1]:
+        for i in range(len(stock_data)):
+            if available_dates[i] >= date:
+                prices.append(stock_data['Close'][i])
+                dates.append(datetime.strptime(available_dates[i], "%Y-%m-%d"))
+                break
+
+    dates.append(datetime.strptime(available_dates[-1], "%Y-%m-%d"))
+    prices.append(stock_data['Close'][-1])
+    
+    return pd.DataFrame({
+        "Start Date": dates[:-1], "End Date": dates[1:],
+        "Start Price": prices[:-1], "End Price": prices[1:]
+    })
+
+
+def get_news(symbol, data):
+    
+    news_list = []
+    
+    for end_date, row in data.iterrows():
+        start_date = row['Start Date'].strftime('%Y-%m-%d')
+        end_date = row['End Date'].strftime('%Y-%m-%d')
+#         print(symbol, ': ', start_date, ' - ', end_date)
+        time.sleep(1) # control qpm
+        weekly_news = finnhub_client.company_news(symbol, _from=start_date, to=end_date)
+        if len(weekly_news) == 0:
+            raise gr.Error(f"No company news found for symbol {symbol} from finnhub!")
+        weekly_news = [
+            {
+                "date": datetime.fromtimestamp(n['datetime']).strftime('%Y%m%d%H%M%S'),
+                "headline": n['headline'],
+                "summary": n['summary'],
+            } for n in weekly_news
+        ]
+        weekly_news.sort(key=lambda x: x['date'])
+        news_list.append(json.dumps(weekly_news))
+    
+    data['News'] = news_list
+    
+    return data
+
+
+def get_company_prompt(symbol):
+
+    profile = finnhub_client.company_profile2(symbol=symbol)
+    if not profile:
+        raise gr.Error(f"Failed to find company profile for symbol {symbol} from finnhub!")
+        
+    company_template = "[Company Introduction]:\n\n{name} is a leading entity in the {finnhubIndustry} sector. Incorporated and publicly traded since {ipo}, the company has established its reputation as one of the key players in the market. As of today, {name} has a market capitalization of {marketCapitalization:.2f} in {currency}, with {shareOutstanding:.2f} shares outstanding." \
+        "\n\n{name} operates primarily in the {country}, trading under the ticker {ticker} on the {exchange}. As a dominant force in the {finnhubIndustry} space, the company continues to innovate and drive progress within the industry."
+
+    formatted_str = company_template.format(**profile)
+    
+    return formatted_str
+
+
+def get_prompt_by_row(symbol, row):
+
+    start_date = row['Start Date'] if isinstance(row['Start Date'], str) else row['Start Date'].strftime('%Y-%m-%d')
+    end_date = row['End Date'] if isinstance(row['End Date'], str) else row['End Date'].strftime('%Y-%m-%d')
+    term = 'increased' if row['End Price'] > row['Start Price'] else 'decreased'
+    head = "From {} to {}, {}'s stock price {} from {:.2f} to {:.2f}. Company news during this period are listed below:\n\n".format(
+        start_date, end_date, symbol, term, row['Start Price'], row['End Price'])
+    
+    news = json.loads(row["News"])
+    news = ["[Headline]: {}\n[Summary]: {}\n".format(
+        n['headline'], n['summary']) for n in news if n['date'][:8] <= end_date.replace('-', '') and \
+        not n['summary'].startswith("Looking for stock market analysis and research with proves results?")]
+
+    basics = json.loads(row['Basics'])
+    if basics:
+        basics = "Some recent basic financials of {}, reported at {}, are presented below:\n\n[Basic Financials]:\n\n".format(
+            symbol, basics['period']) + "\n".join(f"{k}: {v}" for k, v in basics.items() if k != 'period')
+    else:
+        basics = "[Basic Financials]:\n\nNo basic financial reported."
+    
+    return head, news, basics
+
+
+def sample_news(news, k=5):
+    
+    return [news[i] for i in sorted(random.sample(range(len(news)), k))]
+
+
+def get_current_basics(symbol, curday):
+
+    basic_financials = finnhub_client.company_basic_financials(symbol, 'all')
+    if not basic_financials['series']:
+        raise gr.Error(f"Failed to find basic financials for symbol {symbol} from finnhub!")
+        
+    final_basics, basic_list, basic_dict = [], [], defaultdict(dict)
+    
+    for metric, value_list in basic_financials['series']['quarterly'].items():
+        for value in value_list:
+            basic_dict[value['period']].update({metric: value['v']})
+
+    for k, v in basic_dict.items():
+        v.update({'period': k})
+        basic_list.append(v)
+        
+    basic_list.sort(key=lambda x: x['period'])
+    
+    for basic in basic_list[::-1]:
+        if basic['period'] <= curday:
+            break
+            
+    return basic
+    
+
+def get_all_prompts_online(symbol, data, curday, with_basics=True):
+
+    company_prompt = get_company_prompt(symbol)
+
+    prev_rows = []
+
+    for row_idx, row in data.iterrows():
+        head, news, _ = get_prompt_by_row(symbol, row)
+        prev_rows.append((head, news, None))
+        
+    prompt = ""
+    for i in range(-len(prev_rows), 0):
+        prompt += "\n" + prev_rows[i][0]
+        sampled_news = sample_news(
+            prev_rows[i][1],
+            min(5, len(prev_rows[i][1]))
+        )
+        if sampled_news:
+            prompt += "\n".join(sampled_news)
+        else:
+            prompt += "No relative news reported."
+        
+    period = "{} to {}".format(curday, n_weeks_before(curday, -1))
+    
+    if with_basics:
+        basics = get_current_basics(symbol, curday)
+        basics = "Some recent basic financials of {}, reported at {}, are presented below:\n\n[Basic Financials]:\n\n".format(
+            symbol, basics['period']) + "\n".join(f"{k}: {v}" for k, v in basics.items() if k != 'period')
+    else:
+        basics = "[Basic Financials]:\n\nNo basic financial reported."
+
+    info = company_prompt + '\n' + prompt + '\n' + basics
+    prompt = info + f"\n\nBased on all the information before {curday}, let's first analyze the positive developments and potential concerns for {symbol}. Come up with 2-4 most important factors respectively and keep them concise. Most factors should be inferred from company related news. " \
+        f"Then make your prediction of the {symbol} stock price movement for next week ({period}). Provide a summary analysis to support your prediction."
+        
+    return info, prompt
+
+
+def construct_prompt(ticker, curday, n_weeks, use_basics):
+
+    try:
+        steps = [n_weeks_before(curday, n) for n in range(n_weeks + 1)][::-1]
+    except Exception:
+        raise gr.Error(f"Invalid date {curday}!")
+        
+    data = get_stock_data(ticker, steps)
+    data = get_news(ticker, data)
+    data['Basics'] = [json.dumps({})] * len(data)
+    # print(data)
+    
+    info, prompt = get_all_prompts_online(ticker, data, curday, use_basics)
+    
+    prompt = B_INST + B_SYS + SYSTEM_PROMPT + E_SYS + prompt + E_INST
+    # print(prompt)
+    
+    return info, prompt
+
+
+def predict(ticker, date, n_weeks, use_basics):
+
+    print_gpu_utilization()
+
+    info, prompt = construct_prompt(ticker, date, n_weeks, use_basics)
+      
+    inputs = tokenizer(
+        prompt, return_tensors='pt', padding=False
+    )
+    inputs = {key: value.to(model.device) for key, value in inputs.items()}
+
+    print("Inputs loaded onto devices.")
+        
+    res = model.generate(
+        **inputs, max_length=4096, do_sample=True,
+        eos_token_id=tokenizer.eos_token_id,
+        use_cache=True, streamer=streamer
+    )
+    output = tokenizer.decode(res[0], skip_special_tokens=True)
+    answer = re.sub(r'.*\[/INST\]\s*', '', output, flags=re.DOTALL)
+
+    torch.cuda.empty_cache()
+    
+    return info, answer
+
+
+demo = gr.Interface(
+    predict,
+    inputs=[
+        gr.Textbox(
+            label="Ticker",
+            value="AAPL",
+            info="Companys from Dow-30 are recommended"
+        ),
+        gr.Textbox(
+            label="Date",
+            value=get_curday,
+            info="Date from which the prediction is made, use format yyyy-mm-dd"
+        ),
+        gr.Slider(
+            minimum=1,
+            maximum=4,
+            value=3,
+            step=1,
+            label="n_weeks",
+            info="Information of the past n weeks will be utilized, choose between 1 and 4"
+        ),
+        gr.Checkbox(
+            label="Use Latest Basic Financials",
+            value=False,
+            info="If checked, the latest quarterly reported basic financials of the company is taken into account."
+        )
+    ],
+    outputs=[
+        gr.Textbox(
+            label="Information"
+        ),
+        gr.Textbox(
+            label="Response"
+        )
+    ],
+    title="FinGPT-Forecaster",
+    description="""FinGPT-Forecaster takes random market news and optional basic financials related to the specified company from the past few weeks as input and responds with the company's **positive developments** and **potential concerns**. Then it gives out a **prediction** of stock price movement for the coming week and its **analysis** summary.
+This model is finetuned on Llama2-7b-chat-hf with LoRA on the past year's DOW30 market data. Inference in this demo uses fp16 and **welcomes any ticker symbol**.
+Company profile & Market news & Basic financials & Stock prices are retrieved using **yfinance & finnhub**.
+This is just a demo showing what this model is capable of. Results inferred from randomly chosen news can be strongly biased.
+For more detailed and customized implementation, refer to our FinGPT project: <https://github.com/AI4Finance-Foundation/FinGPT>
+**Disclaimer: Nothing herein is financial advice, and NOT a recommendation to trade real money. Please use common sense and always first consult a professional before trading or investing.**
+"""
+)
+
+demo.launch()

config.json

@@ -0,0 +1,33 @@
+{
+    "train_micro_batch_size_per_gpu": "auto",
+    "train_batch_size": "auto",
+    "gradient_accumulation_steps": "auto",
+    "optimizer": {
+      "type": "ZeroOneAdam",
+      "params": {
+        "lr": "auto",
+        "weight_decay": "auto",
+        "bias_correction": false,
+        "var_freeze_step": 1000,
+        "var_update_scaler": 16,
+        "local_step_scaler": 1000,
+        "local_step_clipper": 16,
+        "cuda_aware": true,
+        "comm_backend_name": "nccl"
+        }
+    },
+    "scheduler": {
+        "type": "WarmupLR",
+        "params": {
+            "warmup_min_lr": 0,
+            "warmup_max_lr": "auto",
+            "warmup_num_steps": "auto"
+        }
+    },
+    "fp16": {
+        "enabled": true
+    },
+    "zero_optimization": {
+        "stage": 0
+    }
+}

prepare_data.ipynb

@@ -0,0 +1,1545 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "id": "3c4d096e",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import re\n",
+    "import csv\n",
+    "import math\n",
+    "import time\n",
+    "import json\n",
+    "import random\n",
+    "import finnhub\n",
+    "import datasets\n",
+    "import pandas as pd\n",
+    "import yfinance as yf\n",
+    "from datetime import datetime\n",
+    "from collections import defaultdict\n",
+    "from datasets import Dataset\n",
+    "from openai import OpenAI"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "id": "ace9fdb4",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "START_DATE = \"2022-12-31\"\n",
+    "END_DATE = \"2023-05-31\"\n",
+    "\n",
+    "DATA_DIR = f\"./{START_DATE}_{END_DATE}\"\n",
+    "os.makedirs(DATA_DIR, exist_ok=True)\n",
+    "\n",
+    "finnhub_client = finnhub.Client(api_key=\"your finnhub key\")\n",
+    "\n",
+    "client = OpenAI(api_key = 'your openai key')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2fce2503",
+   "metadata": {},
+   "source": [
+    "# Raw Financial Data Acquisition"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "id": "c6564114",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def bin_mapping(ret):\n",
+    "    \n",
+    "    up_down = 'U' if ret >= 0 else 'D'\n",
+    "    integer = math.ceil(abs(100 * ret))\n",
+    "    \n",
+    "    return up_down + (str(integer) if integer <= 5 else '5+')\n",
+    "\n",
+    "\n",
+    "def get_returns(stock_symbol):\n",
+    "    \n",
+    "    # Download historical stock data\n",
+    "    stock_data = yf.download(stock_symbol, start=START_DATE, end=END_DATE)\n",
+    "    \n",
+    "    weekly_data = stock_data['Adj Close'].resample('W').ffill()\n",
+    "    weekly_returns = weekly_data.pct_change()[1:]\n",
+    "    weekly_start_prices = weekly_data[:-1]\n",
+    "    weekly_end_prices = weekly_data[1:]\n",
+    "\n",
+    "    weekly_data = pd.DataFrame({\n",
+    "        'Start Date': weekly_start_prices.index,\n",
+    "        'Start Price': weekly_start_prices.values,\n",
+    "        'End Date': weekly_end_prices.index,\n",
+    "        'End Price': weekly_end_prices.values,\n",
+    "        'Weekly Returns': weekly_returns.values\n",
+    "    })\n",
+    "    \n",
+    "    weekly_data['Bin Label'] = weekly_data['Weekly Returns'].map(bin_mapping)\n",
+    "\n",
+    "    return weekly_data\n",
+    "\n",
+    "\n",
+    "def get_news(symbol, data):\n",
+    "    \n",
+    "    news_list = []\n",
+    "    \n",
+    "    for end_date, row in data.iterrows():\n",
+    "        start_date = row['Start Date'].strftime('%Y-%m-%d')\n",
+    "        end_date = row['End Date'].strftime('%Y-%m-%d')\n",
+    "        print(symbol, ': ', start_date, ' - ', end_date)\n",
+    "        time.sleep(1) # control qpm\n",
+    "        weekly_news = finnhub_client.company_news(symbol, _from=start_date, to=end_date)\n",
+    "        weekly_news = [\n",
+    "            {\n",
+    "                \"date\": datetime.fromtimestamp(n['datetime']).strftime('%Y%m%d%H%M%S'),\n",
+    "                \"headline\": n['headline'],\n",
+    "                \"summary\": n['summary'],\n",
+    "            } for n in weekly_news\n",
+    "        ]\n",
+    "        weekly_news.sort(key=lambda x: x['date'])\n",
+    "        news_list.append(json.dumps(weekly_news))\n",
+    "    \n",
+    "    data['News'] = news_list\n",
+    "    \n",
+    "    return data\n",
+    "\n",
+    "\n",
+    "def get_basics(symbol, data, always=False):\n",
+    "    \n",
+    "    basic_financials = finnhub_client.company_basic_financials(symbol, 'all')\n",
+    "    \n",
+    "    final_basics, basic_list, basic_dict = [], [], defaultdict(dict)\n",
+    "    \n",
+    "    for metric, value_list in basic_financials['series']['quarterly'].items():\n",
+    "        for value in value_list:\n",
+    "            basic_dict[value['period']].update({metric: value['v']})\n",
+    "\n",
+    "    for k, v in basic_dict.items():\n",
+    "        v.update({'period': k})\n",
+    "        basic_list.append(v)\n",
+    "        \n",
+    "    basic_list.sort(key=lambda x: x['period'])\n",
+    "            \n",
+    "    for i, row in data.iterrows():\n",
+    "        \n",
+    "        start_date = row['End Date'].strftime('%Y-%m-%d')\n",
+    "        last_start_date = START_DATE if i < 2 else data.loc[i-2, 'Start Date'].strftime('%Y-%m-%d')\n",
+    "        \n",
+    "        used_basic = {}\n",
+    "        for basic in basic_list[::-1]:\n",
+    "            if (always and basic['period'] < start_date) or (last_start_date <= basic['period'] < start_date):\n",
+    "                used_basic = basic\n",
+    "                break\n",
+    "        final_basics.append(json.dumps(used_basic))\n",
+    "        \n",
+    "    data['Basics'] = final_basics\n",
+    "    \n",
+    "    return data\n",
+    "    \n",
+    "\n",
+    "def prepare_data_for_company(symbol, with_basics=True):\n",
+    "    \n",
+    "    data = get_returns(symbol)\n",
+    "    data = get_news(symbol, data)\n",
+    "    \n",
+    "    if with_basics:\n",
+    "        data = get_basics(symbol, data)\n",
+    "        data.to_csv(f\"{DATA_DIR}/{symbol}_{START_DATE}_{END_DATE}.csv\")\n",
+    "    else:\n",
+    "        data['Basics'] = [json.dumps({})] * len(data)\n",
+    "        data.to_csv(f\"{DATA_DIR}/{symbol}_{START_DATE}_{END_DATE}_nobasics.csv\")\n",
+    "    \n",
+    "    return data\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 59,
+   "id": "caf02ab7",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "DOW_30 = [\n",
+    "    \"AXP\", \"AMGN\", \"AAPL\", \"BA\", \"CAT\", \"CSCO\", \"CVX\", \"GS\", \"HD\", \"HON\",\n",
+    "    \"IBM\", \"INTC\", \"JNJ\", \"KO\", \"JPM\", \"MCD\", \"MMM\", \"MRK\", \"MSFT\", \"NKE\",\n",
+    "    \"PG\", \"TRV\", \"UNH\", \"CRM\", \"VZ\", \"V\", \"WBA\", \"WMT\", \"DIS\", \"DOW\"\n",
+    "]\n",
+    "\n",
+    "# prepare_data_for_company(\"DOW\", False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 81,
+   "id": "43d65960",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "AXP :  2023-01-08  -  2023-01-15\n",
+      "AXP :  2023-01-15  -  2023-01-22\n",
+      "AXP :  2023-01-22  -  2023-01-29\n",
+      "AXP :  2023-01-29  -  2023-02-05\n",
+      "AXP :  2023-02-05  -  2023-02-12\n",
+      "AXP :  2023-02-12  -  2023-02-19\n",
+      "AXP :  2023-02-19  -  2023-02-26\n",
+      "AXP :  2023-02-26  -  2023-03-05\n",
+      "AXP :  2023-03-05  -  2023-03-12\n",
+      "AXP :  2023-03-12  -  2023-03-19\n",
+      "AXP :  2023-03-19  -  2023-03-26\n",
+      "AXP :  2023-03-26  -  2023-04-02\n",
+      "AXP :  2023-04-02  -  2023-04-09\n",
+      "AXP :  2023-04-09  -  2023-04-16\n",
+      "AXP :  2023-04-16  -  2023-04-23\n",
+      "AXP :  2023-04-23  -  2023-04-30\n",
+      "AXP :  2023-04-30  -  2023-05-07\n",
+      "AXP :  2023-05-07  -  2023-05-14\n",
+      "AXP :  2023-05-14  -  2023-05-21\n",
+      "AXP :  2023-05-21  -  2023-05-28\n",
+      "AXP :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "AMGN :  2023-01-08  -  2023-01-15\n",
+      "AMGN :  2023-01-15  -  2023-01-22\n",
+      "AMGN :  2023-01-22  -  2023-01-29\n",
+      "AMGN :  2023-01-29  -  2023-02-05\n",
+      "AMGN :  2023-02-05  -  2023-02-12\n",
+      "AMGN :  2023-02-12  -  2023-02-19\n",
+      "AMGN :  2023-02-19  -  2023-02-26\n",
+      "AMGN :  2023-02-26  -  2023-03-05\n",
+      "AMGN :  2023-03-05  -  2023-03-12\n",
+      "AMGN :  2023-03-12  -  2023-03-19\n",
+      "AMGN :  2023-03-19  -  2023-03-26\n",
+      "AMGN :  2023-03-26  -  2023-04-02\n",
+      "AMGN :  2023-04-02  -  2023-04-09\n",
+      "AMGN :  2023-04-09  -  2023-04-16\n",
+      "AMGN :  2023-04-16  -  2023-04-23\n",
+      "AMGN :  2023-04-23  -  2023-04-30\n",
+      "AMGN :  2023-04-30  -  2023-05-07\n",
+      "AMGN :  2023-05-07  -  2023-05-14\n",
+      "AMGN :  2023-05-14  -  2023-05-21\n",
+      "AMGN :  2023-05-21  -  2023-05-28\n",
+      "AMGN :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "AAPL :  2023-01-08  -  2023-01-15\n",
+      "AAPL :  2023-01-15  -  2023-01-22\n",
+      "AAPL :  2023-01-22  -  2023-01-29\n",
+      "AAPL :  2023-01-29  -  2023-02-05\n",
+      "AAPL :  2023-02-05  -  2023-02-12\n",
+      "AAPL :  2023-02-12  -  2023-02-19\n",
+      "AAPL :  2023-02-19  -  2023-02-26\n",
+      "AAPL :  2023-02-26  -  2023-03-05\n",
+      "AAPL :  2023-03-05  -  2023-03-12\n",
+      "AAPL :  2023-03-12  -  2023-03-19\n",
+      "AAPL :  2023-03-19  -  2023-03-26\n",
+      "AAPL :  2023-03-26  -  2023-04-02\n",
+      "AAPL :  2023-04-02  -  2023-04-09\n",
+      "AAPL :  2023-04-09  -  2023-04-16\n",
+      "AAPL :  2023-04-16  -  2023-04-23\n",
+      "AAPL :  2023-04-23  -  2023-04-30\n",
+      "AAPL :  2023-04-30  -  2023-05-07\n",
+      "AAPL :  2023-05-07  -  2023-05-14\n",
+      "AAPL :  2023-05-14  -  2023-05-21\n",
+      "AAPL :  2023-05-21  -  2023-05-28\n",
+      "AAPL :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "BA :  2023-01-08  -  2023-01-15\n",
+      "BA :  2023-01-15  -  2023-01-22\n",
+      "BA :  2023-01-22  -  2023-01-29\n",
+      "BA :  2023-01-29  -  2023-02-05\n",
+      "BA :  2023-02-05  -  2023-02-12\n",
+      "BA :  2023-02-12  -  2023-02-19\n",
+      "BA :  2023-02-19  -  2023-02-26\n",
+      "BA :  2023-02-26  -  2023-03-05\n",
+      "BA :  2023-03-05  -  2023-03-12\n",
+      "BA :  2023-03-12  -  2023-03-19\n",
+      "BA :  2023-03-19  -  2023-03-26\n",
+      "BA :  2023-03-26  -  2023-04-02\n",
+      "BA :  2023-04-02  -  2023-04-09\n",
+      "BA :  2023-04-09  -  2023-04-16\n",
+      "BA :  2023-04-16  -  2023-04-23\n",
+      "BA :  2023-04-23  -  2023-04-30\n",
+      "BA :  2023-04-30  -  2023-05-07\n",
+      "BA :  2023-05-07  -  2023-05-14\n",
+      "BA :  2023-05-14  -  2023-05-21\n",
+      "BA :  2023-05-21  -  2023-05-28\n",
+      "BA :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "CAT :  2023-01-08  -  2023-01-15\n",
+      "CAT :  2023-01-15  -  2023-01-22\n",
+      "CAT :  2023-01-22  -  2023-01-29\n",
+      "CAT :  2023-01-29  -  2023-02-05\n",
+      "CAT :  2023-02-05  -  2023-02-12\n",
+      "CAT :  2023-02-12  -  2023-02-19\n",
+      "CAT :  2023-02-19  -  2023-02-26\n",
+      "CAT :  2023-02-26  -  2023-03-05\n",
+      "CAT :  2023-03-05  -  2023-03-12\n",
+      "CAT :  2023-03-12  -  2023-03-19\n",
+      "CAT :  2023-03-19  -  2023-03-26\n",
+      "CAT :  2023-03-26  -  2023-04-02\n",
+      "CAT :  2023-04-02  -  2023-04-09\n",
+      "CAT :  2023-04-09  -  2023-04-16\n",
+      "CAT :  2023-04-16  -  2023-04-23\n",
+      "CAT :  2023-04-23  -  2023-04-30\n",
+      "CAT :  2023-04-30  -  2023-05-07\n",
+      "CAT :  2023-05-07  -  2023-05-14\n",
+      "CAT :  2023-05-14  -  2023-05-21\n",
+      "CAT :  2023-05-21  -  2023-05-28\n",
+      "CAT :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "CSCO :  2023-01-08  -  2023-01-15\n",
+      "CSCO :  2023-01-15  -  2023-01-22\n",
+      "CSCO :  2023-01-22  -  2023-01-29\n",
+      "CSCO :  2023-01-29  -  2023-02-05\n",
+      "CSCO :  2023-02-05  -  2023-02-12\n",
+      "CSCO :  2023-02-12  -  2023-02-19\n",
+      "CSCO :  2023-02-19  -  2023-02-26\n",
+      "CSCO :  2023-02-26  -  2023-03-05\n",
+      "CSCO :  2023-03-05  -  2023-03-12\n",
+      "CSCO :  2023-03-12  -  2023-03-19\n",
+      "CSCO :  2023-03-19  -  2023-03-26\n",
+      "CSCO :  2023-03-26  -  2023-04-02\n",
+      "CSCO :  2023-04-02  -  2023-04-09\n",
+      "CSCO :  2023-04-09  -  2023-04-16\n",
+      "CSCO :  2023-04-16  -  2023-04-23\n",
+      "CSCO :  2023-04-23  -  2023-04-30\n",
+      "CSCO :  2023-04-30  -  2023-05-07\n",
+      "CSCO :  2023-05-07  -  2023-05-14\n",
+      "CSCO :  2023-05-14  -  2023-05-21\n",
+      "CSCO :  2023-05-21  -  2023-05-28\n",
+      "CSCO :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "CVX :  2023-01-08  -  2023-01-15\n",
+      "CVX :  2023-01-15  -  2023-01-22\n",
+      "CVX :  2023-01-22  -  2023-01-29\n",
+      "CVX :  2023-01-29  -  2023-02-05\n",
+      "CVX :  2023-02-05  -  2023-02-12\n",
+      "CVX :  2023-02-12  -  2023-02-19\n",
+      "CVX :  2023-02-19  -  2023-02-26\n",
+      "CVX :  2023-02-26  -  2023-03-05\n",
+      "CVX :  2023-03-05  -  2023-03-12\n",
+      "CVX :  2023-03-12  -  2023-03-19\n",
+      "CVX :  2023-03-19  -  2023-03-26\n",
+      "CVX :  2023-03-26  -  2023-04-02\n",
+      "CVX :  2023-04-02  -  2023-04-09\n",
+      "CVX :  2023-04-09  -  2023-04-16\n",
+      "CVX :  2023-04-16  -  2023-04-23\n",
+      "CVX :  2023-04-23  -  2023-04-30\n",
+      "CVX :  2023-04-30  -  2023-05-07\n",
+      "CVX :  2023-05-07  -  2023-05-14\n",
+      "CVX :  2023-05-14  -  2023-05-21\n",
+      "CVX :  2023-05-21  -  2023-05-28\n",
+      "CVX :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "GS :  2023-01-08  -  2023-01-15\n",
+      "GS :  2023-01-15  -  2023-01-22\n",
+      "GS :  2023-01-22  -  2023-01-29\n",
+      "GS :  2023-01-29  -  2023-02-05\n",
+      "GS :  2023-02-05  -  2023-02-12\n",
+      "GS :  2023-02-12  -  2023-02-19\n",
+      "GS :  2023-02-19  -  2023-02-26\n",
+      "GS :  2023-02-26  -  2023-03-05\n",
+      "GS :  2023-03-05  -  2023-03-12\n",
+      "GS :  2023-03-12  -  2023-03-19\n",
+      "GS :  2023-03-19  -  2023-03-26\n",
+      "GS :  2023-03-26  -  2023-04-02\n",
+      "GS :  2023-04-02  -  2023-04-09\n",
+      "GS :  2023-04-09  -  2023-04-16\n",
+      "GS :  2023-04-16  -  2023-04-23\n",
+      "GS :  2023-04-23  -  2023-04-30\n",
+      "GS :  2023-04-30  -  2023-05-07\n",
+      "GS :  2023-05-07  -  2023-05-14\n",
+      "GS :  2023-05-14  -  2023-05-21\n",
+      "GS :  2023-05-21  -  2023-05-28\n",
+      "GS :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "HD :  2023-01-08  -  2023-01-15\n",
+      "HD :  2023-01-15  -  2023-01-22\n",
+      "HD :  2023-01-22  -  2023-01-29\n",
+      "HD :  2023-01-29  -  2023-02-05\n",
+      "HD :  2023-02-05  -  2023-02-12\n",
+      "HD :  2023-02-12  -  2023-02-19\n",
+      "HD :  2023-02-19  -  2023-02-26\n",
+      "HD :  2023-02-26  -  2023-03-05\n",
+      "HD :  2023-03-05  -  2023-03-12\n",
+      "HD :  2023-03-12  -  2023-03-19\n",
+      "HD :  2023-03-19  -  2023-03-26\n",
+      "HD :  2023-03-26  -  2023-04-02\n",
+      "HD :  2023-04-02  -  2023-04-09\n",
+      "HD :  2023-04-09  -  2023-04-16\n",
+      "HD :  2023-04-16  -  2023-04-23\n",
+      "HD :  2023-04-23  -  2023-04-30\n",
+      "HD :  2023-04-30  -  2023-05-07\n",
+      "HD :  2023-05-07  -  2023-05-14\n",
+      "HD :  2023-05-14  -  2023-05-21\n",
+      "HD :  2023-05-21  -  2023-05-28\n",
+      "HD :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "HON :  2023-01-08  -  2023-01-15\n",
+      "HON :  2023-01-15  -  2023-01-22\n",
+      "HON :  2023-01-22  -  2023-01-29\n",
+      "HON :  2023-01-29  -  2023-02-05\n",
+      "HON :  2023-02-05  -  2023-02-12\n",
+      "HON :  2023-02-12  -  2023-02-19\n",
+      "HON :  2023-02-19  -  2023-02-26\n",
+      "HON :  2023-02-26  -  2023-03-05\n",
+      "HON :  2023-03-05  -  2023-03-12\n",
+      "HON :  2023-03-12  -  2023-03-19\n",
+      "HON :  2023-03-19  -  2023-03-26\n",
+      "HON :  2023-03-26  -  2023-04-02\n",
+      "HON :  2023-04-02  -  2023-04-09\n",
+      "HON :  2023-04-09  -  2023-04-16\n",
+      "HON :  2023-04-16  -  2023-04-23\n",
+      "HON :  2023-04-23  -  2023-04-30\n",
+      "HON :  2023-04-30  -  2023-05-07\n",
+      "HON :  2023-05-07  -  2023-05-14\n",
+      "HON :  2023-05-14  -  2023-05-21\n",
+      "HON :  2023-05-21  -  2023-05-28\n",
+      "HON :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "IBM :  2023-01-08  -  2023-01-15\n",
+      "IBM :  2023-01-15  -  2023-01-22\n",
+      "IBM :  2023-01-22  -  2023-01-29\n",
+      "IBM :  2023-01-29  -  2023-02-05\n",
+      "IBM :  2023-02-05  -  2023-02-12\n",
+      "IBM :  2023-02-12  -  2023-02-19\n",
+      "IBM :  2023-02-19  -  2023-02-26\n",
+      "IBM :  2023-02-26  -  2023-03-05\n",
+      "IBM :  2023-03-05  -  2023-03-12\n",
+      "IBM :  2023-03-12  -  2023-03-19\n",
+      "IBM :  2023-03-19  -  2023-03-26\n",
+      "IBM :  2023-03-26  -  2023-04-02\n",
+      "IBM :  2023-04-02  -  2023-04-09\n",
+      "IBM :  2023-04-09  -  2023-04-16\n",
+      "IBM :  2023-04-16  -  2023-04-23\n",
+      "IBM :  2023-04-23  -  2023-04-30\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "IBM :  2023-04-30  -  2023-05-07\n",
+      "IBM :  2023-05-07  -  2023-05-14\n",
+      "IBM :  2023-05-14  -  2023-05-21\n",
+      "IBM :  2023-05-21  -  2023-05-28\n",
+      "IBM :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "INTC :  2023-01-08  -  2023-01-15\n",
+      "INTC :  2023-01-15  -  2023-01-22\n",
+      "INTC :  2023-01-22  -  2023-01-29\n",
+      "INTC :  2023-01-29  -  2023-02-05\n",
+      "INTC :  2023-02-05  -  2023-02-12\n",
+      "INTC :  2023-02-12  -  2023-02-19\n",
+      "INTC :  2023-02-19  -  2023-02-26\n",
+      "INTC :  2023-02-26  -  2023-03-05\n",
+      "INTC :  2023-03-05  -  2023-03-12\n",
+      "INTC :  2023-03-12  -  2023-03-19\n",
+      "INTC :  2023-03-19  -  2023-03-26\n",
+      "INTC :  2023-03-26  -  2023-04-02\n",
+      "INTC :  2023-04-02  -  2023-04-09\n",
+      "INTC :  2023-04-09  -  2023-04-16\n",
+      "INTC :  2023-04-16  -  2023-04-23\n",
+      "INTC :  2023-04-23  -  2023-04-30\n",
+      "INTC :  2023-04-30  -  2023-05-07\n",
+      "INTC :  2023-05-07  -  2023-05-14\n",
+      "INTC :  2023-05-14  -  2023-05-21\n",
+      "INTC :  2023-05-21  -  2023-05-28\n",
+      "INTC :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "JNJ :  2023-01-08  -  2023-01-15\n",
+      "JNJ :  2023-01-15  -  2023-01-22\n",
+      "JNJ :  2023-01-22  -  2023-01-29\n",
+      "JNJ :  2023-01-29  -  2023-02-05\n",
+      "JNJ :  2023-02-05  -  2023-02-12\n",
+      "JNJ :  2023-02-12  -  2023-02-19\n",
+      "JNJ :  2023-02-19  -  2023-02-26\n",
+      "JNJ :  2023-02-26  -  2023-03-05\n",
+      "JNJ :  2023-03-05  -  2023-03-12\n",
+      "JNJ :  2023-03-12  -  2023-03-19\n",
+      "JNJ :  2023-03-19  -  2023-03-26\n",
+      "JNJ :  2023-03-26  -  2023-04-02\n",
+      "JNJ :  2023-04-02  -  2023-04-09\n",
+      "JNJ :  2023-04-09  -  2023-04-16\n",
+      "JNJ :  2023-04-16  -  2023-04-23\n",
+      "JNJ :  2023-04-23  -  2023-04-30\n",
+      "JNJ :  2023-04-30  -  2023-05-07\n",
+      "JNJ :  2023-05-07  -  2023-05-14\n",
+      "JNJ :  2023-05-14  -  2023-05-21\n",
+      "JNJ :  2023-05-21  -  2023-05-28\n",
+      "JNJ :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "KO :  2023-01-08  -  2023-01-15\n",
+      "KO :  2023-01-15  -  2023-01-22\n",
+      "KO :  2023-01-22  -  2023-01-29\n",
+      "KO :  2023-01-29  -  2023-02-05\n",
+      "KO :  2023-02-05  -  2023-02-12\n",
+      "KO :  2023-02-12  -  2023-02-19\n",
+      "KO :  2023-02-19  -  2023-02-26\n",
+      "KO :  2023-02-26  -  2023-03-05\n",
+      "KO :  2023-03-05  -  2023-03-12\n",
+      "KO :  2023-03-12  -  2023-03-19\n",
+      "KO :  2023-03-19  -  2023-03-26\n",
+      "KO :  2023-03-26  -  2023-04-02\n",
+      "KO :  2023-04-02  -  2023-04-09\n",
+      "KO :  2023-04-09  -  2023-04-16\n",
+      "KO :  2023-04-16  -  2023-04-23\n",
+      "KO :  2023-04-23  -  2023-04-30\n",
+      "KO :  2023-04-30  -  2023-05-07\n",
+      "KO :  2023-05-07  -  2023-05-14\n",
+      "KO :  2023-05-14  -  2023-05-21\n",
+      "KO :  2023-05-21  -  2023-05-28\n",
+      "KO :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "JPM :  2023-01-08  -  2023-01-15\n",
+      "JPM :  2023-01-15  -  2023-01-22\n",
+      "JPM :  2023-01-22  -  2023-01-29\n",
+      "JPM :  2023-01-29  -  2023-02-05\n",
+      "JPM :  2023-02-05  -  2023-02-12\n",
+      "JPM :  2023-02-12  -  2023-02-19\n",
+      "JPM :  2023-02-19  -  2023-02-26\n",
+      "JPM :  2023-02-26  -  2023-03-05\n",
+      "JPM :  2023-03-05  -  2023-03-12\n",
+      "JPM :  2023-03-12  -  2023-03-19\n",
+      "JPM :  2023-03-19  -  2023-03-26\n",
+      "JPM :  2023-03-26  -  2023-04-02\n",
+      "JPM :  2023-04-02  -  2023-04-09\n",
+      "JPM :  2023-04-09  -  2023-04-16\n",
+      "JPM :  2023-04-16  -  2023-04-23\n",
+      "JPM :  2023-04-23  -  2023-04-30\n",
+      "JPM :  2023-04-30  -  2023-05-07\n",
+      "JPM :  2023-05-07  -  2023-05-14\n",
+      "JPM :  2023-05-14  -  2023-05-21\n",
+      "JPM :  2023-05-21  -  2023-05-28\n",
+      "JPM :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "MCD :  2023-01-08  -  2023-01-15\n",
+      "MCD :  2023-01-15  -  2023-01-22\n",
+      "MCD :  2023-01-22  -  2023-01-29\n",
+      "MCD :  2023-01-29  -  2023-02-05\n",
+      "MCD :  2023-02-05  -  2023-02-12\n",
+      "MCD :  2023-02-12  -  2023-02-19\n",
+      "MCD :  2023-02-19  -  2023-02-26\n",
+      "MCD :  2023-02-26  -  2023-03-05\n",
+      "MCD :  2023-03-05  -  2023-03-12\n",
+      "MCD :  2023-03-12  -  2023-03-19\n",
+      "MCD :  2023-03-19  -  2023-03-26\n",
+      "MCD :  2023-03-26  -  2023-04-02\n",
+      "MCD :  2023-04-02  -  2023-04-09\n",
+      "MCD :  2023-04-09  -  2023-04-16\n",
+      "MCD :  2023-04-16  -  2023-04-23\n",
+      "MCD :  2023-04-23  -  2023-04-30\n",
+      "MCD :  2023-04-30  -  2023-05-07\n",
+      "MCD :  2023-05-07  -  2023-05-14\n",
+      "MCD :  2023-05-14  -  2023-05-21\n",
+      "MCD :  2023-05-21  -  2023-05-28\n",
+      "MCD :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "MMM :  2023-01-08  -  2023-01-15\n",
+      "MMM :  2023-01-15  -  2023-01-22\n",
+      "MMM :  2023-01-22  -  2023-01-29\n",
+      "MMM :  2023-01-29  -  2023-02-05\n",
+      "MMM :  2023-02-05  -  2023-02-12\n",
+      "MMM :  2023-02-12  -  2023-02-19\n",
+      "MMM :  2023-02-19  -  2023-02-26\n",
+      "MMM :  2023-02-26  -  2023-03-05\n",
+      "MMM :  2023-03-05  -  2023-03-12\n",
+      "MMM :  2023-03-12  -  2023-03-19\n",
+      "MMM :  2023-03-19  -  2023-03-26\n",
+      "MMM :  2023-03-26  -  2023-04-02\n",
+      "MMM :  2023-04-02  -  2023-04-09\n",
+      "MMM :  2023-04-09  -  2023-04-16\n",
+      "MMM :  2023-04-16  -  2023-04-23\n",
+      "MMM :  2023-04-23  -  2023-04-30\n",
+      "MMM :  2023-04-30  -  2023-05-07\n",
+      "MMM :  2023-05-07  -  2023-05-14\n",
+      "MMM :  2023-05-14  -  2023-05-21\n",
+      "MMM :  2023-05-21  -  2023-05-28\n",
+      "MMM :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "MRK :  2023-01-08  -  2023-01-15\n",
+      "MRK :  2023-01-15  -  2023-01-22\n",
+      "MRK :  2023-01-22  -  2023-01-29\n",
+      "MRK :  2023-01-29  -  2023-02-05\n",
+      "MRK :  2023-02-05  -  2023-02-12\n",
+      "MRK :  2023-02-12  -  2023-02-19\n",
+      "MRK :  2023-02-19  -  2023-02-26\n",
+      "MRK :  2023-02-26  -  2023-03-05\n",
+      "MRK :  2023-03-05  -  2023-03-12\n",
+      "MRK :  2023-03-12  -  2023-03-19\n",
+      "MRK :  2023-03-19  -  2023-03-26\n",
+      "MRK :  2023-03-26  -  2023-04-02\n",
+      "MRK :  2023-04-02  -  2023-04-09\n",
+      "MRK :  2023-04-09  -  2023-04-16\n",
+      "MRK :  2023-04-16  -  2023-04-23\n",
+      "MRK :  2023-04-23  -  2023-04-30\n",
+      "MRK :  2023-04-30  -  2023-05-07\n",
+      "MRK :  2023-05-07  -  2023-05-14\n",
+      "MRK :  2023-05-14  -  2023-05-21\n",
+      "MRK :  2023-05-21  -  2023-05-28\n",
+      "MRK :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "MSFT :  2023-01-08  -  2023-01-15\n",
+      "MSFT :  2023-01-15  -  2023-01-22\n",
+      "MSFT :  2023-01-22  -  2023-01-29\n",
+      "MSFT :  2023-01-29  -  2023-02-05\n",
+      "MSFT :  2023-02-05  -  2023-02-12\n",
+      "MSFT :  2023-02-12  -  2023-02-19\n",
+      "MSFT :  2023-02-19  -  2023-02-26\n",
+      "MSFT :  2023-02-26  -  2023-03-05\n",
+      "MSFT :  2023-03-05  -  2023-03-12\n",
+      "MSFT :  2023-03-12  -  2023-03-19\n",
+      "MSFT :  2023-03-19  -  2023-03-26\n",
+      "MSFT :  2023-03-26  -  2023-04-02\n",
+      "MSFT :  2023-04-02  -  2023-04-09\n",
+      "MSFT :  2023-04-09  -  2023-04-16\n",
+      "MSFT :  2023-04-16  -  2023-04-23\n",
+      "MSFT :  2023-04-23  -  2023-04-30\n",
+      "MSFT :  2023-04-30  -  2023-05-07\n",
+      "MSFT :  2023-05-07  -  2023-05-14\n",
+      "MSFT :  2023-05-14  -  2023-05-21\n",
+      "MSFT :  2023-05-21  -  2023-05-28\n",
+      "MSFT :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "NKE :  2023-01-08  -  2023-01-15\n",
+      "NKE :  2023-01-15  -  2023-01-22\n",
+      "NKE :  2023-01-22  -  2023-01-29\n",
+      "NKE :  2023-01-29  -  2023-02-05\n",
+      "NKE :  2023-02-05  -  2023-02-12\n",
+      "NKE :  2023-02-12  -  2023-02-19\n",
+      "NKE :  2023-02-19  -  2023-02-26\n",
+      "NKE :  2023-02-26  -  2023-03-05\n",
+      "NKE :  2023-03-05  -  2023-03-12\n",
+      "NKE :  2023-03-12  -  2023-03-19\n",
+      "NKE :  2023-03-19  -  2023-03-26\n",
+      "NKE :  2023-03-26  -  2023-04-02\n",
+      "NKE :  2023-04-02  -  2023-04-09\n",
+      "NKE :  2023-04-09  -  2023-04-16\n",
+      "NKE :  2023-04-16  -  2023-04-23\n",
+      "NKE :  2023-04-23  -  2023-04-30\n",
+      "NKE :  2023-04-30  -  2023-05-07\n",
+      "NKE :  2023-05-07  -  2023-05-14\n",
+      "NKE :  2023-05-14  -  2023-05-21\n",
+      "NKE :  2023-05-21  -  2023-05-28\n",
+      "NKE :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "PG :  2023-01-08  -  2023-01-15\n",
+      "PG :  2023-01-15  -  2023-01-22\n",
+      "PG :  2023-01-22  -  2023-01-29\n",
+      "PG :  2023-01-29  -  2023-02-05\n",
+      "PG :  2023-02-05  -  2023-02-12\n",
+      "PG :  2023-02-12  -  2023-02-19\n",
+      "PG :  2023-02-19  -  2023-02-26\n",
+      "PG :  2023-02-26  -  2023-03-05\n",
+      "PG :  2023-03-05  -  2023-03-12\n",
+      "PG :  2023-03-12  -  2023-03-19\n",
+      "PG :  2023-03-19  -  2023-03-26\n",
+      "PG :  2023-03-26  -  2023-04-02\n",
+      "PG :  2023-04-02  -  2023-04-09\n",
+      "PG :  2023-04-09  -  2023-04-16\n",
+      "PG :  2023-04-16  -  2023-04-23\n",
+      "PG :  2023-04-23  -  2023-04-30\n",
+      "PG :  2023-04-30  -  2023-05-07\n",
+      "PG :  2023-05-07  -  2023-05-14\n",
+      "PG :  2023-05-14  -  2023-05-21\n",
+      "PG :  2023-05-21  -  2023-05-28\n",
+      "PG :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "TRV :  2023-01-08  -  2023-01-15\n",
+      "TRV :  2023-01-15  -  2023-01-22\n",
+      "TRV :  2023-01-22  -  2023-01-29\n",
+      "TRV :  2023-01-29  -  2023-02-05\n",
+      "TRV :  2023-02-05  -  2023-02-12\n",
+      "TRV :  2023-02-12  -  2023-02-19\n",
+      "TRV :  2023-02-19  -  2023-02-26\n",
+      "TRV :  2023-02-26  -  2023-03-05\n",
+      "TRV :  2023-03-05  -  2023-03-12\n",
+      "TRV :  2023-03-12  -  2023-03-19\n",
+      "TRV :  2023-03-19  -  2023-03-26\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "TRV :  2023-03-26  -  2023-04-02\n",
+      "TRV :  2023-04-02  -  2023-04-09\n",
+      "TRV :  2023-04-09  -  2023-04-16\n",
+      "TRV :  2023-04-16  -  2023-04-23\n",
+      "TRV :  2023-04-23  -  2023-04-30\n",
+      "TRV :  2023-04-30  -  2023-05-07\n",
+      "TRV :  2023-05-07  -  2023-05-14\n",
+      "TRV :  2023-05-14  -  2023-05-21\n",
+      "TRV :  2023-05-21  -  2023-05-28\n",
+      "TRV :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "UNH :  2023-01-08  -  2023-01-15\n",
+      "UNH :  2023-01-15  -  2023-01-22\n",
+      "UNH :  2023-01-22  -  2023-01-29\n",
+      "UNH :  2023-01-29  -  2023-02-05\n",
+      "UNH :  2023-02-05  -  2023-02-12\n",
+      "UNH :  2023-02-12  -  2023-02-19\n",
+      "UNH :  2023-02-19  -  2023-02-26\n",
+      "UNH :  2023-02-26  -  2023-03-05\n",
+      "UNH :  2023-03-05  -  2023-03-12\n",
+      "UNH :  2023-03-12  -  2023-03-19\n",
+      "UNH :  2023-03-19  -  2023-03-26\n",
+      "UNH :  2023-03-26  -  2023-04-02\n",
+      "UNH :  2023-04-02  -  2023-04-09\n",
+      "UNH :  2023-04-09  -  2023-04-16\n",
+      "UNH :  2023-04-16  -  2023-04-23\n",
+      "UNH :  2023-04-23  -  2023-04-30\n",
+      "UNH :  2023-04-30  -  2023-05-07\n",
+      "UNH :  2023-05-07  -  2023-05-14\n",
+      "UNH :  2023-05-14  -  2023-05-21\n",
+      "UNH :  2023-05-21  -  2023-05-28\n",
+      "UNH :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "CRM :  2023-01-08  -  2023-01-15\n",
+      "CRM :  2023-01-15  -  2023-01-22\n",
+      "CRM :  2023-01-22  -  2023-01-29\n",
+      "CRM :  2023-01-29  -  2023-02-05\n",
+      "CRM :  2023-02-05  -  2023-02-12\n",
+      "CRM :  2023-02-12  -  2023-02-19\n",
+      "CRM :  2023-02-19  -  2023-02-26\n",
+      "CRM :  2023-02-26  -  2023-03-05\n",
+      "CRM :  2023-03-05  -  2023-03-12\n",
+      "CRM :  2023-03-12  -  2023-03-19\n",
+      "CRM :  2023-03-19  -  2023-03-26\n",
+      "CRM :  2023-03-26  -  2023-04-02\n",
+      "CRM :  2023-04-02  -  2023-04-09\n",
+      "CRM :  2023-04-09  -  2023-04-16\n",
+      "CRM :  2023-04-16  -  2023-04-23\n",
+      "CRM :  2023-04-23  -  2023-04-30\n",
+      "CRM :  2023-04-30  -  2023-05-07\n",
+      "CRM :  2023-05-07  -  2023-05-14\n",
+      "CRM :  2023-05-14  -  2023-05-21\n",
+      "CRM :  2023-05-21  -  2023-05-28\n",
+      "CRM :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "VZ :  2023-01-08  -  2023-01-15\n",
+      "VZ :  2023-01-15  -  2023-01-22\n",
+      "VZ :  2023-01-22  -  2023-01-29\n",
+      "VZ :  2023-01-29  -  2023-02-05\n",
+      "VZ :  2023-02-05  -  2023-02-12\n",
+      "VZ :  2023-02-12  -  2023-02-19\n",
+      "VZ :  2023-02-19  -  2023-02-26\n",
+      "VZ :  2023-02-26  -  2023-03-05\n",
+      "VZ :  2023-03-05  -  2023-03-12\n",
+      "VZ :  2023-03-12  -  2023-03-19\n",
+      "VZ :  2023-03-19  -  2023-03-26\n",
+      "VZ :  2023-03-26  -  2023-04-02\n",
+      "VZ :  2023-04-02  -  2023-04-09\n",
+      "VZ :  2023-04-09  -  2023-04-16\n",
+      "VZ :  2023-04-16  -  2023-04-23\n",
+      "VZ :  2023-04-23  -  2023-04-30\n",
+      "VZ :  2023-04-30  -  2023-05-07\n",
+      "VZ :  2023-05-07  -  2023-05-14\n",
+      "VZ :  2023-05-14  -  2023-05-21\n",
+      "VZ :  2023-05-21  -  2023-05-28\n",
+      "VZ :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "V :  2023-01-08  -  2023-01-15\n",
+      "V :  2023-01-15  -  2023-01-22\n",
+      "V :  2023-01-22  -  2023-01-29\n",
+      "V :  2023-01-29  -  2023-02-05\n",
+      "V :  2023-02-05  -  2023-02-12\n",
+      "V :  2023-02-12  -  2023-02-19\n",
+      "V :  2023-02-19  -  2023-02-26\n",
+      "V :  2023-02-26  -  2023-03-05\n",
+      "V :  2023-03-05  -  2023-03-12\n",
+      "V :  2023-03-12  -  2023-03-19\n",
+      "V :  2023-03-19  -  2023-03-26\n",
+      "V :  2023-03-26  -  2023-04-02\n",
+      "V :  2023-04-02  -  2023-04-09\n",
+      "V :  2023-04-09  -  2023-04-16\n",
+      "V :  2023-04-16  -  2023-04-23\n",
+      "V :  2023-04-23  -  2023-04-30\n",
+      "V :  2023-04-30  -  2023-05-07\n",
+      "V :  2023-05-07  -  2023-05-14\n",
+      "V :  2023-05-14  -  2023-05-21\n",
+      "V :  2023-05-21  -  2023-05-28\n",
+      "V :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "WBA :  2023-01-08  -  2023-01-15\n",
+      "WBA :  2023-01-15  -  2023-01-22\n",
+      "WBA :  2023-01-22  -  2023-01-29\n",
+      "WBA :  2023-01-29  -  2023-02-05\n",
+      "WBA :  2023-02-05  -  2023-02-12\n",
+      "WBA :  2023-02-12  -  2023-02-19\n",
+      "WBA :  2023-02-19  -  2023-02-26\n",
+      "WBA :  2023-02-26  -  2023-03-05\n",
+      "WBA :  2023-03-05  -  2023-03-12\n",
+      "WBA :  2023-03-12  -  2023-03-19\n",
+      "WBA :  2023-03-19  -  2023-03-26\n",
+      "WBA :  2023-03-26  -  2023-04-02\n",
+      "WBA :  2023-04-02  -  2023-04-09\n",
+      "WBA :  2023-04-09  -  2023-04-16\n",
+      "WBA :  2023-04-16  -  2023-04-23\n",
+      "WBA :  2023-04-23  -  2023-04-30\n",
+      "WBA :  2023-04-30  -  2023-05-07\n",
+      "WBA :  2023-05-07  -  2023-05-14\n",
+      "WBA :  2023-05-14  -  2023-05-21\n",
+      "WBA :  2023-05-21  -  2023-05-28\n",
+      "WBA :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "WMT :  2023-01-08  -  2023-01-15\n",
+      "WMT :  2023-01-15  -  2023-01-22\n",
+      "WMT :  2023-01-22  -  2023-01-29\n",
+      "WMT :  2023-01-29  -  2023-02-05\n",
+      "WMT :  2023-02-05  -  2023-02-12\n",
+      "WMT :  2023-02-12  -  2023-02-19\n",
+      "WMT :  2023-02-19  -  2023-02-26\n",
+      "WMT :  2023-02-26  -  2023-03-05\n",
+      "WMT :  2023-03-05  -  2023-03-12\n",
+      "WMT :  2023-03-12  -  2023-03-19\n",
+      "WMT :  2023-03-19  -  2023-03-26\n",
+      "WMT :  2023-03-26  -  2023-04-02\n",
+      "WMT :  2023-04-02  -  2023-04-09\n",
+      "WMT :  2023-04-09  -  2023-04-16\n",
+      "WMT :  2023-04-16  -  2023-04-23\n",
+      "WMT :  2023-04-23  -  2023-04-30\n",
+      "WMT :  2023-04-30  -  2023-05-07\n",
+      "WMT :  2023-05-07  -  2023-05-14\n",
+      "WMT :  2023-05-14  -  2023-05-21\n",
+      "WMT :  2023-05-21  -  2023-05-28\n",
+      "WMT :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "DIS :  2023-01-08  -  2023-01-15\n",
+      "DIS :  2023-01-15  -  2023-01-22\n",
+      "DIS :  2023-01-22  -  2023-01-29\n",
+      "DIS :  2023-01-29  -  2023-02-05\n",
+      "DIS :  2023-02-05  -  2023-02-12\n",
+      "DIS :  2023-02-12  -  2023-02-19\n",
+      "DIS :  2023-02-19  -  2023-02-26\n",
+      "DIS :  2023-02-26  -  2023-03-05\n",
+      "DIS :  2023-03-05  -  2023-03-12\n",
+      "DIS :  2023-03-12  -  2023-03-19\n",
+      "DIS :  2023-03-19  -  2023-03-26\n",
+      "DIS :  2023-03-26  -  2023-04-02\n",
+      "DIS :  2023-04-02  -  2023-04-09\n",
+      "DIS :  2023-04-09  -  2023-04-16\n",
+      "DIS :  2023-04-16  -  2023-04-23\n",
+      "DIS :  2023-04-23  -  2023-04-30\n",
+      "DIS :  2023-04-30  -  2023-05-07\n",
+      "DIS :  2023-05-07  -  2023-05-14\n",
+      "DIS :  2023-05-14  -  2023-05-21\n",
+      "DIS :  2023-05-21  -  2023-05-28\n",
+      "DIS :  2023-05-28  -  2023-06-04\n",
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "DOW :  2023-01-08  -  2023-01-15\n",
+      "DOW :  2023-01-15  -  2023-01-22\n",
+      "DOW :  2023-01-22  -  2023-01-29\n",
+      "DOW :  2023-01-29  -  2023-02-05\n",
+      "DOW :  2023-02-05  -  2023-02-12\n",
+      "DOW :  2023-02-12  -  2023-02-19\n",
+      "DOW :  2023-02-19  -  2023-02-26\n",
+      "DOW :  2023-02-26  -  2023-03-05\n",
+      "DOW :  2023-03-05  -  2023-03-12\n",
+      "DOW :  2023-03-12  -  2023-03-19\n",
+      "DOW :  2023-03-19  -  2023-03-26\n",
+      "DOW :  2023-03-26  -  2023-04-02\n",
+      "DOW :  2023-04-02  -  2023-04-09\n",
+      "DOW :  2023-04-09  -  2023-04-16\n",
+      "DOW :  2023-04-16  -  2023-04-23\n",
+      "DOW :  2023-04-23  -  2023-04-30\n",
+      "DOW :  2023-04-30  -  2023-05-07\n",
+      "DOW :  2023-05-07  -  2023-05-14\n",
+      "DOW :  2023-05-14  -  2023-05-21\n",
+      "DOW :  2023-05-21  -  2023-05-28\n",
+      "DOW :  2023-05-28  -  2023-06-04\n"
+     ]
+    }
+   ],
+   "source": [
+    "for symbol in DOW_30:\n",
+    "    prepare_data_for_company(symbol)\n",
+    "#     prepare_data_for_company(symbol, False)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "af655d8b",
+   "metadata": {},
+   "source": [
+    "# Generate Prompt from Financial Data"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 65,
+   "id": "5a53c0ae",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "def get_company_prompt(symbol):\n",
+    "    \n",
+    "    profile = finnhub_client.company_profile2(symbol=symbol)\n",
+    "\n",
+    "    company_template = \"[Company Introduction]:\\n\\n{name} is a leading entity in the {finnhubIndustry} sector. Incorporated and publicly traded since {ipo}, the company has established its reputation as one of the key players in the market. As of today, {name} has a market capitalization of {marketCapitalization:.2f} in {currency}, with {shareOutstanding:.2f} shares outstanding.\" \\\n",
+    "        \"\\n\\n{name} operates primarily in the {country}, trading under the ticker {ticker} on the {exchange}. As a dominant force in the {finnhubIndustry} space, the company continues to innovate and drive progress within the industry.\"\n",
+    "\n",
+    "    formatted_str = company_template.format(**profile)\n",
+    "    \n",
+    "    return formatted_str\n",
+    "\n",
+    "\n",
+    "def get_prompt_by_row(symbol, row):\n",
+    "\n",
+    "    start_date = row['Start Date'] if isinstance(row['Start Date'], str) else row['Start Date'].strftime('%Y-%m-%d')\n",
+    "    end_date = row['End Date'] if isinstance(row['End Date'], str) else row['End Date'].strftime('%Y-%m-%d')\n",
+    "    term = 'increased' if row['End Price'] > row['Start Price'] else 'decreased'\n",
+    "    head = \"From {} to {}, {}'s stock price {} from {:.2f} to {:.2f}. Company news during this period are listed below:\\n\\n\".format(\n",
+    "        start_date, end_date, symbol, term, row['Start Price'], row['End Price'])\n",
+    "    \n",
+    "    news = json.loads(row[\"News\"])\n",
+    "    news = [\"[Headline]: {}\\n[Summary]: {}\\n\".format(\n",
+    "        n['headline'], n['summary']) for n in news if n['date'][:8] <= end_date.replace('-', '') and \\\n",
+    "        not n['summary'].startswith(\"Looking for stock market analysis and research with proves results?\")]\n",
+    "\n",
+    "    basics = json.loads(row['Basics'])\n",
+    "    if basics:\n",
+    "        basics = \"Some recent basic financials of {}, reported at {}, are presented below:\\n\\n[Basic Financials]:\\n\\n\".format(\n",
+    "            symbol, basics['period']) + \"\\n\".join(f\"{k}: {v}\" for k, v in basics.items() if k != 'period')\n",
+    "    else:\n",
+    "        basics = \"[Basic Financials]:\\n\\nNo basic financial reported.\"\n",
+    "    \n",
+    "    return head, news, basics\n",
+    "\n",
+    "\n",
+    "def sample_news(news, k=5):\n",
+    "    \n",
+    "    return [news[i] for i in sorted(random.sample(range(len(news)), k))]\n",
+    "\n",
+    "\n",
+    "def map_bin_label(bin_lb):\n",
+    "    \n",
+    "    lb = bin_lb.replace('U', 'up by ')\n",
+    "    lb = lb.replace('D', 'down by ')\n",
+    "    lb = lb.replace('1', '0-1%')\n",
+    "    lb = lb.replace('2', '1-2%')\n",
+    "    lb = lb.replace('3', '2-3%')\n",
+    "    lb = lb.replace('4', '3-4%')\n",
+    "    if lb.endswith('+'):\n",
+    "        lb = lb.replace('5+', 'more than 5%')\n",
+    "#         lb = lb.replace('5+', '5+%')\n",
+    "    else:\n",
+    "        lb = lb.replace('5', '4-5%')\n",
+    "    \n",
+    "    return lb\n",
+    "\n",
+    "\n",
+    "def get_all_prompts(symbol, min_past_weeks=1, max_past_weeks=3, with_basics=True):\n",
+    "\n",
+    "    \n",
+    "    if with_basics:\n",
+    "        df = pd.read_csv(f'{DATA_DIR}/{symbol}_{START_DATE}_{END_DATE}.csv')\n",
+    "    else:\n",
+    "        df = pd.read_csv(f'{DATA_DIR}/{symbol}_{START_DATE}_{END_DATE}_nobasics.csv')\n",
+    "    \n",
+    "    company_prompt = get_company_prompt(symbol)\n",
+    "\n",
+    "    prev_rows = []\n",
+    "    all_prompts = []\n",
+    "\n",
+    "    for row_idx, row in df.iterrows():\n",
+    "\n",
+    "        prompt = \"\"\n",
+    "        if len(prev_rows) >= min_past_weeks:\n",
+    "            idx = min(random.choice(range(min_past_weeks, max_past_weeks+1)), len(prev_rows))\n",
+    "            for i in range(-idx, 0):\n",
+    "                # Add Price Movement (Head)\n",
+    "                prompt += \"\\n\" + prev_rows[i][0]\n",
+    "                # Add News of previous weeks\n",
+    "                sampled_news = sample_news(\n",
+    "                    prev_rows[i][1],\n",
+    "                    min(5, len(prev_rows[i][1]))\n",
+    "                )\n",
+    "                if sampled_news:\n",
+    "                    prompt += \"\\n\".join(sampled_news)\n",
+    "                else:\n",
+    "                    prompt += \"No relative news reported.\"\n",
+    "\n",
+    "        head, news, basics = get_prompt_by_row(symbol, row)\n",
+    "\n",
+    "        prev_rows.append((head, news, basics))\n",
+    "        if len(prev_rows) > max_past_weeks:\n",
+    "            prev_rows.pop(0)  \n",
+    "\n",
+    "        if not prompt:\n",
+    "            continue\n",
+    "\n",
+    "        prediction = map_bin_label(row['Bin Label'])\n",
+    "        \n",
+    "        prompt = company_prompt + '\\n' + prompt + '\\n' + basics\n",
+    "        prompt += f\"\\n\\nBased on all the information before {row['Start Date']}, let's first analyze the positive developments and potential concerns for {symbol}. Come up with 2-4 most important factors respectively and keep them concise. Most factors should be inferred from company related news. \" \\\n",
+    "            f\"Then let's assume your prediction for next week ({row['Start Date']} to {row['End Date']}) is {prediction}. Provide a summary analysis to support your prediction. The prediction result need to be inferred from your analysis at the end, and thus not appearing as a foundational factor of your analysis.\"\n",
+    "\n",
+    "        all_prompts.append(prompt.strip())\n",
+    "    \n",
+    "    return all_prompts"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "92208b72",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "B_INST, E_INST = \"[INST]\", \"[/INST]\"\n",
+    "B_SYS, E_SYS = \"<<SYS>>\\n\", \"\\n<</SYS>>\\n\\n\"\n",
+    "\n",
+    "\n",
+    "SYSTEM_PROMPT = \"You are a seasoned stock market analyst. Your task is to list the positive developments and potential concerns for companies based on relevant news and basic financials from the past weeks, then provide an analysis and prediction for the companies' stock price movement for the upcoming week. \" \\\n",
+    "    \"Your answer format should be as follows:\\n\\n[Positive Developments]:\\n1. ...\\n\\n[Potential Concerns]:\\n1. ...\\n\\n[Prediction & Analysis]:\\n...\\n\"\n",
+    "\n",
+    "print(SYSTEM_PROMPT)\n",
+    "\n",
+    "# prompts = get_all_prompts(\"AAPL\", 1, 3)\n",
+    "# prompts = get_all_prompts(\"MSFT\", 1, 3, False)\n",
+    "prompts = get_all_prompts(\"TRV\", 1, 4)\n",
+    "\n",
+    "print(prompts[0])\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2b010a45",
+   "metadata": {},
+   "source": [
+    "# Request to GPT-4 for Financial Analysis"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 86,
+   "id": "3e355117",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def append_to_csv(filename, input_data, output_data):\n",
+    "    \n",
+    "    with open(filename, mode='a', newline='') as file:\n",
+    "        writer = csv.writer(file)\n",
+    "        writer.writerow([input_data, output_data])\n",
+    "\n",
+    "        \n",
+    "def initialize_csv(filename):\n",
+    "    \n",
+    "    with open(filename, mode='w', newline='') as file:\n",
+    "        writer = csv.writer(file)\n",
+    "        writer.writerow([\"prompt\", \"answer\"])\n",
+    "\n",
+    "\n",
+    "def query_gpt4(symbol_list, min_past_weeks=1, max_past_weeks=3, with_basics=True):\n",
+    "\n",
+    "    for symbol in symbol_list:\n",
+    "        \n",
+    "        csv_file = f'{DATA_DIR}/{symbol}_{START_DATE}_{END_DATE}_gpt-4.csv' if with_basics else \\\n",
+    "                   f'{DATA_DIR}/{symbol}_{START_DATE}_{END_DATE}_nobasics_gpt-4.csv'\n",
+    "        \n",
+    "        if not os.path.exists(csv_file):\n",
+    "            initialize_csv(csv_file)\n",
+    "            pre_done = 0\n",
+    "        else:\n",
+    "            df = pd.read_csv(csv_file)\n",
+    "            pre_done = len(df)\n",
+    "\n",
+    "        prompts = get_all_prompts(symbol, min_past_weeks, max_past_weeks, with_basics)\n",
+    "\n",
+    "        for i, prompt in enumerate(prompts):\n",
+    "            \n",
+    "            if i < pre_done:\n",
+    "                continue\n",
+    "\n",
+    "            print(f\"{symbol} - {i}\")\n",
+    "            \n",
+    "            cnt = 0\n",
+    "            while cnt < 5:\n",
+    "                try:\n",
+    "                    completion = client.chat.completions.create(\n",
+    "                        model=\"gpt-4\",\n",
+    "                        messages=[\n",
+    "                            {\"role\": \"system\", \"content\": SYSTEM_PROMPT},\n",
+    "                            {\"role\": \"user\", \"content\": prompt}\n",
+    "                          ]\n",
+    "                    )\n",
+    "                    break    \n",
+    "                except Exception:\n",
+    "                    cnt += 1\n",
+    "                    print(f'retry cnt {cnt}')\n",
+    "            \n",
+    "            answer = completion.choices[0].message.content if cnt < 5 else \"\"\n",
+    "            append_to_csv(csv_file, prompt, answer)\n",
+    "      "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 121,
+   "id": "a9ff6ff3",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "WBA - 12\n",
+      "WBA - 13\n",
+      "WBA - 14\n",
+      "WBA - 15\n",
+      "WBA - 16\n",
+      "WBA - 17\n",
+      "WBA - 18\n",
+      "WBA - 19\n"
+     ]
+    }
+   ],
+   "source": [
+    "# query_gpt4(DOW_30, 1, 3)\n",
+    "query_gpt4(DOW_30, 1, 4)\n",
+    "# query_gpt4(['WBA'], 1, 4)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "238ba9f0",
+   "metadata": {},
+   "source": [
+    "# Transform into Llama2 Training Format"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 93,
+   "id": "d2627f5a",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def gpt4_to_llama(symbol, with_basics=True):\n",
+    "    \n",
+    "    csv_file = f'{DATA_DIR}/{symbol}_{START_DATE}_{END_DATE}_gpt-4.csv' if with_basics else \\\n",
+    "                   f'{DATA_DIR}/{symbol}_{START_DATE}_{END_DATE}_nobasics_gpt-4.csv'\n",
+    "    \n",
+    "    df = pd.read_csv(csv_file)\n",
+    "    \n",
+    "    prompts, answers, periods, labels = [], [], [], []\n",
+    "    \n",
+    "    for i, row in df.iterrows():\n",
+    "        \n",
+    "        prompt, answer = row['prompt'], row['answer']\n",
+    "        \n",
+    "        res = re.search(r\"Then let's assume your prediction for next week \\((.*)\\) is ((:?up|down) by .*%).\", prompt)\n",
+    "        \n",
+    "        period, label = res.group(1), res.group(2)\n",
+    "#         label = label.replace('more than 5', '5+')\n",
+    "        \n",
+    "        prompt = re.sub(\n",
+    "            r\"Then let's assume your prediction for next week \\((.*)\\) is (up|down) by ((:?.*)%). Provide a summary analysis to support your prediction. The prediction result need to be inferred from your analysis at the end, and thus not appearing as a foundational factor of your analysis.\", \n",
+    "            f\"Then make your prediction of the {symbol} stock price movement for next week ({period}). Provide a summary analysis to support your prediction.\",\n",
+    "            prompt\n",
+    "        )\n",
+    "        try:\n",
+    "            answer = re.sub(\n",
+    "                r\"\\[Prediction & Analysis\\]:\\s*\",\n",
+    "                f\"[Prediction & Analysis]:\\nPrediction: {label.capitalize()}\\nAnalysis: \",\n",
+    "                answer\n",
+    "            )\n",
+    "        except Exception:\n",
+    "            print(symbol, i)\n",
+    "            print(label)\n",
+    "            print(answer)\n",
+    "            continue\n",
+    "            \n",
+    "        new_system_prompt = SYSTEM_PROMPT.replace(':\\n...', '\\nPrediction: ...\\nAnalysis: ...')\n",
+    "#         new_system_prompt = SYSTEM_PROMPT.replace(':\\n...', '\\nPrediction: {Up|Down} by {1-2|2-3|3-4|4-5|5+}%\\nAnalysis: ...')\n",
+    "        \n",
+    "        prompt = B_INST + B_SYS + new_system_prompt + E_SYS + prompt + E_INST\n",
+    "        \n",
+    "        prompts.append(prompt)\n",
+    "        answers.append(answer)\n",
+    "        periods.append(period)\n",
+    "        labels.append(label)\n",
+    "        \n",
+    "    return {\n",
+    "        \"prompt\": prompts,\n",
+    "        \"answer\": answers,\n",
+    "        \"period\": periods,\n",
+    "        \"label\": labels,\n",
+    "    }\n",
+    "\n",
+    "\n",
+    "def create_dataset(symbol_list, train_ratio=0.8, with_basics=True):\n",
+    "\n",
+    "    train_dataset_list = []\n",
+    "    test_dataset_list = []\n",
+    "\n",
+    "    for symbol in symbol_list:\n",
+    "\n",
+    "        data_dict = gpt4_to_llama(symbol, with_basics)\n",
+    "#         print(data_dict['prompt'][-1])\n",
+    "#         print(data_dict['answer'][-1])\n",
+    "        symbols = [symbol] * len(data_dict['label'])\n",
+    "        data_dict.update({\"symbol\": symbols})\n",
+    "\n",
+    "        dataset = Dataset.from_dict(data_dict)\n",
+    "        train_size = round(train_ratio * len(dataset))\n",
+    "\n",
+    "        train_dataset_list.append(dataset.select(range(train_size)))\n",
+    "        test_dataset_list.append(dataset.select(range(train_size, len(dataset))))\n",
+    "\n",
+    "    train_dataset = datasets.concatenate_datasets(train_dataset_list)\n",
+    "    test_dataset = datasets.concatenate_datasets(test_dataset_list)\n",
+    "\n",
+    "    dataset = datasets.DatasetDict({\n",
+    "        'train': train_dataset,\n",
+    "        'test': test_dataset\n",
+    "    })\n",
+    "    \n",
+    "    return dataset\n",
+    "   "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 129,
+   "id": "e089b1bf",
+   "metadata": {
+    "scrolled": true
+   },
+   "outputs": [],
+   "source": [
+    "# v1\n",
+    "# dow30_dataset = create_dataset(DOW30, True)\n",
+    "# v2\n",
+    "# dow30_nobasic_dataset = create_dataset(DOW_30, 0.8, False)\n",
+    "# v3\n",
+    "dow30_v3_dataset = create_dataset(DOW_30, 0.9)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 130,
+   "id": "123f2db9",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "439535ce3e804a3d847f1e03df02283d",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Saving the dataset (0/1 shards):   0%|          | 0/540 [00:00<?, ? examples/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "373274c4f9b547fabe0027bc696912c3",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "Saving the dataset (0/1 shards):   0%|          | 0/60 [00:00<?, ? examples/s]"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "# dow30_dataset.save_to_disk('fingpt-forecaster-dow30-20230601-20230930-llama')\n",
+    "# dow30_nobasics_dataset.save_to_disk('fingpt-forecaster-dow30nobasics-20230601-20230930-llama')\n",
+    "dow30_v3_dataset.save_to_disk('fingpt-forecaster-dow30v3-20221231-20230531-llama')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 131,
+   "id": "9ed5cf5f",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "DatasetDict({\n",
+       "    train: Dataset({\n",
+       "        features: ['prompt', 'answer', 'period', 'label', 'symbol'],\n",
+       "        num_rows: 540\n",
+       "    })\n",
+       "    test: Dataset({\n",
+       "        features: ['prompt', 'answer', 'period', 'label', 'symbol'],\n",
+       "        num_rows: 60\n",
+       "    })\n",
+       "})"
+      ]
+     },
+     "execution_count": 131,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "dow30_v3_dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "d62e711f",
+   "metadata": {},
+   "source": [
+    "# Test-time Information Fetching"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 90,
+   "id": "292268bc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import yfinance as yf\n",
+    "import pandas as pd\n",
+    "from datetime import date, datetime, timedelta\n",
+    "\n",
+    "\n",
+    "def get_curday():\n",
+    "    \n",
+    "    return date.today().strftime(\"%Y-%m-%d\")\n",
+    "\n",
+    "\n",
+    "def n_weeks_before(date_string, n):\n",
+    "    \n",
+    "    date = datetime.strptime(date_string, \"%Y-%m-%d\") - timedelta(days=7*n)\n",
+    "    \n",
+    "    return date.strftime(\"%Y-%m-%d\")\n",
+    "\n",
+    "\n",
+    "def get_stock_data(stock_symbol, steps):\n",
+    "\n",
+    "    stock_data = yf.download(stock_symbol, steps[0], steps[-1])\n",
+    "    \n",
+    "#     print(stock_data)\n",
+    "    \n",
+    "    dates, prices = [], []\n",
+    "    available_dates = stock_data.index.format()\n",
+    "    \n",
+    "    for date in steps[:-1]:\n",
+    "        for i in range(len(stock_data)):\n",
+    "            if available_dates[i] >= date:\n",
+    "                prices.append(stock_data['Close'][i])\n",
+    "                dates.append(datetime.strptime(available_dates[i], \"%Y-%m-%d\"))\n",
+    "                break\n",
+    "\n",
+    "    dates.append(datetime.strptime(available_dates[-1], \"%Y-%m-%d\"))\n",
+    "    prices.append(stock_data['Close'][-1])\n",
+    "    \n",
+    "    return pd.DataFrame({\n",
+    "        \"Start Date\": dates[:-1], \"End Date\": dates[1:],\n",
+    "        \"Start Price\": prices[:-1], \"End Price\": prices[1:]\n",
+    "    })\n",
+    "\n",
+    "\n",
+    "def get_current_basics(symbol, curday):\n",
+    "\n",
+    "    basic_financials = finnhub_client.company_basic_financials(symbol, 'all')\n",
+    "    \n",
+    "    final_basics, basic_list, basic_dict = [], [], defaultdict(dict)\n",
+    "    \n",
+    "    for metric, value_list in basic_financials['series']['quarterly'].items():\n",
+    "        for value in value_list:\n",
+    "            basic_dict[value['period']].update({metric: value['v']})\n",
+    "\n",
+    "    for k, v in basic_dict.items():\n",
+    "        v.update({'period': k})\n",
+    "        basic_list.append(v)\n",
+    "        \n",
+    "    basic_list.sort(key=lambda x: x['period'])\n",
+    "    \n",
+    "    for basic in basic_list[::-1]:\n",
+    "        if basic['period'] <= curday:\n",
+    "            break\n",
+    "            \n",
+    "    return basic\n",
+    "    \n",
+    "\n",
+    "def get_all_prompts_online(symbol, data, curday, with_basics=True):\n",
+    "\n",
+    "    company_prompt = get_company_prompt(symbol)\n",
+    "\n",
+    "    prev_rows = []\n",
+    "\n",
+    "    for row_idx, row in data.iterrows():\n",
+    "        head, news, _ = get_prompt_by_row(symbol, row)\n",
+    "        prev_rows.append((head, news, None))\n",
+    "        \n",
+    "    prompt = \"\"\n",
+    "    for i in range(-len(prev_rows), 0):\n",
+    "        prompt += \"\\n\" + prev_rows[i][0]\n",
+    "        sampled_news = sample_news(\n",
+    "            prev_rows[i][1],\n",
+    "            min(5, len(prev_rows[i][1]))\n",
+    "        )\n",
+    "        if sampled_news:\n",
+    "            prompt += \"\\n\".join(sampled_news)\n",
+    "        else:\n",
+    "            prompt += \"No relative news reported.\"\n",
+    "        \n",
+    "    period = \"{} to {}\".format(curday, n_weeks_before(curday, -1))\n",
+    "    \n",
+    "    if with_basics:\n",
+    "        basics = get_current_basics(symbol, curday)\n",
+    "        basics = \"Some recent basic financials of {}, reported at {}, are presented below:\\n\\n[Basic Financials]:\\n\\n\".format(\n",
+    "            symbol, basics['period']) + \"\\n\".join(f\"{k}: {v}\" for k, v in basics.items() if k != 'period')\n",
+    "    else:\n",
+    "        basics = \"[Basic Financials]:\\n\\nNo basic financial reported.\"\n",
+    "\n",
+    "    info = company_prompt + '\\n' + prompt + '\\n' + basics\n",
+    "    prompt = info + f\"\\n\\nBased on all the information before {curday}, let's first analyze the positive developments and potential concerns for {symbol}. Come up with 2-4 most important factors respectively and keep them concise. Most factors should be inferred from company related news. \" \\\n",
+    "        f\"Then make your prediction of the {symbol} stock price movement for next week ({period}). Provide a summary analysis to support your prediction.\"\n",
+    "        \n",
+    "    return info, prompt"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 76,
+   "id": "8f48aab1",
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[*********************100%%**********************]  1 of 1 completed\n",
+      "AAPL :  2023-10-25  -  2023-11-01\n",
+      "AAPL :  2023-11-01  -  2023-11-07\n"
+     ]
+    }
+   ],
+   "source": [
+    "ticker = \"AAPL\"\n",
+    "n_weeks = 2\n",
+    "curday = get_curday()\n",
+    "steps = [n_weeks_before(curday, n) for n in range(n_weeks + 1)][::-1]\n",
+    "\n",
+    "data = get_stock_data(ticker, steps)\n",
+    "\n",
+    "data = get_news(ticker, data)\n",
+    "\n",
+    "data['Basics'] = [json.dumps({})] * len(data)\n",
+    "# data = get_basics(ticker, data, always=True)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 91,
+   "id": "84bb302a",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[Company Introduction]:\n",
+      "\n",
+      "Apple Inc is a leading entity in the Technology sector. Incorporated and publicly traded since 1980-12-12, the company has established its reputation as one of the key players in the market. As of today, Apple Inc has a market capitalization of 2809837.86 in USD, with 15634.23 shares outstanding.\n",
+      "\n",
+      "Apple Inc operates primarily in the US, trading under the ticker AAPL on the NASDAQ NMS - GLOBAL MARKET. As a dominant force in the Technology space, the company continues to innovate and drive progress within the industry.\n",
+      "\n",
+      "From 2023-10-25 to 2023-11-01, AAPL's stock price increased from 171.10 to 173.97. Company news during this period are listed below:\n",
+      "\n",
+      "[Headline]: 25 Largest Economies in the World by 2075\n",
+      "[Summary]: In this article, we will be taking a look at the 25 largest economies in the world by 2075. To skip our detailed analysis, you can go directly to see the 5 largest economies in the world by 2075. In both 2022 and 2023, the global economy has struggled significantly after record inflation enveloped most countries across […]\n",
+      "\n",
+      "[Headline]: India opposition accuses govt of trying to hack lawmakers' iPhones\n",
+      "[Summary]: Indian opposition leader Rahul Gandhi on Tuesday accused Prime Minister Narendra Modi's government of trying to hack into senior opposition politicians' mobile phones, after they reported receiving warning messages from Apple.  Some of the lawmakers shared screenshots on social media of a notification quoting the iPhone manufacturer as saying: \"Apple believes you are being targeted by state-sponsored attackers who are trying to remotely compromise the iPhone associated with your Apple ID\".  \"Hack us all you want,\" Gandhi told a news conference in New Delhi, in reference to Modi.\n",
+      "\n",
+      "[Headline]: 39% Of This Apple Insider's Holdings Were Sold\n",
+      "[Summary]: Looking at Apple Inc.'s ( NASDAQ:AAPL ) insider transactions over the last year, we can see that insiders were net...\n",
+      "\n",
+      "[Headline]: Indian opposition MPs accuse government of trying to hack their iPhones\n",
+      "[Summary]: Ruling BJP rejects claims of involvement following Apple notifications of possible ‘state-sponsored’ attacks\n",
+      "\n",
+      "[Headline]: Should You Buy These 2 ‘Magnificent Seven’ Stocks Ahead of Earnings? Apple and Nvidia in Focus\n",
+      "[Summary]: What should investors make of this year’s third-quarter earnings? The Q3 results have been pretty good, with 78% of companies reporting so far beating the forecasts, but stocks are still feeling pressure. One obvious sign of that pressure: the S&P 500 this week hit its lowest point since last May, and is just shy of correction territory. The effect is most clearly seen in the ‘Magnificent Seven,’ a group of Big Tech giants whose gains earlier in the year carried the markets generally – but which\n",
+      "\n",
+      "From 2023-11-01 to 2023-11-07, AAPL's stock price increased from 173.97 to 181.25. Company news during this period are listed below:\n",
+      "\n",
+      "[Headline]: Apple Earnings: Why Guidance Will Be Key\n",
+      "[Summary]: Tech giant Apple (NASDAQ: AAPL) is scheduled to report its fiscal fourth-quarter results on Thursday.  After all, the company's approximately $2.7 trillion market cap is big enough to influence major market indexes like the S&P 500; Apple represents about 7% of the index.  While the company's fiscal fourth-quarter financial performance will definitely be important, investors may pay even closer attention to another metric: management's guidance for its fiscal first-quarter revenue.\n",
+      "\n",
+      "[Headline]: Analysts offer hot takes on Q4 2023 Apple results\n",
+      "[Summary]: Analysts have weighed in on Apple's Q4 2023 financial results, with most taking the view that the quarter is decent-performing, but with caution about a shorter Q1 2024.\n",
+      "\n",
+      "[Headline]: How to run new macOS versions on older Macs with OpenCore\n",
+      "[Summary]: Apple removes support for old Mac hardware in new macOS releases. Here's how to run modern macOS on older Macs using OpenCore.\n",
+      "\n",
+      "[Headline]: Apple Watch import ban: what you need to know\n",
+      "[Summary]: There is a possibility of an import ban in the U.S. on the Apple Watch. Here's what you need to know before it potentially goes into effect on Christmas Day, 2023.\n",
+      "\n",
+      "[Headline]: ChatGPT: Everything you need to know about the AI-powered chatbot\n",
+      "[Summary]: ChatGPT, OpenAI’s text-generating AI chatbot, has taken the world by storm. What started as a tool to hyper-charge productivity through writing essays and code with short text prompts has evolved into a behemoth used by more than 92% of Fortune 500 companies for more wide-ranging needs. While there is a more…nefarious side to ChatGPT, it’s clear that AI tools are not going away anytime soon. Since its initial launch nearly a year ago, ChatGPT has hit 100 million weekly active users, and OpenAI i\n",
+      "\n",
+      "[Basic Financials]:\n",
+      "\n",
+      "No basic financial reported.\n",
+      "\n",
+      "Based on all the information before 2023-11-08, let's first analyze the positive developments and potential concerns for AAPL. Come up with 2-4 most important factors respectively and keep them concise. Most factors should be inferred from company related news. Then make your prediction of the AAPL stock price movement for next week (2023-11-08 to 2023-11-15). Provide a summary analysis to support your prediction.\n"
+     ]
+    }
+   ],
+   "source": [
+    "info, prompt = get_all_prompts_online(ticker, data, curday, False)\n",
+    "\n",
+    "print(prompt)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}

requirements.txt

@@ -0,0 +1,7 @@
+torch==2.0.1
+transformers==4.32.0
+peft==0.5.0
+pandas
+yfinance
+finnhub-python
+nvidia-ml-py3

train.sh

@@ -0,0 +1,21 @@
+export NCCL_IGNORE_DISABLED_P2P=1
+export TRANSFORMERS_NO_ADVISORY_WARNINGS=1
+export TOKENIZERS_PARALLELISM=0
+
+
+deepspeed \
+--include localhost:2,3 \
+train_lora.py \
+--run_name dow30v3-llama2-5e-5lr-qkvogud \
+--base_model llama2 \
+--dataset dow30-20230601-20230930-llama,dow30nobasics-20230601-20230930-llama,dow30v3-20221231-20230531-llama*2 \
+--max_length 4096 \
+--batch_size 1 \
+--gradient_accumulation_steps 16 \
+--learning_rate 5e-5 \
+--num_epochs 5 \
+--log_interval 10 \
+--warmup_ratio 0.03 \
+--scheduler constant \
+--evaluation_strategy steps \
+--ds_config config.json

train_lora.py

@@ -0,0 +1,221 @@
+from transformers.integrations import TensorBoardCallback
+from transformers import AutoTokenizer, AutoModel, AutoModelForCausalLM
+from transformers import TrainingArguments, Trainer, DataCollatorForSeq2Seq
+from transformers import TrainerCallback, TrainerState, TrainerControl
+from transformers.trainer import TRAINING_ARGS_NAME
+from torch.utils.tensorboard import SummaryWriter
+import datasets
+import torch
+import os
+import re
+import sys
+import wandb
+import argparse
+from datetime import datetime
+from functools import partial
+from tqdm import tqdm
+from utils import *
+
+# LoRA
+from peft import (
+    TaskType,
+    LoraConfig,
+    get_peft_model,
+    get_peft_model_state_dict,
+    prepare_model_for_int8_training,
+    set_peft_model_state_dict,   
+)
+
+# Replace with your own api_key and project name
+os.environ['WANDB_API_KEY'] = 'ecf1e5e4f47441d46822d38a3249d62e8fc94db4'
+os.environ['WANDB_PROJECT'] = 'fingpt-forecaster'
+
+
+class GenerationEvalCallback(TrainerCallback):
+    
+    def __init__(self, eval_dataset, ignore_until_epoch=0):
+        self.eval_dataset = eval_dataset
+        self.ignore_until_epoch = ignore_until_epoch
+    
+    def on_evaluate(self, args, state: TrainerState, control: TrainerControl, **kwargs):
+        
+        if state.epoch is None or state.epoch + 1 < self.ignore_until_epoch:
+            return
+            
+        if state.is_local_process_zero:
+            model = kwargs['model']
+            tokenizer = kwargs['tokenizer']
+            generated_texts, reference_texts = [], []
+
+            for feature in tqdm(self.eval_dataset):
+                prompt = feature['prompt']
+                gt = feature['answer']
+                inputs = tokenizer(
+                    prompt, return_tensors='pt',
+                    padding=False, max_length=4096
+                )
+                inputs = {key: value.to(model.device) for key, value in inputs.items()}
+                
+                res = model.generate(
+                    **inputs, 
+                    use_cache=True
+                )
+                output = tokenizer.decode(res[0], skip_special_tokens=True)
+                answer = re.sub(r'.*\[/INST\]\s*', '', output, flags=re.DOTALL)
+
+                generated_texts.append(answer)
+                reference_texts.append(gt)
+
+                # print("GENERATED: ", answer)
+                # print("REFERENCE: ", gt)
+
+            metrics = calc_metrics(reference_texts, generated_texts)
+            
+            # Ensure wandb is initialized
+            if wandb.run is None:
+                wandb.init()
+                
+            wandb.log(metrics, step=state.global_step)
+            torch.cuda.empty_cache()            
+
+
+def main(args):
+        
+    model_name = parse_model_name(args.base_model, args.from_remote)
+    
+    # load model
+    model = AutoModelForCausalLM.from_pretrained(
+        model_name,
+        # load_in_8bit=True,
+        trust_remote_code=True
+    )
+    if args.local_rank == 0:
+        print(model)
+    
+    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
+    tokenizer.pad_token = tokenizer.eos_token
+    tokenizer.padding_side = "right"
+    
+    # load data
+    dataset_list = load_dataset(args.dataset, args.from_remote)
+    
+    dataset_train = datasets.concatenate_datasets([d['train'] for d in dataset_list]).shuffle(seed=42)
+    
+    if args.test_dataset:
+        dataset_list = load_dataset(args.test_dataset, args.from_remote)
+            
+    dataset_test = datasets.concatenate_datasets([d['test'] for d in dataset_list])
+    
+    original_dataset = datasets.DatasetDict({'train': dataset_train, 'test': dataset_test})
+    
+    eval_dataset = original_dataset['test'].shuffle(seed=42).select(range(50))
+    
+    dataset = original_dataset.map(partial(tokenize, args, tokenizer))
+    print('original dataset length: ', len(dataset['train']))
+    dataset = dataset.filter(lambda x: not x['exceed_max_length'])
+    print('filtered dataset length: ', len(dataset['train']))
+    dataset = dataset.remove_columns(
+        ['prompt', 'answer', 'label', 'symbol', 'period', 'exceed_max_length']
+    )
+    
+    current_time = datetime.now()
+    formatted_time = current_time.strftime('%Y%m%d%H%M')
+    
+    training_args = TrainingArguments(
+        output_dir=f'finetuned_models/{args.run_name}_{formatted_time}', # 保存位置
+        logging_steps=args.log_interval,
+        num_train_epochs=args.num_epochs,
+        per_device_train_batch_size=args.batch_size,
+        per_device_eval_batch_size=args.batch_size,
+        gradient_accumulation_steps=args.gradient_accumulation_steps,
+        dataloader_num_workers=args.num_workers,
+        learning_rate=args.learning_rate,
+        weight_decay=args.weight_decay,
+        warmup_ratio=args.warmup_ratio,
+        lr_scheduler_type=args.scheduler,
+        save_steps=args.eval_steps,
+        eval_steps=args.eval_steps,
+        fp16=True,
+        deepspeed=args.ds_config,
+        evaluation_strategy=args.evaluation_strategy,
+        remove_unused_columns=False,
+        report_to='wandb',
+        run_name=args.run_name
+    )
+    
+    model.gradient_checkpointing_enable()
+    model.enable_input_require_grads()
+    model.is_parallelizable = True
+    model.model_parallel = True
+    model.model.config.use_cache = False
+    
+    # model = prepare_model_for_int8_training(model)
+
+    # setup peft
+    peft_config = LoraConfig(
+        task_type=TaskType.CAUSAL_LM,
+        inference_mode=False,
+        r=8,
+        lora_alpha=16,
+        lora_dropout=0.1,
+        target_modules=lora_module_dict[args.base_model],
+        bias='none',
+    )
+    model = get_peft_model(model, peft_config)
+    
+    # Train
+    trainer = Trainer(
+        model=model, 
+        args=training_args, 
+        train_dataset=dataset['train'],
+        eval_dataset=dataset['test'], 
+        tokenizer=tokenizer,
+        data_collator=DataCollatorForSeq2Seq(
+            tokenizer, padding=True,
+            return_tensors="pt"
+        ),
+        callbacks=[
+            GenerationEvalCallback(
+                eval_dataset=eval_dataset,
+                ignore_until_epoch=round(0.3 * args.num_epochs)
+            )
+        ]
+    )
+    
+    if torch.__version__ >= "2" and sys.platform != "win32":
+        model = torch.compile(model)
+    
+    torch.cuda.empty_cache()
+    trainer.train()
+
+    # save model
+    model.save_pretrained(training_args.output_dir)
+
+
+if __name__ == "__main__":
+    
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--local_rank", default=0, type=int)
+    parser.add_argument("--run_name", default='local-test', type=str)
+    parser.add_argument("--dataset", required=True, type=str)
+    parser.add_argument("--test_dataset", type=str)
+    parser.add_argument("--base_model", required=True, type=str, choices=['chatglm2', 'llama2'])
+    parser.add_argument("--max_length", default=512, type=int)
+    parser.add_argument("--batch_size", default=4, type=int, help="The train batch size per device")
+    parser.add_argument("--learning_rate", default=1e-4, type=float, help="The learning rate")
+    parser.add_argument("--weight_decay", default=0.01, type=float, help="weight decay")
+    parser.add_argument("--num_epochs", default=8, type=float, help="The training epochs")
+    parser.add_argument("--num_workers", default=8, type=int, help="dataloader workers")
+    parser.add_argument("--log_interval", default=20, type=int)
+    parser.add_argument("--gradient_accumulation_steps", default=8, type=int)
+    parser.add_argument("--warmup_ratio", default=0.05, type=float)
+    parser.add_argument("--ds_config", default='./config_new.json', type=str)
+    parser.add_argument("--scheduler", default='linear', type=str)
+    parser.add_argument("--instruct_template", default='default')
+    parser.add_argument("--evaluation_strategy", default='steps', type=str)    
+    parser.add_argument("--eval_steps", default=0.1, type=float)    
+    parser.add_argument("--from_remote", default=False, type=bool)    
+    args = parser.parse_args()
+    
+    wandb.login()
+    main(args)

utils.py

@@ -0,0 +1,162 @@
+import re
+import os
+import datasets
+from sklearn.metrics import accuracy_score, mean_squared_error
+from collections import defaultdict
+from rouge_score import rouge_scorer
+
+
+lora_module_dict = {
+    'chatglm2': ['query_key_value'],
+    'llama2': [
+        'q_proj', 'k_proj', 'v_proj',
+        'o_proj', 'gate_proj', 'up_proj', 'down_proj',
+        # 'embed_tokens', 'lm_head',
+    ],
+}
+
+
+def tokenize(args, tokenizer, feature):
+    
+    prompt_ids = tokenizer.encode(
+        feature['prompt'].strip(), padding=False,
+        max_length=args.max_length, truncation=True
+    )
+    
+    target_ids = tokenizer.encode(
+        feature['answer'].strip(), padding=False,
+        max_length=args.max_length, truncation=True, add_special_tokens=False
+    )
+    
+    input_ids = prompt_ids + target_ids
+    exceed_max_length = len(input_ids) >= args.max_length
+    
+     # Add EOS Token
+    if input_ids[-1] != tokenizer.eos_token_id and not exceed_max_length:
+        input_ids.append(tokenizer.eos_token_id)
+    
+    label_ids = [tokenizer.pad_token_id] * len(prompt_ids) + input_ids[len(prompt_ids):]
+    
+    return {
+        "input_ids": input_ids,
+        "labels": label_ids,
+        "exceed_max_length": exceed_max_length
+    }
+
+
+def parse_model_name(name, from_remote=False):
+    
+    if name == 'chatglm2':
+        return 'THUDM/chatglm2-6b' if from_remote else 'base_models/chatglm2-6b'
+    elif name == 'llama2':
+        return 'meta-llama/Llama-2-7b-chat-hf' if from_remote else 'base_models/Llama-2-7b-chat-hf'
+    else:
+        raise ValueError(f"Undefined base model {name}")
+        
+    
+def load_dataset(names, from_remote=False):
+    
+    dataset_names = [d for d in names.split(',')]
+    dataset_list = []
+    
+    for name in dataset_names:
+        rep = 1
+        if not os.path.exists(name):
+            rep = int(name.split('*')[1]) if '*' in name else 1
+            name = ('FinGPT/fingpt-forecaster-' if from_remote else 'data/fingpt-forecaster-') + name.split('*')[0]
+        tmp_dataset = datasets.load_dataset(name) if from_remote else datasets.load_from_disk(name)
+    
+        if 'test' not in tmp_dataset:
+            tmp_dataset = tmp_dataset.train_test_split(0.2, shuffle=True, seed=42)   
+        dataset_list.extend([tmp_dataset] * rep)
+    
+    return dataset_list
+
+
+def parse_answer(answer):
+    
+    match_res = re.match(r"^\s*\[Positive Developments\]:\s*(.*)\s*\[Potential Concerns\]:\s*(.*)\s*\[Prediction & Analysis\]:\s*(.*)\s*$", answer, flags=re.DOTALL)
+    if not match_res:
+        return None
+    
+    pros, cons, pna = match_res.group(1), match_res.group(2), match_res.group(3)
+        
+    match_res = re.match(r'^Prediction:\s*(.*)\s*Analysis:\s*(.*)\s*$', pna, flags=re.DOTALL)
+    if not match_res:
+        return None
+        
+    pred, anal = match_res.group(1), match_res.group(2)
+        
+    if re.search(r'up|increase', pred.lower()):
+        pred_bin = 1
+    elif re.search(r'down|decrease|decline', pred.lower()):
+        pred_bin = -1
+    else:
+        pred_bin = 0
+            
+    match_res = re.search(r'(\d)-(\d)%', pred)
+    if not match_res:
+        match_res = re.search(r'(?:more than )?(\d)+?%', pred)    
+        
+    pred_margin = pred_bin * (int(match_res.group(1)) + 0.5) if match_res else 0.
+        
+    return {
+        "positive developments": pros,
+        "potential concerns": cons,
+        "prediction": pred_margin,
+        "prediction_binary": pred_bin,
+        "analysis": anal
+    }
+    
+
+def calc_rouge_score(references, answers):
+    
+    scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
+        
+    scores_per_pair = [scorer.score(ref, ans) for ref, ans in zip(references, answers)]
+    
+    rouge1 = sum(score['rouge1'].fmeasure for score in scores_per_pair) / len(scores_per_pair)
+    rouge2 = sum(score['rouge2'].fmeasure for score in scores_per_pair) / len(scores_per_pair)
+    rougeL = sum(score['rougeL'].fmeasure for score in scores_per_pair) / len(scores_per_pair)
+    
+    return {'rouge1': rouge1, 'rouge2': rouge2, 'rougeL': rougeL}
+
+    
+def calc_metrics(answers, gts):
+    
+    answers_dict = defaultdict(list)
+    gts_dict = defaultdict(list)
+    
+    for answer, gt in zip(answers, gts):
+        answer_dict = parse_answer(answer)
+        gt_dict = parse_answer(gt)
+        
+        if answer_dict and gt_dict:
+            for k in answer_dict.keys():
+                answers_dict[k].append(answer_dict[k])
+                gts_dict[k].append(gt_dict[k])
+    
+    if not answers_dict['prediction']:
+        return {}
+    
+    bin_acc = accuracy_score(gts_dict['prediction_binary'], answers_dict['prediction_binary'])
+    mse = mean_squared_error(gts_dict['prediction'], answers_dict['prediction'])
+    
+    pros_rouge_scores = calc_rouge_score(gts_dict['positive developments'], answers_dict['positive developments'])
+    cons_rouge_scores = calc_rouge_score(gts_dict['potential concerns'], answers_dict['potential concerns'])
+    anal_rouge_scores = calc_rouge_score(gts_dict['analysis'], answers_dict['analysis'])
+                              
+    print(f"\nBinary Accuracy: {bin_acc:.2f}  |  Mean Square Error: {mse:.2f}")
+    print(f"\nRouge Score of Positive Developments: {pros_rouge_scores}")
+    print(f"\nRouge Score of Potential Concerns: {cons_rouge_scores}")
+    print(f"\nRouge Score of Summary Analysis: {anal_rouge_scores}")
+                              
+    return {
+        "valid_count": len(answers_dict['prediction']),
+        "bin_acc": bin_acc,
+        "mse": mse,
+        "pros_rouge_scores": pros_rouge_scores,
+        "cons_rouge_scores": cons_rouge_scores,
+        "anal_rouge_scores": anal_rouge_scores
+    }
+