Switch back to markdown as easier diffable

.vscode/extensions.json

@@ -1,6 +1,5 @@
 {
     "recommendations": [
-        "janisdd.vscode-edit-csv",
-        "mechatroner.rainbow-csv"
+        "takumii.markdowntable"
     ]
 }

README.md

@@ -1,21 +1,55 @@
 # 🏆 llm-leaderboard
-A joint community effort to create one central leaderboard for LLMs. Contributions and corrections welcome!
-
-## Leaderboard
 
+A joint community effort to create one central leaderboard for LLMs. Contributions and corrections welcome!
 Visit the interactive leaderboard at https://llm-leaderboard.streamlit.app/.
 
-![Screenshot of streamlit application](media/streamlit_screenshot.jpg)
-
 ## How to Contribute
 
 We are always happy for contributions! You can contribute by the following:
 
 - table work:
     - filling missing entries
-    - adding a new model as a new row in `data/leaderboard.csv`, add the source of the evaluation to `data/sources.csv`
-    - adding a new benchmark as a new column in `data/leaderboard.csv` and add the benchmark to `data/benchmarks.csv`
+    - adding a new model as a new row to the leaderboard and add the source of the evaluation to the sources table
+    - adding a new benchmark as a new column in the leaderboard and add the benchmark to the benchmarks table
 - code work:
     - improving the existing code
     - requesting and implementing new features
 
+## Leaderboard
+
+| Model Name              | Chatbot Arena Elo | LAMBADA (zero-shot) | TriviaQA (zero-shot) |
+| ----------------------- | ----------------- | ------------------- | -------------------- |
+| alpaca-13b              | 1008              |                     |                      |
+| cerebras-7b             |                   | 0.636               | 0.141                |
+| cerebras-13b            |                   | 0.635               | 0.146                |
+| chatglm-6b              | 985               |                     |                      |
+| dolly-v2-12b            | 944               |                     |                      |
+| fastchat-t5-3b          | 951               |                     |                      |
+| gpt-neox-20b            |                   | 0.719               | 0.347                |
+| gptj-6b                 |                   | 0.683               | 0.234                |
+| koala-13b               | 1082              |                     |                      |
+| llama-7b                |                   | 0.738               | 0.443                |
+| llama-13b               | 932               |                     |                      |
+| mpt-7b                  |                   | 0.702               | 0.343                |
+| opt-7b                  |                   | 0.677               | 0.227                |
+| opt-13b                 |                   | 0.692               | 0.282                |
+| stablelm-base-alpha-7b  |                   | 0.533               | 0.049                |
+| stablelm-tuned-alpha-7b | 858               |                     |                      |
+| vicuna-13b              | 1169              |                     |                      |
+| oasst-pythia-7b         |                   | 0.667               | 0.198                |
+| oasst-pythia-12b        | 1065              | 0.704               | 0.233                |
+
+## Benchmarks
+
+| Benchmark Name    | Author         | Link                                     | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
+| ----------------- | -------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Chatbot Arena Elo | LMSYS          | https://lmsys.org/blog/2023-05-03-arena/ | "In this blog post, we introduce Chatbot Arena, an LLM benchmark platform featuring anonymous randomized battles in a crowdsourced manner. Chatbot Arena adopts the Elo rating system, which is a widely-used rating system in chess and other competitive games." (Source: https://lmsys.org/blog/2023-05-03-arena/)                                                                                                                                                                                                                                                                 |
+| LAMBADA           | Paperno et al. | https://arxiv.org/abs/1606.06031"        | "The LAMBADA evaluates the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse." (Source: https://huggingface.co/datasets/lambada) |
+| TriviaQA          | Joshi et al.   | https://arxiv.org/abs/1705.03551v2"      | "We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions." (Source: https://arxiv.org/abs/1705.03551v2)                                                                                                                                                                              |
+
+## Sources
+
+| Author   | Link                                     |
+| -------- | ---------------------------------------- |
+| LMSYS    | https://lmsys.org/blog/2023-05-03-arena/ |
+| MOSAICML | https://www.mosaicml.com/blog/mpt-7b     |

data/benchmarks.csv

@@ -1,4 +0,0 @@
-"Benchmark Name"    ,"Author"         ,"Link"                                     ,"Description "
-"Chatbot Arena Elo" ,"LMSYS"          ,"https://lmsys.org/blog/2023-05-03-arena/" ,"In this blog post, we introduce Chatbot Arena, an LLM benchmark platform featuring anonymous randomized battles in a crowdsourced manner. Chatbot Arena adopts the Elo rating system, which is a widely-used rating system in chess and other competitive games. (Source: https://lmsys.org/blog/2023-05-03-arena/)"
-"LAMBADA"           ,"Paperno et al." ,"https://arxiv.org/abs/1606.06031"         ,"The LAMBADA evaluates the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. (Source: https://huggingface.co/datasets/lambada)"
-"TriviaQA"          ,"Joshi et al."   ,"https://arxiv.org/abs/1705.03551v2"       ,"We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. (Source: https://arxiv.org/abs/1705.03551v2)"

data/leaderboard.csv

@@ -1,20 +0,0 @@
-Model Name              ,Chatbot Arena Elo ,LAMBADA (zero-shot) ,TriviaQA (zero-shot)
-alpaca-13b              ,             1008 ,                    ,                
-cerebras-7b             ,                  ,              0.636 ,               0.141
-cerebras-13b            ,                  ,              0.635 ,               0.146
-chatglm-6b              ,              985 ,                    ,                
-dolly-v2-12b            ,              944 ,                    ,                
-fastchat-t5-3b          ,              951 ,                    ,                
-gpt-neox-20b            ,                  ,              0.719 ,               0.347
-gptj-6b                 ,                  ,              0.683 ,               0.234
-koala-13b               ,             1082 ,                    ,                
-llama-7b                ,                  ,              0.738 ,               0.443
-llama-13b               ,              932 ,                    ,                
-mpt-7b                  ,                  ,              0.702 ,               0.343
-opt-7b                  ,                  ,              0.677 ,               0.227
-opt-13b                 ,                  ,              0.692 ,               0.282
-stablelm-base-alpha-7b  ,                  ,              0.533 ,               0.049
-stablelm-tuned-alpha-7b ,              858 ,                    ,                
-vicuna-13b              ,             1169 ,                    ,                
-oasst-pythia-7b         ,                  ,              0.667 ,               0.198
-oasst-pythia-12b        ,             1065 ,              0.704 ,               0.233

data/sources.csv

@@ -1,3 +0,0 @@
-Author   ,Link
-LMSYS    ,https://lmsys.org/blog/2023-05-03-arena/
-MOSAICML ,https://www.mosaicml.com/blog/mpt-7b

streamlit_app.py

@@ -1,25 +1,62 @@
 import pandas as pd
-import requests
 import streamlit as st
+import io
 
-REPO_URL = "https://github.com/LudwigStumpp/llm-leaderboard"
-LEADERBOARD_PATH = "data/leaderboard.csv"
-BENCHMARKS_PATH = "data/benchmarks.csv"
-SOURCES_PATH = "data/sources.csv"
 
+def extract_table_and_format_from_markdown_text(markdown_table: str) -> pd.DataFrame:
+    """Extracts a table from a markdown text and formats it as a pandas DataFrame.
 
-def grab_file_from_repo(repo_url: str, filename: str) -> str:
-    """Grabs a file from a GitHub repository.
+    Args:
+        text (str): Markdown text containing a table.
+
+    Returns:
+        pd.DataFrame: Table as pandas DataFrame.
+    """
+    df = (
+        pd.read_table(io.StringIO(markdown_table), sep="|", header=0, index_col=1)
+        .dropna(axis=1, how="all")  # drop empty columns
+        .iloc[1:]  # drop first row which is the "----" separator of the original markdown table
+        .sort_index(ascending=True)
+        .replace(r"^\s*$", float("nan"), regex=True)
+        .astype(float, errors="ignore")
+    )
+
+    # remove whitespace from column names and index
+    df.columns = df.columns.str.strip()
+    df.index = df.index.str.strip()
+
+    return df
+
+
+def extract_markdown_table_from_multiline(multiline: str, table_headline: str, next_headline_start: str = "#") -> str:
+    """Extracts the markdown table from a multiline string.
 
     Args:
-        repo_url (str): URL of the GitHub repository.
-        filename (str): Name of the file to grab.
+        multiline (str): content of README.md file.
+        table_headline (str): Headline of the table in the README.md file.
+        next_headline_start (str, optional): Start of the next headline. Defaults to "#".
 
     Returns:
-        str: Content of the file.
+        str: Markdown table.
+
+    Raises:
+        ValueError: If the table could not be found.
     """
-    url = repo_url.replace("github.com", "raw.githubusercontent.com") + f"/main/{filename}"
-    return requests.get(url).text
+    # extract everything between the table headline and the next headline
+    table = []
+    start = False
+    for line in multiline.split("\n"):
+        if line.startswith(table_headline):
+            start = True
+        elif line.startswith(next_headline_start):
+            start = False
+        elif start:
+            table.append(line + "\n")
+
+    if len(table) == 0:
+        raise ValueError(f"Could not find table with headline '{table_headline}'")
+
+    return "".join(table)
 
 
 def filter_dataframe(df: pd.DataFrame) -> pd.DataFrame:
@@ -56,7 +93,7 @@ def filter_dataframe(df: pd.DataFrame) -> pd.DataFrame:
 
 
 def setup_basic():
-    title = "LLM-Leaderboard"
+    title = "🏆 LLM-Leaderboard"
 
     st.set_page_config(
         page_title=title,
@@ -73,24 +110,22 @@ def setup_basic():
     )
 
 
-def setup_table():
-    df = pd.read_csv(LEADERBOARD_PATH, index_col=0)
-    df = df.sort_index(ascending=True)
-    df = df.replace(r"^\s*$", float("nan"), regex=True)
-    df = df.astype(float, errors="ignore")
+def setup_leaderboard(readme: str):
+    leaderboard_table = extract_markdown_table_from_multiline(readme, table_headline="## Leaderboard")
+    df_leaderboard = extract_table_and_format_from_markdown_text(leaderboard_table)
 
-    st.markdown("### Leaderboard")
-    st.dataframe(filter_dataframe(df))
+    st.markdown("## Leaderboard")
+    st.dataframe(filter_dataframe(df_leaderboard))
 
 
-def setup_benchmarks():
-    df = pd.read_csv(BENCHMARKS_PATH, index_col=0)
-    df = df.sort_index(ascending=True)
+def setup_benchmarks(readme: str):
+    benchmarks_table = extract_markdown_table_from_multiline(readme, table_headline="## Benchmarks")
+    df_benchmarks = extract_table_and_format_from_markdown_text(benchmarks_table)
 
-    st.markdown("### Covered Benchmarks")
+    st.markdown("## Covered Benchmarks")
 
-    selected_benchmark = st.selectbox("Select a benchmark to learn more:", df.index.unique())
-    df_selected = df.loc[selected_benchmark]
+    selected_benchmark = st.selectbox("Select a benchmark to learn more:", df_benchmarks.index.unique())
+    df_selected = df_benchmarks.loc[selected_benchmark]
     text = [
         f"Name: {selected_benchmark} ",
     ]
@@ -99,14 +134,14 @@ def setup_benchmarks():
     st.markdown("\n".join(text))
 
 
-def setup_sources():
-    df = pd.read_csv(SOURCES_PATH, index_col=0)
-    df = df.sort_index(ascending=True)
+def setup_sources(readme: str):
+    sources_table = extract_markdown_table_from_multiline(readme, table_headline="## Sources")
+    df_sources = extract_table_and_format_from_markdown_text(sources_table)
 
-    st.markdown("### Sources of Above Figures")
+    st.markdown("## Sources of Above Figures")
 
-    selected_source = st.selectbox("Select a source to learn more:", df.index.unique())
-    df_selected = df.loc[selected_source]
+    selected_source = st.selectbox("Select a source to learn more:", df_sources.index.unique())
+    df_selected = df_sources.loc[selected_source]
     text = [
         f"Author: {selected_source} ",
     ]
@@ -126,9 +161,13 @@ def setup_footer():
 
 def main():
     setup_basic()
-    setup_table()
-    setup_benchmarks()
-    setup_sources()
+
+    with open("README.md", "r") as f:
+        readme = f.read()
+
+    setup_leaderboard(readme)
+    setup_benchmarks(readme)
+    setup_sources(readme)
     setup_footer()