improve motivation

Signed-off-by: Yotam-Perlitz <[email protected]>

app.py

@@ -53,7 +53,7 @@ st.markdown(
     """
     The BenchBench leaderboard ranks benchmarks based on their agreement with the *Aggregate Benchmark* – a comprehensive, combined measure of existing benchmark results.
     \n
-    To achive it, we scraped results from multiple benchmarks (citations below) to allow for obtaining benchmark agreement results with a wide range of benchmark using a large set of models.
+    To achive this, we scraped results from multiple benchmarks (citations below) to allow for obtaining benchmark agreement results with a wide range of benchmark using a large set of models.
     \n
     BenchBench is for you if:
     """
@@ -68,11 +68,11 @@ st.markdown(
 
 st.markdown(
     """
-    In our work -- [Benchmark Agreement Testing Done Right](https://arxiv.org/abs/2407.13696), 
+    In our work - [Benchmark Agreement Testing Done Right](https://arxiv.org/abs/2407.13696) and [opensource repo](https://github.com/IBM/benchbench), 
     we standardize BAT and show the importance of its configurations, notably, 
-    the benchmarks we compare to, and the models we use to compare with, check it out int he sidebar.
+    the benchmarks we compare to, and the models we use to compare with (see sidebar).
     \n
-    We show that agreements are best reporesented with the Z Score, the relative agreement of each benchmark to the Aggragate benchmark, as presented below.
+    We also show that agreements are best represented with the relative agreement (Z Score) of each benchmark to the Aggragate benchmark, as presented below in the leaderboard.
     """
 )
 
@@ -325,7 +325,7 @@ reporter = Reporter()
 z_scores = reporter.get_all_z_scores(agreements=agreements, aggragate_name="aggregate")
 z_scores.drop(columns=["n_models_of_corr_with_agg"], inplace=True)
 
-corr_name = f"{'Kendall Tau' if corr_type=='kendall' else 'Per.'} Corr."
+corr_name = f"{'Kendall Tau' if corr_type=='kendall' else 'Per.'} Corr. w/ Agg"
 
 z_scores["z_score"] = z_scores["z_score"].round(2)
 z_scores["corr_with_agg"] = z_scores["corr_with_agg"].round(2)
@@ -699,6 +699,7 @@ with st.expander(label="Citations"):
 
 st.subheader("Benchmark Report Card")
 
+st.markdown("Choose the Benchmark from which you want to get a report.")
 
 benchmarks = allbench.df["scenario"].unique().tolist()
 index_to_use = 1
@@ -742,11 +743,6 @@ fig = px.scatter(
 )
 st.plotly_chart(fig, use_container_width=True)
 
-st.markdown(
-    "BenchBench-Leaderboard complements our study, where we analyzed over 40 prominent benchmarks and introduced standardized practices to enhance the robustness and validity of benchmark evaluations through the [BenchBench Python package](#). "
-    "The BenchBench-Leaderboard serves as a dynamic platform for benchmark comparison and is an essential tool for researchers and practitioners in the language model field aiming to select and utilize benchmarks effectively. "
-)
-
 st.subheader("How did we get the Z Scores?", divider=True)
 
 st.write(r"""
@@ -779,151 +775,65 @@ fig.update_layout(
 # # Plot!
 st.plotly_chart(fig, use_container_width=True)
 
+import streamlit as st
+
 st.subheader("Why should you use the BenchBench Leaderboard?")
 
 st.markdown(
     """
-
-    Current practices in Benchmark Agreement Testing (BAT) often suffer from a lack of standardization and transparency, which can lead to inconsistent results and diminished trust in benchmark evaluations. Several key issues are prevalent in the field:
-
+    Benchmark Agreement Testing (BAT) is crucial for validating new benchmarks and understanding the relationships between existing ones. 
+    However, current BAT practices often lack standardization and transparency, leading to inconsistent results and hindering reliable comparisons. 
+    The BenchBench Leaderboard addresses these challenges by offering a **principled and data-driven approach to benchmark evaluation**. 
+    Let's explore some of the key issues with current BAT practices:
     """
 )
 
 st.markdown(
     """
-    - **Lack of Standard Methodologies:** Unlike other scientific procedures that follow rigorous methodologies, BAT lacks uniform procedures across different studies. Researchers often employ varied criteria for selecting benchmarks and models for comparison, which leads to results that cannot be easily compared or replicated. This variation undermines the reliability of conclusions drawn from BAT and makes it difficult for other researchers to build on existing work.
+    - **Lack of Standard Methodologies:**  BAT lacks standardized procedures for benchmark and model selection, hindering reproducibility and comparability across studies. 
+      Researchers often make arbitrary choices, leading to results that are difficult to interpret and build upon. 
     """
 )
 
 st.image(
     "images/motivation.png",
-    caption="Conclusions depend on the models considered. Kendall-tau correlations between the LMSys Arena benchmark and three other benchmarks: BBH, MMLU, and Alpaca v2. Each group of bars represents the correlation for different sets of top models, specifically the top 5, top 10, and top 15 (overlapping) models (according to the Arena). The results indicate that the degree of agreement between benchmarks varies with the number of top models considered, highlighting that different selections of models can lead to varying conclusions about benchmark agreement.",
+    caption="**Example: Model Selection Impacts BAT Conclusions.** Kendall-tau correlations between the LMSys Arena benchmark and three others demonstrate how agreement varies significantly depending on the subset of models considered. This highlights the need for standardized model selection in BAT.",
     use_column_width=True,
 )
 
 st.markdown(
     """
-    - **Arbitrary Selection of Reference Benchmarks:** One of the most critical decisions in BAT is the choice of reference benchmarks. Currently, this choice is often arbitrary and lacks a clear rationale, influenced by availability or personal preference rather than strategic alignment with the benchmark’s purpose. This can skew the results significantly, as different benchmarks may not be equally representative or relevant to the models being tested.
+    - **Arbitrary Selection of Reference Benchmarks:** The choice of reference benchmarks in BAT is often subjective and lacks a clear rationale. Using different reference benchmarks can lead to widely varying agreement scores, making it difficult to draw robust conclusions about a target benchmark's validity.
     """
 )
 st.markdown(
     """
-    - **Inadequate Model Representation:** BAT frequently relies on a limited subset of models, which may not comprehensively represent the diversity of architectures and training paradigms in modern language models. This selective representation can lead to biased agreement scores that favor certain types of models over others, failing to provide a holistic view of model performance across different benchmarks.
+    - **Inadequate Model Representation:** BAT often relies on a limited set of models that may not adequately represent the diversity of modern language models. This can lead to biased agreement scores that favor certain model types and fail to provide a comprehensive view of benchmark performance.
     """
 )
 
 st.image(
     "images/pointplot_granularity_matters.png",
-    caption="Correlations increase with number of models. Mean correlation (y) between each benchmark (lines) and the rest, given different numbers of models. The Blue and Orange lines are the average of all benchmark pair correlations with models sampled randomly (orange) or in contiguous sets (blue). The shaded lines represents adjacent sampling for the different benchmarks.",
+    caption="**Example: Agreement Varies with Model Range.** Mean correlation between benchmarks shows that agreement tends to increase with the number of models considered and is generally lower for closely ranked models (blue lines). This highlights the importance of considering multiple granularities in BAT.",
     use_column_width=True,
 )
 
 st.markdown(
     """
-    - **Overemphasis on Correlation Metrics:** Current BAT practices tend to over-rely on correlation metrics without adequately considering their limitations and the context of their application. While these metrics can provide useful insights, they are often treated as definitive evidence of agreement without acknowledging that high correlation does not necessarily imply conceptual alignment between benchmarks.
+    - **Overemphasis on Correlation Metrics:** BAT often relies heavily on correlation metrics without fully considering their limitations or the context of their application. While correlation can be informative, it's crucial to remember that high correlation doesn't automatically imply that benchmarks measure the same underlying construct.
     """
 )
 
 st.markdown(
     """
-    To address these issues, there is a critical need for a more structured approach to BAT that includes clear guidelines for benchmark and model selection, a broader consideration of agreement metrics, and an acknowledgment of the evolving nature of technology in this space. By reforming BAT practices, the research community can improve the reliability and utility of benchmarks as tools for evaluating and advancing language models.
+    The BenchBench Leaderboard tackles these challenges by implementing a standardized and transparent approach to BAT, promoting consistency and facilitating meaningful comparisons between benchmarks.
+    By adopting the best practices embedded in the leaderboard, the research community can enhance the reliability and utility of benchmarks for evaluating and advancing language models.
     """
 )
 
 
 st.image(
     "images/ablations.png",
-    caption="Our recommendations substantially reduce the variance of BAT. Ablation analysis for each BAT recommendation separately and their combinations.",
+    caption="**BenchBench's Standardized Approach Reduces Variance.** This ablation study demonstrates that following the best practices implemented in BenchBench significantly reduces the variance of BAT results, leading to more robust and reliable conclusions.",
     use_column_width=True,
 )
-
-
-st.header("The BenchBench package")
-
-st.markdown("""
-### Overview
-
-The BAT package is designed to facilitate benchmark agreement testing for NLP models. It allows users to easily compare multiple models against various benchmarks and generate comprehensive reports on their agreement.
-
-### Installation
-
-To install the BAT package, you can use pip:
-
-```
-pip install bat-package
-```
-
-### Usage Example
-
-Below is a step-by-step example of how to use the BAT package to perform agreement testing.
-
-#### Step 1: Configuration
-
-First, set up the configuration for the tests:
-
-```python
-import pandas as pd
-from bat import Tester, Config, Benchmark, Reporter
-from bat.utils import get_holistic_benchmark
-
-cfg = Config(
-    exp_to_run="example",
-    n_models_taken_list=[0],
-    model_select_strategy_list=["random"],
-    n_exps=10
-)
-```
-
-#### Step 2: Fetch Model Names
-
-Fetch the names of the reference models to be used for scoring:
-
-```python
-tester = Tester(cfg=cfg)
-models_for_benchmark_scoring = tester.fetch_reference_models_names(
-    reference_benchmark=get_holistic_benchmark(), n_models=20
-)
-print(models_for_benchmark_scoring)
-```
-
-#### Step 3: Load and Prepare Benchmark
-
-Load a new benchmark and add an aggregate column:
-
-```python
-newbench_name = "fakebench"
-newbench = Benchmark(
-    pd.read_csv(f"src/bat/assets/{newbench_name}.csv"),
-    data_source=newbench_name,
-)
-newbench.add_aggregate(new_col_name=f"{newbench_name}_mwr")
-```
-
-#### Step 4: Agreement Testing
-
-Perform all-vs-all agreement testing on the new benchmark:
-
-```python
-newbench_agreements = tester.all_vs_all_agreement_testing(newbench)
-reporter = Reporter()
-reporter.draw_agreements(newbench_agreements)
-```
-
-#### Step 5: Extend and Clean Benchmark
-
-Extend the new benchmark with holistic data and clear repeated scenarios:
-
-```python
-allbench = newbench.extend(get_holistic_benchmark())
-allbench.clear_repeated_scenarios(source_to_keep=newbench_name)
-```
-
-#### Step 6: Comprehensive Agreement Testing
-
-Perform comprehensive agreement testing and visualize:
-
-```python
-all_agreements = tester.all_vs_all_agreement_testing(allbench)
-reporter.draw_agreements(all_agreements)
-```
-""")