Add multi-leaderboard support with navigation, enhanced metrics, and correlations

README.md

@@ -8,7 +8,156 @@ sdk_version: 5.27.0
 app_file: app.py
 pinned: false
 license: mit
-short_description: Summary of Results for AutoBench as of 24 April 2025
+short_description: Interactive multi-run leaderboard for AutoBench LLM evaluations with historical navigation
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# AutoBench LLM Leaderboard
+
+Interactive leaderboard for AutoBench, where Large Language Models (LLMs) evaluate and rank responses from other LLMs. This application supports multiple benchmark runs with seamless navigation between different time periods.
+
+## 🌟 Features
+
+### Multi-Run Navigation
+- **📊 Run Selector**: Switch between different AutoBench runs using the dropdown menu
+- **🕐 Historical Data**: View and compare results across different time periods
+- **🔄 Reactive Interface**: All tabs and visualizations update automatically when switching runs
+- **📈 Enhanced Metrics**: Support for evaluation iterations and fail rates in newer runs
+
+### Comprehensive Analysis
+- **Overall Ranking**: Model performance with AutoBench scores, costs, latency, and reliability metrics
+- **Benchmark Comparison**: Correlations with Chatbot Arena, AAI Index, and MMLU benchmarks
+- **Performance Plots**: Interactive scatter plots showing cost vs. performance trade-offs
+- **Cost & Latency Analysis**: Detailed breakdown by domain and response time percentiles
+- **Domain Performance**: Model rankings across specific knowledge areas
+
+### Dynamic Features
+- **📊 Benchmark Correlations**: Displays correlation percentages with other popular benchmarks
+- **💰 Cost Conversion**: Automatic conversion to cents for better readability
+- **⚡ Performance Metrics**: Average and P99 latency measurements
+- **🎯 Fail Rate Tracking**: Model reliability metrics (for supported runs)
+- **🔢 Iteration Counts**: Number of evaluations per model (for supported runs)
+
+## 🚀 How to Use
+
+### Navigation
+1. **Select a Run**: Use the dropdown menu at the top to choose between available benchmark runs
+2. **Explore Tabs**: Navigate through different analysis views using the tab interface
+3. **Interactive Tables**: Sort and filter data by clicking on column headers
+4. **Hover for Details**: Get additional information by hovering over chart elements
+
+### Understanding the Data
+- **AutoBench Score**: Higher scores indicate better performance
+- **Cost**: Lower values are better (displayed in cents per response)
+- **Latency**: Lower response times are better (average and P99 percentiles)
+- **Fail Rate**: Lower percentages indicate more reliable models
+- **Iterations**: Number of evaluation attempts per model
+
+## 🔧 Adding New Runs
+
+### Directory Structure
+```
+runs/
+├── run_YYYY-MM-DD/
+│   ├── metadata.json          # Run information and metadata
+│   ├── correlations.json      # Benchmark correlation data
+│   ├── summary_data.csv       # Main leaderboard data
+│   ├── domain_ranks.csv       # Domain-specific rankings
+│   ├── cost_data.csv          # Cost breakdown by domain
+│   ├── avg_latency.csv        # Average latency by domain
+│   └── p99_latency.csv        # P99 latency by domain
+```
+
+### Required Files
+
+#### 1. metadata.json
+```json
+{
+  "run_id": "run_2025-08-14",
+  "title": "AutoBench Run 3 - August 2025",
+  "date": "2025-08-14",
+  "description": "Latest AutoBench run with enhanced metrics",
+  "blog_url": "https://huggingface.co/blog/PeterKruger/autobench-3rd-run",
+  "model_count": 34,
+  "is_latest": true
+}
+```
+
+#### 2. correlations.json
+```json
+{
+  "correlations": {
+    "Chatbot Arena": 82.51,
+    "Artificial Analysis Intelligence Index": 83.74,
+    "MMLU": 71.51
+  },
+  "description": "Correlation percentages between AutoBench scores and other benchmark scores"
+}
+```
+
+#### 3. summary_data.csv
+Required columns:
+- `Model`: Model name
+- `AutoBench`: AutoBench score
+- `Costs (USD)`: Cost per response in USD
+- `Avg Answer Duration (sec)`: Average response time
+- `P99 Answer Duration (sec)`: 99th percentile response time
+
+Optional columns (for enhanced metrics):
+- `Iterations`: Number of evaluation iterations
+- `Fail Rate %`: Percentage of failed responses
+- `LMArena` or `Chatbot Ar.`: Chatbot Arena scores
+- `MMLU-Pro` or `MMLU Index`: MMLU benchmark scores
+- `AAI Index`: Artificial Analysis Intelligence Index scores
+
+### Adding a New Run
+
+1. **Create Directory**: `mkdir runs/run_YYYY-MM-DD`
+2. **Add Data Files**: Copy your CSV files to the new directory
+3. **Create Metadata**: Add `metadata.json` with run information
+4. **Add Correlations**: Create `correlations.json` with benchmark correlations
+5. **Update Previous Run**: Set `"is_latest": false` in the previous latest run's metadata
+6. **Restart App**: The new run will be automatically discovered
+
+### Column Compatibility
+
+The application automatically adapts to different column structures:
+- **Legacy Runs**: Support basic columns (Model, AutoBench, Cost, Latency)
+- **Enhanced Runs**: Include additional metrics (Iterations, Fail Rate %)
+- **Flexible Naming**: Handles variations in benchmark column names
+
+## 🛠️ Development
+
+### Requirements
+- Python 3.8+
+- Gradio 5.27.0+
+- Pandas
+- Plotly
+
+### Installation
+```bash
+pip install -r requirements.txt
+```
+
+### Running Locally
+```bash
+python app.py
+```
+
+### killing all python processes
+```bash
+taskkill /F /IM python.exe 2>/dev/null || echo "No Python processes to kill"
+```
+
+The app will automatically discover available runs and launch on a local port.
+
+## 📊 Data Sources
+
+AutoBench evaluations are conducted using LLM-generated questions across diverse domains, with responses ranked by evaluation LLMs. For more information about the methodology, visit the [AutoBench blog posts](https://huggingface.co/blog/PeterKruger/autobench).
+
+## 📄 License
+
+MIT License - see LICENSE file for details.
+
+---
+
+Check out the [Hugging Face Spaces configuration reference](https://huggingface.co/docs/hub/spaces-config-reference) for deployment options.

app.py

@@ -1,15 +1,206 @@
 import gradio as gr
 import pandas as pd
 import plotly.express as px
-import os # To check if files exist
+import os
+import json
+from typing import Dict, List
 
 # --- Configuration ---
-DATA_DIR = "." # Assume CSV files are in the same directory as app.py
-SUMMARY_FILE = os.path.join(DATA_DIR, "data/summary_data.csv")
-DOMAIN_RANKS_FILE = os.path.join(DATA_DIR, "data/domain_ranks.csv")
-COST_FILE = os.path.join(DATA_DIR, "data/cost_data.csv")
-AVG_LATENCY_FILE = os.path.join(DATA_DIR, "data/avg_latency.csv")
-P99_LATENCY_FILE = os.path.join(DATA_DIR, "data/p99_latency.csv")
+RUNS_DIR = "runs"
+DATA_DIR = "."  # For backward compatibility
+COST_COLUMN_SUMMARY = 'Costs (USD)'
+NEW_COST_COLUMN_SUMMARY = 'Avg Cost ($ Cents)'
+
+# --- Multi-Run Support Functions ---
+def discover_available_runs() -> List[Dict]:
+    """Scan runs directory and return sorted list of available runs with metadata."""
+    runs = []
+    
+    if not os.path.exists(RUNS_DIR):
+        # Fallback to old structure
+        if os.path.exists("data"):
+            return [{
+                "run_id": "legacy",
+                "title": "AutoBench Run 2 - April 2025",
+                "date": "2025-04-25",
+                "description": "Current run data",
+                "blog_url": "https://huggingface.co/blog/PeterKruger/autobench-2nd-run",
+                "model_count": 27,
+                "is_latest": True,
+                "path": "data"
+            }]
+        return []
+    
+    for run_dir in os.listdir(RUNS_DIR):
+        run_path = os.path.join(RUNS_DIR, run_dir)
+        if os.path.isdir(run_path):
+            metadata_path = os.path.join(run_path, "metadata.json")
+            if os.path.exists(metadata_path):
+                try:
+                    with open(metadata_path, 'r') as f:
+                        metadata = json.load(f)
+                    metadata["path"] = run_path
+                    runs.append(metadata)
+                except Exception as e:
+                    print(f"Error loading metadata for {run_dir}: {e}")
+    
+    # Sort by date, newest first
+    runs.sort(key=lambda x: x.get("date", ""), reverse=True)
+    return runs
+
+def load_run_metadata(run_id: str) -> Dict:
+    """Load metadata for a specific run."""
+    runs = discover_available_runs()
+    for run in runs:
+        if run["run_id"] == run_id:
+            return run
+    return {}
+
+def get_run_file_path(run_path: str, filename: str) -> str:
+    """Get the full path to a data file for a specific run."""
+    return os.path.join(run_path, filename)
+
+
+def load_correlations(run_path: str) -> Dict:
+    """Load correlation data for a specific run."""
+    correlations_file = get_run_file_path(run_path, "correlations.json")
+    if os.path.exists(correlations_file):
+        try:
+            with open(correlations_file, 'r') as f:
+                return json.load(f)
+        except Exception as e:
+            print(f"Error loading correlations from {correlations_file}: {e}")
+    return {}
+
+
+def format_correlations_text(correlations_data: Dict) -> str:
+    """Format correlation data into a readable text string."""
+    if not correlations_data or 'correlations' not in correlations_data:
+        return ""
+    
+    correlations = correlations_data['correlations']
+    if not correlations:
+        return ""
+    
+    # Format the correlation text
+    correlation_parts = []
+    for benchmark, percentage in correlations.items():
+        correlation_parts.append(f"{percentage}% with {benchmark}")
+    
+    if correlation_parts:
+        return f"**Benchmark Correlations:** AutoBench features " + ", ".join(correlation_parts) + "."
+    return ""
+
+def load_run_data(run_id: str) -> Dict[str, pd.DataFrame]:
+    """Load all CSV data for a specific run."""
+    runs = discover_available_runs()
+    run_metadata = None
+    
+    for run in runs:
+        if run["run_id"] == run_id:
+            run_metadata = run
+            break
+    
+    if not run_metadata:
+        print(f"Run {run_id} not found")
+        return {}
+    
+    run_path = run_metadata["path"]
+    
+    # Load all data files
+    data = {}
+    file_mapping = {
+        "summary": "summary_data.csv",
+        "domain": "domain_ranks.csv", 
+        "cost": "cost_data.csv",
+        "avg_latency": "avg_latency.csv",
+        "p99_latency": "p99_latency.csv"
+    }
+    
+    for key, filename in file_mapping.items():
+        filepath = get_run_file_path(run_path, filename)
+        data[key] = load_data(filepath)
+        
+    # Process the data (cost conversion, etc.)
+    data = process_run_data(data)
+    
+    # Load correlations
+    correlations = load_correlations(run_path)
+    data["correlations"] = correlations
+    
+    return data
+
+def process_run_data(data: Dict[str, pd.DataFrame]) -> Dict[str, pd.DataFrame]:
+    """Process and clean the loaded data (cost conversion, sorting, etc.)."""
+    df_summary = data.get("summary", pd.DataFrame())
+    df_cost = data.get("cost", pd.DataFrame())
+    
+    # Convert costs to USD cents (existing logic)
+    if not df_summary.empty and COST_COLUMN_SUMMARY in df_summary.columns:
+        df_summary[COST_COLUMN_SUMMARY] = (pd.to_numeric(df_summary[COST_COLUMN_SUMMARY], errors='coerce') * 100).round(3)
+        df_summary.rename(columns={COST_COLUMN_SUMMARY: NEW_COST_COLUMN_SUMMARY}, inplace=True)
+    
+    # Convert cost breakdown data
+    if not df_cost.empty:
+        model_col_name = 'model_name'
+        cost_cols = [col for col in df_cost.columns if col != model_col_name]
+        for col in cost_cols:
+            df_cost[col] = (pd.to_numeric(df_cost[col], errors='coerce') * 100).round(3)
+    
+    # Rename columns and create display dataframes
+    try:
+        df_summary = df_summary.rename(columns={'Model Name': 'Model'})
+        
+        # Summary display table - include new columns if they exist
+        base_cols = ['Model', 'AutoBench', NEW_COST_COLUMN_SUMMARY, 'Avg Answer Duration (sec)', 'P99 Answer Duration (sec)']
+        
+        # Add new columns at the end: Fail Rate % before Iterations
+        if 'Fail Rate %' in df_summary.columns:
+            base_cols.append('Fail Rate %')
+        if 'Iterations' in df_summary.columns:
+            base_cols.append('Iterations')
+            
+        summary_cols_display = [col for col in base_cols if col in df_summary.columns]
+        df_summary_display = df_summary[summary_cols_display].copy()
+        
+        # Benchmark display table - handle both old and new column names
+        benchmark_cols = ['Model', 'AutoBench']
+        
+        # Handle different column name variations
+        chatbot_col = None
+        mmlu_col = None
+        
+        for col in df_summary.columns:
+            if col in ['Chatbot Ar.', 'LMArena']:
+                chatbot_col = col
+            elif col in ['MMLU Index', 'MMLU-Pro']:
+                mmlu_col = col
+        
+        if chatbot_col:
+            benchmark_cols.append(chatbot_col)
+        if 'AAI Index' in df_summary.columns:
+            benchmark_cols.append('AAI Index')
+        if mmlu_col:
+            benchmark_cols.append(mmlu_col)
+            
+        benchmark_cols = [col for col in benchmark_cols if col in df_summary.columns]
+        df_benchmark_display = df_summary[benchmark_cols].copy()
+        
+        # Sort by AutoBench score
+        for df in [df_summary_display, df_benchmark_display]:
+            if 'AutoBench' in df.columns:
+                df['AutoBench'] = pd.to_numeric(df['AutoBench'], errors='coerce')
+                df.sort_values(by='AutoBench', ascending=False, inplace=True)
+        
+        data["summary_display"] = df_summary_display
+        data["benchmark_display"] = df_benchmark_display
+        
+    except Exception as e:
+        print(f"Error processing display data: {e}")
+        data["summary_display"] = df_summary.copy()
+        data["benchmark_display"] = pd.DataFrame()
+    
+    return data
 
 # --- Helper Function to Load Data ---
 def load_data(filepath, separator=','):
@@ -33,281 +224,343 @@ def load_data(filepath, separator=','):
         print(f"Error loading {filepath}: {e}")
         return pd.DataFrame()
 
-# --- Load All Data ---
-print("Loading data...")
-df_summary = load_data(SUMMARY_FILE)
-df_domain = load_data(DOMAIN_RANKS_FILE)
-df_cost = load_data(COST_FILE)
-df_avg_latency = load_data(AVG_LATENCY_FILE)
-df_p99_latency = load_data(P99_LATENCY_FILE)
-print("Data loading complete.")
+# --- Initialize Multi-Run System ---
+print("Discovering available runs...")
+available_runs = discover_available_runs()
+if not available_runs:
+    print("No runs found! Please check the runs/ directory structure.")
+    exit(1)
 
-# --- *** NEW: Convert Costs to USD Cents *** ---
-COST_COLUMN_SUMMARY = 'Costs (USD)' # IMPORTANT: Check this matches your summary_data.csv header EXACTLY
-NEW_COST_COLUMN_SUMMARY = 'Avg Cost ($ Cents)' # This is the new name we'll use
-
-# Convert summary cost
-if not df_summary.empty and COST_COLUMN_SUMMARY in df_summary.columns:
-    df_summary[COST_COLUMN_SUMMARY] = (pd.to_numeric(df_summary[COST_COLUMN_SUMMARY], errors='coerce') * 100).round(3) # <-- ADDED .round(3)
-    df_summary.rename(columns={COST_COLUMN_SUMMARY: NEW_COST_COLUMN_SUMMARY}, inplace=True)
-    print(f"Converted '{COST_COLUMN_SUMMARY}' to $ Cents and renamed to '{NEW_COST_COLUMN_SUMMARY}' in df_summary.")
-else:
-    print(f"Warning: Column '{COST_COLUMN_SUMMARY}' not found in df_summary for conversion.")
-
-# Convert cost breakdown data
-if not df_cost.empty:
-    # IMPORTANT: Check if your model name column in cost_data.csv is 'model_name' or 'Model Name' etc.
-    model_col_name = 'model_name' # Adjust if needed
-    cost_cols = [col for col in df_cost.columns if col != model_col_name]
-    for col in cost_cols:
-        # Handle potential non-numeric data gracefully before multiplying
-        df_cost[col] = (pd.to_numeric(df_cost[col], errors='coerce') * 100).round(3) # <-- ADDED .round(3)
-    print("Converted cost breakdown columns to $ Cents in df_cost.")
-# --- *** End of Cost Conversion *** ---
-
-# Rename columns for clarity if needed (example for summary)
-# Make sure the original names match your CSV headers EXACTLY
-try:
-    df_summary = df_summary.rename(columns={
-        'Model Name': 'Model', # If your CSV uses 'Model Name'
-        # Add other renames here if your CSV headers differ from the target names below
-        # 'Costs (USD)': 'Avg Cost (USD/response)',
-        # 'Avg Answer Duration (sec)': 'Avg Latency (s)',
-        # 'P99 Answer Duration (sec)': 'P99 Latency (s)'
-    })
-    # Select and reorder columns for the main table - REMOVED BENCHMARK COLUMNS
-    summary_cols_display = ['Model', 'AutoBench', NEW_COST_COLUMN_SUMMARY, 'Avg Answer Duration (sec)', 'P99 Answer Duration (sec)']
-    # Filter to only columns that actually exist after loading and renaming
-    summary_cols_display = [col for col in summary_cols_display if col in df_summary.columns]
-    df_summary_display = df_summary[summary_cols_display].copy() # Use .copy() to avoid SettingWithCopyWarning
-
-    # Select columns for the new benchmark comparison table
-    benchmark_cols = ['Model', 'AutoBench', 'Chatbot Ar.', 'AAI Index', 'MMLU Index']
-    benchmark_cols = [col for col in benchmark_cols if col in df_summary.columns] # Filter existing
-    df_benchmark_display = df_summary[benchmark_cols].copy() # Use .copy()
-
-    # Ensure AutoBench score is numeric for sorting BOTH display tables
-    if 'AutoBench' in df_summary_display.columns:
-        df_summary_display['AutoBench'] = pd.to_numeric(df_summary_display['AutoBench'], errors='coerce')
-        df_summary_display.sort_values(by='AutoBench', ascending=False, inplace=True) # Use inplace=True
-    else:
-        print("Warning: 'AutoBench' column not found for sorting summary table.")
-
-    if 'AutoBench' in df_benchmark_display.columns:
-        df_benchmark_display['AutoBench'] = pd.to_numeric(df_benchmark_display['AutoBench'], errors='coerce')
-        df_benchmark_display.sort_values(by='AutoBench', ascending=False, inplace=True) # Use inplace=True
-    else:
-        print("Warning: 'AutoBench' column not found for sorting benchmark table.")
-
-except KeyError as e:
-    print(f"Error preparing display columns: Missing key {e}. Check CSV headers and rename mapping.")
-    df_summary_display = df_summary.copy() # Fallback
-    df_benchmark_display = pd.DataFrame() # Fallback to empty for benchmark table
+# Get the latest run as default
+latest_run = available_runs[0]
+print(f"Found {len(available_runs)} run(s). Latest: {latest_run['title']}")
 
+# Initialize with latest run data
+print("Loading latest run data...")
+current_data = load_run_data(latest_run["run_id"])
+print("Data loading complete.")
+
+# --- Plotting Functions ---
+def create_cost_scatter_plot(data: Dict[str, pd.DataFrame]) -> tuple:
+    """Create the cost vs rank scatter plot."""
+    df_summary = data.get("summary", pd.DataFrame())
+    
+    if df_summary.empty or 'AutoBench' not in df_summary.columns or NEW_COST_COLUMN_SUMMARY not in df_summary.columns:
+        return None, "_(Insufficient data for Rank vs Cost plot)_"
+    
+    plot_df = df_summary.dropna(subset=['AutoBench', NEW_COST_COLUMN_SUMMARY, 'Model']).copy()
+    plot_df[NEW_COST_COLUMN_SUMMARY] = pd.to_numeric(plot_df[NEW_COST_COLUMN_SUMMARY], errors='coerce')
+    plot_df = plot_df.dropna(subset=[NEW_COST_COLUMN_SUMMARY])
+    
+    if plot_df.empty:
+        return None, "_(No valid data for Rank vs Cost plot)_"
+    
+    fig_cost = px.scatter(
+        plot_df,
+        x=NEW_COST_COLUMN_SUMMARY,
+        y="AutoBench",
+        text="Model",
+        log_x=True,
+        title="AutoBench Rank vs. Average Cost per Response ($ Cents - Log Scale)",
+        labels={'AutoBench': 'AutoBench Rank', NEW_COST_COLUMN_SUMMARY: 'Avg Cost ($ Cents) - Log Scale'},
+        hover_data=['Model', 'AutoBench', NEW_COST_COLUMN_SUMMARY, 'Avg Answer Duration (sec)']
+    )
+    fig_cost.update_traces(textposition='top center')
+    fig_cost.update_layout(
+        xaxis_title="Avg Cost ($ Cents) - Log Scale",
+        yaxis_title="AutoBench Rank",
+        width=1000,
+        height=800,
+        xaxis2=dict(
+            overlaying='x',
+            matches='x',
+            side='top',
+            showticklabels=True,
+            showline=True,
+            title=None
+        )
+    )
+    return fig_cost, ""
+
+def create_avg_latency_plot(data: Dict[str, pd.DataFrame]) -> tuple:
+    """Create the average latency vs rank scatter plot."""
+    df_summary = data.get("summary", pd.DataFrame())
+    
+    if df_summary.empty or 'AutoBench' not in df_summary.columns or 'Avg Answer Duration (sec)' not in df_summary.columns:
+        return None, "_(Insufficient data for Rank vs Avg Latency plot)_"
+    
+    plot_df = df_summary.dropna(subset=['AutoBench', 'Avg Answer Duration (sec)', 'Model']).copy()
+    plot_df['Avg Answer Duration (sec)'] = pd.to_numeric(plot_df['Avg Answer Duration (sec)'], errors='coerce')
+    plot_df = plot_df.dropna(subset=['Avg Answer Duration (sec)'])
+    
+    if plot_df.empty:
+        return None, "_(No valid data for Rank vs Avg Latency plot)_"
+    
+    fig_latency = px.scatter(
+        plot_df,
+        x="Avg Answer Duration (sec)",
+        y="AutoBench",
+        text="Model",
+        log_x=True,
+        title="AutoBench Rank vs. Average Latency (Log Scale)",
+        labels={'AutoBench': 'AutoBench Rank', 'Avg Answer Duration (sec)': 'Avg Latency (s) - Log Scale'},
+        hover_data=['Model', 'AutoBench', 'Avg Answer Duration (sec)', NEW_COST_COLUMN_SUMMARY]
+    )
+    fig_latency.update_traces(textposition='top center')
+    fig_latency.update_layout(
+        xaxis_title="Avg Latency (s) - Log Scale", 
+        yaxis_title="AutoBench Rank", 
+        width=1000, 
+        height=800
+    )
+    return fig_latency, ""
+
+def create_p99_latency_plot(data: Dict[str, pd.DataFrame]) -> tuple:
+    """Create the P99 latency vs rank scatter plot."""
+    df_summary = data.get("summary", pd.DataFrame())
+    
+    if df_summary.empty or 'AutoBench' not in df_summary.columns or 'P99 Answer Duration (sec)' not in df_summary.columns:
+        return None, "_(Insufficient data for Rank vs P99 Latency plot)_"
+    
+    plot_df = df_summary.dropna(subset=['AutoBench', 'P99 Answer Duration (sec)', 'Model']).copy()
+    plot_df['P99 Answer Duration (sec)'] = pd.to_numeric(plot_df['P99 Answer Duration (sec)'], errors='coerce')
+    plot_df = plot_df.dropna(subset=['P99 Answer Duration (sec)'])
+    
+    if plot_df.empty:
+        return None, "_(No valid data for Rank vs P99 Latency plot)_"
+    
+    fig_p99 = px.scatter(
+        plot_df,
+        x="P99 Answer Duration (sec)",
+        y="AutoBench",
+        text="Model",
+        log_x=True,
+        title="AutoBench Rank vs. P99 Latency (Log Scale)",
+        labels={'AutoBench': 'AutoBench Rank', 'P99 Answer Duration (sec)': 'P99 Latency (s) - Log Scale'},
+        hover_data=['Model', 'AutoBench', 'P99 Answer Duration (sec)', 'Avg Answer Duration (sec)', NEW_COST_COLUMN_SUMMARY]
+    )
+    fig_p99.update_traces(textposition='top center')
+    fig_p99.update_layout(
+        xaxis_title="P99 Latency (s) - Log Scale",
+        yaxis_title="AutoBench Rank",
+        width=1000,
+        height=800
+    )
+    return fig_p99, ""
+
+def update_leaderboard_data(selected_run_id: str) -> tuple:
+    """Update all leaderboard components when run selection changes."""
+    if not selected_run_id:
+        # Return empty/default values for all outputs
+        empty_df = pd.DataFrame()
+        return (
+            empty_df, empty_df, empty_df, empty_df, empty_df, empty_df,  # DataFrames
+            None, "", None, "", None, "",  # Plots and messages
+            "No run selected", ""  # Info message, correlations text
+        )
+    
+    # Load data for selected run
+    data = load_run_data(selected_run_id)
+    run_metadata = load_run_metadata(selected_run_id)
+    
+    if not data:
+        empty_df = pd.DataFrame()
+        return (
+            empty_df, empty_df, empty_df, empty_df, empty_df, empty_df,
+            None, "Error loading data", None, "Error loading data", None, "Error loading data",
+            f"Error loading run: {selected_run_id}", ""
+        )
+    
+    # Get DataFrames
+    summary_display = data.get("summary_display", pd.DataFrame())
+    benchmark_display = data.get("benchmark_display", pd.DataFrame())
+    cost_df = data.get("cost", pd.DataFrame())
+    avg_latency_df = data.get("avg_latency", pd.DataFrame())
+    p99_latency_df = data.get("p99_latency", pd.DataFrame())
+    domain_df = data.get("domain", pd.DataFrame())
+    
+    # Create rank display (rename AutoBench to Rank for overall ranking tab)
+    overall_rank_display = summary_display.copy()
+    if 'AutoBench' in overall_rank_display.columns:
+        overall_rank_display.rename(columns={'AutoBench': 'Rank'}, inplace=True)
+    
+    # Prepare cost and latency displays with model_name first
+    def prepare_table_display(df, model_col='model_name'):
+        if df.empty:
+            return df
+        if model_col in df.columns:
+            cols = [model_col] + [col for col in df.columns if col != model_col]
+            return df[cols]
+        return df
+    
+    cost_display = prepare_table_display(cost_df)
+    avg_latency_display = prepare_table_display(avg_latency_df)
+    p99_latency_display = prepare_table_display(p99_latency_df)
+    
+    # Prepare domain display
+    domain_display = domain_df.copy()
+    if 'Model Name' in domain_display.columns:
+        cols = ['Model Name'] + [col for col in domain_display.columns if col != 'Model Name']
+        domain_display = domain_display[cols]
+    
+    # Create plots
+    cost_plot, cost_msg = create_cost_scatter_plot(data)
+    avg_latency_plot, avg_latency_msg = create_avg_latency_plot(data)
+    p99_latency_plot, p99_latency_msg = create_p99_latency_plot(data)
+    
+    # Create info message
+    info_msg = f"**Current Run:** {run_metadata.get('title', 'Unknown')} ({run_metadata.get('date', 'Unknown date')})"
+    if 'model_count' in run_metadata:
+        info_msg += f" - {run_metadata['model_count']} models"
+    
+    # Get correlation text
+    correlations_text = format_correlations_text(data.get("correlations", {}))
+    
+    return (
+        overall_rank_display, benchmark_display, cost_display, avg_latency_display, p99_latency_display, domain_display,
+        cost_plot, cost_msg, avg_latency_plot, avg_latency_msg, p99_latency_plot, p99_latency_msg,
+        info_msg, correlations_text
+    )
 
 # --- Build Gradio App ---
 with gr.Blocks(theme=gr.themes.Soft()) as app:
     gr.Markdown("# AutoBench LLM Leaderboard")
     gr.Markdown(
         "Interactive leaderboard for AutoBench, where LLMs rank LLMs' responses. "
-        "Includes performance, cost, and latency metrics."
-        "Data updated on April 25, 2025."
-        "\n\nMore info for this benchmark run: [AutoBench Run 2 Results](https://huggingface.co/blog/PeterKruger/autobench-2nd-run). "
-        "If you want to know more about AutoBench: [AutoBench Release](https://huggingface.co/blog/PeterKruger/autobench)."
-        )
+        "Includes performance, cost, and latency metrics. "
+        "Use the dropdown below to navigate between different benchmark runs."
+    )
+    
+    # --- Navigation Section ---
+    with gr.Row():
+        with gr.Column(scale=3):
+            # Create dropdown choices
+            run_choices = [(f"{run['date']} - {run['title']}", run['run_id']) for run in available_runs]
+            run_selector = gr.Dropdown(
+                choices=run_choices,
+                value=latest_run["run_id"],
+                label="📊 Select AutoBench Run",
+                info="Choose a benchmark run to view its results"
+            )
+        with gr.Column(scale=2):
+            current_run_info = gr.Markdown(
+                f"**Current Run:** {latest_run['title']} ({latest_run['date']})" + 
+                (f" - {latest_run['model_count']} models" if 'model_count' in latest_run else "")
+            )
+    
+    gr.Markdown("---")
 
     # --- Tab 1: Overall Ranking ---
     with gr.Tab("Overall Ranking"):
         gr.Markdown("## Overall Model Performance")
-        # REMOVED benchmark correlations from Markdown
-        gr.Markdown("Models ranked by AutoBench score. Lower cost ($ Cents) and latency (s) are better.")
-        # Check if df_summary_display has data before rendering
-        if not df_summary_display.empty:
-             # Create a copy specifically for this tab's display and rename the column
-             df_overall_rank_display = df_summary_display.copy()
-             if 'AutoBench' in df_overall_rank_display.columns:
-                 df_overall_rank_display.rename(columns={'AutoBench': 'Rank'}, inplace=True)
-
-             gr.DataFrame(
-                df_overall_rank_display, # Pass the renamed DF
-                # Adjust datatype length based on potentially fewer columns
-                datatype=['str'] + ['number'] * (len(df_overall_rank_display.columns) - 1),
-                interactive=True, # Allows sorting
-                # height=600 # Adjust height as needed
-             )
-        else:
-             gr.Markdown("_(Summary data failed to load or is empty. Please check `summary_data.csv`)_")
+        gr.Markdown("Models ranked by AutoBench score. Lower cost ($ Cents), latency (s), and fail rate (%) are better. Iterations shows the number of evaluations per model.")
+        
+        # Add correlations display
+        initial_correlations = format_correlations_text(current_data.get("correlations", {}))
+        correlations_display = gr.Markdown(value=initial_correlations)
+        
+        overall_ranking_table = gr.DataFrame(
+            current_data.get("summary_display", pd.DataFrame()).copy().rename(columns={'AutoBench': 'Rank'}) if 'AutoBench' in current_data.get("summary_display", pd.DataFrame()).columns else current_data.get("summary_display", pd.DataFrame()),
+            interactive=True,
+            label="Overall Rankings"
+        )
 
-    # --- NEW Tab 1.5: Benchmark Comparison ---
+    # --- Tab 2: Benchmark Comparison ---
     with gr.Tab("Benchmark Comparison"):
         gr.Markdown("## Benchmark Comparison")
         gr.Markdown("Comparison of AutoBench scores with other popular benchmarks. AutoBench features 82.51% correlation with Chatbot Arena, 83.74% with Artificial Analysis Intelligence Index, and 71.51% with MMLU. Models sorted by AutoBench score.")
-        if not df_benchmark_display.empty:
-            gr.DataFrame(
-                df_benchmark_display,
-                datatype=['str'] + ['number'] * (len(df_benchmark_display.columns) - 1),
-                interactive=True # Allow sorting
-            )
-        else:
-            gr.Markdown("_(Benchmark comparison data could not be prepared. Check `summary_data.csv` for 'Chatbot Ar.', 'AAI Index', 'MMLU Index' columns.)_")
+        
+        benchmark_comparison_table = gr.DataFrame(
+            current_data.get("benchmark_display", pd.DataFrame()),
+            interactive=True,
+            label="Benchmark Comparison"
+        )
 
-    # --- Tab 2: Performance Plots ---
+    # --- Tab 3: Performance Plots ---
     with gr.Tab("Performance Plots"):
         gr.Markdown("## Performance Visualizations")
         gr.Markdown("Exploring relationships between AutoBench Rank, Latency, and Cost.")
 
-        # Scatter Plot 1 (using summary data)
+        # Scatter Plot 1: Cost vs Rank
         gr.Markdown("### Rank vs. Average Cost") 
-        if not df_summary.empty and 'AutoBench' in df_summary.columns and NEW_COST_COLUMN_SUMMARY in df_summary.columns:   
-            # Filter out rows where essential plot data might be missing
-            plot_df = df_summary.dropna(subset=['AutoBench', NEW_COST_COLUMN_SUMMARY, 'Model']).copy()
-            plot_df[NEW_COST_COLUMN_SUMMARY] = pd.to_numeric(plot_df[NEW_COST_COLUMN_SUMMARY], errors='coerce')
-            plot_df = plot_df.dropna(subset=[NEW_COST_COLUMN_SUMMARY]) # Drop if cost conversion failed
-
-            if not plot_df.empty:
-                fig_cost = px.scatter(
-                    plot_df,
-                    x=NEW_COST_COLUMN_SUMMARY,
-                    y="AutoBench",
-                    text="Model", # Show model name near point
-                    log_x=True, # Use log scale for cost
-                    title="AutoBench Rank vs. Average Cost per Response ($ Cents - Log Scale)",
-                    labels={'AutoBench': 'AutoBench Rank', NEW_COST_COLUMN_SUMMARY: 'Avg Cost ($ Cents) - Log Scale'},
-                    hover_data=['Model', 'AutoBench', NEW_COST_COLUMN_SUMMARY, 'Avg Answer Duration (sec)'] # Show details on hover
-                )
-                fig_cost.update_traces(textposition='top center')
-                fig_cost.update_layout(
-                    xaxis_title="Avg Cost ($ Cents) - Log Scale", # Keep bottom axis title
-                    yaxis_title="AutoBench Rank",
-                    width=1000, # Your existing width
-                    height=800, # Your existing height (if you added it)
-                    # --- ADD THE FOLLOWING ---
-                    xaxis2=dict(
-                        overlaying='x',     # Link to primary x-axis
-                        matches='x',        # Explicitly match primary x-axis properties (like type='log')
-                        side='top',         # Position on top
-                        showticklabels=True,# Show the labels (numbers)
-                        showline=True,      # Explicitly show the axis line itself
-                        title=None          # No title for the top axis
-                    )
-                    # --- END OF ADDITION ---
-                )                
-                gr.Plot(fig_cost)
-            else:
-                gr.Markdown("_(Insufficient valid data for Rank vs Cost plot. Check 'AutoBench' and NEW_COST_COLUMN_SUMMARY columns in `summary_data.csv`)_")
-        else:
-             gr.Markdown("_(Summary data failed to load or essential columns missing for Rank vs Cost plot)_")
+        initial_cost_plot, initial_cost_msg = create_cost_scatter_plot(current_data)
+        cost_plot = gr.Plot(value=initial_cost_plot)
+        cost_plot_msg = gr.Markdown(value=initial_cost_msg)
 
         # Plot 2: Rank vs Average Latency
         gr.Markdown("### Rank vs. Average Latency")
-        if not df_summary.empty and 'AutoBench' in df_summary.columns and 'Avg Answer Duration (sec)' in df_summary.columns:
-            # Filter out rows where essential plot data might be missing
-            plot_df_avg_latency = df_summary.dropna(subset=['AutoBench', 'Avg Answer Duration (sec)', 'Model']).copy()
-            plot_df_avg_latency['Avg Answer Duration (sec)'] = pd.to_numeric(plot_df_avg_latency['Avg Answer Duration (sec)'], errors='coerce')
-            plot_df_avg_latency = plot_df_avg_latency.dropna(subset=['Avg Answer Duration (sec)']) # Drop if conversion failed
-
-            if not plot_df_avg_latency.empty:
-                fig_avg_latency = px.scatter(
-                    plot_df_avg_latency,
-                    x="Avg Answer Duration (sec)",
-                    y="AutoBench",
-                    text="Model",
-                    log_x=True,  # Use log scale for latency - adjust if not desired
-                    title="AutoBench Rank vs. Average Latency (Log Scale)",
-                    labels={'AutoBench': 'AutoBench Rank', 'Avg Answer Duration (sec)': 'Avg Latency (s) - Log Scale'},
-                    hover_data=['Model', 'AutoBench', 'Avg Answer Duration (sec)', NEW_COST_COLUMN_SUMMARY]
-                )
-                fig_avg_latency.update_traces(textposition='top center')
-                fig_avg_latency.update_layout(xaxis_title="Avg Latency (s) - Log Scale", yaxis_title="AutoBench Rank", width=1000, height=800)
-                gr.Plot(fig_avg_latency)
-            else:
-                gr.Markdown("_(Insufficient valid data for Rank vs Avg Latency plot. Check 'AutoBench' and 'Avg Answer Duration (sec)' columns in `summary_data.csv`)_")
-        else:
-             gr.Markdown("_(Summary data failed to load or essential columns missing for Rank vs Avg Latency plot)_")
-
+        initial_avg_latency_plot, initial_avg_latency_msg = create_avg_latency_plot(current_data)
+        avg_latency_plot = gr.Plot(value=initial_avg_latency_plot)
+        avg_latency_plot_msg = gr.Markdown(value=initial_avg_latency_msg)
 
         # Plot 3: Rank vs P99 Latency
         gr.Markdown("### Rank vs. P99 Latency")
-        if not df_summary.empty and 'AutoBench' in df_summary.columns and 'P99 Answer Duration (sec)' in df_summary.columns:
-            # Filter out rows where essential plot data might be missing
-            plot_df_p99_latency = df_summary.dropna(subset=['AutoBench', 'P99 Answer Duration (sec)', 'Model']).copy()
-            plot_df_p99_latency['P99 Answer Duration (sec)'] = pd.to_numeric(plot_df_p99_latency['P99 Answer Duration (sec)'], errors='coerce')
-            plot_df_p99_latency = plot_df_p99_latency.dropna(subset=['P99 Answer Duration (sec)']) # Drop if conversion failed
-
-            if not plot_df_p99_latency.empty:
-                fig_p99_latency = px.scatter(
-                    plot_df_p99_latency,
-                    x="P99 Answer Duration (sec)",
-                    y="AutoBench",
-                    text="Model",
-                    log_x=True,  # Use log scale for latency - adjust if not desired
-                    title="AutoBench Rank vs. P99 Latency (Log Scale)",
-                    labels={'AutoBench': 'AutoBench Rank', 'P99 Answer Duration (sec)': 'P99 Latency (s) - Log Scale'},
-                    hover_data=['Model', 'AutoBench', 'P99 Answer Duration (sec)', 'Avg Answer Duration (sec)', NEW_COST_COLUMN_SUMMARY]
-                )
-                fig_p99_latency.update_traces(textposition='top center')
-                fig_p99_latency.update_layout(xaxis_title="P99 Latency (s) - Log Scale", yaxis_title="AutoBench Rank", width=1000, height=800)
-                gr.Plot(fig_p99_latency)
-            else:
-                gr.Markdown("_(Insufficient valid data for Rank vs P99 Latency plot. Check 'AutoBench' and 'P99 Answer Duration (sec)' columns in `summary_data.csv`)_")
-        else:
-             gr.Markdown("_(Summary data failed to load or essential columns missing for Rank vs P99 Latency plot)_")
+        initial_p99_latency_plot, initial_p99_latency_msg = create_p99_latency_plot(current_data)
+        p99_latency_plot = gr.Plot(value=initial_p99_latency_plot)
+        p99_latency_plot_msg = gr.Markdown(value=initial_p99_latency_msg)
 
-    # --- Tab 3: Cost & Latency Analysis ---
+    # --- Tab 4: Cost & Latency Analysis ---
     with gr.Tab("Cost & Latency Analysis"):
         gr.Markdown("## Performance vs. Cost/Latency Trade-offs")
 
         # Cost Breakdown Table
-        gr.Markdown("### Cost Breakdown per Domain ($ Cents/Response)") # <-- MODIFIED
-        if not df_cost.empty:
-            # Make model name the first column if it exists
-            if 'model_name' in df_cost.columns:
-                 cols = ['model_name'] + [col for col in df_cost.columns if col != 'model_name']
-                 df_cost_display = df_cost[cols]
-            else:
-                 df_cost_display = df_cost # Use as is if 'model_name' isn't found
-            gr.DataFrame(df_cost_display, interactive=True)
+        gr.Markdown("### Cost Breakdown per Domain ($ Cents/Response)")
+        cost_df = current_data.get("cost", pd.DataFrame())
+        if not cost_df.empty and 'model_name' in cost_df.columns:
+            cols = ['model_name'] + [col for col in cost_df.columns if col != 'model_name']
+            initial_cost_display = cost_df[cols]
         else:
-             gr.Markdown("_(Cost breakdown data failed to load or is empty. Please check `cost_data.csv`)_")
+            initial_cost_display = cost_df
+        cost_breakdown_table = gr.DataFrame(
+            value=initial_cost_display,
+            interactive=True,
+            label="Cost Breakdown"
+        )
 
         # Latency Breakdown Tables
         gr.Markdown("### Average Latency Breakdown per Domain (Seconds)")
-        if not df_avg_latency.empty:
-            if 'model_name' in df_avg_latency.columns:
-                 cols = ['model_name'] + [col for col in df_avg_latency.columns if col != 'model_name']
-                 df_avg_latency_display = df_avg_latency[cols]
-            else:
-                 df_avg_latency_display = df_avg_latency
-            gr.DataFrame(df_avg_latency_display, interactive=True)
+        avg_latency_df = current_data.get("avg_latency", pd.DataFrame())
+        if not avg_latency_df.empty and 'model_name' in avg_latency_df.columns:
+            cols = ['model_name'] + [col for col in avg_latency_df.columns if col != 'model_name']
+            initial_avg_latency_display = avg_latency_df[cols]
         else:
-             gr.Markdown("_(Average latency data failed to load or is empty. Please check `avg_latency.csv`)_")
+            initial_avg_latency_display = avg_latency_df
+        avg_latency_breakdown_table = gr.DataFrame(
+            value=initial_avg_latency_display,
+            interactive=True,
+            label="Average Latency Breakdown"
+        )
 
         gr.Markdown("### P99 Latency Breakdown per Domain (Seconds)")
-        if not df_p99_latency.empty:
-            if 'model_name' in df_p99_latency.columns:
-                 cols = ['model_name'] + [col for col in df_p99_latency.columns if col != 'model_name']
-                 df_p99_latency_display = df_p99_latency[cols]
-            else:
-                 df_p99_latency_display = df_p99_latency
-            gr.DataFrame(df_p99_latency_display, interactive=True)
+        p99_latency_df = current_data.get("p99_latency", pd.DataFrame())
+        if not p99_latency_df.empty and 'model_name' in p99_latency_df.columns:
+            cols = ['model_name'] + [col for col in p99_latency_df.columns if col != 'model_name']
+            initial_p99_latency_display = p99_latency_df[cols]
         else:
-            gr.Markdown("_(P99 latency data failed to load or is empty. Please check `p99_latency.csv`)_")
+            initial_p99_latency_display = p99_latency_df
+        p99_latency_breakdown_table = gr.DataFrame(
+            value=initial_p99_latency_display,
+            interactive=True,
+            label="P99 Latency Breakdown"
+        )
 
 
-    # --- Tab 4: Domain Performance ---
+    # --- Tab 5: Domain Performance ---
     with gr.Tab("Domain Performance"):
         gr.Markdown("## Performance Across Different Domains")
         gr.Markdown("Model ranks within specific knowledge or task areas. Higher is better.")
-        if not df_domain.empty:
-            if 'Model Name' in df_domain.columns:
-                 # Attempt to make Model Name first col
-                 cols = ['Model Name'] + [col for col in df_domain.columns if col != 'Model Name']
-                 df_domain_display = df_domain[cols]
-            else:
-                 df_domain_display = df_domain # Use as is
-            gr.DataFrame(df_domain_display, interactive=True)
+        
+        domain_df = current_data.get("domain", pd.DataFrame())
+        if not domain_df.empty and 'Model Name' in domain_df.columns:
+            cols = ['Model Name'] + [col for col in domain_df.columns if col != 'Model Name']
+            initial_domain_display = domain_df[cols]
         else:
-            gr.Markdown("_(Domain ranks data failed to load or is empty. Please check `domain_ranks.csv`)_")
+            initial_domain_display = domain_df
+        domain_performance_table = gr.DataFrame(
+            value=initial_domain_display,
+            interactive=True,
+            label="Domain Performance"
+        )
 
     # --- Tab 5: About ---
     with gr.Tab("About AutoBench"):
@@ -339,8 +592,36 @@ with gr.Blocks(theme=gr.themes.Soft()) as app:
 
         **Disclaimer:** Benchmark results provide one perspective on model capabilities. Performance can vary based on specific tasks, prompts, and API conditions. Costs are estimates and subject to change by providers. Latency depends on server load and geographic location.
         """)
+    
+    # --- Event Handlers ---
+    # Set up reactive data loading when run selection changes
+    run_selector.change(
+        fn=update_leaderboard_data,
+        inputs=[run_selector],
+        outputs=[
+            overall_ranking_table,
+            benchmark_comparison_table, 
+            cost_breakdown_table,
+            avg_latency_breakdown_table,
+            p99_latency_breakdown_table,
+            domain_performance_table,
+            cost_plot,
+            cost_plot_msg,
+            avg_latency_plot,
+            avg_latency_plot_msg,
+            p99_latency_plot,
+            p99_latency_plot_msg,
+            current_run_info,
+            correlations_display
+        ]
+    )
+    
+    # Note: Initial data is already loaded via value parameters above
 
 # --- Launch the App ---
 print("Launching Gradio app...")
-app.launch()
+app.launch(
+    favicon_path="static/manifest.json" if os.path.exists("static/manifest.json") else None,
+    show_error=True
+)
 print("Gradio app launched.")

runs/run_2025-04-25/correlations.json

@@ -0,0 +1,8 @@
+{
+  "correlations": {
+    "Chatbot Arena": 82.51,
+    "Artificial Analysis Intelligence Index": 83.74,
+    "MMLU-Plus": 71.51
+  },
+  "description": "Correlation percentages between AutoBench scores and other benchmark scores"
+}

runs/run_2025-04-25/metadata.json

@@ -0,0 +1,9 @@
+{
+  "run_id": "run_2025-04-25",
+  "title": "AutoBench Run 2 - April 2025",
+  "date": "2025-04-25",
+  "description": "Second major AutoBench run with o4-mini, GPT-4.1-mini, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet:thinking, etc.",
+  "blog_url": "https://huggingface.co/blog/PeterKruger/autobench-2nd-run",
+  "model_count": 27,
+  "is_latest": false
+}

runs/run_2025-08-14/avg_latency.csv

@@ -0,0 +1,34 @@
+model_name,coding,creative writing,current news,general culture,grammar,history,logics,math,science,technology,Average (All Topics)
+claude-3.5-haiku,17.2096,9.5021,13.1592,11.2295,9.2722,13.7099,7.9069,9.5013,10.8395,11.2754,11.51902452
+claude-opus-4-1,97.988,31.7501,44.9092,36.3513,30.166,53.8647,38.4643,32.1546,48.1318,52.2831,48.62490598
+claude-sonnet-4,75.4533,18.119,34.977,23.5653,18.9687,35.1883,19.9602,24.1877,36.5889,34.5893,33.66639032
+deepSeek-R1-0528,238.2024,38.1537,62.9681,48.2227,51.7076,61.9942,302.2976,271.5234,62.5406,66.1298,119.174235
+deepSeek-V3-0324,55.0977,17.8165,32.0701,23.041,24.2812,31.001,88.9007,51.2641,29.3776,43.4225,40.30336432
+gemini-2.5-flash,95.8841,16.7289,32.1003,18.3471,24.8943,29.1208,133.8567,52.6201,31.5185,36.1925,48.7078753
+gemini-2.5-flash-lite,26.1777,2.8563,5.6249,3.6215,3.9845,6.8374,86.5258,41.5112,6.4054,9.5453,19.15509939
+gemini-2.5-pro,83.393,29.2572,49.1594,36.6978,36.9932,47.2089,166.9207,93.4631,49.4694,52.5929,65.03115036
+gemma-3-27b-it,62.2708,12.4686,27.5139,18.3155,19.2946,25.0176,24.3704,40.4719,35.3548,23.2773,29.7215
+GLM-4.5,147.0394,20.9164,43.4854,31.8055,38.4103,42.5121,224.8967,165.5218,43.1608,49.7589,80.74437254
+GLM-4.5-Air,105.9142,13.7172,31.2173,19.5371,45.6936,30.9208,206.3031,140.9465,38.1786,44.156,68.34050587
+gpt-4.1,58.0164,11.4923,24.717,14.4706,15.9619,23.1579,80.4132,46.8127,21.9879,23.3983,32.86274006
+gpt-5,120.9672,50.0373,78.8355,55.0956,56.6508,73.3966,156.5955,151.2006,83.2029,76.5455,89.99818067
+gpt-5-mini,102.5153,25.7197,48.4212,31.5236,35.4217,44.406,159.7168,84.2748,52.1692,56.7156,65.89701176
+gpt-5-nano,98.4218,38.2174,52.23,38.2242,48.2844,54.4789,136.6573,86.4832,48.3263,53.912,66.4959839
+gpt-oss-120b,49.9796,11.3648,28.9885,18.2911,14.4024,22.6241,25.4515,36.6005,27.8547,28.8135,27.00733404
+grok-3-mini,45.7707,12.9635,17.7188,13.6096,20.4385,20.4502,40.5232,32.54,25.8136,23.9,26.12147499
+grok-4,92.6961,28.4581,49.7663,33.7181,41.4523,48.4394,138.4335,124.3858,48.7183,50.103,60.95525411
+Kimi-K2-Instruct,69.4739,29.2635,65.4032,45.9564,46.6415,75.2439,114.3645,58.6696,72.5513,57.1711,65.0222057
+llama-3_1-Nemotron-Ultra-253B-v1,88.5095,28.8905,29.2174,22.7454,32.2121,36.1473,174.681,134.8929,38.2931,32.3295,61.53657957
+Llama-3_3-Nemotron-Super-49B-v1,46.3574,15.1811,28.7779,22.9057,21.5446,27.2551,67.4261,33.0439,31.8722,25.0215,32.63831081
+llama-4-maverick,21.5707,4.76,7.8189,5.8389,6.1681,8.3187,21.0604,11.3275,8.7158,6.8338,10.65014104
+llama-4-Scout-17B-16E-Instruct,20.2177,6.4339,9.1924,7.529,7.9851,9.8133,11.1171,13.0721,12.2545,8.2341,10.86684261
+magistral-small-2506,11.4551,7.1952,7.5178,5.7532,6.1051,8.6988,79.2722,37.9617,7.0901,7.2786,17.53939687
+mistral-large-2411,51.7739,14.2815,23.6025,17.3517,13.3736,25.8432,18.1355,24.6234,25.1484,21.9005,24.36368715
+nova-lite-v1,7.1014,4.7882,5.846,4.7061,4.3402,5.5093,4.8806,4.861,4.9134,5.3275,5.288625128
+nova-pro-v1,12.4833,7.5838,7.52,5.6658,6.5254,7.3418,5.8645,7.2712,6.6838,6.7792,7.528192069
+o3,70.4202,25.9427,46.4619,29.613,26.5293,42.6644,194.8085,112.9362,41.2548,46.7826,63.89621339
+o4-mini,56.98,16.3976,26.7274,19.8134,21.7084,23.2641,116.3436,41.5349,28.6513,26.4233,39.05469579
+phi-4,10.8373,5.9498,6.7808,6.3085,5.9981,7.1457,7.7569,12.1669,7.0431,7.4096,7.744667446
+Qwen3-14B,67.7342,19.9239,31.3204,32.178,32.2363,31.2024,197.4205,132.5492,40.1656,31.5221,61.11544056
+Qwen3-235B-A22B-Thinking-2507,180.1429,33.7386,65.2237,45.3004,54.6603,53.7109,122.6611,138.2941,60.427,72.9058,78.79346155
+Qwen3-30B-A3B,119.9895,27.907,34.7837,25.8461,38.8109,37.0577,204.2344,157.6709,38.1969,41.6341,72.64171253

runs/run_2025-08-14/correlations.json

@@ -0,0 +1,8 @@
+{
+  "correlations": {
+    "LMArena": 86.85,
+    "Artificial Analysis Intelligence Index": 92.17,
+    "MMLU": 75.44
+  },
+  "description": "Correlation percentages between AutoBench scores and other benchmark scores"
+}

runs/run_2025-08-14/cost_data.csv

@@ -0,0 +1,34 @@
+model_name,coding,creative writing,current news,general culture,grammar,history,logics,math,science,technology,Average (All Topics)
+claude-3.5-haiku,0.01433364,0.00596023,0.00783403,0.00687656,0.00700846,0.00828981,0.00670457,0.0086506,0.00762436,0.00770914,0.008262832
+claude-opus-4-1,0.1854432,0.058055,0.07953592,0.07093359,0.06338077,0.091515,0.07756427,0.08967857,0.08113929,0.08624571,0.091256434
+claude-sonnet-4,0.03735534,0.00986831,0.01471113,0.01164281,0.01133654,0.01593981,0.01523071,0.0181211,0.01523195,0.01548664,0.017099466
+deepSeek-R1-0528,0.01314631,0.00240864,0.00361531,0.0028925,0.00321982,0.00348657,0.01360484,0.01570098,0.00335462,0.00345558,0.006382309
+deepSeek-V3-0324,0.00179803,0.00076659,0.0010843,0.00076717,0.0009645,0.00102303,0.00176716,0.00161885,0.00101519,0.00100663,0.00119639
+gemini-2.5-flash,0.01009034,0.00149757,0.00386032,0.00235404,0.00327702,0.00373479,0.00399646,0.00651437,0.0042002,0.00425649,0.004512314
+gemini-2.5-flash-lite,0.00162929,0.00022925,0.00043297,0.0002676,0.00036001,0.00074135,0.00284197,0.00300094,0.00046755,0.00076045,0.001052718
+gemini-2.5-pro,0.0277487,0.0072651,0.01509773,0.01122996,0.01217529,0.01489145,0.01518476,0.02214276,0.0161067,0.01515628,0.015866994
+gemma-3-27b-it,0.00044567,0.00017776,0.00028088,0.00018688,0.00021991,0.00027132,0.00025736,0.0004104,0.00026623,0.00026777,0.00028134
+GLM-4.5,0.01215121,0.00224537,0.00373941,0.00300118,0.00374437,0.0037643,0.01209785,0.01441371,0.00379349,0.00407664,0.00629521
+GLM-4.5-Air,0.0064693,0.00115415,0.00196991,0.0013692,0.00310372,0.00188536,0.00667506,0.00810732,0.00243774,0.00270236,0.003611243
+gpt-4.1,0.0127812,0.00475815,0.00799795,0.00526612,0.00723564,0.00747097,0.01396449,0.01694627,0.007576,0.0074939,0.009144648
+gpt-5,0.06007105,0.02747785,0.03786611,0.03067602,0.03328859,0.03684483,0.0619945,0.07589079,0.03760841,0.03823268,0.043676351
+gpt-5-mini,0.00837399,0.00360921,0.00562431,0.00423233,0.00503336,0.00521421,0.00885311,0.01098604,0.00557968,0.00584673,0.006324841
+gpt-5-nano,0.00322071,0.00188066,0.00179711,0.00158044,0.00258222,0.00217255,0.00350634,0.00381871,0.00171226,0.00193456,0.002414388
+gpt-oss-120b,0.00204009,0.00075235,0.0013959,0.00103317,0.00088107,0.00124919,0.00158693,0.00206557,0.00120019,0.00136573,0.001361942
+grok-3-mini,0.00135005,0.00053652,0.000744,0.00052427,0.00074544,0.00073632,0.00146137,0.00130681,0.00070784,0.00073552,0.000895661
+grok-4,0.0510268,0.01488985,0.02296405,0.01615106,0.02757851,0.02201092,0.05122846,0.05969514,0.02105693,0.02263429,0.029202342
+Kimi-K2-Instruct,0.0028794,0.00174574,0.00235794,0.00221802,0.00168557,0.00288834,0.00275612,0.00216027,0.00241402,0.00230324,0.002379296
+llama-3_1-Nemotron-Ultra-253B-v1,0.00518958,0.00210383,0.00166994,0.00144186,0.00232614,0.00206072,0.00771408,0.0084011,0.00209152,0.00188066,0.003451924
+Llama-3_3-Nemotron-Super-49B-v1,0.00061338,0.00029891,0.0003921,0.00035633,0.00037593,0.00042511,0.00067755,0.00059406,0.00041489,0.00037202,0.000455359
+llama-4-maverick,0.00073583,0.0003091,0.00044116,0.00034576,0.0004487,0.00045704,0.00067601,0.00068581,0.00041993,0.00040759,0.000496574
+llama-4-Scout-17B-16E-Instruct,0.00052871,0.00029487,0.00041163,0.00037054,0.00037521,0.00042884,0.00043854,0.00049328,0.00037896,0.00036888,0.000410484
+magistral-small-2506,0.00209764,0.00099426,0.00100711,0.00073844,0.00105969,0.00101964,0.00613572,0.00519953,0.0010145,0.00099893,0.0019781
+mistral-large-2411,0.00963771,0.004614,0.00608889,0.00459562,0.00456308,0.00608816,0.00568067,0.0072976,0.00577418,0.00586219,0.006101138
+nova-lite-v1,0.00028518,0.00013034,0.0001725,0.00012972,0.00014658,0.00016856,0.00020718,0.0002494,0.00016274,0.00015855,0.00018322
+nova-pro-v1,0.00286661,0.00146443,0.00169594,0.00126732,0.00182435,0.00161185,0.00167262,0.00247683,0.00151393,0.00145992,0.001800498
+o3,0.01830136,0.00943369,0.01515805,0.01014862,0.01051015,0.0133993,0.03758871,0.05157964,0.01258745,0.01355057,0.018504711
+o4-mini,0.01033712,0.00629079,0.00714936,0.00606561,0.00734834,0.00687444,0.01597855,0.01248218,0.0066754,0.00744339,0.008704364
+phi-4,0.00034647,0.00019431,0.00021457,0.00017458,0.00019261,0.00022752,0.00024432,0.00038072,0.00021456,0.00020954,0.000240436
+Qwen3-14B,0.00098626,0.00036762,0.00045077,0.00043898,0.00056254,0.00047318,0.0018592,0.00189101,0.0005008,0.00047842,0.000789184
+Qwen3-235B-A22B-Thinking-2507,0.00447462,0.00278202,0.00439546,0.003571,0.0041131,0.00377739,0.00673892,0.00660976,0.00369867,0.00371842,0.00416518
+Qwen3-30B-A3B,0.00120847,0.00039109,0.00046062,0.00038347,0.00054571,0.00046903,0.00150189,0.00185211,0.00046031,0.0004507,0.000763337

runs/run_2025-08-14/domain_ranks.csv

@@ -0,0 +1,34 @@
+model_name,coding,creative writing,current news,general culture,grammar,history,logics,math,science,technology,Average (All Topics)
+claude-3.5-haiku,3.4733,3.8572,3.7443,3.995,3.7371,3.8701,2.8162,2.7809,3.736,3.8134,3.586292962
+claude-opus-4-1,4.2931,4.5071,4.3035,4.4302,4.2258,4.441,3.5738,3.5758,4.4164,4.4833,4.239909895
+claude-sonnet-4,4.1894,4.3647,4.3026,4.3258,4.2532,4.3497,3.5475,3.4817,4.3953,4.3884,4.171968576
+deepSeek-R1-0528,3.9481,4.3147,4.3493,4.4062,4.3139,4.4007,3.5649,3.6287,4.3876,4.4032,4.18112906
+deepSeek-V3-0324,3.8756,4.1946,4.0724,4.0561,4.0467,4.0888,3.3667,3.4401,4.1442,4.0976,3.945669087
+gemini-2.5-flash,4.4225,4.1694,4.3729,4.3287,4.3781,4.4165,4.0091,4.2283,4.4283,4.3877,4.32099389
+gemini-2.5-flash-lite,4.1092,4.1468,4.0836,4.1655,4.0513,4.1563,3.3399,3.546,4.2419,4.1944,4.017202952
+gemini-2.5-pro,4.5248,4.3916,4.4224,4.4873,4.4508,4.477,4.0868,4.2425,4.5154,4.516,4.416904571
+gemma-3-27b-it,3.5655,4.2891,4.112,4.1677,3.973,4.1788,3.0395,3.0841,4.1903,4.1951,3.881640548
+GLM-4.5,3.8921,4.2566,4.3827,4.4219,4.3206,4.4692,3.4848,3.4666,4.4781,4.4931,4.176558031
+GLM-4.5-Air,3.8049,3.993,4.1921,4.193,4.0296,4.2921,3.4191,3.3372,4.2467,4.2717,3.98464018
+gpt-4.1,4.2419,4.3243,4.2551,4.186,4.2089,4.2291,3.7372,3.7882,4.2677,4.3183,4.165890881
+gpt-5,4.5821,4.5178,4.5866,4.6213,4.359,4.6423,4.2072,4.1718,4.6466,4.6634,4.511567341
+gpt-5-mini,4.545,4.5442,4.4995,4.5239,4.442,4.5635,4.1788,4.2467,4.6203,4.6302,4.486571107
+gpt-5-nano,4.4143,4.3848,4.3524,4.403,4.3042,4.4475,3.88,4.134,4.3676,4.5159,4.325926956
+gpt-oss-120b,4.612,4.4161,4.5248,4.4229,4.4453,4.5703,4.162,4.2461,4.6282,4.634,4.479287977
+grok-3-mini,4.0184,4.1848,4.1622,4.2055,4.1589,4.2079,3.5142,3.4923,4.2585,4.263,4.055940505
+grok-4,4.3075,4.3543,4.3302,4.3823,4.3368,4.3983,4.0111,3.8461,4.4058,4.4033,4.308828831
+Kimi-K2-Instruct,4.119,4.5362,4.2929,4.3457,4.1929,4.5009,3.3958,3.5199,4.4592,4.4092,4.177138663
+llama-3_1-Nemotron-Ultra-253B-v1,3.7715,4.268,4.1844,4.2204,4.0862,4.2596,3.4729,3.434,4.2285,4.2351,4.020264345
+Llama-3_3-Nemotron-Super-49B-v1,3.8343,4.0472,4.0449,4.1217,3.9872,4.1624,3.0436,3.2692,4.1175,4.1322,3.883310532
+llama-4-maverick,3.5884,3.7355,3.7029,3.7833,3.8314,3.8241,3.1303,3.0961,3.8317,3.7879,3.640194992
+llama-4-Scout-17B-16E-Instruct,3.3725,3.8585,3.6597,3.8316,3.8462,3.8386,3.0535,3.0033,3.8224,3.8368,3.614481399
+magistral-small-2506,3.7448,3.2301,3.9232,3.8931,3.8409,3.9707,3.2159,3.2791,4.0028,3.941,3.713933337
+mistral-large-2411,3.4967,3.9724,3.8286,3.9329,3.7992,3.919,3.1123,3.1132,3.9871,3.9484,3.714675671
+nova-lite-v1,3.322,3.7767,3.7078,3.7683,3.5057,3.7565,2.9917,2.9507,3.8237,3.7503,3.538201832
+nova-pro-v1,3.3633,3.8403,3.5455,3.723,3.5315,3.633,2.9588,2.8514,3.7492,3.6051,3.490835422
+o3,4.4254,4.2963,4.5626,4.4951,4.3871,4.5722,3.9576,4.1618,4.579,4.6123,4.409851586
+o4-mini,4.3056,4.2389,4.3587,4.3787,4.2565,4.3495,3.8986,3.8362,4.4838,4.5318,4.27410734
+phi-4,3.4825,3.9651,3.7302,3.849,3.6624,3.8171,3.1286,3.1995,3.8704,3.8465,3.657791802
+Qwen3-14B,3.833,4.2818,4.1127,4.0911,4.0235,4.149,3.4441,3.3376,4.2349,4.1606,3.976245179
+Qwen3-235B-A22B-Thinking-2507,4.3112,4.4415,4.4798,4.5551,4.4452,4.5117,3.8366,3.9413,4.4376,4.5403,4.394399183
+Qwen3-30B-A3B,3.8019,4.1652,4.0949,4.1569,3.9359,4.145,3.4857,3.3944,4.1322,4.1468,3.952481327

runs/run_2025-08-14/metadata.json

@@ -0,0 +1,9 @@
+{
+  "run_id": "run_2025-08-14",
+  "title": "AutoBench Run 3 - August 2025",
+  "date": "2025-08-14",
+  "description": "Latest AutoBench run with enhanced metrics including evaluation iterations and fail rates",
+  "blog_url": "https://huggingface.co/blog/PeterKruger/autobench-3rd-run",
+  "model_count": 34,
+  "is_latest": true
+}

runs/run_2025-08-14/p99_latency.csv

@@ -0,0 +1,34 @@
+model_name,coding,creative writing,current news,general culture,grammar,history,logics,math,science,technology,Average (All Topics)
+GLM-4.5,336.5838,46.1539,115.4247,61.5081,183.8738,94.3667,955.208,354.6674,147.6845,164.993,246.0464
+GLM-4.5-Air,294.8103,49.5964,75.2865,47.3232,188.0317,122.7531,934.9063,326.1799,192.6793,173.4233,240.499
+Kimi-K2-Instruct,328.1743,127.0559,498.215,155.9084,411.2174,374.8414,919.1317,342.6293,464.4898,283.0215,390.4685
+Llama-3_3-Nemotron-Super-49B-v1,215.9947,28.1436,78.589,55.0476,69.6513,64.6465,672.344,82.6381,150.7472,96.6115,151.4413
+Qwen3-14B,291.1851,58.0622,88.5603,117.7344,88.0704,115.429,952.7636,353.0349,203.2735,124.2363,239.235
+Qwen3-235B-A22B-Thinking-2507,666.7428,62.6109,184.4234,141.1792,188.1752,130.8262,431.5208,488.4891,231.8124,312.6447,283.8425
+Qwen3-30B-A3B,302.3182,64.1154,92.9827,77.8548,121.5989,100.3155,973.6302,352.3788,165.0523,180.4958,243.0743
+claude-3.5-haiku,41.3124,14.9481,33.5752,18.5466,17.7297,36.6532,15.8231,40.0595,17.8959,16.9296,25.3473
+claude-opus-4-1,411.8235,66.277,99.7769,67.8099,73.13,140.073,240.5076,85.4884,194.4168,172.176,155.1479
+claude-sonnet-4,372.4893,48.2756,92.4428,50.91,54.3981,124.7965,53.7468,57.4402,181.6048,159.8677,119.5972
+deepSeek-R1-0528,516.7523,74.2918,114.0859,72.9274,112.0078,127.0512,839.4571,432.4486,182.3988,184.1239,265.5545
+deepSeek-V3-0324,186.8884,51.2986,88.1105,69.8525,65.7204,121.314,755.2438,202.0879,100.6333,355.9609,199.711
+gemini-2.5-flash,712.3231,38.3328,103.7981,43.5623,74.993,117.5583,944.4005,122.8572,135.3642,147.9211,244.1111
+gemini-2.5-flash-lite,190.8247,6.2927,19.3252,12.1569,13.2804,47.8915,602.4712,222.2597,49.1969,110.7355,127.4435
+gemini-2.5-pro,240.7699,52.3828,97.3472,57.9532,71.3161,111.1321,714.4278,300.57,170.1939,177.3255,199.3419
+gemma-3-27b-it,375.5933,26.3165,60.8123,46.1448,66.8804,99.0397,180.0912,228.1774,193.2272,68.8631,134.5146
+gpt-4.1,373.0435,20.9727,103.6544,34.6418,45.1164,68.0267,580.148,268.4173,151.755,161.6432,180.7419
+gpt-5,379.0229,104.1814,151.0171,109.2785,163.0493,141.3236,655.5064,536.5325,304.2283,232.5815,277.6722
+gpt-5-mini,420.1856,55.5139,107.6471,65.403,94.1406,96.4831,710.367,304.143,221.4093,238.5054,231.3798
+gpt-5-nano,452.6803,63.8721,123.3952,95.2713,98.3822,131.3145,649.7221,349.2375,145.5956,209.7373,231.9208
+gpt-oss-120b,219.9099,40.9979,88.2543,59.7898,50.0398,66.35,154.4168,213.958,154.396,143.3909,119.1503
+grok-3-mini,324.7266,28.8678,38.3573,27.4405,58.7259,56.7006,303.2076,79.7071,164.6423,78.6302,116.1006
+grok-4,330.6722,73.9998,112.4368,75.3662,148.4834,118.3984,908.2874,484.3592,205.0356,168.1656,262.5205
+llama-3_1-Nemotron-Ultra-253B-v1,299.2445,64.7177,80.6145,53.2406,111.7416,114.9641,677.2227,364.4893,179.9651,73.4696,201.967
+llama-4-Scout-17B-16E-Instruct,119.4443,17.908,21.721,15.137,15.5893,21.4368,21.6394,35.6977,109.5036,18.1442,39.6221
+llama-4-maverick,258.2904,12.3067,28.3245,14.1619,15.9541,23.4693,237.7464,50.0955,44.6007,26.4254,71.1375
+magistral-small-2506,50.6671,23.6896,23.0028,17.8342,14.257,27.6318,461.2929,227.3066,22.4139,27.139,89.5235
+mistral-large-2411,320.7227,28.5833,76.1094,50.5788,34.0307,104.2414,69.1314,52.9922,161.3657,71.1036,96.8859
+nova-lite-v1,17.362,9.0387,11.6702,10.1896,8.0435,9.778,8.2672,7.7491,9.4719,11.1956,10.2766
+nova-pro-v1,55.831,13.3815,14.2866,9.7714,24.3369,15.6141,14.2894,23.7601,15.2509,15.0352,20.1557
+o3,370.6262,215.2039,126.7157,84.4048,96.7106,130.4733,970.1118,427.7559,179.8835,165.4601,276.7346
+o4-mini,317.1998,49.1399,78.8689,45.2952,67.6997,52.9028,768.0834,246.0076,143.2397,86.9799,185.5417
+phi-4,28.1176,10.3654,12.3853,13.4812,13.4604,12.3491,14.116,39.9468,13.6159,34.0316,19.1869

runs/run_2025-08-14/summary_data.csv

@@ -0,0 +1,34 @@
+Model,Iterations,AutoBench,LMArena,AAI Index,MMLU-Pro,Costs (USD),Avg Answer Duration (sec),P99 Answer Duration (sec),Fail Rate %
+claude-3.5-haiku,393,3.586292962,1317,23326,0.634,0.008262832,11.51902452,17.98,4.15%
+claude-opus-4-1,387,4.239909895,1446,58830,,0.091256434,48.62490598,32.86,5.61%
+claude-sonnet-4,393,4.171968576,1399,61000,0.842,0.017099466,33.66639032,82.6,4.15%
+deepSeek-R1-0528,385,4.18112906,1418,58740,0.849,0.006382309,119.174235,223.47,6.10%
+deepSeek-V3-0324,392,3.945669087,1390,43990,0.819,0.00119639,40.30336432,106.53,4.39%
+gemini-2.5-flash,387,4.32099389,1409,58430,0.759,0.004512314,48.7078753,140.54,5.61%
+gemini-2.5-flash-lite,389,4.017202952,1351,44348,0.832,0.001052718,19.15509939,8.82,5.12%
+gemini-2.5-pro,388,4.416904571,1458,64630,0.862,0.015866994,65.03115036,64.18,5.37%
+gemma-3-27b-it,393,3.881640548,1363,25220,0.669,0.00028134,29.7215,79.12,4.15%
+GLM-4.5,389,4.176558031,1414,56080,0.835,0.00629521,80.74437254,29.19,5.12%
+GLM-4.5-Air,392,3.98464018,1379,49475,0.815,0.003611243,68.34050587,21.75,4.39%
+gpt-4.1,392,4.165890881,1406,46770,0.806,0.009144648,32.86274006,23.32,4.39%
+gpt-5,385,4.511567341,1481,68950,0.871,0.043676351,89.99818067,69.79,6.10%
+gpt-5-mini,392,4.486571107,,63700,0.828,0.006324841,65.89701176,48.74,4.39%
+gpt-5-nano,390,4.325926956,,53780,0.772,0.002414388,66.4959839,73.7,4.88%
+gpt-oss-120b,388,4.479287977,1356,61340,0.808,0.001361942,27.00733404,94.45,5.37%
+grok-3-mini,391,4.055940505,1360,58010,0.828,0.000895661,26.12147499,23.11,4.63%
+grok-4,360,4.308828831,1430,67520,0.866,0.029202342,60.95525411,13.82,12.20%
+Kimi-K2-Instruct,325,4.177138663,1420,48560,0.824,0.002379296,65.0222057,96.77,20.73%
+llama-3_1-Nemotron-Ultra-253B-v1,391,4.020264345,1345,46420,0.825,0.003451924,61.53657957,29.62,4.63%
+llama-3_3-Nemotron-Super-49B-v1,392,3.883310532,1324,40473,0.698,0.000455359,32.63831081,12.47,4.39%
+llama-4-maverick,388,3.640194992,1330,41730,0.809,0.000496574,10.65014104,9.93,5.37%
+llama-4-Scout,393,3.614481399,1318,33060,0.752,0.000410484,10.86684261,23.67,4.15%
+magistral-small-2506,390,3.713933337,1347,35950,0.746,0.0019781,17.53939687,52.3,4.88%
+mistral-large-2411,392,3.714675671,1313,27013,0.697,0.006101138,24.36368715,66.7,4.39%
+nova-lite-v1,393,3.538201832,1262,24540,0.59,0.00018322,5.288625128,21.75,4.15%
+nova-pro-v1,389,3.490835422,1289,28830,0.691,0.001800498,7.528192069,23.32,5.12%
+o3,391,4.409851586,1451,67070,0.853,0.018504711,63.89621339,69.79,4.63%
+o4-mini,393,4.27410734,1398,65050,0.832,0.008704364,39.05469579,48.74,4.15%
+phi-4,392,3.657791802,1258,27950,0.714,0.000240436,7.744667446,73.7,4.39%
+Qwen3-14B,392,3.976245179,,45235,0.774,0.000789184,61.11544056,94.45,4.39%
+Qwen3-235B-A22B-Thinking-2507,331,4.394399183,1401,63590,0.843,0.00416518,78.79346155,23.11,19.27%
+Qwen3-30B-A3B,390,3.952481327,1380,42340,0.777,0.000763337,72.64171253,13.82,4.88%

static/manifest.json

@@ -0,0 +1,10 @@
+{
+  "name": "AutoBench Leaderboard",
+  "short_name": "AutoBench",
+  "description": "Interactive leaderboard for AutoBench LLM evaluations",
+  "start_url": "/",
+  "display": "standalone",
+  "background_color": "#ffffff",
+  "theme_color": "#000000",
+  "icons": []
+}