experiments/README.md

# Experiments
Below are the instructions for running experiments using our novel ChestAgentBench and the previous SoTA CheXbench. ChestAgentBench is a comprehensive benchmark containing over 2,500 complex medical queries across 8 diverse categories.

### ChestAgentBench

To run gpt-4o on ChestAgentBench, enter the `experiments` directory and run the following script:
```bash

python benchmark_gpt4o.py

```

To run llama 3.2 vision 90B on ChestAgentBench, run the following:
```bash

python benchmark_llama.py

```

To run chexagent on ChestAgentBench, run the following:
```bash

python benchmark_chexagent.py

```

To run llava-med on ChestAgentBench, you'll need to clone their repo and copy the following script into it, after you follow their setup instructions.
```bash

mv benchmark_llavamed.py ~/LLaVA-Med/llava/serve

python -m llava.serve.benchmark_llavamed --model-name llava-med-v1.5-mistral-7b --controller http://localhost:10000

```

If you want to inspect the logs, you can run the following. It will select the most recent log file by default.
```bash

python inspect_logs.py [optional: log-file] -n [num-logs]

```

Finally, to analyze results, run:
```bash

python analyze_axes.py results/[logfile].json ../benchmark/questions/ --model [gpt4|llama|chexagent|llava-med] --max-questions [optional:int]

```

### CheXbench

To run the models on chexbench, you can use `chexbench_gpt4.py` as a reference. You'll need to download the dataset files locally, and upload them for each request. Rad-ReStruct and Open-I use the same set of images, so you can download the `NLMCXR.zip` file just once and copy the images to both directories.

You can find the datasets here:
1. [SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering](https://www.med-vqa.com/slake/). Save this to `MedMAX/data/slake`.
2. [Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reporting](https://github.com/ChantalMP/Rad-ReStruct). Save the images to `MedMAX/data/rad-restruct/images`.
3. [Open-I Service of the National Library of Medicine](https://openi.nlm.nih.gov/faq). Save the images to `MedMAX/data/openi/images`.

Once you're finished, you'll want to fix the paths in the `chexbench.json` file to your local paths using the `MedMax/data/fix_chexbench.py` script.


### Compare Runs
Analyze a single file based on overall accuracy and along different axes
```

python compare_runs.py results/medmax.json

```

For a direct evaluation comparing **2** models, on the exact same questions 
```

python compare_runs.py results/medmax.json results/gpt4o.json

```

For a direct evaluation comparing **ALL** models, on the exact same questions (add as many model log files as you want).
```

python compare_runs.py results/medmax.json results/gpt4o.json results/llama.json results/chexagent.json results/llavamed.json

```