ybchen928/oncall-guide-ai at 3edd46d038e73c2478d330cf56a00d48feee8549

oncall-guide-ai / evaluation

Ctrl+K

6 contributors

History: 19 commits

YanBoChen

Enhance direct LLM evaluation with retry mechanism for 504 timeouts and improved guidance format

3edd46d 22 days ago

modules
feat(evaluation): add comprehensive hospital customization evaluation system 23 days ago
old
Before Run the 1st Evalation: Add Precision & MRR Chart Generator and a sample test query 23 days ago
results
Add comprehensive evaluation reports and execution time breakdown for Hospital Customization System 22 days ago
README_HOSPITAL_CUSTOMIZATION.md

10.2 kB

feat(evaluation): add comprehensive hospital customization evaluation system 23 days ago
direct_llm_evaluator.py
22.2 kB

Enhance direct LLM evaluation with retry mechanism for 504 timeouts and improved guidance format 22 days ago
generate_combined_comparison_chart.py

8.56 kB

feat(evaluation): add visualization generators for generating png files 23 days ago
generate_comparison_report.py

18.8 kB

feat(evaluation): add comprehensive hospital customization evaluation system 23 days ago
generate_execution_time_table.py

7.6 kB

feat(evaluation): add visualization generators for generating png files 23 days ago
generate_individual_analysis_charts.py
8.96 kB

feat(evaluation): add visualization generators for generating png files 23 days ago
generate_individual_rag_vs_direct_charts.py

12.9 kB

feat(evaluation): add visualization generators for generating png files 23 days ago
hospital_customization_evaluator.py

26.5 kB

feat(evaluation): add comprehensive hospital customization evaluation system 23 days ago
latency_evaluator.py
41.4 kB

Update query file references for full evaluation and correct typo in pre_user_query_evaluate.txt for pre-test. 22 days ago
metric1_latency_chart_generator.py

13.6 kB

Before Run the 1st Evalation: Add Precision & MRR Chart Generator and a sample test query 23 days ago
metric2_extraction_chart_generator.py

8.63 kB

Before Run the 1st Evalation: Add Precision & MRR Chart Generator and a sample test query 23 days ago
metric3_relevance_chart_generator.py

9.93 kB

Update threshold values in latency evaluator and coverage chart generator; enhance precision and MRR analysis with corrected thresholds and new chart generator for detailed metrics visualization. 22 days ago
metric4_coverage_chart_generator.py

9.32 kB

Update threshold values in latency evaluator and coverage chart generator; enhance precision and MRR analysis with corrected thresholds and new chart generator for detailed metrics visualization. 22 days ago
metric5_6_judge_evaluator_manual.md

9.86 kB

Add multi-system evaluation support for clinical actionability and evidence quality metrics 23 days ago
metric5_6_llm_judge_chart_generator.py
19.5 kB

Add multi-system evaluation support for clinical actionability and evidence quality metrics 23 days ago
metric5_6_llm_judge_evaluator.py

30.3 kB

Enhance Direct LLM Evaluator and Judge Evaluator: 22 days ago
metric7_8_precision_MRR.py
16.8 kB

Add adaptive relevance thresholds for query complexity in PrecisionMRRAnalyzer; fix typo in condition mapping for postpartum hemorrhage 22 days ago
metric7_8_precision_mrr_chart_generator.py

23.8 kB

Update threshold values in latency evaluator and coverage chart generator; enhance precision and MRR analysis with corrected thresholds and new chart generator for detailed metrics visualization. 22 days ago
pre_user_query_evaluate.txt

330 Bytes

Update query file references for full evaluation and correct typo in pre_user_query_evaluate.txt for pre-test. 22 days ago
run_hospital_evaluation.py

3.58 kB

feat(evaluation): add comprehensive hospital customization evaluation system 23 days ago
run_rag_vs_direct_comparison.py
17.3 kB

feat(evaluation): add comprehensive hospital customization evaluation system 23 days ago
single_test_query.txt

127 Bytes

Add comprehensive evaluation reports and execution time breakdown for Hospital Customization System 22 days ago
test_hospital_customization_pipeline.py
15.4 kB

feat(evaluation): add comprehensive hospital customization evaluation system 23 days ago
user_query.txt
2.06 kB

fix(mild bug): enhance user query prompts (more robust dealing process with .txt or .json) and add postpartum hemorrhage condition mapping 23 days ago