feat: add emergency subset analysis plots and statistics report

dataset/analysis/stats/analysis_stats_emergency_subset_opt.json

@@ -0,0 +1,55 @@
+{
+  "basic_statistics": {
+    "total_records": 11914,
+    "avg_length": 23847.07579318449
+  },
+  "keyword_statistics": {
+    "Acute abdomen": 52,
+    "Acute bleeding": 31,
+    "Acute Coronary Syndrome": 351,
+    "Acute Kidney Injury": 202,
+    "Acute pancreatitis": 214,
+    "Acute respiratory distress syndrome": 231,
+    "Acute stroke": 67,
+    "Anaphylaxis": 1016,
+    "Anaphylactic Shock": 153,
+    "Arrhythmia": 1564,
+    "Atrial fibrillation": 771,
+    "Atrial flutter": 146,
+    "Bradycardia": 884,
+    "Cardiac arrest": 614,
+    "Cardiogenic Shock": 196,
+    "Chest pain": 1434,
+    "Dyspnea": 1319,
+    "Fever": 4279,
+    "Gastrointestinal Hemorrhage": 158,
+    "GI bleeding": 105,
+    "Hemorrhage": 1621,
+    "Hemorrhagic stroke": 117,
+    "Hyperthermia": 305,
+    "Hypovolemic Shock": 63,
+    "Hypotension": 1929,
+    "Hypothermia": 356,
+    "Internal bleeding": 70,
+    "Intracranial Hemorrhages": 6,
+    "Ischemic stroke": 225,
+    "Loss of consciousness": 422,
+    "Myocardial Infarction": 1710,
+    "MI": 11773,
+    "Pulmonary Edema": 487,
+    "Pulmonary Embolism": 654,
+    "Respiratory distress": 730,
+    "Respiratory failure": 579,
+    "Sepsis": 1188,
+    "Severe Sepsis": 81,
+    "Septic Shock": 244,
+    "Shock": 1892,
+    "Status Epilepticus": 150,
+    "Syncope": 834,
+    "Tachycardia": 1651,
+    "Tachypnea": 268,
+    "Traumatic Brain Injury": 171,
+    "Ventricular Tachycardia": 492,
+    "Ventricular fibrillation": 295
+  }
+}

dataset/analysis/subset_comparison_first10_records_20250726_163149.md

@@ -0,0 +1,198 @@
+# Optimized Subsets Comparison Report
+
+Generated on: 2025-07-26 16:31:49
+
+File format: CSV
+
+
+## Basic Statistics
+
+- Emergency subset total records: 11914
+- Emergency+Treatment subset total records: 11023
+- Avg Emergency Text Length: 23847.08
+- Avg Treatment Text Length: 25408.64
+- Avg Emergency Keywords: 2.85
+- Avg Treatment Keywords: 2.97
+
+## Emergency Subset (First 10 Records)
+
+
+### Record 1
+```
+Text preview: # Section 1: Recommendations
+
+# RECOMMENDATIONS Recommendation 1: General Measures Committee Respons...
+Matched keywords: shock
+Text length: 37792
+```
+
+
+### Record 2
+```
+Text preview: Evidence-based Series 4-9 Version 2 A Quality Initiative of the Program in Evidence-based Care (PEBC...
+Matched keywords: hemorrhage
+Text length: 7559
+```
+
+
+### Record 3
+```
+Text preview: Neuroendocrine tumours (NETs) constitute a heterogeneous group of neoplasms: they include epithelial...
+Matched keywords: ards|pulmonary embolism
+Text length: 11731
+```
+
+
+### Record 4
+```
+Text preview: Given the potential toxicities associated with alemtuzumab, and given the limited nature of the clin...
+Matched keywords: fever|dyspnea|hypotension|sepsis
+Text length: 46087
+```
+
+
+### Record 5
+```
+Text preview: Although the incidence and mortality of gastric cancer has been steadily decreasing in Canadian men ...
+Matched keywords: hyperthermia
+Text length: 35302
+```
+
+
+### Record 6
+```
+Text preview: There are various definitions for palliative care, but most people would agree that "it focuses on c...
+Matched keywords: hemorrhage|dyspnea
+Text length: 16186
+```
+
+
+### Record 7
+```
+Text preview: # GUIDELINE OBJECTIVES
+The objective of this guideline is to update a previous guideline on chemothe...
+Matched keywords: hemorrhage
+Text length: 7551
+```
+
+
+### Record 8
+```
+Text preview: Anthracyclines have been established to be superior to some non-anthracycline chemotherapy regimens ...
+Matched keywords: mi
+Text length: 50729
+```
+
+
+### Record 9
+```
+Text preview: # GUIDELINE OBJECTIVE
+This guideline was written to provide guidance on the most appropriate follow-...
+Matched keywords: hemorrhage
+Text length: 4299
+```
+
+
+### Record 10
+```
+Text preview: PDT is a local treatment. It utilizes the local, selective, cytotoxic reaction produced by photosens...
+Matched keywords: dyspnea|mi|hemorrhage|respiratory_failure|cva|hypotension|sepsis|ards
+Text length: 54427
+```
+
+
+## Emergency+Treatment Subset (First 10 Records)
+
+
+### Record 1
+```
+Text preview: # Section 1: Recommendations
+
+# RECOMMENDATIONS Recommendation 1: General Measures Committee Respons...
+Emergency keywords: shock
+Treatment keywords: management|medication|procedure|fluid|monitoring|iv|administer|dose
+Text length: 37792
+```
+
+
+### Record 2
+```
+Text preview: Evidence-based Series 4-9 Version 2 A Quality Initiative of the Program in Evidence-based Care (PEBC...
+Emergency keywords: hemorrhage
+Treatment keywords: Therapy|treatment|x-ray|us|ct
+Text length: 7559
+```
+
+
+### Record 3
+```
+Text preview: Neuroendocrine tumours (NETs) constitute a heterogeneous group of neoplasms: they include epithelial...
+Emergency keywords: ards|pulmonary embolism
+Treatment keywords: dopamine|therapy|treatment|surgery|iv|intervention|dose
+Text length: 11731
+```
+
+
+### Record 4
+```
+Text preview: Given the potential toxicities associated with alemtuzumab, and given the limited nature of the clin...
+Emergency keywords: fever|dyspnea|hypotension|sepsis
+Treatment keywords: treatment|iv|therapy|treat|management|intervention|supportive care|dose
+Text length: 46087
+```
+
+
+### Record 5
+```
+Text preview: Although the incidence and mortality of gastric cancer has been steadily decreasing in Canadian men ...
+Emergency keywords: hyperthermia
+Treatment keywords: surgery|treatment|therapy|treat|dose|ct
+Text length: 35302
+```
+
+
+### Record 6
+```
+Text preview: There are various definitions for palliative care, but most people would agree that "it focuses on c...
+Emergency keywords: hemorrhage|dyspnea
+Treatment keywords: therapy|management|treatment|morphine|dose
+Text length: 16186
+```
+
+
+### Record 7
+```
+Text preview: # GUIDELINE OBJECTIVES
+The objective of this guideline is to update a previous guideline on chemothe...
+Emergency keywords: hemorrhage
+Treatment keywords: therapy|treatment|surgery
+Text length: 7551
+```
+
+
+### Record 8
+```
+Text preview: Anthracyclines have been established to be superior to some non-anthracycline chemotherapy regimens ...
+Emergency keywords: mi
+Treatment keywords: iv|Dose|therapy|administer|surgery|treatment|treat|medication|ecg
+Text length: 50729
+```
+
+
+### Record 9
+```
+Text preview: # GUIDELINE OBJECTIVE
+This guideline was written to provide guidance on the most appropriate follow-...
+Emergency keywords: hemorrhage
+Treatment keywords: treatment|ct
+Text length: 4299
+```
+
+
+### Record 10
+```
+Text preview: PDT is a local treatment. It utilizes the local, selective, cytotoxic reaction produced by photosens...
+Emergency keywords: dyspnea|mi|hemorrhage|respiratory_failure|cva|hypotension|sepsis|ards
+Treatment keywords: treatment|oxygen|iv|dose|therapy|surgery|x-ray|administer|procedure|management
+Text length: 54427
+```

dataset/analysis/subset_comparison_first10_records_20250726_163158.md

@@ -0,0 +1,198 @@
+# Optimized Subsets Comparison Report
+
+Generated on: 2025-07-26 16:31:58
+
+File format: JSONL
+
+
+## Basic Statistics
+
+- Emergency subset total records: 11914
+- Emergency+Treatment subset total records: 11023
+- Avg Emergency Text Length: 23847.08
+- Avg Treatment Text Length: 25408.64
+- Avg Emergency Keywords: 2.85
+- Avg Treatment Keywords: 2.97
+
+## Emergency Subset (First 10 Records)
+
+
+### Record 1
+```
+Text preview: # Section 1: Recommendations
+
+# RECOMMENDATIONS Recommendation 1: General Measures Committee Respons...
+Matched keywords: shock
+Text length: 37792
+```
+
+
+### Record 2
+```
+Text preview: Evidence-based Series 4-9 Version 2 A Quality Initiative of the Program in Evidence-based Care (PEBC...
+Matched keywords: hemorrhage
+Text length: 7559
+```
+
+
+### Record 3
+```
+Text preview: Neuroendocrine tumours (NETs) constitute a heterogeneous group of neoplasms: they include epithelial...
+Matched keywords: ards|pulmonary embolism
+Text length: 11731
+```
+
+
+### Record 4
+```
+Text preview: Given the potential toxicities associated with alemtuzumab, and given the limited nature of the clin...
+Matched keywords: fever|dyspnea|hypotension|sepsis
+Text length: 46087
+```
+
+
+### Record 5
+```
+Text preview: Although the incidence and mortality of gastric cancer has been steadily decreasing in Canadian men ...
+Matched keywords: hyperthermia
+Text length: 35302
+```
+
+
+### Record 6
+```
+Text preview: There are various definitions for palliative care, but most people would agree that "it focuses on c...
+Matched keywords: hemorrhage|dyspnea
+Text length: 16186
+```
+
+
+### Record 7
+```
+Text preview: # GUIDELINE OBJECTIVES
+The objective of this guideline is to update a previous guideline on chemothe...
+Matched keywords: hemorrhage
+Text length: 7551
+```
+
+
+### Record 8
+```
+Text preview: Anthracyclines have been established to be superior to some non-anthracycline chemotherapy regimens ...
+Matched keywords: mi
+Text length: 50729
+```
+
+
+### Record 9
+```
+Text preview: # GUIDELINE OBJECTIVE
+This guideline was written to provide guidance on the most appropriate follow-...
+Matched keywords: hemorrhage
+Text length: 4299
+```
+
+
+### Record 10
+```
+Text preview: PDT is a local treatment. It utilizes the local, selective, cytotoxic reaction produced by photosens...
+Matched keywords: dyspnea|mi|hemorrhage|respiratory_failure|cva|hypotension|sepsis|ards
+Text length: 54427
+```
+
+
+## Emergency+Treatment Subset (First 10 Records)
+
+
+### Record 1
+```
+Text preview: # Section 1: Recommendations
+
+# RECOMMENDATIONS Recommendation 1: General Measures Committee Respons...
+Emergency keywords: shock
+Treatment keywords: management|medication|procedure|fluid|monitoring|iv|administer|dose
+Text length: 37792
+```
+
+
+### Record 2
+```
+Text preview: Evidence-based Series 4-9 Version 2 A Quality Initiative of the Program in Evidence-based Care (PEBC...
+Emergency keywords: hemorrhage
+Treatment keywords: Therapy|treatment|x-ray|us|ct
+Text length: 7559
+```
+
+
+### Record 3
+```
+Text preview: Neuroendocrine tumours (NETs) constitute a heterogeneous group of neoplasms: they include epithelial...
+Emergency keywords: ards|pulmonary embolism
+Treatment keywords: dopamine|therapy|treatment|surgery|iv|intervention|dose
+Text length: 11731
+```
+
+
+### Record 4
+```
+Text preview: Given the potential toxicities associated with alemtuzumab, and given the limited nature of the clin...
+Emergency keywords: fever|dyspnea|hypotension|sepsis
+Treatment keywords: treatment|iv|therapy|treat|management|intervention|supportive care|dose
+Text length: 46087
+```
+
+
+### Record 5
+```
+Text preview: Although the incidence and mortality of gastric cancer has been steadily decreasing in Canadian men ...
+Emergency keywords: hyperthermia
+Treatment keywords: surgery|treatment|therapy|treat|dose|ct
+Text length: 35302
+```
+
+
+### Record 6
+```
+Text preview: There are various definitions for palliative care, but most people would agree that "it focuses on c...
+Emergency keywords: hemorrhage|dyspnea
+Treatment keywords: therapy|management|treatment|morphine|dose
+Text length: 16186
+```
+
+
+### Record 7
+```
+Text preview: # GUIDELINE OBJECTIVES
+The objective of this guideline is to update a previous guideline on chemothe...
+Emergency keywords: hemorrhage
+Treatment keywords: therapy|treatment|surgery
+Text length: 7551
+```
+
+
+### Record 8
+```
+Text preview: Anthracyclines have been established to be superior to some non-anthracycline chemotherapy regimens ...
+Emergency keywords: mi
+Treatment keywords: iv|Dose|therapy|administer|surgery|treatment|treat|medication|ecg
+Text length: 50729
+```
+
+
+### Record 9
+```
+Text preview: # GUIDELINE OBJECTIVE
+This guideline was written to provide guidance on the most appropriate follow-...
+Emergency keywords: hemorrhage
+Treatment keywords: treatment|ct
+Text length: 4299
+```
+
+
+### Record 10
+```
+Text preview: PDT is a local treatment. It utilizes the local, selective, cytotoxic reaction produced by photosens...
+Emergency keywords: dyspnea|mi|hemorrhage|respiratory_failure|cva|hypotension|sepsis|ards
+Treatment keywords: treatment|oxygen|iv|dose|therapy|surgery|x-ray|administer|procedure|management
+Text length: 54427
+```