Update README and UFC data, retrain models

Expanded and clarified the README with detailed usage instructions for scraping, prediction, and pipeline execution. Updated ufc_fights.csv with new event results, added output/last_event.json, and refreshed model artifacts and results to reflect retraining on the latest data.

README.md

@@ -19,20 +19,50 @@ pinned: false
 ```bash
 pip install -r requirements.txt
 ```
-## Scraping:
-Scrape ALL fight and fighter data from [ufcstats.com](http://ufcstats.com) up to the latest event and save them in `.csv` format 
 
-2. Then run the main script to scrape all data:
+## Usage
 
+### 1. Data Scraping
+
+**Initial Setup (First Time):**
+```bash
+python -m src.main --pipeline scrape --scrape-mode full
+```
+Scrapes all historical fight data from ufcstats.com.
+
+**Update Data (Regular Use):**
+```bash
+python -m src.main --pipeline scrape --scrape-mode update
+```
+Adds only the latest events to existing data.
+
+### 2. Fight Prediction
+
+**Use Existing Models (Fast):**
 ```bash
-python -m src.scrape.main
+python -m src.main --pipeline predict
 ```
-This command will execute the entire scraping and processing pipeline, saving the final CSV files in the `output/` directory.
+Loads saved models if available and retrains if new data available.
 
-## Train and save ML models:
+**Force Retrain Models:**
+```bash
+python -m src.main --pipeline predict --force-retrain
+```
+Always retrains all models from scratch with latest data. This is useful for when the way training models changes
 
-This trains a different set of ML models and saves them in `output/models`.
+### 3. Complete Pipeline
 
 ```bash
-python -m src.predict.main
+python -m src.main --pipeline all --scrape-mode update
 ```
+Runs scraping (update mode), analysis, and prediction in sequence.
+
+## Model Performance
+
+The system tests on the latest UFC event for realistic accuracy scores (typically 50-70% for fight prediction).
+
+## Output
+
+- **Data:** `output/ufc_fights.csv`, `output/ufc_fighters.csv`
+- **Models:** `output/models/*.joblib`
+- **Results:** `output/model_results.json`

output/last_event.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a6437dbe76de54ac99372958849c4fda0baab3fe5dae46844de8201f4df7ea50
+size 168

output/model_results.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3ac3da79a015fe96d6a70000dea70be81cd626208bfdc05a79b2c7d444d68a59
-size 222959
+oid sha256:bf8df1ba9e26fa98e34bfb1c773e66576cbf89152087c55b70921269c84f39d5
+size 27286

output/models/BernoulliNBModel.joblib

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:48e363229ce42b62cc80eaa694e53906527f17faecd49fc952ec8b70753bec39
-size 5338648
+oid sha256:5ff1f1701e009137de1325c65eda57ff32444f723b07d6bc9bf0dd5b87d4dd01
+size 5344949

output/models/LGBMModel.joblib

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e91f0d2d056c0a2d0d19866cac1a547498cf1c5d819e34b842880554befd30bc
-size 6649369
+oid sha256:a2acd855ed50d393d06119fc0a3cff73e7a2e1affe2d387e631169b52e8083dd
+size 6657224

output/models/LogisticRegressionModel.joblib

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0db74028309dac730252143b3df7fc115c145dfab7da1f1dc1b25f55c1c3f65a
-size 5511435
+oid sha256:7a773552b7f1b166858ab1ff7bdf472e24b293279a8e24871de773b1a3de46e1
+size 5517988

output/models/RandomForestModel.joblib

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b1dfa2fa6240d5979ebaf66aa933b0d5c10f0919cf14c56e65047cd89ebd5259
-size 49556539
+oid sha256:100ab12c17d233b9ac97e75a8d81cf339c0d7cbd7f17050005f535f2965a67cd
+size 49715610

output/models/SVCModel.joblib

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:063b4df8b8c95679fb861498247120466f125a54245abd6498acaf5fb4c73a93
-size 7193785
+oid sha256:e4db6a11d4082ffa4d8626e485959c42868553380a7dabfc93db55bceaecd873
+size 7204520

output/models/XGBoostModel.joblib

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cbbb33909efa675e5dc7f2860c0ae32a90d4721ff92175aa03728bfa793af979
-size 6060855
+oid sha256:901938289fd8ac976f04be6ae72ba6ea9df9dcda4d6d37955f47bb9fdf2acd30
+size 6070396

src/config.py

@@ -5,6 +5,6 @@ MODELS_DIR = os.path.join(OUTPUT_DIR, 'models')
 MODEL_RESULTS_PATH = os.path.join(OUTPUT_DIR, 'model_results.json')
 FIGHTS_CSV_PATH = os.path.join(OUTPUT_DIR, 'ufc_fights.csv')
 FIGHTERS_CSV_PATH = os.path.join(OUTPUT_DIR, 'ufc_fighters.csv')
-UPCOMING_EVENTS_JSON_PATH = os.path.join(OUTPUT_DIR, 'upcoming_events.json')
 EVENTS_JSON_PATH = os.path.join(OUTPUT_DIR, 'events.json')
-
+FIGHTERS_JSON_PATH = os.path.join(OUTPUT_DIR, 'fighters.json')
+LAST_EVENT_JSON_PATH = os.path.join(OUTPUT_DIR, 'last_event.json')

src/main.py

@@ -1,5 +1,96 @@
+import argparse
+import sys
+import os
 
+# Add the current directory to Python path for imports
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
 
-# Run scrape.main
-# Run analysis.elo to add elo
-# Run predict.main for ML models
+def main():
+    """
+    Main entry point for the UFC data pipeline.
+    Supports scraping, analysis, and prediction workflows.
+    """
+    parser = argparse.ArgumentParser(description="UFC Data Pipeline")
+    parser.add_argument(
+        '--pipeline', 
+        type=str, 
+        default='scrape', 
+        choices=['scrape', 'analysis', 'predict', 'all'],
+        help="Pipeline to run: 'scrape', 'analysis', 'predict', or 'all'"
+    )
+    parser.add_argument(
+        '--scrape-mode', 
+        type=str, 
+        default='full', 
+        choices=['full', 'update'],
+        help="Scraping mode: 'full' (complete scraping) or 'update' (latest events only)"
+    )
+    parser.add_argument(
+        '--num-events', 
+        type=int, 
+        default=5,
+        help="Number of latest events to scrape in update mode (default: 5)"
+    )
+    # Model management arguments for prediction pipeline
+    parser.add_argument(
+        '--use-existing-models',
+        action='store_true',
+        default=True,
+        help="Use existing saved models if available and no new data (default: True)."
+    )
+    parser.add_argument(
+        '--no-use-existing-models',
+        action='store_true',
+        default=False,
+        help="Force retrain all models from scratch, ignoring existing saved models."
+    )
+    parser.add_argument(
+        '--force-retrain',
+        action='store_true',
+        default=False,
+        help="Force retrain all models even if no new data is available."
+    )
+    
+    args = parser.parse_args()
+    
+    if args.pipeline in ['scrape', 'all']:
+        print("=== Running Scraping Pipeline ===")
+        from scrape.main import main as scrape_main
+        
+        # Override sys.argv to pass arguments to scrape.main
+        original_argv = sys.argv
+        sys.argv = ['scrape_main', '--mode', args.scrape_mode, '--num-events', str(args.num_events)]
+        try:
+            scrape_main()
+        finally:
+            sys.argv = original_argv
+    
+    if args.pipeline in ['analysis', 'all']:
+        print("\n=== Running ELO Analysis ===")
+        from analysis.elo import main as elo_main
+        elo_main()
+    
+    if args.pipeline in ['predict', 'all']:
+        print("\n=== Running Prediction Pipeline ===")
+        from predict.main import main as predict_main
+        
+        # Override sys.argv to pass model management arguments to predict.main
+        original_argv = sys.argv
+        predict_args = ['predict_main']
+        
+        if args.no_use_existing_models:
+            predict_args.append('--no-use-existing-models')
+        elif args.use_existing_models:
+            predict_args.append('--use-existing-models')
+            
+        if args.force_retrain:
+            predict_args.append('--force-retrain')
+            
+        sys.argv = predict_args
+        try:
+        predict_main()
+        finally:
+            sys.argv = original_argv
+
+if __name__ == '__main__':
+    main()

src/predict/main.py

@@ -1,6 +1,8 @@
 import argparse
-from .pipeline import PredictionPipeline
-from .models import (
+
+# Use absolute imports to avoid relative import issues
+from src.predict.pipeline import PredictionPipeline
+from src.predict.models import (
     EloBaselineModel, 
     LogisticRegressionModel, 
     XGBoostModel,
@@ -23,8 +25,37 @@ def main():
         choices=['detailed', 'summary'],
         help="Type of report to generate: 'detailed' (file) or 'summary' (console)."
     )
+    parser.add_argument(
+        '--use-existing-models',
+        action='store_true',
+        default=True,
+        help="Use existing saved models if available and no new data (default: True)."
+    )
+    parser.add_argument(
+        '--no-use-existing-models',
+        action='store_true',
+        default=False,
+        help="Force retrain all models from scratch, ignoring existing saved models."
+    )
+    parser.add_argument(
+        '--force-retrain',
+        action='store_true',
+        default=False,
+        help="Force retrain all models even if no new data is available."
+    )
     args = parser.parse_args()
 
+    # Handle conflicting arguments
+    use_existing_models = not args.no_use_existing_models and args.use_existing_models
+    force_retrain = args.force_retrain
+
+    if args.no_use_existing_models:
+        print("No-use-existing-models flag set: All models will be retrained from scratch.")
+    elif force_retrain:
+        print("Force-retrain flag set: All models will be retrained regardless of new data.")
+    elif use_existing_models:
+        print("Using existing models if available and no new data detected.")
+
     # --- Define Models to Run ---
     # Instantiate all the models you want to evaluate here.
     models_to_run = [
@@ -38,7 +69,11 @@ def main():
     ]
     # --- End of Model Definition ---
 
-    pipeline = PredictionPipeline(models=models_to_run)
+    pipeline = PredictionPipeline(
+        models=models_to_run, 
+        use_existing_models=use_existing_models,
+        force_retrain=force_retrain
+    )
     
     try:
         pipeline.run(detailed_report=(args.report == 'detailed'))

src/predict/models.py

@@ -1,7 +1,6 @@
 from abc import ABC, abstractmethod
 import sys
 import os
-from ..analysis.elo import process_fights_for_elo, INITIAL_ELO
 import pandas as pd
 from sklearn.linear_model import LogisticRegression
 from sklearn.svm import SVC
@@ -9,8 +8,17 @@ from sklearn.naive_bayes import BernoulliNB
 from sklearn.ensemble import RandomForestClassifier
 from xgboost import XGBClassifier
 from lightgbm import LGBMClassifier
-from ..config import FIGHTERS_CSV_PATH
-from .preprocess import preprocess_for_ml, _get_fighter_history_stats, _calculate_age
+
+# Use absolute imports to avoid relative import issues
+try:
+    from src.analysis.elo import process_fights_for_elo, INITIAL_ELO
+    from src.config import FIGHTERS_CSV_PATH
+    from src.predict.preprocess import preprocess_for_ml, _get_fighter_history_stats, _calculate_age
+except ImportError:
+    # Fallback for when running directly
+    from ..analysis.elo import process_fights_for_elo, INITIAL_ELO
+    from ..config import FIGHTERS_CSV_PATH
+    from .preprocess import preprocess_for_ml, _get_fighter_history_stats, _calculate_age
 
 class BaseModel(ABC):
     """

src/predict/pipeline.py

@@ -6,22 +6,139 @@ from collections import OrderedDict
 import json
 import joblib
 
-from ..config import FIGHTS_CSV_PATH, MODEL_RESULTS_PATH, MODELS_DIR
+# Use absolute imports to avoid relative import issues
+try:
+    from src.config import FIGHTS_CSV_PATH, MODEL_RESULTS_PATH, MODELS_DIR, LAST_EVENT_JSON_PATH
+except ImportError:
+    # Fallback for when running directly
+    from ..config import FIGHTS_CSV_PATH, MODEL_RESULTS_PATH, MODELS_DIR, LAST_EVENT_JSON_PATH
+
 from .models import BaseModel
 
 class PredictionPipeline:
     """
     Orchestrates the model training, evaluation, and reporting pipeline.
     """
-    def __init__(self, models):
+    def __init__(self, models, use_existing_models=True, force_retrain=False):
         if not all(isinstance(m, BaseModel) for m in models):
             raise TypeError("All models must be instances of BaseModel.")
         self.models = models
         self.train_fights = []
         self.test_fights = []
         self.results = {}
+        self.use_existing_models = use_existing_models
+        self.force_retrain = force_retrain
+
+    def _get_last_trained_event(self):
+        """Get the last event that models were trained on."""
+        if not os.path.exists(LAST_EVENT_JSON_PATH):
+            return None
+        try:
+            with open(LAST_EVENT_JSON_PATH, 'r', encoding='utf-8') as f:
+                last_event_data = json.load(f)
+                if isinstance(last_event_data, list) and len(last_event_data) > 0:
+                    return last_event_data[0].get('name'), last_event_data[0].get('date')
+                return None, None
+        except (json.JSONDecodeError, FileNotFoundError):
+            return None, None
+
+    def _save_last_trained_event(self, event_name, event_date):
+        """Save the last event that models were trained on."""
+        last_event_data = [{
+            "name": event_name,
+            "date": event_date,
+            "training_timestamp": datetime.now().isoformat()
+        }]
+        try:
+            with open(LAST_EVENT_JSON_PATH, 'w', encoding='utf-8') as f:
+                json.dump(last_event_data, f, indent=4)
+        except Exception as e:
+            print(f"Warning: Could not save last trained event: {e}")
+
+    def _has_new_data_since_last_training(self):
+        """Check if there's new fight data since the last training."""
+        last_event_name, last_event_date = self._get_last_trained_event()
+        if not last_event_name or not last_event_date:
+            return True  # No previous training record, consider as new data
+        
+        if not os.path.exists(FIGHTS_CSV_PATH):
+            return False
+        
+        with open(FIGHTS_CSV_PATH, 'r', encoding='utf-8') as f:
+            fights = list(csv.DictReader(f))
+        
+        if not fights:
+            return False
+        
+        # Sort fights by date to get the latest event
+        fights.sort(key=lambda x: datetime.strptime(x['event_date'], '%B %d, %Y'))
+        latest_fight = fights[-1]
+        latest_event_name = latest_fight['event_name']
+        latest_event_date = latest_fight['event_date']
+        
+        # Check if we have new events since last training
+        if latest_event_name != last_event_name:
+            print(f"New data detected: Latest event '{latest_event_name}' differs from last trained event '{last_event_name}'")
+            return True
+        
+        return False
 
-    def _load_and_split_data(self, num_test_events=10):
+    def _model_exists(self, model):
+        """Check if a saved model file exists and can be loaded successfully."""
+        model_name = model.__class__.__name__
+        file_name = f"{model_name}.joblib"
+        save_path = os.path.join(MODELS_DIR, file_name)
+        
+        if not os.path.exists(save_path):
+            return False
+        
+        # Verify the model can actually be loaded
+        try:
+            joblib.load(save_path)
+            return True
+        except Exception as e:
+            print(f"Warning: Model file {file_name} exists but cannot be loaded ({e}). Will retrain.")
+            return False
+
+    def _load_existing_model(self, model_class):
+        """Load an existing model from disk."""
+        model_name = model_class.__name__
+        file_name = f"{model_name}.joblib"
+        load_path = os.path.join(MODELS_DIR, file_name)
+        
+        try:
+            loaded_model = joblib.load(load_path)
+            print(f"Loaded existing model: {model_name}")
+            return loaded_model
+        except Exception as e:
+            print(f"Error loading model {model_name}: {e}")
+            return None
+
+    def _should_retrain_models(self):
+        """Determine if models should be retrained."""
+        if self.force_retrain:
+            print("Force retrain flag is set. Retraining all models.")
+            return True
+        
+        if not self.use_existing_models:
+            print("Use existing models flag is disabled. Retraining all models.")
+            return True
+        
+        # Check if any model files are missing
+        missing_models = [m for m in self.models if not self._model_exists(m)]
+        if missing_models:
+            missing_names = [m.__class__.__name__ for m in missing_models]
+            print(f"Missing model files for: {missing_names}. Retraining all models.")
+            return True
+        
+        # Check if there's new data since last training
+        if self._has_new_data_since_last_training():
+            return True
+        
+        print("No new data detected and all model files exist. Using existing models.")
+        return False
+
+    def _load_and_split_data(self, num_test_events=1):
         """Loads and splits the data into chronological training and testing sets."""
         print("\n--- Loading and Splitting Data ---")
         if not os.path.exists(FIGHTS_CSV_PATH):
@@ -41,7 +158,7 @@ class PredictionPipeline:
         self.train_fights = [f for f in fights if f['event_name'] not in test_event_names]
         self.test_fights = [f for f in fights if f['event_name'] in test_event_names]
         print(f"Data loaded. {len(self.train_fights)} training fights, {len(self.test_fights)} testing fights.")
-        print(f"Testing on the last {num_test_events} events.")
+        print(f"Testing on the last {num_test_events} event(s): {', '.join(test_event_names)}")
 
     def run(self, detailed_report=True):
         """Executes the full pipeline: load, train, evaluate, report and save models."""
@@ -52,10 +169,24 @@ class PredictionPipeline:
             print("No fights with definitive outcomes in the test set. Aborting.")
             return
 
-        for model in self.models:
+        should_retrain = self._should_retrain_models()
+        
+        for i, model in enumerate(self.models):
             model_name = model.__class__.__name__
             print(f"\n--- Evaluating Model: {model_name} ---")
             
+            if should_retrain:
+                print(f"Training {model_name}...")
+                model.train(self.train_fights)
+            else:
+                # Try to load existing model, fall back to training if loading fails
+                loaded_model = self._load_existing_model(model.__class__)
+                if loaded_model is not None:
+                    # Replace the model instance with the loaded one
+                    self.models[i] = loaded_model
+                    model = loaded_model
+                else:
+                    print(f"Failed to load {model_name}, training new model...")
             model.train(self.train_fights)
             
             correct_predictions = 0
@@ -84,10 +215,12 @@ class PredictionPipeline:
                 })
                 
             accuracy = (correct_predictions / len(eval_fights)) * 100
+            model_status = "retrained" if should_retrain else "loaded from disk"
             self.results[model_name] = {
                 'accuracy': accuracy, 
                 'predictions': predictions,
-                'total_fights': len(eval_fights)
+                'total_fights': len(eval_fights),
+                'model_status': model_status
             }
 
         if detailed_report:
@@ -95,7 +228,9 @@ class PredictionPipeline:
         else:
             self._report_summary()
 
-        self._train_and_save_models()
+        # Only train and save models if retraining was performed
+        if should_retrain:
+            self._train_and_save_models()
 
     def _train_and_save_models(self):
         """Trains all models on the full dataset and saves them."""
@@ -114,6 +249,13 @@ class PredictionPipeline:
             os.makedirs(MODELS_DIR)
             print(f"Created directory: {MODELS_DIR}")
 
+        # Get the latest event info for tracking
+        if all_fights:
+            all_fights.sort(key=lambda x: datetime.strptime(x['event_date'], '%B %d, %Y'))
+            latest_fight = all_fights[-1]
+            latest_event_name = latest_fight['event_name']
+            latest_event_date = latest_fight['event_date']
+
         for model in self.models:
             model_name = model.__class__.__name__
             print(f"\n--- Training: {model_name} ---")
@@ -125,14 +267,20 @@ class PredictionPipeline:
             joblib.dump(model, save_path)
             print(f"Model saved successfully to {save_path}")
 
+        # Save the last trained event info
+        if all_fights:
+            self._save_last_trained_event(latest_event_name, latest_event_date)
+            print(f"Updated last trained event: {latest_event_name} ({latest_event_date})")
+
     def _report_summary(self):
         """Prints a concise summary of model performance."""
         print("\n\n--- Prediction Pipeline Summary ---")
-        print(f"{'Model':<25} | {'Accuracy':<10} | {'Fights Evaluated':<20}")
-        print("-" * 65)
+        print(f"{'Model':<25} | {'Accuracy':<10} | {'Fights Evaluated':<20} | {'Status':<15}")
+        print("-" * 80)
         for model_name, result in self.results.items():
-            print(f"{model_name:<25} | {result['accuracy']:<9.2f}% | {result['total_fights']:<20}")
-        print("-" * 65)
+            status = result.get('model_status', 'unknown')
+            print(f"{model_name:<25} | {result['accuracy']:<9.2f}% | {result['total_fights']:<20} | {status:<15}")
+        print("-" * 80)
 
     def _save_report_to_json(self, file_path=MODEL_RESULTS_PATH):
         """Saves the detailed prediction results to a JSON file."""
@@ -153,6 +301,7 @@ class PredictionPipeline:
                 report[model_name] = {
                     "overall_accuracy": f"{result['accuracy']:.2f}%",
                     "total_fights_evaluated": result['total_fights'],
+                    "model_status": result.get('model_status', 'unknown'),
                     "predictions_by_event": predictions_by_event
                 }
 

src/predict/predict_new.py

@@ -3,7 +3,12 @@ import os
 import joblib
 from datetime import datetime
 
-from ..config import MODELS_DIR
+# Use absolute imports to avoid relative import issues
+try:
+    from src.config import MODELS_DIR
+except ImportError:
+    # Fallback for when running directly
+    from ..config import MODELS_DIR
 
 def predict_new_fight(fighter1_name, fighter2_name, model_path):
     """

src/predict/preprocess.py

@@ -2,6 +2,12 @@ import pandas as pd
 import os
 import sys
 from datetime import datetime
+
+# Use absolute imports to avoid relative import issues
+try:
+    from src.config import FIGHTERS_CSV_PATH
+except ImportError:
+    # Fallback for when running directly
 from ..config import FIGHTERS_CSV_PATH
 
 def _clean_numeric_column(series):
@@ -232,6 +238,11 @@ def preprocess_for_ml(fights_to_process, fighters_csv_path):
     return X, y, metadata
 
 if __name__ == '__main__':
+    # Use absolute imports to avoid relative import issues
+    try:
+        from src.predict.pipeline import PredictionPipeline
+    except ImportError:
+        # Fallback for when running directly
     from .pipeline import PredictionPipeline
     
     print("--- Running Preprocessing Example ---")

src/scrape/main.py

@@ -1,6 +1,8 @@
 import os
 import json
-from .scrape_fights import scrape_all_events
+import argparse
+import pandas as pd
+from .scrape_fights import scrape_all_events, scrape_latest_events
 from .scrape_fighters import scrape_all_fighters
 from .to_csv import json_to_csv, fighters_json_to_csv
 from .preprocess import preprocess_fighters_csv
@@ -8,17 +10,46 @@ from .. import config
 
 def main():
     """
-    Main function to run the complete scraping and preprocessing pipeline.
+    Main function to run the scraping and preprocessing pipeline.
+    Supports both full scraping and incremental updates.
     """
+    parser = argparse.ArgumentParser(description="UFC Data Scraping Pipeline")
+    parser.add_argument(
+        '--mode', 
+        type=str, 
+        default='full', 
+        choices=['full', 'update'],
+        help="Scraping mode: 'full' (complete scraping) or 'update' (latest events + sync from last_event.json)"
+    )
+    parser.add_argument(
+        '--num-events', 
+        type=int, 
+        default=5,
+        help="Number of latest events to scrape in update mode (default: 5)"
+    )
+    
+    args = parser.parse_args()
+    
     # Ensure the output directory exists
     if not os.path.exists(config.OUTPUT_DIR):
         os.makedirs(config.OUTPUT_DIR)
         print(f"Created directory: {config.OUTPUT_DIR}")
 
+    if args.mode == 'full':
+        run_full_pipeline()
+    elif args.mode == 'update':
+        run_update_pipeline(args.num_events)
+
+def run_full_pipeline():
+    """
+    Runs the complete scraping and preprocessing pipeline.
+    """
+    print("\n=== Running FULL scraping pipeline ===")
+    
     # --- Step 1: Scrape all data from the website ---
     # This will generate fighters.json and events.json
-    scrape_all_fighters()
-    scrape_all_events()
+    scrape_all_fighters(config.FIGHTERS_JSON_PATH)
+    scrape_all_events(config.EVENTS_JSON_PATH)
 
     # --- Step 2: Convert the scraped JSON data to CSV format ---
     # This will generate fighters.csv and fights.csv
@@ -42,7 +73,133 @@ def main():
     except OSError as e:
         print(f"Error deleting JSON files: {e}")
 
-    print("\n\n--- Scraping and Preprocessing Pipeline Finished ---")
+    print("\n\n--- Full Scraping and Preprocessing Pipeline Finished ---")
+
+def run_update_pipeline(num_events=5):
+    """
+    Runs the incremental update pipeline to scrape only the latest events.
+    Also adds any events from last_event.json that aren't already in the CSV.
+    
+    Args:
+        num_events (int): Number of latest events to scrape
+    """
+    print(f"\n=== Running UPDATE pipeline for latest {num_events} events ===")
+    
+    # --- Step 1: Scrape latest events only ---
+    latest_events = scrape_latest_events(config.LAST_EVENT_JSON_PATH, num_events)
+    
+    # --- Step 2: Save latest events to last_event.json (even if empty) ---
+    if latest_events:
+        with open(config.LAST_EVENT_JSON_PATH, 'w') as f:
+            json.dump(latest_events, f, indent=4)
+        print(f"Latest {len(latest_events)} events saved to {config.LAST_EVENT_JSON_PATH}")
+    
+    # --- Step 3: Always check and update from last_event.json ---
+    update_fights_csv_from_last_event()
+    
+    print(f"\n--- Update Pipeline Finished ---")
+
+def update_fights_csv_from_last_event():
+    """
+    Updates the existing fights CSV with any events from last_event.json that aren't already present.
+    Ensures latest events are on top and preserves data types.
+    """
+    # Check if last_event.json exists
+    if not os.path.exists(config.LAST_EVENT_JSON_PATH):
+        print(f"No {config.LAST_EVENT_JSON_PATH} found. Nothing to update.")
+        return
+    
+    # Load events from last_event.json
+    try:
+        with open(config.LAST_EVENT_JSON_PATH, 'r') as f:
+            events_from_json = json.load(f)
+        
+        if not events_from_json:
+            print("No events found in last_event.json.")
+            return
+            
+        print(f"Found {len(events_from_json)} events in last_event.json")
+        
+    except Exception as e:
+        print(f"Error reading last_event.json: {e}")
+        return
+    
+    try:
+        # Check if main CSV exists
+        if os.path.exists(config.FIGHTS_CSV_PATH):
+            existing_df = pd.read_csv(config.FIGHTS_CSV_PATH)
+            existing_event_names = set(existing_df['event_name'].unique())
+        else:
+            print(f"Main fights CSV ({config.FIGHTS_CSV_PATH}) not found. Creating new CSV from last_event.json.")
+            json_to_csv(config.LAST_EVENT_JSON_PATH, config.FIGHTS_CSV_PATH)
+            return
+        
+        # Create temporary CSV from events in last_event.json
+        temp_json_path = os.path.join(config.OUTPUT_DIR, 'temp_latest.json')
+        temp_csv_path = os.path.join(config.OUTPUT_DIR, 'temp_latest.csv')
+        
+        with open(temp_json_path, 'w') as f:
+            json.dump(events_from_json, f, indent=4)
+        
+        json_to_csv(temp_json_path, temp_csv_path)
+        
+        # Read the new CSV
+        new_df = pd.read_csv(temp_csv_path)
+        
+        # Filter out events that already exist
+        new_events_df = new_df[~new_df['event_name'].isin(existing_event_names)]
+        
+        if len(new_events_df) > 0:
+            # Add new events to the TOP of the CSV (latest first)
+            combined_df = pd.concat([new_events_df, existing_df], ignore_index=True)
+            
+            # Convert date column to datetime for proper sorting
+            combined_df['event_date_parsed'] = pd.to_datetime(combined_df['event_date'])
+            
+            # Sort by date descending (latest first)
+            combined_df = combined_df.sort_values('event_date_parsed', ascending=False)
+            
+            # Drop the temporary date column
+            combined_df = combined_df.drop('event_date_parsed', axis=1)
+            
+            # Fix data types to remove .0 from numbers
+            fix_data_types(combined_df)
+            
+            combined_df.to_csv(config.FIGHTS_CSV_PATH, index=False)
+            print(f"Added {len(new_events_df)} new fights from {new_events_df['event_name'].nunique()} events to the TOP of {config.FIGHTS_CSV_PATH}")
+        else:
+            print("No new events found that aren't already in the existing CSV.")
+        
+        # Clean up temporary files
+        if os.path.exists(temp_json_path):
+            os.remove(temp_json_path)
+        if os.path.exists(temp_csv_path):
+            os.remove(temp_csv_path)
+            
+    except Exception as e:
+        print(f"Error updating fights CSV: {e}")
+        print("Falling back to creating new CSV from last_event.json only.")
+        json_to_csv(config.LAST_EVENT_JSON_PATH, config.FIGHTS_CSV_PATH)
+
+def fix_data_types(df):
+    """
+    Fix data types in the dataframe to remove .0 from numbers and preserve original format.
+    
+    Args:
+        df (pandas.DataFrame): DataFrame to fix
+    """
+    for col in df.columns:
+        if df[col].dtype == 'float64':
+            # Check if the column contains only whole numbers (no actual decimals)
+            if df[col].notna().all() and (df[col] % 1 == 0).all():
+                df[col] = df[col].astype('int64')
+            elif df[col].isna().any():
+                # Handle columns with missing values - keep as string to avoid .0
+                df[col] = df[col].fillna('').astype(str)
+                # Remove .0 from string representations
+                df[col] = df[col].str.replace(r'\.0$', '', regex=True)
+                # Convert empty strings back to original empty values
+                df[col] = df[col].replace('', '')
 
 if __name__ == '__main__':
     main()

src/scrape/scrape_fighters.py

@@ -68,7 +68,7 @@ def process_fighter(fighter_data):
     time.sleep(REQUEST_DELAY)
     return fighter_data
 
-def scrape_all_fighters():
+def scrape_all_fighters(json_path):
     """Scrapes all fighters from a-z pages using parallel processing."""
     
     # Step 1: Sequentially scrape all fighter list pages. This is fast.
@@ -129,14 +129,14 @@ def scrape_all_fighters():
 
             if (i + 1) > 0 and (i + 1) % 50 == 0:
                 fighters_with_details.sort(key=lambda x: (x['last_name'], x['first_name']))
-                with open(config.FIGHTERS_JSON_PATH, 'w') as f:
+                with open(json_path, 'w') as f:
                     json.dump(fighters_with_details, f, indent=4)
                 
     fighters_with_details.sort(key=lambda x: (x['last_name'], x['first_name']))
     return fighters_with_details
 
 if __name__ == "__main__":
-    all_fighters_data = scrape_all_fighters()
+    all_fighters_data = scrape_all_fighters(config.FIGHTERS_JSON_PATH)
     if not os.path.exists(config.OUTPUT_DIR):
         os.makedirs(config.OUTPUT_DIR)
 

src/scrape/scrape_fights.py

@@ -3,7 +3,7 @@ from bs4 import BeautifulSoup
 import json
 import time
 import concurrent.futures
-from ..config import EVENTS_JSON_PATH
+from .. import config
 
 # --- Configuration ---
 # The number of parallel threads to use for scraping fight details.
@@ -175,7 +175,7 @@ def scrape_event_details(event_url):
     event_details['fights'] = completed_fights
     return event_details
 
-def scrape_all_events():
+def scrape_all_events(json_path):
     soup = get_soup(BASE_URL)
     events = []
 
@@ -204,15 +204,60 @@ def scrape_all_events():
 
             if (i + 1) % 10 == 0:
                 print(f"--- Saving progress: {i + 1} of {total_events} events saved. ---")
-                with open(EVENTS_JSON_PATH, 'w') as f:
+                with open(json_path, 'w') as f:
                     json.dump(events, f, indent=4)
         except Exception as e:
             print(f"Could not process event {event_url}. Error: {e}")
 
     return events
 
+def scrape_latest_events(json_path, num_events=5):
+    """
+    Scrapes only the latest N events from UFC stats.
+    This is useful for incremental updates to avoid re-scraping all data.
+    
+    Args:
+        json_path (str): Path to save the latest events JSON file
+        num_events (int): Number of latest events to scrape (default: 5)
+    
+    Returns:
+        list: List of scraped event data
+    """
+    soup = get_soup(BASE_URL)
+    events = []
+
+    table = soup.find('table', class_='b-statistics__table-events')
+    if not table:
+        print("Could not find events table on the page.")
+        return []
+
+    event_rows = [row for row in table.find_all('tr', class_='b-statistics__table-row') if row.find('td')]
+    
+    # Limit to the latest N events (events are ordered chronologically with most recent first)
+    latest_event_rows = event_rows[:num_events]
+    total_events = len(latest_event_rows)
+    print(f"Found {len(event_rows)} total events. Scraping latest {total_events} events.")
+
+    for i, row in enumerate(latest_event_rows):
+        event_link_tag = row.find('a', class_='b-link b-link_style_black')
+        if not event_link_tag or not event_link_tag.has_attr('href'):
+            continue
+        
+        event_url = event_link_tag['href']
+        
+        try:
+            event_data = scrape_event_details(event_url)
+            if event_data:
+                events.append(event_data)
+            
+            print(f"Progress: {i+1}/{total_events} latest events scraped.")
+        except Exception as e:
+            print(f"Could not process event {event_url}. Error: {e}")
+
+    return events
+
 if __name__ == "__main__":
-    all_events_data = scrape_all_events()
-    with open(EVENTS_JSON_PATH, 'w') as f:
+    all_events_data = scrape_all_events(config.EVENTS_JSON_PATH)
+    with open(config.EVENTS_JSON_PATH, 'w') as f:
         json.dump(all_events_data, f, indent=4)
-    print(f"\nScraping complete. Final data saved to {EVENTS_JSON_PATH}")
+    print(f"\nScraping complete. Final data saved to {config.EVENTS_JSON_PATH}")