PROJECT_OVERVIEW.md

Project Overview - Medi Scape

Medi Scape is a healthcare application that allows users to upload prescription images, which are then processed to extract and analyze the text. The key components include image processing, text recognition, and context understanding using machine learning models.

Workflow Steps

1. User Uploads Prescription Image

   • Frontend: User uploads an image via the Streamlit app.
   • Action: The image is sent to the backend for processing.

2. Backend Receives Image

   • Backend: The Django server receives the uploaded image.
   • Action: The backend calls the Doctor's Handwriting Detection API to detect handwriting.

3. Doctor's Handwriting Detection API

   • API: The uploaded image is processed to detect bounding boxes and classify the handwritten text.
   • Action: The API returns the bounding boxes and classified text to the backend.

4. Backend Calls OCR Model

   • Backend: Extract bounding boxes and classes from the API response.
   • Action: Send the bounding boxes to an OCR model to convert the handwriting to text.

5. Backend Processes OCR Text

   • Backend: The OCR model returns the detected text.
   • Action: Send the detected text to the Falcon LLM model via the AI71 API for context understanding.

6. Falcon LLM Model via AI71 API

   • API: The detected text is processed for context understanding by the Falcon LLM model.
   • Action: The processed text is returned to the backend.

7. Backend Sends Final Results to Frontend

   • Backend: The final processed text received from the AI71 API.
   • Action: Send the final results back to the frontend (Streamlit app) for user display.

Checklist / To-Do List

1. Frontend Development:

   • Implement image upload functionality in the Streamlit app.
   • Ensure the image is correctly sent to the backend.

2. Backend Development:

   • Set up Django server to receive and handle uploaded images.
   • Integrate Doctor's Handwriting Detection API to process the images.
   • Extract bounding boxes and classes from the API response.
   • Integrate OCR model to convert bounding boxes to text.
   • Process the OCR text using the Falcon LLM model via AI71 API.
   • Send the final processed text to the frontend.

3. API Integration:

   • Ensure proper integration of the Doctor's Handwriting Detection API.
   • Ensure proper integration of the Falcon LLM model via AI71 API.
   • Implement fallback solutions (like iframes) for Streamlit if API integration fails.

4. Deployment:

   • Deploy the handwriting classification API on HuggingFace.
   • Deploy the Streamlit app and ensure it is linked correctly with the backend.
   • Hide the HuggingFace name in the deployed interface for security reasons.

5. Testing and Validation:

   • Test the entire workflow from image upload to final text display.
   • Validate the accuracy of the handwriting detection and OCR models.
   • Ensure the Falcon LLM model provides accurate and contextually relevant text analysis.

6. Documentation and Communication:

   • Document the workflow and API integration steps.
   • Communicate with the team leader for the exact prompt required for the healthcare automation part.
   • Keep the team updated on progress and any issues encountered.

7. Deadline Management:

   • Ensure all tasks are completed by the deadline (August 6).