Upload 8 files

README.md

@@ -1,14 +1,172 @@
----
-title: Multi Modal O1 Chatbot
-emoji: 🔥
-colorFrom: green
-colorTo: blue
-sdk: gradio
-sdk_version: 5.32.0
-app_file: app.py
-pinned: false
-license: mit
-short_description: Omni-1 and Omni-3 Chatbot
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+---
+title: Multi Modal Omni Chatbot
+emoji: 🐠
+colorFrom: blue
+colorTo: pink
+sdk: gradio
+sdk_version: 5.20.1
+app_file: app.py
+pinned: false
+license: mit
+short_description: A chatbot that supports text, image, voice and pdf chat.
+---
+
+Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+
+---
+
+# Building a Multimodal Chatbot with Gradio and OpenAI
+
+In recent years, the field of artificial intelligence (AI) has seen an exciting leap in multimodal capabilities. Multimodal systems can understand and generate multiple types of input — like text and images — to provide richer, more dynamic responses. One such example is a multimodal chatbot that can process both text and image inputs using the OpenAI API.
+
+In this article, we’ll walk through how to create a multimodal chatbot using **Gradio** and the **OpenAI API** that allows users to input both text and images, interact with the model, and receive insightful responses.
+
+## Key Components
+
+Before we dive into the code, let's break down the core components of this chatbot:
+
+- **Gradio**: A simple, open-source Python library for building UIs for machine learning models. It allows you to quickly create and deploy interfaces for any ML model, including those that take images, text, or audio as input.
+  
+- **OpenAI API**: This is the engine behind our chatbot. OpenAI provides models like `gpt-3.5`, `gpt-4`, and specialized image models such as `o1` for handling multimodal tasks (image and text inputs).
+
+- **Python and PIL**: To handle image preprocessing, we use Python's `PIL` (Python Imaging Library) to convert uploaded images into a format that can be passed into the OpenAI model.
+
+## The Chatbot Overview
+
+The chatbot can take two main types of input:
+1. **Text Input**: Ask a question or give a prompt to the model.
+2. **Image Input**: Upload an image, and the model will interpret the image and provide a response based on its content.
+
+The interface offers the user the ability to adjust two main settings:
+- **Reasoning Effort**: This controls how complex or detailed the assistant’s answers should be. The options are `low`, `medium`, and `high`.
+- **Model Choice**: Users can select between two models: `o1` (optimized for image input) and `o3-mini` (focused on text input).
+
+The interface is simple, intuitive, and interactive, with the chat history displayed on the side.
+
+## Step-by-Step Code Explanation
+
+### 1. Set Up Gradio UI
+
+Gradio makes it easy to create beautiful interfaces for your AI models. We start by defining a custom interface with the following components:
+
+- **Textbox for OpenAI API Key**: Users provide their OpenAI API key to authenticate their request.
+- **Image Upload and Text Input Fields**: Users can choose to upload an image or input text.
+- **Dropdowns for Reasoning Effort and Model Selection**: Choose the complexity of the responses and the model to use.
+- **Submit and Clear Buttons**: These trigger the logic to process user inputs and clear chat history, respectively.
+
+```python
+with gr.Blocks(css=custom_css) as demo:
+    gr.Markdown("""
+        <div class="gradio-header">
+            <h1>Multimodal Chatbot (Text + Image)</h1>
+            <h3>Interact with a chatbot using text or image inputs</h3>
+        </div>
+    """)
+    
+    # User inputs and chat history
+    openai_api_key = gr.Textbox(label="Enter OpenAI API Key", type="password", placeholder="sk-...", interactive=True)
+    image_input = gr.Image(label="Upload an Image", type="pil")
+    input_text = gr.Textbox(label="Enter Text Question", placeholder="Ask a question or provide text", lines=2)
+    
+    # Reasoning effort and model selection
+    reasoning_effort = gr.Dropdown(label="Reasoning Effort", choices=["low", "medium", "high"], value="medium")
+    model_choice = gr.Dropdown(label="Select Model", choices=["o1", "o3-mini"], value="o1")
+    
+    submit_btn = gr.Button("Ask!", elem_id="submit-btn")
+    clear_btn = gr.Button("Clear History", elem_id="clear-history")
+    
+    # Chat history display
+    chat_history = gr.Chatbot()
+```
+
+### 2. Handle Image and Text Inputs
+
+The function `generate_response` processes both image and text inputs by sending them to OpenAI’s API. If an image is uploaded, it gets converted into a **base64 string** so it can be sent as part of the prompt.
+
+For text inputs, the prompt is directly passed to the model.
+
+```python
+def generate_response(input_text, image, openai_api_key, reasoning_effort="medium", model_choice="o1"):
+    openai.api_key = openai_api_key
+    
+    if image:
+        image_info = get_base64_string_from_image(image)
+        input_text = f"data:image/png;base64,{image_info}"
+
+    if model_choice == "o1":
+        messages = [{"role": "user", "content": [{"type": "image_url", "image_url": {"url": input_text}}]}]
+    elif model_choice == "o3-mini":
+        messages = [{"role": "user", "content": [{"type": "text", "text": input_text}]}]
+    
+    # API request
+    response = openai.ChatCompletion.create(
+        model=model_choice,
+        messages=messages,
+        reasoning_effort=reasoning_effort,
+        max_completion_tokens=2000
+    )
+    return response["choices"][0]["message"]["content"]
+```
+
+### 3. Image-to-Base64 Conversion
+
+To ensure the image is properly formatted, we convert it into a **base64** string. This string can then be embedded directly into the OpenAI request. This conversion is handled by the `get_base64_string_from_image` function.
+
+```python
+def get_base64_string_from_image(pil_image):
+    buffered = io.BytesIO()
+    pil_image.save(buffered, format="PNG")
+    img_bytes = buffered.getvalue()
+    base64_str = base64.b64encode(img_bytes).decode("utf-8")
+    return base64_str
+```
+
+### 4. Chat History and Interaction
+
+The chat history is stored and displayed using Gradio’s `gr.Chatbot`. Each time the user submits a question or image, the conversation history is updated, showing both user and assistant responses in an easy-to-read format.
+
+```python
+def chatbot(input_text, image, openai_api_key, reasoning_effort, model_choice, history=[]):
+    response = generate_response(input_text, image, openai_api_key, reasoning_effort, model_choice)
+    history.append((f"User: {input_text}", f"Assistant: {response}"))
+    return "", history
+```
+
+### 5. Clear History Function
+
+To reset the conversation, we include a simple function that clears the chat history when the "Clear History" button is clicked.
+
+```python
+def clear_history():
+    return "", []
+```
+
+### 6. Custom CSS for Styling
+
+To ensure a visually appealing interface, custom CSS is applied. The design includes animations for chat messages and custom button styles to make the interaction smoother.
+
+```css
+/* Custom CSS for the chat interface */
+.gradio-container { ... }
+.gradio-header { ... }
+.gradio-chatbot { ... }
+```
+
+### 7. Launch the Interface
+
+Finally, we call the `create_interface()` function to launch the Gradio interface. This allows users to start interacting with the chatbot by uploading images, entering text, and receiving responses based on the selected model and reasoning effort.
+
+```python
+if __name__ == "__main__":
+    demo = create_interface()
+    demo.launch()
+```
+
+## Conclusion
+
+This multimodal chatbot can handle both text and image inputs, offering a rich conversational experience. By combining the power of **Gradio** for building intuitive UIs and **OpenAI’s powerful models** for natural language processing and image recognition, this application demonstrates how to seamlessly integrate multiple forms of input into a single, easy-to-use interface.
+
+Feel free to try it out yourself and experiment with different settings, including reasoning effort and model selection. Whether you're building a customer support bot or an image-based query system, this framework provides a flexible foundation for creating powerful, multimodal applications.
+
+---
+

abc.txt

@@ -0,0 +1,118 @@
+import gradio as gr
+import openai
+import base64
+from PIL import Image
+import io
+
+# Function to send the request to OpenAI API with an image or text input
+def generate_response(input_text, image, openai_api_key, reasoning_effort="medium", model_choice="o1"):
+    if not openai_api_key:
+        return "Error: No API key provided."
+
+    openai.api_key = openai_api_key
+
+    # Process the input depending on whether it's text or an image
+    if image:
+        # Convert the image to base64 string
+        image_info = get_base64_string_from_image(image)
+        input_text = f"data:image/png;base64,{image_info}"
+
+    # Prepare the messages for OpenAI API
+    if model_choice == "o1":
+        messages = [
+            {"role": "user", "content": [{"type": "image_url", "image_url": {"url": input_text}}]}
+        ]
+    elif model_choice == "o3-mini":
+        messages = [
+            {"role": "user", "content": [{"type": "text", "text": input_text}]}
+        ]
+    
+    try:
+        # Call OpenAI API with the selected model
+        response = openai.ChatCompletion.create(
+            model=model_choice,  # Dynamically choose the model (o1 or o3-mini)
+            messages=messages,
+            reasoning_effort=reasoning_effort,  # Set reasoning_effort for the response
+            max_completion_tokens=2000  # Limit response tokens to 2000
+        )
+
+        return response["choices"][0]["message"]["content"]
+    except Exception as e:
+        return f"Error calling OpenAI API: {str(e)}"
+
+# Function to convert an uploaded image to a base64 string
+def get_base64_string_from_image(pil_image):
+    # Convert PIL Image to bytes
+    buffered = io.BytesIO()
+    pil_image.save(buffered, format="PNG")
+    img_bytes = buffered.getvalue()
+    base64_str = base64.b64encode(img_bytes).decode("utf-8")
+    return base64_str
+
+# The function that will be used by Gradio interface
+def chatbot(input_text, image, openai_api_key, reasoning_effort, model_choice, history=[]):
+    response = generate_response(input_text, image, openai_api_key, reasoning_effort, model_choice)
+    
+    # Append the response to the history
+    history.append((f"User: {input_text}", f"Assistant: {response}"))
+    
+    return "", history
+
+# Function to clear the chat history
+def clear_history():
+    return "", []
+
+# Gradio interface setup
+def create_interface():
+    with gr.Blocks() as demo:
+        gr.Markdown("# Multimodal Chatbot (Text + Image)")
+
+        # Add a description after the title
+        gr.Markdown("""
+        ### Description:
+        This is a multimodal chatbot that can handle both text and image inputs. 
+        - You can ask questions or provide text, and the assistant will respond.
+        - You can also upload an image, and the assistant will process it and answer questions about the image.
+        - Enter your OpenAI API key to start interacting with the model.
+        - You can use the 'Clear History' button to remove the conversation history.
+        - "o1" is for image chat and "o3-mini" is for text chat.
+        ### Reasoning Effort:
+        The reasoning effort controls how complex or detailed the assistant's answers should be. 
+        - **Low**: Provides quick, concise answers with minimal reasoning or details.
+        - **Medium**: Offers a balanced response with a reasonable level of detail and thought.
+        - **High**: Produces more detailed, analytical, or thoughtful responses, requiring deeper reasoning.
+        """)
+
+        with gr.Row():
+            openai_api_key = gr.Textbox(label="Enter OpenAI API Key", type="password", placeholder="sk-...", interactive=True)
+        
+        with gr.Row():
+            image_input = gr.Image(label="Upload an Image", type="pil")  # Image upload input
+            input_text = gr.Textbox(label="Enter Text Question", placeholder="Ask a question or provide text", lines=2)
+        
+        with gr.Row():
+            reasoning_effort = gr.Dropdown(
+                label="Reasoning Effort",
+                choices=["low", "medium", "high"],
+                value="medium"
+            )
+            model_choice = gr.Dropdown(
+                label="Select Model",
+                choices=["o1", "o3-mini"],
+                value="o1"  # Default to 'o1' for image-related tasks
+            )
+            submit_btn = gr.Button("Send")
+            clear_btn = gr.Button("Clear History")
+
+        chat_history = gr.Chatbot()
+
+        # Button interactions
+        submit_btn.click(fn=chatbot, inputs=[input_text, image_input, openai_api_key, reasoning_effort, model_choice, chat_history], outputs=[input_text, chat_history])
+        clear_btn.click(fn=clear_history, inputs=[], outputs=[chat_history, chat_history])
+
+    return demo
+
+# Run the interface
+if __name__ == "__main__":
+    demo = create_interface()
+    demo.launch()

abc2.txt

@@ -0,0 +1,299 @@
+import gradio as gr
+import openai
+import base64
+from PIL import Image
+import io
+
+# Function to send the request to OpenAI API with an image or text input
+def generate_response(input_text, image, openai_api_key, reasoning_effort="medium", model_choice="o1"):
+    if not openai_api_key:
+        return "Error: No API key provided."
+
+    openai.api_key = openai_api_key
+
+    # Process the input depending on whether it's text or an image
+    if image:
+        # Convert the image to base64 string
+        image_info = get_base64_string_from_image(image)
+        input_text = f"data:image/png;base64,{image_info}"
+
+    # Prepare the messages for OpenAI API
+    if model_choice == "o1":
+        if image:
+            messages = [
+                {"role": "user", "content": [{"type": "image_url", "image_url": {"url": input_text}}]}
+            ]
+        else:
+            messages = [
+                {"role": "user", "content": [{"type": "text", "text": input_text}]}
+            ]
+    elif model_choice == "o3-mini":
+        messages = [
+            {"role": "user", "content": [{"type": "text", "text": input_text}]}
+        ]
+    
+    try:
+        # Call OpenAI API with the selected model
+        response = openai.ChatCompletion.create(
+            model=model_choice,  # Dynamically choose the model (o1 or o3-mini)
+            messages=messages,
+            reasoning_effort=reasoning_effort,  # Set reasoning_effort for the response
+            max_completion_tokens=2000  # Limit response tokens to 2000
+        )
+
+        return response["choices"][0]["message"]["content"]
+    except Exception as e:
+        return f"Error calling OpenAI API: {str(e)}"
+
+# Function to convert an uploaded image to a base64 string
+def get_base64_string_from_image(pil_image):
+    # Convert PIL Image to bytes
+    buffered = io.BytesIO()
+    pil_image.save(buffered, format="PNG")
+    img_bytes = buffered.getvalue()
+    base64_str = base64.b64encode(img_bytes).decode("utf-8")
+    return base64_str
+
+# Function to transcribe audio to text using OpenAI Whisper API
+def transcribe_audio(audio, openai_api_key):
+    if not openai_api_key:
+        return "Error: No API key provided."
+    
+    openai.api_key = openai_api_key
+    
+    try:
+        # Open the audio file and pass it as a file object
+        with open(audio, 'rb') as audio_file:
+            audio_file_content = audio_file.read()
+        
+        # Use the correct transcription API call
+        audio_file_obj = io.BytesIO(audio_file_content)
+        audio_file_obj.name = 'audio.wav'  # Set a name for the file object (as OpenAI expects it)
+
+        # Transcribe the audio to text using OpenAI's whisper model
+        audio_file_transcription = openai.Audio.transcribe(file=audio_file_obj, model="whisper-1")
+        return audio_file_transcription['text']
+    except Exception as e:
+        return f"Error transcribing audio: {str(e)}"
+
+# The function that will be used by Gradio interface
+def chatbot(input_text, image, audio, openai_api_key, reasoning_effort, model_choice, history=[]):
+    # If there's audio, transcribe it to text
+    if audio:
+        input_text = transcribe_audio(audio, openai_api_key)
+
+    response = generate_response(input_text, image, openai_api_key, reasoning_effort, model_choice)
+    
+    # Append the response to the history
+    history.append((f"User: {input_text}", f"Assistant: {response}"))
+    
+    return "", history
+
+# Function to clear the chat history
+def clear_history():
+    return "", []
+
+# Custom CSS styles with animations and button colors
+custom_css = """
+    /* General body styles */
+    .gradio-container {
+        font-family: 'Arial', sans-serif;
+        background-color: #f8f9fa;
+        color: #333;
+    }
+    /* Header styles */
+    .gradio-header {
+        background-color: #007bff;
+        color: white;
+        padding: 20px;
+        text-align: center;
+        border-radius: 8px;
+        box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);
+        animation: fadeIn 1s ease-out;
+    }
+    .gradio-header h1 {
+        font-size: 2.5rem;
+    }
+    .gradio-header h3 {
+        font-size: 1.2rem;
+        margin-top: 10px;
+    }
+    /* Chatbot container styles */
+    .gradio-chatbot {
+        background-color: #fff;
+        border-radius: 10px;
+        padding: 20px;
+        box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);
+        max-height: 500px;
+        overflow-y: auto;
+        animation: fadeIn 2s ease-out;
+    }
+    /* Input field styles */
+    .gradio-textbox, .gradio-dropdown, .gradio-image, .gradio-audio {
+        border-radius: 8px;
+        border: 2px solid #ccc;
+        padding: 10px;
+        margin-bottom: 10px;
+        width: 100%;
+        font-size: 1rem;
+        transition: all 0.3s ease;
+    }
+    .gradio-textbox:focus, .gradio-dropdown:focus, .gradio-image:focus, .gradio-audio:focus {
+        border-color: #007bff;
+    }
+    /* Button styles */
+    /* Send Button: Sky Blue */
+    #submit-btn {
+        background-color: #00aaff; /* Sky blue */
+        color: white;
+        border: none;
+        border-radius: 8px;
+        padding: 10px 19px;
+        font-size: 1.1rem;
+        cursor: pointer;
+        transition: all 0.3s ease;
+        margin-left: auto;
+        margin-right: auto;
+        display: block;
+        margin-top: 10px;
+    }
+    #submit-btn:hover {
+        background-color: #0099cc; /* Slightly darker blue */
+    }
+    #submit-btn:active {
+        transform: scale(0.95);
+    }
+    #clear-history {
+        background-color: #f04e4e; /* Slightly Darker red */
+        color: white;
+        border: none;
+        border-radius: 8px;
+        padding: 10px 13px;
+        font-size: 1.1rem;
+        cursor: pointer;
+        transition: all 0.3s ease;
+        margin-top: 10px;
+    }
+    #clear-history:hover {
+        background-color: #f5a4a4; /* Light red */
+    }
+    #clear-history:active {
+        transform: scale(0.95);
+    }
+    /* Chat history styles */
+    .gradio-chatbot .message {
+        margin-bottom: 10px;
+    }
+    .gradio-chatbot .user {
+        background-color: #007bff;
+        color: white;
+        padding: 10px;
+        border-radius: 12px;
+        max-width: 70%;
+        animation: slideInUser 0.5s ease-out;
+    }
+    .gradio-chatbot .assistant {
+        background-color: #f1f1f1;
+        color: #333;
+        padding: 10px;
+        border-radius: 12px;
+        max-width: 70%;
+        margin-left: auto;
+        animation: slideInAssistant 0.5s ease-out;
+    }
+    /* Animation keyframes */
+    @keyframes fadeIn {
+        0% { opacity: 0; }
+        100% { opacity: 1; }
+    }
+    @keyframes slideInUser {
+        0% { transform: translateX(-100%); }
+        100% { transform: translateX(0); }
+    }
+    @keyframes slideInAssistant {
+        0% { transform: translateX(100%); }
+        100% { transform: translateX(0); }
+    }
+    /* Mobile responsiveness */
+    @media (max-width: 768px) {
+        .gradio-header h1 {
+            font-size: 1.8rem;
+        }
+        .gradio-header h3 {
+            font-size: 1rem;
+        }
+        .gradio-chatbot {
+            max-height: 400px;
+        }
+        .gradio-textbox, .gradio-dropdown, .gradio-image, .gradio-audio {
+            width: 100%;
+        }
+        #submit-btn, #clear-history {
+            width: 100%;
+            margin-left: 0;
+        }
+    }
+"""
+
+# Gradio interface setup
+def create_interface():
+    with gr.Blocks(css=custom_css) as demo:
+        gr.Markdown("""
+            <div class="gradio-header">
+                <h1>Multimodal Chatbot (Text + Image + Voice)</h1>
+                <h3>Interact with a chatbot using text, image, or voice inputs</h3>
+            </div>
+        """)
+
+        # Add a description with an expandable accordion
+        with gr.Accordion("Click to expand for details", open=False):
+            gr.Markdown("""
+            ### Description:
+            This is a multimodal chatbot that can handle text, image, and voice inputs. 
+            - You can ask questions or provide text, and the assistant will respond.
+            - You can also upload an image, and the assistant will process it and answer questions about the image.
+            - Voice input is supported: You can upload or record an audio file, and it will be transcribed to text and sent to the assistant.
+            - Enter your OpenAI API key to start interacting with the model.
+            - You can use the 'Clear History' button to remove the conversation history.
+            - "o1" is for image chat and "o3-mini" is for text chat.
+            ### Reasoning Effort:
+            The reasoning effort controls how complex or detailed the assistant's answers should be. 
+            - **Low**: Provides quick, concise answers with minimal reasoning or details.
+            - **Medium**: Offers a balanced response with a reasonable level of detail and thought.
+            - **High**: Produces more detailed, analytical, or thoughtful responses, requiring deeper reasoning.
+            """)
+
+        with gr.Row():
+            openai_api_key = gr.Textbox(label="Enter OpenAI API Key", type="password", placeholder="sk-...", interactive=True)
+
+        with gr.Row():
+            image_input = gr.Image(label="Upload an Image", type="pil")  # Image upload input
+            input_text = gr.Textbox(label="Enter Text Question", placeholder="Ask a question or provide text", lines=2)
+            audio_input = gr.Audio(label="Upload or Record Audio", type="filepath")  # Audio upload or record input (using filepath)
+
+        with gr.Row():
+            reasoning_effort = gr.Dropdown(
+                label="Reasoning Effort",
+                choices=["low", "medium", "high"],
+                value="medium"
+            )
+            model_choice = gr.Dropdown(
+                label="Select Model",
+                choices=["o1", "o3-mini"],
+                value="o1"  # Default to 'o1' for image-related tasks
+            )
+            submit_btn = gr.Button("Ask!", elem_id="submit-btn")
+            clear_btn = gr.Button("Clear History", elem_id="clear-history")
+
+        chat_history = gr.Chatbot()
+
+        # Button interactions
+        submit_btn.click(fn=chatbot, inputs=[input_text, image_input, audio_input, openai_api_key, reasoning_effort, model_choice, chat_history], outputs=[input_text, chat_history])
+        clear_btn.click(fn=clear_history, inputs=[], outputs=[chat_history, chat_history])
+
+    return demo
+
+# Run the interface
+if __name__ == "__main__":
+    demo = create_interface()
+    demo.launch()

abc3.txt

@@ -0,0 +1,523 @@
+import gradio as gr
+import openai
+import base64
+from PIL import Image
+import io
+import os
+import tempfile
+import fitz  # PyMuPDF for PDF handling
+
+# Function to extract text from PDF files
+def extract_text_from_pdf(pdf_file):
+    try:
+        text = ""
+        pdf_document = fitz.open(pdf_file)
+        
+        for page_num in range(len(pdf_document)):
+            page = pdf_document[page_num]
+            text += page.get_text()
+            
+        pdf_document.close()
+        return text
+    except Exception as e:
+        return f"Error extracting text from PDF: {str(e)}"
+
+# Function to generate MCQ quiz from PDF content
+def generate_mcq_quiz(pdf_content, num_questions, openai_api_key, model_choice):
+    if not openai_api_key:
+        return "Error: No API key provided."
+    
+    openai.api_key = openai_api_key
+    
+    # Limit content length to avoid token limits
+    limited_content = pdf_content[:8000] if len(pdf_content) > 8000 else pdf_content
+    
+    prompt = f"""Based on the following document content, generate {num_questions} multiple-choice quiz questions.
+For each question:
+1. Create a clear question based on key concepts in the document
+2. Provide 4 possible answers (A, B, C, D)
+3. Indicate the correct answer
+4. Briefly explain why the answer is correct
+Format the output clearly with each question numbered and separated.
+Document content:
+{limited_content}
+"""
+    
+    try:
+        messages = [
+            {"role": "user", "content": prompt}
+        ]
+        
+        response = openai.ChatCompletion.create(
+            model=model_choice,
+            messages=messages
+        )
+        
+        return response.choices[0].message.content
+    except Exception as e:
+        return f"Error generating quiz: {str(e)}"
+
+# Function to send the request to OpenAI API with an image, text or PDF input
+def generate_response(input_text, image, pdf_content, openai_api_key, reasoning_effort="medium", model_choice="o1"):
+    if not openai_api_key:
+        return "Error: No API key provided."
+
+    openai.api_key = openai_api_key
+
+    # Process the input depending on whether it's text, image, or a PDF-related query
+    if pdf_content and input_text:
+        # For PDF queries, we combine the PDF content with the user's question
+        prompt = f"Based on the following document content, please answer this question: '{input_text}'\n\nDocument content:\n{pdf_content}"
+        input_content = prompt
+    elif image:
+        # Convert the image to base64 string
+        image_info = get_base64_string_from_image(image)
+        input_content = f"data:image/png;base64,{image_info}"
+    else:
+        # Plain text input
+        input_content = input_text
+
+    # Prepare the messages for OpenAI API
+    if model_choice == "o1":
+        if image and not pdf_content:
+            messages = [
+                {"role": "user", "content": [{"type": "image_url", "image_url": {"url": input_content}}]}
+            ]
+        else:
+            messages = [
+                {"role": "user", "content": input_content}
+            ]
+    elif model_choice == "o3-mini":
+        messages = [
+            {"role": "user", "content": input_content}
+        ]
+    
+    try:
+        # Call OpenAI API with the selected model
+        response = openai.ChatCompletion.create(
+            model=model_choice,
+            messages=messages,
+            max_completion_tokens=2000
+        )
+
+        return response.choices[0].message.content
+    except Exception as e:
+        return f"Error calling OpenAI API: {str(e)}"
+
+# Function to convert an uploaded image to a base64 string
+def get_base64_string_from_image(pil_image):
+    # Convert PIL Image to bytes
+    buffered = io.BytesIO()
+    pil_image.save(buffered, format="PNG")
+    img_bytes = buffered.getvalue()
+    base64_str = base64.b64encode(img_bytes).decode("utf-8")
+    return base64_str
+
+# Function to transcribe audio to text using OpenAI Whisper API
+def transcribe_audio(audio, openai_api_key):
+    if not openai_api_key:
+        return "Error: No API key provided."
+    
+    openai.api_key = openai_api_key
+    
+    try:
+        # Open the audio file and pass it as a file object
+        with open(audio, 'rb') as audio_file:
+            audio_file_content = audio_file.read()
+        
+        # Use the correct transcription API call
+        audio_file_obj = io.BytesIO(audio_file_content)
+        audio_file_obj.name = 'audio.wav'  # Set a name for the file object (as OpenAI expects it)
+
+        # Transcribe the audio to text using OpenAI's whisper model
+        audio_file_transcription = openai.Audio.transcribe(file=audio_file_obj, model="whisper-1")
+        return audio_file_transcription.text
+    except Exception as e:
+        return f"Error transcribing audio: {str(e)}"
+
+# The function that will be used by Gradio interface
+def chatbot(input_text, image, audio, pdf_file, openai_api_key, reasoning_effort, model_choice, pdf_content, num_quiz_questions, pdf_quiz_mode, history):
+    if history is None:
+        history = []
+        
+    # If there's audio, transcribe it to text
+    if audio:
+        input_text = transcribe_audio(audio, openai_api_key)
+    
+    # If a new PDF is uploaded, extract its text
+    new_pdf_content = pdf_content
+    if pdf_file is not None:
+        new_pdf_content = extract_text_from_pdf(pdf_file)
+    
+    # Check if we're in PDF quiz mode
+    if pdf_quiz_mode:
+        if new_pdf_content:
+            # Generate MCQ quiz questions
+            quiz_response = generate_mcq_quiz(new_pdf_content, int(num_quiz_questions), openai_api_key, model_choice)
+            history.append((f"User: [Uploaded PDF for Quiz - {int(num_quiz_questions)} questions]", f"Assistant: {quiz_response}"))
+        else:
+            history.append(("User: [Attempted to generate quiz without PDF]", "Assistant: Please upload a PDF file to generate quiz questions."))
+    else:
+        # Regular chat mode - generate the response
+        response = generate_response(input_text, image, new_pdf_content, openai_api_key, reasoning_effort, model_choice)
+        
+        # Append the response to the history
+        if input_text:
+            history.append((f"User: {input_text}", f"Assistant: {response}"))
+        elif image is not None:
+            history.append((f"User: [Uploaded image]", f"Assistant: {response}"))
+        elif pdf_file is not None:
+            history.append((f"User: [Uploaded PDF]", f"Assistant: {response}"))
+        else:
+            history.append((f"User: [No input provided]", f"Assistant: Please provide some input (text, image, or PDF) for me to respond to."))
+    
+    return "", None, None, None, new_pdf_content, history
+
+# Function to clear the chat history and PDF content
+def clear_history():
+    return "", None, None, None, "", []
+
+# Function to process a newly uploaded PDF
+def process_pdf(pdf_file):
+    if pdf_file is None:
+        return ""
+    return extract_text_from_pdf(pdf_file)
+
+# Function to update visible components based on input type selection
+def update_input_type(choice):
+    if choice == "Text":
+        return gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(value=False)
+    elif choice == "Image":
+        return gr.update(visible=True), gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(value=False)
+    elif choice == "Voice":
+        return gr.update(visible=False), gr.update(visible=False), gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(value=False)
+    elif choice == "PDF":
+        return gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(visible=True), gr.update(visible=False), gr.update(value=False)
+    elif choice == "PDF(QUIZ)":
+        return gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(visible=True), gr.update(visible=True), gr.update(value=True)
+    
+# Custom CSS styles with animations and button colors
+custom_css = """
+    /* General body styles */
+    .gradio-container {
+        font-family: 'Arial', sans-serif;
+        background-color: #f8f9fa;
+        color: #333;
+    }
+    /* Header styles */
+    .gradio-header {
+        background-color: #007bff;
+        color: white;
+        padding: 20px;
+        text-align: center;
+        border-radius: 8px;
+        box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);
+        animation: fadeIn 1s ease-out;
+    }
+    .gradio-header h1 {
+        font-size: 2.5rem;
+    }
+    .gradio-header h3 {
+        font-size: 1.2rem;
+        margin-top: 10px;
+    }
+    /* Chatbot container styles */
+    .gradio-chatbot {
+        background-color: #fff;
+        border-radius: 10px;
+        padding: 20px;
+        box-shadow: 0 4px 8px rgba(0, 0, 0, 0.1);
+        max-height: 500px;
+        overflow-y: auto;
+        animation: fadeIn 2s ease-out;
+    }
+    /* Input field styles */
+    .gradio-textbox, .gradio-dropdown, .gradio-image, .gradio-audio, .gradio-file, .gradio-slider {
+        border-radius: 8px;
+        border: 2px solid #ccc;
+        padding: 10px;
+        margin-bottom: 10px;
+        width: 100%;
+        font-size: 1rem;
+        transition: all 0.3s ease;
+    }
+    .gradio-textbox:focus, .gradio-dropdown:focus, .gradio-image:focus, .gradio-audio:focus, .gradio-file:focus, .gradio-slider:focus {
+        border-color: #007bff;
+    }
+    /* Button styles */
+    /* Send Button: Sky Blue */
+    #submit-btn {
+        background-color: #00aaff; /* Sky blue */
+        color: white;
+        border: none;
+        border-radius: 8px;
+        padding: 10px 19px;
+        font-size: 1.1rem;
+        cursor: pointer;
+        transition: all 0.3s ease;
+        margin-left: auto;
+        margin-right: auto;
+        display: block;
+        margin-top: 10px;
+    }
+    #submit-btn:hover {
+        background-color: #0099cc; /* Slightly darker blue */
+    }
+    #submit-btn:active {
+        transform: scale(0.95);
+    }
+    #clear-history {
+        background-color: #f04e4e; /* Slightly Darker red */
+        color: white;
+        border: none;
+        border-radius: 8px;
+        padding: 10px 13px;
+        font-size: 1.1rem;
+        cursor: pointer;
+        transition: all 0.3s ease;
+        margin-top: 10px;
+    }
+    #clear-history:hover {
+        background-color: #f5a4a4; /* Light red */
+    }
+    #clear-history:active {
+        transform: scale(0.95);
+    }
+    /* Input type selector buttons */
+    #input-type-group {
+        display: flex;
+        justify-content: center;
+        gap: 10px;
+        margin-bottom: 20px;
+    }
+    .input-type-btn {
+        background-color: #6c757d;
+        color: white;
+        border: none;
+        border-radius: 8px;
+        padding: 10px 15px;
+        font-size: 1rem;
+        cursor: pointer;
+        transition: all 0.3s ease;
+    }
+    .input-type-btn.selected {
+        background-color: #007bff;
+    }
+    .input-type-btn:hover {
+        background-color: #5a6268;
+    }
+    /* Chat history styles */
+    .gradio-chatbot .message {
+        margin-bottom: 10px;
+    }
+    .gradio-chatbot .user {
+        background-color: #007bff;
+        color: white;
+        padding: 10px;
+        border-radius: 12px;
+        max-width: 70%;
+        animation: slideInUser 0.5s ease-out;
+    }
+    .gradio-chatbot .assistant {
+        background-color: #f1f1f1;
+        color: #333;
+        padding: 10px;
+        border-radius: 12px;
+        max-width: 70%;
+        margin-left: auto;
+        animation: slideInAssistant 0.5s ease-out;
+    }
+    /* Animation keyframes */
+    @keyframes fadeIn {
+        0% { opacity: 0; }
+        100% { opacity: 1; }
+    }
+    @keyframes slideInUser {
+        0% { transform: translateX(-100%); }
+        100% { transform: translateX(0); }
+    }
+    @keyframes slideInAssistant {
+        0% { transform: translateX(100%); }
+        100% { transform: translateX(0); }
+    }
+    /* Mobile responsiveness */
+    @media (max-width: 768px) {
+        .gradio-header h1 {
+            font-size: 1.8rem;
+        }
+        .gradio-header h3 {
+            font-size: 1rem;
+        }
+        .gradio-chatbot {
+            max-height: 400px;
+        }
+        .gradio-textbox, .gradio-dropdown, .gradio-image, .gradio-audio, .gradio-file, .gradio-slider {
+            width: 100%;
+        }
+        #submit-btn, #clear-history {
+            width: 100%;
+            margin-left: 0;
+        }
+    }
+"""
+
+# Gradio interface setup
+def create_interface():
+    with gr.Blocks(css=custom_css) as demo:
+        gr.Markdown("""
+            <div class="gradio-header">
+                <h1>Multimodal Chatbot (Text + Image + Voice + PDF + Quiz)</h1>
+                <h3>Interact with a chatbot using text, image, voice, or PDF inputs</h3>
+            </div>
+        """)
+
+        # Add a description with an expandable accordion
+        with gr.Accordion("Click to expand for details", open=False):
+            gr.Markdown("""
+            ### Description:
+            This is a multimodal chatbot that can handle text, image, voice, PDF inputs, and generate quizzes from PDFs. 
+            - You can ask questions or provide text, and the assistant will respond.
+            - You can upload an image, and the assistant will process it and answer questions about the image.
+            - Voice input is supported: You can upload or record an audio file, and it will be transcribed to text and sent to the assistant.
+            - PDF support: Upload a PDF and ask questions about its content.
+            - PDF Quiz: Upload a PDF and specify how many MCQ questions you want generated based on the content.
+            - Enter your OpenAI API key to start interacting with the model.
+            - You can use the 'Clear History' button to remove the conversation history.
+            - "o1" is for image, voice, PDF and text chat and "o3-mini" is for text, PDF and voice chat only.
+            ### Reasoning Effort:
+            The reasoning effort controls how complex or detailed the assistant's answers should be. 
+            - **Low**: Provides quick, concise answers with minimal reasoning or details.
+            - **Medium**: Offers a balanced response with a reasonable level of detail and thought.
+            - **High**: Produces more detailed, analytical, or thoughtful responses, requiring deeper reasoning.
+            """)
+
+        # Store PDF content as a state variable
+        pdf_content = gr.State("")
+
+        with gr.Row():
+            openai_api_key = gr.Textbox(label="Enter OpenAI API Key", type="password", placeholder="sk-...", interactive=True)
+
+        # Input type selector
+        with gr.Row():
+            input_type = gr.Radio(
+                ["Text", "Image", "Voice", "PDF", "PDF(QUIZ)"], 
+                label="Choose Input Type", 
+                value="Text"
+            )
+
+        # Create the input components (initially text is visible, others are hidden)
+        with gr.Row():
+            # Text input
+            input_text = gr.Textbox(
+                label="Enter Text Question", 
+                placeholder="Ask a question or provide text", 
+                lines=2,
+                visible=True
+            )
+            
+            # Image input
+            image_input = gr.Image(
+                label="Upload an Image", 
+                type="pil",
+                visible=False
+            )
+            
+            # Audio input
+            audio_input = gr.Audio(
+                label="Upload or Record Audio", 
+                type="filepath",
+                visible=False
+            )
+            
+            # PDF input
+            pdf_input = gr.File(
+                label="Upload your PDF",
+                file_types=[".pdf"],
+                visible=False
+            )
+            
+            # Quiz specific components
+            quiz_questions_slider = gr.Slider(
+                minimum=1, 
+                maximum=20, 
+                value=5, 
+                step=1, 
+                label="Number of Quiz Questions", 
+                visible=False
+            )
+            
+            # Hidden state for quiz mode
+            quiz_mode = gr.Checkbox(
+                label="Quiz Mode", 
+                visible=False,
+                value=False
+            )
+
+        with gr.Row():
+            reasoning_effort = gr.Dropdown(
+                label="Reasoning Effort",
+                choices=["low", "medium", "high"],
+                value="medium"
+            )
+            model_choice = gr.Dropdown(
+                label="Select Model",
+                choices=["o1", "o3-mini"],
+                value="o1"  # Default to 'o1' for image-related tasks
+            )
+            submit_btn = gr.Button("Ask!", elem_id="submit-btn")
+            clear_btn = gr.Button("Clear History", elem_id="clear-history")
+
+        chat_history = gr.Chatbot()
+
+        # Connect the input type selector to the update function
+        input_type.change(
+            fn=update_input_type,
+            inputs=[input_type],
+            outputs=[input_text, image_input, audio_input, pdf_input, quiz_questions_slider, quiz_mode]
+        )
+        
+        # Process PDF when uploaded
+        pdf_input.change(
+            fn=process_pdf,
+            inputs=[pdf_input],
+            outputs=[pdf_content]
+        )
+
+        # Button interactions
+        submit_btn.click(
+            fn=chatbot, 
+            inputs=[
+                input_text, 
+                image_input, 
+                audio_input, 
+                pdf_input, 
+                openai_api_key, 
+                reasoning_effort, 
+                model_choice, 
+                pdf_content, 
+                quiz_questions_slider, 
+                quiz_mode, 
+                chat_history
+            ], 
+            outputs=[
+                input_text, 
+                image_input, 
+                audio_input, 
+                pdf_input, 
+                pdf_content, 
+                chat_history
+            ]
+        )
+        
+        clear_btn.click(
+            fn=clear_history, 
+            inputs=[], 
+            outputs=[input_text, image_input, audio_input, pdf_input, pdf_content, chat_history]
+        )
+
+    return demo
+
+# Run the interface
+if __name__ == "__main__":
+    demo = create_interface()
+    demo.launch()

app.py

@@ -0,0 +1,522 @@
+import gradio as gr
+import openai
+import base64
+from PIL import Image
+import io
+import os
+import tempfile
+import fitz  # PyMuPDF for PDF handling
+
+# Function to extract text from PDF files
+def extract_text_from_pdf(pdf_file):
+    try:
+        text = ""
+        pdf_document = fitz.open(pdf_file)
+        
+        for page_num in range(len(pdf_document)):
+            page = pdf_document[page_num]
+            text += page.get_text()
+            
+        pdf_document.close()
+        return text
+    except Exception as e:
+        return f"Error extracting text from PDF: {str(e)}"
+
+# Function to generate MCQ quiz from PDF content
+def generate_mcq_quiz(pdf_content, num_questions, openai_api_key, model_choice):
+    if not openai_api_key:
+        return "Error: No API key provided."
+    
+    openai.api_key = openai_api_key
+    
+    # Limit content length to avoid token limits
+    limited_content = pdf_content[:8000] if len(pdf_content) > 8000 else pdf_content
+    
+    prompt = f"""Based on the following document content, generate {num_questions} multiple-choice quiz questions.
+For each question:
+1. Create a clear question based on key concepts in the document
+2. Provide 4 possible answers (A, B, C, D)
+3. Indicate the correct answer
+4. Briefly explain why the answer is correct
+
+Format the output clearly with each question numbered and separated.
+
+Document content:
+{limited_content}
+"""
+    
+    try:
+        messages = [
+            {"role": "user", "content": prompt}
+        ]
+        
+        response = openai.ChatCompletion.create(
+            model=model_choice,
+            messages=messages
+        )
+        
+        return response.choices[0].message.content
+    except Exception as e:
+        return f"Error generating quiz: {str(e)}"
+
+# Function to send the request to OpenAI API with an image, text or PDF input
+def generate_response(input_text, image, pdf_content, openai_api_key, reasoning_effort="medium", model_choice="o1"):
+    if not openai_api_key:
+        return "Error: No API key provided."
+
+    openai.api_key = openai_api_key
+
+    # Process the input depending on whether it's text, image, or a PDF-related query
+    if pdf_content and input_text:
+        # For PDF queries, we combine the PDF content with the user's question
+        prompt = f"Based on the following document content, please answer this question: '{input_text}'\n\nDocument content:\n{pdf_content}"
+        input_content = prompt
+    elif image:
+        # Convert the image to base64 string
+        image_info = get_base64_string_from_image(image)
+        input_content = f"data:image/png;base64,{image_info}"
+    else:
+        # Plain text input
+        input_content = input_text
+
+    # Prepare the messages for OpenAI API
+    if model_choice == "o1":
+        if image and not pdf_content:
+            messages = [
+                {"role": "user", "content": [{"type": "image_url", "image_url": {"url": input_content}}]}
+            ]
+        else:
+            messages = [
+                {"role": "user", "content": input_content}
+            ]
+    elif model_choice == "o3-mini":
+        messages = [
+            {"role": "user", "content": input_content}
+        ]
+    
+    try:
+        # Call OpenAI API with the selected model
+        response = openai.ChatCompletion.create(
+            model=model_choice,
+            messages=messages,
+            max_completion_tokens=2000
+        )
+
+        return response.choices[0].message.content
+    except Exception as e:
+        return f"Error calling OpenAI API: {str(e)}"
+
+# Function to convert an uploaded image to a base64 string
+def get_base64_string_from_image(pil_image):
+    # Convert PIL Image to bytes
+    buffered = io.BytesIO()
+    pil_image.save(buffered, format="PNG")
+    img_bytes = buffered.getvalue()
+    base64_str = base64.b64encode(img_bytes).decode("utf-8")
+    return base64_str
+
+# Function to transcribe audio to text using OpenAI Whisper API
+def transcribe_audio(audio, openai_api_key):
+    if not openai_api_key:
+        return "Error: No API key provided."
+    
+    openai.api_key = openai_api_key
+    
+    try:
+        # Open the audio file and pass it as a file object
+        with open(audio, 'rb') as audio_file:
+            audio_file_content = audio_file.read()
+        
+        # Use the correct transcription API call
+        audio_file_obj = io.BytesIO(audio_file_content)
+        audio_file_obj.name = 'audio.wav'  # Set a name for the file object (as OpenAI expects it)
+
+        # Transcribe the audio to text using OpenAI's whisper model
+        audio_file_transcription = openai.Audio.transcribe(file=audio_file_obj, model="whisper-1")
+        return audio_file_transcription.text
+    except Exception as e:
+        return f"Error transcribing audio: {str(e)}"
+
+# The function that will be used by Gradio interface
+def chatbot(input_text, image, audio, pdf_file, openai_api_key, reasoning_effort, model_choice, pdf_content, num_quiz_questions, pdf_quiz_mode, history):
+    if history is None:
+        history = []
+        
+    # If there's audio, transcribe it to text
+    if audio:
+        input_text = transcribe_audio(audio, openai_api_key)
+    
+    # If a new PDF is uploaded, extract its text
+    new_pdf_content = pdf_content
+    if pdf_file is not None:
+        new_pdf_content = extract_text_from_pdf(pdf_file)
+    
+    # Check if we're in PDF quiz mode
+    if pdf_quiz_mode:
+        if new_pdf_content:
+            # Generate MCQ quiz questions
+            quiz_response = generate_mcq_quiz(new_pdf_content, int(num_quiz_questions), openai_api_key, model_choice)
+            history.append((f"👤: [Uploaded PDF for Quiz - {int(num_quiz_questions)} questions]", f"🤖: {quiz_response}"))
+        else:
+            history.append(("👤: [Attempted to generate quiz without PDF]", "🤖: Please upload a PDF file to generate quiz questions."))
+    else:
+        # Regular chat mode - generate the response
+        response = generate_response(input_text, image, new_pdf_content, openai_api_key, reasoning_effort, model_choice)
+        
+        # Append the response to the history
+        if input_text:
+            history.append((f"👤: {input_text}", f"🤖: {response}"))
+        elif image is not None:
+            history.append((f"👤: [Uploaded image]", f"🤖: {response}"))
+        elif pdf_file is not None:
+            history.append((f"👤: [Uploaded PDF]", f"🤖: {response}"))
+        else:
+            history.append((f"👤: [No input provided]", f"🤖: Please provide some input (text, image, or PDF) for me to respond to."))
+    
+    return "", None, None, None, new_pdf_content, history
+
+# Function to clear the chat history and PDF content
+def clear_history():
+    return "", None, None, None, "", []
+
+# Function to process a newly uploaded PDF
+def process_pdf(pdf_file):
+    if pdf_file is None:
+        return ""
+    return extract_text_from_pdf(pdf_file)
+
+# Function to update visible components based on input type selection
+def update_input_type(choice):
+    if choice == "Text":
+        return gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(value=False)
+    elif choice == "Image":
+        return gr.update(visible=True), gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(value=False)
+    elif choice == "Voice":
+        return gr.update(visible=False), gr.update(visible=False), gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(value=False)
+    elif choice == "PDF":
+        return gr.update(visible=True), gr.update(visible=False), gr.update(visible=False), gr.update(visible=True), gr.update(visible=False), gr.update(value=False)
+    elif choice == "PDF(QUIZ)":
+        return gr.update(visible=False), gr.update(visible=False), gr.update(visible=False), gr.update(visible=True), gr.update(visible=True), gr.update(value=True)
+    
+# Custom CSS styles with animations and button colors
+custom_css = """
+    /* General body styles */
+    .gradio-container {
+        font-family: 'Arial', sans-serif;
+        background-color: #f0f4f8; /* Lighter blue-gray background */
+        color: #2d3748;;
+    }
+    /* Header styles */
+    .gradio-header {
+        background: linear-gradient(135deg, #4a00e0 0%, #8e2de2 100%); /* Purple gradient */
+        color: white;
+        padding: 20px;
+        text-align: center;
+        border-radius: 8px;
+        box-shadow: 0 4px 15px rgba(0, 0, 0, 0.2);
+        animation: fadeIn 1s ease-out;
+    }
+    .gradio-header h1 {
+        font-size: 2.5rem;
+    }
+    .gradio-header h3 {
+        font-size: 1.2rem;
+        margin-top: 10px;
+    }
+    /* Chatbot container styles */
+    .gradio-chatbot {
+        background-color: #fff;
+        border-radius: 10px;
+        padding: 20px;
+        box-shadow: 0 6px 18px rgba(0, 0, 0, 0.1);
+        border-left: 4px solid #4a00e0; /* Accent border */
+    }
+    /* Input field styles */
+    .gradio-textbox, .gradio-dropdown, .gradio-image, .gradio-audio, .gradio-file, .gradio-slider {
+        border-radius: 8px;
+        border: 2px solid #e2e8f0;
+        background-color: #f8fafc;
+    }
+    .gradio-textbox:focus, .gradio-dropdown:focus, .gradio-image:focus, .gradio-audio:focus, .gradio-file:focus, .gradio-slider:focus {
+        border-color: #8e2de2;
+        box-shadow: 0 0 0 3px rgba(142, 45, 226, 0.2);
+    }
+    /* Button styles */
+    /* Send Button: Sky Blue */
+    #submit-btn {
+        background: linear-gradient(135deg, #4a00e0 0%, #8e2de2 100%); /* Purple gradient */
+        color: white;
+        border: none;
+        border-radius: 8px;
+        padding: 10px 19px;
+        font-size: 1.1rem;
+        cursor: pointer;
+        transition: all 0.3s ease;
+        margin-left: auto;
+        margin-right: auto;
+        display: block;
+        margin-top: 10px;
+    }
+    #submit-btn:hover {
+        background: linear-gradient(135deg, #5b10f1 0%, #9f3ef3 100%); /* Slightly lighter */
+        box-shadow: 0 6px 8px rgba(74, 0, 224, 0.4);
+    }
+    #submit-btn:active {
+        transform: scale(0.95);
+    }
+    #clear-history {
+        background: linear-gradient(135deg, #e53e3e 0%, #f56565 100%); /* Red gradient */
+        color: white;
+        border: none;
+        border-radius: 8px;
+        padding: 10px 13px;
+        font-size: 1.1rem;
+        cursor: pointer;
+        transition: all 0.3s ease;
+        margin-top: 10px;
+    }
+    #clear-history:hover {
+        background: linear-gradient(135deg, #c53030 0%, #e53e3e 100%); /* Slightly darker red gradient on hover */
+        box-shadow: 0 6px 8px rgba(229, 62, 62, 0.4);
+    }
+    #clear-history:active {
+        transform: scale(0.95);
+    }
+    /* Input type selector buttons */
+    #input-type-group {
+        display: flex;
+        justify-content: center;
+        gap: 10px;
+        margin-bottom: 20px;
+    }
+    .input-type-btn {
+        background-color: #718096; /* Slate gray */
+        color: white;
+        border: none;
+        border-radius: 8px;
+        padding: 10px 15px;
+        font-size: 1rem;
+        cursor: pointer;
+        transition: all 0.3s ease;
+    }
+    .input-type-btn.selected {
+        background-color: linear-gradient(135deg, #4a00e0 0%, #8e2de2 100%); /* Purple gradient */
+    }
+    .input-type-btn:hover {
+        background-color: #4a5568; /* Darker slate */
+    }
+    /* Chat history styles */
+    .gradio-chatbot .message {
+        margin-bottom: 10px;
+    }
+    .gradio-chatbot .user {
+        background-color: linear-gradient(135deg, #4a00e0 0%, #8e2de2 100%); /* Purple gradient */
+        color: white;
+        padding: 10px;
+        border-radius: 12px;
+        max-width: 70%;
+        animation: slideInUser 0.5s ease-out;
+    }
+    .gradio-chatbot .assistant {
+        background-color: #f0f4f8; /* Light blue-gray */
+        color: #2d3748;
+        padding: 10px;
+        border-radius: 12px;
+        max-width: 70%;
+        margin-left: auto;
+        animation: slideInAssistant 0.5s ease-out;
+    }
+    /* Animation keyframes */
+    @keyframes fadeIn {
+        0% { opacity: 0; }
+        100% { opacity: 1; }
+    }
+    @keyframes slideInUser {
+        0% { transform: translateX(-100%); }
+        100% { transform: translateX(0); }
+    }
+    @keyframes slideInAssistant {
+        0% { transform: translateX(100%); }
+        100% { transform: translateX(0); }
+    }
+    /* Mobile responsiveness */
+    @media (max-width: 768px) {
+        .gradio-header h1 {
+            font-size: 1.8rem;
+        }
+        .gradio-header h3 {
+            font-size: 1rem;
+        }
+        .gradio-chatbot {
+            max-height: 400px;
+        }
+        .gradio-textbox, .gradio-dropdown, .gradio-image, .gradio-audio, .gradio-file, .gradio-slider {
+            width: 100%;
+        }
+        #submit-btn, #clear-history {
+            width: 100%;
+            margin-left: 0;
+        }
+    }
+"""
+
+# Gradio interface setup
+def create_interface():
+    with gr.Blocks(css=custom_css) as demo:
+        gr.Markdown("""
+            <div class="gradio-header">
+                <h1>Multimodal Chatbot (Text + Image + Voice + PDF + Quiz)</h1>
+                <h3>Interact with a chatbot using text, image, voice, or PDF inputs</h3>
+            </div>
+        """)
+
+        # Add a description with an expandable accordion
+        with gr.Accordion("Click to expand for details", open=False):
+            gr.Markdown("""
+            ### Description:
+            This is a multimodal chatbot that can handle text, image, voice, PDF inputs, and generate quizzes from PDFs. 
+            - You can ask questions or provide text, and the assistant will respond.
+            - You can upload an image, and the assistant will process it and answer questions about the image.
+            - Voice input is supported: You can upload or record an audio file, and it will be transcribed to text and sent to the assistant.
+            - PDF support: Upload a PDF and ask questions about its content.
+            - PDF Quiz: Upload a PDF and specify how many MCQ questions you want generated based on the content.
+            - Enter your OpenAI API key to start interacting with the model.
+            - You can use the 'Clear History' button to remove the conversation history.
+            - "o1" is for image, voice, PDF and text chat and "o3-mini" is for text, PDF and voice chat only.
+            ### Reasoning Effort:
+            The reasoning effort controls how complex or detailed the assistant's answers should be. 
+            - **Low**: Provides quick, concise answers with minimal reasoning or details.
+            - **Medium**: Offers a balanced response with a reasonable level of detail and thought.
+            - **High**: Produces more detailed, analytical, or thoughtful responses, requiring deeper reasoning.
+            """)
+
+        # Store PDF content as a state variable
+        pdf_content = gr.State("")
+
+        with gr.Row():
+            openai_api_key = gr.Textbox(label="Enter OpenAI API Key", type="password", placeholder="sk-...", interactive=True)
+
+        # Input type selector
+        with gr.Row():
+            input_type = gr.Radio(
+                ["Text", "Image", "Voice", "PDF", "PDF(QUIZ)"], 
+                label="Choose Input Type", 
+                value="Text"
+            )
+
+        # Create the input components (initially text is visible, others are hidden)
+        with gr.Row():
+            # Text input
+            input_text = gr.Textbox(
+                label="Enter Text Question", 
+                placeholder="Ask a question or provide text", 
+                lines=2,
+                visible=True
+            )
+            
+            # Image input
+            image_input = gr.Image(
+                label="Upload an Image", 
+                type="pil",
+                visible=False
+            )
+            
+            # Audio input
+            audio_input = gr.Audio(
+                label="Upload or Record Audio", 
+                type="filepath",
+                visible=False
+            )
+            
+            # PDF input
+            pdf_input = gr.File(
+                label="Upload your PDF",
+                file_types=[".pdf"],
+                visible=False
+            )
+            
+            # Quiz specific components
+            quiz_questions_slider = gr.Slider(
+                minimum=1, 
+                maximum=20, 
+                value=5, 
+                step=1, 
+                label="Number of Quiz Questions", 
+                visible=False
+            )
+            
+            # Hidden state for quiz mode
+            quiz_mode = gr.Checkbox(
+                label="Quiz Mode", 
+                visible=False,
+                value=False
+            )
+
+        with gr.Row():
+            reasoning_effort = gr.Dropdown(
+                label="Reasoning Effort",
+                choices=["low", "medium", "high"],
+                value="medium"
+            )
+            model_choice = gr.Dropdown(
+                label="Select Model",
+                choices=["o1", "o3-mini"],
+                value="o1"  # Default to 'o1' for image-related tasks
+            )
+            submit_btn = gr.Button("Ask!", elem_id="submit-btn")
+            clear_btn = gr.Button("Clear History", elem_id="clear-history")
+
+        chat_history = gr.Chatbot()
+
+        # Connect the input type selector to the update function
+        input_type.change(
+            fn=update_input_type,
+            inputs=[input_type],
+            outputs=[input_text, image_input, audio_input, pdf_input, quiz_questions_slider, quiz_mode]
+        )
+        
+        # Process PDF when uploaded
+        pdf_input.change(
+            fn=process_pdf,
+            inputs=[pdf_input],
+            outputs=[pdf_content]
+        )
+
+        # Button interactions
+        submit_btn.click(
+            fn=chatbot, 
+            inputs=[
+                input_text, 
+                image_input, 
+                audio_input, 
+                pdf_input, 
+                openai_api_key, 
+                reasoning_effort, 
+                model_choice, 
+                pdf_content, 
+                quiz_questions_slider, 
+                quiz_mode, 
+                chat_history
+            ], 
+            outputs=[
+                input_text, 
+                image_input, 
+                audio_input, 
+                pdf_input, 
+                pdf_content, 
+                chat_history
+            ]
+        )
+        
+        clear_btn.click(
+            fn=clear_history, 
+            inputs=[], 
+            outputs=[input_text, image_input, audio_input, pdf_input, pdf_content, chat_history]
+        )
+
+    return demo
+
+# Run the interface
+if __name__ == "__main__":
+    demo = create_interface()
+    demo.launch()

generate_answer.py

@@ -0,0 +1,95 @@
+import os
+from glob import glob
+import openai
+from dotenv import load_dotenv
+
+from langchain.embeddings import OpenAIEmbeddings
+from langchain.vectorstores import Chroma
+from langchain.document_loaders import PyPDFLoader
+from langchain.text_splitter import RecursiveCharacterTextSplitter
+
+from langchain_community.chat_models import ChatOpenAI
+from langchain.chains import RetrievalQA
+from langchain.memory import ConversationBufferMemory
+
+load_dotenv()
+api_key = os.getenv("OPENAI_API_KEY")
+openai.api_key = api_key
+
+# Helper function to validate response completeness
+def is_response_complete(response: str) -> bool:
+    return response.strip()[-1] in ".!?"
+
+# Retry mechanism for incomplete responses
+def retry_response(messages):
+    response = openai.ChatCompletion.create(
+        model="gpt-4o-mini",
+        messages=messages
+    ).choices[0].message['content']
+    if not is_response_complete(response):
+        response += " This is the end of the response. Please let me know if you need further clarification."
+    return response
+
+def base_model_chatbot(messages):
+    system_message = [
+        {"role": "system", "content": "You are a helpful AI chatbot that provides clear, complete, and coherent responses to User's questions. Ensure your answers are in full sentences and complete the thought or idea."}
+    ]
+    messages = system_message + messages
+    response = openai.ChatCompletion.create(
+        model="gpt-4o-mini",
+        messages=messages
+    ).choices[0].message['content']
+    # Validate response completeness
+    if not is_response_complete(response):
+        response = retry_response(messages)
+    return response
+
+class VectorDB:
+    """Class to manage document loading and vector database creation."""
+    
+    def __init__(self, docs_directory: str):
+        self.docs_directory = docs_directory
+
+    def create_vector_db(self):
+        text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
+
+        files = glob(os.path.join(self.docs_directory, "*.pdf"))
+
+        loadPDFs = [PyPDFLoader(pdf_file) for pdf_file in files]
+
+        pdf_docs = list()
+        for loader in loadPDFs:
+            pdf_docs.extend(loader.load())
+        chunks = text_splitter.split_documents(pdf_docs)
+            
+        return Chroma.from_documents(chunks, OpenAIEmbeddings()) 
+    
+class ConversationalRetrievalChain:
+    """Class to manage the QA chain setup."""
+
+    def __init__(self, model_name="gpt-3.5-turbo", temperature=0):
+        self.model_name = model_name
+        self.temperature = temperature
+
+    def create_chain(self):
+        model = ChatOpenAI(
+            model_name=self.model_name, 
+            temperature=self.temperature,
+            system_prompt="You are a knowledgeable AI that answers questions based on provided documents. Always give responses in clear, complete sentences."
+        )
+        memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)
+        vector_db = VectorDB('docs/')
+        retriever = vector_db.create_vector_db().as_retriever(search_type="similarity", search_kwargs={"k": 2})
+        return RetrievalQA.from_chain_type(
+            llm=model,
+            retriever=retriever,
+            memory=memory,
+        )
+    
+def with_pdf_chatbot(messages):
+    query = messages[-1]['content'].strip()
+    qa_chain = ConversationalRetrievalChain().create_chain()
+    result = qa_chain({"query": query})
+    if not is_response_complete(result['result']):
+        result['result'] += " This is the end of the response. Let me know if you need further clarification."
+    return result['result']

helpers.py

@@ -0,0 +1,42 @@
+import base64
+import streamlit as st
+import os
+import openai
+from dotenv import load_dotenv
+from gtts import gTTS
+
+# Function to accept OpenAI API Key as input from the user
+def get_api_key():
+    api_key = st.text_input("Enter your OpenAI API Key", type="password")
+    if api_key:
+        openai.api_key = api_key
+        return api_key
+    else:
+        return None
+
+def speech_to_text(audio_data):
+    """Transcribes audio data to text using OpenAI's API."""
+    with open(audio_data, "rb") as audio_file:
+        transcript = openai.Audio.transcribe(
+            model="whisper-1",
+            file=audio_file
+        )
+    return transcript["text"]
+
+def text_to_speech(input_text):
+    """Generates a TTS audio file from the input text."""
+    tts = gTTS(text=input_text, lang="en")
+    audio_file_path = "temp_audio_play.mp3"
+    tts.save(audio_file_path)
+    return audio_file_path
+
+def autoplay_audio(file_path: str):
+    with open(file_path, "rb") as f:
+        data = f.read()
+    b64 = base64.b64encode(data).decode("utf-8")
+    md = f"""
+    <audio autoplay>
+    <source src="data:audio/mp3;base64,{b64}" type="audio/mp3">
+    </audio>
+    """
+    st.markdown(md, unsafe_allow_html=True)

requirements.txt

@@ -0,0 +1,3 @@
+openai==0.28
+pillow
+PyMuPDF