Add initial database setup and comprehensive documentation for SmartQuery application; include Dockerfile for containerized deployment

Dockerfile

@@ -0,0 +1,32 @@
+# Use the official Python image from the Docker Hub
+FROM python:3.11
+
+# Create a new user with a specific UID and set it as the default user
+RUN useradd -m -u 1000 user
+
+# Switch to the new user
+USER user
+
+# Set environment variables
+ENV HOME=/home/user \
+  PATH=/home/user/.local/bin:$PATH
+
+# Set the working directory
+WORKDIR $HOME/app
+
+# Copy the requirements file and install the dependencies
+COPY --chown=user ./requirements.txt $HOME/app/requirements.txt
+RUN pip install --upgrade pip && \
+  pip install -r requirements.txt
+
+# Copy project files to the working directory
+COPY --chown=user . $HOME/app
+
+# Set PYTHONPATH to include the project directory
+ENV PYTHONPATH=$HOME/app
+
+# Initialize the database
+RUN python smartquery/init_db.py
+
+# Set the command to run the application
+CMD ["chainlit", "run", "smartquery/app.py", "--port", "7860"]

README.md

@@ -4,3 +4,83 @@ colorFrom: blue
 colorTo: yellow
 sdk: docker
 pinned: false
+
+# SmartQuery
+
+SmartQuery is an intelligent assistant designed to provide seamless interaction with your database. Built on top of LangChain and Chainlit, and using the OpenAI API, SmartQuery allows users to query their database using natural language, either through text or voice commands.
+
+## Features
+
+- **Natural Language Querying:** Interact with your database using plain English, no SQL required.
+- **Voice Commands:** Ask questions out loud and get verbal responses.
+- **Rich Insights:** Get detailed answers and insights from your data.
+- **User-Friendly Interface:** Simple chat-based interaction for ease of use.
+
+## Setup
+
+Clone the repository:
+
+```bash
+git clone https://github.com/your-repo/SmartQuery.git
+cd SmartQuery
+```
+
+Install the required dependencies using pip:
+
+```bash
+pip install -r requirements.txt
+```
+
+Create and populate the database:
+
+```bash
+sqlite3 database/Chinook.db
+.read database/Chinook_Sqlite.sql
+.exit
+```
+
+Create a .env file and add your environment variables:
+
+```bash
+OPENAI_API_KEY=your-openai-key-here
+ELEVENLABS_API_KEY=your-elevenlabs-key-here
+ELEVENLABS_VOICE_ID=your-elevenlabs-voice-id-here
+```
+
+Run the application using the following command:
+
+    ```bash
+    chainlit run app.py
+    ```
+
+## Usage
+
+Start a chat session and ask questions related to the content of the Chinook database. The application supports both text and voice input. For voice input, press the microphone button, speak your question, and let SmartQuery process your query. It might take some time to answer complex questions (usually less than 1 minute), so please be patient.
+
+### Sample Questions to Try:
+
+- "What is the most expensive track?"
+- "List the total sales per country. Which country's customers spent the most?"
+- "Who are the 3 most listened artists and what is their average revenue?"
+
+## About the Database
+
+The database powering SmartQuery is the Chinook Database, a sample database representing a digital media store. It contains tables for:
+
+- **Artists**: Information about music artists.
+- **Albums**: Details of albums released by artists.
+- **Tracks**: Data on individual tracks, including their length and price.
+- **Genres**: Different genres of music.
+- **Customers**: Information about the customers of the store.
+- **Invoices**: Purchase records containing information on sales transactions.
+- **InvoiceLines**: Details about each item in an invoice.
+- **Employees**: Data on employees managing the store.
+- **Playlists**: User-generated playlists.
+- **PlaylistTracks**: Mapping of tracks to playlists.
+- **MediaTypes**: Types of media the tracks are available in.
+
+This structure allows you to ask a wide range of questions about sales, customer preferences, artist performance, and more. Feel free to explore the richness of the data and uncover valuable insights.
+
+## Acknowledgements
+
+This project uses technologies including LangChain, OpenAI's GPT models, FAISS for vector storage, Eleven Labs for speech-to-text and text-to-speech, and ChainLit for building interactive AI applications. Thanks to all open-source contributors and organizations that make these tools available.

chainlit.md

@@ -4,12 +4,44 @@ Hello and welcome to SmartQuery, your intelligent assistant designed to help you
 
 ## Getting Started
 
-## Sample Questions to Try:
+### Sample Questions to Try:
+
+- "What is the most expensive track?"
+- "List the total sales per country. Which country's customers spent the most?"
+- "Who are the 3 most listened artists and what is their average revenue?"
+
+## About the Database 🎶💽
+
+The database powering SmartQuery is the Chinook Database, a sample database representing a digital media store. It contains tables for:
+
+- **Artists**: Information about music artists.
+- **Albums**: Details of albums released by artists.
+- **Tracks**: Data on individual tracks, including their length and price.
+- **Genres**: Different genres of music.
+- **Customers**: Information about the customers of the store.
+- **Invoices**: Purchase records containing information on sales transactions.
+- **InvoiceLines**: Details about each item in an invoice.
+- **Employees**: Data on employees managing the store.
+- **Playlists**: User-generated playlists.
+- **PlaylistTracks**: Mapping of tracks to playlists.
+- **MediaTypes**: Types of media the tracks are available in.
+
+This structure allows you to ask a wide range of questions about sales, customer preferences, artist performance, and more. Feel free to explore the richness of the data and uncover valuable insights.
 
 ## How It Works 🧠
 
+SmartQuery is an agentic application that leverages advanced natural language processing (NLP) techniques to understand your queries in plain English and convert them into SQL queries to retrieve the desired information from the database. You can interact with SmartQuery via text or voice, making it incredibly versatile and user-friendly.
+
 ## The Tech Behind It 💡🤖
 
+SmartQuery is built using state-of-the-art technologies, including:
+
+- **LangChain**: For chaining together multiple language model prompts and responses.
+- **OpenAI GPT-4o**: For understanding and processing natural language queries.
+- **FAISS**: For efficient similarity search and retrieval.
+- **Chainlit**: For building interactive AI applications front-end.
+- **Eleven Labs**: For speech-to-text and text-to-speech functionalities, enabling voice interactions.
+
 ## Ready to Query?
 
 With SmartQuery, transform your database interactions into a seamless experience. Dive into your data, uncover insights, and make data-driven decisions more effectively than ever before. Happy querying!

old_app.py

@@ -1,24 +0,0 @@
-import chainlit as cl
-from langchain.schema.runnable.config import RunnableConfig
-from sql_agent import SQLAgent
-
-# ChainLit Integration
-@cl.on_chat_start
-async def on_chat_start():
-    cl.user_session.set("agent", SQLAgent)
-
-@cl.on_message
-async def on_message(message: cl.Message):
-    agent = cl.user_session.get("agent")  # Get the agent from the session
-    cb = cl.AsyncLangchainCallbackHandler(stream_final_answer=True)
-    config = RunnableConfig(callbacks=[cb])
-
-    async with cl.Step(name="SmartQuery Agent", root=True) as step:
-        step.input = message.content
-        result = await agent.ainvoke(message.content, config=config)
-
-        # Assuming the result is a dictionary with a key 'output' containing the final answer
-        final_answer = result.get('output', 'No answer returned')
-
-        # Stream the final answer as a token to the step
-        await step.stream_token(final_answer)

app.py → smartquery/app.py

@@ -4,7 +4,7 @@ import chainlit as cl
 import httpx
 from dotenv import load_dotenv
 from langchain.schema.runnable.config import RunnableConfig
-from sql_agent import SQLAgent
+from smartquery.sql_agent import SQLAgent
 from openai import AsyncOpenAI
 from chainlit.element import Audio
 
@@ -29,47 +29,13 @@ async def speech_to_text(audio_file):
 
 @cl.step(type="tool")
 async def generate_text_answer(transcription, images):
-    model = "gpt-4-turbo"
+    model = "gpt-4o"
     messages = [{"role": "user", "content": transcription}]
     response = await client.chat.completions.create(
         messages=messages, model=model, temperature=0.3
     )
     return response.choices[0].message.content
 
-@cl.step(type="tool")
-async def text_to_speech(text: str, mime_type: str):
-    CHUNK_SIZE = 1024
-    url = f"https://api.elevenlabs.io/v1/text-to-speech/{ELEVENLABS_VOICE_ID}"
-
-    headers = {
-        "Accept": mime_type,
-        "Content-Type": "application/json",
-        "xi-api-key": ELEVENLABS_API_KEY
-    }
-
-    data = {
-        "text": text,
-        "model_id": "eleven_monolingual_v1",
-        "voice_settings": {
-            "stability": 0.5,
-            "similarity_boost": 0.5
-        }
-    }
-
-    async with httpx.AsyncClient(timeout=25.0) as client:
-        response = await client.post(url, json=data, headers=headers)
-        response.raise_for_status()  # Ensure we notice bad responses
-
-        buffer = BytesIO()
-        buffer.name = f"output_audio.{mime_type.split('/')[1]}"
-
-        async for chunk in response.aiter_bytes(chunk_size=CHUNK_SIZE):
-            if chunk:
-                buffer.write(chunk)
-
-        buffer.seek(0)
-        return buffer.name, buffer.read()
-
 @cl.on_chat_start
 async def on_chat_start():
     cl.user_session.set("agent", SQLAgent)
@@ -114,6 +80,11 @@ async def on_audio_end(elements: list[Audio]):
 
     await process_message(transcription, answer_message, audio_mime_type)
 
+    # Reset audio buffer and mime type
+    cl.user_session.set("audio_buffer", None)
+    cl.user_session.set("audio_mime_type", None)
+    print("Audio buffer reset")
+
 async def process_message(content: str, answer_message=None, mime_type=None):
     agent = cl.user_session.get("agent")
     cb = cl.AsyncLangchainCallbackHandler(stream_final_answer=True)
@@ -127,15 +98,6 @@ async def process_message(content: str, answer_message=None, mime_type=None):
 
         await step.stream_token(final_answer)
 
-        if mime_type:
-            output_name, output_audio = await text_to_speech(final_answer, mime_type)
-            output_audio_el = Audio(
-                name=output_name,
-                auto_play=True,
-                mime=mime_type,
-                content=output_audio,
-            )
-            answer_message.elements = [output_audio_el]
+        if answer_message:
+            answer_message.content = final_answer
             await answer_message.update()
-        else:
-            await cl.Message(content=final_answer).send()

smartquery/init_db.py

@@ -0,0 +1,13 @@
+import sqlite3
+
+def initialize_database():
+    conn = sqlite3.connect('database/Chinook.db')
+    cursor = conn.cursor()
+    with open('database/Chinook_Sqlite.sql', 'r') as f:
+        sql_script = f.read()
+    cursor.executescript(sql_script)
+    conn.commit()
+    conn.close()
+
+if __name__ == "__main__":
+    initialize_database()

sql_agent.py → smartquery/sql_agent.py

@@ -9,14 +9,14 @@ from langchain_core.prompts import ChatPromptTemplate, FewShotPromptTemplate, Me
 from langchain_openai import OpenAIEmbeddings
 from langchain_openai import ChatOpenAI
 from langchain_community.utilities import SQLDatabase
-from prompt_templates import few_shot_examples, system_prefix
+from smartquery.prompt_templates import few_shot_examples, system_prefix
 
 
 # Load the .env file
 load_dotenv()
 
 # Initialize the SQL database
-db = SQLDatabase.from_uri("sqlite:///Chinook.db")
+db = SQLDatabase.from_uri("sqlite:///database/Chinook.db")
 
 # Check the database connection
 print(db.dialect)