DL4DS Tutor 🏃

Check out the configuration reference at Hugging Face Spaces Config Reference.

You can find an implementation of the Tutor at DL4DS Tutor on Hugging Face, which is hosted on Hugging Face here.

Running Locally

Clone the Repository

git clone https://github.com/DL4DS/dl4ds_tutor

Put your data under the storage/data directory
- Add URLs in the urls.txt file.
- Add other PDF files in the storage/data directory.

To test Data Loading (Optional)

cd code
python -m modules.dataloader.data_loader

Create the Vector Database
```
cd code
python -m modules.vectorstore.store_manager
```
- Note: You need to run the above command when you add new data to the storage/data directory, or if the storage/data/urls.txt file is updated.
- Alternatively, you can set ["vectorstore"]["embedd_files"] to True in the code/modules/config/config.yaml file, which will embed files from the storage directory every time you run the below chainlit command.
Run the Chainlit App
```
chainlit run main.py
```

See the docs for more information.

File Structure

code/
 ├── modules
 │   ├── chat                # Contains the chatbot implementation
 │   ├── chat_processor      # Contains the implementation to process and log the conversations
 │   ├── config              # Contains the configuration files
 │   ├── dataloader          # Contains the implementation to load the data from the storage directory
 │   ├── retriever           # Contains the implementation to create the retriever
 │   └── vectorstore         # Contains the implementation to create the vector database
 ├── public
 │   ├── logo_dark.png       # Dark theme logo
 │   ├── logo_light.png      # Light theme logo
 │   └── test.css            # Custom CSS file
 └── main.py

 
docs/                        # Contains the documentation to the codebase and methods used

storage/
 ├── data                    # Store files and URLs here
 ├── logs                    # Logs directory, includes logs on vector DB creation, tutor logs, and chunks logged in JSON files
 └── models                  # Local LLMs are loaded from here

vectorstores/                # Stores the created vector databases

.env                         # This needs to be created, store the API keys here