|
--- |
|
license: mit |
|
language: |
|
- en |
|
--- |
|
## Objective |
|
|
|
Normalizer is a tool that processes various file types to extract prompt-response pairs for finetuning LLM models on unstructured data. |
|
|
|
The end-state will be a GUI where users of any technical level can drag and drop multiple files at once into the application, then they hit a 'create fine-tuning data' button and they'll receive a .csv with a system prompt and a series of matched pair prompt-completion responses. |
|
|
|
## Working Repository |
|
|
|
https://github.com/anarchy-ai/normalizer |
|
|
|
## Input Formats |
|
|
|
- **Text**: .txt, .md, .mdx |
|
- **Documents**: .pdf, .doc, .docx |
|
- **Spreadsheets**: .xlsx, .csv |
|
- **Code**: .py, .js, .html, .java |
|
- **Images**: .jpg, .png (will require OCR) |
|
|
|
## Output Format |
|
|
|
A .csv file with 3 columns: |
|
|
|
| System Prompt | User Prompt | Response | |
|
| -------------------------------------- | ------------------------------------- | ---------------------------------------------| |
|
| Hi, how can I help you today? | | |
|
| | What do these lab results suggest? | These lab results suggest that the patient is healthy, as no anomalous data has been detected. | |
|
| | What is the sentiment of the last 5 customers who came into support chat? | The last five customers have a neutral to positive sentiment. | |
|
| | ... | ... | |
|
|
|
## Required Libraries |
|
|
|
```pip install PyPDF2 python-docx pandas openpyxl pillow pytesseract beautifulsoup4 transformers datasets``` |
|
|
|
## Project Structure |
|
|
|
```CSS |
|
normalizer/ |
|
│ |
|
├── src/ |
|
│ ├── __init__.py |
|
│ ├── file_ingest.py |
|
│ ├── prompt_extractor.py |
|
│ ├── main.py |
|
│ |
|
├── app.py |
|
├── requirements.txt |
|
├── README.md |
|
└── .gitignore |
|
``` |
