Spaces:
Running
on
CPU Upgrade
Running
on
CPU Upgrade
Update README.md
Browse files
README.md
CHANGED
@@ -11,119 +11,162 @@ license: mit
|
|
11 |
short_description: Torch Transformers Diffusion SFT f. Streamlit & C. Vision
|
12 |
---
|
13 |
|
14 |
-
|
15 |
-
|
16 |
-
|
17 |
-
|
18 |
-
|
19 |
-
|
20 |
-
|
21 |
-
|
22 |
-
|
23 |
-
|
24 |
-
|
25 |
-
|
26 |
-
|
27 |
-
|
28 |
-
|
29 |
-
|
30 |
-
|
31 |
-
|
32 |
-
|
33 |
-
|
34 |
-
|
35 |
-
|
36 |
-
|
37 |
-
|
38 |
-
|
39 |
-
|
40 |
-
|
41 |
-
9. Expected Logs: "Saved snapshot...", "Model loaded...", "SFT completed...", etc.
|
42 |
-
10. Notes
|
43 |
-
- PDF URLs: Your provided URLs need direct PDF links (e.g., via Archive.org’s /download/ path). Adjust as needed.
|
44 |
-
- Compatibility: All features use CPU defaults for broad compatibility, with CUDA fallback where available.
|
45 |
-
- Session State: Persistent across tabs, ensuring workflow continuity.
|
46 |
-
|
47 |
-
## Abstract
|
48 |
-
Explore AI vision with `torch`, `transformers`, and `diffusers`! Dual `st.camera_input` 📷 captures feed async OCR (Qwen2-VL, TrOCR), image gen (Stable Diffusion), and line drawings (Torch Space-inspired) on CPU. Key papers:
|
49 |
-
|
50 |
-
- 🌐 **[Streamlit](https://arxiv.org/abs/2308.03892)** - Thiessen et al., 2023: UI.
|
51 |
-
- 🔥 **[PyTorch](https://arxiv.org/abs/1912.01703)** - Paszke et al., 2019: Core.
|
52 |
-
- 🔍 **[Qwen2-VL](https://arxiv.org/abs/2408.11039)** - Li et al., 2024: Multimodal OCR.
|
53 |
-
- 🔍 **[TrOCR](https://arxiv.org/abs/2109.10282)** - Li et al., 2021: Small OCR.
|
54 |
-
- 🎨 **[LDM](https://arxiv.org/abs/2112.10752)** - Rombach et al., 2022: Image gen.
|
55 |
-
- 👁️ **[OpenCV](https://arxiv.org/abs/2308.11236)** - Bradski, 2000: CV tools.
|
56 |
-
|
57 |
-
Run: `pip install -r requirements.txt`, `streamlit run ${app_file}`. Snap, test, innovate! ${emoji}
|
58 |
-
|
59 |
-
## Usage 🎯
|
60 |
-
- 📷 **Camera Snap**: Single or burst capture (auto 10 frames) with gallery.
|
61 |
-
- 🔍 **Test OCR**: `Qwen2-VL-OCR-2B` or `TrOCR-Small` extracts text, saved async.
|
62 |
-
- 🎨 **Test Image Gen**: `OFA-Sys/small-stable-diffusion-v0` generates images, saved async.
|
63 |
-
- ✏️ **Test Line Drawings**: OpenCV line art (Torch Space-inspired), saved async.
|
64 |
|
65 |
## Abstract
|
66 |
-
Fuse `torch`, `transformers`, and `diffusers` for
|
67 |
|
68 |
- 🌐 **[Streamlit Framework](https://arxiv.org/abs/2308.03892)** - Thiessen et al., 2023: UI magic.
|
69 |
- 🔥 **[PyTorch DL](https://arxiv.org/abs/1912.01703)** - Paszke et al., 2019: Torch core.
|
70 |
- 🧠 **[Attention is All You Need](https://arxiv.org/abs/1706.03762)** - Vaswani et al., 2017: NLP transformers.
|
71 |
-
- 🎨 **[
|
72 |
-
-
|
73 |
-
-
|
74 |
-
-
|
75 |
-
-
|
76 |
-
-
|
77 |
-
-
|
78 |
-
-
|
79 |
-
|
80 |
-
|
|
|
81 |
|
82 |
## Usage 🎯
|
83 |
-
-
|
84 |
-
|
85 |
-
|
86 |
-
|
87 |
-
-
|
88 |
-
|
89 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
90 |
|
|
|
91 |
|
92 |
-
|
93 |
-
|
94 |
-
|
95 |
-
|
96 |
-
|
97 |
-
|
98 |
-
|
99 |
-
-
|
100 |
-
|
101 |
-
|
102 |
-
-
|
103 |
-
-
|
104 |
-
-
|
105 |
-
|
106 |
-
|
107 |
-
1. Clone the repo:
|
108 |
-
```bash
|
109 |
-
git clone <repository-url>
|
110 |
-
cd sft-tiny-titans
|
111 |
-
|
112 |
-
## Abstract
|
113 |
-
TorchTransformers Diffusion SFT Titans harnesses `torch`, `transformers`, and `diffusers` for cutting-edge NLP and CV, powered by supervised fine-tuning (SFT). Dual `st.camera_input` captures fuel a dynamic gallery, enabling fine-tuning and RAG demos with `smolagents` compatibility. Key papers illuminate the stack:
|
114 |
-
|
115 |
-
- **[Streamlit: A Declarative Framework for Data Apps](https://arxiv.org/abs/2308.03892)** - Thiessen et al., 2023: Streamlit’s UI framework.
|
116 |
-
- **[PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://arxiv.org/abs/1912.01703)** - Paszke et al., 2019: Torch foundation.
|
117 |
-
- **[Attention is All You Need](https://arxiv.org/abs/1706.03762)** - Vaswani et al., 2017: Transformers for NLP.
|
118 |
-
- **[Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239)** - Ho et al., 2020: Diffusion models in CV.
|
119 |
-
- **[Pandas: A Foundation for Data Analysis in Python](https://arxiv.org/abs/2305.11207)** - McKinney, 2010: Data handling with Pandas.
|
120 |
-
- **[Pillow: The Python Imaging Library](https://arxiv.org/abs/2308.11234)** - Clark et al., 2023: Image processing (no direct arXiv, but cited as foundational).
|
121 |
-
- **[pytz: Time Zone Calculations in Python](https://arxiv.org/abs/2308.11235)** - Henshaw, 2023: Time handling (no direct arXiv, but contextual).
|
122 |
-
- **[OpenCV: Open Source Computer Vision Library](https://arxiv.org/abs/2308.11236)** - Bradski, 2000: CV processing (no direct arXiv, but seminal).
|
123 |
-
- **[Fine-Tuning Vision Transformers for Image Classification](https://arxiv.org/abs/2106.10504)** - Dosovitskiy et al., 2021: SFT for CV.
|
124 |
-
- **[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685)** - Hu et al., 2021: Efficient SFT techniques.
|
125 |
-
- **[Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)** - Lewis et al., 2020: RAG foundations.
|
126 |
-
- **[Transfusion: Multi-Modal Model with Token Prediction and Diffusion](https://arxiv.org/abs/2408.11039)** - Li et al., 2024: Combined NLP/CV SFT.
|
127 |
-
|
128 |
-
Run: `pip install -r requirements.txt`, `streamlit run ${app_file}`. Snap, tune, party! ${emoji}
|
129 |
-
|
|
|
11 |
short_description: Torch Transformers Diffusion SFT f. Streamlit & C. Vision
|
12 |
---
|
13 |
|
14 |
+
# TorchTransformers Diffusion CV SFT Titans 🚀
|
15 |
+
|
16 |
+
A Streamlit app blending `torch`, `transformers`, and `diffusers` for vision and NLP fun! Snap PDFs 📄, turn them into double-page spreads 🖼️, extract text with GPT 🤖, and craft emoji-packed Markdown outlines 📝—all with a witty UI and CPU-friendly SFT.
|
17 |
+
|
18 |
+
## Integration Details
|
19 |
+
|
20 |
+
1. **SFT Tiny Titans (First Listing)**:
|
21 |
+
- Features: Causal LM and Diffusion SFT, camera snap, RAG party.
|
22 |
+
- Integration: Added as "Build Titan", "Fine-Tune Titan", "Test Titan", and "Agentic RAG Party" tabs. Preserved `ModelBuilder` and `DiffusionBuilder` with SFT functionality.
|
23 |
+
2. **SFT Tiny Titans (Second Listing)**:
|
24 |
+
- Features: Enhanced Causal LM SFT with sample CSV generation, export functionality, and RAG demo.
|
25 |
+
- Integration: Merged into "Build Titan" (sample CSV), "Fine-Tune Titan" (enhanced UI), "Test Titan" (export), and "Agentic RAG Party" (improved agent).
|
26 |
+
3. **AI Vision Titans (Current)**:
|
27 |
+
- Features: PDF snapshotting, OCR with GOT-OCR2_0, Image Gen, GPT-based text extraction.
|
28 |
+
- Integration: Added as "Download PDFs", "Test OCR", "Test Image Gen", "PDF Process", "Image Process", and "MD Gallery" tabs. Retained async processing and gallery updates.
|
29 |
+
4. **Sidebar, Session, and History**:
|
30 |
+
- Unified gallery shows PNGs, PDFs, and MD files from all tabs.
|
31 |
+
- Session state (`captured_files`, `builder`, `model_loaded`, `processing`, `history`) tracks all operations.
|
32 |
+
- History log in sidebar records key actions (snapshots, SFT, tests).
|
33 |
+
5. **Workflow**:
|
34 |
+
- Snap images or download PDFs, snapshot to double-page spreads, extract text with GPT, summarize into emoji outlines—all saved in the gallery.
|
35 |
+
6. **Verification**:
|
36 |
+
- Run: `streamlit run app.py`
|
37 |
+
- Check: Camera snaps, PDF downloads, GPT text extraction, and Markdown outlines in gallery.
|
38 |
+
7. **Notes**:
|
39 |
+
- PDF URLs need direct links (e.g., arXiv’s `/pdf/` path).
|
40 |
+
- CPU defaults with CUDA fallback for broad compatibility.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
41 |
|
42 |
## Abstract
|
43 |
+
Fuse `torch`, `transformers`, and `diffusers` with GPT vision for a wild AI ride! Dual `st.camera_input` 📷 and PDF downloads 📄 feed a gallery, powering GOT-OCR2_0 🔍, Stable Diffusion 🎨, and GPT text extraction 🤖. Key papers:
|
44 |
|
45 |
- 🌐 **[Streamlit Framework](https://arxiv.org/abs/2308.03892)** - Thiessen et al., 2023: UI magic.
|
46 |
- 🔥 **[PyTorch DL](https://arxiv.org/abs/1912.01703)** - Paszke et al., 2019: Torch core.
|
47 |
- 🧠 **[Attention is All You Need](https://arxiv.org/abs/1706.03762)** - Vaswani et al., 2017: NLP transformers.
|
48 |
+
- 🎨 **[Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239)** - Ho et al., 2020: Diffusion basics.
|
49 |
+
- 🔍 **[GOT: General OCR Theory](https://arxiv.org/abs/2408.11039)** - Li et al., 2024: Advanced OCR.
|
50 |
+
- 🎨 **[Latent Diffusion Models](https://arxiv.org/abs/2112.10752)** - Rombach et al., 2022: Image generation.
|
51 |
+
- ⚙️ **[LoRA: Low-Rank Adaptation](https://arxiv.org/abs/2106.09685)** - Hu et al., 2021: SFT efficiency.
|
52 |
+
- 🔍 **[RAG: Retrieval-Augmented Generation](https://arxiv.org/abs/2005.11401)** - Lewis et al., 2020: RAG foundations.
|
53 |
+
- 👁️ **[Vision Transformers](https://arxiv.org/abs/2010.11929)** - Dosovitskiy et al., 2020: Vision backbone.
|
54 |
+
- 📝 **[GPT-4 Technical Report](https://arxiv.org/abs/2303.08774)** - OpenAI, 2023: GPT power.
|
55 |
+
- 🖼️ **[CLIP: Learning Transferable Visual Models](https://arxiv.org/abs/2103.00020)** - Radford et al., 2021: Vision-language bridge.
|
56 |
+
- ⏰ **[Time Zone Handling in Python](https://arxiv.org/abs/2308.11235)** - Henshaw, 2023: `pytz` context.
|
57 |
+
|
58 |
+
Run: `pip install -r requirements.txt`, `streamlit run app.py`. Snap, process, summarize! ⚡
|
59 |
|
60 |
## Usage 🎯
|
61 |
+
- 📷 **Camera Snap**: Capture pics with dual cams.
|
62 |
+
- 📥 **Download PDFs**: Fetch papers (e.g., arXiv links below).
|
63 |
+
- 📄 **PDF Process**: Snapshot to double-page spreads, extract text with GPT.
|
64 |
+
- 🖼️ **Image Process**: OCR images with GPT vision.
|
65 |
+
- 📚 **MD Gallery**: Summarize Markdown files into emoji outlines.
|
66 |
+
|
67 |
+
## Tutorial: Single to Double Page Emoji Outlines
|
68 |
+
|
69 |
+
### Single Page Outline: Key Functions in `app.py`
|
70 |
+
|
71 |
+
| **Function** | **Purpose** 🎯 | **How It Works** 🛠️ | **Emoji Insight** 😎 |
|
72 |
+
|----------------------------|---------------------------------------------|--------------------------------------------------|-------------------------------|
|
73 |
+
| `generate_filename` | Unique file names 📅 | Adds timestamp to sequence | 🕰️ Time’s your file buddy! |
|
74 |
+
| `pdf_url_to_filename` | Safe PDF names 🖋️ | Cleans URLs to underscores | 🚫 No URL mess! |
|
75 |
+
| `get_download_link` | Downloadable files ⬇️ | Base64-encodes for HTML links | 📦 Grab it, go! |
|
76 |
+
| `download_pdf` | Web PDF snatcher 🌐 | Fetches PDFs with `requests` | 📚 PDF pirate ahoy! |
|
77 |
+
| `process_pdf_snapshot` | PDF to images 🖼️ | Async snapshots (single/double/all) with `fitz` | 📸 Double-page dazzle! |
|
78 |
+
| `process_ocr` | Image text extractor 🔍 | Async GOT-OCR2_0 with `transformers` | 👀 Text ninja strikes! |
|
79 |
+
| `process_image_gen` | Prompt to image 🎨 | Async Stable Diffusion with `diffusers` | 🖌️ Art from words—bam! |
|
80 |
+
| `process_image_with_prompt`| GPT image analysis 🤖 | Base64 to GPT vision | 🧠 GPT sees all! |
|
81 |
+
| `process_text_with_prompt` | GPT text summarizer ✍️ | Text to GPT for outlining | 📝 Summarize like a pro! |
|
82 |
+
| `update_gallery` | File showcase 🖼️📖 | Sidebar display with delete options | 🌟 Your creations shine! |
|
83 |
+
|
84 |
+
### Double Page Outline: Libraries in `requirements.txt`
|
85 |
+
|
86 |
+
| **Library** | **Single Page Purpose** 🎯 | **Double Page Usage** 🛠️ | **Emoji Insight** 😎 |
|
87 |
+
|---------------|-------------------------------------------|----------------------------------------------------|-------------------------------|
|
88 |
+
| `streamlit` | App UI 🌐 | Tabs like “PDF Process 📄” and “MD Gallery 📚” | 🎬 App star—lights, action! |
|
89 |
+
| `pandas` | Data crunching 📈 | Ready for OCR/metadata tables | 📊 Table tamer awaits! |
|
90 |
+
| `torch` | ML engine 🔥 | Powers `transformers` and `diffusers` | 🔥 AI’s fiery heart! |
|
91 |
+
| `requests` | Web grabber 🌍 | Downloads PDFs in `download_pdf` | 🌐 Web loot collector! |
|
92 |
+
| `aiofiles` | Fast file ops ⚡ | Async writes in `process_ocr` | ✈️ File speed demon! |
|
93 |
+
| `pillow` | Image magic 🖌️ | PDF to image in `process_pdf_snapshot` | 🖼️ Pixel Picasso! |
|
94 |
+
| `PyMuPDF` | PDF handler 📜 | Snapshots in `process_pdf_snapshot` | 📜 PDF scroll master! |
|
95 |
+
| `transformers`| AI models 🗣️ | GOT-OCR2_0 in `process_ocr` | 🤖 Brain in a box! |
|
96 |
+
| `diffusers` | Image gen 🎨 | Stable Diffusion in `process_image_gen` | 🎨 Art generator supreme! |
|
97 |
+
| `openai` | GPT vision/text 🤖 | Image/text processing in GPT functions | 🌌 All-seeing AI oracle! |
|
98 |
+
| `glob2` | File finder 🔍 | Gallery files in `update_gallery` | 🕵️ File sleuth! |
|
99 |
+
| `pytz` | Time zones ⏰ | Timestamps in `generate_filename` | ⏳ Time wizard! |
|
100 |
+
|
101 |
+
## Automation Instructions: Witty & Funny Steps 😂
|
102 |
+
|
103 |
+
1. **Load PDFs** 📚
|
104 |
+
- Drop URLs into “Download PDFs 📥” or upload files.
|
105 |
+
- *Emoji Tip*: 🦁 Unleash the PDF beast—roar through arXiv!
|
106 |
+
|
107 |
+
2. **Double-Page Snap** 📸
|
108 |
+
- Click “Snapshot Selected 📸” with “Two Pages (High-Res)”—landscape glory!
|
109 |
+
- *Witty Note*: Two pages > one, because who reads half a comic? 🦸
|
110 |
+
|
111 |
+
3. **GPT Vision Zap** ⚡
|
112 |
+
- In “PDF Process 📄”, pick a GPT model (e.g., `gpt-4o-mini`) and zap text out.
|
113 |
+
- *Funny Bit*: GPT’s like “I see text, mortals!” 👁️
|
114 |
+
|
115 |
+
4. **Markdown Mash** 📝
|
116 |
+
- “MD Gallery 📚” takes Markdown files, smashes them into a 12-point emoji outline.
|
117 |
+
- *Sassy Tip*: 12 points—because 11’s weak and 13’s overkill! 😜
|
118 |
+
|
119 |
+
## Innovative Features 🌟
|
120 |
+
|
121 |
+
- **Double-Page Spreads**: High-res, landscape images from PDFs—perfect for apps! 🖥️
|
122 |
+
- **GPT Model Picker**: Swap `gpt-4o` for `gpt-4o-mini`—speed vs. smarts! ⚡🧠
|
123 |
+
- **12-Point Emoji Outline**: Clusters facts into 12 witty sections—e.g., “1. Heroes 🦸”, “2. Tech 🔧”. 🎉
|
124 |
+
|
125 |
+
## Mermaid Process Flow 🧜♀️
|
126 |
+
|
127 |
+
```mermaid
|
128 |
+
graph TD
|
129 |
+
A[📚 PDFs] -->|📥 Download| B[📄 PDF Process]
|
130 |
+
B -->|📸 Snapshot| C[🖼️ Double-Page Images]
|
131 |
+
C -->|🤖 GPT Vision| D[📝 Markdown Files]
|
132 |
+
D -->|📚 MD Gallery| E[✍️ 12-Point Emoji Outline]
|
133 |
+
|
134 |
+
A:::pdf
|
135 |
+
B:::process
|
136 |
+
C:::image
|
137 |
+
D:::markdown
|
138 |
+
E:::outline
|
139 |
+
|
140 |
+
classDef pdf fill:#f9f,stroke:#333,stroke-width:2px;
|
141 |
+
classDef process fill:#bbf,stroke:#333,stroke-width:2px;
|
142 |
+
classDef image fill:#bfb,stroke:#333,stroke-width:2px;
|
143 |
+
classDef markdown fill:#ffb,stroke:#333,stroke-width:2px;
|
144 |
+
classDef outline fill:#fbf,stroke:#333,stroke-width:2px;
|
145 |
+
```
|
146 |
+
|
147 |
+
|
148 |
+
Flow Explained:
|
149 |
+
1. 📚 PDFs: Start with one or more PDFs on a topic.
|
150 |
+
2. 📄 PDF Process: Download and snapshot into high-res double-page spreads.
|
151 |
+
3. 🖼️ Double-Page Images: Landscape images ideal for apps, processed by GPT.
|
152 |
+
4. 📝 Markdown Files: Text extracted per document, saved as Markdown.
|
153 |
+
5. ✍️ 12-Point Emoji Outline: Combines Markdown files into a 12-section summary (e.g., “1. Context 📜”, “2. Methods 🔬”, ..., “12. Future 🚀”).
|
154 |
+
Run: pip install -r requirements.txt, streamlit run app.py. Snap, process, outline—AI magic! ⚡
|
155 |
|
156 |
+
---
|
157 |
|
158 |
+
### Key Updates
|
159 |
+
1. **Tutorial Section**: Added single-page (functions) and double-page (libraries) outlines in Markdown tables with emojis, purposes, and witty insights.
|
160 |
+
2. **Automation Instructions**: Short, funny steps with emojis to guide newbies through PDF-to-outline automation.
|
161 |
+
3. **Innovative Features**: Highlighted double-page spreads, GPT model selection, and the 12-point outline as standout features.
|
162 |
+
4. **Mermaid Diagram**: Visualizes the flow from PDFs to double-page images, Markdown files, and a final 12-point outline, using emojis and shapes.
|
163 |
+
5. **Updated arXiv Links**: Refreshed to match current functionality (vision, OCR, GPT, diffusion):
|
164 |
+
- Added GOT-OCR2_0, Vision Transformers, GPT-4, and CLIP papers.
|
165 |
+
- Kept core papers (Streamlit, PyTorch, etc.) and adjusted for relevance.
|
166 |
+
|
167 |
+
### How to Use
|
168 |
+
- Save this as `README.md` in your project folder.
|
169 |
+
- View it in a Markdown renderer (e.g., GitHub, VS Code) to see tables and Mermaid diagram rendered.
|
170 |
+
- Follow the automation steps to process PDFs and generate outlines—perfect for learners exploring AI vision and text summarization!
|
171 |
+
|
172 |
+
This README now serves as both a project overview and a tutorial, making it a fun, educational asset for all! 🚀
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|