Create DEPLOYMENT.md
Browse files- DEPLOYMENT.md +592 -0
DEPLOYMENT.md
ADDED
@@ -0,0 +1,592 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
# π AI Dataset Studio - Complete Deployment Guide
|
2 |
+
|
3 |
+
**Deploy your AI-powered dataset creation platform with Perplexity integration**
|
4 |
+
|
5 |
+
---
|
6 |
+
|
7 |
+
## π Pre-Deployment Checklist
|
8 |
+
|
9 |
+
### β
**Required Files**
|
10 |
+
Ensure you have all these files ready:
|
11 |
+
|
12 |
+
```
|
13 |
+
ai-dataset-studio/
|
14 |
+
βββ app.py # Main application with Perplexity integration
|
15 |
+
βββ perplexity_client.py # Perplexity AI client
|
16 |
+
βββ config.py # Configuration management
|
17 |
+
βββ requirements.txt # Dependencies
|
18 |
+
βββ README.md # Documentation
|
19 |
+
βββ DEPLOYMENT.md # This guide
|
20 |
+
βββ utils.py # Utility functions (optional)
|
21 |
+
```
|
22 |
+
|
23 |
+
### β
**API Keys & Environment**
|
24 |
+
- [ ] **Perplexity API Key** - Get from [Perplexity AI](https://www.perplexity.ai/)
|
25 |
+
- [ ] **HuggingFace Account** - For Space hosting
|
26 |
+
- [ ] **Optional**: HuggingFace Token for private datasets
|
27 |
+
|
28 |
+
---
|
29 |
+
|
30 |
+
## π― Deployment Options
|
31 |
+
|
32 |
+
### **Option 1: Full AI-Powered Deployment (Recommended)**
|
33 |
+
*Best for: Professional use, maximum features*
|
34 |
+
|
35 |
+
#### Hardware: **T4 Small** ($0.60/hour)
|
36 |
+
- β
GPU acceleration for AI models
|
37 |
+
- β
Fast processing (5-15s per article)
|
38 |
+
- β
All Perplexity features enabled
|
39 |
+
- β
Production-ready performance
|
40 |
+
|
41 |
+
#### **Step-by-Step:**
|
42 |
+
|
43 |
+
1. **Create HuggingFace Space**
|
44 |
+
```bash
|
45 |
+
# Go to: https://huggingface.co/new-space
|
46 |
+
Space Name: ai-dataset-studio
|
47 |
+
SDK: Gradio
|
48 |
+
Hardware: T4 Small
|
49 |
+
Visibility: Public (or Private)
|
50 |
+
```
|
51 |
+
|
52 |
+
2. **Upload Files**
|
53 |
+
- Copy all files from artifacts above
|
54 |
+
- Ensure `app.py` is the main file
|
55 |
+
- Keep file structure intact
|
56 |
+
|
57 |
+
3. **Set Environment Variables**
|
58 |
+
```bash
|
59 |
+
# In Space Settings β Repository secrets:
|
60 |
+
PERPLEXITY_API_KEY = your_perplexity_api_key_here
|
61 |
+
|
62 |
+
# Optional:
|
63 |
+
HF_TOKEN = your_huggingface_token
|
64 |
+
LOG_LEVEL = INFO
|
65 |
+
DEBUG = false
|
66 |
+
```
|
67 |
+
|
68 |
+
4. **Deploy & Test**
|
69 |
+
- Space will build automatically (2-3 minutes)
|
70 |
+
- Test Perplexity integration first
|
71 |
+
- Verify all templates work
|
72 |
+
|
73 |
+
---
|
74 |
+
|
75 |
+
### **Option 2: Budget-Friendly Deployment**
|
76 |
+
*Best for: Testing, learning, cost-conscious users*
|
77 |
+
|
78 |
+
#### Hardware: **CPU Basic** (Free)
|
79 |
+
- β‘ Basic functionality available
|
80 |
+
- β οΈ Slower AI processing (30-60s per article)
|
81 |
+
- β
Perplexity discovery still works
|
82 |
+
- β
Perfect for getting started
|
83 |
+
|
84 |
+
#### **Step-by-Step:**
|
85 |
+
|
86 |
+
1. **Create Space with CPU Basic**
|
87 |
+
```bash
|
88 |
+
Space Name: ai-dataset-studio
|
89 |
+
SDK: Gradio
|
90 |
+
Hardware: CPU Basic (Free)
|
91 |
+
```
|
92 |
+
|
93 |
+
2. **Upload Core Files**
|
94 |
+
```bash
|
95 |
+
# Essential files only:
|
96 |
+
app.py
|
97 |
+
perplexity_client.py
|
98 |
+
requirements.txt
|
99 |
+
README.md
|
100 |
+
config.py
|
101 |
+
```
|
102 |
+
|
103 |
+
3. **Set API Key**
|
104 |
+
```bash
|
105 |
+
PERPLEXITY_API_KEY = your_api_key
|
106 |
+
```
|
107 |
+
|
108 |
+
4. **Gradual Upgrade Path**
|
109 |
+
- Start with CPU Basic
|
110 |
+
- Test functionality
|
111 |
+
- Upgrade to T4 Small when ready
|
112 |
+
|
113 |
+
---
|
114 |
+
|
115 |
+
### **Option 3: Enterprise Deployment**
|
116 |
+
*Best for: High-volume usage, team collaboration*
|
117 |
+
|
118 |
+
#### Hardware: **A10G Small** ($1.05/hour)
|
119 |
+
- π Maximum performance (3-8s per article)
|
120 |
+
- πͺ Handle large batch processing
|
121 |
+
- π Support multiple concurrent users
|
122 |
+
- π Production-scale capabilities
|
123 |
+
|
124 |
+
#### **Additional Setup:**
|
125 |
+
|
126 |
+
1. **Persistent Storage**
|
127 |
+
```bash
|
128 |
+
# In Space settings:
|
129 |
+
Storage: Small Persistent ($5/month)
|
130 |
+
# Enables data persistence between restarts
|
131 |
+
```
|
132 |
+
|
133 |
+
2. **Advanced Configuration**
|
134 |
+
```bash
|
135 |
+
# Environment variables:
|
136 |
+
MAX_SOURCES_PER_SEARCH = 50
|
137 |
+
BATCH_SIZE = 16
|
138 |
+
ENABLE_CACHING = true
|
139 |
+
CONCURRENT_REQUESTS = 5
|
140 |
+
```
|
141 |
+
|
142 |
+
3. **Monitoring Setup**
|
143 |
+
```bash
|
144 |
+
# Enable detailed logging:
|
145 |
+
LOG_LEVEL = DEBUG
|
146 |
+
ENABLE_METRICS = true
|
147 |
+
```
|
148 |
+
|
149 |
+
---
|
150 |
+
|
151 |
+
## π§ Configuration Details
|
152 |
+
|
153 |
+
### **Perplexity API Setup**
|
154 |
+
|
155 |
+
1. **Get API Key**
|
156 |
+
```bash
|
157 |
+
# Visit: https://www.perplexity.ai/
|
158 |
+
# Sign up for account
|
159 |
+
# Navigate to API section
|
160 |
+
# Generate new API key
|
161 |
+
# Copy key for environment setup
|
162 |
+
```
|
163 |
+
|
164 |
+
2. **Test API Key**
|
165 |
+
```python
|
166 |
+
# Quick test script:
|
167 |
+
import requests
|
168 |
+
|
169 |
+
headers = {
|
170 |
+
'Authorization': 'Bearer YOUR_API_KEY',
|
171 |
+
'Content-Type': 'application/json'
|
172 |
+
}
|
173 |
+
|
174 |
+
response = requests.post(
|
175 |
+
'https://api.perplexity.ai/chat/completions',
|
176 |
+
headers=headers,
|
177 |
+
json={
|
178 |
+
"model": "llama-3.1-sonar-large-128k-online",
|
179 |
+
"messages": [{"role": "user", "content": "Test message"}]
|
180 |
+
}
|
181 |
+
)
|
182 |
+
|
183 |
+
print("API Status:", response.status_code)
|
184 |
+
```
|
185 |
+
|
186 |
+
### **Hardware Requirements by Use Case**
|
187 |
+
|
188 |
+
| Use Case | Hardware | Monthly Cost | Performance | Best For |
|
189 |
+
|----------|----------|--------------|-------------|----------|
|
190 |
+
| **Learning** | CPU Basic | Free | Basic | Students, hobbyists |
|
191 |
+
| **Development** | CPU Upgrade | $22 | Good | Developers, testing |
|
192 |
+
| **Production** | T4 Small | $432 | Excellent | Businesses, researchers |
|
193 |
+
| **Enterprise** | A10G Small | $756 | Maximum | High-volume, teams |
|
194 |
+
|
195 |
+
### **Memory & Storage Planning**
|
196 |
+
|
197 |
+
```bash
|
198 |
+
# Model Memory Usage:
|
199 |
+
BART Summarization: ~1.5GB
|
200 |
+
RoBERTa Sentiment: ~500MB
|
201 |
+
BERT NER: ~400MB
|
202 |
+
Base Application: ~200MB
|
203 |
+
Total GPU Memory: ~2.5GB (T4 Small = 16GB, plenty of headroom)
|
204 |
+
|
205 |
+
# Storage Usage:
|
206 |
+
Application Files: ~50MB
|
207 |
+
Model Cache: ~2GB
|
208 |
+
Temporary Data: ~100MB per project
|
209 |
+
Persistent Storage: Optional, recommended for large projects
|
210 |
+
```
|
211 |
+
|
212 |
+
---
|
213 |
+
|
214 |
+
## π§ͺ Testing Your Deployment
|
215 |
+
|
216 |
+
### **Basic Functionality Test**
|
217 |
+
|
218 |
+
1. **Launch Application**
|
219 |
+
```bash
|
220 |
+
# Your Space URL: https://huggingface.co/spaces/YOUR_USERNAME/ai-dataset-studio
|
221 |
+
# Wait for "Running" status
|
222 |
+
# Interface should load within 30-60 seconds
|
223 |
+
```
|
224 |
+
|
225 |
+
2. **Test Project Creation**
|
226 |
+
```bash
|
227 |
+
Project Name: "Test Sentiment Analysis"
|
228 |
+
Template: Sentiment Analysis
|
229 |
+
Description: "Testing the deployment"
|
230 |
+
Click: "Create Project"
|
231 |
+
Expected: "β
Project created successfully"
|
232 |
+
```
|
233 |
+
|
234 |
+
3. **Test Perplexity Integration**
|
235 |
+
```bash
|
236 |
+
AI Search Description: "Product reviews for sentiment analysis"
|
237 |
+
Search Type: General
|
238 |
+
Max Sources: 10
|
239 |
+
Click: "Discover Sources with AI"
|
240 |
+
Expected: List of relevant URLs with quality scores
|
241 |
+
```
|
242 |
+
|
243 |
+
### **Advanced Testing**
|
244 |
+
|
245 |
+
4. **Test Complete Workflow**
|
246 |
+
```bash
|
247 |
+
# Use discovered sources from step 3
|
248 |
+
Click: "Use These Sources"
|
249 |
+
Click: "Start Scraping"
|
250 |
+
Wait: Processing to complete
|
251 |
+
Click: "Process Data"
|
252 |
+
Select: Same template as project
|
253 |
+
Click: "Export Dataset"
|
254 |
+
Format: JSON
|
255 |
+
Expected: Downloadable dataset file
|
256 |
+
```
|
257 |
+
|
258 |
+
5. **Performance Benchmarks**
|
259 |
+
```bash
|
260 |
+
# Timing expectations:
|
261 |
+
AI Source Discovery: 5-15 seconds
|
262 |
+
Scraping 10 URLs: 30-120 seconds
|
263 |
+
Processing Data: 30-180 seconds (depends on hardware)
|
264 |
+
Export: 5-10 seconds
|
265 |
+
```
|
266 |
+
|
267 |
+
---
|
268 |
+
|
269 |
+
## π¨ Troubleshooting
|
270 |
+
|
271 |
+
### **Common Issues & Solutions**
|
272 |
+
|
273 |
+
#### β **"Perplexity API key not found"**
|
274 |
+
```bash
|
275 |
+
# Problem: Environment variable not set
|
276 |
+
# Solution:
|
277 |
+
1. Go to Space Settings β Repository secrets
|
278 |
+
2. Add: PERPLEXITY_API_KEY = your_key_here
|
279 |
+
3. Restart Space
|
280 |
+
4. Check logs for "β
Perplexity AI client initialized"
|
281 |
+
```
|
282 |
+
|
283 |
+
#### β **"No sources found" from AI discovery**
|
284 |
+
```bash
|
285 |
+
# Problem: Search query too specific or API limits
|
286 |
+
# Solutions:
|
287 |
+
1. Make description more general
|
288 |
+
2. Try different search types
|
289 |
+
3. Check API key has sufficient credits
|
290 |
+
4. Use manual URL entry as fallback
|
291 |
+
```
|
292 |
+
|
293 |
+
#### β **"Model loading failed"**
|
294 |
+
```bash
|
295 |
+
# Problem: Insufficient memory or network issues
|
296 |
+
# Solutions:
|
297 |
+
1. Upgrade to T4 Small for GPU memory
|
298 |
+
2. Wait 2-3 minutes for model downloads
|
299 |
+
3. Check Space logs for specific errors
|
300 |
+
4. Restart Space if persistent
|
301 |
+
```
|
302 |
+
|
303 |
+
#### β **"Scraping failed" for multiple URLs**
|
304 |
+
```bash
|
305 |
+
# Problem: Rate limiting or blocked access
|
306 |
+
# Solutions:
|
307 |
+
1. Reduce concurrent requests
|
308 |
+
2. Check robots.txt compliance
|
309 |
+
3. Use more diverse sources
|
310 |
+
4. Verify URLs are publicly accessible
|
311 |
+
```
|
312 |
+
|
313 |
+
### **Debug Mode**
|
314 |
+
|
315 |
+
Enable detailed logging for troubleshooting:
|
316 |
+
|
317 |
+
```bash
|
318 |
+
# Environment variables:
|
319 |
+
DEBUG = true
|
320 |
+
LOG_LEVEL = DEBUG
|
321 |
+
|
322 |
+
# Then check Space logs for detailed information
|
323 |
+
```
|
324 |
+
|
325 |
+
### **Health Check Script**
|
326 |
+
|
327 |
+
```python
|
328 |
+
# Add this to test basic functionality:
|
329 |
+
def health_check():
|
330 |
+
"""Test all components"""
|
331 |
+
|
332 |
+
# Test imports
|
333 |
+
try:
|
334 |
+
import gradio
|
335 |
+
print("β
Gradio imported")
|
336 |
+
except ImportError:
|
337 |
+
print("β Gradio import failed")
|
338 |
+
|
339 |
+
# Test Perplexity
|
340 |
+
try:
|
341 |
+
from perplexity_client import PerplexityClient
|
342 |
+
client = PerplexityClient()
|
343 |
+
if client._validate_api_key():
|
344 |
+
print("β
Perplexity API key valid")
|
345 |
+
else:
|
346 |
+
print("β Perplexity API key invalid")
|
347 |
+
except Exception as e:
|
348 |
+
print(f"β Perplexity error: {e}")
|
349 |
+
|
350 |
+
# Test models
|
351 |
+
try:
|
352 |
+
from transformers import pipeline
|
353 |
+
print("β
Transformers available")
|
354 |
+
except ImportError:
|
355 |
+
print("β οΈ Transformers not available (CPU fallback)")
|
356 |
+
|
357 |
+
# Run health check in your Space
|
358 |
+
```
|
359 |
+
|
360 |
+
---
|
361 |
+
|
362 |
+
## π Maintenance & Updates
|
363 |
+
|
364 |
+
### **Regular Maintenance Tasks**
|
365 |
+
|
366 |
+
1. **Monitor API Usage**
|
367 |
+
```bash
|
368 |
+
# Check Perplexity dashboard for:
|
369 |
+
- API calls remaining
|
370 |
+
- Rate limit status
|
371 |
+
- Billing usage
|
372 |
+
```
|
373 |
+
|
374 |
+
2. **Update Dependencies**
|
375 |
+
```bash
|
376 |
+
# Periodically update requirements.txt:
|
377 |
+
gradio>=4.44.0 # Check for latest version
|
378 |
+
transformers>=4.30.0
|
379 |
+
# Test thoroughly after updates
|
380 |
+
```
|
381 |
+
|
382 |
+
3. **Performance Monitoring**
|
383 |
+
```bash
|
384 |
+
# Monitor Space metrics:
|
385 |
+
- CPU/GPU usage
|
386 |
+
- Memory consumption
|
387 |
+
- Request response times
|
388 |
+
- Error rates
|
389 |
+
```
|
390 |
+
|
391 |
+
### **Backup Strategy**
|
392 |
+
|
393 |
+
```bash
|
394 |
+
# Important data to backup:
|
395 |
+
1. Configuration files (app.py, config.py)
|
396 |
+
2. Custom templates or modifications
|
397 |
+
3. API keys and environment variables
|
398 |
+
4. Any persistent data or datasets
|
399 |
+
|
400 |
+
# HuggingFace Spaces automatically versions your files
|
401 |
+
# Use git commands to manage versions
|
402 |
+
```
|
403 |
+
|
404 |
+
---
|
405 |
+
|
406 |
+
## π Scaling & Optimization
|
407 |
+
|
408 |
+
### **Performance Optimization**
|
409 |
+
|
410 |
+
1. **Model Optimization**
|
411 |
+
```python
|
412 |
+
# In config.py, adjust for your needs:
|
413 |
+
batch_size = 16 # Increase for better GPU utilization
|
414 |
+
max_sequence_length = 256 # Reduce for faster processing
|
415 |
+
confidence_threshold = 0.8 # Higher for better quality
|
416 |
+
```
|
417 |
+
|
418 |
+
2. **Caching Strategy**
|
419 |
+
```python
|
420 |
+
# Enable model caching:
|
421 |
+
cache_models = True
|
422 |
+
model_cache_dir = "./model_cache"
|
423 |
+
|
424 |
+
# Cache API responses:
|
425 |
+
cache_api_responses = True
|
426 |
+
cache_ttl_hours = 24
|
427 |
+
```
|
428 |
+
|
429 |
+
3. **Resource Management**
|
430 |
+
```python
|
431 |
+
# Optimize memory usage:
|
432 |
+
clear_cache_after_processing = True
|
433 |
+
max_concurrent_requests = 3
|
434 |
+
timeout_per_url = 10 # seconds
|
435 |
+
```
|
436 |
+
|
437 |
+
### **Cost Optimization**
|
438 |
+
|
439 |
+
1. **Auto-Sleep Configuration**
|
440 |
+
```bash
|
441 |
+
# HuggingFace Spaces auto-sleep after 1 hour idle
|
442 |
+
# No additional configuration needed
|
443 |
+
# Automatically resumes on next request
|
444 |
+
```
|
445 |
+
|
446 |
+
2. **Hardware Scheduling**
|
447 |
+
```bash
|
448 |
+
# Strategy: Start with CPU Basic
|
449 |
+
# Upgrade to T4 Small during processing
|
450 |
+
# Downgrade back to CPU Basic when idle
|
451 |
+
```
|
452 |
+
|
453 |
+
3. **API Cost Management**
|
454 |
+
```bash
|
455 |
+
# Perplexity API optimization:
|
456 |
+
- Cache search results for similar queries
|
457 |
+
- Use more specific search terms
|
458 |
+
- Implement request batching
|
459 |
+
- Set reasonable max_sources limits
|
460 |
+
```
|
461 |
+
|
462 |
+
---
|
463 |
+
|
464 |
+
## π Best Practices
|
465 |
+
|
466 |
+
### **Security Best Practices**
|
467 |
+
|
468 |
+
1. **API Key Management**
|
469 |
+
```bash
|
470 |
+
β
Store in HuggingFace Spaces secrets
|
471 |
+
β
Never commit to git repositories
|
472 |
+
β
Rotate keys periodically
|
473 |
+
β
Monitor usage for anomalies
|
474 |
+
```
|
475 |
+
|
476 |
+
2. **Safe Scraping**
|
477 |
+
```bash
|
478 |
+
β
Respect robots.txt
|
479 |
+
β
Implement rate limiting
|
480 |
+
β
Use appropriate user agents
|
481 |
+
β
Avoid private/internal networks
|
482 |
+
```
|
483 |
+
|
484 |
+
3. **Data Privacy**
|
485 |
+
```bash
|
486 |
+
β
No persistent data storage by default
|
487 |
+
β
Clear temporary files after processing
|
488 |
+
β
Respect copyright and fair use
|
489 |
+
β
Provide clear data source attribution
|
490 |
+
```
|
491 |
+
|
492 |
+
### **Development Best Practices**
|
493 |
+
|
494 |
+
1. **Testing Strategy**
|
495 |
+
```bash
|
496 |
+
# Test with small datasets first
|
497 |
+
# Verify each step of the pipeline
|
498 |
+
# Use diverse source types
|
499 |
+
# Test error conditions
|
500 |
+
```
|
501 |
+
|
502 |
+
2. **Version Control**
|
503 |
+
```bash
|
504 |
+
# Use git for file management
|
505 |
+
# Tag stable releases
|
506 |
+
# Document changes and updates
|
507 |
+
# Keep rollback capability
|
508 |
+
```
|
509 |
+
|
510 |
+
3. **Documentation**
|
511 |
+
```bash
|
512 |
+
# Keep README.md updated
|
513 |
+
# Document custom configurations
|
514 |
+
# Provide usage examples
|
515 |
+
# Include troubleshooting guides
|
516 |
+
```
|
517 |
+
|
518 |
+
---
|
519 |
+
|
520 |
+
## π Getting Help
|
521 |
+
|
522 |
+
### **Support Channels**
|
523 |
+
|
524 |
+
1. **HuggingFace Community**
|
525 |
+
- Discussions: Share issues and solutions
|
526 |
+
- Discord: Real-time help from community
|
527 |
+
|
528 |
+
2. **GitHub Issues**
|
529 |
+
- Bug reports and feature requests
|
530 |
+
- Include logs and configuration details
|
531 |
+
|
532 |
+
3. **Documentation**
|
533 |
+
- README.md: Complete usage guide
|
534 |
+
- DEPLOYMENT.md: This guide
|
535 |
+
- Code comments: Inline documentation
|
536 |
+
|
537 |
+
### **Information to Include When Asking for Help**
|
538 |
+
|
539 |
+
```bash
|
540 |
+
1. Deployment type (CPU Basic, T4 Small, etc.)
|
541 |
+
2. Error messages (exact text)
|
542 |
+
3. Space logs (relevant sections)
|
543 |
+
4. Configuration details (without API keys)
|
544 |
+
5. Steps to reproduce the issue
|
545 |
+
6. Expected vs actual behavior
|
546 |
+
```
|
547 |
+
|
548 |
+
---
|
549 |
+
|
550 |
+
## π Success Indicators
|
551 |
+
|
552 |
+
Your deployment is successful when you see:
|
553 |
+
|
554 |
+
```bash
|
555 |
+
β
Space builds without errors
|
556 |
+
β
Interface loads within 60 seconds
|
557 |
+
β
Perplexity AI discovery works
|
558 |
+
β
Can create projects and scrape URLs
|
559 |
+
β
AI processing generates quality data
|
560 |
+
β
Export produces valid dataset files
|
561 |
+
β
No persistent errors in logs
|
562 |
+
```
|
563 |
+
|
564 |
+
---
|
565 |
+
|
566 |
+
## π What's Next?
|
567 |
+
|
568 |
+
After successful deployment:
|
569 |
+
|
570 |
+
1. **Create Your First Dataset**
|
571 |
+
- Start with a simple sentiment analysis project
|
572 |
+
- Use AI discovery to find sources
|
573 |
+
- Process and export a small dataset
|
574 |
+
|
575 |
+
2. **Explore Advanced Features**
|
576 |
+
- Try different templates
|
577 |
+
- Experiment with search types
|
578 |
+
- Test batch processing
|
579 |
+
|
580 |
+
3. **Optimize for Your Use Case**
|
581 |
+
- Adjust configurations
|
582 |
+
- Create custom templates
|
583 |
+
- Integrate with your ML pipeline
|
584 |
+
|
585 |
+
4. **Share and Collaborate**
|
586 |
+
- Make Space public to help others
|
587 |
+
- Contribute improvements
|
588 |
+
- Share success stories
|
589 |
+
|
590 |
+
**Your AI Dataset Studio is now ready to revolutionize how you create training datasets!** π―
|
591 |
+
|
592 |
+
*From idea to ML-ready dataset in minutes, not weeks.*
|