MagicMeWizard commited on
Commit
dcb20f6
Β·
verified Β·
1 Parent(s): ccc5d44

Create DEPLOYMENT.md

Browse files
Files changed (1) hide show
  1. DEPLOYMENT.md +592 -0
DEPLOYMENT.md ADDED
@@ -0,0 +1,592 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # πŸš€ AI Dataset Studio - Complete Deployment Guide
2
+
3
+ **Deploy your AI-powered dataset creation platform with Perplexity integration**
4
+
5
+ ---
6
+
7
+ ## πŸ“‹ Pre-Deployment Checklist
8
+
9
+ ### βœ… **Required Files**
10
+ Ensure you have all these files ready:
11
+
12
+ ```
13
+ ai-dataset-studio/
14
+ β”œβ”€β”€ app.py # Main application with Perplexity integration
15
+ β”œβ”€β”€ perplexity_client.py # Perplexity AI client
16
+ β”œβ”€β”€ config.py # Configuration management
17
+ β”œβ”€β”€ requirements.txt # Dependencies
18
+ β”œβ”€β”€ README.md # Documentation
19
+ β”œβ”€β”€ DEPLOYMENT.md # This guide
20
+ └── utils.py # Utility functions (optional)
21
+ ```
22
+
23
+ ### βœ… **API Keys & Environment**
24
+ - [ ] **Perplexity API Key** - Get from [Perplexity AI](https://www.perplexity.ai/)
25
+ - [ ] **HuggingFace Account** - For Space hosting
26
+ - [ ] **Optional**: HuggingFace Token for private datasets
27
+
28
+ ---
29
+
30
+ ## 🎯 Deployment Options
31
+
32
+ ### **Option 1: Full AI-Powered Deployment (Recommended)**
33
+ *Best for: Professional use, maximum features*
34
+
35
+ #### Hardware: **T4 Small** ($0.60/hour)
36
+ - βœ… GPU acceleration for AI models
37
+ - βœ… Fast processing (5-15s per article)
38
+ - βœ… All Perplexity features enabled
39
+ - βœ… Production-ready performance
40
+
41
+ #### **Step-by-Step:**
42
+
43
+ 1. **Create HuggingFace Space**
44
+ ```bash
45
+ # Go to: https://huggingface.co/new-space
46
+ Space Name: ai-dataset-studio
47
+ SDK: Gradio
48
+ Hardware: T4 Small
49
+ Visibility: Public (or Private)
50
+ ```
51
+
52
+ 2. **Upload Files**
53
+ - Copy all files from artifacts above
54
+ - Ensure `app.py` is the main file
55
+ - Keep file structure intact
56
+
57
+ 3. **Set Environment Variables**
58
+ ```bash
59
+ # In Space Settings β†’ Repository secrets:
60
+ PERPLEXITY_API_KEY = your_perplexity_api_key_here
61
+
62
+ # Optional:
63
+ HF_TOKEN = your_huggingface_token
64
+ LOG_LEVEL = INFO
65
+ DEBUG = false
66
+ ```
67
+
68
+ 4. **Deploy & Test**
69
+ - Space will build automatically (2-3 minutes)
70
+ - Test Perplexity integration first
71
+ - Verify all templates work
72
+
73
+ ---
74
+
75
+ ### **Option 2: Budget-Friendly Deployment**
76
+ *Best for: Testing, learning, cost-conscious users*
77
+
78
+ #### Hardware: **CPU Basic** (Free)
79
+ - ⚑ Basic functionality available
80
+ - ⚠️ Slower AI processing (30-60s per article)
81
+ - βœ… Perplexity discovery still works
82
+ - βœ… Perfect for getting started
83
+
84
+ #### **Step-by-Step:**
85
+
86
+ 1. **Create Space with CPU Basic**
87
+ ```bash
88
+ Space Name: ai-dataset-studio
89
+ SDK: Gradio
90
+ Hardware: CPU Basic (Free)
91
+ ```
92
+
93
+ 2. **Upload Core Files**
94
+ ```bash
95
+ # Essential files only:
96
+ app.py
97
+ perplexity_client.py
98
+ requirements.txt
99
+ README.md
100
+ config.py
101
+ ```
102
+
103
+ 3. **Set API Key**
104
+ ```bash
105
+ PERPLEXITY_API_KEY = your_api_key
106
+ ```
107
+
108
+ 4. **Gradual Upgrade Path**
109
+ - Start with CPU Basic
110
+ - Test functionality
111
+ - Upgrade to T4 Small when ready
112
+
113
+ ---
114
+
115
+ ### **Option 3: Enterprise Deployment**
116
+ *Best for: High-volume usage, team collaboration*
117
+
118
+ #### Hardware: **A10G Small** ($1.05/hour)
119
+ - πŸš€ Maximum performance (3-8s per article)
120
+ - πŸ’ͺ Handle large batch processing
121
+ - πŸ”„ Support multiple concurrent users
122
+ - πŸ“ˆ Production-scale capabilities
123
+
124
+ #### **Additional Setup:**
125
+
126
+ 1. **Persistent Storage**
127
+ ```bash
128
+ # In Space settings:
129
+ Storage: Small Persistent ($5/month)
130
+ # Enables data persistence between restarts
131
+ ```
132
+
133
+ 2. **Advanced Configuration**
134
+ ```bash
135
+ # Environment variables:
136
+ MAX_SOURCES_PER_SEARCH = 50
137
+ BATCH_SIZE = 16
138
+ ENABLE_CACHING = true
139
+ CONCURRENT_REQUESTS = 5
140
+ ```
141
+
142
+ 3. **Monitoring Setup**
143
+ ```bash
144
+ # Enable detailed logging:
145
+ LOG_LEVEL = DEBUG
146
+ ENABLE_METRICS = true
147
+ ```
148
+
149
+ ---
150
+
151
+ ## πŸ”§ Configuration Details
152
+
153
+ ### **Perplexity API Setup**
154
+
155
+ 1. **Get API Key**
156
+ ```bash
157
+ # Visit: https://www.perplexity.ai/
158
+ # Sign up for account
159
+ # Navigate to API section
160
+ # Generate new API key
161
+ # Copy key for environment setup
162
+ ```
163
+
164
+ 2. **Test API Key**
165
+ ```python
166
+ # Quick test script:
167
+ import requests
168
+
169
+ headers = {
170
+ 'Authorization': 'Bearer YOUR_API_KEY',
171
+ 'Content-Type': 'application/json'
172
+ }
173
+
174
+ response = requests.post(
175
+ 'https://api.perplexity.ai/chat/completions',
176
+ headers=headers,
177
+ json={
178
+ "model": "llama-3.1-sonar-large-128k-online",
179
+ "messages": [{"role": "user", "content": "Test message"}]
180
+ }
181
+ )
182
+
183
+ print("API Status:", response.status_code)
184
+ ```
185
+
186
+ ### **Hardware Requirements by Use Case**
187
+
188
+ | Use Case | Hardware | Monthly Cost | Performance | Best For |
189
+ |----------|----------|--------------|-------------|----------|
190
+ | **Learning** | CPU Basic | Free | Basic | Students, hobbyists |
191
+ | **Development** | CPU Upgrade | $22 | Good | Developers, testing |
192
+ | **Production** | T4 Small | $432 | Excellent | Businesses, researchers |
193
+ | **Enterprise** | A10G Small | $756 | Maximum | High-volume, teams |
194
+
195
+ ### **Memory & Storage Planning**
196
+
197
+ ```bash
198
+ # Model Memory Usage:
199
+ BART Summarization: ~1.5GB
200
+ RoBERTa Sentiment: ~500MB
201
+ BERT NER: ~400MB
202
+ Base Application: ~200MB
203
+ Total GPU Memory: ~2.5GB (T4 Small = 16GB, plenty of headroom)
204
+
205
+ # Storage Usage:
206
+ Application Files: ~50MB
207
+ Model Cache: ~2GB
208
+ Temporary Data: ~100MB per project
209
+ Persistent Storage: Optional, recommended for large projects
210
+ ```
211
+
212
+ ---
213
+
214
+ ## πŸ§ͺ Testing Your Deployment
215
+
216
+ ### **Basic Functionality Test**
217
+
218
+ 1. **Launch Application**
219
+ ```bash
220
+ # Your Space URL: https://huggingface.co/spaces/YOUR_USERNAME/ai-dataset-studio
221
+ # Wait for "Running" status
222
+ # Interface should load within 30-60 seconds
223
+ ```
224
+
225
+ 2. **Test Project Creation**
226
+ ```bash
227
+ Project Name: "Test Sentiment Analysis"
228
+ Template: Sentiment Analysis
229
+ Description: "Testing the deployment"
230
+ Click: "Create Project"
231
+ Expected: "βœ… Project created successfully"
232
+ ```
233
+
234
+ 3. **Test Perplexity Integration**
235
+ ```bash
236
+ AI Search Description: "Product reviews for sentiment analysis"
237
+ Search Type: General
238
+ Max Sources: 10
239
+ Click: "Discover Sources with AI"
240
+ Expected: List of relevant URLs with quality scores
241
+ ```
242
+
243
+ ### **Advanced Testing**
244
+
245
+ 4. **Test Complete Workflow**
246
+ ```bash
247
+ # Use discovered sources from step 3
248
+ Click: "Use These Sources"
249
+ Click: "Start Scraping"
250
+ Wait: Processing to complete
251
+ Click: "Process Data"
252
+ Select: Same template as project
253
+ Click: "Export Dataset"
254
+ Format: JSON
255
+ Expected: Downloadable dataset file
256
+ ```
257
+
258
+ 5. **Performance Benchmarks**
259
+ ```bash
260
+ # Timing expectations:
261
+ AI Source Discovery: 5-15 seconds
262
+ Scraping 10 URLs: 30-120 seconds
263
+ Processing Data: 30-180 seconds (depends on hardware)
264
+ Export: 5-10 seconds
265
+ ```
266
+
267
+ ---
268
+
269
+ ## 🚨 Troubleshooting
270
+
271
+ ### **Common Issues & Solutions**
272
+
273
+ #### ❌ **"Perplexity API key not found"**
274
+ ```bash
275
+ # Problem: Environment variable not set
276
+ # Solution:
277
+ 1. Go to Space Settings β†’ Repository secrets
278
+ 2. Add: PERPLEXITY_API_KEY = your_key_here
279
+ 3. Restart Space
280
+ 4. Check logs for "βœ… Perplexity AI client initialized"
281
+ ```
282
+
283
+ #### ❌ **"No sources found" from AI discovery**
284
+ ```bash
285
+ # Problem: Search query too specific or API limits
286
+ # Solutions:
287
+ 1. Make description more general
288
+ 2. Try different search types
289
+ 3. Check API key has sufficient credits
290
+ 4. Use manual URL entry as fallback
291
+ ```
292
+
293
+ #### ❌ **"Model loading failed"**
294
+ ```bash
295
+ # Problem: Insufficient memory or network issues
296
+ # Solutions:
297
+ 1. Upgrade to T4 Small for GPU memory
298
+ 2. Wait 2-3 minutes for model downloads
299
+ 3. Check Space logs for specific errors
300
+ 4. Restart Space if persistent
301
+ ```
302
+
303
+ #### ❌ **"Scraping failed" for multiple URLs**
304
+ ```bash
305
+ # Problem: Rate limiting or blocked access
306
+ # Solutions:
307
+ 1. Reduce concurrent requests
308
+ 2. Check robots.txt compliance
309
+ 3. Use more diverse sources
310
+ 4. Verify URLs are publicly accessible
311
+ ```
312
+
313
+ ### **Debug Mode**
314
+
315
+ Enable detailed logging for troubleshooting:
316
+
317
+ ```bash
318
+ # Environment variables:
319
+ DEBUG = true
320
+ LOG_LEVEL = DEBUG
321
+
322
+ # Then check Space logs for detailed information
323
+ ```
324
+
325
+ ### **Health Check Script**
326
+
327
+ ```python
328
+ # Add this to test basic functionality:
329
+ def health_check():
330
+ """Test all components"""
331
+
332
+ # Test imports
333
+ try:
334
+ import gradio
335
+ print("βœ… Gradio imported")
336
+ except ImportError:
337
+ print("❌ Gradio import failed")
338
+
339
+ # Test Perplexity
340
+ try:
341
+ from perplexity_client import PerplexityClient
342
+ client = PerplexityClient()
343
+ if client._validate_api_key():
344
+ print("βœ… Perplexity API key valid")
345
+ else:
346
+ print("❌ Perplexity API key invalid")
347
+ except Exception as e:
348
+ print(f"❌ Perplexity error: {e}")
349
+
350
+ # Test models
351
+ try:
352
+ from transformers import pipeline
353
+ print("βœ… Transformers available")
354
+ except ImportError:
355
+ print("⚠️ Transformers not available (CPU fallback)")
356
+
357
+ # Run health check in your Space
358
+ ```
359
+
360
+ ---
361
+
362
+ ## πŸ”„ Maintenance & Updates
363
+
364
+ ### **Regular Maintenance Tasks**
365
+
366
+ 1. **Monitor API Usage**
367
+ ```bash
368
+ # Check Perplexity dashboard for:
369
+ - API calls remaining
370
+ - Rate limit status
371
+ - Billing usage
372
+ ```
373
+
374
+ 2. **Update Dependencies**
375
+ ```bash
376
+ # Periodically update requirements.txt:
377
+ gradio>=4.44.0 # Check for latest version
378
+ transformers>=4.30.0
379
+ # Test thoroughly after updates
380
+ ```
381
+
382
+ 3. **Performance Monitoring**
383
+ ```bash
384
+ # Monitor Space metrics:
385
+ - CPU/GPU usage
386
+ - Memory consumption
387
+ - Request response times
388
+ - Error rates
389
+ ```
390
+
391
+ ### **Backup Strategy**
392
+
393
+ ```bash
394
+ # Important data to backup:
395
+ 1. Configuration files (app.py, config.py)
396
+ 2. Custom templates or modifications
397
+ 3. API keys and environment variables
398
+ 4. Any persistent data or datasets
399
+
400
+ # HuggingFace Spaces automatically versions your files
401
+ # Use git commands to manage versions
402
+ ```
403
+
404
+ ---
405
+
406
+ ## πŸ“ˆ Scaling & Optimization
407
+
408
+ ### **Performance Optimization**
409
+
410
+ 1. **Model Optimization**
411
+ ```python
412
+ # In config.py, adjust for your needs:
413
+ batch_size = 16 # Increase for better GPU utilization
414
+ max_sequence_length = 256 # Reduce for faster processing
415
+ confidence_threshold = 0.8 # Higher for better quality
416
+ ```
417
+
418
+ 2. **Caching Strategy**
419
+ ```python
420
+ # Enable model caching:
421
+ cache_models = True
422
+ model_cache_dir = "./model_cache"
423
+
424
+ # Cache API responses:
425
+ cache_api_responses = True
426
+ cache_ttl_hours = 24
427
+ ```
428
+
429
+ 3. **Resource Management**
430
+ ```python
431
+ # Optimize memory usage:
432
+ clear_cache_after_processing = True
433
+ max_concurrent_requests = 3
434
+ timeout_per_url = 10 # seconds
435
+ ```
436
+
437
+ ### **Cost Optimization**
438
+
439
+ 1. **Auto-Sleep Configuration**
440
+ ```bash
441
+ # HuggingFace Spaces auto-sleep after 1 hour idle
442
+ # No additional configuration needed
443
+ # Automatically resumes on next request
444
+ ```
445
+
446
+ 2. **Hardware Scheduling**
447
+ ```bash
448
+ # Strategy: Start with CPU Basic
449
+ # Upgrade to T4 Small during processing
450
+ # Downgrade back to CPU Basic when idle
451
+ ```
452
+
453
+ 3. **API Cost Management**
454
+ ```bash
455
+ # Perplexity API optimization:
456
+ - Cache search results for similar queries
457
+ - Use more specific search terms
458
+ - Implement request batching
459
+ - Set reasonable max_sources limits
460
+ ```
461
+
462
+ ---
463
+
464
+ ## πŸŽ“ Best Practices
465
+
466
+ ### **Security Best Practices**
467
+
468
+ 1. **API Key Management**
469
+ ```bash
470
+ βœ… Store in HuggingFace Spaces secrets
471
+ βœ… Never commit to git repositories
472
+ βœ… Rotate keys periodically
473
+ βœ… Monitor usage for anomalies
474
+ ```
475
+
476
+ 2. **Safe Scraping**
477
+ ```bash
478
+ βœ… Respect robots.txt
479
+ βœ… Implement rate limiting
480
+ βœ… Use appropriate user agents
481
+ βœ… Avoid private/internal networks
482
+ ```
483
+
484
+ 3. **Data Privacy**
485
+ ```bash
486
+ βœ… No persistent data storage by default
487
+ βœ… Clear temporary files after processing
488
+ βœ… Respect copyright and fair use
489
+ βœ… Provide clear data source attribution
490
+ ```
491
+
492
+ ### **Development Best Practices**
493
+
494
+ 1. **Testing Strategy**
495
+ ```bash
496
+ # Test with small datasets first
497
+ # Verify each step of the pipeline
498
+ # Use diverse source types
499
+ # Test error conditions
500
+ ```
501
+
502
+ 2. **Version Control**
503
+ ```bash
504
+ # Use git for file management
505
+ # Tag stable releases
506
+ # Document changes and updates
507
+ # Keep rollback capability
508
+ ```
509
+
510
+ 3. **Documentation**
511
+ ```bash
512
+ # Keep README.md updated
513
+ # Document custom configurations
514
+ # Provide usage examples
515
+ # Include troubleshooting guides
516
+ ```
517
+
518
+ ---
519
+
520
+ ## πŸ†˜ Getting Help
521
+
522
+ ### **Support Channels**
523
+
524
+ 1. **HuggingFace Community**
525
+ - Discussions: Share issues and solutions
526
+ - Discord: Real-time help from community
527
+
528
+ 2. **GitHub Issues**
529
+ - Bug reports and feature requests
530
+ - Include logs and configuration details
531
+
532
+ 3. **Documentation**
533
+ - README.md: Complete usage guide
534
+ - DEPLOYMENT.md: This guide
535
+ - Code comments: Inline documentation
536
+
537
+ ### **Information to Include When Asking for Help**
538
+
539
+ ```bash
540
+ 1. Deployment type (CPU Basic, T4 Small, etc.)
541
+ 2. Error messages (exact text)
542
+ 3. Space logs (relevant sections)
543
+ 4. Configuration details (without API keys)
544
+ 5. Steps to reproduce the issue
545
+ 6. Expected vs actual behavior
546
+ ```
547
+
548
+ ---
549
+
550
+ ## πŸŽ‰ Success Indicators
551
+
552
+ Your deployment is successful when you see:
553
+
554
+ ```bash
555
+ βœ… Space builds without errors
556
+ βœ… Interface loads within 60 seconds
557
+ βœ… Perplexity AI discovery works
558
+ βœ… Can create projects and scrape URLs
559
+ βœ… AI processing generates quality data
560
+ βœ… Export produces valid dataset files
561
+ βœ… No persistent errors in logs
562
+ ```
563
+
564
+ ---
565
+
566
+ ## πŸš€ What's Next?
567
+
568
+ After successful deployment:
569
+
570
+ 1. **Create Your First Dataset**
571
+ - Start with a simple sentiment analysis project
572
+ - Use AI discovery to find sources
573
+ - Process and export a small dataset
574
+
575
+ 2. **Explore Advanced Features**
576
+ - Try different templates
577
+ - Experiment with search types
578
+ - Test batch processing
579
+
580
+ 3. **Optimize for Your Use Case**
581
+ - Adjust configurations
582
+ - Create custom templates
583
+ - Integrate with your ML pipeline
584
+
585
+ 4. **Share and Collaborate**
586
+ - Make Space public to help others
587
+ - Contribute improvements
588
+ - Share success stories
589
+
590
+ **Your AI Dataset Studio is now ready to revolutionize how you create training datasets!** 🎯
591
+
592
+ *From idea to ML-ready dataset in minutes, not weeks.*