DEPLOYMENT.md

# 🚀 AI Dataset Studio - Complete Deployment Guide

**Deploy your AI-powered dataset creation platform with Perplexity integration**

---

## 📋 Pre-Deployment Checklist

### ✅ **Required Files**
Ensure you have all these files ready:

```
ai-dataset-studio/
├── app.py                    # Main application with Perplexity integration
├── perplexity_client.py     # Perplexity AI client
├── config.py               # Configuration management
├── requirements.txt        # Dependencies
├── README.md              # Documentation
├── DEPLOYMENT.md          # This guide
└── utils.py               # Utility functions (optional)
```

### ✅ **API Keys & Environment**
- [ ] **Perplexity API Key** - Get from [Perplexity AI](https://www.perplexity.ai/)
- [ ] **HuggingFace Account** - For Space hosting
- [ ] **Optional**: HuggingFace Token for private datasets

---

## 🎯 Deployment Options

### **Option 1: Full AI-Powered Deployment (Recommended)**
*Best for: Professional use, maximum features*

#### Hardware: **T4 Small** ($0.60/hour)
- ✅ GPU acceleration for AI models
- ✅ Fast processing (5-15s per article)
- ✅ All Perplexity features enabled
- ✅ Production-ready performance

#### **Step-by-Step:**

1. **Create HuggingFace Space**
   ```bash
   # Go to: https://huggingface.co/new-space
   Space Name: ai-dataset-studio
   SDK: Gradio
   Hardware: T4 Small
   Visibility: Public (or Private)
   ```

2. **Upload Files**
   - Copy all files from artifacts above
   - Ensure `app.py` is the main file
   - Keep file structure intact

3. **Set Environment Variables**
   ```bash
   # In Space Settings → Repository secrets:
   PERPLEXITY_API_KEY = your_perplexity_api_key_here
   
   # Optional:
   HF_TOKEN = your_huggingface_token
   LOG_LEVEL = INFO
   DEBUG = false
   ```

4. **Deploy & Test**
   - Space will build automatically (2-3 minutes)
   - Test Perplexity integration first
   - Verify all templates work

---

### **Option 2: Budget-Friendly Deployment**
*Best for: Testing, learning, cost-conscious users*

#### Hardware: **CPU Basic** (Free)
- ⚡ Basic functionality available
- ⚠️ Slower AI processing (30-60s per article)
- ✅ Perplexity discovery still works
- ✅ Perfect for getting started

#### **Step-by-Step:**

1. **Create Space with CPU Basic**
   ```bash
   Space Name: ai-dataset-studio
   SDK: Gradio
   Hardware: CPU Basic (Free)
   ```

2. **Upload Core Files**
   ```bash
   # Essential files only:
   app.py
   perplexity_client.py
   requirements.txt
   README.md
   config.py
   ```

3. **Set API Key**
   ```bash
   PERPLEXITY_API_KEY = your_api_key
   ```

4. **Gradual Upgrade Path**
   - Start with CPU Basic
   - Test functionality
   - Upgrade to T4 Small when ready

---

### **Option 3: Enterprise Deployment**
*Best for: High-volume usage, team collaboration*

#### Hardware: **A10G Small** ($1.05/hour)
- 🚀 Maximum performance (3-8s per article)
- 💪 Handle large batch processing
- 🔄 Support multiple concurrent users
- 📈 Production-scale capabilities

#### **Additional Setup:**

1. **Persistent Storage**
   ```bash
   # In Space settings:
   Storage: Small Persistent ($5/month)
   # Enables data persistence between restarts
   ```

2. **Advanced Configuration**
   ```bash
   # Environment variables:
   MAX_SOURCES_PER_SEARCH = 50
   BATCH_SIZE = 16
   ENABLE_CACHING = true
   CONCURRENT_REQUESTS = 5
   ```

3. **Monitoring Setup**
   ```bash
   # Enable detailed logging:
   LOG_LEVEL = DEBUG
   ENABLE_METRICS = true
   ```

---

## 🔧 Configuration Details

### **Perplexity API Setup**

1. **Get API Key**
   ```bash
   # Visit: https://www.perplexity.ai/
   # Sign up for account
   # Navigate to API section
   # Generate new API key
   # Copy key for environment setup
   ```

2. **Test API Key**
   ```python
   # Quick test script:
   import requests
   
   headers = {
       'Authorization': 'Bearer YOUR_API_KEY',
       'Content-Type': 'application/json'
   }
   
   response = requests.post(
       'https://api.perplexity.ai/chat/completions',
       headers=headers,
       json={
           "model": "llama-3.1-sonar-large-128k-online",
           "messages": [{"role": "user", "content": "Test message"}]
       }
   )
   
   print("API Status:", response.status_code)
   ```

### **Hardware Requirements by Use Case**

| Use Case | Hardware | Monthly Cost | Performance | Best For |
|----------|----------|--------------|-------------|----------|
| **Learning** | CPU Basic | Free | Basic | Students, hobbyists |
| **Development** | CPU Upgrade | $22 | Good | Developers, testing |
| **Production** | T4 Small | $432 | Excellent | Businesses, researchers |
| **Enterprise** | A10G Small | $756 | Maximum | High-volume, teams |

### **Memory & Storage Planning**

```bash
# Model Memory Usage:
BART Summarization: ~1.5GB
RoBERTa Sentiment: ~500MB
BERT NER: ~400MB
Base Application: ~200MB
Total GPU Memory: ~2.5GB (T4 Small = 16GB, plenty of headroom)

# Storage Usage:
Application Files: ~50MB
Model Cache: ~2GB
Temporary Data: ~100MB per project
Persistent Storage: Optional, recommended for large projects
```

---

## 🧪 Testing Your Deployment

### **Basic Functionality Test**

1. **Launch Application**
   ```bash
   # Your Space URL: https://huggingface.co/spaces/YOUR_USERNAME/ai-dataset-studio
   # Wait for "Running" status
   # Interface should load within 30-60 seconds
   ```

2. **Test Project Creation**
   ```bash
   Project Name: "Test Sentiment Analysis"
   Template: Sentiment Analysis
   Description: "Testing the deployment"
   Click: "Create Project"
   Expected: "✅ Project created successfully"
   ```

3. **Test Perplexity Integration**
   ```bash
   AI Search Description: "Product reviews for sentiment analysis"
   Search Type: General
   Max Sources: 10
   Click: "Discover Sources with AI"
   Expected: List of relevant URLs with quality scores
   ```

### **Advanced Testing**

4. **Test Complete Workflow**
   ```bash
   # Use discovered sources from step 3
   Click: "Use These Sources"
   Click: "Start Scraping"
   Wait: Processing to complete
   Click: "Process Data"
   Select: Same template as project
   Click: "Export Dataset"
   Format: JSON
   Expected: Downloadable dataset file
   ```

5. **Performance Benchmarks**
   ```bash
   # Timing expectations:
   AI Source Discovery: 5-15 seconds
   Scraping 10 URLs: 30-120 seconds
   Processing Data: 30-180 seconds (depends on hardware)
   Export: 5-10 seconds
   ```

---

## 🚨 Troubleshooting

### **Common Issues & Solutions**

#### ❌ **"Perplexity API key not found"**
```bash
# Problem: Environment variable not set
# Solution:
1. Go to Space Settings → Repository secrets
2. Add: PERPLEXITY_API_KEY = your_key_here
3. Restart Space
4. Check logs for "✅ Perplexity AI client initialized"
```

#### ❌ **"No sources found" from AI discovery**
```bash
# Problem: Search query too specific or API limits
# Solutions:
1. Make description more general
2. Try different search types
3. Check API key has sufficient credits
4. Use manual URL entry as fallback
```

#### ❌ **"Model loading failed"**
```bash
# Problem: Insufficient memory or network issues
# Solutions:
1. Upgrade to T4 Small for GPU memory
2. Wait 2-3 minutes for model downloads
3. Check Space logs for specific errors
4. Restart Space if persistent
```

#### ❌ **"Scraping failed" for multiple URLs**
```bash
# Problem: Rate limiting or blocked access
# Solutions:
1. Reduce concurrent requests
2. Check robots.txt compliance
3. Use more diverse sources
4. Verify URLs are publicly accessible
```

### **Debug Mode**

Enable detailed logging for troubleshooting:

```bash
# Environment variables:
DEBUG = true
LOG_LEVEL = DEBUG

# Then check Space logs for detailed information
```

### **Health Check Script**

```python
# Add this to test basic functionality:
def health_check():
    """Test all components"""
    
    # Test imports
    try:
        import gradio
        print("✅ Gradio imported")
    except ImportError:
        print("❌ Gradio import failed")
    
    # Test Perplexity
    try:
        from perplexity_client import PerplexityClient
        client = PerplexityClient()
        if client._validate_api_key():
            print("✅ Perplexity API key valid")
        else:
            print("❌ Perplexity API key invalid")
    except Exception as e:
        print(f"❌ Perplexity error: {e}")
    
    # Test models
    try:
        from transformers import pipeline
        print("✅ Transformers available")
    except ImportError:
        print("⚠️ Transformers not available (CPU fallback)")

# Run health check in your Space
```

---

## 🔄 Maintenance & Updates

### **Regular Maintenance Tasks**

1. **Monitor API Usage**
   ```bash
   # Check Perplexity dashboard for:
   - API calls remaining
   - Rate limit status
   - Billing usage
   ```

2. **Update Dependencies**
   ```bash
   # Periodically update requirements.txt:
   gradio>=4.44.0  # Check for latest version
   transformers>=4.30.0
   # Test thoroughly after updates
   ```

3. **Performance Monitoring**
   ```bash
   # Monitor Space metrics:
   - CPU/GPU usage
   - Memory consumption
   - Request response times
   - Error rates
   ```

### **Backup Strategy**

```bash
# Important data to backup:
1. Configuration files (app.py, config.py)
2. Custom templates or modifications
3. API keys and environment variables
4. Any persistent data or datasets

# HuggingFace Spaces automatically versions your files
# Use git commands to manage versions
```

---

## 📈 Scaling & Optimization

### **Performance Optimization**

1. **Model Optimization**
   ```python
   # In config.py, adjust for your needs:
   batch_size = 16  # Increase for better GPU utilization
   max_sequence_length = 256  # Reduce for faster processing
   confidence_threshold = 0.8  # Higher for better quality
   ```

2. **Caching Strategy**
   ```python
   # Enable model caching:
   cache_models = True
   model_cache_dir = "./model_cache"
   
   # Cache API responses:
   cache_api_responses = True
   cache_ttl_hours = 24
   ```

3. **Resource Management**
   ```python
   # Optimize memory usage:
   clear_cache_after_processing = True
   max_concurrent_requests = 3
   timeout_per_url = 10  # seconds
   ```

### **Cost Optimization**

1. **Auto-Sleep Configuration**
   ```bash
   # HuggingFace Spaces auto-sleep after 1 hour idle
   # No additional configuration needed
   # Automatically resumes on next request
   ```

2. **Hardware Scheduling**
   ```bash
   # Strategy: Start with CPU Basic
   # Upgrade to T4 Small during processing
   # Downgrade back to CPU Basic when idle
   ```

3. **API Cost Management**
   ```bash
   # Perplexity API optimization:
   - Cache search results for similar queries
   - Use more specific search terms
   - Implement request batching
   - Set reasonable max_sources limits
   ```

---

## 🎓 Best Practices

### **Security Best Practices**

1. **API Key Management**
   ```bash
   ✅ Store in HuggingFace Spaces secrets
   ✅ Never commit to git repositories
   ✅ Rotate keys periodically
   ✅ Monitor usage for anomalies
   ```

2. **Safe Scraping**
   ```bash
   ✅ Respect robots.txt
   ✅ Implement rate limiting
   ✅ Use appropriate user agents
   ✅ Avoid private/internal networks
   ```

3. **Data Privacy**
   ```bash
   ✅ No persistent data storage by default
   ✅ Clear temporary files after processing
   ✅ Respect copyright and fair use
   ✅ Provide clear data source attribution
   ```

### **Development Best Practices**

1. **Testing Strategy**
   ```bash
   # Test with small datasets first
   # Verify each step of the pipeline
   # Use diverse source types
   # Test error conditions
   ```

2. **Version Control**
   ```bash
   # Use git for file management
   # Tag stable releases
   # Document changes and updates
   # Keep rollback capability
   ```

3. **Documentation**
   ```bash
   # Keep README.md updated
   # Document custom configurations
   # Provide usage examples
   # Include troubleshooting guides
   ```

---

## 🆘 Getting Help

### **Support Channels**

1. **HuggingFace Community**
   - Discussions: Share issues and solutions
   - Discord: Real-time help from community

2. **GitHub Issues**
   - Bug reports and feature requests
   - Include logs and configuration details

3. **Documentation**
   - README.md: Complete usage guide
   - DEPLOYMENT.md: This guide
   - Code comments: Inline documentation

### **Information to Include When Asking for Help**

```bash
1. Deployment type (CPU Basic, T4 Small, etc.)
2. Error messages (exact text)
3. Space logs (relevant sections)
4. Configuration details (without API keys)
5. Steps to reproduce the issue
6. Expected vs actual behavior
```

---

## 🎉 Success Indicators

Your deployment is successful when you see:

```bash
✅ Space builds without errors
✅ Interface loads within 60 seconds
✅ Perplexity AI discovery works
✅ Can create projects and scrape URLs
✅ AI processing generates quality data
✅ Export produces valid dataset files
✅ No persistent errors in logs
```

---

## 🚀 What's Next?

After successful deployment:

1. **Create Your First Dataset**
   - Start with a simple sentiment analysis project
   - Use AI discovery to find sources
   - Process and export a small dataset

2. **Explore Advanced Features**
   - Try different templates
   - Experiment with search types
   - Test batch processing

3. **Optimize for Your Use Case**
   - Adjust configurations
   - Create custom templates
   - Integrate with your ML pipeline

4. **Share and Collaborate**
   - Make Space public to help others
   - Contribute improvements
   - Share success stories

**Your AI Dataset Studio is now ready to revolutionize how you create training datasets!** 🎯

*From idea to ML-ready dataset in minutes, not weeks.*