🚀 AI Dataset Studio - Complete Deployment Guide

Deploy your AI-powered dataset creation platform with Perplexity integration

📋 Pre-Deployment Checklist

✅ Required Files

Ensure you have all these files ready:

ai-dataset-studio/
├── app.py                    # Main application with Perplexity integration
├── perplexity_client.py     # Perplexity AI client
├── config.py               # Configuration management
├── requirements.txt        # Dependencies
├── README.md              # Documentation
├── DEPLOYMENT.md          # This guide
└── utils.py               # Utility functions (optional)

✅ API Keys & Environment

Perplexity API Key - Get from Perplexity AI
HuggingFace Account - For Space hosting
Optional: HuggingFace Token for private datasets

🎯 Deployment Options

Option 1: Full AI-Powered Deployment (Recommended)

Best for: Professional use, maximum features

Hardware: T4 Small ($0.60/hour)

✅ GPU acceleration for AI models
✅ Fast processing (5-15s per article)
✅ All Perplexity features enabled
✅ Production-ready performance

Step-by-Step:

Create HuggingFace Space

# Go to: https://huggingface.co/new-space
Space Name: ai-dataset-studio
SDK: Gradio
Hardware: T4 Small
Visibility: Public (or Private)

Upload Files
- Copy all files from artifacts above
- Ensure app.py is the main file
- Keep file structure intact

Set Environment Variables

# In Space Settings → Repository secrets:
PERPLEXITY_API_KEY = your_perplexity_api_key_here

# Optional:
HF_TOKEN = your_huggingface_token
LOG_LEVEL = INFO
DEBUG = false

Deploy & Test
- Space will build automatically (2-3 minutes)
- Test Perplexity integration first
- Verify all templates work

Option 2: Budget-Friendly Deployment

Best for: Testing, learning, cost-conscious users

Hardware: CPU Basic (Free)

⚡ Basic functionality available
⚠️ Slower AI processing (30-60s per article)
✅ Perplexity discovery still works
✅ Perfect for getting started

Step-by-Step:

Create Space with CPU Basic

Space Name: ai-dataset-studio
SDK: Gradio
Hardware: CPU Basic (Free)

Upload Core Files

# Essential files only:
app.py
perplexity_client.py
requirements.txt
README.md
config.py

Set API Key
```
PERPLEXITY_API_KEY = your_api_key
```
Gradual Upgrade Path
- Start with CPU Basic
- Test functionality
- Upgrade to T4 Small when ready

Option 3: Enterprise Deployment

Best for: High-volume usage, team collaboration

Hardware: A10G Small ($1.05/hour)

🚀 Maximum performance (3-8s per article)
💪 Handle large batch processing
🔄 Support multiple concurrent users
📈 Production-scale capabilities

Additional Setup:

Persistent Storage

# In Space settings:
Storage: Small Persistent ($5/month)
# Enables data persistence between restarts

Advanced Configuration

# Environment variables:
MAX_SOURCES_PER_SEARCH = 50
BATCH_SIZE = 16
ENABLE_CACHING = true
CONCURRENT_REQUESTS = 5

Monitoring Setup

# Enable detailed logging:
LOG_LEVEL = DEBUG
ENABLE_METRICS = true

🔧 Configuration Details

Perplexity API Setup

Get API Key

# Visit: https://www.perplexity.ai/
# Sign up for account
# Navigate to API section
# Generate new API key
# Copy key for environment setup

Test API Key

# Quick test script:
import requests

headers = {
    'Authorization': 'Bearer YOUR_API_KEY',
    'Content-Type': 'application/json'
}

response = requests.post(
    'https://api.perplexity.ai/chat/completions',
    headers=headers,
    json={
        "model": "llama-3.1-sonar-large-128k-online",
        "messages": [{"role": "user", "content": "Test message"}]
    }
)

print("API Status:", response.status_code)

Hardware Requirements by Use Case

Use Case	Hardware	Monthly Cost	Performance	Best For
Learning	CPU Basic	Free	Basic	Students, hobbyists
Development	CPU Upgrade	$22	Good	Developers, testing
Production	T4 Small	$432	Excellent	Businesses, researchers
Enterprise	A10G Small	$756	Maximum	High-volume, teams

Memory & Storage Planning

# Model Memory Usage:
BART Summarization: ~1.5GB
RoBERTa Sentiment: ~500MB
BERT NER: ~400MB
Base Application: ~200MB
Total GPU Memory: ~2.5GB (T4 Small = 16GB, plenty of headroom)

# Storage Usage:
Application Files: ~50MB
Model Cache: ~2GB
Temporary Data: ~100MB per project
Persistent Storage: Optional, recommended for large projects

🧪 Testing Your Deployment

Basic Functionality Test

Launch Application

# Your Space URL: https://huggingface.co/spaces/YOUR_USERNAME/ai-dataset-studio
# Wait for "Running" status
# Interface should load within 30-60 seconds

Test Project Creation

Project Name: "Test Sentiment Analysis"
Template: Sentiment Analysis
Description: "Testing the deployment"
Click: "Create Project"
Expected: "✅ Project created successfully"

Test Perplexity Integration

AI Search Description: "Product reviews for sentiment analysis"
Search Type: General
Max Sources: 10
Click: "Discover Sources with AI"
Expected: List of relevant URLs with quality scores

Advanced Testing

Test Complete Workflow

# Use discovered sources from step 3
Click: "Use These Sources"
Click: "Start Scraping"
Wait: Processing to complete
Click: "Process Data"
Select: Same template as project
Click: "Export Dataset"
Format: JSON
Expected: Downloadable dataset file

Performance Benchmarks

# Timing expectations:
AI Source Discovery: 5-15 seconds
Scraping 10 URLs: 30-120 seconds
Processing Data: 30-180 seconds (depends on hardware)
Export: 5-10 seconds

🚨 Troubleshooting

Common Issues & Solutions

❌ "Perplexity API key not found"

# Problem: Environment variable not set
# Solution:
1. Go to Space Settings → Repository secrets
2. Add: PERPLEXITY_API_KEY = your_key_here
3. Restart Space
4. Check logs for "✅ Perplexity AI client initialized"

❌ "No sources found" from AI discovery

# Problem: Search query too specific or API limits
# Solutions:
1. Make description more general
2. Try different search types
3. Check API key has sufficient credits
4. Use manual URL entry as fallback

❌ "Model loading failed"

# Problem: Insufficient memory or network issues
# Solutions:
1. Upgrade to T4 Small for GPU memory
2. Wait 2-3 minutes for model downloads
3. Check Space logs for specific errors
4. Restart Space if persistent

❌ "Scraping failed" for multiple URLs

# Problem: Rate limiting or blocked access
# Solutions:
1. Reduce concurrent requests
2. Check robots.txt compliance
3. Use more diverse sources
4. Verify URLs are publicly accessible

Debug Mode

Enable detailed logging for troubleshooting:

# Environment variables:
DEBUG = true
LOG_LEVEL = DEBUG

# Then check Space logs for detailed information

Health Check Script

# Add this to test basic functionality:
def health_check():
    """Test all components"""
    
    # Test imports
    try:
        import gradio
        print("✅ Gradio imported")
    except ImportError:
        print("❌ Gradio import failed")
    
    # Test Perplexity
    try:
        from perplexity_client import PerplexityClient
        client = PerplexityClient()
        if client._validate_api_key():
            print("✅ Perplexity API key valid")
        else:
            print("❌ Perplexity API key invalid")
    except Exception as e:
        print(f"❌ Perplexity error: {e}")
    
    # Test models
    try:
        from transformers import pipeline
        print("✅ Transformers available")
    except ImportError:
        print("⚠️ Transformers not available (CPU fallback)")

# Run health check in your Space

🔄 Maintenance & Updates

Regular Maintenance Tasks

Monitor API Usage

# Check Perplexity dashboard for:
- API calls remaining
- Rate limit status
- Billing usage

Update Dependencies

# Periodically update requirements.txt:
gradio>=4.44.0  # Check for latest version
transformers>=4.30.0
# Test thoroughly after updates

Performance Monitoring

# Monitor Space metrics:
- CPU/GPU usage
- Memory consumption
- Request response times
- Error rates

Backup Strategy

# Important data to backup:
1. Configuration files (app.py, config.py)
2. Custom templates or modifications
3. API keys and environment variables
4. Any persistent data or datasets

# HuggingFace Spaces automatically versions your files
# Use git commands to manage versions

📈 Scaling & Optimization

Performance Optimization

Model Optimization

# In config.py, adjust for your needs:
batch_size = 16  # Increase for better GPU utilization
max_sequence_length = 256  # Reduce for faster processing
confidence_threshold = 0.8  # Higher for better quality

Caching Strategy

# Enable model caching:
cache_models = True
model_cache_dir = "./model_cache"

# Cache API responses:
cache_api_responses = True
cache_ttl_hours = 24

Resource Management

# Optimize memory usage:
clear_cache_after_processing = True
max_concurrent_requests = 3
timeout_per_url = 10  # seconds

Cost Optimization

Auto-Sleep Configuration

# HuggingFace Spaces auto-sleep after 1 hour idle
# No additional configuration needed
# Automatically resumes on next request

Hardware Scheduling

# Strategy: Start with CPU Basic
# Upgrade to T4 Small during processing
# Downgrade back to CPU Basic when idle

API Cost Management

# Perplexity API optimization:
- Cache search results for similar queries
- Use more specific search terms
- Implement request batching
- Set reasonable max_sources limits

🎓 Best Practices

Security Best Practices

API Key Management

✅ Store in HuggingFace Spaces secrets
✅ Never commit to git repositories
✅ Rotate keys periodically
✅ Monitor usage for anomalies

Safe Scraping

✅ Respect robots.txt
✅ Implement rate limiting
✅ Use appropriate user agents
✅ Avoid private/internal networks

Data Privacy

✅ No persistent data storage by default
✅ Clear temporary files after processing
✅ Respect copyright and fair use
✅ Provide clear data source attribution

Development Best Practices

Testing Strategy

# Test with small datasets first
# Verify each step of the pipeline
# Use diverse source types
# Test error conditions

Version Control

# Use git for file management
# Tag stable releases
# Document changes and updates
# Keep rollback capability

Documentation

# Keep README.md updated
# Document custom configurations
# Provide usage examples
# Include troubleshooting guides

🆘 Getting Help

Support Channels

HuggingFace Community
- Discussions: Share issues and solutions
- Discord: Real-time help from community
GitHub Issues
- Bug reports and feature requests
- Include logs and configuration details
Documentation
- README.md: Complete usage guide
- DEPLOYMENT.md: This guide
- Code comments: Inline documentation

Information to Include When Asking for Help

1. Deployment type (CPU Basic, T4 Small, etc.)
2. Error messages (exact text)
3. Space logs (relevant sections)
4. Configuration details (without API keys)
5. Steps to reproduce the issue
6. Expected vs actual behavior

🎉 Success Indicators

Your deployment is successful when you see:

✅ Space builds without errors
✅ Interface loads within 60 seconds
✅ Perplexity AI discovery works
✅ Can create projects and scrape URLs
✅ AI processing generates quality data
✅ Export produces valid dataset files
✅ No persistent errors in logs

🚀 What's Next?

After successful deployment:

Create Your First Dataset
- Start with a simple sentiment analysis project
- Use AI discovery to find sources
- Process and export a small dataset
Explore Advanced Features
- Try different templates
- Experiment with search types
- Test batch processing
Optimize for Your Use Case
- Adjust configurations
- Create custom templates
- Integrate with your ML pipeline
Share and Collaborate
- Make Space public to help others
- Contribute improvements
- Share success stories

Your AI Dataset Studio is now ready to revolutionize how you create training datasets! 🎯

From idea to ML-ready dataset in minutes, not weeks.