quantumtaskai-caprover/CAPROVER_SCALING_OPTIMIZATION.md

CapRover Auto-scaling and Resource Optimization

1. Container Resource Limits

Update captain-definition for Resource Management

{
  "schemaVersion": 2,
  "dockerfilePath": "./Dockerfile.captain",
  "containerHttpPort": 80,
  "resources": {
    "memory": "512m",
    "memoryReservation": "256m",
    "cpu": 0.5,
    "cpuReservation": 0.25
  },
  "healthcheck": {
    "test": ["CMD", "python", "manage.py", "check"],
    "interval": "30s",
    "timeout": "10s",
    "retries": 3,
    "startPeriod": "40s"
  }
}

CapRover App Configuration

1. CapRover Dashboard → Apps → quantumtaskai → App Configs
2. Resources section:
   • Memory Limit: 512MB
   • Memory Reservation: 256MB
   • CPU Limit: 0.5 (50% of one CPU core)
   • CPU Reservation: 0.25 (25% guaranteed)

2. Horizontal Scaling (Multiple Instances)

Load Balancing Setup

1. App Configs → Enable "Load Balancer"
2. Set "Instance Count" to 2-3 instances
3. Configure "Health Check Path" to /health/

Session Affinity (Important for Django)

Since you're using Redis for sessions, sticky sessions aren't needed:

• Disable session affinity
• Enable Redis session storage (already configured)
• Sessions persist across all instances

Database Connection Pooling

Update database settings for multiple instances:

# In settings.py
if config('CAPROVER_GIT_COMMIT_SHA', default=''):
    # CapRover environment - optimize for multiple instances
    DATABASES['default']['CONN_MAX_AGE'] = 300  # Shorter connection lifetime
    DATABASES['default']['OPTIONS']['MAX_CONNS'] = 10  # Fewer connections per instance

3. Vertical Scaling (Resource Monitoring)

Memory Optimization

# Add to Django settings
if not DEBUG:
    # Production memory optimizations
    MIDDLEWARE = [
        'django.middleware.gzip.GZipMiddleware',  # Compress responses
    ] + MIDDLEWARE
    
    # Enable template caching
    TEMPLATES[0]['OPTIONS']['loaders'] = [
        ('django.template.loaders.cached.Loader', [
            'django.template.loaders.filesystem.Loader',
            'django.template.loaders.app_directories.Loader',
        ]),
    ]

Database Query Optimization

# Add to apps/core/middleware.py
class DatabaseOptimizationMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        from django.db import connection, reset_queries
        
        # Reset queries for this request
        reset_queries()
        
        response = self.get_response(request)
        
        # Log slow or numerous queries in production
        if not settings.DEBUG and len(connection.queries) > 10:
            logger.warning(f"High query count: {len(connection.queries)} queries for {request.path}")
        
        return response

4. Auto-scaling Triggers

CPU-based Scaling

# Monitor CPU usage
docker stats [container-id]

# Scale up when CPU > 70% for 5 minutes
# Scale down when CPU < 30% for 10 minutes

Memory-based Scaling

# Monitor memory usage  
docker exec [container-id] free -m

# Scale up when memory > 80% for 5 minutes
# Scale down when memory < 40% for 10 minutes

Custom Metrics Scaling

Monitor application-specific metrics:

• Active user sessions (Redis keys)
• Agent execution queue length
• Database connection pool usage
• Response time averages

5. Performance Monitoring Scripts

Create Monitoring Script

#!/bin/bash
# monitoring.sh

CONTAINER_ID=$(docker ps | grep quantumtaskai | awk '{print $1}')

# Get resource usage
CPU_USAGE=$(docker stats --no-stream $CONTAINER_ID | tail -1 | awk '{print $3}' | sed 's/%//')
MEM_USAGE=$(docker stats --no-stream $CONTAINER_ID | tail -1 | awk '{print $4}' | sed 's/%//')

echo "CPU Usage: $CPU_USAGE%"
echo "Memory Usage: $MEM_USAGE%"

# Alert if high usage
if (( $(echo "$CPU_USAGE > 80" | bc -l) )); then
    echo "HIGH CPU ALERT: $CPU_USAGE%"
    # Send notification (email, webhook, etc.)
fi

if (( $(echo "$MEM_USAGE > 85" | bc -l) )); then
    echo "HIGH MEMORY ALERT: $MEM_USAGE%"
    # Send notification (email, webhook, etc.)
fi

# Check application health
HEALTH_CHECK=$(curl -s -o /dev/null -w "%{http_code}" https://quantumtaskai.captain.your-domain.com/health/)
if [ "$HEALTH_CHECK" != "200" ]; then
    echo "APPLICATION HEALTH ALERT: HTTP $HEALTH_CHECK"
    # Send notification
fi

Cron Job for Monitoring

# Add to crontab
*/5 * * * * /path/to/monitoring.sh >> /var/log/quantum-monitor.log 2>&1

6. Caching Strategy for Scale

Multi-layer Caching

# In views.py - example caching strategy
from django.core.cache import cache
from django.views.decorators.cache import cache_page
from django.utils.decorators import method_decorator

@method_decorator(cache_page(60 * 5), name='dispatch')  # 5 minutes
class AgentListView(ListView):
    model = Agent
    
    def get_queryset(self):
        cache_key = f"agents_list_{self.request.user.id}"
        queryset = cache.get(cache_key)
        
        if queryset is None:
            queryset = Agent.objects.select_related().prefetch_related('category')
            cache.set(cache_key, queryset, 60 * 10)  # 10 minutes
        
        return queryset

Redis Cluster for High Availability (Advanced)

For very high load, consider Redis Cluster:

1. Deploy multiple Redis instances
2. Configure Redis Cluster
3. Update Django Redis settings for cluster mode

7. Load Testing and Optimization

Load Testing Tools

# Install Apache Bench
sudo apt-get install apache2-utils

# Test with concurrent users
ab -n 1000 -c 10 https://quantumtaskai.captain.your-domain.com/

# Test specific endpoints
ab -n 500 -c 5 https://quantumtaskai.captain.your-domain.com/agents/

# Load test with POST data
ab -n 100 -c 5 -p post_data.json -T application/json https://quantumtaskai.captain.your-domain.com/agents/api/execute/

Performance Benchmarks

Target performance metrics:

• Response time: < 200ms for cached pages
• Database queries: < 50ms per query
• Memory usage: < 400MB per instance
• CPU usage: < 60% average
• Concurrent users: 100+ simultaneous users

8. Auto-scaling Scripts

Simple Auto-scaler Script

#!/bin/bash
# auto-scaler.sh

APP_NAME="quantumtaskai"
MIN_INSTANCES=1
MAX_INSTANCES=5
CPU_THRESHOLD_UP=70
CPU_THRESHOLD_DOWN=30

# Get current instance count
CURRENT_INSTANCES=$(docker ps | grep $APP_NAME | wc -l)

# Get average CPU usage
AVG_CPU=$(docker stats --no-stream $(docker ps -q --filter name=$APP_NAME) | awk 'NR>1 {sum += $3; count++} END {print sum/count}' | sed 's/%//')

echo "Current instances: $CURRENT_INSTANCES"
echo "Average CPU: $AVG_CPU%"

# Scale up logic
if (( $(echo "$AVG_CPU > $CPU_THRESHOLD_UP" | bc -l) )) && [ $CURRENT_INSTANCES -lt $MAX_INSTANCES ]; then
    echo "Scaling UP: CPU at $AVG_CPU%"
    # Implement scaling up logic (CapRover API call)
    curl -X POST https://captain.your-domain.com/api/v2/user/apps/appData/quantumtaskai \
         -H "x-captain-auth: $CAPTAIN_TOKEN" \
         -d '{"instanceCount": '$((CURRENT_INSTANCES + 1))'}'
fi

# Scale down logic  
if (( $(echo "$AVG_CPU < $CPU_THRESHOLD_DOWN" | bc -l) )) && [ $CURRENT_INSTANCES -gt $MIN_INSTANCES ]; then
    echo "Scaling DOWN: CPU at $AVG_CPU%"
    # Implement scaling down logic (CapRover API call)
    curl -X POST https://captain.your-domain.com/api/v2/user/apps/appData/quantumtaskai \
         -H "x-captain-auth: $CAPTAIN_TOKEN" \
         -d '{"instanceCount": '$((CURRENT_INSTANCES - 1))'}'
fi

9. Cost Optimization

Resource Right-sizing

• Start small: 512MB RAM, 0.5 CPU
• Monitor usage: Scale up only when needed
• Regular reviews: Monthly resource usage analysis

Efficient Resource Usage

• Shared services: Use shared PostgreSQL and Redis
• Image optimization: Multi-stage Docker builds
• Caching: Reduce database load with strategic caching
• Compression: Enable gzip compression

Schedule-based Scaling

# Scale up during peak hours (9 AM - 6 PM)
0 9 * * 1-5 /scripts/scale-up.sh

# Scale down during off-hours  
0 18 * * 1-5 /scripts/scale-down.sh
0 0 * * 6-7 /scripts/scale-down.sh

This comprehensive scaling strategy ensures your Quantum Tasks AI application can handle varying loads efficiently while maintaining cost-effectiveness.