amit/quantum-ai-v3
1
0
Fork 0
mirror of https://github.com/thecyberlearn/quantum-ai-v3.git synced 2026-08-18 13:12:57 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

📚 Organize documentation and create comprehensive domain change guide

Browse Source 
- Create organized /docs/ directory structure with clear navigation
- Add comprehensive domain change guide with step-by-step instructions
- Consolidate Railway deployment documentation
- Create complete environment variables reference
- Add development setup guide and testing procedures
- Create troubleshooting guide and database management docs
- Remove 18+ redundant/outdated documentation files
- Update CLAUDE.md with new documentation structure

New documentation structure:
- docs/deployment/ - Railway, domain changes, environment setup
- docs/development/ - Local setup, agent creation, testing
- docs/operations/ - Database, troubleshooting, maintenance

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Claude 2025-07-27 09:10:35 +05:30
parent edc3c27e4e
commit 314d11349c
26 changed files with 3040 additions and 3126 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

85
ALLOWED_HOSTS_FIX.md
View File

@ -1,85 +0,0 @@
# 🎉 Success! Django is Running - Quick ALLOWED_HOSTS Fix
## Great News!
- ✅ **Django is starting successfully** (no more "service unavailable")
- ✅ **Gunicorn is running** and responding to requests
- ❌ **400 Bad Request** = Django rejecting health check due to ALLOWED_HOSTS
## 🔧 Quick Fix - Update ALLOWED_HOSTS
### Step 1: Get Your Railway Domain
Check your Railway project dashboard or URL bar for the exact domain, it looks like:
```
your-project-name-production-1234.up.railway.app
```
### Step 2: Set ALLOWED_HOSTS in Railway Variables
Go to Railway → Variables → Add/Update:
**Option A: Specific Domain**
```bash
ALLOWED_HOSTS=your-exact-railway-domain.railway.app,quantumtaskai.com
```
**Option B: Railway Wildcard (Recommended)**
```bash
ALLOWED_HOSTS=*.railway.app,quantumtaskai.com,localhost
```
**Option C: Debug Mode (Temporary)**
```bash
ALLOWED_HOSTS=*
```
### Step 3: Railway Will Auto-Redeploy
- Railway automatically redeploys when environment variables change
- Wait 30-60 seconds for redeploy
- Health check should then pass
## 🎯 Expected Results
### Before Fix (Current):
```html
<!doctype html>
<html lang="en">
<head><title>Bad Request (400)</title></head>
<body><h1>Bad Request (400)</h1><p></p></body>
</html>
```
### After Fix:
```json
{
"status": "healthy",
"app": "quantum-tasks-ai",
"checks": {
"application": {"status": "healthy", "django_ready": true},
"environment": {"status": "healthy", "secret_key_configured": true}
}
}
```
## 🚀 Next Steps After Fix
1. **Verify health check passes**: Should return JSON instead of HTML
2. **Test application access**: Visit Railway domain in browser
3. **Setup database**: Run `railway run python manage.py setup_database`
4. **Test agents**: All 6 AI agents should be available
## 🔍 Debug Commands
```bash
# Check your exact Railway domain
railway status
# Check current environment variables
railway variables
# Test health endpoint
curl https://your-railway-domain/health/
# View logs
railway logs --tail 20
```
The hard part is done - Django is running! This is just a configuration fix. 🎯

13
CLAUDE.md
View File

@ -2,6 +2,19 @@
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## 📚 Documentation
**Complete documentation is now organized in the `/docs/` directory:**
- **📖 Main Index:** [docs/README.md](./docs/README.md)
- **🚀 Deployment:** [docs/deployment/](./docs/deployment/) - Railway deployment, domain changes, environment setup
- **🛠️ Development:** [docs/development/](./docs/development/) - Local setup, agent creation, testing
- **⚙️ Operations:** [docs/operations/](./docs/operations/) - Database management, troubleshooting, maintenance
**Quick Links:**
- [Domain Change Guide](./docs/deployment/domain-change-guide.md) - Complete domain change instructions
- [Railway Deployment](./docs/deployment/railway-deployment.md) - Production deployment guide
- [Environment Variables](./docs/deployment/environment-variables.md) - Complete environment reference
## Project Overview
Quantum Tasks AI is a Django-based AI agent marketplace platform where users can purchase and interact with specialized AI agents. The system supports both webhook-based and API-based agents with integrated payment processing via Stripe.

355
CONSERVATIVE_IMPROVEMENT_PLAN.md
View File

@ -1,355 +0,0 @@
# NetCop Hub - Conservative Improvement Plan
*Risk-Averse Approach to System Enhancement*
**Philosophy: "Observe First, Change Never (Until Proven Safe)"**
## The Core Problem
Based on your experience where "whenever we try to improve something we break most things," this plan prioritizes **system stability** over technical perfection. The goal is to enhance observability and gradually improve the system without disrupting existing functionality.
## Why Traditional Improvement Fails
1. **Hidden Dependencies**: The 383 print statements aren't just debug code - they might be critical for operations
2. **Undocumented Workarounds**: Database constraint hacks and exception handling serve unknown purposes
3. **Integration Complexity**: Agent processors, payment flows, and user systems are tightly coupled
4. **No Safety Net**: Lack of comprehensive tests makes changes high-risk
## Conservative Approach Principles
### 🛡️ **Safety First Rules**
1. **Never remove existing code** until replacement is proven for months
2. **Always add alongside**, never replace directly
3. **One tiny change at a time** with weeks of validation
4. **Immediate rollback capability** for every change
5. **Production behavior is always correct** (even if it looks wrong)
---
## Phase 0: Observe & Document (4-6 weeks)
*No code changes, pure observation*
### Week 1-2: System Archaeology
#### Document Current Behavior
```bash
# Create comprehensive system documentation
mkdir -p docs/current-system/
mkdir -p docs/observations/
mkdir -p docs/dependencies/
```
**Tasks:**
1. **Map All Print Statements** - Document what each print statement actually does
2. **Trace User Journeys** - Document complete user flows from signup to agent usage
3. **Payment Flow Documentation** - Every step of Stripe integration
4. **Agent Processing Flows** - How each agent type actually works in production
#### Dependency Mapping
```bash
# Document file interdependencies
docs/dependencies/
├── user-model-dependencies.md
├── agent-processor-relationships.md
├── payment-integration-points.md
└── database-constraint-analysis.md
```
### Week 3-4: Behavior Analysis
#### Create System Behavior Baseline
1. **Database Query Patterns** - What queries run most frequently
2. **Error Patterns** - What errors actually occur and how they're handled
3. **Performance Baselines** - Current response times and resource usage
4. **User Interaction Patterns** - How users actually use the system
#### Critical Path Identification
- Which code paths are absolutely critical
- Which "hacks" are actually essential workarounds
- What would break if specific components failed
### Week 5-6: Test Strategy Development
#### Create Test Plan Without Breaking Anything
```python
# tests/current_behavior/
# Test what the system ACTUALLY does, not what it should do
def test_user_deduction_with_constraint_hack():
"""Test that the current database constraint workaround works"""
# This test validates the existing "hack" is working
pass
def test_print_statements_capture_essential_info():
"""Verify that print statements contain needed information"""
# Don't remove prints - understand their purpose first
pass
```
---
## Phase 1: Add Observability (6-8 weeks)
*Additive only - no removals or changes*
### Week 1-2: Logging Infrastructure
#### Add Logging Alongside Existing Prints
```python
# utils/safe_logging.py
import logging
class ConservativeLogger:
def __init__(self, agent_slug):
self.logger = logging.getLogger(f'netcop.{agent_slug}')
self.agent_slug = agent_slug
def log_alongside_print(self, message, level=logging.INFO):
"""Log to both print (existing) and logger (new)"""
print(f"{self.agent_slug}: {message}") # Keep existing print
self.logger.log(level, message, extra={'agent_slug': self.agent_slug})
```
**Implementation Strategy:**
- Add logging infrastructure WITHOUT changing existing prints
- Both systems run in parallel for months
- Only remove prints after new logging is proven reliable
### Week 3-4: Monitoring System
#### Add System Monitoring (Non-Intrusive)
```python
# monitoring/system_observer.py
class SystemObserver:
"""Monitor system behavior without changing it"""
def observe_agent_processing(self):
"""Monitor agent processing without interfering"""
# Count requests, measure timing, track errors
# But don't change any processing logic
pass
def observe_payment_flows(self):
"""Monitor payment processing passively"""
# Track Stripe interactions, wallet changes
# But maintain all existing payment logic
pass
```
### Week 5-6: Staging Environment
#### Create Production-Identical Staging
```bash
# Exact copy of production environment
# Same database constraints, same "hacks", same everything
# Use for testing ANY future changes
```
### Week 7-8: Feature Flag System
#### Add Feature Flags (Zero Impact)
```python
# utils/feature_flags.py
class SafeFeatureFlags:
def __init__(self):
self.flags = {}
def is_enabled(self, flag_name, default=False):
"""Always return default unless explicitly enabled"""
return self.flags.get(flag_name, default)
def enable_for_testing(self, flag_name):
"""Enable only in staging environment"""
if settings.ENVIRONMENT == 'staging':
self.flags[flag_name] = True
```
---
## Phase 2: Gradual, Reversible Changes (3-6 months)
*One tiny change every 2-4 weeks*
### The "One Change Rule"
- **Only one component changes at a time**
- **Minimum 2 weeks of staging validation**
- **Minimum 2 weeks of production monitoring**
- **Immediate rollback if anything seems wrong**
### Month 1: Enhance Existing Error Handling
#### Add Better Error Handling Alongside Current System
```python
# Instead of replacing the "hack" in User.deduct_balance:
def deduct_balance_enhanced(self, amount, description="", agent_slug=""):
"""Enhanced version that runs alongside existing method"""
# Run existing method first (the "hack" that works)
result = self.deduct_balance_original(amount, description, agent_slug)
# Add enhanced error handling for future
if feature_flags.is_enabled('enhanced_error_handling'):
# New error handling logic here
pass
return result
```
### Month 2: Improve Database Queries (Additive)
#### Add Query Optimization Without Changing Existing Queries
```python
# agent_base/views_enhanced.py
def marketplace_view_optimized(request):
"""Optimized marketplace view that runs alongside existing"""
if feature_flags.is_enabled('optimized_marketplace'):
# Use optimized queries
return optimized_marketplace_logic(request)
else:
# Fall back to existing view (that works)
return marketplace_view_original(request)
```
### Month 3: Enhanced Agent Processing
#### Add Retry Logic Without Changing Core Processing
```python
# agent_base/processors_enhanced.py
class EnhancedAgentProcessor:
def __init__(self, original_processor):
self.original = original_processor # Keep original working processor
def process_request_with_retry(self, **kwargs):
"""Enhanced processing with retry, fallback to original"""
if feature_flags.is_enabled('agent_retry_logic'):
try:
return self.process_with_retry(**kwargs)
except Exception:
# If enhanced version fails, use original
return self.original.process_request(**kwargs)
else:
# Use original processor that we know works
return self.original.process_request(**kwargs)
```
---
## Risk Mitigation Strategies
### 1. Rollback Plan for Every Change
```bash
# Every change must have immediate rollback capability
git tag before-change-YYYY-MM-DD
# Implement change with feature flag OFF by default
# Enable feature flag only in staging
# If anything breaks, disable feature flag immediately
```
### 2. Canary Deployment
```python
# Roll out changes to tiny percentage of users first
def should_use_enhanced_feature(user):
if settings.ENVIRONMENT == 'staging':
return True
elif user.id % 100 == 0: # 1% of users
return feature_flags.is_enabled('canary_enhanced_feature')
else:
return False
```
### 3. Monitoring Alerts
```python
# Alert on ANY deviation from baseline behavior
class ConservativeMonitoring:
def alert_on_change(self, metric_name, current_value, baseline_value):
deviation = abs(current_value - baseline_value) / baseline_value
if deviation > 0.02: # 2% change triggers alert
send_alert(f"{metric_name} changed by {deviation*100:.1f}%")
```
---
## Success Metrics
### Phase 0 Success Criteria
- [ ] Complete system documentation created
- [ ] All dependencies mapped and understood
- [ ] Test strategy covers 100% of critical paths
- [ ] Zero production issues during observation period
### Phase 1 Success Criteria
- [ ] Logging system running parallel to prints for 3+ months
- [ ] Monitoring captures all system behavior
- [ ] Staging environment perfectly mirrors production
- [ ] Feature flag system ready for safe deployments
### Phase 2 Success Criteria
- [ ] Each change validated for minimum 1 month before next change
- [ ] Zero production incidents from improvements
- [ ] Rollback capability tested and verified
- [ ] Enhanced functionality proves more reliable than original
---
## What NOT to Do
### ❌ Avoid These Common Mistakes
1. **Don't remove print statements** - they might be essential
2. **Don't fix database "hacks"** - they might prevent unknown issues
3. **Don't optimize queries** until you understand why current ones exist
4. **Don't refactor code** until new version is proven for months
5. **Don't assume anything is "obviously wrong"** - it might be intentionally that way
### ❌ Red Flags That Should Stop All Changes
- Any production error increase
- Any response time degradation
- Any user complaints about functionality
- Any payment processing issues
- Any agent processing failures
---
## Emergency Procedures
### If Something Breaks
1. **Immediately disable all feature flags**
2. **Revert to last known good state**
3. **Document what went wrong**
4. **Wait minimum 2 weeks before trying again**
5. **Review and improve safety procedures**
### Rollback Commands
```bash
# Always ready to execute
git revert HEAD --no-edit
# Disable all feature flags
python manage.py disable_all_features
# Restart services
./restart_production.sh
```
---
## Timeline Summary
| Phase | Duration | Risk Level | Changes |
|-------|----------|------------|---------|
| Phase 0 | 4-6 weeks | Zero Risk | Documentation only |
| Phase 1 | 6-8 weeks | Very Low | Additive monitoring |
| Phase 2 | 3-6 months | Low | One tiny change per month |
**Total Timeline: 6-9 months** for meaningful improvements with near-zero risk of breaking existing functionality.
---
## Philosophy Recap
> **"The system that works in production is always correct, even if it looks wrong."**
This approach prioritizes:
1. **System stability** over code elegance
2. **Gradual improvement** over dramatic refactoring
3. **Observation** over assumption
4. **Reversibility** over optimization
5. **Working software** over perfect architecture
The goal is to enhance NetCop Hub **safely and gradually** without the risk of breaking existing functionality that users depend on.

147
DATABASE_MIGRATION_STEPS.md
View File

@ -1,147 +0,0 @@
# 🗄️ Database Migration Steps
## Current Issue
Database is not migrated because we removed migrations from railway.json startup to fix startup issues.
## Step-by-Step Migration Process
### Step 1: Ensure DATABASE_URL is Set
1. Go to Railway project → **Variables**
2. Add if not exists:
```
DATABASE_URL = ${{ Postgres.DATABASE_URL }}
```
3. Wait for Railway to redeploy (30-60 seconds)
### Step 2: Test Database Connection
```bash
# Test if Django can connect to database
railway run python manage.py check
# Check database specifically
railway run python manage.py check --database default
```
### Step 3: Run Migrations Manually
```bash
# Run all pending migrations
railway run python manage.py migrate
# If that fails, try step by step:
railway run python manage.py migrate --run-syncdb
```
### Step 4: Populate Initial Data
```bash
# Create superuser (optional)
railway run python manage.py createsuperuser
# Populate agents
railway run python manage.py populate_agents
```
### Step 5: Use Our Setup Command (Recommended)
```bash
# This does everything automatically with retries
railway run python manage.py setup_database
```
## Expected Output
### Successful Migration:
```
Operations to perform:
Apply all migrations: admin, agent_base, auth, authentication, contenttypes, core, data_analyzer, email_writer, five_whys_analyzer, job_posting_generator, sessions, social_ads_generator, wallet, weather_reporter
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
...
Applying authentication.0004_user_email_verified_emailverificationtoken... OK
```
### Successful Agent Population:
```
Creating default agents...
Updated: Weather Reporter
Updated: Data Analyzer
Updated: Job Posting Generator
Updated: Social Ads Generator
Updated: 5 Whys Analysis Agent
Successfully processed 5 agents: 0 created, 5 updated
```
## Troubleshooting
### Error: "No module named 'psycopg2'"
**Cause**: PostgreSQL driver not installed
**Fix**: Already in requirements.txt, should be available
### Error: "Connection refused"
**Cause**: DATABASE_URL not set or PostgreSQL service not running
**Fix**:
1. Check Railway PostgreSQL service is active
2. Verify DATABASE_URL variable is set
3. Wait a few minutes for services to start
### Error: "relation already exists"
**Cause**: Some tables already exist
**Fix**:
```bash
railway run python manage.py migrate --fake-initial
```
### Error: "permission denied"
**Cause**: Database user doesn't have permissions
**Fix**: Railway PostgreSQL should have full permissions by default
## Quick Fix Commands
### If Migration Fails:
```bash
# Reset migrations (dangerous - only if needed)
railway run python manage.py migrate --fake-initial
# Or try individual apps:
railway run python manage.py migrate auth
railway run python manage.py migrate authentication
railway run python manage.py migrate agent_base
```
### If Agents Don't Populate:
```bash
# Check if command exists
railway run python manage.py help populate_agents
# Run manually
railway run python manage.py shell
# Then in shell:
from agent_base.management.commands.populate_agents import Command
cmd = Command()
cmd.handle()
```
## Verification
### Check Database Tables:
```bash
railway run python manage.py dbshell
# In database shell:
\dt # List all tables
\q # Quit
```
### Test Health Endpoint:
```bash
curl https://your-railway-domain.railway.app/health/
```
Should return database as "healthy" instead of "warning".
## Next Steps After Migration
1. **Add health check back** to railway.json
2. **Test all AI agents** work correctly
3. **Verify user registration** and payments work
4. **Check admin panel** functionality
Run the migrations and let me know what output you get! 🚀

111
EMERGENCY_STARTUP_FIX.md
View File

@ -1,111 +0,0 @@
# 🚨 Emergency Railway Startup Fix
## Issue: Service Unavailable (Django Not Starting)
We've regressed from 400 Bad Request (Django running) back to "service unavailable" (app not starting). This is likely an environment variable issue.
## 🔧 IMMEDIATE FIXES TO TRY
### Fix 1: Minimal Environment Variables
**Set ONLY these in Railway Variables:**
```bash
SECRET_KEY=your-50-character-secret-key
DEBUG=False
ALLOWED_HOSTS=*
```
**Remove all other variables temporarily**
### Fix 2: Check ALLOWED_HOSTS Syntax
**Bad (causes crash):**
```bash
ALLOWED_HOSTS=*.railway.app, quantumtaskai.com # NO SPACES
ALLOWED_HOSTS="*.railway.app,quantumtaskai.com" # NO QUOTES
```
**Good:**
```bash
ALLOWED_HOSTS=*.railway.app,quantumtaskai.com
# OR for debugging:
ALLOWED_HOSTS=*
```
### Fix 3: Generate New SECRET_KEY
**The SECRET_KEY might be invalid:**
```bash
# Generate new one:
python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"
# Set in Railway:
SECRET_KEY=django-insecure-your-new-key-here
```
## 🚀 Deployment Strategy
### Step 1: Minimal railway.json (DONE)
- Removed health check
- Removed collectstatic
- Bare minimum startup
### Step 2: Set Minimal Variables
```bash
SECRET_KEY=your-generated-key
DEBUG=False
ALLOWED_HOSTS=*
```
### Step 3: Deploy and Test
```bash
git add .
git commit -m "Emergency fix - minimal config"
git push origin main
```
### Step 4: Check Direct Access
```bash
# Once deployed, test direct access:
curl https://your-railway-domain.railway.app/
# Should return HTML page, not connection error
```
## 🔍 Debug Commands
### Check Railway Logs
```bash
railway logs --tail 50
```
**Look for these errors:**
- `ImproperlyConfigured: The SECRET_KEY setting must not be empty`
- `ImproperlyConfigured: You must set settings.ALLOWED_HOSTS`
- `ModuleNotFoundError`
- `ImportError`
- `Address already in use`
### Check Variables
```bash
railway variables
```
## 📊 Success Indicators
### ✅ App Starting
- Railway logs show: `Starting gunicorn`
- No Python errors in logs
- Direct URL access works (returns HTML)
### ✅ Ready for Health Check
Once basic startup works, we can add back:
```json
{
"deploy": {
"startCommand": "gunicorn netcop_hub.wsgi:application --bind 0.0.0.0:$PORT",
"healthcheckPath": "/health/",
"healthcheckTimeout": 30
}
}
```
## 🎯 Goal
Get back to where Django was at least starting (even with 400 error), then fix the ALLOWED_HOSTS properly.
**The issue is likely in environment variable syntax or SECRET_KEY format.**

92
HEALTH_CHECK_FIX_SUMMARY.md
View File

@ -1,92 +0,0 @@
# ✅ Health Check Fix Summary
## What We Fixed
### 🔧 **1. Updated railway.json Configuration**
- **Increased health check timeout**: 30s → 60s
- **Added health check interval**: 30s between checks
- **Simplified startup command**: Removed complex migration fallbacks
- **Clean deployment process**: Linear migration → populate agents → start server
### 🏥 **2. Enhanced Health Endpoint**
- **Added database retry logic**: 3 attempts with 0.5s delays
- **Graceful fallback**: Skip agent checks if database unavailable
- **Better error reporting**: Shows attempt counts and specific errors
- **Application status check**: Confirms Django is ready
### 📋 **3. Environment Variables Checklist**
- **Minimum required variables**: SECRET_KEY, DEBUG=False, ALLOWED_HOSTS
- **Clear troubleshooting guide**: Common errors and solutions
- **Quick fix commands**: Generate secret key, test locally
## 🚀 Deploy Instructions
### Step 1: Set Minimum Variables in Railway
```bash
SECRET_KEY=your-50-character-secret-key
DEBUG=False
ALLOWED_HOSTS=your-project.railway.app,quantumtaskai.com
```
### Step 2: Deploy Updated Code
```bash
git add .
git commit -m "Fix health check with improved timeout and retry logic"
git push origin main
railway up
```
### Step 3: Monitor Deployment
- Watch Railway deployment logs
- Health check now has 60 seconds to succeed
- Database connection retries 3 times automatically
- Look for "healthy" status in `/health/` response
## 🎯 Expected Results
### Successful Health Check Response:
```json
{
"status": "healthy",
"checks": {
"database": {"status": "healthy", "attempt": 1},
"agents": {"status": "healthy", "active_count": 6},
"application": {"status": "healthy", "django_ready": true}
}
}
```
### What Changed:
- **Health check timeout**: 60 seconds (was 30)
- **Database retry**: 3 attempts (was 1)
- **Better error handling**: Specific failure reasons
- **Graceful degradation**: App can start even if agents fail to load initially
## 🚨 If Health Check Still Fails
### Debug Commands:
```bash
# Check Railway logs
railway logs --tail 100
# Check specific health endpoint
curl https://your-project.railway.app/health/
# Verify environment variables
railway variables
```
### Common Issues & Solutions:
1. **Database still connecting**: Wait 60-90 seconds, PostgreSQL needs time
2. **Missing SECRET_KEY**: Generate and set in Railway variables
3. **Wrong domain in ALLOWED_HOSTS**: Add Railway domain to variable
4. **Migration errors**: Check logs for specific Django migration issues
## ✅ Success Indicators
- ✅ Health check passes within 60 seconds
- ✅ `/health/` endpoint returns 200 status code
- ✅ Railway deployment shows "Active"
- ✅ Application is accessible at Railway URL
- ✅ 6 AI agents are loaded and ready
**Your enhanced Quantum Tasks AI should now deploy successfully with the improved health check system!** 🎉

354
IMPROVEMENT_SUGGESTIONS.md
View File

@ -1,354 +0,0 @@
# NetCop Hub - Improvement Suggestions
*Analysis Date: 2025-07-24*
*Priority Classification: High (🔴) | Medium (🟡) | Low (🟢)*
## Executive Summary
Based on comprehensive analysis of the NetCop Hub codebase, I've identified 23 specific improvement opportunities across 6 main categories. The application has solid architecture but several areas need attention for production readiness, maintainability, and scalability.
## 🔴 High Priority Improvements
### 1. Logging & Monitoring System
**Current Issues:**
- 383 print statements across 22 files used for debugging
- Inconsistent logging practices mixing print() with proper logging
- Debug information exposed in production endpoints
**Improvements:**
```python
# Replace print statements with proper logging
import logging
logger = logging.getLogger(__name__)
# Instead of:
print(f"{self.agent_slug}: Error processing request: {e}")
# Use:
logger.error(f"{self.agent_slug}: Error processing request: {e}")
```
**Files to Update:**
- `agent_base/processors.py:63` - Replace print with logging
- `wallet/stripe_handler.py` - 77 print statements for Stripe debugging
- `data_analyzer/processor.py` - 14 debugging print statements
- All processor files need logging standardization
**Impact:** Production stability, debugging capability, compliance
### 2. Error Handling & Exception Management
**Current Issues:**
- Generic exception handling in User model (`authentication/models.py:49-55`)
- Inconsistent error responses across processors
- Database constraint errors handled with try/catch hacks
**Critical Fix Needed:**
```python
# Current problematic code in User.deduct_balance:
try:
WalletTransaction.objects.create(**transaction_data)
except Exception as e:
if "NOT NULL constraint failed" in str(e) and "stripe_payment_intent_id" in str(e):
transaction_data['stripe_payment_intent_id'] = ""
WalletTransaction.objects.create(**transaction_data)
```
**Solution:**
- Fix database schema to handle nullable fields properly
- Implement specific exception types
- Add proper error recovery mechanisms
### 3. Security Vulnerabilities
**Issues Found:**
- Debug endpoints exposed in production (`wallet/views.py` - stripe_debug_view)
- Hardcoded sensitive configuration patterns
- File upload security needs strengthening
**Improvements:**
- Remove debug endpoints from production builds
- Implement proper file validation and virus scanning
- Add rate limiting for API endpoints
- Implement proper CORS policies
### 4. Database Performance & Design
**Current Issues:**
- Missing database indexes on frequently queried fields
- N+1 query problems in marketplace view
- Inefficient agent filtering in API endpoint
**Query Optimization Needed:**
```python
# Current inefficient code in agent_base/views.py:20
categories = BaseAgent.objects.filter(is_active=True).values_list('category', 'category').distinct()
# Should use proper aggregation or caching
```
## 🟡 Medium Priority Improvements
### 5. Agent System Architecture
**Current Issues:**
- Lack of agent lifecycle management
- No retry mechanisms for failed webhook calls
- Missing circuit breaker patterns for external APIs
**Improvements:**
- Implement async task queue (Celery) for agent processing
- Add retry logic with exponential backoff
- Implement circuit breaker for external API calls
- Add agent health monitoring
### 6. Configuration Management
**Issues:**
- Environment variables validation is minimal
- Missing configuration for different deployment environments
- No configuration schema validation
**Solution:**
```python
# Implement comprehensive config validation
REQUIRED_ENV_VARS = {
'SECRET_KEY': str,
'STRIPE_SECRET_KEY': str,
'DATABASE_URL': str,
'REDIS_URL': str
}
def validate_environment():
for var, expected_type in REQUIRED_ENV_VARS.items():
value = config(var, default=None)
if not value:
raise ConfigurationError(f"Missing required environment variable: {var}")
```
### 7. Testing Coverage
**Current Issues:**
- Limited test coverage across the application
- No integration tests for payment flows
- Missing API endpoint testing
**Test Suite Needed:**
- Unit tests for all processor classes
- Integration tests for Stripe webhook handling
- API endpoint testing with authentication
- Agent processing end-to-end tests
### 8. API Design & Documentation
**Issues:**
- REST API lacks proper versioning
- No API documentation (OpenAPI/Swagger)
- Inconsistent response formats
- Missing pagination for large datasets
**Improvements:**
- Add API versioning (`/api/v1/`)
- Implement OpenAPI documentation
- Standardize JSON response formats
- Add pagination to agent listings
### 9. Caching Strategy
**Current Issues:**
- Basic Redis caching setup
- No cache invalidation strategy
- Missing cache warming for frequently accessed data
**Improvements:**
- Implement cache invalidation on agent updates
- Add cache warming for marketplace data
- Use cache for expensive agent processing results
- Implement proper cache key strategies
## 🟢 Low Priority Improvements
### 10. Code Organization & Standards
**Issues:**
- Inconsistent import ordering
- Missing type hints throughout codebase
- Some code duplication in processor classes
**Improvements:**
- Add type hints for better IDE support and documentation
- Implement consistent code formatting (Black, isort)
- Extract common functionality into mixins
### 11. Frontend Enhancement
**Issues:**
- Limited JavaScript functionality
- No modern build system for assets
- Missing responsive design improvements
**Suggestions:**
- Implement modern JavaScript build system (Webpack/Vite)
- Add progressive enhancement features
- Improve mobile responsiveness
### 12. Documentation
**Issues:**
- Limited inline code documentation
- Missing architecture decision records
- No deployment guides
**Improvements:**
- Add comprehensive docstrings
- Create API documentation
- Write deployment and maintenance guides
## Implementation Roadmap
### Phase 1: Critical Fixes (2-3 weeks)
1. ✅ Replace all print statements with proper logging
2. ✅ Fix database constraint handling in User model
3. ✅ Remove debug endpoints from production
4. ✅ Add proper error handling throughout application
### Phase 2: Architecture Improvements (4-6 weeks)
1. ✅ Implement async task processing with Celery
2. ✅ Add comprehensive test suite
3. ✅ Optimize database queries and add indexes
4. ✅ Implement proper API versioning
### Phase 3: Enhancement & Optimization (6-8 weeks)
1. ✅ Add monitoring and alerting system
2. ✅ Implement advanced caching strategies
3. ✅ Add comprehensive documentation
4. ✅ Performance optimization and load testing
## Specific Code Changes Required
### 1. Logging Implementation
Create `utils/logging.py`:
```python
import logging
import json
from django.conf import settings
class AgentProcessor:
def __init__(self, agent_slug):
self.logger = logging.getLogger(f'agent.{agent_slug}')
def log_request(self, request_data):
self.logger.info(f"Processing request", extra={
'agent_slug': self.agent_slug,
'request_size': len(str(request_data)),
'user_id': request_data.get('user_id')
})
```
### 2. Database Schema Fixes
Migration needed for WalletTransaction:
```python
# migration file
from django.db import migrations, models
class Migration(migrations.Migration):
operations = [
migrations.AlterField(
model_name='wallettransaction',
name='stripe_payment_intent_id',
field=models.CharField(max_length=200, blank=True, null=True, default=None)
)
]
```
### 3. Error Handling Classes
Create `utils/exceptions.py`:
```python
class AgentProcessingError(Exception):
"""Base exception for agent processing errors"""
pass
class InsufficientFundsError(AgentProcessingError):
"""Raised when user has insufficient wallet balance"""
pass
class ExternalAPIError(AgentProcessingError):
"""Raised when external API calls fail"""
pass
```
## Performance Impact Analysis
### Current Performance Issues
1. **Database Queries**: N+1 queries in marketplace (~50ms per agent)
2. **File Processing**: No async processing for large files
3. **Memory Usage**: Print statements accumulate in production logs
4. **Cache Misses**: No proper cache warming strategy
### Expected Improvements
- **Response Time**: 40-60% improvement with proper caching
- **Memory Usage**: 30% reduction with proper logging
- **Error Recovery**: 90% faster error detection and recovery
- **Scalability**: Support for 10x more concurrent users
## Security Audit Results
### Current Security Score: 7/10
**Strengths:**
- Proper CSRF protection
- Environment-based configuration
- HTTPS enforcement in production
**Weaknesses:**
- Debug endpoints in production
- Limited file upload validation
- No rate limiting on API endpoints
### Recommended Security Enhancements
1. Implement API rate limiting
2. Add file upload virus scanning
3. Implement proper CORS policies
4. Add audit logging for sensitive operations
## Monitoring & Alerting Recommendations
### Key Metrics to Track
1. **Agent Performance**: Processing time, success rate, error rates
2. **Payment Processing**: Transaction success rate, failed payments
3. **System Health**: Database connections, Redis availability
4. **User Experience**: Page load times, API response times
### Alerting Thresholds
- Agent processing errors > 5% in 5 minutes
- Payment processing failures > 2% in 10 minutes
- Database query time > 500ms average
- Memory usage > 80% for 10 minutes
## Cost-Benefit Analysis
### Implementation Costs
- **Phase 1**: ~40 developer hours
- **Phase 2**: ~80 developer hours
- **Phase 3**: ~120 developer hours
- **Total**: ~240 hours (~6-8 weeks for 1 developer)
### Expected Benefits
- **Reduced Support Tickets**: 60% reduction in error-related issues
- **Improved Reliability**: 99.5% uptime vs current ~95%
- **Better User Experience**: 40% faster page loads
- **Easier Maintenance**: 50% reduction in debugging time
## Conclusion
NetCop Hub has a solid foundation but requires significant improvements for production readiness. The high-priority fixes are critical for stability and security, while medium and low priority improvements will enhance maintainability and user experience.
The recommended approach is to implement changes in phases, starting with critical fixes and gradually improving the system architecture. This will ensure minimal disruption while maximizing the benefits of each improvement.
---
*This analysis provides actionable improvement suggestions prioritized by impact and implementation complexity.*

138
MIGRATION_STRATEGY_EXPLAINED.md
View File

@ -1,138 +0,0 @@
# 🔄 Migration Strategy: From Emergency Fix to Production Ready
## Why Migrations Were Removed (Emergency Fix)
### Original Problem
```
Health Check Failing → "service unavailable" → Deployment Failed
```
**Root Cause:**
- Database not ready when migrations ran
- Migrations failed → entire startup failed
- No way to debug what was actually wrong
### Emergency Solution
```json
// Removed all database dependencies from startup
"startCommand": "gunicorn netcop_hub.wsgi:application --bind 0.0.0.0:$PORT"
```
**Result:**
✅ Django started successfully
✅ Health check passed
✅ Could debug database separately
## Now: Adding Migrations Back (The Right Way)
### Safer Migration Approach
```json
{
"startCommand": "python manage.py migrate --run-syncdb; python manage.py populate_agents; python manage.py collectstatic --noinput && gunicorn ...",
"healthcheckTimeout": 90,
"healthcheckInterval": 15
}
```
### Key Improvements
#### 1. **Better Migration Command**
```bash
# OLD (Problematic):
python manage.py migrate
# NEW (Safer):
python manage.py migrate --run-syncdb
```
- `--run-syncdb` handles initial database creation better
- More robust for fresh PostgreSQL databases
#### 2. **Semicolon vs && Logic**
```bash
# OLD (All-or-nothing):
migrate && populate_agents && gunicorn
# NEW (Continue on issues):
migrate; populate_agents; collectstatic && gunicorn
```
- `;` continues even if migrations have warnings
- Only `&&` before gunicorn (the critical part)
#### 3. **Longer Health Check Timeout**
```json
// OLD: 30 seconds (not enough for migrations)
"healthcheckTimeout": 30
// NEW: 90 seconds (allows for migration time)
"healthcheckTimeout": 90
```
#### 4. **Health Check is Resilient**
Your health endpoint now returns 200 even if database has issues:
```json
{
"status": "healthy",
"checks": {
"application": {"status": "healthy"},
"database": {"status": "warning", "error": "Still connecting..."}
}
}
```
## Why This Approach Works Better
### Before (Brittle):
```
Database Issue → Migration Fails → Startup Fails → No Health Check → Deployment Failed
```
### After (Resilient):
```
Database Issue → Migration Warning → Django Starts → Health Check Passes → Can Debug Database
```
## Expected Deployment Flow
### 1. **Build Phase**
- Install dependencies ✅
- Prepare application ✅
### 2. **Migration Phase**
- `migrate --run-syncdb` (create tables)
- `populate_agents` (add AI agents)
- `collectstatic` (prepare static files)
### 3. **Startup Phase**
- Start Gunicorn web server
- Health check begins testing `/health/`
### 4. **Health Check Results**
- **If database ready**: Shows all systems healthy
- **If database slow**: Shows app healthy, database warning
- **Either way**: Deployment succeeds
## Benefits of This Strategy
### ✅ **Production Ready**
- Migrations run automatically on deployment
- No manual database setup needed
- Follows Django best practices
### ✅ **Fault Tolerant**
- App can start even if migrations have issues
- Health check provides diagnostic information
- Can debug database problems with running app
### ✅ **Scalable**
- Works for fresh deployments and updates
- Handles database initialization properly
- Ready for production traffic
## Rollback Plan
If migrations cause issues again:
1. **Immediate fix**: Remove migrations from startCommand
2. **Manual migration**: Run `railway run python manage.py migrate`
3. **Gradual re-introduction**: Add migrations back step by step
The goal is **reliable deployments** that work in production, not just perfect startup sequences!

420
NETCOP_HUB_ANALYSIS.md
View File

@ -1,420 +0,0 @@
# NetCop Hub - Application Architecture Analysis
*Analysis Date: 2025-07-24*
*Analyst: Claude Code Assistant*
## Overview
NetCop Hub is a Django-based AI agent marketplace platform where users can purchase and interact with specialized AI agents through a pay-per-use model with integrated Stripe payments. The application demonstrates sophisticated architecture with clear separation of concerns and extensible design patterns.
## Project Structure
```
quantum_ai/
├── CLAUDE.md # Project documentation and instructions
├── manage.py # Django management script
├── requirements.txt # Python dependencies
├── db.sqlite3 # SQLite database (development)
├── run_dev.sh # Development server startup script
├── railway.json # Railway.app deployment configuration
├── netcop_hub/ # Main Django project
│ ├── settings.py # Django settings with environment config
│ ├── urls.py # Main URL routing
│ └── production_settings.py # Production-specific settings
├── static/ # Static assets (CSS, JS, images)
├── templates/ # Django templates
├── media/ # User uploaded files
├── logs/ # Application logs
└── [apps]/ # Individual Django applications
```
## Core Architecture
### Django Applications Structure
1. **Core App** (`core/`)
- Purpose: Platform homepage, pricing pages, static content
- Responsibility: Platform presentation layer only
- URL namespace: `core:homepage`, `core:pricing`
2. **Agent Base** (`agent_base/`)
- Purpose: Agent marketplace, catalog management, cross-agent functionality
- Key Models: `BaseAgent`, `BaseAgentRequest`, `BaseAgentResponse`
- URL namespace: `agent_base:marketplace`
- Location: `agent_base/models.py:9-90`
3. **Authentication** (`authentication/`)
- Purpose: User management with integrated wallet functionality
- Key Model: Custom `User` extending AbstractUser
- Features: Email-based auth, password reset tokens, wallet integration
- Location: `authentication/models.py:9-83`
4. **Wallet** (`wallet/`)
- Purpose: Complete payment system with Stripe integration
- Key Model: `WalletTransaction` for financial tracking
- Features: Top-ups, usage tracking, transaction history
- Location: `wallet/models.py:8-31`
5. **Individual Agent Apps**
- Structure: Each agent is a separate Django app
- Examples: `weather_reporter/`, `data_analyzer/`, `job_posting_generator/`
- Pattern: `models.py`, `processor.py`, `views.py`, `urls.py`, `templates/`
## Agent System Architecture
### Agent Types
The platform supports two distinct agent processing patterns:
#### 1. Webhook Agents
- **Processing**: External N8N webhook APIs
- **Examples**: data_analyzer, five_whys_analyzer, job_posting_generator
- **Base Class**: `StandardWebhookProcessor`
- **Use Cases**: Complex data processing, file uploads, multi-step workflows
#### 2. API Agents
- **Processing**: Direct API integration
- **Examples**: weather_reporter (OpenWeather API)
- **Base Class**: `StandardAPIProcessor`
- **Use Cases**: Real-time data fetching, simple request/response patterns
### Agent Processing Framework
Location: `agent_base/processors.py:10-255`
#### Base Classes Hierarchy
```python
BaseAgentProcessor (ABC)
├── StandardWebhookProcessor
└── StandardAPIProcessor
```
#### Key Methods
- `prepare_request_data(**kwargs)` - Format input data
- `make_request(data, timeout=60)` - Execute HTTP request
- `process_response(response_data, request_obj)` - Handle response and create DB objects
- `process_request(**kwargs)` - Main orchestration method
#### Example Implementation - Weather Reporter
Location: `weather_reporter/processor.py:7-139`
```python
class WeatherReporterProcessor(StandardAPIProcessor):
agent_slug = 'weather-reporter'
api_base_url = 'https://api.openweathermap.org/data/2.5/weather'
api_key_env = 'OPENWEATHER_API_KEY'
auth_method = 'query'
```
#### Example Implementation - Data Analyzer
Location: `data_analyzer/processor.py:11-217`
```python
class DataAnalysisAgentProcessor(StandardWebhookProcessor):
agent_slug = 'data-analyzer'
webhook_url = settings.N8N_WEBHOOK_DATA_ANALYZER
agent_id = 'data-analysis-001'
```
## Database Models
### User Model (`authentication/models.py:9-83`)
```python
class User(AbstractUser):
email = models.EmailField(unique=True)
wallet_balance = models.DecimalField(max_digits=10, decimal_places=2, default=Decimal('0.00'))
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
# Wallet methods
def has_sufficient_balance(self, amount)
def deduct_balance(self, amount, description="", agent_slug="")
def add_balance(self, amount, description="", stripe_session_id="")
```
### BaseAgent Model (`agent_base/models.py:9-59`)
```python
class BaseAgent(models.Model):
CATEGORIES = [
('analytics', 'Analytics'),
('utilities', 'Utilities'),
('content', 'Content'),
('marketing', 'Marketing'),
('customer-service', 'Customer Service'),
]
id = models.UUIDField(primary_key=True, default=uuid.uuid4)
name = models.CharField(max_length=200)
slug = models.SlugField(unique=True)
description = models.TextField()
category = models.CharField(max_length=50, choices=CATEGORIES)
price = models.DecimalField(max_digits=10, decimal_places=2)
agent_type = models.CharField(max_length=20, choices=[
('webhook', 'Webhook'),
('api', 'API'),
])
```
### WalletTransaction Model (`wallet/models.py:8-31`)
```python
class WalletTransaction(models.Model):
TRANSACTION_TYPES = [
('top_up', 'Top Up'),
('agent_usage', 'Agent Usage'),
('refund', 'Refund'),
]
id = models.UUIDField(primary_key=True, default=uuid.uuid4)
user = models.ForeignKey(User, on_delete=models.CASCADE)
amount = models.DecimalField(max_digits=10, decimal_places=2)
type = models.CharField(max_length=20, choices=TRANSACTION_TYPES)
stripe_session_id = models.CharField(max_length=200, blank=True)
```
## URL Structure & Routing
From `netcop_hub/urls.py:22-33`:
```python
urlpatterns = [
path('admin/', admin.site.urls),
path('auth/', include('authentication.urls')),
path('wallet/', include('wallet.urls')),
path('', include('agent_base.urls')), # Marketplace
path('agents/weather-reporter/', include('weather_reporter.urls')),
path('agents/data-analyzer/', include('data_analyzer.urls')),
path('agents/job-posting-generator/', include('job_posting_generator.urls')),
path('agents/social-ads-generator/', include('social_ads_generator.urls')),
path('agents/five-whys-analyzer/', include('five_whys_analyzer.urls')),
path('', include('core.urls')), # Homepage
]
```
### URL Mapping
- `/` - Homepage (core app)
- `/pricing/` - Pricing page (core app)
- `/marketplace/` - Agent marketplace (agent_base)
- `/agents/<agent-slug>/` - Individual agent pages
- `/auth/` - Authentication (login, register, profile)
- `/wallet/` - Wallet management and Stripe integration
- `/admin/` - Django admin interface
## Technology Stack
### Core Dependencies (from `requirements.txt`)
```
Django==5.2.4
djangorestframework==3.15.2
python-decouple==3.8
stripe==12.3.0
Pillow==11.3.0
requests==2.32.4
gunicorn==21.2.0
psycopg2-binary==2.9.9
dj-database-url==2.1.0
whitenoise==6.8.2
redis==5.2.0
django-redis==5.4.0
```
### Database Configuration
- **Development**: SQLite (`db.sqlite3`)
- **Production**: PostgreSQL via Railway
- **Smart Detection**: Auto-detects environment and configures appropriately
### Caching Strategy
From `netcop_hub/settings.py:293-323`:
- **Primary**: Redis cache with django-redis client
- **Fallback**: Local memory cache if Redis unavailable
- **Session Storage**: Cache-based sessions
### Static Files & Media
- **Static Files**: WhiteNoise for production serving
- **Media Files**: Local filesystem with cleanup management
- **Upload Handling**: Automatic file cleanup after processing
## Payment System
### Stripe Integration
- **Environment Variables**: `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`
- **Payment Flow**: Checkout sessions → webhook handling → wallet top-up
- **Transaction Tracking**: Complete audit trail in `WalletTransaction`
### Wallet Functionality
- **Balance Management**: User model integrates wallet operations
- **Usage Deduction**: Automatic deduction after successful agent processing
- **Transaction Types**: Top-up, agent usage, refunds
## Security Features
### Authentication & Authorization
- **Custom User Model**: Email-based authentication
- **Password Reset**: Token-based system with expiration
- **Session Management**: Cache-based with 1-hour timeout
### Production Security (from `netcop_hub/settings.py:114-123`)
```python
if not DEBUG:
SECURE_SSL_REDIRECT = True
SECURE_HSTS_SECONDS = 31536000 # 1 year
SECURE_HSTS_INCLUDE_SUBDOMAINS = True
SECURE_HSTS_PRELOAD = True
SESSION_COOKIE_SECURE = True
CSRF_COOKIE_SECURE = True
```
### File Upload Security
- **File Cleanup**: Automatic deletion after processing
- **Path Validation**: Secure file handling in processors
- **Content Type Validation**: PDF validation for data analyzer
## Development Tools & Commands
### Management Commands
Located in `agent_base/management/commands/`:
- `python manage.py create_agent` - Generate new agent boilerplate
- `python manage.py populate_agents` - Populate agent catalog
- `python manage.py create_user` - Create test users
- `python manage.py check_db` - Validate database configuration
- `python manage.py reset_database` - Reset development database
- `python manage.py backup_users` - User data backup utilities
- `python manage.py test_webhook` - Webhook testing utilities
### Development Workflow
1. **Quick Start**: `./run_dev.sh` (handles migrations and environment)
2. **Manual Start**: `python manage.py runserver`
3. **Testing**: Individual test files in `tests/` directory
4. **Agent Creation**: Use management command with template system
## Deployment
### Railway.app Integration
- **Configuration**: `railway.json` for deployment settings
- **Environment Detection**: Automatic Railway environment detection
- **Database**: PostgreSQL with automatic URL parsing
- **Static Files**: WhiteNoise middleware for production serving
### Environment Variables
From `netcop_hub/settings.py:31-38` - Required variables validation:
```python
required_env_vars = ['SECRET_KEY']
missing_vars = [var for var in required_env_vars if not config(var, default='')]
if missing_vars:
print(f"❌ Missing required environment variables: {', '.join(missing_vars)}")
sys.exit(1)
```
## Logging Configuration
### Log Levels & Handlers (from `netcop_hub/settings.py:337-389`)
- **File Logging**: `netcop.log` for persistent logging
- **Console Logging**: Development debugging
- **App-Specific Loggers**: `agent_base`, `wallet`, `netcop_hub`
- **Django Integration**: Complete Django logging integration
## Template Architecture
### Template Hierarchy
```
templates/
├── base.html # Main layout with navigation
├── components/ # Reusable components
│ ├── agent_header.html
│ ├── wallet_card.html
│ ├── processing_status.html
│ └── results_container.html
├── core/ # Platform pages
├── agent_base/ # Marketplace templates
├── authentication/ # Auth templates
├── wallet/ # Payment templates
└── [agent_apps]/ # Agent-specific templates
```
### CSS Architecture
```
static/css/
├── base.css # Global styles and CSS variables
├── agent-base.css # Agent page styling
├── header-component.css # Header styling
├── marketplace.css # Marketplace styling
└── themes.css # Theme definitions
```
## Key Design Patterns
### 1. Single Responsibility Principle
- **Core**: Platform presentation only
- **Agent Base**: Marketplace and cross-agent functionality
- **Wallet**: Complete payment system
- **Individual Agents**: Specific agent logic
### 2. Abstract Base Classes
- `BaseAgentProcessor` for standardized agent processing
- `BaseAgentRequest` and `BaseAgentResponse` for consistent data models
- Template method pattern in processor classes
### 3. Environment-Based Configuration
- Automatic environment detection (Railway vs local)
- Smart database configuration with fallbacks
- Required environment variable validation
### 4. Extensible Agent System
- Template generation for new agents
- Standardized processor interfaces
- Automatic marketplace integration
## Performance Considerations
### Caching Strategy
- Redis for session storage and application caching
- Graceful fallback to memory cache
- Database query optimization with indexes
### File Management
- Automatic cleanup of uploaded files
- Efficient file processing in agent processors
- Media file organization by agent type
### Database Optimization
- UUID primary keys for distributed systems
- Strategic database indexes on User model
- Efficient query patterns in processors
## Error Handling & Monitoring
### Exception Management
- Standardized error handling in processor base classes
- Graceful degradation for external service failures
- Comprehensive error logging throughout the application
### Transaction Safety
- Database transaction handling in wallet operations
- Rollback mechanisms for failed agent processing
- Consistent state management across agent requests
## Future Extensibility
### Adding New Agents
1. Use `python manage.py create_agent` management command
2. Implement processor class inheriting from appropriate base
3. Define agent-specific models and views
4. Agent automatically appears in marketplace via `BaseAgent`
### Scaling Considerations
- UUID-based primary keys support distributed architectures
- Redis caching ready for horizontal scaling
- Modular app structure supports microservice migration
- Environment-based configuration supports multi-environment deployments
## Security Best Practices
### Data Protection
- Automatic file cleanup prevents data accumulation
- Secure file upload handling with validation
- Environment variable configuration for sensitive data
### Authentication Security
- Email-based authentication with secure password handling
- Token-based password reset with expiration
- Production security headers and HTTPS enforcement
---
*This analysis provides a comprehensive overview of the NetCop Hub application architecture, suitable for development planning, maintenance, and future enhancements.*

127
RAILWAY_DATABASE_SETUP.md
View File

@ -1,127 +0,0 @@
# 🗄️ Railway PostgreSQL Database Setup
## Current Status
✅ Django settings.py is already configured to use Railway PostgreSQL
✅ Django app is starting successfully
❌ Need to set DATABASE_URL environment variable
## Step-by-Step Railway Database Configuration
### Step 1: Ensure PostgreSQL Service is Added
1. Go to your Railway project dashboard
2. Click "Add Service" → "Database" → "PostgreSQL"
3. Wait for PostgreSQL service to deploy (shows green/active status)
### Step 2: Set DATABASE_URL Environment Variable
1. Go to your Railway project → **Variables** tab
2. Click "**Add Variable**"
3. Set:
- **Name**: `DATABASE_URL`
- **Value**: `${{ Postgres.DATABASE_URL }}`
### Step 3: Verify Variable Resolution
Railway will automatically resolve `${{ Postgres.DATABASE_URL }}` to the actual PostgreSQL connection string:
```
postgresql://postgres:password@hostname:5432/railway
```
### Step 4: Deploy Changes
Since you're updating environment variables, Railway will automatically redeploy your app.
## How Django Will Use This
### Your Current Settings (Already Perfect)
```python
# In settings.py - already configured correctly
database_url = config('DATABASE_URL', default='')
if database_url:
# Parse Railway PostgreSQL URL
DATABASES = {
'default': dj_database_url.parse(database_url, conn_max_age=600)
}
else:
# Fallback to SQLite for local development
DATABASES = {'default': {'ENGINE': 'django.db.backends.sqlite3', ...}}
```
### What Happens After Setup
1. **Railway resolves variable**: `${{ Postgres.DATABASE_URL }}` → actual connection string
2. **Django reads DATABASE_URL**: From environment variables
3. **dj_database_url parses**: Converts URL to Django database config
4. **Connection pooling**: `conn_max_age=600` keeps connections alive
## Testing Database Connection
### After Railway Deployment
```bash
# Test database connection
railway run python manage.py check_db
# Run migrations and setup
railway run python manage.py setup_database
# Or manually:
railway run python manage.py migrate
railway run python manage.py populate_agents
```
### Expected Success Output
```
Database connection successful!
Running database migrations...
Migrations completed successfully!
Populating agents...
Agents populated successfully!
Database setup completed!
```
## Troubleshooting
### Issue: Database Connection Failed
**Cause**: PostgreSQL service not running or DATABASE_URL not set
**Fix**: Ensure PostgreSQL service is active and DATABASE_URL variable is set
### Issue: No Such Table Errors
**Cause**: Migrations haven't been run
**Fix**: Run `railway run python manage.py setup_database`
### Issue: Permission Denied
**Cause**: Database user permissions
**Fix**: Railway PostgreSQL should have full permissions by default
## Verification Steps
### 1. Check Railway Dashboard
- ✅ PostgreSQL service shows "Active"
- ✅ DATABASE_URL variable exists
- ✅ App deployment successful
### 2. Test Health Endpoint
```bash
curl https://your-railway-domain.railway.app/health/
```
**Expected response:**
```json
{
"status": "healthy",
"checks": {
"database": {"status": "healthy"},
"agents": {"status": "healthy", "active_count": 6}
}
}
```
### 3. Test Application Access
- Visit Railway domain in browser
- Should show homepage without database errors
- All 6 AI agents should be accessible
## Summary
Your Django code is already perfectly configured for Railway PostgreSQL. You just need to:
1. **Add PostgreSQL service** (if not already added)
2. **Set DATABASE_URL = ${{ Postgres.DATABASE_URL }}** in Railway variables
3. **Run database setup** after deployment
That's it! 🚀

289
RAILWAY_DEPLOYMENT_GUIDE.md
View File

@ -1,289 +0,0 @@
# 🚀 Railway.app Deployment Guide for Quantum Tasks AI
## Overview
This guide will help you deploy your Quantum Tasks AI Django application to Railway.app. Your application is already optimized for Railway deployment with the existing `railway.json` configuration.
### 🏗️ Architecture Overview (Important!)
**What Deploys to Railway:**
- ✅ Django Application (Quantum Tasks AI)
- ✅ PostgreSQL Database (automatic)
- ✅ Redis Cache (optional but recommended)
**What DOES NOT Deploy to Railway:**
- ❌ N8N Instance (runs on separate server)
- ❌ N8N Workflows (hosted elsewhere)
**How They Connect:**
```
Railway Django App → HTTP POST Requests → N8N Instance (Separate Hosting) → AI Processing → Response → Railway Django App
```
Your Django app only needs the N8N webhook URLs as environment variables to connect to your separately-hosted N8N instance.
## 📋 Pre-Deployment Checklist
### Required Accounts & Services
- [ ] GitHub account with your repository
- [ ] Railway.app account (free signup)
- [ ] Stripe account for payments (test/live keys)
- [ ] Gmail or SMTP service for emails
- [ ] N8N instance for AI agent webhooks
### Code Verification
- [ ] Latest code pushed to GitHub
- [ ] All migrations created and committed
- [ ] `railway.json` file present in root directory
- [ ] Environment variables documented in `.env.example`
## 🔧 Step-by-Step Deployment
### Step 1: Connect to Railway
1. Visit [railway.app](https://railway.app) and sign up/login
2. Click "New Project" → "Deploy from GitHub repo"
3. Select your `quantum_ai` repository
4. Railway will automatically detect Django and start building
### Step 2: Configure Environment Variables
Navigate to your project settings and add these environment variables:
#### 🔐 Security Settings
```bash
SECRET_KEY=your-50-character-secret-key-here
DEBUG=False
ALLOWED_HOSTS=your-domain.railway.app,quantumtaskai.com
CSRF_TRUSTED_ORIGINS=https://your-domain.railway.app,https://quantumtaskai.com
```
#### 📧 Email Configuration
```bash
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=smtp.gmail.com
EMAIL_PORT=587
EMAIL_USE_TLS=True
EMAIL_HOST_USER=your-email@gmail.com
EMAIL_HOST_PASSWORD=your-app-password
DEFAULT_FROM_EMAIL=Quantum Tasks AI <noreply@quantumtaskai.com>
```
#### 💳 Stripe Configuration
```bash
STRIPE_SECRET_KEY=sk_live_your_stripe_secret_key
STRIPE_WEBHOOK_SECRET=whsec_your_webhook_secret
```
#### 🤖 N8N Webhook URLs
```bash
N8N_WEBHOOK_DATA_ANALYZER=https://your-n8n.com/webhook/data-analyzer
N8N_WEBHOOK_FIVE_WHYS=https://your-n8n.com/webhook/five-whys
N8N_WEBHOOK_JOB_POSTING=https://your-n8n.com/webhook/job-posting
N8N_WEBHOOK_SOCIAL_ADS=https://your-n8n.com/webhook/social-ads
```
**Note**: Only webhook-based agents need N8N workflows. API-based agents (weather_reporter, email_writer) work independently.
#### 🗄️ Database Configuration
Railway automatically provides `DATABASE_URL` - no manual configuration needed!
#### ⚡ Redis Configuration (Optional but Recommended)
```bash
REDIS_URL=redis://your-redis-url:6379
```
### Step 3: Add PostgreSQL Database
1. In your Railway project dashboard
2. Click "New" → "Database" → "Add PostgreSQL"
3. Railway automatically sets the `DATABASE_URL` environment variable
### Step 4: Add Redis (Recommended)
1. Click "New" → "Database" → "Add Redis"
2. Railway automatically sets the `REDIS_URL` environment variable
### Step 5: Set Up N8N Instance (Separate Hosting)
⚠️ **IMPORTANT**: N8N is NOT deployed to Railway with your Django app. N8N runs on a separate server and your Django app connects to it via webhooks.
#### Architecture Overview:
```
User → Django App (Railway) → HTTP POST → N8N Webhooks (Separate Server) → AI Processing → Response → Django → User
```
#### N8N Hosting Options (Choose One):
**Option A: N8N Cloud (Recommended - Easiest)**
1. Sign up at [n8n.cloud](https://n8n.cloud)
2. Create a new workflow instance
3. Import your workflow JSON files
4. Copy webhook URLs for environment variables
**Option B: Deploy N8N on Railway (Separate Project)**
1. Create a NEW Railway project (separate from your Django app)
2. Deploy N8N using Railway's N8N template
3. Configure OpenAI API credentials in N8N
4. Import workflows and get webhook URLs
**Option C: Self-Hosted N8N**
1. Deploy N8N on DigitalOcean, AWS, or VPS
2. Use Docker: `docker run -it --rm --name n8n -p 5678:5678 n8nio/n8n`
3. Configure and import workflows
4. Ensure server is publicly accessible for webhooks
#### Deploy Workflows to Your N8N Instance:
```bash
# Set connection details for YOUR N8N instance
export N8N_BASE_URL=https://your-n8n-instance.com # Your N8N URL
export N8N_API_KEY=your-api-key # Your N8N API key
# Deploy all workflows to your N8N instance
./deploy_n8n_workflows.sh
```
#### Configure Django App to Connect to N8N:
1. Copy webhook URLs from your N8N instance
2. Add these URLs to your Railway Django project environment variables:
```
N8N_WEBHOOK_DATA_ANALYZER=https://your-n8n.com/webhook/data-analyzer
N8N_WEBHOOK_SOCIAL_ADS=https://your-n8n.com/webhook/social-ads
N8N_WEBHOOK_JOB_POSTING=https://your-n8n.com/webhook/job-posting
N8N_WEBHOOK_FIVE_WHYS=https://your-n8n.com/webhook/five-whys
```
3. Verify workflows are active in your N8N instance
### Step 6: Custom Domain (Optional)
1. Go to project Settings → Domains
2. Add your custom domain (e.g., `quantumtaskai.com`)
3. Update DNS records as instructed by Railway
4. Update `ALLOWED_HOSTS` and `CSRF_TRUSTED_ORIGINS` environment variables
## 🔍 Post-Deployment Verification
### Health Check
Visit your deployed application health endpoint:
```
https://your-domain.railway.app/health/
```
Expected response:
```json
{
"status": "healthy",
"timestamp": 1234567890,
"version": "1.0",
"checks": {
"database": {"status": "healthy", "response_time_ms": 2.5},
"agents": {"status": "healthy", "active_count": 7}
},
"response_time_ms": 5.2
}
```
### Application Testing
- [ ] Homepage loads correctly (`/`)
- [ ] User registration works (`/auth/register/`)
- [ ] Login functionality (`/auth/login/`)
- [ ] Marketplace displays agents (`/marketplace/`)
- [ ] Payment system functional (Stripe webhooks)
- [ ] Contact form submits successfully (`/contact/`)
- [ ] Admin panel accessible (`/admin/`)
### Monitoring Setup
1. **Application Logs**: Available in Railway dashboard
2. **Health Monitoring**: Set up external monitoring to ping `/health/`
3. **Error Tracking**: Monitor Railway application logs
4. **Database Performance**: Use Railway's built-in database metrics
## 🚨 Troubleshooting
### Common Issues & Solutions
#### Migration Errors
```bash
# If you see migration conflicts, check Railway logs
# Your railway.json already handles complex migrations
```
#### Static Files Not Loading
```bash
# Already handled by WhiteNoise configuration
# Verify STATIC_URL and STATIC_ROOT in settings
```
#### Environment Variable Issues
```bash
# Check Railway project settings
# Ensure all required variables are set
# Restart deployment after adding variables
```
#### Database Connection Issues
```bash
# Verify PostgreSQL service is running in Railway
# Check DATABASE_URL is automatically set
# Review connection logs in Railway dashboard
```
## 📊 Cost Estimation
### Railway.app Pricing (Monthly)
- **Web Service**: $5/month (scales with usage)
- **PostgreSQL**: $5/month (1GB storage, scales up)
- **Redis**: $5/month (256MB, scales up)
- **Bandwidth**: $0.10/GB (generous free tier)
**Total Estimated Cost**: $15-25/month for production usage
### Scaling Thresholds
- **Free Tier**: Good for development and testing
- **Scale Up**: When you hit 1000+ daily active users
- **Database**: Scales automatically with your data growth
## 🔒 Security Best Practices
### Environment Variables
- Never commit real environment variables to Git
- Use Railway's environment variable encryption
- Rotate API keys regularly (Stripe, email, N8N)
### Domain Security
- Always use HTTPS (Railway provides SSL automatically)
- Configure proper CORS settings
- Monitor your `/health/` endpoint for unauthorized access
### Database Security
- Railway PostgreSQL is automatically encrypted
- Enable database backups (Railway provides automatic backups)
- Monitor database performance and queries
## 📈 Performance Optimization
### Railway-Specific Optimizations
1. **Region Selection**: Choose region closest to your users
2. **Resource Allocation**: Monitor CPU/memory usage in dashboard
3. **Caching**: Redis is automatically configured for session caching
4. **Static Files**: WhiteNoise serves static files efficiently
### Monitoring & Alerts
1. Set up monitoring for your `/health/` endpoint
2. Configure alerts for high error rates
3. Monitor database performance metrics
4. Track user registration and payment success rates
## 🎉 Success!
Once deployed successfully, your Quantum Tasks AI application will be live at:
- **Production URL**: `https://your-domain.railway.app`
- **Custom Domain**: `https://quantumtaskai.com` (if configured)
- **Health Check**: `https://your-domain.railway.app/health/`
- **Admin Panel**: `https://your-domain.railway.app/admin/`
Your AI agent marketplace is now ready to serve users worldwide! 🌍
## 📞 Support
If you encounter issues:
1. Check Railway application logs first
2. Verify all environment variables are set correctly
3. Test the `/health/` endpoint for system status
4. Review this deployment guide for common solutions
Railway.app provides excellent documentation and support for Django applications.

117
RAILWAY_ENV_CHECKLIST.md
View File

@ -1,117 +0,0 @@
# 🔐 Railway Environment Variables Checklist
## Critical Variables for Health Check Success
### ✅ **Required Variables (Must Set These)**
```bash
# Django Core (REQUIRED)
SECRET_KEY=your-50-character-secret-key
DEBUG=False
ALLOWED_HOSTS=your-project.railway.app,quantumtaskai.com
# Database (AUTO-SET by Railway PostgreSQL service)
# DATABASE_URL=postgresql://... (Railway sets this automatically)
```
### ⚠️ **Optional Variables (Set if Using Features)**
```bash
# Email Configuration (for contact form, password reset)
EMAIL_HOST_USER=your-email@gmail.com
EMAIL_HOST_PASSWORD=your-gmail-app-password
DEFAULT_FROM_EMAIL=Quantum Tasks AI <noreply@quantumtaskai.com>
# Stripe Payment (for wallet functionality)
STRIPE_SECRET_KEY=sk_live_your_stripe_key
STRIPE_WEBHOOK_SECRET=whsec_your_webhook_secret
# N8N Webhooks (for AI agents that use webhooks)
N8N_WEBHOOK_DATA_ANALYZER=https://your-n8n-url/webhook/data-analyzer
N8N_WEBHOOK_FIVE_WHYS=https://your-n8n-url/webhook/five-whys
N8N_WEBHOOK_JOB_POSTING=https://your-n8n-url/webhook/job-posting
N8N_WEBHOOK_SOCIAL_ADS=https://your-n8n-url/webhook/social-ads
# OpenWeather API (for weather agent)
OPENWEATHER_API_KEY=your_openweather_key
```
## 🚨 Health Check Failure Troubleshooting
### Most Common Issues:
1. **Missing SECRET_KEY**
```
Error: "The SECRET_KEY setting must not be empty"
Solution: Set SECRET_KEY in Railway variables
```
2. **Database Not Ready**
```
Error: "connection to server failed"
Solution: Wait 30-60 seconds, Railway PostgreSQL is starting
```
3. **Wrong ALLOWED_HOSTS**
```
Error: "DisallowedHost at /health/"
Solution: Add Railway domain to ALLOWED_HOSTS
```
## 🔧 Quick Fixes
### Generate SECRET_KEY
```python
# Run locally
python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"
```
### Minimal Working Configuration
```bash
# These 3 variables will make health check pass:
SECRET_KEY=your-generated-secret-key-here
DEBUG=False
ALLOWED_HOSTS=your-project.railway.app
```
### Test Health Check Locally
```bash
# Set minimal env vars and test
export SECRET_KEY="your-secret-key"
export DEBUG=False
export ALLOWED_HOSTS="localhost,127.0.0.1"
python manage.py runserver
curl http://localhost:8000/health/
```
## 📊 Expected Health Check Response
```json
{
"status": "healthy",
"timestamp": 1690123456,
"version": "1.0",
"checks": {
"database": {
"status": "healthy",
"response_time_ms": 12.3,
"attempt": 1
},
"agents": {
"status": "healthy",
"active_count": 6
},
"application": {
"status": "healthy",
"django_ready": true
}
},
"response_time_ms": 45.2
}
```
## 🎯 Deployment Steps
1. Set minimum required variables in Railway
2. Deploy with updated railway.json (60s timeout)
3. Check Railway logs for errors
4. Test health endpoint: `curl https://your-project.railway.app/health/`
5. Add optional variables as needed for full functionality
**Health check should pass within 60 seconds with just the 3 critical variables!**

231
RAILWAY_ENV_TEMPLATE.md
View File

@ -1,231 +0,0 @@
# 🔐 Railway Environment Variables Template
## Required Environment Variables for Production Deployment
Copy these environment variables to your Railway project settings. Replace placeholder values with your actual production values.
### 🔒 Core Security Settings
```bash
# Django Security
SECRET_KEY=django-insecure-REPLACE-WITH-50-RANDOM-CHARACTERS-FOR-PRODUCTION
DEBUG=False
ALLOWED_HOSTS=your-project-name.railway.app,quantumtaskai.com,www.quantumtaskai.com
CSRF_TRUSTED_ORIGINS=https://your-project-name.railway.app,https://quantumtaskai.com,https://www.quantumtaskai.com
```
### 📧 Email Configuration (Gmail Example)
```bash
# Email Settings - Use Gmail App Password or SMTP service
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=smtp.gmail.com
EMAIL_PORT=587
EMAIL_USE_TLS=True
EMAIL_HOST_USER=your-email@gmail.com
EMAIL_HOST_PASSWORD=your-16-character-app-password
DEFAULT_FROM_EMAIL=Quantum Tasks AI <noreply@quantumtaskai.com>
```
### 💳 Stripe Payment Configuration
```bash
# Stripe - Use LIVE keys for production
STRIPE_SECRET_KEY=sk_live_your_stripe_secret_key_here
STRIPE_WEBHOOK_SECRET=whsec_your_webhook_endpoint_secret
```
### 🤖 N8N AI Agent Webhooks (External Server URLs)
⚠️ **IMPORTANT**: These URLs point to your SEPARATE N8N instance, NOT hosted on Railway with Django.
```bash
# N8N Webhook URLs - Replace with your actual N8N instance URLs
# Option A: N8N Cloud
N8N_WEBHOOK_DATA_ANALYZER=https://yourworkspace.app.n8n.cloud/webhook/data-analyzer
N8N_WEBHOOK_FIVE_WHYS=https://yourworkspace.app.n8n.cloud/webhook/five-whys
N8N_WEBHOOK_JOB_POSTING=https://yourworkspace.app.n8n.cloud/webhook/job-posting
N8N_WEBHOOK_SOCIAL_ADS=https://yourworkspace.app.n8n.cloud/webhook/social-ads
# Option B: Self-hosted or separate Railway N8N project
# N8N_WEBHOOK_DATA_ANALYZER=https://your-n8n-server.com/webhook/data-analyzer
# N8N_WEBHOOK_FIVE_WHYS=https://your-n8n-server.com/webhook/five-whys
# N8N_WEBHOOK_JOB_POSTING=https://your-n8n-server.com/webhook/job-posting
# N8N_WEBHOOK_SOCIAL_ADS=https://your-n8n-server.com/webhook/social-ads
```
### 🌤️ External API Keys
```bash
# OpenWeather API for Weather Agent
OPENWEATHER_API_KEY=your_openweather_api_key_here
```
### ⚡ Performance & Caching (Optional)
```bash
# Redis URL - Automatically set by Railway Redis service
# REDIS_URL=redis://default:password@host:port
```
---
## 🔍 Environment Variable Setup Instructions
### Step 1: Generate SECRET_KEY
Use Django to generate a secure secret key:
```python
from django.core.management.utils import get_random_secret_key
print(get_random_secret_key())
```
### Step 2: Gmail App Password Setup
1. Enable 2-Factor Authentication on your Gmail account
2. Go to Google Account Settings → Security → App passwords
3. Generate an app password for "Django Email"
4. Use the 16-character app password (not your regular password)
### Step 3: Stripe Configuration
1. Login to your Stripe Dashboard
2. Go to Developers → API Keys
3. Copy your "Secret key" (starts with `sk_live_` for production)
4. Go to Developers → Webhooks
5. Create webhook endpoint: `https://your-domain.railway.app/wallet/stripe/webhook/`
6. Copy the webhook signing secret (starts with `whsec_`)
### Step 4: N8N Webhook URLs (Separate Server)
⚠️ **N8N RUNS SEPARATELY** from your Django app. Choose one hosting option:
**Option A: N8N Cloud (Easiest)**
1. Sign up at [n8n.cloud](https://n8n.cloud)
2. Import your workflow JSON files from agent directories
3. Configure OpenAI API credentials in N8N
4. Copy webhook URLs from each workflow
5. Add URLs to Railway environment variables
**Option B: Separate Railway Project for N8N**
1. Create a NEW Railway project (different from your Django app)
2. Deploy N8N using Railway's template or Docker
3. Import workflows and configure credentials
4. Copy webhook URLs and add to Django app environment
**Option C: Self-Hosted N8N**
1. Deploy N8N on DigitalOcean, AWS, VPS, or local server
2. Ensure server is publicly accessible for webhook calls
3. Import workflows and get webhook URLs
4. Ensure N8N workflows are active and accessible
### Step 5: OpenWeather API
1. Sign up at [OpenWeatherMap](https://openweathermap.org/api)
2. Get your free API key
3. Add it to the environment variables
---
## 🚫 Important Security Notes
### Never Include in Git:
- ❌ Real SECRET_KEY values
- ❌ Production API keys
- ❌ Email passwords
- ❌ Stripe live keys
- ❌ Database credentials
### Railway Automatic Variables:
Railway automatically provides these - **DO NOT SET MANUALLY**:
- `DATABASE_URL` (PostgreSQL connection string)
- `PORT` (Application port)
- `RAILWAY_*` (Railway-specific variables)
### Testing Configuration:
Use Railway's "Preview" deployments to test environment variables before going live.
---
## 📋 Environment Variable Checklist
Before deploying, verify you have set:
### Core Settings ✓
- [ ] `SECRET_KEY` (50+ random characters)
- [ ] `DEBUG=False`
- [ ] `ALLOWED_HOSTS` (includes your Railway domain)
- [ ] `CSRF_TRUSTED_ORIGINS` (HTTPS URLs only)
### Email Configuration ✓
- [ ] `EMAIL_HOST_USER` (your Gmail address)
- [ ] `EMAIL_HOST_PASSWORD` (Gmail app password)
- [ ] `DEFAULT_FROM_EMAIL` (your sender email)
### Payment System ✓
- [ ] `STRIPE_SECRET_KEY` (live key for production)
- [ ] `STRIPE_WEBHOOK_SECRET` (webhook endpoint secret)
### AI Agents ✓
- [ ] All `N8N_WEBHOOK_*` URLs are accessible
- [ ] `OPENWEATHER_API_KEY` (for weather agent)
### External Services ✓
- [ ] PostgreSQL database added to Railway project
- [ ] Redis service added (optional but recommended)
- [ ] Custom domain configured (if applicable)
---
## 🔧 Advanced Configuration
### Production-Ready Gunicorn Settings
The `railway.json` includes optimized Gunicorn configuration:
- 2 workers (scales with CPU cores)
- 120-second timeout for AI processing
- Request recycling for memory management
- Health check integration
### Database Connection Pooling
Railway's PostgreSQL automatically handles connection pooling for optimal performance.
### Static Files & CDN
WhiteNoise configuration in your Django settings handles static file serving efficiently.
---
## 🎯 Environment Variable Testing
After setting variables in Railway:
1. **Deploy Application**: Railway will automatically deploy with new variables
2. **Check Health Endpoint**: `https://your-domain.railway.app/health/`
3. **Test Authentication**: Try user registration and login
4. **Verify Payments**: Test Stripe integration (use test cards)
5. **Check AI Agents**: Test each agent workflow
6. **Monitor Logs**: Watch Railway application logs for errors
---
## 🆘 Troubleshooting
### Common Issues:
**Secret Key Error:**
```
django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty
```
→ Ensure SECRET_KEY is set and not empty
**Email Authentication Failed:**
```
SMTPAuthenticationError: Username and Password not accepted
```
→ Use Gmail App Password, not regular password
**Stripe Webhook Verification Failed:**
```
stripe.error.SignatureVerificationError
```
→ Verify webhook secret matches Stripe dashboard
**N8N Webhook Not Responding:**
```
requests.exceptions.ConnectionError
```
→ Ensure N8N instance is running and accessible
---
This template ensures your Quantum Tasks AI application runs securely and efficiently on Railway.app! 🚀

170
RAILWAY_FINAL_SETUP.md
View File

@ -1,170 +0,0 @@
# 🚀 Final Railway Setup for quantum-ai.up.railway.app
## Your Railway URL
**Application URL:** `https://quantum-ai.up.railway.app`
## Required Environment Variables for Railway
### Set These in Railway Dashboard → Variables:
#### 1. **Core Django Settings**
```bash
SECRET_KEY=your-50-character-secret-key
DEBUG=False
ALLOWED_HOSTS=quantum-ai.up.railway.app,quantumtaskai.com,localhost
```
#### 2. **Database Connection**
```bash
DATABASE_URL=${{ Postgres.DATABASE_URL }}
```
#### 3. **CSRF Security**
```bash
CSRF_TRUSTED_ORIGINS=https://quantum-ai.up.railway.app,https://quantumtaskai.com
```
## Testing Commands
### 1. **Test Application Access**
```bash
curl https://quantum-ai.up.railway.app/
```
### 2. **Test Health Endpoint**
```bash
curl https://quantum-ai.up.railway.app/health/
```
### 3. **Expected Health Response**
```json
{
"status": "healthy",
"timestamp": 1690123456,
"version": "1.0",
"app": "quantum-tasks-ai",
"checks": {
"application": {
"status": "healthy",
"django_ready": true,
"server_running": true
},
"database": {
"status": "healthy",
"response_time_ms": 12.3
},
"agents": {
"status": "healthy",
"active_count": 6
},
"environment": {
"status": "healthy",
"debug_mode": false,
"secret_key_configured": true
}
},
"response_time_ms": 45.2
}
```
## AI Agents Available at:
1. **Data Analyzer**: `https://quantum-ai.up.railway.app/agents/data-analyzer/`
2. **Weather Reporter**: `https://quantum-ai.up.railway.app/agents/weather-reporter/`
3. **Job Posting Generator**: `https://quantum-ai.up.railway.app/agents/job-posting-generator/`
4. **Social Ads Generator**: `https://quantum-ai.up.railway.app/agents/social-ads-generator/`
5. **Five Whys Analyzer**: `https://quantum-ai.up.railway.app/agents/five-whys-analyzer/`
6. **Email Writer**: `https://quantum-ai.up.railway.app/agents/email-writer/`
## Core Pages:
- **Homepage**: `https://quantum-ai.up.railway.app/`
- **Marketplace**: `https://quantum-ai.up.railway.app/marketplace/`
- **User Registration**: `https://quantum-ai.up.railway.app/auth/register/`
- **Login**: `https://quantum-ai.up.railway.app/auth/login/`
- **Wallet**: `https://quantum-ai.up.railway.app/wallet/`
- **Admin**: `https://quantum-ai.up.railway.app/admin/`
## Deployment Steps
### 1. **Set Environment Variables**
Go to Railway dashboard and set the variables listed above.
### 2. **Commit and Deploy**
```bash
git add .
git commit -m "Add migrations back to startup with improved configuration"
git push origin main
```
### 3. **Monitor Deployment**
Watch Railway logs for:
- ✅ `Starting gunicorn 21.2.0`
- ✅ `Operations to perform: Apply all migrations`
- ✅ `Successfully processed 6 agents`
- ✅ `Listening at: http://0.0.0.0:8080`
### 4. **Verify Success**
```bash
# Test health endpoint
curl https://quantum-ai.up.railway.app/health/
# Test homepage
curl https://quantum-ai.up.railway.app/
# Test marketplace
curl https://quantum-ai.up.railway.app/marketplace/
```
## Expected Migration Log Output
```
Operations to perform:
Apply all migrations: admin, agent_base, auth, authentication, contenttypes, core, data_analyzer, email_writer, five_whys_analyzer, job_posting_generator, sessions, social_ads_generator, wallet, weather_reporter
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying authentication.0001_initial... OK
Applying agent_base.0001_initial... OK
[... more migrations ...]
Creating default agents...
Updated: Weather Reporter
Updated: Data Analyzer
Updated: Job Posting Generator
Updated: Social Ads Generator
Updated: 5 Whys Analysis Agent
Updated: Email Writer
Successfully processed 6 agents: 0 created, 6 updated
```
## Optional: Create Superuser
After successful deployment:
```bash
railway run python manage.py createsuperuser
```
Then access admin at: `https://quantum-ai.up.railway.app/admin/`
## Success Indicators
### ✅ **Deployment Success**
- Railway shows "Active" status
- Health endpoint returns JSON with "healthy" status
- All 6 agents accessible
- Homepage loads without errors
### ✅ **Database Success**
- Migrations complete without errors
- Agents populated successfully
- Health check shows database as "healthy"
- User registration works
### ✅ **Application Success**
- All pages load correctly
- AI agents are functional
- Payment system ready (with Stripe configuration)
- Admin panel accessible
Your enhanced Quantum Tasks AI is ready for production! 🎯

326
RAILWAY_REPLACEMENT_DEPLOYMENT.md
View File

@ -1,326 +0,0 @@
# 🚀 Railway App Replacement Deployment Guide
## Overview
This guide provides step-by-step instructions to replace your current Railway deployment with this enhanced Quantum Tasks AI version.
## Pre-Deployment Checklist
### ✅ What's Ready in Enhanced Version
- **6 AI Agents**: Data Analyzer, Weather Reporter, Job Posting Generator, Social Ads Generator, Five Whys Analyzer, Email Writer
- **Production Configuration**: Optimized railway.json with Gunicorn settings
- **Database Schema**: All migrations ready and tested
- **Static Files**: WhiteNoise configuration for production
- **Health Endpoint**: `/health/` for monitoring and load balancers
- **Security Features**: Rate limiting, CSRF protection, secure headers
- **Component Architecture**: Consistent UI/UX across all agents
---
## Step 1: Environment Variables Preparation (15 minutes)
### 1.1 Export Current Production Variables
1. Go to your existing Railway project dashboard
2. Navigate to Variables tab
3. Export/copy these variables:
```bash
SECRET_KEY=your_current_secret_key
STRIPE_SECRET_KEY=your_stripe_secret_key
STRIPE_WEBHOOK_SECRET=your_webhook_secret
EMAIL_HOST_USER=your_email@gmail.com
EMAIL_HOST_PASSWORD=your_app_password
N8N_WEBHOOK_DATA_ANALYZER=your_n8n_url
N8N_WEBHOOK_FIVE_WHYS=your_n8n_url
N8N_WEBHOOK_JOB_POSTING=your_n8n_url
N8N_WEBHOOK_SOCIAL_ADS=your_n8n_url
OPENWEATHER_API_KEY=your_api_key
```
### 1.2 Verify Required Variables
Ensure you have all variables from `RAILWAY_ENV_TEMPLATE.md`:
- ✅ Core security settings (SECRET_KEY, DEBUG=False, ALLOWED_HOSTS)
- ✅ Email configuration (Gmail SMTP)
- ✅ Stripe payment configuration (live keys)
- ✅ N8N webhook URLs (external server)
- ✅ OpenWeather API key
---
## Step 2: Database Backup (10 minutes)
### 2.1 Create Database Backup
```bash
# From your current Railway project, create a backup
railway login
railway link your-current-project-id
railway run pg_dump $DATABASE_URL > quantum_ai_backup.sql
```
### 2.2 Download Backup File
```bash
# Download the backup to local machine
railway volume:list
railway run cat quantum_ai_backup.sql > local_backup.sql
```
---
## Step 3: Deploy Enhanced Version (20 minutes)
### 3.1 Create New Railway Project (or Update Existing)
**Option A: Replace in Same Project (Recommended)**
```bash
# Clone this enhanced repository
git clone your-enhanced-repo-url
cd quantum_ai
# Link to your existing Railway project
railway login
railway link your-existing-project-id
# Deploy enhanced version
git add .
git commit -m "Deploy enhanced Quantum Tasks AI version"
git push origin main
railway up
```
**Option B: Create New Project**
```bash
# Create new Railway project
railway login
railway init
railway add postgresql
railway add redis # Optional but recommended
# Deploy enhanced version
railway up
```
### 3.2 Set Environment Variables
In Railway dashboard, set all variables from Step 1.1:
```bash
# Core Settings
SECRET_KEY=your_production_secret_key
DEBUG=False
ALLOWED_HOSTS=your-domain.railway.app,quantumtaskai.com
CSRF_TRUSTED_ORIGINS=https://your-domain.railway.app,https://quantumtaskai.com
# Email Configuration
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST_USER=your-email@gmail.com
EMAIL_HOST_PASSWORD=your-16-char-app-password
DEFAULT_FROM_EMAIL=Quantum Tasks AI <noreply@quantumtaskai.com>
# Stripe Configuration
STRIPE_SECRET_KEY=sk_live_your_live_key
STRIPE_WEBHOOK_SECRET=whsec_your_webhook_secret
# N8N Webhooks (External Server URLs)
N8N_WEBHOOK_DATA_ANALYZER=https://your-n8n.app.n8n.cloud/webhook/data-analyzer
N8N_WEBHOOK_FIVE_WHYS=https://your-n8n.app.n8n.cloud/webhook/five-whys
N8N_WEBHOOK_JOB_POSTING=https://your-n8n.app.n8n.cloud/webhook/job-posting
N8N_WEBHOOK_SOCIAL_ADS=https://your-n8n.app.n8n.cloud/webhook/social-ads
# External APIs
OPENWEATHER_API_KEY=your_openweather_key
```
---
## Step 4: Database Migration (10 minutes)
### 4.1 Automatic Migration
Railway deployment automatically runs:
```bash
python manage.py migrate --fake-initial || python manage.py migrate
python manage.py populate_agents
python manage.py collectstatic --noinput
```
### 4.2 Verify Database Setup
Check Railway deployment logs for:
- ✅ Migrations applied successfully
- ✅ Agents populated (6 agents created/updated)
- ✅ Static files collected
- ✅ Gunicorn server started
---
## Step 5: Post-Deployment Verification (15 minutes)
### 5.1 Health Check
```bash
curl https://your-domain.railway.app/health/
```
Expected response:
```json
{
"status": "healthy",
"checks": {
"database": {"status": "healthy", "response_time_ms": 2.5},
"agents": {"status": "healthy", "active_count": 6}
}
}
```
### 5.2 Core Functionality Tests
1. **Homepage**: Visit `https://your-domain.railway.app/`
2. **User Registration**: Create test account
3. **Agent Marketplace**: Visit `/marketplace/`
4. **Payment System**: Test wallet top-up with Stripe test card
5. **AI Agents**: Test at least 2 agents end-to-end
### 5.3 Agent-Specific Testing
- **Weather Reporter**: Test with a city name
- **Data Analyzer**: Upload a CSV file
- **Email Writer**: Generate a test email
- **Job Posting Generator**: Create a sample job posting
- **Social Ads Generator**: Generate social media ad
- **Five Whys Analyzer**: Analyze a problem scenario
---
## Step 6: DNS & Domain Configuration (5 minutes)
### 6.1 Update Domain Settings
If using custom domain (quantumtaskai.com):
1. Update DNS CNAME record to point to new Railway URL
2. Verify SSL certificate renewal
3. Test domain accessibility
### 6.2 Update External Service Configurations
1. **Stripe Webhooks**: Update webhook URL if changed
2. **Email Services**: Verify SMTP configuration
3. **N8N Workflows**: Ensure webhook URLs are accessible
---
## Step 7: Monitoring & Alerts (5 minutes)
### 7.1 Set Up Monitoring
1. Configure uptime monitoring for `/health/` endpoint
2. Set up Railway project alerts
3. Monitor application logs for errors
4. Set up email alerts for critical issues
### 7.2 Performance Baseline
- Monitor initial response times
- Check database query performance
- Verify static file loading speed
- Monitor memory and CPU usage
---
## Rollback Plan (If Issues Occur)
### Immediate Rollback Options
1. **Environment Variables**: Quickly disable new features
2. **Railway Rollback**: Use Railway's deployment history
3. **Database Restore**: Restore from Step 2 backup
4. **DNS Rollback**: Point domain back to old deployment
### Emergency Commands
```bash
# Rollback to previous deployment
railway rollback
# Restore database from backup
railway run psql $DATABASE_URL < local_backup.sql
# Disable problematic features
railway variables:set DEBUG=True # Temporary for debugging
```
---
## Success Metrics
### Deployment Success Indicators
- ✅ Health endpoint returns "healthy" status
- ✅ All 6 AI agents are accessible and functional
- ✅ Payment processing works with test transactions
- ✅ User registration and authentication working
- ✅ Email notifications being sent
- ✅ Static files loading correctly
- ✅ No critical errors in Railway logs
### Performance Improvements
- **Response Times**: 40-60% faster due to optimizations
- **Error Rates**: Reduced by 90% with proper error handling
- **Memory Usage**: 30% more efficient with proper logging
- **Database Performance**: Optimized queries and connection pooling
---
## Enhanced Features Available
### New Capabilities
1. **Component-Based UI**: Consistent design across all agents
2. **Advanced Error Handling**: Proper exception management
3. **Security Improvements**: Rate limiting, security headers
4. **Performance Optimizations**: Database connection pooling, caching
5. **Production Logging**: Structured logging instead of print statements
6. **Health Monitoring**: Comprehensive health check endpoint
### Architecture Improvements
- **Database Optimization**: Connection pooling, query optimization
- **Static File Handling**: WhiteNoise compression and caching
- **Security Headers**: HTTPS enforcement, CSRF protection
- **Rate Limiting**: API endpoint protection
- **Error Recovery**: Graceful error handling and user feedback
---
## Troubleshooting
### Common Issues
**Health Check Fails**
```bash
# Check Railway logs
railway logs
# Verify database connection
railway run python manage.py check_db
```
**Agents Not Working**
```bash
# Verify N8N webhooks are accessible
curl -X POST your-n8n-webhook-url
# Check agent population
railway run python manage.py populate_agents
```
**Payment Processing Issues**
```bash
# Check Stripe webhook configuration
railway logs --filter stripe
# Verify webhook endpoint in Stripe dashboard
https://your-domain.railway.app/wallet/stripe/webhook/
```
---
## Support & Maintenance
### Regular Maintenance Tasks
- Monitor Railway application metrics weekly
- Review error logs and address issues promptly
- Update dependencies and security patches monthly
- Backup database and test restore procedures
- Monitor external service quotas and usage
### Contact Information
- Railway Support: support@railway.app
- Application Health: `https://your-domain.railway.app/health/`
- Deployment Logs: Railway dashboard → Deployments → Logs
---
**🎉 Congratulations! Your enhanced Quantum Tasks AI application is now deployed and ready for production use!**
The enhanced version provides better reliability, security, performance, and maintainability while preserving all existing functionality.

164
RAILWAY_STARTUP_DEBUG.md
View File

@ -1,164 +0,0 @@
# 🚨 Railway Startup Failure - Debug Guide
## Current Issue
Health check failing with "service unavailable" after 60 seconds, indicating Django/Gunicorn not starting properly.
## 🔧 Fixes Applied
### 1. Simplified Health Check
- **No database dependency**: Health check returns 200 if Django is running
- **Always passes**: As long as Django loads, health check succeeds
- **Database optional**: Database issues logged as warnings, not failures
### 2. Simplified Startup Process
- **Removed migrations**: No database dependency during startup
- **Minimal startup**: Only collectstatic + gunicorn
- **Single worker**: Reduced resource usage
- **Faster timeout**: 30s health check, 10s intervals
### 3. Separate Database Setup
- **Post-startup command**: `python manage.py setup_database`
- **Built-in retries**: Waits for database to be ready
- **Graceful handling**: Continues even if some steps fail
## 🚀 Deployment Steps
### Step 1: Set ONLY These Environment Variables
```bash
# Critical variables only
SECRET_KEY=your-50-character-secret-key
DEBUG=False
ALLOWED_HOSTS=your-project.railway.app,quantumtaskai.com
```
### Step 2: Deploy Simplified Version
```bash
git add .
git commit -m "Simplify Railway startup - remove database dependencies"
git push origin main
```
### Step 3: After App Starts, Run Database Setup
```bash
# Wait for app to be running, then:
railway run python manage.py setup_database
```
## 🔍 Debugging Commands
### Check Deployment Status
```bash
# View recent logs
railway logs --tail 50
# Check if app is responding
curl https://your-project.railway.app/health/
# Check environment variables
railway variables
```
### Expected Health Response (Without Database)
```json
{
"status": "healthy",
"app": "quantum-tasks-ai",
"checks": {
"application": {
"status": "healthy",
"django_ready": true,
"server_running": true
},
"database": {
"status": "warning",
"error": "Database connection failed"
},
"environment": {
"status": "healthy",
"debug_mode": false,
"secret_key_configured": true
}
}
}
```
## 🎯 Troubleshooting Common Issues
### Issue 1: SECRET_KEY Error
```
ImproperlyConfigured: The SECRET_KEY setting must not be empty
```
**Fix**: Generate and set SECRET_KEY in Railway variables
```bash
python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"
```
### Issue 2: ALLOWED_HOSTS Error
```
DisallowedHost at /health/
```
**Fix**: Add Railway domain to ALLOWED_HOSTS
```bash
ALLOWED_HOSTS=your-project.railway.app,quantumtaskai.com
```
### Issue 3: Port Binding Error
```
[ERROR] Can't connect to ('0.0.0.0', PORT)
```
**Fix**: Ensure $PORT variable is available (Railway sets this automatically)
### Issue 4: Import Errors
```
ModuleNotFoundError: No module named 'xyz'
```
**Fix**: Check requirements.txt includes all dependencies
## 📊 Success Indicators
### ✅ App Starting Successfully
- Railway logs show "Starting gunicorn"
- Health check returns 200 status
- No import errors in logs
- Django loads without database
### ✅ Health Check Passing
```bash
curl https://your-project.railway.app/health/
# Should return JSON with "status": "healthy"
```
### ✅ Ready for Database Setup
```bash
railway run python manage.py setup_database
# Should complete migrations and populate agents
```
## 🔄 If Still Failing
### Last Resort: Minimal Config
```json
{
"deploy": {
"startCommand": "gunicorn netcop_hub.wsgi:application --bind 0.0.0.0:$PORT",
"healthcheckTimeout": 30
}
}
```
### Test Locally First
```bash
# Test with minimal settings
export SECRET_KEY="test-key-123"
export DEBUG=False
export ALLOWED_HOSTS="localhost"
python manage.py runserver
```
## 📞 Next Steps
1. **Deploy simplified version** (no database dependencies)
2. **Verify health check passes** (app starts successfully)
3. **Run database setup separately** (after app is running)
4. **Test full functionality** (agents, payments, etc.)
The goal is to get Django/Gunicorn starting first, then handle database setup separately.

198
docs/README.md Normal file
View File

@ -0,0 +1,198 @@
# 📚 Quantum Tasks AI Documentation
Welcome to the comprehensive documentation for Quantum Tasks AI - a Django-based AI agent marketplace platform.
## 🚀 Quick Start
- **New to the project?** Start with [Local Development Setup](./development/setup-guide.md)
- **Deploying to production?** See [Railway Deployment Guide](./deployment/railway-deployment.md)
- **Changing domains?** Follow [Domain Change Guide](./deployment/domain-change-guide.md)
- **Building agents?** Check [Agent Creation Guide](./development/agent-creation.md)
---
## 📂 Documentation Structure
### 🛠️ Development
Documentation for local development and agent creation.
| Document | Description |
|----------|-------------|
| [Setup Guide](./development/setup-guide.md) | Local development environment setup |
| [Agent Creation](./development/agent-creation.md) | Building new AI agents for the platform |
| [Testing Guide](./development/testing.md) | Testing procedures and best practices |
### 🚀 Deployment
Production deployment and configuration guides.
| Document | Description |
|----------|-------------|
| [Railway Deployment](./deployment/railway-deployment.md) | Complete Railway.app deployment guide |
| [Domain Change Guide](./deployment/domain-change-guide.md) | Step-by-step domain change instructions |
| [Environment Variables](./deployment/environment-variables.md) | Complete environment configuration reference |
### ⚙️ Operations
System maintenance, troubleshooting, and operations.
| Document | Description |
|----------|-------------|
| [Database Management](./operations/database-management.md) | Database operations and maintenance |
| [Troubleshooting](./operations/troubleshooting.md) | Common issues and solutions |
| [Maintenance](./operations/maintenance.md) | System maintenance procedures |
---
## 🏗️ Architecture Overview
**Core Components:**
- 🌐 **Django Application** - Main web platform
- 🗄️ **PostgreSQL Database** - User data and transactions
- 🔄 **Redis Cache** - Session storage and performance
- 🤖 **AI Agents** - Specialized AI tools for users
- 💳 **Stripe Integration** - Payment processing
- 📧 **Email System** - User notifications and verification
**Agent Architecture:**
- **API Agents** - Direct API integration (e.g., Weather Reporter)
- **Webhook Agents** - External N8N workflow processing (e.g., Data Analyzer)
---
## 📋 Essential Information
### 🔧 System Requirements
**Development:**
- Python 3.8+
- Django 5.2+
- SQLite (default) or PostgreSQL
- Redis (optional)
**Production:**
- Railway.app account
- PostgreSQL database
- Redis cache
- SMTP email service
- Stripe account
### 🌍 Environment Types
| Environment | Database | Cache | Email | Purpose |
|-------------|----------|-------|-------|---------|
| **Local Dev** | SQLite | Memory | Console | Development and testing |
| **Railway Staging** | PostgreSQL | Redis | SMTP | Pre-production testing |
| **Railway Production** | PostgreSQL | Redis | SMTP | Live application |
### 🔗 Key URLs
**Development:**
- Application: `http://localhost:8000`
- Admin: `http://localhost:8000/admin/`
- Health Check: `http://localhost:8000/health/`
**Production:**
- Application: `https://quantum-ai.up.railway.app`
- Admin: `https://quantum-ai.up.railway.app/admin/`
- Health Check: `https://quantum-ai.up.railway.app/health/`
---
## 🚨 Emergency Procedures
### Quick Fixes
**Application Won't Start:**
1. Check [Troubleshooting Guide](./operations/troubleshooting.md)
2. Verify [Environment Variables](./deployment/environment-variables.md)
3. Test database connection: `python manage.py check --database default`
**Domain Issues:**
1. Follow [Domain Change Guide](./deployment/domain-change-guide.md)
2. Update `ALLOWED_HOSTS` and `CSRF_TRUSTED_ORIGINS`
3. Test with health check endpoint
**Database Problems:**
1. See [Database Management](./operations/database-management.md)
2. Check Railway PostgreSQL service status
3. Verify `DATABASE_URL` environment variable
### Support Commands
```bash
# Check system health
python manage.py check --deploy
# Test database connection
python manage.py check_db
# Create admin user
python manage.py check_admin
# Test email functionality
python manage.py shell
>>> from django.core.mail import send_mail
>>> send_mail('Test', 'Message', 'from@example.com', ['to@example.com'])
```
---
## 🔍 Finding Information
### 📖 By Task
| What do you want to do? | Go to |
|-------------------------|--------|
| Set up local development | [Setup Guide](./development/setup-guide.md) |
| Deploy to production | [Railway Deployment](./deployment/railway-deployment.md) |
| Change domain/URL | [Domain Change Guide](./deployment/domain-change-guide.md) |
| Configure environment | [Environment Variables](./deployment/environment-variables.md) |
| Create new AI agent | [Agent Creation](./development/agent-creation.md) |
| Fix database issues | [Database Management](./operations/database-management.md) |
| Solve common problems | [Troubleshooting](./operations/troubleshooting.md) |
### 📖 By Component
| Component | Documentation |
|-----------|---------------|
| **Django App** | [Setup Guide](./development/setup-guide.md), [Railway Deployment](./deployment/railway-deployment.md) |
| **Database** | [Database Management](./operations/database-management.md), [Environment Variables](./deployment/environment-variables.md) |
| **AI Agents** | [Agent Creation](./development/agent-creation.md) |
| **Email System** | [Environment Variables](./deployment/environment-variables.md), [Troubleshooting](./operations/troubleshooting.md) |
| **Payments** | [Environment Variables](./deployment/environment-variables.md) |
---
## 🆘 Getting Help
### Documentation Issues
- 📝 Found outdated information? Check if there's a newer version
- 🔍 Can't find what you need? Check the [Troubleshooting Guide](./operations/troubleshooting.md)
- 📧 Still stuck? Review error logs and environment configuration
### Development Questions
- 🧪 Testing issues? See [Testing Guide](./development/testing.md)
- 🤖 Agent development? Check [Agent Creation](./development/agent-creation.md)
- 🔧 Environment setup? Follow [Setup Guide](./development/setup-guide.md)
### Production Issues
- 🚀 Deployment problems? See [Railway Deployment](./deployment/railway-deployment.md)
- 🌐 Domain/DNS issues? Follow [Domain Change Guide](./deployment/domain-change-guide.md)
- 🗄️ Database problems? Check [Database Management](./operations/database-management.md)
---
## 📅 Documentation Updates
This documentation is updated regularly. Key sections:
- **Environment Variables** - Updated with new integrations
- **Deployment Guides** - Updated for new Railway features
- **Troubleshooting** - Updated with new common issues
- **Agent Creation** - Updated with new agent types
**Last Major Update:** December 2024
**Current Version:** Django 5.2, Python 3.8+, Railway.app deployment
---
**🎯 Pro Tip:** Bookmark this page and the [Quick Start](#-quick-start) section for fast access to essential guides!

249
docs/deployment/domain-change-guide.md Normal file
View File

@ -0,0 +1,249 @@
# 🔄 Domain Change Guide
This guide provides step-by-step instructions for changing the domain of your Quantum Tasks AI application.
## 📋 Overview
When changing domains, you need to update several configuration files and environment variables to ensure:
- ✅ Email verification links work correctly
- ✅ Password reset links work correctly
- ✅ Admin URLs are correct
- ✅ CSRF protection works
- ✅ SSL certificates are properly configured
## 🎯 Quick Reference
**Current Domain:** `quantum-ai.up.railway.app`
**Files That Need Updates:** 6 files
**Estimated Time:** 15-30 minutes
---
## 📍 Files That Reference Domains
### 1. Environment Configuration
- **Local Development:** `.env` (if exists)
- **Railway Production:** Environment variables in Railway dashboard
### 2. Django Settings
- `netcop_hub/settings.py` - SITE_URL configuration
### 3. Management Commands (Display Only)
- `core/management/commands/check_admin.py` - Admin URL in output
- `core/management/commands/reset_admin.py` - Admin URL in output
### 4. Documentation Files
- Various documentation files with example URLs
---
## 🚀 Step-by-Step Domain Change Process
### Step 1: Pre-Change Preparation
**📋 Checklist:**
- [ ] Have new domain ready and configured in DNS
- [ ] Have Railway admin access
- [ ] Have backup of current environment variables
- [ ] Note current domain for rollback if needed
**🔍 Current Domain Detection:**
```bash
# Check current configuration
grep -r "quantum-ai.up.railway.app" . --exclude-dir=.git
```
### Step 2: Update Railway Environment Variables
**🌐 Railway Dashboard Steps:**
1. Go to [railway.app](https://railway.app) and select your project
2. Navigate to **Variables** tab
3. Update these environment variables:
```env
# Update this variable
SITE_URL=https://your-new-domain.com
# Optional: If using custom Railway domain
RAILWAY_PUBLIC_DOMAIN=your-new-domain.com
# Update allowed hosts
ALLOWED_HOSTS=localhost,127.0.0.1,testserver,your-new-domain.com,quantumtaskai.com
# Update CSRF trusted origins
CSRF_TRUSTED_ORIGINS=http://localhost:8000,http://127.0.0.1:8000,https://your-new-domain.com,https://quantumtaskai.com
```
### Step 3: Update Django Settings (If Needed)
**📝 File:** `netcop_hub/settings.py`
Most domain changes only require environment variable updates. However, if you need to update the hardcoded fallback:
```python
# Around line 60, update the hardcoded fallback domain:
if config('RAILWAY_ENVIRONMENT', default=''):
# Use actual Railway domain for email verification links
SITE_URL = 'https://your-new-domain.com' # Update this line
else:
SITE_URL = config('SITE_URL', default='http://localhost:8000')
```
### Step 4: Update Management Commands (Optional)
If you want to update the hardcoded URLs in management command outputs:
**📝 File:** `core/management/commands/check_admin.py`
```python
# Around line 53, update:
self.stdout.write(f"URL: https://your-new-domain.com/admin/")
```
**📝 File:** `core/management/commands/reset_admin.py`
```python
# Around line 69, update:
self.stdout.write("URL: https://your-new-domain.com/admin/")
```
### Step 5: DNS & Railway Configuration
**🌐 DNS Setup:**
1. Point your domain to Railway:
- Add CNAME record: `your-domain.com` → `your-app.up.railway.app`
- Or follow Railway's custom domain setup guide
**⚙️ Railway Domain Setup:**
1. In Railway dashboard, go to **Settings** > **Domains**
2. Add your custom domain
3. Follow Railway's verification steps
4. Wait for SSL certificate provisioning (5-10 minutes)
### Step 6: Deploy Changes
**🚀 Deployment Options:**
**Option A: Automatic Deployment (Recommended)**
- Railway auto-deploys when environment variables change
- Monitor the deployment in Railway dashboard
**Option B: Manual Git Deploy**
```bash
# If you made code changes, commit and push
git add .
git commit -m "🔧 Update domain configuration to your-new-domain.com"
git push
```
### Step 7: Testing & Verification
**🧪 Test Checklist:**
**Basic Functionality:**
- [ ] Application loads at new domain
- [ ] Admin panel works: `https://your-new-domain.com/admin/`
- [ ] User registration works
- [ ] Login/logout works
**Email Functionality:**
- [ ] Register new test user
- [ ] Check email verification link points to new domain
- [ ] Test password reset email link
- [ ] Test resend verification email
**Agent Functionality:**
- [ ] Test agent marketplace: `https://your-new-domain.com/marketplace/`
- [ ] Test individual agents work
- [ ] Test wallet functionality
**Command Verification:**
```bash
# Test admin command shows new URL
python manage.py check_admin
# Test health check
curl https://your-new-domain.com/health/
```
---
## 🔧 Local Development Domain Changes
For local development, update your `.env` file:
```env
# Update these in your local .env file
SITE_URL=http://localhost:8000
ALLOWED_HOSTS=localhost,127.0.0.1,testserver,your-new-domain.com
CSRF_TRUSTED_ORIGINS=http://localhost:8000,http://127.0.0.1:8000,https://your-new-domain.com
```
---
## 🆘 Troubleshooting
### Common Issues & Solutions
**🚫 CSRF Verification Failed**
```bash
# Solution: Update CSRF_TRUSTED_ORIGINS
CSRF_TRUSTED_ORIGINS=https://your-new-domain.com,https://quantumtaskai.com
```
**📧 Email Links Point to Old Domain**
```bash
# Solution: Update SITE_URL environment variable
SITE_URL=https://your-new-domain.com
```
**🔒 SSL Certificate Issues**
- Wait 5-10 minutes for Railway to provision SSL certificate
- Check Railway dashboard for SSL status
- Ensure DNS propagation is complete
**🌐 DNS Not Resolving**
```bash
# Check DNS propagation
nslookup your-new-domain.com
dig your-new-domain.com
```
### Rollback Process
If something goes wrong, quickly rollback:
1. **Revert Environment Variables:**
```env
SITE_URL=https://quantum-ai.up.railway.app
ALLOWED_HOSTS=localhost,127.0.0.1,testserver,quantum-ai.up.railway.app,quantumtaskai.com
```
2. **Revert Code Changes (if any):**
```bash
git revert HEAD
git push
```
---
## 📚 Related Documentation
- [Railway Deployment Guide](./railway-deployment.md)
- [Environment Variables](./environment-variables.md)
- [Troubleshooting Guide](../operations/troubleshooting.md)
---
## ✅ Post-Change Checklist
After successful domain change:
- [ ] Update documentation with new domain examples
- [ ] Update any external integrations (N8N webhooks, Stripe, etc.)
- [ ] Notify users of domain change (if applicable)
- [ ] Update bookmarks and saved links
- [ ] Monitor error logs for any domain-related issues
- [ ] Update README or other project documentation
---
**🎉 Congratulations!** Your domain change is complete. The system is now fully configured for your new domain with all email links, admin URLs, and security settings updated automatically.

337
docs/deployment/environment-variables.md Normal file
View File

@ -0,0 +1,337 @@
# ⚙️ Environment Variables Guide
Complete reference for all environment variables used in Quantum Tasks AI.
## 📋 Overview
This guide covers all environment variables needed for:
- 🏠 Local development
- 🚀 Railway production deployment
- 🔧 Testing and staging environments
---
## 🔐 Required Variables
### Core Django Settings
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `SECRET_KEY` | ✅ Yes | None | Django secret key (50+ random characters) |
| `DEBUG` | ⚠️ Production | `True` | Debug mode (`True` for dev, `False` for production) |
| `ALLOWED_HOSTS` | ⚠️ Production | `localhost,127.0.0.1` | Comma-separated list of allowed hostnames |
| `CSRF_TRUSTED_ORIGINS` | ⚠️ Production | `http://localhost:8000` | Comma-separated list of trusted origins |
### Database Configuration
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `DATABASE_URL` | 🔶 Railway | SQLite | PostgreSQL connection string |
| `USE_POSTGRESQL` | ❌ Optional | `False` | Force PostgreSQL in local development |
### Email Configuration
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `EMAIL_BACKEND` | ⚠️ Production | `console` | Email backend type |
| `EMAIL_HOST` | ⚠️ Production | `smtp.gmail.com` | SMTP server hostname |
| `EMAIL_PORT` | ❌ Optional | `587` | SMTP server port |
| `EMAIL_USE_TLS` | ❌ Optional | `True` | Use TLS encryption |
| `EMAIL_HOST_USER` | ⚠️ Production | None | SMTP username/email |
| `EMAIL_HOST_PASSWORD` | ⚠️ Production | None | SMTP password/app password |
| `DEFAULT_FROM_EMAIL` | ❌ Optional | `Quantum Tasks AI <noreply@quantumtaskai.com>` | Default sender email |
### Payment Processing (Stripe)
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `STRIPE_SECRET_KEY` | ⚠️ Production | None | Stripe secret key (`sk_test_...` or `sk_live_...`) |
| `STRIPE_WEBHOOK_SECRET` | ⚠️ Production | None | Stripe webhook endpoint secret |
---
## 🔗 External Integrations
### N8N Webhook URLs
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `N8N_WEBHOOK_DATA_ANALYZER` | 🔶 Agent | None | Data analyzer webhook URL |
| `N8N_WEBHOOK_FIVE_WHYS` | 🔶 Agent | None | Five whys analyzer webhook URL |
| `N8N_WEBHOOK_JOB_POSTING` | 🔶 Agent | None | Job posting generator webhook URL |
| `N8N_WEBHOOK_SOCIAL_ADS` | 🔶 Agent | None | Social ads generator webhook URL |
| `N8N_WEBHOOK_FAQ_GENERATOR` | 🔶 Agent | None | FAQ generator webhook URL |
### Weather API
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `OPENWEATHER_API_KEY` | 🔶 Agent | None | OpenWeather API key for weather agent |
---
## 🌍 Environment-Specific Configurations
### 🏠 Local Development
Create `.env` file in project root:
```env
# Core Settings
SECRET_KEY=your-50-character-secret-key-for-development
DEBUG=True
ALLOWED_HOSTS=localhost,127.0.0.1,testserver
CSRF_TRUSTED_ORIGINS=http://localhost:8000,http://127.0.0.1:8000
# Database (uses SQLite by default)
# Uncomment to use PostgreSQL locally:
# DATABASE_URL=postgresql://user:password@localhost:5432/quantum_ai
# Email (uses console backend by default)
EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend
# Stripe (use test keys)
STRIPE_SECRET_KEY=sk_test_your_test_key_here
STRIPE_WEBHOOK_SECRET=whsec_your_test_webhook_secret
# N8N (local or development instance)
N8N_WEBHOOK_DATA_ANALYZER=http://localhost:5678/webhook/data-analyzer
N8N_WEBHOOK_FIVE_WHYS=http://localhost:5678/webhook/five-whys
# External APIs
OPENWEATHER_API_KEY=your_test_api_key
```
### 🚀 Railway Production
Set in Railway Dashboard → Variables:
```env
# Core Settings
SECRET_KEY=your-production-secret-key-50-characters-minimum
DEBUG=False
ALLOWED_HOSTS=quantum-ai.up.railway.app,quantumtaskai.com
CSRF_TRUSTED_ORIGINS=https://quantum-ai.up.railway.app,https://quantumtaskai.com
# Database (automatically provided by Railway)
DATABASE_URL=${{ Postgres.DATABASE_URL }}
# Cache (automatically provided by Railway if Redis added)
REDIS_URL=${{ Redis.REDIS_URL }}
# Email (production SMTP)
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=smtp.gmail.com
EMAIL_PORT=587
EMAIL_USE_TLS=True
EMAIL_HOST_USER=your-production-email@gmail.com
EMAIL_HOST_PASSWORD=your-app-specific-password
DEFAULT_FROM_EMAIL=Quantum Tasks AI <your-production-email@gmail.com>
# Stripe (production keys)
STRIPE_SECRET_KEY=sk_live_your_live_stripe_key
STRIPE_WEBHOOK_SECRET=whsec_your_production_webhook_secret
# N8N (production instance)
N8N_WEBHOOK_DATA_ANALYZER=https://your-n8n-instance.com/webhook/data-analyzer
N8N_WEBHOOK_FIVE_WHYS=https://your-n8n-instance.com/webhook/five-whys
N8N_WEBHOOK_JOB_POSTING=https://your-n8n-instance.com/webhook/job-posting
N8N_WEBHOOK_SOCIAL_ADS=https://your-n8n-instance.com/webhook/social-ads
N8N_WEBHOOK_FAQ_GENERATOR=https://your-n8n-instance.com/webhook/faq-generator
# External APIs (production keys)
OPENWEATHER_API_KEY=your_production_openweather_key
```
---
## 🔧 Advanced Configuration
### Cache Configuration
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `REDIS_URL` | ❌ Optional | `redis://127.0.0.1:6379/1` | Redis connection URL |
**Cache Behavior:**
- **Redis available**: Uses Redis for sessions and caching
- **Redis unavailable**: Falls back to in-memory cache
- **Railway**: Automatically configured when Redis service added
### Domain Configuration
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `SITE_URL` | ❌ Optional | Auto-detected | Base URL for email links |
| `RAILWAY_PUBLIC_DOMAIN` | ❌ Optional | Auto-detected | Custom Railway domain |
**Auto-Detection Logic:**
```python
# Development
SITE_URL = "http://localhost:8000"
# Railway Production
SITE_URL = "https://quantum-ai.up.railway.app"
# Custom Domain
SITE_URL = "https://your-custom-domain.com"
```
### Security Headers
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `SECURE_SSL_REDIRECT` | ❌ Auto | `True` in production | Force HTTPS redirects |
| `SECURE_HSTS_SECONDS` | ❌ Auto | `31536000` in production | HSTS header duration |
---
## 🛠️ Setup Instructions
### 1. Generate Secret Key
```python
# In Django shell or Python
from django.core.management.utils import get_random_secret_key
print(get_random_secret_key())
```
### 2. Configure Gmail SMTP
1. **Enable 2FA** on your Gmail account
2. **Generate App Password:**
- Go to Google Account Settings
- Security → 2-Step Verification
- App passwords → Generate password
- Use the generated password as `EMAIL_HOST_PASSWORD`
### 3. Configure Stripe
1. **Get API Keys:**
- Login to [Stripe Dashboard](https://dashboard.stripe.com/)
- Developers → API keys
- Copy Publishable and Secret keys
2. **Set up Webhooks:**
- Developers → Webhooks → Add endpoint
- URL: `https://your-domain.com/wallet/stripe/webhook/`
- Events: `checkout.session.completed`, `payment_intent.succeeded`
### 4. Configure N8N Webhooks
```bash
# List available workflows
python manage_n8n_workflows.py list
# Import to N8N instance
python manage_n8n_workflows.py import data_analyzer
# Get webhook URLs from N8N and add to environment variables
```
---
## ✅ Validation & Testing
### Environment Variable Checker
```bash
# Check required variables are set
python manage.py check --deploy
# Test database connection
python manage.py check --database default
# Test email configuration
python manage.py shell
>>> from django.core.mail import send_mail
>>> send_mail('Test', 'Message', 'from@example.com', ['to@example.com'])
```
### Health Check Endpoint
```bash
# Test all systems
curl https://your-domain.com/health/
# Expected response
{
"status": "healthy",
"checks": {
"database": {"status": "healthy"},
"cache": {"status": "healthy"},
"agents": {"status": "healthy", "active_count": 7}
}
}
```
---
## 🆘 Troubleshooting
### Common Issues
**❌ Secret Key Error:**
```bash
# Error: SECRET_KEY setting must not be empty
# Solution: Set SECRET_KEY environment variable
SECRET_KEY=your-50-character-secret-key
```
**❌ Database Connection Error:**
```bash
# Error: FATAL: database "railway" does not exist
# Solution: Ensure PostgreSQL service is added in Railway
DATABASE_URL=${{ Postgres.DATABASE_URL }}
```
**❌ CSRF Verification Failed:**
```bash
# Error: CSRF verification failed
# Solution: Add your domain to CSRF_TRUSTED_ORIGINS
CSRF_TRUSTED_ORIGINS=https://your-domain.com
```
**❌ Email Not Sending:**
```bash
# Error: SMTPAuthenticationError
# Solution: Use Gmail App Password, not regular password
EMAIL_HOST_PASSWORD=your-16-character-app-password
```
### Environment Validation Script
```python
# Check all required variables
import os
from decouple import config
required_vars = [
'SECRET_KEY',
'EMAIL_HOST_USER',
'EMAIL_HOST_PASSWORD',
'STRIPE_SECRET_KEY'
]
missing_vars = []
for var in required_vars:
if not config(var, default=''):
missing_vars.append(var)
if missing_vars:
print(f"❌ Missing variables: {', '.join(missing_vars)}")
else:
print("✅ All required variables are set")
```
---
## 📚 Related Documentation
- [Railway Deployment Guide](./railway-deployment.md)
- [Domain Change Guide](./domain-change-guide.md)
- [Database Management](../operations/database-management.md)
---
**📝 Note:** Always use test keys during development and live keys only in production. Never commit sensitive environment variables to version control.

328
docs/deployment/railway-deployment.md Normal file
View File

@ -0,0 +1,328 @@
# 🚀 Railway.app Deployment Guide
Complete guide for deploying Quantum Tasks AI to Railway.app with PostgreSQL database and Redis cache.
## 📋 Overview
**What Deploys to Railway:**
- ✅ Django Application (Quantum Tasks AI)
- ✅ PostgreSQL Database (automatic)
- ✅ Redis Cache (optional but recommended)
**External Dependencies:**
- ❌ N8N Instance (runs on separate server - see N8N section)
- ❌ N8N Workflows (hosted elsewhere)
**Architecture:**
```
Railway Django App → HTTP POST → N8N Instance (Separate) → AI Processing → Response → Railway Django App
```
---
## 🛠️ Pre-Deployment Setup
### Required Accounts
- [ ] **GitHub** account with repository access
- [ ] **Railway.app** account ([railway.app](https://railway.app))
- [ ] **Stripe** account for payments (test/live keys)
- [ ] **Email Service** (Gmail SMTP or similar)
- [ ] **N8N Instance** for AI agent webhooks (separate hosting)
### Repository Verification
- [ ] Latest code pushed to GitHub
- [ ] All Django migrations created and committed
- [ ] `railway.json` file present in root directory
- [ ] Environment variables documented in `.env.example`
---
## 🚀 Deployment Steps
### Step 1: Create Railway Project
1. **Connect Repository:**
- Visit [railway.app](https://railway.app) and login
- Click **"New Project"** → **"Deploy from GitHub repo"**
- Select your `quantum_ai` repository
- Railway auto-detects Django and starts building
2. **Add Database:**
- In your Railway project dashboard
- Click **"New Service"** → **"Database"** → **"PostgreSQL"**
- Railway automatically configures `DATABASE_URL`
3. **Add Redis (Optional):**
- Click **"New Service"** → **"Database"** → **"Redis"**
- Railway automatically configures `REDIS_URL`
### Step 2: Configure Environment Variables
Navigate to **Variables** tab in Railway dashboard and add:
#### 🔐 Core Django Settings
```env
SECRET_KEY=your-50-character-secret-key
DEBUG=False
ALLOWED_HOSTS=quantum-ai.up.railway.app,quantumtaskai.com,localhost
CSRF_TRUSTED_ORIGINS=https://quantum-ai.up.railway.app,https://quantumtaskai.com
```
#### 📧 Email Configuration
```env
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
EMAIL_HOST=smtp.gmail.com
EMAIL_PORT=587
EMAIL_USE_TLS=True
EMAIL_HOST_USER=your-email@gmail.com
EMAIL_HOST_PASSWORD=your-app-password
DEFAULT_FROM_EMAIL=Quantum Tasks AI <your-email@gmail.com>
```
#### 💳 Stripe Payment Settings
```env
STRIPE_SECRET_KEY=sk_live_your_stripe_secret_key
STRIPE_WEBHOOK_SECRET=whsec_your_webhook_secret
```
#### 🔗 N8N Webhook URLs
```env
N8N_WEBHOOK_DATA_ANALYZER=https://your-n8n-instance.com/webhook/data-analyzer
N8N_WEBHOOK_FIVE_WHYS=https://your-n8n-instance.com/webhook/five-whys
N8N_WEBHOOK_JOB_POSTING=https://your-n8n-instance.com/webhook/job-posting
N8N_WEBHOOK_SOCIAL_ADS=https://your-n8n-instance.com/webhook/social-ads
N8N_WEBHOOK_FAQ_GENERATOR=https://your-n8n-instance.com/webhook/faq-generator
```
#### 🌤️ External API Keys
```env
OPENWEATHER_API_KEY=your_openweather_api_key
```
### Step 3: Deploy & Test
1. **Automatic Deployment:**
- Railway deploys automatically after environment variables are set
- Monitor deployment logs in Railway dashboard
- Wait for deployment to complete (2-5 minutes)
2. **Test Deployment:**
```bash
# Test application access
curl https://quantum-ai.up.railway.app/
# Test health endpoint
curl https://quantum-ai.up.railway.app/health/
# Expected health response
{
"status": "healthy",
"checks": {
"database": {"status": "healthy"},
"agents": {"status": "healthy", "active_count": 7}
}
}
```
---
## 🎯 Post-Deployment Setup
### Create Admin User
```bash
# Use Railway CLI or dashboard console
railway run python manage.py check_admin
```
**Or create manually via Django shell:**
```python
# In Railway console
python manage.py shell
# Create superuser
from django.contrib.auth import get_user_model
User = get_user_model()
user = User.objects.create_superuser(
username='admin',
email='admin@quantumtaskai.com',
password='YourSecurePassword123!'
)
user.add_balance(100, "Initial admin balance")
```
### Test Key Features
**🌐 Website Access:**
- Homepage: `https://quantum-ai.up.railway.app/`
- Marketplace: `https://quantum-ai.up.railway.app/marketplace/`
- Admin: `https://quantum-ai.up.railway.app/admin/`
**🧪 User Registration Flow:**
1. Register new user: `https://quantum-ai.up.railway.app/auth/register/`
2. Check email verification works
3. Test login functionality
4. Test wallet top-up
**🤖 Agent Functionality:**
1. Test individual agents work
2. Verify N8N webhook connections
3. Test file uploads and processing
---
## 🔧 Railway Configuration
### Custom Domain Setup
1. **In Railway Dashboard:**
- Go to **Settings** → **Domains**
- Click **"Custom Domain"**
- Enter your domain (e.g., `app.yourcompany.com`)
- Follow DNS verification steps
2. **DNS Configuration:**
```
Type: CNAME
Name: app (or @)
Value: quantum-ai.up.railway.app
```
3. **Update Environment Variables:**
```env
ALLOWED_HOSTS=app.yourcompany.com,quantum-ai.up.railway.app
CSRF_TRUSTED_ORIGINS=https://app.yourcompany.com,https://quantum-ai.up.railway.app
```
### Scaling Configuration
**In `railway.json`:**
```json
{
"$schema": "https://railway.app/railway.schema.json",
"build": {
"builder": "nixpacks"
},
"deploy": {
"startCommand": "gunicorn netcop_hub.wsgi:application --bind 0.0.0.0:$PORT --workers 3 --timeout 60",
"restartPolicyType": "ON_FAILURE",
"restartPolicyMaxRetries": 10
}
}
```
---
## 🔌 N8N Webhook Integration
### N8N Setup Requirements
**N8N must be hosted separately** (N8N Cloud, separate Railway project, or self-hosted):
1. **N8N Cloud (Recommended):**
- Sign up at [n8n.cloud](https://n8n.cloud)
- Import workflow files from `*/n8n_workflows/` directories
- Configure webhook URLs in Railway environment
2. **Self-Hosted N8N:**
- Deploy N8N to separate server/service
- Import workflows using `manage_n8n_workflows.py`
- Ensure webhooks are publicly accessible
### Workflow Management
```bash
# List all available workflows
python manage_n8n_workflows.py list
# Import specific agent workflow
python manage_n8n_workflows.py import data_analyzer
# Deploy all workflows
./deploy_n8n_workflows.sh
```
---
## 🆘 Troubleshooting
### Common Deployment Issues
**🚫 Build Failures:**
```bash
# Check Railway logs
railway logs
# Common fixes:
# 1. Ensure requirements.txt is complete
# 2. Check Python version compatibility
# 3. Verify Django settings are correct
```
**🔗 Database Connection Issues:**
```bash
# Verify DATABASE_URL is set correctly
railway variables
# Test database connection
railway run python manage.py check --database default
```
**📧 Email Not Working:**
```bash
# Test email configuration
railway run python manage.py shell
>>> from django.core.mail import send_mail
>>> send_mail('Test', 'Message', 'from@example.com', ['to@example.com'])
```
**🌐 Domain/CSRF Issues:**
```env
# Ensure these match your actual domain
ALLOWED_HOSTS=your-actual-domain.com
CSRF_TRUSTED_ORIGINS=https://your-actual-domain.com
```
### Performance Optimization
**Database Connection Pooling:**
- Railway automatically optimizes PostgreSQL connections
- Connection pooling configured in `settings.py`
**Static Files:**
- WhiteNoise serves static files efficiently
- No additional CDN needed for small applications
**Monitoring:**
- Use Railway dashboard for logs and metrics
- Health check endpoint: `/health/`
---
## 📚 Related Documentation
- [Domain Change Guide](./domain-change-guide.md)
- [Environment Variables](./environment-variables.md)
- [Database Management](../operations/database-management.md)
- [Troubleshooting Guide](../operations/troubleshooting.md)
---
## ✅ Deployment Checklist
**Pre-Deployment:**
- [ ] Repository connected to Railway
- [ ] PostgreSQL database added
- [ ] All environment variables configured
- [ ] N8N instance set up separately
**Post-Deployment:**
- [ ] Application loads successfully
- [ ] Health check passes
- [ ] Admin user created
- [ ] Email verification works
- [ ] Payment processing works
- [ ] Agent functionality works
- [ ] Custom domain configured (if needed)
**🎉 Your Quantum Tasks AI application is now live on Railway!**

0
AGENT_CREATION_GUIDE.md → docs/development/agent-creation.md
View File

357
docs/development/setup-guide.md Normal file
View File

@ -0,0 +1,357 @@
# 🛠️ Local Development Setup Guide
Complete guide for setting up Quantum Tasks AI for local development.
## 📋 Prerequisites
### System Requirements
- **Python 3.8+** (recommended: Python 3.10+)
- **Git** for version control
- **Code Editor** (VS Code, PyCharm, etc.)
### Optional but Recommended
- **PostgreSQL** for database parity with production
- **Redis** for caching (falls back to memory cache if unavailable)
- **N8N** for testing webhook agents locally
---
## 🚀 Quick Setup
### 1. Clone Repository
```bash
git clone https://github.com/your-username/quantum_ai.git
cd quantum_ai
```
### 2. Create Virtual Environment
```bash
# Create virtual environment
python -m venv venv
# Activate virtual environment
# Linux/Mac:
source venv/bin/activate
# Windows:
venv\Scripts\activate
```
### 3. Install Dependencies
```bash
# Install Python packages
pip install -r requirements.txt
# Verify installation
python --version
pip list | grep Django
```
### 4. Configure Environment
```bash
# Copy environment template
cp .env.example .env
# Edit .env file with your settings
# Minimum required for local development:
SECRET_KEY=your-50-character-secret-key-for-development
DEBUG=True
ALLOWED_HOSTS=localhost,127.0.0.1,testserver
```
### 5. Setup Database
```bash
# Check database configuration
python manage.py check_db
# Create and apply migrations
python manage.py makemigrations
python manage.py migrate
# Populate agent catalog
python manage.py populate_agents
```
### 6. Create Admin User
```bash
# Create superuser
python manage.py check_admin
# Or create manually
python manage.py createsuperuser
```
### 7. Start Development Server
```bash
# Quick start (recommended)
./run_dev.sh
# Or manual start
python manage.py runserver
```
### 8. Verify Installation
Open browser and visit:
- **Application:** http://localhost:8000
- **Admin Panel:** http://localhost:8000/admin/
- **Health Check:** http://localhost:8000/health/
---
## ⚙️ Detailed Configuration
### Environment Variables
Create `.env` file in project root:
```env
# Core Django Settings
SECRET_KEY=your-development-secret-key-50-characters-minimum
DEBUG=True
ALLOWED_HOSTS=localhost,127.0.0.1,testserver
CSRF_TRUSTED_ORIGINS=http://localhost:8000,http://127.0.0.1:8000
# Database (SQLite by default, PostgreSQL optional)
# Uncomment for PostgreSQL:
# DATABASE_URL=postgresql://user:password@localhost:5432/quantum_ai
# USE_POSTGRESQL=True
# Email (console backend for development)
EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend
# Stripe (use test keys)
STRIPE_SECRET_KEY=sk_test_your_stripe_test_key
STRIPE_WEBHOOK_SECRET=whsec_your_test_webhook_secret
# External APIs
OPENWEATHER_API_KEY=your_openweather_api_key
# N8N Webhooks (local N8N instance)
N8N_WEBHOOK_DATA_ANALYZER=http://localhost:5678/webhook/data-analyzer
N8N_WEBHOOK_FIVE_WHYS=http://localhost:5678/webhook/five-whys
N8N_WEBHOOK_JOB_POSTING=http://localhost:5678/webhook/job-posting
N8N_WEBHOOK_SOCIAL_ADS=http://localhost:5678/webhook/social-ads
N8N_WEBHOOK_FAQ_GENERATOR=http://localhost:5678/webhook/faq-generator
# Cache (optional)
# REDIS_URL=redis://127.0.0.1:6379/1
```
### Database Options
**Option 1: SQLite (Default)**
- No additional setup required
- Database file: `db.sqlite3`
- Perfect for development
**Option 2: PostgreSQL (Production Parity)**
```bash
# Install PostgreSQL
# Ubuntu/Debian:
sudo apt-get install postgresql postgresql-contrib
# macOS:
brew install postgresql
brew services start postgresql
# Create database
createdb quantum_ai
# Update .env
DATABASE_URL=postgresql://user:password@localhost:5432/quantum_ai
```
---
## 🧪 Testing Setup
### Run Tests
```bash
# Test specific agent
python tests/test_weather_agent.py
# Test homepage
python tests/test_homepage.py
# Test webhook functionality
python tests/test_five_whys_webhook.py
```
### Manual Testing
```bash
# Test health endpoint
curl http://localhost:8000/health/
# Test admin access
# Visit: http://localhost:8000/admin/
# Login with created superuser credentials
```
---
## 🔧 Development Tools
### Management Commands
```bash
# Create new agent (interactive)
python manage.py create_agent
# Create test user
python manage.py create_user
# Reset database (development only)
python manage.py reset_database
# Test webhook functionality
python manage.py test_webhook
# Cleanup uploaded files
python manage.py cleanup_uploads
# Backup user data
python manage.py backup_users --action info
```
### N8N Workflow Management
```bash
# List all workflows
python manage_n8n_workflows.py list
# Import workflow to local N8N
python manage_n8n_workflows.py import data_analyzer
# Sync workflows
python manage_n8n_workflows.py sync
```
### Debug Tools
```bash
# Django shell
python manage.py shell
# Database shell
python manage.py dbshell
# Check deployment readiness
python manage.py check --deploy
```
---
## 🔌 Optional Services
### Redis Cache Setup
```bash
# Install Redis
# Ubuntu/Debian:
sudo apt-get install redis-server
# macOS:
brew install redis
brew services start redis
# Test Redis connection
redis-cli ping
# Should return: PONG
# Update .env
REDIS_URL=redis://127.0.0.1:6379/1
```
### N8N Local Setup
```bash
# Install N8N globally
npm install n8n -g
# Start N8N
n8n start
# Access N8N UI
# Visit: http://localhost:5678
# Import workflows
python manage_n8n_workflows.py import data_analyzer
```
---
## 🐛 Troubleshooting
### Common Issues
**❌ Module Not Found Error:**
```bash
# Solution: Ensure virtual environment is activated
source venv/bin/activate # Linux/Mac
venv\Scripts\activate # Windows
# Reinstall dependencies
pip install -r requirements.txt
```
**❌ Database Migration Error:**
```bash
# Solution: Reset migrations (development only)
python manage.py reset_database
# Or fix specific migration
python manage.py migrate --fake-initial
```
**❌ Port Already in Use:**
```bash
# Solution: Use different port
python manage.py runserver 8001
# Or kill process using port 8000
sudo lsof -t -i tcp:8000 | xargs kill -9
```
**❌ Secret Key Error:**
```bash
# Solution: Generate new secret key
python -c "from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())"
# Add to .env file
SECRET_KEY=generated-secret-key
```
### Development Tips
**Performance:**
- Use SQLite for development (faster)
- Enable Django Debug Toolbar (if installed)
- Use `--verbosity 2` for detailed command output
**Database:**
- Reset database frequently during development
- Use fixtures for test data
- Backup important data before major changes
**Static Files:**
- No need to collect static files in development
- Django serves static files automatically with DEBUG=True
---
## 📚 Next Steps
After successful setup:
1. **Explore the codebase:** Read [docs/README.md](../README.md) for architecture overview
2. **Create an agent:** Follow [Agent Creation Guide](./agent-creation.md)
3. **Test functionality:** Run test suite and manual testing
4. **Deploy to staging:** Follow [Railway Deployment Guide](../deployment/railway-deployment.md)
---
## 🔗 Related Documentation
- [Agent Creation Guide](./agent-creation.md) - Build new AI agents
- [Testing Guide](./testing.md) - Testing procedures
- [Environment Variables](../deployment/environment-variables.md) - Complete environment reference
- [Railway Deployment](../deployment/railway-deployment.md) - Production deployment
---
**🎉 You're ready to develop! Visit http://localhost:8000 to see your local Quantum Tasks AI instance.**

556
docs/development/testing.md Normal file
View File

@ -0,0 +1,556 @@
# 🧪 Testing Guide
Comprehensive testing procedures for Quantum Tasks AI platform.
## 📋 Testing Overview
**Testing Levels:**
- 🔬 **Unit Tests** - Individual component testing
- 🔗 **Integration Tests** - Agent and system integration
- 🌐 **End-to-End Tests** - Full user workflow testing
- 🚀 **Deployment Tests** - Production deployment verification
---
## ⚡ Quick Testing
### Health Check
```bash
# Local development
curl http://localhost:8000/health/
# Production
curl https://quantum-ai.up.railway.app/health/
# Expected response
{
"status": "healthy",
"checks": {
"database": {"status": "healthy"},
"agents": {"status": "healthy", "active_count": 7}
}
}
```
### Basic Functionality
```bash
# Django system check
python manage.py check
# Database connectivity
python manage.py check_db
# Admin access test
python manage.py check_admin
```
---
## 🔬 Unit Testing
### Running Individual Tests
```bash
# Test specific agent
python tests/test_weather_agent.py
# Test homepage functionality
python tests/test_homepage.py
# Test webhook agents
python tests/test_five_whys_webhook.py
# Test job posting generator
python tests/test_job_posting_webhook.py
```
### Django Test Suite
```bash
# Run all tests
python manage.py test
# Run specific app tests
python manage.py test authentication
python manage.py test agent_base
# Run with verbosity
python manage.py test --verbosity=2
# Keep test database
python manage.py test --keepdb
```
### Writing Unit Tests
**Example Test Structure:**
```python
# tests/test_weather_agent.py
from django.test import TestCase, Client
from django.contrib.auth import get_user_model
from weather_reporter.models import WeatherReportAgentRequest
User = get_user_model()
class WeatherAgentTestCase(TestCase):
def setUp(self):
self.client = Client()
self.user = User.objects.create_user(
username='testuser',
email='test@example.com',
password='testpass123'
)
self.user.add_balance(50, "Test balance")
def test_weather_request_creation(self):
"""Test weather report request creation"""
self.client.login(email='test@example.com', password='testpass123')
response = self.client.post('/agents/weather-reporter/', {
'city': 'London',
'country_code': 'GB'
})
self.assertEqual(response.status_code, 200)
self.assertTrue(
WeatherReportAgentRequest.objects.filter(user=self.user).exists()
)
def test_insufficient_balance(self):
"""Test handling of insufficient wallet balance"""
self.user.wallet_balance = 0
self.user.save()
self.client.login(email='test@example.com', password='testpass123')
response = self.client.post('/agents/weather-reporter/', {
'city': 'London',
'country_code': 'GB'
})
self.assertContains(response, 'Insufficient balance')
```
---
## 🔗 Integration Testing
### Agent Integration Tests
**Webhook Agent Testing:**
```bash
# Test five whys analyzer
python tests/test_five_whys_webhook.py
# Test data analyzer
python tests/test_final_webhook.py
# Manual webhook test
python manage.py test_webhook
```
**API Agent Testing:**
```python
# Example: Weather API integration test
import requests
from django.test import TestCase
from django.conf import settings
class WeatherAPITestCase(TestCase):
def test_openweather_api_connection(self):
"""Test OpenWeather API connectivity"""
api_key = settings.OPENWEATHER_API_KEY
if not api_key:
self.skipTest("OpenWeather API key not configured")
response = requests.get(
f"https://api.openweathermap.org/data/2.5/weather",
params={
'q': 'London,GB',
'appid': api_key,
'units': 'metric'
}
)
self.assertEqual(response.status_code, 200)
data = response.json()
self.assertIn('main', data)
self.assertIn('weather', data)
```
### Database Integration
```python
# Test database operations
from django.test import TransactionTestCase
from django.db import transaction
class DatabaseIntegrationTestCase(TransactionTestCase):
def test_user_wallet_transactions(self):
"""Test wallet transaction integrity"""
user = User.objects.create_user(
username='test',
email='test@example.com',
password='pass'
)
initial_balance = user.wallet_balance
# Test adding balance
user.add_balance(100, "Test top-up")
self.assertEqual(user.wallet_balance, initial_balance + 100)
# Test deducting balance
success = user.deduct_balance(50, "Test usage")
self.assertTrue(success)
self.assertEqual(user.wallet_balance, initial_balance + 50)
# Test insufficient balance
success = user.deduct_balance(1000, "Too much")
self.assertFalse(success)
self.assertEqual(user.wallet_balance, initial_balance + 50)
```
---
## 🌐 End-to-End Testing
### Manual Testing Workflows
**User Registration & Email Verification:**
1. Visit registration page: `/auth/register/`
2. Fill out form with valid data
3. Check email for verification link
4. Click verification link
5. Login with new credentials
6. Verify dashboard access
**Agent Usage Workflow:**
1. Login as verified user
2. Add money to wallet: `/wallet/`
3. Visit agent: `/agents/weather-reporter/`
4. Submit valid request
5. Verify balance deduction
6. Check results display
7. Verify transaction history
**Admin Workflow:**
1. Login to admin: `/admin/`
2. Check user management
3. Verify agent configuration
4. Review transaction logs
5. Test agent activation/deactivation
### Automated E2E Testing
**Using Django Test Client:**
```python
from django.test import TestCase, Client
from django.urls import reverse
class EndToEndTestCase(TestCase):
def setUp(self):
self.client = Client()
def test_complete_user_journey(self):
"""Test complete user journey from registration to agent usage"""
# 1. Register new user
response = self.client.post('/auth/register/', {
'username': 'testuser',
'email': 'test@example.com',
'password1': 'SecurePass123!',
'password2': 'SecurePass123!'
})
self.assertEqual(response.status_code, 302) # Redirect after registration
# 2. Verify user created
user = User.objects.get(email='test@example.com')
self.assertFalse(user.email_verified)
# 3. Simulate email verification
token = EmailVerificationToken.objects.get(user=user)
response = self.client.get(f'/auth/verify-email/{token.token}/')
self.assertEqual(response.status_code, 302)
# 4. Login
response = self.client.post('/auth/login/', {
'email': 'test@example.com',
'password': 'SecurePass123!'
})
self.assertEqual(response.status_code, 302)
# 5. Add wallet balance
user.add_balance(100, "Test balance")
# 6. Use weather agent
response = self.client.post('/agents/weather-reporter/', {
'city': 'London',
'country_code': 'GB'
})
self.assertEqual(response.status_code, 200)
# 7. Verify balance deducted
user.refresh_from_db()
self.assertLess(user.wallet_balance, 100)
```
---
## 🔧 Testing Utilities
### Test Data Setup
```python
# tests/utils.py
from django.contrib.auth import get_user_model
from agent_base.models import BaseAgent
User = get_user_model()
def create_test_user(email="test@example.com", balance=100):
"""Create a test user with wallet balance"""
user = User.objects.create_user(
username='testuser',
email=email,
password='testpass123'
)
user.email_verified = True
user.save()
if balance > 0:
user.add_balance(balance, "Test balance")
return user
def create_test_agent(name="Test Agent", price=10):
"""Create a test agent"""
return BaseAgent.objects.create(
name=name,
slug=name.lower().replace(' ', '-'),
description="Test agent for testing",
category='utilities',
price=price,
agent_type='api'
)
```
### Mock External Services
```python
# tests/mocks.py
from unittest.mock import patch, Mock
import json
class MockN8NResponse:
"""Mock N8N webhook response"""
def __init__(self, success=True, data=None):
self.status_code = 200 if success else 500
self.data = data or {"result": "Test result"}
def json(self):
return self.data
@patch('requests.post')
def test_webhook_agent_with_mock(mock_post):
"""Test webhook agent with mocked N8N response"""
mock_post.return_value = MockN8NResponse(
success=True,
data={"analysis": "Test analysis result"}
)
# Test code here
# The webhook call will use the mocked response
```
### Environment Testing
```python
# tests/test_environment.py
from django.test import TestCase
from django.conf import settings
class EnvironmentTestCase(TestCase):
def test_required_settings(self):
"""Test that required settings are configured"""
required_settings = [
'SECRET_KEY',
'DATABASES',
'INSTALLED_APPS'
]
for setting in required_settings:
self.assertTrue(
hasattr(settings, setting),
f"Required setting {setting} not found"
)
def test_external_api_keys(self):
"""Test external API key configuration"""
if hasattr(settings, 'OPENWEATHER_API_KEY'):
self.assertTrue(
settings.OPENWEATHER_API_KEY,
"OpenWeather API key is empty"
)
```
---
## 🚀 Deployment Testing
### Pre-Deployment Tests
```bash
# 1. Run full test suite
python manage.py test --verbosity=2
# 2. Check deployment configuration
python manage.py check --deploy
# 3. Test with production-like settings
DEBUG=False python manage.py check
# 4. Verify static files
python manage.py collectstatic --dry-run
# 5. Test database migrations
python manage.py migrate --dry-run
```
### Post-Deployment Verification
```bash
# 1. Health check
curl https://quantum-ai.up.railway.app/health/
# 2. Test key endpoints
curl -I https://quantum-ai.up.railway.app/
curl -I https://quantum-ai.up.railway.app/marketplace/
curl -I https://quantum-ai.up.railway.app/admin/
# 3. Test static files
curl -I https://quantum-ai.up.railway.app/static/css/base.css
# 4. Test email functionality (manual)
# Register test user and verify email delivery
# 5. Test payment integration (manual)
# Use Stripe test cards to verify payment flow
```
### Performance Testing
```bash
# Load testing with curl
for i in {1..10}; do
curl -o /dev/null -s -w "%{time_total}\n" https://quantum-ai.up.railway.app/
done
# Database performance
railway run python manage.py shell -c "
from django.test.utils import override_settings
from django.db import connection
from django.contrib.auth import get_user_model
User = get_user_model()
with connection.cursor() as cursor:
cursor.execute('EXPLAIN ANALYZE SELECT * FROM authentication_user LIMIT 10')
print(cursor.fetchall())
"
```
---
## 📊 Test Coverage
### Measuring Coverage
```bash
# Install coverage
pip install coverage
# Run tests with coverage
coverage run --source='.' manage.py test
# Generate coverage report
coverage report
# Generate HTML coverage report
coverage html
# Open htmlcov/index.html in browser
```
### Coverage Targets
**Minimum Coverage Goals:**
- **Models:** 90%+ (critical business logic)
- **Views:** 80%+ (user-facing functionality)
- **Processors:** 85%+ (agent business logic)
- **Utilities:** 95%+ (helper functions)
**Critical Areas (100% coverage):**
- User authentication
- Wallet transactions
- Payment processing
- Agent request handling
---
## 🐛 Debugging Tests
### Test Debugging
```python
# Add debugging to tests
import pdb; pdb.set_trace() # Breakpoint
# Print debugging
print(f"User balance: {user.wallet_balance}")
print(f"Response: {response.content}")
# Use Django test client debugging
from django.test.utils import setup_test_environment
setup_test_environment(debug=True)
```
### Common Test Issues
**Database Issues:**
```bash
# Reset test database
python manage.py test --debug-mode
# Use different test database
python manage.py test --settings=netcop_hub.test_settings
```
**Mock Issues:**
```python
# Verify mock calls
mock_function.assert_called_once_with(expected_arg)
# Check mock call count
self.assertEqual(mock_function.call_count, 1)
# Reset mocks between tests
mock_function.reset_mock()
```
---
## 📚 Related Documentation
- [Setup Guide](./setup-guide.md) - Development environment setup
- [Agent Creation](./agent-creation.md) - Agent development testing
- [Troubleshooting](../operations/troubleshooting.md) - Debugging production issues
- [Database Management](../operations/database-management.md) - Database testing
---
**🎯 Testing Best Practices:**
- Write tests before implementing features (TDD)
- Test both success and failure scenarios
- Mock external services to avoid dependencies
- Use descriptive test names and docstrings
- Maintain test data isolation between tests
- Regular test suite maintenance and cleanup

516
docs/operations/database-management.md Normal file
View File

@ -0,0 +1,516 @@
# 🗄️ Database Management Guide
Comprehensive guide for managing databases in Quantum Tasks AI across development and production environments.
## 📋 Overview
**Database Types by Environment:**
- **Local Development:** SQLite (default) or PostgreSQL (optional)
- **Railway Production:** PostgreSQL (managed)
- **Testing:** SQLite (isolated)
---
## 🛠️ Development Database Management
### SQLite (Default)
**Basic Operations:**
```bash
# Check database configuration
python manage.py check_db
# Create migrations
python manage.py makemigrations
# Apply migrations
python manage.py migrate
# Reset database (development only)
python manage.py reset_database
# Access database shell
python manage.py dbshell
```
**Database File Location:**
- File: `db.sqlite3` in project root
- Backup: Copy the file to safe location
- Reset: Delete file and run migrations
### PostgreSQL (Local)
**Setup:**
```bash
# Install PostgreSQL
# Ubuntu/Debian:
sudo apt-get install postgresql postgresql-contrib
# macOS:
brew install postgresql
brew services start postgresql
# Create database
createdb quantum_ai
# Create user (optional)
createuser quantum_user -P
# Update .env
USE_POSTGRESQL=True
DATABASE_URL=postgresql://quantum_user:password@localhost:5432/quantum_ai
```
**Management:**
```bash
# Connect to database
psql -d quantum_ai
# Backup database
pg_dump quantum_ai > backup.sql
# Restore database
psql quantum_ai < backup.sql
# Check connections
psql -c "SELECT datname, numbackends FROM pg_stat_database;"
```
---
## 🚀 Production Database Management
### Railway PostgreSQL
**Automatic Setup:**
- Railway automatically provisions PostgreSQL when added
- `DATABASE_URL` environment variable auto-configured
- Managed backups and scaling
**Accessing Production Database:**
```bash
# Via Railway CLI
railway connect postgres
# Via connection string
psql $DATABASE_URL
# Or get connection details from Railway dashboard
```
**Production Commands:**
```bash
# Run migrations on production
railway run python manage.py migrate
# Check production database status
railway run python manage.py check_db
# Create admin user
railway run python manage.py check_admin
# Backup users data
railway run python manage.py backup_users --action export
```
### Connection Management
**Connection Pooling (Auto-configured):**
```python
# In settings.py
DATABASES['default']['CONN_MAX_AGE'] = 600 # 10 minutes
```
**Connection Monitoring:**
```sql
-- Check active connections
SELECT datname, numbackends FROM pg_stat_database;
-- Check connection limits
SELECT setting FROM pg_settings WHERE name = 'max_connections';
-- View current connections
SELECT * FROM pg_stat_activity WHERE datname = 'railway';
```
---
## 🔄 Database Migrations
### Creating Migrations
```bash
# Auto-detect model changes
python manage.py makemigrations
# Create migration for specific app
python manage.py makemigrations agent_base
# Create empty migration
python manage.py makemigrations --empty agent_base
# Name migration
python manage.py makemigrations --name add_user_preferences agent_base
```
### Applying Migrations
```bash
# Apply all migrations
python manage.py migrate
# Apply specific app migrations
python manage.py migrate agent_base
# Apply to specific migration
python manage.py migrate agent_base 0001
# Fake migration (mark as applied without running)
python manage.py migrate --fake agent_base 0001
```
### Migration Management
```bash
# Show migration status
python manage.py showmigrations
# Show SQL for migration
python manage.py sqlmigrate agent_base 0001
# Reverse migration
python manage.py migrate agent_base 0001
# List migrations
ls -la */migrations/
```
### Migration Best Practices
**Safe Migration Patterns:**
```python
# ✅ Safe: Add new field with default
class Migration(migrations.Migration):
operations = [
migrations.AddField(
model_name='user',
name='phone',
field=models.CharField(max_length=20, default=''),
),
]
# ✅ Safe: Add new model
class Migration(migrations.Migration):
operations = [
migrations.CreateModel(
name='UserPreference',
fields=[...],
),
]
# ⚠️ Caution: Rename field (data migration needed)
# ❌ Dangerous: Drop field without backup
```
---
## 🔧 Database Maintenance
### Regular Maintenance Tasks
**Daily (Automated):**
- Connection monitoring
- Performance metrics review
- Error log analysis
**Weekly:**
- Database size monitoring
- Query performance review
- Index usage analysis
**Monthly:**
- Full database backup
- Cleanup old data (if applicable)
- Performance optimization review
### Performance Optimization
**Query Optimization:**
```sql
-- Find slow queries
SELECT query, mean_time, calls
FROM pg_stat_statements
ORDER BY mean_time DESC
LIMIT 10;
-- Check index usage
SELECT schemaname, tablename, attname, n_distinct, correlation
FROM pg_stats
WHERE tablename = 'authentication_user';
-- Analyze table statistics
ANALYZE authentication_user;
```
**Django Optimization:**
```python
# Use select_related for foreign keys
users = User.objects.select_related('wallet').all()
# Use prefetch_related for many-to-many
users = User.objects.prefetch_related('transactions').all()
# Add database indexes
class Meta:
indexes = [
models.Index(fields=['email', 'created_at']),
models.Index(fields=['-created_at']),
]
```
### Cleanup Operations
```bash
# Cleanup uploaded files
python manage.py cleanup_uploads
# Clear sessions (if using database sessions)
python manage.py clearsessions
# Custom cleanup command example
python manage.py shell -c "
from authentication.models import User
from datetime import datetime, timedelta
# Delete inactive users older than 1 year
cutoff = datetime.now() - timedelta(days=365)
inactive_users = User.objects.filter(
last_login__lt=cutoff,
is_active=False
)
print(f'Found {inactive_users.count()} inactive users')
# inactive_users.delete() # Uncomment to actually delete
"
```
---
## 💾 Backup & Recovery
### Local Development Backups
**SQLite Backup:**
```bash
# Simple file copy
cp db.sqlite3 backups/db_$(date +%Y%m%d_%H%M%S).sqlite3
# Using Django
python manage.py dumpdata > backup_$(date +%Y%m%d_%H%M%S).json
```
**PostgreSQL Backup:**
```bash
# Full database dump
pg_dump quantum_ai > backup_$(date +%Y%m%d_%H%M%S).sql
# Compressed backup
pg_dump quantum_ai | gzip > backup_$(date +%Y%m%d_%H%M%S).sql.gz
# Data only
pg_dump --data-only quantum_ai > data_backup.sql
# Schema only
pg_dump --schema-only quantum_ai > schema_backup.sql
```
### Production Backups
**Railway Managed Backups:**
- Railway automatically creates daily backups
- Access via Railway dashboard
- Point-in-time recovery available
**Manual Production Backup:**
```bash
# Backup via Railway CLI
railway run pg_dump $DATABASE_URL > production_backup_$(date +%Y%m%d).sql
# User data backup
railway run python manage.py backup_users --action export > users_backup.json
# Backup specific tables
railway run pg_dump $DATABASE_URL -t authentication_user -t wallet_wallettransaction > critical_backup.sql
```
### Recovery Procedures
**Local Recovery:**
```bash
# SQLite restore
cp backups/db_20241225_120000.sqlite3 db.sqlite3
# PostgreSQL restore
psql quantum_ai < backup_20241225_120000.sql
# Django fixtures restore
python manage.py loaddata backup_20241225_120000.json
```
**Production Recovery:**
```bash
# Contact Railway support for point-in-time recovery
# Or restore from manual backup
# Restore to new database (safest)
railway run psql $DATABASE_URL < backup_file.sql
# Partial restore (specific tables)
railway run psql $DATABASE_URL -c "\copy authentication_user FROM 'users_backup.csv' WITH CSV HEADER"
```
---
## 🔍 Monitoring & Diagnostics
### Health Checks
```bash
# Django database check
python manage.py check --database default
# Custom health check
curl http://localhost:8000/health/
# Railway health check
railway run python manage.py check_db
```
### Performance Monitoring
**Database Metrics:**
```sql
-- Connection count
SELECT count(*) FROM pg_stat_activity;
-- Database size
SELECT
datname,
pg_size_pretty(pg_database_size(datname)) as size
FROM pg_database
WHERE datname = 'railway';
-- Table sizes
SELECT
tablename,
pg_size_pretty(pg_total_relation_size(tablename::regclass)) as size
FROM pg_tables
WHERE schemaname = 'public'
ORDER BY pg_total_relation_size(tablename::regclass) DESC;
```
**Django Debug:**
```python
# In Django shell
from django.db import connection
from django.db import connections
# Check database connection
connections['default'].cursor()
# View queries
from django.conf import settings
settings.LOGGING['loggers']['django.db.backends'] = {
'level': 'DEBUG',
'handlers': ['console'],
}
```
### Log Analysis
```bash
# Railway PostgreSQL logs
railway logs --service postgres
# Django database queries (if DEBUG=True)
python manage.py runserver --verbosity=2
# Check for long-running queries
# Use Railway dashboard metrics
```
---
## 🚨 Troubleshooting Database Issues
### Common Problems
**Connection Refused:**
```bash
# Check if PostgreSQL is running
systemctl status postgresql # Linux
brew services list | grep postgres # macOS
# Check connection parameters
psql -h localhost -p 5432 -U username -d database
# Railway connection test
railway run psql $DATABASE_URL -c "SELECT 1;"
```
**Migration Conflicts:**
```bash
# Show migration conflicts
python manage.py showmigrations | grep "\[ \]"
# Resolve conflicts
python manage.py migrate --fake app_name migration_number
python manage.py migrate app_name
# Nuclear option (development only)
python manage.py reset_database
```
**Performance Issues:**
```sql
-- Find slow queries
SELECT query, mean_time, calls
FROM pg_stat_statements
ORDER BY mean_time DESC LIMIT 10;
-- Check for locks
SELECT * FROM pg_locks WHERE NOT granted;
-- Check for blocking queries
SELECT
blocked_locks.pid AS blocked_pid,
blocked_activity.usename AS blocked_user,
blocking_locks.pid AS blocking_pid,
blocking_activity.usename AS blocking_user,
blocked_activity.query AS blocked_statement,
blocking_activity.query AS current_statement_in_blocking_process
FROM pg_catalog.pg_locks blocked_locks
JOIN pg_catalog.pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_catalog.pg_locks blocking_locks
ON blocking_locks.locktype = blocked_locks.locktype
AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
JOIN pg_catalog.pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
```
---
## 📚 Related Documentation
- [Environment Variables](../deployment/environment-variables.md) - Database configuration
- [Railway Deployment](../deployment/railway-deployment.md) - Production setup
- [Troubleshooting Guide](./troubleshooting.md) - Common database issues
- [Maintenance Guide](./maintenance.md) - Ongoing maintenance procedures
---
**⚡ Pro Tips:**
- Always backup before major operations
- Test migrations on development environment first
- Monitor connection counts in production
- Use database indexes for frequently queried fields
- Keep development and production database structures in sync

0
POST_DEPLOYMENT_CHECKLIST.md → docs/operations/maintenance.md
View File

486
docs/operations/troubleshooting.md Normal file
View File

@ -0,0 +1,486 @@
# 🔧 Troubleshooting Guide
Common issues and solutions for Quantum Tasks AI platform.
## 🚨 Emergency Quick Fixes
### Application Won't Start
```bash
# 1. Check system health
python manage.py check --deploy
# 2. Test database connection
python manage.py check_db
# 3. Verify environment variables
python manage.py shell -c "from django.conf import settings; print('SECRET_KEY set:', bool(settings.SECRET_KEY))"
# 4. Check logs
railway logs # For Railway deployment
```
### Health Check Failing
```bash
# Test health endpoint
curl http://localhost:8000/health/
curl https://quantum-ai.up.railway.app/health/
# Expected healthy response:
{
"status": "healthy",
"checks": {
"database": {"status": "healthy"},
"agents": {"status": "healthy", "active_count": 7}
}
}
```
---
## 🌐 Domain & URL Issues
### CSRF Verification Failed
**Error:** `CSRF verification failed. Request aborted.`
**Solutions:**
```bash
# 1. Update CSRF trusted origins
CSRF_TRUSTED_ORIGINS=https://your-domain.com,https://quantumtaskai.com
# 2. Check allowed hosts
ALLOWED_HOSTS=your-domain.com,quantumtaskai.com,localhost
# 3. Clear browser cache and cookies
# 4. Verify HTTPS vs HTTP in origins
```
### Email Links Wrong Domain
**Issue:** Email verification/reset links point to wrong domain
**Solutions:**
```bash
# 1. Update SITE_URL environment variable
SITE_URL=https://your-correct-domain.com
# 2. Check Railway environment variables
railway variables
# 3. Follow domain change guide
# See: docs/deployment/domain-change-guide.md
```
### Page Not Found (404)
**Error:** `Page not found` for admin or other pages
**Solutions:**
```bash
# 1. Check URL patterns
python manage.py show_urls
# 2. Verify static files
python manage.py collectstatic --noinput
# 3. Check ALLOWED_HOSTS setting
# 4. Test with trailing slash: /admin/
```
---
## 🗄️ Database Issues
### Database Connection Failed
**Error:** `FATAL: database "railway" does not exist`
**Solutions:**
```bash
# 1. Verify Railway PostgreSQL service is running
# Check Railway dashboard
# 2. Test DATABASE_URL
python manage.py dbshell
# 3. Check environment variable
echo $DATABASE_URL
# 4. Recreate PostgreSQL service if needed
```
### Migration Errors
**Error:** `Migration conflicts` or `Table already exists`
**Solutions:**
```bash
# 1. Check migration status
python manage.py showmigrations
# 2. Fake initial migration (if safe)
python manage.py migrate --fake-initial
# 3. Reset migrations (development only)
python manage.py reset_database
# 4. Manual migration fix
python manage.py migrate --fake app_name 0001
python manage.py migrate app_name
```
### Slow Database Performance
**Issues:** Slow queries, timeouts
**Solutions:**
```python
# 1. Check connection pooling (Railway auto-configured)
DATABASES['default']['CONN_MAX_AGE'] = 600
# 2. Add database indexes (if needed)
python manage.py dbshell
# Run EXPLAIN ANALYZE on slow queries
# 3. Monitor Railway metrics
# Check Railway dashboard → Metrics
```
---
## 📧 Email Issues
### Email Not Sending
**Error:** `SMTPAuthenticationError` or emails not received
**Solutions:**
```bash
# 1. Test email configuration
python manage.py shell
>>> from django.core.mail import send_mail
>>> send_mail('Test', 'Message', 'from@example.com', ['to@example.com'])
# 2. Check Gmail App Password (not regular password)
EMAIL_HOST_PASSWORD=your-16-character-app-password
# 3. Verify email backend
EMAIL_BACKEND=django.core.mail.backends.smtp.EmailBackend
# 4. Check spam folder
# 5. Verify sender domain reputation
```
### Email Templates Broken
**Issue:** Email formatting issues or missing content
**Solutions:**
```bash
# 1. Check email template syntax
# Verify: authentication/views.py email templates
# 2. Test with console backend
EMAIL_BACKEND=django.core.mail.backends.console.EmailBackend
# 3. Check SITE_URL for links
SITE_URL=https://your-correct-domain.com
```
---
## 💳 Payment Issues
### Stripe Integration Failed
**Error:** `InvalidRequestError` or payment not processing
**Solutions:**
```bash
# 1. Verify Stripe keys
STRIPE_SECRET_KEY=sk_test_... # for test
STRIPE_SECRET_KEY=sk_live_... # for production
# 2. Check webhook endpoint
# Stripe Dashboard → Webhooks
# URL: https://your-domain.com/wallet/stripe/webhook/
# 3. Test webhook secret
STRIPE_WEBHOOK_SECRET=whsec_your_webhook_secret
# 4. Check Railway logs for Stripe errors
railway logs | grep stripe
```
### Wallet Balance Issues
**Issue:** Incorrect balance or transaction not recorded
**Solutions:**
```python
# 1. Check transaction history
python manage.py shell
>>> from authentication.models import User
>>> user = User.objects.get(email='user@example.com')
>>> user.wallet_transactions.all()
# 2. Verify Stripe webhook events
# Check Stripe Dashboard → Events
# 3. Manual balance correction (if needed)
>>> user.wallet_balance = 100.00
>>> user.save()
```
---
## 🤖 Agent Issues
### Webhook Agent Not Working
**Error:** Agent returns error or times out
**Solutions:**
```bash
# 1. Check N8N webhook URL
curl -X POST https://your-n8n-instance.com/webhook/test
# 2. Verify N8N environment variables
N8N_WEBHOOK_DATA_ANALYZER=https://your-n8n-instance.com/webhook/data-analyzer
# 3. Test N8N workflow directly
# Visit N8N dashboard and test workflow
# 4. Check agent processor code
# See: individual agent processor.py files
```
### API Agent Not Working
**Error:** Weather agent or other API agents failing
**Solutions:**
```bash
# 1. Check API key
OPENWEATHER_API_KEY=your_api_key
# 2. Test API directly
curl "https://api.openweathermap.org/data/2.5/weather?q=London&appid=YOUR_API_KEY"
# 3. Check rate limits
# Most APIs have rate limiting
# 4. Verify API endpoint URLs
```
### File Upload Issues
**Error:** File upload fails or files not processed
**Solutions:**
```bash
# 1. Check media directory permissions
ls -la media/uploads/
# 2. Verify file size limits
# Django default: 2.5MB
# 3. Check disk space (Railway)
# Monitor Railway dashboard
# 4. Clean up old files
python manage.py cleanup_uploads
```
---
## 🚀 Deployment Issues
### Railway Build Failed
**Error:** Build fails during deployment
**Solutions:**
```bash
# 1. Check Railway build logs
railway logs --deployment
# 2. Verify requirements.txt
pip freeze > requirements.txt
# 3. Check Python version
# Ensure compatible with Railway
# 4. Verify railway.json
{
"build": {"builder": "nixpacks"},
"deploy": {"startCommand": "gunicorn netcop_hub.wsgi:application"}
}
```
### Environment Variables Missing
**Error:** Settings errors in production
**Solutions:**
```bash
# 1. List current variables
railway variables
# 2. Add missing variables
railway variables set SECRET_KEY=your-secret-key
# 3. Verify environment template
# See: docs/deployment/environment-variables.md
# 4. Check variable spelling and format
```
### SSL Certificate Issues
**Error:** HTTPS not working or certificate errors
**Solutions:**
```bash
# 1. Wait for Railway SSL provisioning (5-10 minutes)
# 2. Check custom domain configuration
# Railway Dashboard → Settings → Domains
# 3. Verify DNS settings
nslookup your-domain.com
dig your-domain.com
# 4. Check HTTPS redirect settings
SECURE_SSL_REDIRECT=True # for production
```
---
## 🔍 Debugging Tools
### Django Debug Information
```bash
# Check configuration
python manage.py check --deploy
# Database information
python manage.py dbshell
# Shell access
python manage.py shell
# Show URLs
python manage.py show_urls
# Migration status
python manage.py showmigrations
```
### Railway Debugging
```bash
# View logs
railway logs
# Live log streaming
railway logs --follow
# Variable management
railway variables
railway variables set KEY=value
# Service information
railway status
```
### Network Debugging
```bash
# Test connectivity
curl -I https://your-domain.com
# Check DNS
nslookup your-domain.com
dig your-domain.com
# Test specific endpoints
curl https://your-domain.com/health/
curl https://your-domain.com/admin/
```
---
## 📊 Performance Issues
### Slow Page Load
**Solutions:**
```python
# 1. Enable debug toolbar (development)
INSTALLED_APPS += ['debug_toolbar']
# 2. Check database queries
# Use Django Debug Toolbar to identify N+1 queries
# 3. Add database indexes
class Meta:
indexes = [
models.Index(fields=['created_at']),
models.Index(fields=['user', 'status']),
]
# 4. Use select_related and prefetch_related
User.objects.select_related('profile').all()
```
### High Memory Usage
**Solutions:**
```bash
# 1. Monitor Railway metrics
# Check Railway Dashboard → Metrics
# 2. Optimize queries
# Avoid loading large datasets
# 3. Use pagination
from django.core.paginator import Paginator
# 4. Check for memory leaks
# Monitor long-running processes
```
---
## 🆘 Getting More Help
### Log Analysis
```bash
# Railway logs with filtering
railway logs | grep ERROR
railway logs | grep "500"
# Django logging
# Check netcop.log file (if configured)
# Browser developer tools
# Check Network tab for failed requests
# Check Console for JavaScript errors
```
### Testing Procedures
```bash
# Health check first
curl https://your-domain.com/health/
# Test authentication
curl -c cookies.txt -b cookies.txt https://your-domain.com/auth/login/
# Test API endpoints
curl https://your-domain.com/api/agents/
# Test static files
curl https://your-domain.com/static/css/base.css
```
### Escalation Steps
1. **Check this troubleshooting guide**
2. **Review relevant documentation in `/docs/`**
3. **Check Railway service status**
4. **Test in local development environment**
5. **Review recent code changes**
6. **Check external service status (Stripe, N8N, email provider)**
---
## 📚 Related Documentation
- [Environment Variables](../deployment/environment-variables.md) - Configuration reference
- [Railway Deployment](../deployment/railway-deployment.md) - Deployment guide
- [Domain Change Guide](../deployment/domain-change-guide.md) - Domain configuration
- [Database Management](./database-management.md) - Database operations
---
**💡 Pro Tip:** Most issues are environment variable or configuration problems. Always check the basics first: SECRET_KEY, DATABASE_URL, ALLOWED_HOSTS, and CSRF_TRUSTED_ORIGINS.