quantum-ai-v2/docs/ERROR_PREVENTION_GUIDE.md

Error Prevention Guide for Agent Creation

🎯 Based on 5 Whys Analyzer Debugging Experience

This guide documents all the common errors encountered during agent development and their proven solutions, based on extensive debugging work that led to the successful 5 Whys Analyzer implementation.

---

📋 Table of Contents

1. Template Loading Errors
2. URL Routing Issues
3. Database Migration Conflicts
4. Wallet Integration Problems
5. Session Management Issues
6. Status Tracking Problems
7. Error Handling Failures
8. Environment Variable Issues
9. N8N Webhook Problems
10. Performance and Index Issues

---

1. Template Loading Errors

❌ Common Error

TemplateDoesNotExist: detail.html
django.template.loader.TemplateDoesNotExist: detail.html

🔍 Root Cause Analysis

• Template in wrong directory structure
• Django server cache holding old template paths
• Missing app in INSTALLED_APPS
• Incorrect template naming convention

✅ 5 Whys Learned Solution

Correct Template Structure:

# ✅ Correct - 5 Whys pattern
agent_five_whys_analyzer/
└── templates/
    └── five_whys_analyzer/
        └── detail.html

# ❌ Wrong - causes TemplateDoesNotExist
agent_five_whys_analyzer/
└── templates/
    └── detail.html  # Missing app subdirectory

Template Path Validation Script:

# Test template loading before starting server
python manage.py shell -c "
from django.template.loader import get_template
try:
    template = get_template('five_whys_analyzer/detail.html')
    print('✅ Template found:', template.origin.name)
except Exception as e:
    print('❌ Template error:', e)
"

Critical Fix Steps:

1. Create proper directory structure
2. Move template to correct location
3. RESTART Django server (cache issue)
4. Verify template loading with shell command

🛡️ Prevention Strategy

# Template creation checklist
mkdir -p [agent_name]/templates/[agent_name]/
cp existing_working_template.html [agent_name]/templates/[agent_name]/detail.html
# Always restart server after template changes

---

2. URL Routing Issues

❌ Common Errors

NoReverseMatch: Reverse for 'wallet' not found
django.urls.exceptions.NoReverseMatch at /agents/five-whys-analyzer/

🔍 Root Cause Analysis

• Missing URL namespaces in templates
• Incorrect URL registration order
• Agent URLs placed after catch-all core URLs

✅ 5 Whys Learned Solution

Correct URL Namespacing in Templates:

<!-- ❌ Wrong - causes NoReverseMatch -->
<a href="{% url 'wallet' %}">Wallet</a>
<a href="{% url 'homepage' %}">Home</a>

<!-- ✅ Correct - 5 Whys pattern -->
<a href="{% url 'core:wallet' %}">Wallet</a>
<a href="{% url 'core:homepage' %}">Home</a>
<a href="{% url 'authentication:login' %}">Login</a>

Correct URL Registration Order:

# netcop_hub/urls.py - CRITICAL ORDER
urlpatterns = [
    path('admin/', admin.site.urls),
    path('auth/', include('authentication.urls')),
    
    # ✅ Agent URLs MUST come before core URLs
    path('agents/weather-reporter/', include('weather_reporter.urls')),
    path('agents/five-whys-analyzer/', include('five_whys_analyzer.urls')),
    
    # ❌ Core URLs with catch-all pattern must be LAST
    path('', include('core.urls')),  # This catches everything - put LAST
]

URL Testing Commands:

# Test URL resolution
python manage.py shell -c "
from django.urls import reverse
try:
    url = reverse('core:agent_detail', args=['five-whys-analyzer'])
    print('✅ URL resolved:', url)
except Exception as e:
    print('❌ URL error:', e)
"

🛡️ Prevention Strategy

• Always use namespaced URLs in templates
• Register agent URLs before core URLs
• Test URL resolution after each agent creation

---

3. Database Migration Conflicts

❌ Common Errors

django.db.utils.ProgrammingError: relation "five_whys_analyzer_requests" already exists
django.db.migrations.exceptions.InconsistentMigrationHistory

🔍 Root Cause Analysis

• Django migration state out of sync with actual database
• Manually created tables conflicting with migrations
• Migration dependencies missing or circular

✅ 5 Whys Learned Solution

Manual Migration Sync Fix:

# 1. Check current migration state
python manage.py showmigrations five_whys_analyzer

# 2. Create empty migration to sync state
python manage.py makemigrations five_whys_analyzer --empty --name fix_migration_sync

# 3. Edit the migration file to match current state
# migrations/000X_fix_migration_sync.py
from django.db import migrations

class Migration(migrations.Migration):
    dependencies = [
        ('five_whys_analyzer', '0001_initial'),
    ]
    operations = [
        # Empty operations - just sync Django state
    ]

# 4. Apply migration
python manage.py migrate five_whys_analyzer

Conflict Resolution Pattern:

# If migration conflicts persist
python manage.py migrate five_whys_analyzer --fake-initial
python manage.py migrate five_whys_analyzer

🛡️ Prevention Strategy

• Always run makemigrations immediately after model changes
• Test migrations on clean database before production
• Keep migration files in version control

---

4. Wallet Integration Problems

❌ Common Errors

AttributeError: 'User' object has no attribute 'deduct_balance'
decimal.InvalidOperation: [<class 'decimal.ConversionSyntax'>]
Wallet balance incorrectly deducted for failed requests

🔍 Root Cause Analysis

• Deducting balance before processing completion
• Incorrect decimal handling for currency
• Missing wallet methods in User model

✅ 5 Whys Learned Solution

Delayed Deduction Pattern (Critical):

# ❌ Wrong - deduct before processing
def process_view(request):
    # Bad: deduct immediately
    request.user.deduct_balance(agent.price, description, agent_slug)
    result = process_request()  # What if this fails?
    return result

# ✅ Correct - 5 Whys pattern (deduct after success)
def process_report_response(self, response_data, request_obj):
    try:
        # Process first
        final_report = response_data.get('output', '')
        success = bool(final_report) and response_data.get('success', True)
        
        if success:
            # Save successful response
            response_obj.final_report = final_report
            response_obj.save()
            
            # ONLY deduct after confirmed success
            request_obj.user.deduct_balance(
                request_obj.cost,
                f"5 Whys Analysis Agent - Final Report",
                'five-whys-analyzer'
            )
            request_obj.status = 'completed'
        else:
            request_obj.status = 'failed'
            # No wallet deduction for failures
            
        request_obj.save()
        return response_obj
        
    except Exception as e:
        request_obj.status = 'failed'
        request_obj.save()
        # No wallet deduction for exceptions
        raise Exception(f"Failed to process: {e}")

Decimal Handling:

# ✅ Correct decimal usage
from decimal import Decimal

# Always use Decimal for currency
agent.price = Decimal('8.00')
request_obj.cost = Decimal('8.00')

# Check balance properly
if request.user.wallet_balance >= agent.price:
    # Proceed

🛡️ Prevention Strategy

• Never deduct balance before processing completion
• Always use Decimal for currency calculations
• Implement balance checks before processing
• Test wallet integration with both success and failure scenarios

---

5. Session Management Issues

❌ Common Errors

KeyError: 'session_id'
Multiple chat sessions created for same user
Session state lost between requests

🔍 Root Cause Analysis

• Missing session ID handling
• No persistent session storage
• Poor session lifecycle management

✅ 5 Whys Learned Solution

Session-Based Model Pattern:

# 5 Whys session management pattern
class AgentRequest(BaseAgentRequest):
    # Session management
    session_id = models.CharField(max_length=100, default=uuid.uuid4, db_index=True)
    
    # Session state tracking
    chat_messages = models.JSONField(default=list)
    chat_active = models.BooleanField(default=True)
    report_generated = models.BooleanField(default=False)
    
    class Meta:
        indexes = [
            models.Index(fields=['session_id']),
            models.Index(fields=['user', 'chat_active']),
        ]

Session Retrieval Pattern:

# Safe session handling
def handle_chat_message(self, **kwargs):
    user = kwargs.get('user')
    session_id = kwargs.get('session_id', str(uuid.uuid4()))
    
    # Get or create session
    request_obj, created = AgentRequest.objects.get_or_create(
        user=user,
        session_id=session_id,
        chat_active=True,
        defaults={
            'agent': agent,
            'cost': 0,  # No cost for chat
            'status': 'pending'
        }
    )
    
    # Add message to history
    chat_messages = request_obj.chat_messages
    chat_messages.append({
        'role': 'user',
        'message': user_message,
        'timestamp': timezone.now().isoformat()
    })
    request_obj.chat_messages = chat_messages
    request_obj.save()

🛡️ Prevention Strategy

• Always use UUID for session IDs
• Index session_id field for performance
• Implement session cleanup for old sessions
• Test session persistence across requests

---

6. Status Tracking Problems

❌ Common Errors

Requests stuck in 'processing' status
Status not updated after completion
Inconsistent status across request lifecycle

🔍 Root Cause Analysis

• Missing status updates in error paths
• No status transitions defined
• Exception handling bypassing status updates

✅ 5 Whys Learned Solution

Status Lifecycle Pattern:

# 5 Whys status tracking pattern
def process_response(self, response_data, request_obj):
    try:
        # Always update status to processing
        request_obj.status = 'processing'
        request_obj.save()
        
        # Process the request
        success = self.extract_and_validate_response(response_data)
        
        # Update status based on result
        if success:
            request_obj.status = 'completed'
            # Handle successful response
        else:
            request_obj.status = 'failed'
            # Handle failed response
            
    except Exception as e:
        # Always handle errors with status update
        request_obj.status = 'failed'
        request_obj.save()
        raise
    finally:
        # Always set processed timestamp
        request_obj.processed_at = timezone.now()
        request_obj.save()

Status Validation:

# Status transition validation
VALID_STATUS_TRANSITIONS = {
    'pending': ['processing', 'failed'],
    'processing': ['completed', 'failed'],
    'completed': [],  # Terminal state
    'failed': [],     # Terminal state
}

def update_status(self, request_obj, new_status):
    current_status = request_obj.status
    if new_status not in VALID_STATUS_TRANSITIONS.get(current_status, []):
        raise ValueError(f"Invalid status transition: {current_status} -> {new_status}")
    request_obj.status = new_status
    request_obj.save()

🛡️ Prevention Strategy

• Define clear status lifecycle
• Always update status in exception handlers
• Use try-finally blocks for cleanup
• Monitor requests stuck in processing status

---

7. Error Handling Failures

❌ Common Errors

Unhandled exceptions breaking request flow
Users see raw Django error pages
No error logging for debugging

🔍 Root Cause Analysis

• Missing try-catch blocks
• No graceful error recovery
• Poor error messaging to users

✅ 5 Whys Learned Solution

Comprehensive Error Handling Pattern:

# 5 Whys error handling pattern
def process_request(self, **kwargs):
    request_obj = None
    try:
        # Create request object
        request_obj = self.create_request_object(**kwargs)
        
        # Process the request
        response_data = self.make_api_call(**kwargs)
        
        # Handle response
        return self.process_response(response_data, request_obj)
        
    except ValidationError as e:
        # User input error - don't log as system error
        self.handle_user_error(request_obj, f"Invalid input: {e}")
        raise Exception(f"Please check your input: {e}")
        
    except requests.RequestException as e:
        # External API error - log and retry
        self.log_api_error(e, request_obj)
        self.handle_api_error(request_obj, "External service temporarily unavailable")
        raise Exception("Service temporarily unavailable. Please try again later.")
        
    except Exception as e:
        # Unknown error - log everything for debugging
        self.log_system_error(e, request_obj, **kwargs)
        self.handle_system_error(request_obj, "An unexpected error occurred")
        raise Exception("An unexpected error occurred. Please contact support.")

def handle_user_error(self, request_obj, message):
    if request_obj:
        request_obj.status = 'failed'
        request_obj.save()
    # Don't log user errors as system issues

def handle_api_error(self, request_obj, message):
    if request_obj:
        request_obj.status = 'failed'
        request_obj.save()
    # Log API errors for monitoring
    print(f"API Error: {message}")

def handle_system_error(self, request_obj, message):
    if request_obj:
        request_obj.status = 'failed'
        request_obj.save()
    # Log system errors with full context
    print(f"SYSTEM ERROR: {message}")

def log_system_error(self, error, request_obj, **kwargs):
    """Log system errors with full context for debugging"""
    error_context = {
        'error': str(error),
        'request_id': str(request_obj.id) if request_obj else 'None',
        'user_id': kwargs.get('user', {}).get('id', 'None'),
        'agent_slug': self.agent_slug,
        'kwargs': kwargs
    }
    print(f"SYSTEM ERROR CONTEXT: {error_context}")

User-Friendly Error Messages:

# Map internal errors to user-friendly messages
ERROR_MESSAGES = {
    'insufficient_balance': "Insufficient wallet balance. Please top up your wallet.",
    'file_too_large': "File size exceeds limit. Please upload a smaller file.",
    'invalid_format': "Unsupported file format. Please upload a valid file.",
    'api_timeout': "Request timed out. Please try again.",
    'service_unavailable': "Service temporarily unavailable. Please try again later.",
    'unknown_error': "An unexpected error occurred. Please contact support."
}

def get_user_friendly_error(self, error_code):
    return ERROR_MESSAGES.get(error_code, ERROR_MESSAGES['unknown_error'])

🛡️ Prevention Strategy

• Wrap all external calls in try-catch blocks
• Provide user-friendly error messages
• Log errors with sufficient context for debugging
• Test error scenarios during development

---

8. Environment Variable Issues

❌ Common Errors

KeyError: 'N8N_WEBHOOK_5_WHYS'
API authentication failures
Webhook URLs not found

🔍 Root Cause Analysis

• Environment variables not loaded
• Variable name mismatches
• Missing .env file in production

✅ 5 Whys Learned Solution

Environment Variable Pattern:

# Safe environment variable loading
import os
from django.conf import settings

class AgentProcessor:
    def __init__(self):
        # Safe environment variable access
        self.webhook_url = self.get_env_var('N8N_WEBHOOK_5_WHYS')
        self.api_key = self.get_env_var('EXTERNAL_API_KEY')
    
    def get_env_var(self, var_name, default=None):
        """Safely get environment variable with validation"""
        value = os.getenv(var_name, default)
        if not value and default is None:
            raise Exception(f"Required environment variable '{var_name}' not found")
        return value
    
    def validate_configuration(self):
        """Validate all required environment variables"""
        required_vars = [
            'N8N_WEBHOOK_5_WHYS',
            'DATABASE_URL',
            'SECRET_KEY'
        ]
        
        missing_vars = []
        for var in required_vars:
            if not os.getenv(var):
                missing_vars.append(var)
        
        if missing_vars:
            raise Exception(f"Missing required environment variables: {missing_vars}")

Environment Variable Validation Command:

# Create validation script
python manage.py shell -c "
import os
required_vars = ['N8N_WEBHOOK_5_WHYS', 'OPENWEATHER_API_KEY', 'DATABASE_URL']
missing = [var for var in required_vars if not os.getenv(var)]
if missing:
    print('❌ Missing variables:', missing)
else:
    print('✅ All required variables present')
"

🛡️ Prevention Strategy

• Create environment variable validation script
• Use safe access patterns with defaults
• Document all required variables
• Test with missing variables to ensure graceful failure

---

9. N8N Webhook Problems

❌ Common Errors

Connection refused to N8N webhook
Webhook timeout errors
Invalid webhook response format

🔍 Root Cause Analysis

• N8N workflow not active
• Network connectivity issues
• Response format mismatches

✅ 5 Whys Learned Solution

Webhook Validation Pattern:

# 5 Whys webhook handling pattern
class WebhookProcessor(StandardWebhookProcessor):
    def make_request(self, payload):
        """Make webhook request with comprehensive error handling"""
        try:
            # Validate webhook URL
            if not self.webhook_url:
                raise Exception("Webhook URL not configured")
            
            # Test connectivity first
            self.test_webhook_connectivity()
            
            # Make request with timeout
            response = requests.post(
                self.webhook_url,
                json=payload,
                timeout=30,  # 30 second timeout
                headers={'Content-Type': 'application/json'}
            )
            
            # Validate response
            if response.status_code != 200:
                raise Exception(f"Webhook returned status {response.status_code}: {response.text}")
            
            # Validate response format
            try:
                response_data = response.json()
            except ValueError:
                raise Exception("Webhook returned invalid JSON")
            
            return response_data
            
        except requests.ConnectionError:
            raise Exception("Cannot connect to N8N webhook. Check N8N service status.")
        except requests.Timeout:
            raise Exception("Webhook request timed out. Try again later.")
        except Exception as e:
            raise Exception(f"Webhook error: {e}")
    
    def test_webhook_connectivity(self):
        """Test webhook connectivity before making actual request"""
        try:
            test_response = requests.get(
                self.webhook_url.replace('/webhook/', '/ping/'),
                timeout=5
            )
            return True
        except:
            # Webhook connectivity test failed - continue anyway
            return False

Webhook Response Validation:

def validate_webhook_response(self, response_data):
    """Validate webhook response format"""
    required_fields = ['output', 'success']
    
    if not isinstance(response_data, dict):
        raise Exception("Webhook response must be JSON object")
    
    missing_fields = [field for field in required_fields if field not in response_data]
    if missing_fields:
        raise Exception(f"Webhook response missing fields: {missing_fields}")
    
    return True

🛡️ Prevention Strategy

• Always test webhook connectivity
• Implement proper timeout handling
• Validate webhook response format
• Have fallback mechanisms for webhook failures

---

10. Performance and Index Issues

❌ Common Errors

Slow database queries
Missing indexes on frequently queried fields
Session lookup timeouts

🔍 Root Cause Analysis

• Missing database indexes
• Inefficient query patterns
• No query optimization

✅ 5 Whys Learned Solution

Database Index Pattern:

# 5 Whys performance optimization
class AgentRequest(BaseAgentRequest):
    session_id = models.CharField(max_length=100, default=uuid.uuid4, db_index=True)
    
    class Meta:
        indexes = [
            # Session-based queries
            models.Index(fields=['session_id']),
            models.Index(fields=['user', 'chat_active']),
            
            # Status and time-based queries
            models.Index(fields=['status', 'created_at']),
            models.Index(fields=['user', 'status']),
            
            # Agent-specific queries
            models.Index(fields=['agent', 'created_at']),
        ]

Query Optimization Pattern:

# Efficient query patterns
def get_user_active_session(self, user, agent_slug):
    """Optimized session lookup"""
    return AgentRequest.objects.select_related('agent', 'user').filter(
        user=user,
        agent__slug=agent_slug,
        chat_active=True
    ).first()

def get_recent_requests(self, user, limit=10):
    """Optimized recent requests lookup"""
    return AgentRequest.objects.select_related('agent').filter(
        user=user
    ).order_by('-created_at')[:limit]

🛡️ Prevention Strategy

• Add indexes for all frequently queried fields
• Use select_related for foreign key queries
• Monitor slow queries in production
• Test with realistic data volumes

---

🛡️ Overall Prevention Strategy

Pre-Development Checklist

• Study 5 Whys Analyzer patterns before starting
• Plan session management if needed
• Design delayed wallet deduction flow
• Plan comprehensive error handling

During Development Checklist

• Use proper template directory structure
• Always use namespaced URLs
• Implement delayed wallet deduction
• Add comprehensive error handling
• Create proper database indexes

Post-Development Checklist

• Test all error scenarios
• Validate template loading
• Test URL routing
• Verify wallet integration
• Test session management
• Validate environment variables

Production Deployment Checklist

• Run migration validation
• Test webhook connectivity
• Verify environment variables
• Monitor error rates
• Check performance metrics

---

🎯 Key Takeaways from 5 Whys Debugging

1. Template Organization is Critical: Always use proper directory structure
2. URL Namespaces Prevent Errors: Always use namespaced URLs
3. Delayed Wallet Deduction: Never deduct before processing success
4. Session Management: Use UUID-based sessions for complex agents
5. Status Tracking: Implement proper lifecycle management
6. Error Handling: Wrap everything in try-catch blocks
7. Environment Variables: Validate all required variables
8. Performance: Add indexes for frequently queried fields

Following these patterns from the 5 Whys success ensures error-free agent creation.