mirror of
https://github.com/thecyberlearn/chat-backend.git
synced 2026-08-18 06:32:52 +00:00
Features: - Django REST API backend - Multi-strategy web scraping system (Beautiful Soup, Playwright, Firecrawl) - Anti-detection features (proxy rotation, user-agent rotation) - Data export functionality (JSON, CSV, TXT) - Business and CrawledPage models - Enhanced crawling service with fallback mechanisms 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
1.9 KiB
1.9 KiB
Quick Start Guide
Getting Back Up and Running
1. Navigate to Project
cd /home/amit/projects/chat-backend
2. Activate Virtual Environment
source venv/bin/activate
3. Start Server
python manage.py runserver
4. Access Application
- Open: http://localhost:8000
- Admin: http://localhost:8000/admin/
What We Built
Fresh Start Results
- ✅ Removed old 718-line messy codebase
- ✅ Created clean Django project from scratch
- ✅ Built modern Firecrawl integration (~80 lines)
- ✅ Working website scraper with clean UI
- ✅ Proper environment configuration
File Structure
chat-backend/
├── .env # Environment variables
├── requirements.txt # Dependencies
├── manage.py # Django management
├── chat_backend/ # Project settings
│ ├── settings.py # Clean settings with env vars
│ └── urls.py # URL routing
└── scraping/ # Main app
├── models.py # Site + ScrapedContent models
├── services.py # Clean Firecrawl service
├── views.py # Simple view functions
├── urls.py # App URLs
├── admin.py # Admin interface
└── templates/ # Clean HTML templates
Key Features
- Add Sites: Enter website name and URL
- Scrape Content: Uses Firecrawl extract API
- View Results: Company name, description, full content
- Status Tracking: Real-time scraping progress
Configuration
- Set FIRECRAWL_API_KEY in .env file
- Uses demo mode if API key not configured
- Clean environment variable handling
Next Steps for Your Vision
Based on your original plan, ready to add:
- Content Management (PDFs, videos, forms)
- AI Q&A Generation
- Export to business.json
This is now a professional foundation you can actually build on!