amit/chat-backend
1
0
Fork 0
mirror of https://github.com/thecyberlearn/chat-backend.git synced 2026-08-18 07:32:52 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
2 Commits 1 Branch 0 Tags 64 MiB
Python 80.7%
JavaScript 8.4%
Cython 3.6%
C 2.8%
HTML 2.3%
Other 2.1%
6d11b80c03
Go to file
Django Template 6d11b80c03 Add comprehensive AI chat integration and URL management features 
Features Added:
- AI_CHAT_INTEGRATION.md: Complete reference for Django → Vue.js AI chat integration
- URL update functionality with validation and accessibility testing
- Business edit interface with re-crawl capabilities
- Enhanced export functionality documentation
- AI-ready data export specifications (JSONL, OpenAI training format, knowledge base)
- URL_UPDATE_GUIDE.md: Comprehensive URL management documentation
- Edit business template with real-time validation

AI Integration Planning:
- Complete data flow architecture (Django scraping → AI processing → Vue.js chat)
- Export format specifications for AI training and knowledge base sync
- Implementation roadmap for seamless backend-frontend integration
- Code examples and API specifications for both systems

URL Management Features:
- Smart URL validation with accessibility testing
- Business information editing with URL update capabilities
- Optional re-crawling after URL changes
- API endpoints for programmatic URL updates
- Enhanced UI with edit buttons and quick actions

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-09-21 12:11:03 +05:30
chat_backend Initial commit: Django chat backend with enhanced web scraping 2025-09-21 11:01:50 +05:30
scraping Add comprehensive AI chat integration and URL management features 2025-09-21 12:11:03 +05:30
venv Initial commit: Django chat backend with enhanced web scraping 2025-09-21 11:01:50 +05:30
.env Initial commit: Django chat backend with enhanced web scraping 2025-09-21 11:01:50 +05:30
AI_CHAT_INTEGRATION.md Add comprehensive AI chat integration and URL management features 2025-09-21 12:11:03 +05:30
db.sqlite3 Initial commit: Django chat backend with enhanced web scraping 2025-09-21 11:01:50 +05:30
info.md Add comprehensive AI chat integration and URL management features 2025-09-21 12:11:03 +05:30
manage.py Initial commit: Django chat backend with enhanced web scraping 2025-09-21 11:01:50 +05:30
QUICK_START.md Initial commit: Django chat backend with enhanced web scraping 2025-09-21 11:01:50 +05:30
README.md Initial commit: Django chat backend with enhanced web scraping 2025-09-21 11:01:50 +05:30
requirements.txt Initial commit: Django chat backend with enhanced web scraping 2025-09-21 11:01:50 +05:30

README.md

Chat Backend - Clean Website Scraper

A clean, simple Django application for scraping websites using Firecrawl AI.

Features

  • Simple Site Management: Add websites to scrape
  • Modern Firecrawl Integration: Uses the latest extract API for structured data
  • Clean Architecture: ~150 lines total instead of 718 lines of legacy code
  • Live Status Updates: Real-time scraping progress
  • Structured Data: Extracts company name, description, and content

Quick Start

  1. Install Dependencies

    pip install -r requirements.txt

  2. Configure Environment

    # Edit .env file
FIRECRAWL_API_KEY=your-actual-api-key

  3. Setup Database

    python manage.py migrate

  4. Run Server

    python manage.py runserver

  5. Visit Application

Architecture

Clean & Simple

  • Models: Site + ScrapedContent (simple relationship)
  • Service: Single ScrapingService class (~80 lines)
  • Views: Clean view functions with proper separation
  • Templates: Responsive, minimal UI

Key Files

  • scraping/models.py - Simple data models
  • scraping/services.py - Clean Firecrawl integration
  • scraping/views.py - Business logic separation
  • scraping/templates/ - Clean HTML templates

How It Works

  1. Add Site: Enter website name and URL
  2. Scrape: Uses Firecrawl extract API with schema
  3. Extract: AI extracts company name, description, content
  4. Store: Saves structured data for later use

Next Steps

This foundation supports your original plan:

  • ✅ Website scraping (completed)
  • 🔄 Content management (PDF, videos, forms)
  • 🔄 AI Q&A generation
  • 🔄 Export to business.json

Benefits Over Previous Version

  • 80% less code (150 vs 718 lines)
  • Actually works with current Firecrawl API
  • Easy to extend for additional features
  • Clean separation of concerns
  • Proper error handling
  • Security best practices

Built from scratch to be professional and maintainable.