System Design
Microservices Architecture
A monorepo structure with logically distinct services sharing common utilities and a unified database schema.
PostgreSQL
Gumtree Monitor
gumtree-monitor/
Notifier Service
notifier-service/
Admin Dashboard
admin-dashboard/
Shared Library
shared/
Core Components
Gumtree Monitor
gumtree-monitor/The primary data ingestion engine. Two-phase scraping (Lite → Heavy) with multi-category support.
- Two-phase: Lite scrape → Heavy scrape
- PropertyScraper & VehiclesScraper
- Phone extraction with login
- Image upload to DO Spaces
Notifier Service
notifier-service/The alert delivery mechanism. Database-driven recipients with advanced filtering and WhatsApp delivery.
- Database-driven RecipientManager
- Twilio Content Templates
- Quiet hours enforcement
- Lead quality routing
Admin Dashboard
admin-dashboard/The command center. Manage listings, sellers, vehicles, and WhatsApp recipients.
- Next.js 15 App Router
- Property & Vehicle management
- WhatsApp recipient CRUD
- Server Actions for mutations
Shared Library
shared/Common utilities including BaseScraper, DatabaseManager, and ScraperConfig shared across services.
- BaseScraper → GumtreeBaseScraper
- DatabaseManager base class
- ScraperConfig dataclass
- Utility functions & logging
Data Flow Pipeline
Gumtree SASearch & listing pages
Lite ScrapeCollect URLs
Heavy ScrapeExtract details
DO SpacesStore images
PostgreSQLUPSERT data
NotifierFilter & match
WhatsAppDeliver alerts