Web Scraping & Data Collection

You are a data engineer building a web scraping and data collection system. Based on the following requirements, create a comprehensive scraping pipeline.

Target Data: [WHAT DATA YOU NEED TO COLLECT - prices, reviews, job listings, news articles]
Source Websites: [TYPES OF WEBSITES - static HTML, JavaScript-rendered SPAs, APIs]
Collection Volume: [NUMBER OF PAGES OR RECORDS TO COLLECT]
Frequency: [ONE-TIME COLLECTION/DAILY/WEEKLY/REAL-TIME MONITORING]
Storage Destination: [DATABASE/CSV/DATA LAKE/API]
Legal Considerations: [TERMS OF SERVICE STATUS AND robots.txt COMPLIANCE]

Build the scraping system across these six sections:

1. ETHICAL AND LEGAL FRAMEWORK
Establish the ethical and legal boundaries for the scraping project. Check robots.txt compliance for each target site and respect disallowed paths. Review Terms of Service for explicit prohibitions on automated access. Implement rate limiting that does not burden the target server, typically one to two seconds between requests as a minimum. Set proper User-Agent headers identifying your scraper rather than impersonating browsers. Comply with copyright law by using collected data only for permitted purposes. Address regional regulations including GDPR implications when scraping personal data from European sites. Create a compliance checklist that the team reviews before starting any new scraping project. Document the legal justification for the data collection activity.

2. SCRAPING ARCHITECTURE AND TOOL SELECTION
Design the scraping architecture based on the target site characteristics. For static HTML sites, implement scraping with requests and BeautifulSoup or lxml for fast parsing. For JavaScript-rendered content, use Playwright or Selenium with proper wait conditions and headless browser configuration. For sites with APIs, reverse-engineer the API endpoints using browser developer tools and make direct API calls which is more efficient and reliable than HTML parsing. Design a modular scraper architecture with separate components for URL generation, page fetching, content parsing, and data storage. Implement a URL frontier that manages which pages to visit next and tracks already-visited pages. Provide code templates for each scraping approach.

3. ROBUST EXTRACTION AND PARSING
Build resilient parsing logic that handles real-world HTML variability. Implement multiple selector strategies using CSS selectors as primary and XPath as fallback for complex structures. Build extraction functions that handle missing elements gracefully rather than crashing. Implement schema validation for extracted data to catch parsing errors immediately. Create extraction configurations that separate the parsing rules from the code so new sites can be added without code changes. Handle pagination patterns including next-page links, infinite scroll with API-based loading, and URL pattern-based pagination. Address anti-scraping measures including CAPTCHAs, IP blocking, and dynamic class names by implementing appropriate countermeasures like proxy rotation and request header randomization.

4. ERROR HANDLING AND RELIABILITY
Build a fault-tolerant scraping system that handles failures gracefully. Implement retry logic with exponential backoff for transient HTTP errors like 429 rate limiting and 503 service unavailable. Create a proxy rotation system for large-scale scraping that distributes requests across multiple IP addresses. Implement request session management with proper cookie handling for sites that require login or session state. Build checkpointing so long-running scraping jobs can be resumed from where they left off after a failure. Create a dead URL queue that tracks consistently failing URLs for manual investigation. Implement circuit breakers that pause scraping when error rates spike to avoid being permanently blocked. Log every request and response status for debugging and monitoring.

5. DATA PROCESSING AND STORAGE
Process and store the scraped data for downstream use. Implement data cleaning immediately after extraction including HTML entity decoding, whitespace normalization, and type conversion for prices, dates, and numbers. Build deduplication logic that identifies when the same content has been scraped multiple times using content hashing or unique identifiers. Design the storage schema optimized for the intended use case including full-text fields for search, normalized relational tables for analysis, and raw HTML storage for reprocessing. Implement incremental updates that detect changed content without re-scraping everything by using Last-Modified headers, ETags, or content checksums. Create data validation reports showing collection completeness, field coverage, and anomaly detection.

6. SCHEDULING, MONITORING, AND MAINTENANCE
Operationalize the scraping system for ongoing collection. Set up scheduled execution using cron, Airflow, or a cloud scheduler with proper dependency management. Implement monitoring dashboards showing pages scraped, success rates, data volume trends, and proxy health. Create alerts for site structure changes that break parsers, which is detected by a sudden drop in extracted fields or increase in null values. Build a site health check that runs a small sample scrape before the full job to verify the site is accessible and the parsing rules still work. Establish a maintenance process for updating parsers when sites redesign. Create documentation for each scraper including target URL patterns, extraction rules, known limitations, and the owner responsible for maintenance.