# Scrapling Scrapling is an adaptive web scraping framework for [[Python]], built by web scrapers for web scrapers. It handles everything from single HTTP requests to full-scale concurrent crawls, with a focus on resilience against website changes and anti-bot protections. > An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl. ## Key Features ### Adaptive Element Tracking Scrapling's standout feature is its ability to automatically relocate scraped elements when websites change their structure. It uses similarity algorithms to re-identify elements even after DOM changes, making scrapers significantly more resilient and reducing maintenance overhead. ### Anti-Bot Bypass Out-of-the-box bypassing of [[Cloudflare]] Turnstile and other anti-bot systems. Achieves this through browser TLS fingerprint impersonation, stealthy request headers, and HTTP/3 support. ### Fetcher Types Scrapling provides multiple fetcher backends depending on the use case: - **Fetcher**: Fast HTTP requests with browser impersonation - **StealthyFetcher**: Advanced stealth mode with Cloudflare bypass - **DynamicFetcher**: Full browser automation via [[Playwright]]/Chrome - **AsyncFetcher**: Async variants of all the above ### Spider Framework A Scrapy-like API with `start_urls` and async `parse` callbacks. Supports concurrent crawling with configurable limits, multi-session routing (different fetcher types per request), pause/resume with checkpoint persistence, streaming mode, automatic blocked-request detection and retry, and built-in JSON/JSONL export. ### Performance Lightning-fast by Python standards. Uses optimized data structures, 10x faster JSON serialization than the standard library, and memory-efficient internals. ### Developer Experience - CSS selectors, XPath, regex, text search, and filter-based queries - Interactive IPython shell integration - CLI interface for no-code scraping - Auto selector generation for elements - Full type hints coverage - Docker images with all browsers included ## Combining Scrapling with AI Agents Scrapling becomes especially powerful when paired with AI agents like [[OpenClaw]]. Where Scrapling handles the low-level mechanics of fetching, parsing, and adapting to website changes, an AI agent provides the intelligence layer that decides what to scrape, interprets the results, and acts on them. Practical combinations include: - **Autonomous data gathering**: An agent identifies what information it needs, delegates the actual scraping to Scrapling, and processes the results; all without human intervention - **Self-healing scraping pipelines**: When Scrapling's adaptive tracking detects structural changes, an AI agent can evaluate whether the data still makes sense semantically and adjust the scraping strategy accordingly - **Browser automation orchestration**: OpenClaw's browser automation capabilities complement Scrapling's DynamicFetcher; the agent navigates complex multi-step flows (login, form filling, navigation) while Scrapling extracts structured data from the resulting pages - **MCP integration**: Scrapling ships with a built-in MCP server, making it directly accessible as a tool for AI agents that support the [[Model Context Protocol (MCP)]] This combination effectively turns web scraping from a brittle, maintenance-heavy process into an adaptive, agent-driven capability. ## References - GitHub: https://github.com/D4Vinci/Scrapling ## Related - [[AI Agents Web Browsing]] - [[Browser Use]] - [[Agent Reach]] - [[Web Scraping techniques]] - [[Python]] - [[Playwright]] - [[AI Agents]] - [[OpenClaw]] - [[Cloudflare]] - [[Model Context Protocol (MCP)]]