🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

What is the project about?

Crawl4AI is an open-source web crawler and scraper designed specifically for extracting data in a format that's friendly for Large Language Models (LLMs), AI agents, and data pipelines.

What problem does it solve?

It solves the problem of efficiently and effectively extracting clean, structured data from websites in a way that is optimized for use with LLMs. Traditional web scraping often produces noisy or unstructured data that requires significant cleaning and preprocessing before it can be used for AI tasks. Crawl4AI addresses this by providing tools to generate concise Markdown, extract structured data, and manage the crawling process with features tailored for AI applications. It also addresses the frustration of closed-source or expensive web scraping solutions by providing a free, open-source, and community-driven alternative.

What are the features of the project?

Markdown Generation: Creates clean, structured, and "fit" Markdown optimized for LLMs, including citations and references. Offers customizable strategies and uses algorithms like BM25 for filtering.
Structured Data Extraction: Supports LLM-driven and CSS-based extraction, chunking strategies, and cosine similarity for relevant content. Allows users to define custom schemas.
Browser Integration: Offers managed and remote browser control, session management, proxy support, and multi-browser compatibility (Chromium, Firefox, WebKit). Handles dynamic viewport adjustments.
Crawling & Scraping: Supports media extraction, dynamic content handling, screenshots, raw data crawling, link extraction, customizable hooks, caching, metadata extraction, iframe content extraction, lazy loading handling, and full-page scanning.
Deployment: Provides Dockerized setup, API gateway, scalable architecture, and DigitalOcean deployment configurations.
Additional Features: Stealth mode, tag-based content extraction, link analysis, error handling, CORS & static serving, and clear documentation.
Streaming Mode: Process results as they arrive.
Robots.txt Compliance: Respect website rules.
Proxy Rotation: Built-in support for dynamic proxy switching.
URL Redirection Tracking: Captures the final destination after any redirects.

What are the technologies used in the project?

Python: The primary programming language.
Playwright: For asynchronous web crawling and browser automation.
Selenium: (Deprecated, but previously used) For synchronous web crawling.
Docker: For containerization and deployment.
LLMs (various): OpenAI, Ollama, and others supported via Litellm, used for structured data extraction and content filtering.
BM25: An algorithm used for filtering and extracting core information.
CSS Selectors/XPath: For schema-based data extraction.
LXML: For fast web scraping (experimental).
PyTorch/Transformers: (Optional) For advanced NLP features.

What are the benefits of the project?

LLM-Friendly: Output is optimized for use with LLMs.
Fast: High-performance crawling and scraping.
Flexible: Highly configurable and customizable.
Open Source: Free to use and modify, with no API keys required.
Community-Driven: Actively maintained and supported by a community.
Deployable: Easy to deploy using Docker and cloud platforms.
Scalable: Designed for large-scale data extraction.

What are the use cases of the project?

Building Datasets for LLMs: Creating training data for fine-tuning or RAG (Retrieval-Augmented Generation) applications.
Data Extraction for AI Agents: Providing structured information to AI agents for tasks like question answering or content summarization.
Web Scraping for Data Analysis: Gathering data for market research, competitive analysis, or other data-driven projects.
Content Aggregation: Collecting information from multiple websites for news feeds, content curation, or knowledge bases.
Website Monitoring: Tracking changes on websites for price tracking, content updates, or compliance monitoring.
Website Mirroring: Creating copies of websites.
Automated Testing: Testing websites.