unclecode/crawl4ai repository overview

⁠🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper.

Crawl4AI is the #1 trending GitHub repository, actively maintained by a vibrant community. It delivers blazing-fast, AI-ready web crawling tailored for LLMs, AI agents, and data pipelines. Open source, flexible, and built for real-time performance, Crawl4AI empowers developers with unmatched speed, precision, and deployment ease.

⁠🧪 Try It

docker run -d \
  -p 11235:11235 \
  --name crawl4ai \
  --shm-size=3g \
  unclecode/crawl4ai:latest

Now open http://localhost:11235/playground⁠ a UI for testing crawls, tweaking configs, and generating JSON for your agents.

⁠🔥 Sample API Call

curl -X POST http://localhost:11235/crawl \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"]}'

Or stream results:

curl -N -X POST http://localhost:11235/crawl/stream \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"], "crawler_config": {"type": "CrawlerRunConfig", "params": {"stream": true}}}'

⁠📦 From Docker Hub

docker pull unclecode/crawl4ai:latest
# Or specific version
docker pull unclecode/crawl4ai:0.6.0rc1-r1

Multi-arch support (amd64, arm64), .llm.env support for OpenAI, Claude, Groq, and others.

⁠📚 Docs

Full docs at docs.crawl4ai.com⁠ – learn how to tweak configs, use browser profiles, write custom filters, or deploy at scale.

⁠❤️ Support the Project

• ⭐ Star on GitHub⁠
• 🧠 Follow @unclecode on X⁠
• 💬 Join our Discord⁠

🖖 Live Long and import Crawl4ai