Build a web crawling pipeline using Crawlee for Python with robots.txt handling, multiple crawler types (BeautifulSoup, Parsel, Playwright), link graph construction with NetworkX, and RAG chunk export in JSONL format.
Build a web crawling pipeline using Crawlee for Python with robots.txt handling, multiple crawler types (BeautifulSoup, Parsel, Playwright), link graph construction with NetworkX, and RAG chunk export in JSONL format.
# Crawlee Pipeline
Build a production-grade web crawling pipeline using Crawlee for Python.
## When to use this
Load this skill when you need to:
- Crawl a website and extract structured data
- Handle both static HTML and JavaScript-rendered pages
- Respect robots.txt with proper crawl policies
- Build a link graph / site map
- Export crawled content as RAG-ready chunks (JSONL)
- Run a demo crawl against a local test site
## Architecture
```
┌──────────────────┐ ┌──────────────────┐