{"full_name":"apify/crawlee-python","owner":"apify","name":"crawlee-python","shard":"1","url":"https://github.com/apify/crawlee-python","description":"Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract structured data with LLMs via Pydantic AI, or control the browser in natural language with Stagehand. Works with BeautifulSoup, Parsel, Playwright, and raw HTTP. Download HTML, PDF, images, and other files. Proxy rotation included.","homepage":"https://crawlee.dev/python/","topics":["apify","automation","beautifulsoup","crawler","crawling","headless","headless-chrome","pip","playwright","python","scraper","scraping","web-crawler","web-crawling","web-scraping"],"tag_pages":["automation","crawler","crawling","headless","headless-chrome","pip","playwright","python","scraper","scraping","web-crawler","web-scraping"],"language":"Python","license":"Apache-2.0","avatar":"https://avatars.githubusercontent.com/u/24586296?v=4","created_at":"2024-01-10T08:44:47Z","pushed_at":"2026-11-13T08:10:22Z","metrics":{"stars":9675,"forks":833,"watchers":44,"issues":636,"prs":1725,"releases":82,"issues_open":71,"issues_closed":565,"prs_open":18,"prs_closed":207,"prs_merged":1500,"discussions":72},"growth":{"day":{"stars":3,"forks":1,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[9657,9666,9668,9668,9668,9672,9675],"forks":[826,830,831,831,832,832,833],"watchers":[null,45,45,45,45,44,44],"issues_open":[null,72,71,71,71,71,71],"prs_open":[null,12,8,7,10,10,18]},"boards":{},"same_owner":[{"full_name":"apify/apify-mcp-server","name":"apify-mcp-server","shard":"1","boards":{}},{"full_name":"apify/awesome-skills","name":"awesome-skills","shard":"1","boards":{}}],"issues":[{"number":925,"title":"Add get_run_state method to BasicCrawler","comments":10,"created_at":"2025-01-21T09:34:55Z"},{"number":2242,"title":"Keep a single `EventManager` per process (event loop)","comments":9,"created_at":"2026-09-19T12:18:19Z"},{"number":2020,"title":"Docs search is broken on crawlee.dev/python","comments":4,"created_at":"2026-07-05T09:54:46Z"},{"number":1842,"title":"Possible deadlock in `SitemapRequestLoader`","comments":4,"created_at":"2026-04-08T16:13:10Z"},{"number":1600,"title":"Pass router as request_handler to crawler - is this intended?","comments":4,"created_at":"2025-12-05T12:56:43Z"},{"number":1953,"title":"Feature Request: Add serpbase.dev as a URL discovery source for research-style crawls","comments":3,"created_at":"2026-06-08T04:29:42Z"},{"number":1744,"title":"Redesign browser pool and the whole browsers subpackage","comments":3,"created_at":"2026-02-12T14:57:50Z"},{"number":1253,"title":"Importing the crawler class takes over 2 seconds","comments":3,"created_at":"2025-06-17T08:06:55Z"}],"discussions":[{"number":2333,"title":"Showcase: ATS Zero-Browser Gateway — Fast protocol-level scraper on Apify","comments":0,"category":"Show and tell","created_at":"2026-10-15T20:50:52Z"},{"number":2281,"title":"Using crawlee-python with a fingerprint-patched Chromium (tested on 1.10.4, Windows, 0 failed sannysoft checks)","comments":0,"category":"Show and tell","created_at":"2026-10-06T20:20:04Z"},{"number":2256,"title":"python/docs/introduction","comments":0,"category":"Comments","created_at":"2026-09-25T14:43:53Z"},{"number":2180,"title":"We’re building a context-aware code review tool and would love some feedback","comments":0,"category":"Show and tell","created_at":"2026-08-20T14:03:56Z"},{"number":2028,"title":"How to keep cookies on request retry?","comments":3,"category":"Q&A","created_at":"2026-07-06T18:21:08Z"},{"number":2019,"title":"Search on your docs page doesn't seem to work","comments":2,"category":"General","created_at":"2026-07-04T09:34:26Z"},{"number":2006,"title":"Realistic Firefox binary to pair with browser_type='firefox'","comments":6,"category":"Ideas","created_at":"2026-07-02T12:37:07Z"},{"number":1967,"title":"add_requests with a token that has an expiry time","comments":1,"category":"Q&A","created_at":"2026-06-15T09:14:27Z"}],"latest_release":{"name":"1.11.2","tag":"v1.11.2","published_at":"2026-11-10T10:12:52Z","body":"## [1.11.2](https://github.com/apify/crawlee-python/releases/tag/v1.11.2) (2026-11-10)\n\n### 🐛 Bug Fixes\n\n- Stop truncating fractional robots.txt crawl-delay to an integer ([#2392](https://github.com/apify/crawlee-python/pull/2392)) ([fa386bb](https://github.com/apify/crawlee-python/commit/fa386bb5a54d641cefbe40b72a61c6ced97a3e87)) by @RaphaelFakhri\n- Fit the default `desired_concurrency` within `min_concurrency` and `max_concurrency` ([#2397](https://github.com/apify/crawlee-python/pull/2397)) ([b71d5af](https://github.com/apify/crawlee-python/commit/b71d5afb8ee888b5d12d0a47519038d3040a10e5)) by @PrashanthLande\n- Pass the crawler state to the rendering type detection run of `AdaptivePlaywrightCrawler` ([#2417](https://github.com/apify/crawlee-python/pull/2417)) ([4d3f989](https://github.com/apify/crawlee-python/commit/4d3f989ed0c435a391a876b7fea1120dc9474646)) by @Mantisus\n- Accept `None` for `proxy_configuration` in `BasicCrawlerOptions` ([#2416](https://github.com/apify/crawlee-python/pull/2416)) ([16858fd](https://github.com/apify/crawlee-python/commit/16858fdeb87616b9db3d8c7bf6ea7f16d1d9f951)) by @vdusek\n- **file-system:** Stop KVS pagination from skipping or repeating keys ([#2431](https://github.com/apify/crawlee-python/pull/2431)) ([7d271c0](https://github.com/apify/crawlee-python/commit/7d271c0db0e3e011d88effaeab5b3dc4d806ef4d)) by @peterolkhov\n\n","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}