{"full_name":"yujiosaka/headless-chrome-crawler","owner":"yujiosaka","name":"headless-chrome-crawler","shard":"1","url":"https://github.com/yujiosaka/headless-chrome-crawler","description":"Distributed crawler powered by Headless Chrome","homepage":"","topics":["headless-chrome","puppeteer","jquery","crawler","crawling","scraper","scraping","chrome","chromium","promise"],"tag_pages":["chrome","chromium","crawler","crawling","headless-chrome","jquery","promise","puppeteer","scraper","scraping"],"language":"JavaScript","license":"MIT","avatar":"https://avatars.githubusercontent.com/u/2261067?u=65289216fa6e896d4c00eab96d8dd92f1bf054cf&v=4","created_at":"2017-12-02T07:12:12Z","pushed_at":"2023-04-29T20:48:51Z","metrics":{"stars":5631,"forks":404,"watchers":112,"issues":135,"prs":256,"releases":20,"issues_open":29,"issues_closed":106,"prs_open":4,"prs_closed":44,"prs_merged":208,"discussions":0},"growth":{"day":{"stars":2,"forks":0,"base_day":"2026-11-15"}},"history":{"dates":["2026-11-10","2026-11-11","2026-11-12","2026-11-13","2026-11-14","2026-11-15","2026-11-16"],"stars":[5631,5631,5631,5630,5629,5629,5631],"forks":[403,403,403,403,404,404,404],"watchers":[null,112,112,112,112,112,112],"issues_open":[null,29,29,29,29,29,29],"prs_open":[null,4,4,4,4,4,4]},"boards":{},"same_owner":[],"issues":[{"number":314,"title":"Stuck for no reason?","comments":6,"created_at":"2018-08-31T08:37:44Z"},{"number":302,"title":"duplicated url are crawled twice","comments":6,"created_at":"2018-07-31T14:58:56Z"},{"number":148,"title":"[Feature Request] Add support for multiple sitemaps","comments":6,"created_at":"2018-03-08T13:54:15Z"},{"number":338,"title":"Crawl stops on non-www URLs","comments":5,"created_at":"2019-01-09T17:25:57Z"},{"number":377,"title":"Is there a way to scroll?","comments":4,"created_at":"2021-01-13T04:14:25Z"},{"number":293,"title":"Queueing same url on multiple workers in cluster with Redis cache results in duplicates","comments":4,"created_at":"2018-07-12T17:02:57Z"},{"number":118,"title":"[Feature Request] Add support for WARC file format","comments":4,"created_at":"2018-02-22T23:39:15Z"},{"number":376,"title":"Crawling site with maxDepth > 2 causes hang","comments":3,"created_at":"2020-12-18T02:56:50Z"}],"discussions":[],"latest_release":{"name":"1.8.0","tag":"1.8.0","published_at":"2018-06-11T01:27:32Z","body":"### Added\r\n\r\n- Set `previousUrl` to `onSuccess` argument.\r\n- Set `options`, `depth`, `previousUrl` to errors.\r\n- Support `customCrawl` for [HCCrawler.connect()](https://github.com/yujiosaka/headless-chrome-crawler/blob/master/docs/API.md#hccrawlerconnectoptions) and [HCCrawler.launch()](https://github.com/yujiosaka/headless-chrome-crawler/blob/master/docs/API.md#hccrawlerlaunchoptions)'s options.\r\n- Add [Dockerfile](https://github.com/yujiosaka/headless-chrome-crawler/blob/master/Dockerfile) and [tips](https://github.com/yujiosaka/headless-chrome-crawler/blob/master/docs/TIPS.md) for using [Docker](https://www.docker.com/).\r\n\r\n### Changed\r\n\r\n- Drop `newpage` event.\r\n- Update [Puppeteer](https://github.com/GoogleChrome/puppeteer) version to 1.5.0.\r\n\r\n### Fixed\r\n\r\n- Fix a bug of not marking skipped requests correctly.\r\n- Fix `requestfinished` event's argument as described in the [API reference](https://github.com/yujiosaka/headless-chrome-crawler/blob/master/docs/API.md).","is_prerelease":false},"readme":true,"readme_zh":false,"news":[],"stale":false,"snapshot_at":"2026-11-16T09:23:14Z","history_start":"2026-09-06"}