crawler17.5kMIT2.0.2Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.
bda-researchIl y a 4 monthscrawler, javascript, spider, scraper @nodelib/fs.walk238.4mMIT3.0.1A library for efficiently walking a directory recursively
nodelibIl y a 9 monthscrawler, NodeLib, fs, FileSystem fdir131mMIT6.5.0The fastest directory crawler & globbing alternative to glob, fast-glob, & tiny-glob. Crawls 1m files in < 1s
thecodrrIl y a 28 dayscrawler, util, os, sys isbot7mUnlicense5.1.30🤖/👨🦰 Recognise bots/crawlers/spiders using the user agent string.
omrilotanIl y a 24 dayscrawlers, bot, spiders, googlebot pdf-parse5mMIT1.1.1Pure javascript cross-platform module to extract text from PDFs.
autokentIl y a over 3 yearspdf-crawler, pdf-parse, xpdf, pdf.js