01
doc-scraper — a docs site crawled into Markdown, `llms.txt` and a SQLite index your agent searches offline
The Show HN (5 points, no comments, posted at 07:33 UTC by devlithic) is a Go crawler that has been shipping for seventeen months and reached the Show HN list with its MCP server already in the registry. doc-scraper (Apache-2.0, Go 1.26; 98 stars, 251 commits, 247 of them by the author Sriram-PR, earliest commit on main May 6, 2025, titled "refactored codebase") reads a config.yaml of sites, each with a start URL, an allowed domain and path prefix, and a CSS selector for the content, and writes each page as GitHub-flavored Markdown in the site's own directory hierarchy with internal links rewritten to relative paths, plus, when JSONL output is on, pages.jsonl, an llms.txt manifest and an llms-full.txt concatenation in the llmstxt.org shape, regenerated on every crawl. It respects robots.txt, walks sitemaps, rate-limits per host with jitter, retries with backoff, persists crawl state in BadgerDB so crawl --resume picks up where a killed run stopped, and crawl -incremental re-fetches every known page and re-checks it for changes. v2.9.0, on Friday, added the two commands that make it usable without reading the config reference: add, which probes a site "with a handful of polite requests (the page, robots.txt, llms.txt, the sitemap)," detects the framework across "30+ frameworks" by generator tag, DOM and asset signatures (Docusaurus, MkDocs, Sphinx, GitBook and ReadTheDocs among them), proposes a scope clustered from the sitemap with page counts, previews the extracted Markdown "with code-block fidelity numbers," and appends the entry only when you confirm, preserving the rest of the file "byte-for-byte, comments included"; and search, a ranked BM25 query over the corpus with section anchors. mcp-server (stdio only; SSE was removed in 2.x) exposes eleven tools, of which search_docs (SQLite FTS5, stemming, snippets), read_page, list_pages, get_freshness and diff_crawl run "without network access," and crawl_site, get_job_status and cancel_crawl run the crawler as background jobs; a run command takes a JSON task spec on stdin "for orchestration agents that would rather build a JSON payload than assemble shell flags," and rejects unknown fields. Watch mode re-crawls on an interval and persists its schedule. Binaries ship for macOS, Linux and Windows.
The weekend's changelog is what earns the bottom rung of the ladder. v2.8.2 (Friday morning) added tool annotations and "a privacy policy with a README section," and the policy is one sentence: "doc-scraper collects nothing: no telemetry, no analytics, no accounts. All output and state stays on your machine, and the only network requests it makes are the crawls and fetches you explicitly ask for," with a PRIVACY.md behind it; the same release automated publishing to the MCP registry via OIDC and attached a .mcpb bundle to the GitHub release, which has 58 downloads against one or two per platform tarball. v2.9.1 (Friday afternoon) added fuzz targets "for the untrusted-input parsers," hardened the config writer "to refuse degenerate YAML layouts found by fuzzing instead of silently losing site entries," and bumped x/text for "a reachable infinite-loop DoS in Unicode normalization of untrusted HTML." v2.9.2 (Sunday) made a fresh crawl fail "when every attempted page fetch fails, so a fully unreachable site exits nonzero instead of reporting an empty corpus as success," and marked the read-only tools "explicitly non-destructive and idempotent so clients and directory scanners no longer see spec-default destructive hints on them." Four releases between Friday morning and Sunday morning, every entry a bug the author found in his own tool, and no maker text in the Show HN at all. The comparison the README does not draw is with Context7 and the hosted docs MCPs: those serve someone's crawl of the docs from their server, on their schedule; this serves your crawl from your disk, and get_freshness tells your agent how old it is. Machine-facing docs: PRIVACY.md, and a server.json for the registry; no AGENTS.md.
Reach for it when your agent keeps fetching the same framework docs live, page by page, and you would rather it grep a local corpus that a cron job refreshes; it replaces the hosted docs MCP and the "read this URL" habit. Delete the docs-fetching tool calls from the session and the tokens they cost. Tradeoffs: a YAML entry and a CSS selector per site, now drafted for you but still yours to maintain; the corpus is exactly as fresh as the last crawl, which is why the freshness tool exists; stdio only, so it runs where the files are; one author, three contributors, 98 stars, and a Show HN nobody has commented on; and a documentation site that blocks crawlers will simply not be in it, which v2.9.2 at least now reports as a failure.