# AI Hacker Daily — 2026-08-18 An agent can generate any benchmark now, so grade the ones that can lose.

A runtime gate for agent tool calls, `pip install doberman-core`, Apache-2.0, Python 3.11+. Two deployment shapes: an MCP proxy that wraps tool servers, or native pre-execution hooks into Claude Code, Codex CLI and OpenClaw. Every tool call gets exactly one verdict before it executes — PASS, AUTH (pause for a human), or BLOCK — and two properties are stated as non-negotiable: "fail closed," where any error or unhandled case denies the action, and "raise-only learning," where guardrails can auto-tighten but never silently loosen. That is a reasonable design and it is not why this is the top pick. It is the top pick because `docs/BENCHMARKS.md` is the best evidence document this newsletter has graded, and it opens by telling you what it is for: "Doberman's protection claims should resolve to a number you can reproduce, not an adjective," with "the results — **failure cases before wins**, on purpose." It separates `asr` from `asr_strict` because an AUTH is not a block — "a human still has to say no" — and then it does the thing nobody does, which is put that human in the eval: `asr_under_fatigue` and `auth_burden` model an operator who rubber-stamps some fraction of prompts, with the flat statement that "an AUTH-heavy defense is only as strong as the human answering it." That is the 66.3% approval-accuracy study from our 08-07 edition and Anthropic's own 97%-approval figure from 08-10 turned into a parameter of the measurement instead of a caveat under it. The in-repo detection corpus is ~137 labeled rows across injection, exfiltration, secrets, destructive, encoded and benign, deterministic and reproducible from a cold clone with no external dependency — and it "is *not* filtered to cases the engine wins. Pure natural-language injection scores **TPR 0.0** — the objective layer is structurally blind to it... and the corpus says so rather than hiding it." Calibration also surfaced that reading an `.env.example` template gets over-blocked because the secret-path rule matches `.env.*` fail-closed, and that is in the docs too. On the AgentDojo suite they caught a leak in their own harness: the adapter sets `source_context` by ground truth, so a provenance-driven result would measure the adapter's labels rather than the guard, and they neutralize the field and report the leaky arm "**only** to quantify that leak, never as a headline." The poisoning eval ships two arms and an honesty control — a benign action that does normalize, "so the eval cannot pass vacuously by reporting 'nothing normalizes'." The docs then list the ways around it: homoglyph limits, bare high-entropy hex false negatives, oversized-blob detection evadable by splitting, static egress classification bypassable through runtime command construction or DNS rebinding. The punchline is in the thread, from the author: the README intro was his, "the rest was meant to just be something agents ingest and use to set up Doberman hahaha, didn't think anyone would read the full thing." The most rigorous disclosure on the board was written for machines to read. Reach for it if you have agents holding credentials against a production system and your current control is a permission prompt at 4pm. Delete the hand-rolled regex denylist in your hook script, and delete the belief that an approval dialog is a control rather than a queue. Tradeoffs: it scores TPR 0.0 on pure natural-language injection and says so, so this is a floor and not a perimeter; the strongest published numbers cover the deterministic layer, while the adaptive layer's AUC comes with a warm-sufficiency caveat telling you to read every result next to its bucket size; the AgentDojo run needs an operator-supplied package, so only the synthetic and corpus suites reproduce from a cold clone; 207 stars against 23 forks and **zero watchers**, 43 open issues, one author at 304 of roughly 360 commits; PyPI reports no license field while the repo is plainly Apache-2.0, the sixth time this quarter the registry has been the unreliable half; the repo was pushed at 01:45 UTC this morning and shipped 0.18.1 three days ago, so pin it; and the thread's sharpest comment is free and correct — "Give the agent a read-only postgres user. Boom, I just saved you a startup," which is the right first move before any of this.

Five accounts, $100,000 of simulated capital each, started 27 July: GPT-5.6, Claude Fable 5, Grok 4.6, Gemini 3.1 (joined 4 August), and a rules-based System baseline that does not think. As of yesterday afternoon the baseline is at **+15.92%**, GPT-5.6 at +5.95%, Claude at +3.84%, Grok at +2.50%, Gemini at **-0.08%**, and the S&P 500 at +4.58%. So the frozen rulebook beats every model and the index; three of four models beat the index; one is underwater after three weeks of a rising market. Every trade goes on a public ledger with "entry, exit, and the exact reason it closed — target hit, stopped out, signal flip," and each model reviews its closed trades daily and rewrites its own playbook, which means the models are getting the adaptive advantage and losing anyway. This is in the slate because of what it is willing to report, not because of what it proves. Three weeks of paper trading is not a finding about model capability, and the operators say so — "paper trading is simulated; past simulated results never promise anything about real money." What it is, is the rarest artifact in the category: a live scoreboard whose null hypothesis is winning, published continuously by the people running it, in a format where they cannot quietly restate the baseline later. Contrast that with every vendor eval we have graded since May, where the comparison arm is chosen after the result is known. The honest way to read today's standings is not "LLMs cannot trade" — it is that a fixed rulebook is the control condition almost nobody runs, and when someone does run it in public, it is competitive enough to be embarrassing. It went up on Show HN yesterday and drew 12 points and three comments, so nobody has stress-tested the ledger either. Reach for it as a reference implementation of a public control condition, and as an argument to run one before you put a model in a decision loop. Delete the internal deck claiming your agent beats a heuristic, if the heuristic was never actually run. Tradeoffs: this is a scoreboard, not something you install; the landing page does not publish fees, slippage, fill assumptions, data source or rebalance cadence, and those choices can move a 22-day result more than the models do; five accounts and one baseline is a sample size that cannot separate skill from variance over three weeks; the operators built and run the thing they are measuring; and a "daily learning loop" where each model rewrites its own playbook means the strategies are not fixed, so what is being compared drifts day to day.

Mermaid is a fine format and an unpleasant thing to nudge by hand when a label is one word too long. Visimer is the missing half: it renders with Mermaid.js unmodified, then maps the resulting SVG elements back to a concrete syntax tree so it can correlate what you clicked to a position in the source. Double-click a label and type on the diagram, nodes grow as you type; drag to connect or reorder. The property that makes it usable in a repo rather than a toy is the round-trip: "unknown syntax is preserved verbatim; your diff is only what you changed." Install is `npm i @visimer/core @visimer/dom mermaid` and it is an embeddable component library rather than a hosted service, so it drops into your own markdown editor. It earns its rung on this ladder for a small thing that most projects skip: it ships a capability matrix that names its own gaps. Twenty-two of twenty-three Mermaid diagram types support full editing; zenuml is view-only; and seven more — journey, timeline, quadrant, kanban, mindmap, treemap, packet — support viewing and item editing but not structural editing. That is a table where eight of the boxes are unflattering, published by the people who would have gotten away with "supports Mermaid diagrams." It is a much smaller act of disclosure than the top pick's, and it is the same species: a claim written so a reader can catch it being wrong. Reach for it if you keep architecture diagrams as Mermaid in the repo and the diffs are unreadable because your editor rewrites the whole block. Delete the round-trip through a drawing app and the exported PNG that immediately goes stale. Tradeoffs: 38 stars, zero forks, **zero watchers**, created 22 July, so there is no production track record; structural editing is missing for eight diagram types including mindmap and timeline, which are common in exactly the docs people want to edit visually; it is a library you integrate rather than an app you open, so budget real work to get value; the npm packages went 0.1.0 to 1.1.1 in four weeks with the last release two days ago; and the repo was pushed at 01:33 UTC this morning, so the version you install today is not the one anyone has been running.

The template builds Foreman, a four-station pipeline that takes an issue and returns a draft pull request: a Classifier judging type, priority and actionability; an Analyst producing an implementation plan with acceptance criteria; an Implementer that executes and runs verification; and a Reviewer that independently validates the result. Humans review the draft PR before merge, and "local runs are treated as untrusted, so changes to GitHub wait for your approval." MIT, TypeScript, wired to GitHub, optionally Linear, and deployed on Vercel with Blob storage. It has taken 891 stars in six days. The reason it sits at this specific rung is the `evals/` directory, which is more thoughtful than most shipped agent products and completely unpopulated with outcomes. There are twelve eval files: a full-pipeline eval, three routing evals (`classifier-first`, `labels-follow-classification`, `needs-clarification`), and seven safety evals — `prompt-injection`, `no-direct-push-to-main`, `write-requires-approval`, `ship-gate-approve-resume`, `ship-gate-parks`, `factory-brain-write-parks`, `read-only-question`. Somebody sat down and wrote the failure modes of an autonomous PR-writing pipeline as executable tests, which is the correct instinct and a real contribution to read. What is nowhere in the repo is a single run of them. No pass rate on the prompt-injection eval, no number on how often the ship gate holds, no results file, no tags, no releases. This is the second time in five editions we have found a full rig with no published run — NVIDIA's Switchyard on 08-14 shipped pinned agent versions, dataset allowlists and a control condition, and also no results — and it is becoming the characteristic artifact of the moment: the measurement apparatus is now cheap enough to ship as a template, and running it in public still is not. Reach for it if you want a readable, opinionated reference for how to decompose an agent pipeline into stations with an approval boundary, and especially for the eval suite as a checklist of what to test. Delete the single mega-prompt that reads an issue and opens a PR in one shot. Tradeoffs: no published eval results at all, which is the whole point of the rung it occupies; **891 stars against one watcher** and 58 forks, a ratio that reads as reach rather than review, the same shape flagged on Paperclip on 08-10; it is a template rather than a product, so you own it after `git clone`; hard dependencies on Vercel, Vercel Blob and a GitHub connector make "self-hosted" a stretch; created 12 August with no tags or releases; and a pipeline whose output is a draft PR relocates the review burden rather than removing it, which is precisely what today's Snowflake disclosure is about.

OpenRouter for agent harnesses rather than models: one API in front of Codex, Claude Code and Hermes, with sessions, streaming, files, cancellation and failure handling, under something the project calls the Unified Harness Protocol. The community edition is Apache-2.0 and self-hosted as a single ~700 MB Docker image running three services — a Next.js console on 3000, a Responses-API gateway on 8080, a runner on 8081 — with SQLite and files on a volume. The harnesses are installed at first runtime rather than bundled, for licensing reasons, and the licensing is stated honestly: Codex Apache-2.0, Claude Code under Anthropic's terms, Hermes "license unspecified by upstream." The docs also carry a known issue that reads like a confession: "any value that leaves out `hermes` makes the container exit immediately with status 1 and no error message." Where it lands on today's ladder is the benchmark. There is one — the co-founder posted in the thread that across harness/model/tool combinations "one combination is 99.8% cheaper, and one combination is 3.2× faster," with a link — and it lives in a Hacker News comment rather than in the repository you install. Grade that against the two standards this newsletter has set: Ante's eval page on 08-11 (pinned build per row, standard error, dollar cost, raw run link) and the grafana-agento11y README on 08-17, where the disclosure ships inside the thing you install. A routing product's entire value proposition is a comparison, so the comparison is the artifact, and a figure that strong belongs where a reader trips over it. The thread also did the harder work on the premise: Bnjoroge asked three separate times what routing across coding harnesses actually buys you, and the maker conceded it — "You are right, for coding scenario, I also stick with one (CC in my case)" — landing on the real case, which is packaging a harness into a product backend where you need pre-configured instructions, MCP tools and skills. That is a narrower and more believable pitch than the tagline. Unanswered in the thread: kiops asking how it handles harness-specific features "without becoming a lowest-common-denominator API," which is the question the whole abstraction rests on. Reach for it if you are shipping an agent as part of a backend and want the harness to be a swappable dependency rather than a rewrite. Delete the bespoke subprocess wrapper you wrote around one CLI, if you have already discovered you need a second. Tradeoffs: the cost and latency numbers are in a comment and on the cloud product's page, not in the community repo, and nothing in the repo reproduces them; the community edition is the free tier of a paid cloud that sells the tracing and managed sandboxes, so the interesting half is the upsell; default credentials are `harnessrouter` / `harnessrouter`; 112 stars, 7 forks, one watcher, zero open issues on a repo created 9 August; it crossed our pool on 24 July on Product Hunt and is back today as a 9-point Show HN, so this is a relaunch rather than a debut; and the maker's own answer says single-harness users — most readers — do not need it.

The claim is the loudest on the board: "compresses Key-Value (KV) cache tensors by up to 384x while maintaining >94% semantic retention," via FSQ with dynamic 16-dimensional latent mapping, in something described as a hardware-software co-designed memory fabric. The benchmark table reports VRAM going from 48.00 MB to 0.12 MB at 4,096 tokens, semantic retention between 94.15% and 95.84%, and pipeline latency of roughly 14–20 ms marked "CUDA Event Verified." AGPL-3.0 with a commercial dual-license, Python 3.10+, PyTorch 2.0+. Read the metric column and the whole thing dissolves. Semantic retention is **cosine similarity**, and there is no model named, no dataset named, no task named — no perplexity, no retrieval accuracy, no long-context recall, nothing a compressed KV cache could actually fail at. The quickstart compresses a *random tensor*. So the headline is a reconstruction-fidelity score on noise, which is a measurement that structurally cannot come out badly: you compress something with no semantics and report how much of the no-semantics survived. The latency figure carries no batch size, sequence length or hardware. The repository description says 384x while the Show HN title says ~300x, and neither is anchored to a workload. The thread is exactly one comment, and it is the entire audit — colingauvin: "This should be trivially demonstrable if it actually works. Cosine similarity is not a good metric, however, for attention compression." One person, one sentence, and the claim is closed. To be fair to it: the code is published under AGPL, so unlike most things in this category it is checkable, and 300x-class KV compression is a real research direction rather than nonsense. The problem is the shape of the evidence, not the ambition. Reach for it only if you want to run the missing experiment yourself — load a real model, compress a real cache, and report perplexity or a retrieval score, which is a weekend and would be the first genuine number this project has. Delete nothing on the strength of what is published today. Tradeoffs: four stars, zero forks, one watcher, created 13 August, the repo is a Jupyter notebook; the headline metric is not a task metric; AGPL-3.0 means any network-accessible service using it must open-source its backend, so the commercial license is the actual path for most readers; and this is the specimen Dan Luu's essay describes — a plausible mechanism, a spectacular ratio, and a benchmark selected so that failure was never on the table.

**Also on the board.** Two Mac terminals for supervising coding agents launched within hours of each other and neither published a number: Saggar took 40 points and 45 comments on Show HN, and Shepherd Terminal took #6 on Product Hunt with 89 votes. Both are closed source, both are macOS-only, Saggar requires an account and macOS 26 Tahoe, and the Saggar thread named four more competitors unprompted — CMUX, herdr, captain-miao and a commenter's own scrolling-window-manager build — which is what a commoditized category sounds like. A commenter also found the linked GitHub repo, Releases and Issues URLs all 404ing, and the best exchange in the edition came when rafram quoted the site's own copy ("What says no is the account: nothing pairs that is not signed in as you") with "I am begging Claude to stop treating the English language this way," and the maker conceded that "load bearing colon is doing a lot of work there." Elsewhere: a live feed scoring Hacker News comments for guideline violations with a fine-tuned embedding model publishes no accuracy, precision, recall or false-positive rate, which is a hard drop on a day like today; Deepmark took #12 on Product Hunt at $10/month claiming sub-100ms search over a 10k-item library with no methodology and no repo; and the day's fourth-biggest story is a report that Israel built a fake think tank apparently to seed AI chatbots, which is the poisoning attack aimed one layer below every number above. **Pool:** 84 candidates — Hacker News front 29, Product Hunt 24, Show HN 19, GitHub trending 12 across four language feeds. Reddit returned nothing for the seventeenth consecutive week.