# Is Passive Workers actually beneficial? — measured, on real hardware

This is an honest evidence report, run on **2026-06-14** against the **real engine** on local Ollama
(`qwen2.5:14b`, `mistral-small:22b`, `gemma3:12b`, `gemma2:9b`). It is not "the tests pass" — it's the
software doing its job and being scored. Where the evidence is thin, that's said plainly.

## Bottom line
**Yes — for what it actually claims to be**, and not beyond that:
- On **time-sensitive questions**, the local council with live web **beats a frontier model
  (`gpt-5-chat`) answering from memory** — measured head-to-head. That is the core promise.
- On **stable knowledge**, they **tie** — so this is *not* a "we beat frontier models" tool, and it
  doesn't pretend to be. The honest edge is **currency, privacy, grounding, and cost**, not raw IQ.
- Its citations are **actually grounded in their sources**, not decorative.

So it is genuinely beneficial for: people who **can't or won't** send their data to a cloud, who need
**current** answers a frozen chatbot gets wrong, who have **no API budget**, or who want **re-checkable
citations** — and (opt-in) a commons of computers doing that work for each other.

---

## The evidence

### 1. Currency moat — council (live web) vs frontier (memory), head-to-head
`scripts/eval_currency_gap.py --run` · frontier = `openai/gpt-5-chat` (paid baseline, no web by
design) · local blind grader · 10 questions · scored 0–10 vs curated references.

| window (currency) | council, live web | frontier, memory | gap |
|---|---|---|---|
| **static** (fairness control) | 10.00 | 10.00 | **+0.00** — tie, as it should be |
| **recent** | 5.00 | 2.50 | **+2.50** |
| **breaking** | 4.00 | 1.00 | +3.00 ⚠ (n=2) |
| **overall** | 6.80 | 5.20 | **+1.60** |

**Read it honestly:** the **static tie is the point** — when currency doesn't matter, a local 14–22B
council does *not* out-think a frontier model. The benefit appears exactly where it's claimed: on
**recent** (+2.50) and **breaking** (+3.00) questions, live grounding wins. The frontier answers
without web *by design* — this measures the value of live grounding, not model weakness.
*Caveats: small samples (recent n=4, breaking n=2 ⚠ — noise at that size), an LLM grader, one curated
reference per question, and quick-depth council. It's a real signal, not a universal verdict.*

> **Replication at deeper depth (2026-06-14, same 10 verified questions, `--depth standard --analysts 2`):**
> static **−0.25** (9.75 vs 10.00, control still ~tie), recent **+2.25** (5.00 vs 2.75), breaking
> **+2.00 ⚠** (2.00 vs 0.00, n=2), overall **+1.20** (6.30 vs 5.10). The moat **replicates** — the
> council wins on recent/breaking and ~ties on the control at both depths. Notably, deeper depth did
> **not** raise the overall score (6.30 vs 6.80 at quick) — honest signal that the edge is **currency
> (live web), not compute depth**. We re-ran the *verified* question bank deeper rather than fabricate
> new "breaking" reference answers we couldn't vouch for (a wrong reference silently corrupts the gap).

### 2. Citations are grounded, not decorative
`scripts/eval_citation_fidelity.py` on a fresh report ("current US federal funds rate…"):
- **Grounded rate: 5/5 verifiable cited claims (100%)**, **mean content overlap 86%**.
- 3 claims unverifiable on live re-fetch (page drift/paywall) — reported separately, never as failures.
- Honest framing the tool itself prints: *grounded = "not obviously fabricated", not "verified true."*
  It's a floor against the common, damaging failure (off-topic/fabricated citations) — and it holds.

### 3. The actual output is good
A real `pw research` run (quick, 2 analysts, **2.7 min, 16 sources**) produced a report that is
**current** (June-2026-dated sources lead), **cited** (`[S#]` with real URLs + dates), and — notably —
**preserves disagreement**: the two analysts differed on the next FOMC date and the report *says so*
rather than faking a consensus. Full sample: [`sample-report.md`](sample-report.md).

### 4. Private-document retrieval works
`scripts/bench_rag.py`: **recall@1 = 7/7** (both dense and hybrid BM25⊕RRF). The corpus is small and a
strong local embedder saturates it (honest caveat) — hybrid is robustness insurance for the long tail.
The benefit that matters: documents are embedded **locally** (`nomic-embed-text`) and never uploaded.

### 5. The "council" (multi-model) value — honestly bounded
`scripts/merge_eval.py` does a **length-controlled, position-swapped** comparison (so a longer answer
can't win just by being longer). A **fresh run today** (3 questions): **raw win-rate 1/3, but
length-controlled win-rate 2/3** — i.e. the merge beats the best single answer even after removing the
verbosity advantage. That's the honest shape: the diversity dividend is real **per word** but small
(n=3) — the council's value is dissent-preservation + currency, not a large raw-quality jump. It does
*not* inflate the win by being longer (the merge was consistently ~half the length of the best single).

---

## What this is NOT (the honest limits)
- **Not a frontier-beater on stable knowledge** — the static tie proves it; don't use it for math/code/
  explanations where a frontier chatbot is better.
- **Small samples** in the currency eval — directional, not a benchmark league table.
- **The network is the maturing track.** The screenshots below are a **real but small** deployment:
  a brand-new operator joined with `pw join` and answered one real job (it even shows up on the
  leaderboard as `mac AE`, rep 5/10). It's genuine, not seeded — but it's *one* node; the map and
  leaderboard fill out only as real operators join (still invite-only). The single-player engine is
  the verified flagship; the federation is real, working code that's early.

## Validated by dogfooding on a real VPS (2026-06-14)
We ran the **actual operator onboarding on a Hetzner VPS** (Ubuntu, a machine we don't develop on):
`pip install passiveworkers` (7s) → `pw join` registered the node and wrote `~/.passiveworkers/join.json`
**owner-only (0600)**. This surfaced — and we fixed — **two real bugs that made `pw join` unusable**:
(1) enrolled nodes were rejected on every authenticated call because those endpoints also demanded the
shared admin token a `pw join` operator never has (now the per-node secret authenticates on its own);
(2) a lone operator defaulted to *not* judging, so jobs failed "no judge node online" (judging is now
on by default). After the fixes, the full loop completes end-to-end (answer → judge → cited result) —
verified locally with a real `done` job; on the VPS it reached the judge stage but hit the 600s deadline
purely because that box was at load ~20 (other workloads), not a code issue.

## See it (screenshots of the real running app)
Desktop + mobile, captured with Playwright against the live UIs — **all real** (a real generated
report; a real answered job; a real single-node operator dashboard).

| Surface | Desktop | Mobile |
|---|---|---|
| Research desk (real report) | ![](img/research-desk-desktop.png) | ![](img/research-desk-mobile.png) |
| Marketplace (council answer) | ![](img/marketplace-desktop.png) | ![](img/marketplace-mobile.png) |
| Operator dashboard (geo + leaderboard) | ![](img/dashboard-desktop.png) | ![](img/dashboard-mobile.png) |

## Reproduce it yourself (keyless unless noted)
```bash
pw research "What is the current US federal funds rate target range?" --quick --analysts 2
python scripts/eval_citation_fidelity.py --report reports/<that-report>.md
python scripts/bench_rag.py
python scripts/merge_eval.py
python scripts/eval_currency_gap.py            # $0 dry run (validate + estimate)
OPENROUTER_API_KEY=… python scripts/eval_currency_gap.py --run   # paid frontier baseline (~$0.10)
```