Methodology

Goals

• Help an agent (or human) pick a default provider per job-to-be-done (JTBD) quickly.
• Surface tradeoffs marketing pages hide: lock-in, privacy defaults, reliability under load, DX for programmatic agents.
• Prefer real URLs and honest confidence over fake precision.

Dimensions

Overall formula

Weighted sum of dimensions. lockInRisk is inverted in the aggregate: contribution = (10 − lockInRisk) × weight. Raw lockInRisk displayed in UI remains higher = more lock-in.

overall = 0.25*capabilityFit + 0.18*pricePerformance + 0.15*latencyReliability + 0.15*developerDx + 0.10*privacyPosture + 0.07*(10-lockInRisk) + 0.10*maturity. Raw lockInRisk stays higher=worse; only the aggregate inverts.

What we never do

• Invent live benchmark numbers without measurement
• Hide as-of dates or confidence
• Let unverified agent insights silently change official scores

The full list, with what Depot says instead and where each rule is enforced: What Depot refuses to say.

How the record is kept

• Freshness. Every score, price and free row carries ageDays and stale against a published policy (/api/freshness). A stale score is still the current editorial score and is labelled stale.
• History (append-only). Each source re-check (npm run check:sources) appends what the stored catalog said and what the official page answered to data/history/. Read it at /api/history and /api/price-changes. A price only changes when a human edits the catalog; re-checks never edit amounts.
• Price history: observed vs catalog. Every pricing row keeps two dated series. Catalog is the amount Depot stores and quotes. Observed is the amount the official source page itself stated for that SKU, in that unit, on that run. It is read only when it is unambiguous: an exact raw API field, a schema.org offer named for the SKU in the row's own unit and backed by the visible page text, or the exact catalog amount with the SKU name on the page. Otherwise it is recorded as null. Units are never converted or blended. When the observed amount moves, a source_price_changed event is logged with an evidence snippet, and the row joins pendingReprice. Re-pricing the catalog is a separate, reviewed step (an amount_changed event). The record is append-only and kept per run: every run has a runId and an observedAt time, a second run on the same day is a second observation (never a replacement), past observations and events are never edited or backfilled, and a check (scripts/check-history-append-only.mjs) compares every change with git HEAD and fails on any same-day replacement. Days are summaries of runs (/api/history?runs=1). Per-row timeline: /api/history/{rowId}.
• Attestations (provable priority). After every pricing-watch run, scripts/attest.mjs appends one entry to data/history/attestations.json, served at /api/history/attestations. An entry holds the sha256 of the canonical JSON (RFC 8785 JCS rules: keys sorted, no whitespace) of price-history.json, check-runs.json and page-status.json, then dataHash = sha256(canonical(files)), prevHash (the previous entry's hash) and entryHash = sha256(canonical(entry without entryHash)). Editing any past history byte breaks every later hash. Entries of kind git_backfill were computed from the files at past git commits, dated with the commit time (recorded by git, not notarised). Entries of kind run are written by the run itself. To verify:
  1. Fetch /api/history/attestations and keep the head.entryHash you saw (that is your receipt).
  2. Recompute the chain: for each entry, sha256(canonical(entry minus entryHash)) must equal entryHash, and prevHash must equal the previous entry's entryHash (64 zeros for the first entry).
  3. Recompute the data: fetch the three history files from the repo (or a licensed copy), parse them, take sha256(canonical(json)) of each, and compare them with head.files. The response also carries the server's own verification result.
  4. Later, check that the entry you kept still appears unchanged at the same seq. If it does, nothing recorded up to then has been rewritten. In the repo: node scripts/attest.mjs --verify. npm run validate-data also fails if the chain breaks, an entry is altered or removed compared with git, or the history changed without a new attestation.

  Bitcoin timestamps (OpenTimestamps, since v0.11.1). Each chain head's entryHash is timestamped with OpenTimestamps on its free public calendars, by the same daily pricing-watch run. Only that 32-byte hash leaves the box: a random 16-byte nonce is appended and hashed first, so each calendar receives 32 bytes and never any data or file. Proofs are stored append-only as data/history/ots/<attestationId>.ots (attestationId = 4-digit seq + the first 16 hex characters of entryHash). A proof is pending until a calendar's Bitcoin transaction confirms (usually within hours); the next run runs ots upgrade and records confirmed, the block height and the block time. Every entry on /api/history/attestations carries ots.status and ots.download. Publishing these fingerprints was approved by Ethan Hawkes on 2026-10-21 at 01:52 ET (approvalRef: chat, answering a card: "Yes, publish the daily history fingerprints to OpenTimestamps").

  Honest limit. A confirmed proof shows the entryHash existed no later than the confirming block's time, and because every entryHash commits to all earlier entries, the chain before it did too. The eight backfilled entries (seq 0 to 7, kind git_backfill) were stamped on 2026-10-21 (stampBasis: stamped_later): OTS proves they existed by then, not at their earlier at (git commit times, which stay unnotarised evidence).

  1. Install the client: pip install opentimestamps-client. Download the proof from ots.download (for example /api/history/attestations/ots/0007-b67e9d0f2b6939e1.ots) and note the entry's entryHash; recompute it first (step 2 above) so you are not trusting the server.
  2. Run ots verify -d <entryHash> <attestationId>.ots. With a local Bitcoin Core node (pruned is fine) it prints "Success! Bitcoin block N attests existence as of …". Equivalent: write canonical(entry minus entryHash) to a file and run ots verify -f that-file <attestationId>.ots, since entryHash is the sha256 of exactly those bytes.
  3. Without a node: ots --no-bitcoin verify -d <entryHash> file.ots prints the block height and the merkle root to look up; compare it with that block's header in any block explorer. While a proof is still pending, ots upgrade file.ots fetches the completed proof from the calendars.
  4. In the repo: node scripts/ots.mjs --verify checks that every attestation since the first stamped one has a proof of its own entryHash and that proofs are append-only (an upgrade may only add attestations); npm run validate-data runs the same check. A calendar outage never fails the daily run: it warns, and the next run retries.
• Failure modes per job. Via agent classes, and via direct rules that quote the failure definition they rely on (rules). Jobs no rule fits keep a reason code.
• Corrections. Every data fix is logged with before/after and source: corrections log. Feeds: RSS, Atom, JSON Feed.