docs: owner decisions on statistics, and the sweep and statistics plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2

docs/ROADMAP.md

@@ -12,6 +12,7 @@ Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federati
 - [x] P4 Groups and privacy features: v1.6.0, deployed 2026-10-01; communities, circles and local-only located posts verified by tests. v1.6.1 adds the pasture (`tools/pasture/`): live interop with GoToSocial 0.22.1 passes all 25 checks, three runs in a row. Lemmy and a live Mastodon circle member are not run yet; the pasture has GoToSocial only
 - [x] P5 Lose nothing: v1.7.0 to v1.9.1, deployed 2026-10-01; parsing, typed details, provenance, downvotes, tombstones, federated blocks, all checked live against GoToSocial. Book reviews and forum threads keep their raw form only (typed in P7 and P8)
 - [x] P6 Emoji, polls, quotes, reactions, cards, players: v1.10.0 to v1.15.0, deployed 2026-10-01; polls, link cards, ranged video streaming and quote policies checked live against GoToSocial
+- [ ] T/M Test sweep, the full pasture and the interaction ledger (see "Sweep and statistics" below)
 - [ ] P7 Threads, communities, moderation, the social graph
 - [ ] P8 Signatures, discovery, the long tail
 
@@ -131,6 +132,20 @@ and circles (see Owner decisions).
 | Who may quote a persona (default) | **Anyone, automatically**, as on Mastodon, for public and unlisted posts only. Each persona can change its default (followers only, or nobody) and each post can be changed; a granted quote can be revoked. Followers-only posts and DMs can never be quoted. |
 | Quote permission address | `/peasants/{name}/parrot-licences/{id}` |
 
+### Owner decisions on statistics (2026-10-03)
+
+| Question | Decision |
+|---|---|
+| Fediverse statistics | **Recorded now, published later as per-server aggregates only.** The following are each kept for 90 days as an event that names the remote *server*, never a remote account or a local persona: every inbox answer, every processed activity, every delivery attempt and every outbound request. Each day folds into per-server counters kept indefinitely, plus weekly server snapshots. A future public page, `/stargazing`, shows per-server aggregates for educational use, never per account and never per persona. |
+| Remote accounts in statistics | **Never stored.** Distinct accounts per server per day are counted with a keyed hash. Its key is made for that day, kept only until the day's rollup and then destroyed, so the hashes can be neither reversed nor linked across days. |
+| Local side in statistics | **Only the kind of local actor** (person, group, application), **and only on public and unlisted traffic.** DMs, followers-only and circle traffic are one "private" class, never broken out per server in public. Circles are never named, whether as a kind or as a reason. Fetches of our own documents are counted per day, never per server. |
+| Reading-driven traffic | **Counted per day, never logged per event:** the media proxy, lookups a client asks for, and the client API per endpoint group (admin only). Client app names are not recorded. |
+| Describing servers | **Every server we exchange activities with is described weekly**, from its NodeInfo (including the user counts it publishes) and its Mastodon instance API, never its contact account. These requests are unsigned, because they are not ActivityPub documents. Never on read. |
+| Server locations | **City and network (ASN) from the offline DB-IP Lite databases** (CC BY 4.0, attributed), downloaded monthly outside the app. The location comes from the address we connected to; an inbound sender's address is never recorded, and no address is stored. In public: city and network only for servers reporting at least 10 users and not behind a CDN; the country otherwise; only the CDN's name for CDN-fronted servers. The admin sees everything. |
+| Crawler | **Off by default** (`Statistics:Crawler:Enabled`). When on, it identifies as `PrivaPub-Stargazer/<version> (+https://privapub.thepra.dev/stargazing)`, where `/stargazing` explains it and how to opt out. It honours robots.txt (an unreachable robots.txt means "keep out") and domain blocks. It visits one server a minute, each at most weekly, and at most 5000 servers. It reads only robots.txt, NodeInfo, the instance API and the peers list, never accounts, posts or directories. Crawled servers stay marked as crawled. |
+| A remote account deletes itself | **Its posts are kept but hidden everywhere** (`Post.AuthorGone`): from timelines, profiles, search and lookups by id. Its follows and timeline rows go, as before. |
+| Signed-in smoke check in production | **An undiscoverable persona**, whose read-only token is the Gitea secret `PRIVAPUB_SMOKE_TOKEN`; the deploy checks `verify_credentials`, home and notifications with it. The owner creates both. |
+
 ## Libraries (researched; no maintained .NET ActivityPub library exists, so Letterbook and Iceshrimp.NET both wrote their own)
 
 | Area | Choice |
@@ -521,6 +536,25 @@ it, raw where it doesn't.
   - Pixelfed `place` (display only, never re-federated);
   - Bluesky bridging per persona.
 
+### Sweep and statistics (T and M, planned 2026-10-03, between P6 and P7)
+
+Two tracks, interleaved so that the test host exists before the ledger, and the ledger starts collecting early, because
+statistics gain value with every day recorded.
+
+| Step | Content | Tag |
+|---|---|---|
+| T1–T4 | Tests stop sharing state they don't own. CI runs every test against a throwaway mongod. The whole server runs under test (`WebApplicationFactory`). Nothing answers 500. | v1.15.1 |
+| M1–M6 | The interaction ledger: inbox answers, handler verdicts, delivery attempts, outbound requests, served and client traffic. Provenance fixes (fetched records, stored extensions, group-wrapped Update/Delete). | v1.16.0 |
+| T5–T8 | Coverage sweep over HTTP: OAuth, `/clientapi`, the Mastodon API, federation GETs, inbox gaps, jobs, migrations, pages. | v1.17.0 |
+| M7–M10 | Daily rollups (`InstanceDay`, `ServerDay`). Every touched server described weekly (NodeInfo usage, instance API, snapshots). Geolocation (DB-IP Lite city and ASN). Admin statistics API under `/clientapi/admin/statistics`. | v1.18.0 |
+| T9–T14 | The pasture as plugins. GoToSocial gaps, then Mastodon, Misskey/Sharkey, Akkoma and Lemmy 1.0, each run also checking that peer's statistics. | v1.18.x |
+| M11 | The opt-in crawler (`PrivaPub-Stargazer`) and the `/stargazing` explainer. | v1.19.0 |
+
+Later, and not part of these steps: the public `/stargazing` statistics. It is anonymous, cached and rate-limited, and
+covers per-server software, self-published counts, location as projected by the rule above, availability buckets and
+the public activity mix. Per-server activity is shown only as order-of-magnitude buckets, and only for servers that
+report at least 10 users; smaller ones fold into one "small servers" aggregate.
+
 ### Cut or deferred (deliberately)
 - **Cut:**
   - Link-preview cards *fetched from the linked page* by default. Cards built from the object, or from FEP-8967