The town: a fake community across the pasture, checked for coherence

tools/pasture/town/ (run through tools/pasture/town.sh) seeds a fake
community across every running peer and checks that all of them, and
PrivaPub, agree on what happened:

- drivers per platform on four dialect bases (Mastodon API, Misskey API,
  Lemmy API, PrivaPub with /clientapi), each with a selftest against
  its own server; what a server holds is read from its database, never
  by making it fetch;
- a deterministic generator (specs/village.json: 23 accounts on seven
  servers, roots with several personas, circles and communities, a
  cross-server follow graph, posts of every kind and visibility, reply
  rounds, likes, boosts, reactions, votes, edits, deletes, blocks,
  mutes and a report) and a seeder that keeps a ledger of what happened;
- a sweep that expects delivery and confinement per server, what each
  account sees, counts, threads, edits, deletes, follows and privacy
  rows (sibling keys, published days, canary root credentials in every
  peer's database, located posts that never leave), with what the peers
  do on purpose modelled (Misskey drops orphan replies, Lemmy keeps only
  community content, edits go to the post's own audience);
- known gaps (gaps.json) turn failures into xfail and passes into xpass;
  a self-contained report.html, and docs/INTEROP-BACKLOG.md.

The pasture moves to a public-looking subnet (peers with no private
address switch can join), takes PASTURE_PORT when 6971 is in use, adds
peers to a running pasture (run.sh add, Caddy recreated with its CA
kept), removes its volumes on down, writes every scenario check to
out/scenarios.jsonl, serves decePub as decepub.test for its e2e tests,
lifts GoToSocial's and Lemmy's own rate limits, trusts Caddy in
Mastodon (TRUSTED_PROXY_IP) and gains Hollo (Fedify), whose one login
owning several accounts is the nearest peer to PrivaPub's personas.

The first village found the four PrivaPub bugs fixed in the commits
before this one; the second run, on the fixed server, passes 2319 checks
with 11 failures left, all between peers or from Lemmy's send worker,
which the seeder now warms up first. ROADMAP records the owner's
decisions of 2026-10-04 (the town, and P9 back from the cut list).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw

.gitignore

@@ -405,3 +405,4 @@ tools/pasture/.publish/
 .claude/worktrees/
 tools/pasture/.ca/
 tools/pasture/.state/
+tools/pasture/out/

CLAUDE.md

@@ -469,7 +469,7 @@ tools/pasture/run.sh down                                            # removes e
   deliveries are checked through `/api/v1/accounts/:id/statuses` of the sender as Mastodon knows them, or
   `Status.exists?` through `rails runner`. Its actors are numbered (`/ap/users/<id>`), so look URIs up rather than
   build them. Circle posts and a followers-only post survive its signed refetch (`ActivityPub::FetchRemoteStatusService`
-  through `rails runner`). Inbound Block is the one expected failure until P7.
+  through `rails runner`). Its block of a persona is checked through our `blocked_by`.
 - **Misskey (2026.10.0):** one container on the shared Postgres and Redis. A new Misskey federates with nobody
   (`federation: none`) until `admin/update-meta` says `all`, which `misskey_up` does. Its API is `POST /api/<endpoint>`
   with the token as `i` (`mk` in the scenario); `users/relation` answers a list, `users/notes` leaves replies out unless
@@ -498,6 +498,43 @@ tools/pasture/run.sh down                                            # removes e
   run.sh up mastodon`, then `interop.sh crawler`. `PRIVAPUB_ENV` passes any setting to the PrivaPub container.
 - `run.sh up` replaces every container, Mongo included, so each run starts clean. To keep the data, republish into
   `tools/pasture/.publish` and `podman restart pasture-privapub`; that is how a migration is tried on dirty data.
+- `run.sh add <peer>` starts more peers next to a running pasture; `run.sh down` removes every pasture container with
+  its volumes (`--purge` drops Caddy's CA too); `run.sh stats` shows each container's memory.
+- **The network looks public** (`PASTURE_SUBNET`, default `11.42.0.0/24`), so peers that refuse private addresses with no
+  switch (Pixelfed, Mbin, WordPress, Discourse, Fedify) federate. Mastodon then needs `TRUSTED_PROXY_IP` for Caddy, or it
+  answers every request with an IP spoofing error.
+- **`PASTURE_PORT`** moves PrivaPub's workstation port off 6971 when something else holds it; `lib/interop.sh` follows.
+- Every `ok`/`ko`/`xf` is also a line of `out/scenarios.jsonl`, which the town's report shows.
+
+#### The town (`tools/pasture/town.sh`, `tools/pasture/town/`)
+
+A fake community seeded across every running peer, then checked for coherence: what each server holds (read from its
+database, never by making it fetch), what each account sees through its API, counts, threads, edits, deletes, follows
+and the privacy rows. Python 3 standard library only, tabs.
+
+```bash
+tools/pasture/town.sh selftest [peer...]   # each driver against its own server; run after changing a driver or a pin
+tools/pasture/town.sh seed village         # specs/village.json: 23 accounts on seven servers, about 11 minutes
+tools/pasture/town.sh check village        # out/<run>/results.json; --deadline=0 re-sweeps an old run at once
+tools/pasture/town.sh report village       # out/<run>/report.html, publishable (fake data, no tokens)
+tools/pasture/town.sh backlog --write      # docs/INTEROP-BACKLOG.md
+```
+
+- **Drivers** (`dialects/`): one class per platform on a dialect base (`mastodon_api`, `misskey_api`, `lemmy_api`,
+  `privapub`). `stored()` reads the database (shared Postgres through `psql`, GoToSocial's sqlite copied out, Mongo);
+  `seen()` asks the API as one account. A driver that cannot do something raises `Unsupported`, and the planner never
+  asks for it (`gen.CAPS`).
+- **The plan** (`gen.py`) is a pure function of the spec and its seed; each run keeps its own `plan.json`, `ledger.jsonl`
+  (what happened, one fact per line) and `results.json`. `seed --resume` redoes only failed steps.
+- **Cells** are `feature|origin|observer|direction` (`in`, `out`, `via`, `control`, `local`). A known gap is a
+  `gaps.json` entry: matching failures are `xfail`, matching passes `xpass` (close the gap).
+- **What the peers do on purpose**, so the checker expects it: Misskey and Sharkey drop a reply whose parent they cannot
+  fetch; Lemmy keeps only community content and private messages; servers send edits and deletes to the post's own
+  audience only; a Misskey heart reaction arrives as a Like.
+- Peers' own limits are lifted for the town: GoToSocial `GTS_ADVANCED_RATE_LIMIT_REQUESTS=0`, Lemmy's
+  `local_site_rate_limit`. Mastodon reserves usernames that contain `mastodon`.
+- decePubClient's e2e tests (`decePubClient/tests/e2e/run.sh`) publish the client with the Pasture environment, serve
+  it as `https://decepub.test` (`peers/decepub.sh`) and write their cells to `out/client.jsonl`.
 
 After the pasture, verify.funfedi.dev and the owner's GoToSocial at social.arasaka.software. **Ask before acting from
 the owner's GoToSocial account.**

docs/INTEROP-BACKLOG.md

@@ -0,0 +1,40 @@
+# Interop backlog
+
+Generated by `tools/pasture/town.sh backlog --write` from `village-20261004-234452` on 2026-10-04. Do not edit by hand: fix the cause, or record a known gap in `tools/pasture/town/gaps.json`, and generate it again.
+
+Checks: 2392 pass, 23 fail, 1 known gaps, 19 known gaps now passing.
+
+## New failures
+
+- **`deliver.community|lemmy|privapub|in`**: 3 failing. Example: p81 expected `held`, got `missing`.
+- **`deliver.community|lemmy|akkoma|via`**: 3 failing. Example: p84 expected `held`, got `missing`.
+- **`deliver.followers|akkoma|mastodon|control`**: 1 failing. Example: p36 expected `held`, got `missing`.
+- **`deliver.community|lemmy|mastodon|via`**: 1 failing. Example: p166 expected `held`, got `missing`.
+- **`count.boost.public|*|misskey|control`**: 1 failing. Example: p19 expected `>=1`, got `0`.
+- **`count.like.community|*|mastodon|control`**: 1 failing. Example: p161 expected `>=1`, got `0`.
+- **`count.boost.public|*|sharkey|control`**: 1 failing. Example: p175 expected `>=1`, got `0`.
+- **`client.render.text.public|akkoma|decepub|in`**: 1 failing. Example: p1 expected `None`, got `Timeout 30000ms exceeded. Call log: - waiting for Locator("article[data-uri='https://akkoma.test/objects/c7942cfe-4fee-4127-a0c3-20974b8c1f9`.
+- **`client.render.cw.public|sharkey|decepub|in`**: 1 failing. Example: p20 expected `None`, got `Locator expected to contain text 'map food' But was: 'Authoracct xavi_sk33@sharkey.testuri https://sharkey.test/users/arxwyny1ewcv0007create`.
+- **`client.render.mention.public|akkoma|decepub|in`**: 1 failing. Example: p54 expected `None`, got `Timeout 30000ms exceeded. Call log: - waiting for Locator("article[data-uri='https://akkoma.test/objects/d75be046-217f-4081-bf4d-46c979893d5`.
+- **`client.render.image.public|sharkey|decepub|in`**: 1 failing. Example: p77 expected `None`, got `Locator expected to be visible Error: element(s) not found Call log: - Expect "ToBeVisibleAsync" with timeout 5000ms - waiting for Locator("`.
+- **`client.render.image.public|akkoma|decepub|in`**: 1 failing. Example: p18 expected `None`, got `Locator expected to be visible Error: element(s) not found Call log: - Expect "ToBeVisibleAsync" with timeout 5000ms - waiting for Locator("`.
+- **`client.render.poll.public|akkoma|decepub|in`**: 1 failing. Example: p11 expected `None`, got `Timeout 30000ms exceeded. Call log: - waiting for Locator("article[data-uri='https://akkoma.test/objects/85cd4293-d8fb-4f68-92fa-909a2d2a7de`.
+- **`client.render.text.unlisted|akkoma|decepub|in`**: 1 failing. Example: p32 expected `None`, got `Timeout 30000ms exceeded. Call log: - waiting for Locator("article[data-uri='https://akkoma.test/objects/38ba2cb1-37f8-4c58-9a17-ef3d71418f6`.
+- **`client.action.boost|misskey|decepub|in`**: 1 failing. Example: p19 expected `None`, got `undoing the boost never reached misskey`.
+- **`client.action.boost|akkoma|decepub|in`**: 1 failing. Example: p1 expected `None`, got `Timeout 30000ms exceeded. Call log: - waiting for Locator("article[data-uri='https://akkoma.test/objects/c7942cfe-4fee-4127-a0c3-20974b8c1f9`.
+- **`client.action.reply|gts|decepub|in`**: 1 failing. Example: p28 expected `None`, got `the reply never landed under the post on gts`.
+- **`client.action.reply|akkoma|decepub|in`**: 1 failing. Example: p1 expected `None`, got `Timeout 30000ms exceeded. Call log: - waiting for Locator("article[data-uri='https://akkoma.test/objects/c7942cfe-4fee-4127-a0c3-20974b8c1f9`.
+- **`client.action.favourite|akkoma|decepub|in`**: 1 failing. Example: p1 expected `None`, got `Timeout 30000ms exceeded. Call log: - waiting for Locator("article[data-uri='https://akkoma.test/objects/c7942cfe-4fee-4127-a0c3-20974b8c1f9`.
+
+## Known gaps still failing
+
+- **G-0003** (server, P7): Votes and moderation that a Lemmy community relays inside Announce (Like, Dislike, Undo, Delete, Block) are dropped. 1 checks. Code: `PrivaPub/Federation/Inbox/Handlers/AnnounceHandler.cs:203`.
+
+## Known gaps now passing (close them)
+
+- **G-0001**: Likes, reactions, votes and downvotes from followers on followers-only posts, and from members on circle posts, are dropped. 16 checks pass.
+- **G-0003**: Votes and moderation that a Lemmy community relays inside Announce (Like, Dislike, Undo, Delete, Block) are dropped. 3 checks pass.
+
+## Scenario failures
+
+- akkoma: undo of like or boost not applied on Akkoma

docs/ROADMAP.md

@@ -50,8 +50,19 @@ Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federati
     - Mastodon routes that answer, grouped notifications;
     - community announce ids that resolve, a hashtag page;
     - remote accounts' real counts.
+- [ ] W The town (planned 2026-10-04, owner decisions below): every self-hostable fediverse platform in the pasture, a
+  fake community seeded across all of them, its coherence checked server by server and through decePubClient, and every
+  gap it finds fixed. Runs alongside P7, P8 and P9. Done so far:
+  - the town toolkit (`tools/pasture/town/`): drivers for PrivaPub and the six peers, the seeded `village` (23 accounts,
+    205 objects, 970 steps), the coherence sweep with known gaps (`gaps.json`), the report and `docs/INTEROP-BACKLOG.md`;
+  - decePubClient's end-to-end tests against the pasture (`decePubClient/tests/e2e`, Playwright for .NET);
+  - three bugs the first village found, fixed: likes, reactions, votes and downvotes from followers on followers-only
+    posts and from circle members were dropped (G-0001); a Follow refused with Mastodon's first-contact 422 was lost for
+    good; a persona addressed without being named could not see the post (silent mentions).
 - [ ] P7 Threads, communities, moderation, the social graph
 - [ ] P8 Signatures, discovery, the long tail
+- [ ] P9 Reading at volume (owner decision 2026-10-04, back from "Cut"): lists, server-side filters, scheduled posts,
+  followed hashtags, trends and a directory
 
 ## Intent
 
@@ -206,6 +217,16 @@ and circles (see Owner decisions).
 | CDN-fronted servers | **Kept apart, and followed through time.** Their place stays hidden (the address is the CDN's edge). They are grouped per CDN, named by the CDN's own domain, with week by week how many servers were behind it and who joined or left. Each server's weekly history (place, CDN, size) is public, including where it was before it went behind a CDN. |
 | Server locations on the instance API | **Public, as decided for public server locations** (2026-10-03, corrected): city, coordinates to 0.1° and network for every located server whatever its size, the CDN's name for a CDN-fronted one, with DB-IP's attribution. decePubClient's globe draws posts at their author's server. This server's own place comes from its host's address, or from `Statistics:Geo:Self` when the owner sets it (a server behind a proxy). |
 
+### Owner decisions on the town (2026-10-04)
+
+| Question | Decision |
+|---|---|
+| Scope | **Every self-hostable fediverse platform** in the pasture, accounts of every kind, a fake community across them, and coherence of state between all of them and PrivaPub, checked from both the servers and decePubClient. What the town finds is fixed until PrivaPub is the best federating server and decePubClient the best client for the fediverse at volume. |
+| The "Cut" list | **Lists, server-side filters, scheduled posts, followed hashtags, trends and a directory come back** (P9). The directory lists only discoverable personas; trends count only public posts PrivaPub already holds. Translation, local custom emoji, the Mastodon admin API, API registration, S3 and JSON-LD processing stay cut. |
+| decePubClient tests | **Playwright for .NET**, against the pasture, never in CI. |
+| How work lands | Commits on master in both repositories; nothing is pushed, tagged or deployed without asking. |
+| Still asked first | Anything new that tells another server something about a persona, Web Push (traffic to push services), any admin endpoint in production, and anything acting from production or the owner's accounts. |
+
 ## Libraries (researched; no maintained .NET ActivityPub library exists, so Letterbook and Iceshrimp.NET both wrote their own)
 
 | Area | Choice |
@@ -622,8 +643,8 @@ buckets and only for servers reporting at least 10 users; the owner never intend
 - **Cut:**
   - Link-preview cards *fetched from the linked page* by default. Cards built from the object, or from FEP-8967
     `preview` data, are in P6; fetching the page is an owner decision (INTEROP §6.1).
-  - Translation, trends, directory and lists (filters stay as stubs).
-  - Scheduled posts.
+  - Translation. (Trends, the directory, lists, filters and scheduled posts were cut too, until the owner brought them
+    back on 2026-10-04: P9.)
   - The Mastodon admin API: moderation stays on `/clientapi`.
   - `POST /api/v1/accounts` registration: avatars are created through `/clientapi`.
   - Local custom emoji.

tools/pasture/Caddyfile

@@ -37,3 +37,13 @@ akkoma.test {
 	tls internal
 	reverse_proxy pasture-akkoma:4000
 }
+
+decepub.test {
+	tls internal
+	reverse_proxy pasture-decepub:80
+}
+
+hollo.test {
+	tls internal
+	reverse_proxy pasture-hollo:3000
+}

tools/pasture/interop.sh

@@ -5,9 +5,11 @@
 set -uo pipefail
 here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 . "$here/lib/interop.sh"
+: > "$INTEROP_JSONL"
 
 for peer in "${@:-gts}"; do
 	[ -f "$here/scenarios/$peer.sh" ] || { echo "no scenario for $peer" >&2; exit 2; }
+	INTEROP_PEER=$peer
 	. "$here/scenarios/$peer.sh"
 done
 

tools/pasture/lib/interop.sh

@@ -1,10 +1,15 @@
 # Shared by interop.sh and the scenarios: check helpers, PrivaPub personas and tokens, the statistics check.
-P=http://127.0.0.1:6971
+P=http://127.0.0.1:${PASTURE_PORT:-6971}
 work=$(mktemp -d); trap 'rm -rf "$work"' EXIT
 pass=0; fail=0; expected=0
-ok() { echo "  ok   $*"; pass=$((pass+1)); }
-ko() { echo "  FAIL $*"; fail=$((fail+1)); }
-xf() { echo "  xf   $* (expected to fail until a later phase)"; expected=$((expected+1)); }
+# every check is also appended to $INTEROP_JSONL (out/scenarios.jsonl), which the town's report reads next to its own
+INTEROP_JSONL=${INTEROP_JSONL:-$here/out/scenarios.jsonl}
+mkdir -p "$(dirname "$INTEROP_JSONL")"
+record() { python3 -c 'import json,sys,time; print(json.dumps({"source":"scenario","peer":sys.argv[1],"check":sys.argv[2],"status":sys.argv[3],"t":time.strftime("%Y-%m-%dT%H:%M:%SZ",time.gmtime())}))' \
+	"${INTEROP_PEER:-}" "$1" "$2" >> "$INTEROP_JSONL"; }
+ok() { echo "  ok   $*"; pass=$((pass+1)); record "$*" pass; }
+ko() { echo "  FAIL $*"; fail=$((fail+1)); record "$*" fail; }
+xf() { echo "  xf   $* (expected to fail until a later phase)"; expected=$((expected+1)); record "$*" xfail; }
 j() { python3 -c "import sys,json
 try: d=json.load(sys.stdin)
 except Exception: d=None

tools/pasture/lib/pasture.sh

@@ -1,6 +1,10 @@
 # Shared by run.sh: the network, Caddy (its CA kept in a volume and copied to .ca/root.crt), Mongo and PrivaPub.
 here="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"; repo="$(cd "$here/../.." && pwd)"
+port=${PASTURE_PORT:-6971}  # PrivaPub straight from the workstation; PASTURE_PORT when 6971 is taken
 net=privapub-pasture; publish="$here/.publish"; ca="$here/.ca"
+# The network looks public (11.42.0.0/24 is never reached for real from inside the pasture), so peers that refuse private
+# addresses with no switch to say otherwise (Pixelfed, Mbin, WordPress, Discourse, Fedify) federate too.
+subnet=${PASTURE_SUBNET:-11.42.0.0/24}
 
 site() { curl -k --resolve "$1:6443:127.0.0.1" "${@:2}"; }
 
@@ -13,21 +17,16 @@ pasture_base_up() {
 	local dotnet=${DOTNET:-$(command -v dotnet || echo ~/.dotnet/dotnet)}
 	"$dotnet" publish "$repo/PrivaPub/PrivaPub.csproj" -c Release -r linux-x64 --self-contained true -o "$publish" -v quiet
 	cp "$here/appsettings.Pasture.json" "$publish/"
-	podman network exists $net || podman network create $net >/dev/null
+	podman network exists $net || podman network create --subnet "$subnet" $net >/dev/null
 	podman volume exists pasture-caddy-data || podman volume create pasture-caddy-data >/dev/null
 	podman run -d --replace --name pasture-mongo --network $net --network-alias mongo docker.io/library/mongo:8 --quiet >/dev/null
-	local aliases=""
-	for site in privapub gts mastodon akkoma misskey sharkey lemmy; do aliases="$aliases --network-alias $site.test"; done
-	# shellcheck disable=SC2086
-	podman run -d --replace --name pasture-caddy --network $net $aliases \
-		-p 127.0.0.1:6443:443 --sysctl net.ipv4.ip_unprivileged_port_start=0 -v "$here/Caddyfile:/etc/caddy/Caddyfile:Z,ro" \
-		-v pasture-caddy-data:/data docker.io/library/caddy:2 >/dev/null
+	caddy_up
 	local extra=()
 	for setting in ${PRIVAPUB_ENV:-}; do extra+=(-e "$setting"); done
-	podman run -d --replace --name pasture-privapub --network $net -p 127.0.0.1:6971:80 \
+	podman run -d --replace --name pasture-privapub --network $net -p "127.0.0.1:$port:80" \
 		--sysctl net.ipv4.ip_unprivileged_port_start=0 -e ASPNETCORE_ENVIRONMENT=Pasture "${extra[@]}" -w /app -v "$publish:/app:Z,ro" \
 		mcr.microsoft.com/dotnet/runtime-deps:10.0 /app/PrivaPub >/dev/null
-	wait_http http://127.0.0.1:6971/build.json
+	wait_http "http://127.0.0.1:$port/build.json"
 	rm -rf "$ca"; mkdir -p "$ca"
 	for _ in $(seq 1 60); do
 		podman cp pasture-caddy:/data/caddy/pki/authorities/local/root.crt "$ca/root.crt" 2>/dev/null && [ -s "$ca/root.crt" ] && break
@@ -41,9 +40,43 @@ pasture_base_up() {
 	chmod 644 "$ca/bundle.pem"
 }
 
+# Caddy, with every site the Caddyfile serves as a name on the network, so a peer is added with its Caddyfile block alone.
+# Its CA lives in the pasture-caddy-data volume, so recreating it (run.sh add, for a new site) keeps every peer's trust.
+caddy_up() {
+	local aliases=""
+	for site in $(caddy_sites); do aliases="$aliases --network-alias $site"; done
+	# shellcheck disable=SC2086
+	podman run -d --replace --name pasture-caddy --network $net $aliases \
+		-p 127.0.0.1:6443:443 --sysctl net.ipv4.ip_unprivileged_port_start=0 -v "$here/Caddyfile:/etc/caddy/Caddyfile:Z,ro" \
+		-v pasture-caddy-data:/data docker.io/library/caddy:2 >/dev/null
+}
+
+# caddy_current: whether the running Caddy already answers every site of the Caddyfile on the network
+caddy_current() {
+	local have
+	have=$(podman inspect pasture-caddy --format '{{range .NetworkSettings.Networks}}{{range .Aliases}}{{.}} {{end}}{{end}}' 2>/dev/null)
+	for site in $(caddy_sites); do case " $have " in *" $site "*) ;; *) return 1 ;; esac; done
+}
+
+# the sites the Caddyfile serves: "a.test {" and "a.test, b.test {" lines
+caddy_sites() {
+	grep -E '^[a-z0-9.-]+\.test(, *[a-z0-9.-]+\.test)* *\{' "$here/Caddyfile" | sed 's/ *{.*//; s/,/ /g'
+}
+
+# pasture_down [--purge]: removes every pasture container with its anonymous volumes (images declare volumes, and
+# without -v each run left them behind), the named volumes and the network. Caddy's CA is kept unless --purge, so a peer
+# that was told to trust it still does after the next up.
 pasture_down() {
-	podman ps -a --format '{{.Names}}' | grep '^pasture-' | xargs -r podman rm -f >/dev/null 2>&1 || true
-	podman volume ls --format '{{.Name}}' | grep '^pasture-' | xargs -r podman volume rm -f >/dev/null 2>&1 || true
+	podman ps -a --format '{{.Names}}' | grep '^pasture-' | xargs -r podman rm -f -v >/dev/null 2>&1 || true
+	local keep='^pasture-caddy-data$'
+	[ "${1:-}" = "--purge" ] && keep='^$'
+	podman volume ls --format '{{.Name}}' | grep '^pasture-' | grep -v "$keep" | xargs -r podman volume rm -f >/dev/null 2>&1 || true
 	podman network rm $net >/dev/null 2>&1 || true
 	rm -rf "$here/.state" "$ca"
 }
+
+# pasture_stats: memory and CPU of every pasture container, largest first
+pasture_stats() {
+	podman stats --no-stream --format '{{.Name}}\t{{.MemUsage}}\t{{.CPUPerc}}' $(podman ps --format '{{.Names}}' | grep '^pasture-') 2>/dev/null \
+		| sort -t$'\t' -k2 -h -r
+}

tools/pasture/peers/decepub.sh

@@ -0,0 +1,28 @@
+# decePubClient as tests/e2e/run.sh publishes it (environment Pasture: its API is https://privapub.test), served like the
+# production vhost (deploy/nginx in decePubClient): static files, the SPA fallback, nothing cached that must not be.
+DECEPUB_SITE=${DECEPUB_SITE:-$repo/../decePubClient/tests/e2e/.publish/wwwroot}
+
+decepub_up() {
+	[ -f "$DECEPUB_SITE/index.html" ] || { echo "no published decePub in $DECEPUB_SITE (decePubClient/tests/e2e/run.sh publishes it)" >&2; return 1; }
+	mkdir -p "$here/.state"
+	cat > "$here/.state/decepub.Caddyfile" <<'CADDY'
+:80 {
+	root * /srv
+	header {
+		X-Content-Type-Options nosniff
+		Referrer-Policy strict-origin-when-cross-origin
+		X-Frame-Options SAMEORIGIN
+	}
+	@fresh path /index.html /appsettings.json /appsettings.Pasture.json /service-worker.js /service-worker-assets.js /build.json
+	header @fresh Cache-Control no-cache
+	try_files {path} /index.html
+	file_server {
+		precompressed br gzip
+	}
+}
+CADDY
+	podman run -d --replace --name pasture-decepub --network $net -v "$DECEPUB_SITE:/srv:z,ro" \
+		-v "$here/.state/decepub.Caddyfile:/etc/caddy/Caddyfile:z,ro" docker.io/library/caddy:2 >/dev/null
+	podman exec pasture-caddy caddy reload --config /etc/caddy/Caddyfile >/dev/null 2>&1 || true
+	echo "decepub: https://decepub.test:6443"
+}

tools/pasture/peers/gts.sh

@@ -7,7 +7,7 @@ gts_up() {
 		-e GTS_HOST=gts.test -e GTS_PROTOCOL=https -e GTS_PORT=80 -e GTS_BIND_ADDRESS=0.0.0.0 -e GTS_HTTP_CLIENT_TLS_INSECURE_SKIP_VERIFY=true \
 		-e GTS_DB_TYPE=sqlite -e GTS_DB_ADDRESS=/gotosocial/storage/sqlite.db -e GTS_STORAGE_LOCAL_BASE_PATH=/gotosocial/storage \
 		-e GTS_LETSENCRYPT_ENABLED=false -e GTS_HTTP_CLIENT_ALLOW_IPS=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 \
-		-e GTS_TRUSTED_PROXIES=0.0.0.0/0 -e GTS_LOG_LEVEL=info -e GTS_INSTANCE_EXPOSE_PUBLIC_TIMELINE=true \
+		-e GTS_TRUSTED_PROXIES=0.0.0.0/0 -e GTS_LOG_LEVEL=info -e GTS_ADVANCED_RATE_LIMIT_REQUESTS=0 -e GTS_INSTANCE_EXPOSE_PUBLIC_TIMELINE=true \
 		$GTS_IMAGE >/dev/null
 	wait_http http://127.0.0.1:6972/api/v1/instance
 	podman exec pasture-gts /gotosocial/gotosocial admin account create --username gtsuser --email gtsuser@gts.test --password 'Gts-Pasture-Pass-1!' >/dev/null 2>&1 || true

tools/pasture/peers/hollo.sh

@@ -0,0 +1,26 @@
+# Hollo 0.9 (Fedify): one login owning several accounts, the nearest thing to PrivaPub's personas. RFC 9421 signatures
+# first (cavage after a refusal), FEP-8b32 proofs, FEP-521a keys. On the shared Postgres (database hollo). The town's
+# driver makes the login and the accounts through its forms (dialects/hollo.py); ALLOW_PRIVATE_ADDRESS only matters if
+# the pasture's subnet is made private again.
+HOLLO_IMAGE=${HOLLO_IMAGE:-ghcr.io/fedify-dev/hollo:0.9.19}
+. "$here/peers/shared.sh"
+
+hollo_up() {
+	shared_postgres_up
+	pg_db hollo
+	mkdir -p "$here/.state/hollo/media" && chmod a+rwx "$here/.state/hollo/media"
+	podman run -d --replace --name pasture-hollo --network $net -v "$here/.state/hollo/media:/var/lib/hollo:z" \
+		-e DATABASE_URL=postgres://pasture:pasture@postgres:5432/hollo -e SECRET_KEY=pasture-hollo-not-a-secret-0123456789abcdefghijklmn \
+		-e BEHIND_PROXY=true -e ALLOW_PRIVATE_ADDRESS=true -e NODE_EXTRA_CA_CERTS=/pasture/ca/root.crt -e LOG_LEVEL=info \
+		-e DRIVE_DISK=fs -e FS_STORAGE_PATH=/var/lib/hollo -e STORAGE_URL_BASE=https://hollo.test/assets/ \
+		-v "$ca:/pasture/ca:z,ro" $HOLLO_IMAGE >/dev/null
+	for _ in $(seq 1 90); do
+		site hollo.test -s -o /dev/null -w '%{http_code}' https://hollo.test:6443/.well-known/nodeinfo 2>/dev/null | grep -q 200 && break
+		sleep 2
+	done
+	# the one login: its accounts are made by the town as it needs them
+	# its forms compare Origin with the Host, so the Host leaves out the workstation's port
+	site hollo.test -s -o /dev/null -X POST https://hollo.test:6443/setup -H 'Origin: https://hollo.test' -H 'Host: hollo.test' \
+		--data-urlencode email=owner@hollo.test --data-urlencode 'password=Hollo-Pasture-Pass-1' --data-urlencode 'password_confirm=Hollo-Pasture-Pass-1'
+	echo "hollo: https://hollo.test:6443"
+}

tools/pasture/peers/lemmy.sh

@@ -29,8 +29,16 @@ HJSON
 		site lemmy.test -s -o /dev/null -w '%{http_code}' https://lemmy.test:6443/api/v4/site 2>/dev/null | grep -q 200 && break
 		sleep 2
 	done
+	# Lemmy's default limits (6 posts or comments per 10 minutes, 10 sign-ups an hour) would throttle a scripted town;
+	# Lemmy reads them again when its site is edited
+	podman exec pasture-postgres psql -U pasture -d lemmy -qc "update local_site_rate_limit set message_max_requests=100000,
+		message_interval_seconds=1, post_max_requests=100000, post_interval_seconds=1, register_max_requests=100000,
+		register_interval_seconds=1, image_max_requests=100000, image_interval_seconds=1, comment_max_requests=100000,
+		comment_interval_seconds=1, search_max_requests=100000, search_interval_seconds=1" >/dev/null
 	site lemmy.test -s -X POST https://lemmy.test:6443/api/v4/account/auth/login -H 'Content-Type: application/json' \
 		-d '{"username_or_email":"lemmyuser","password":"Lemmy-Pasture-Pass-1"}' \
 		| python3 -c "import sys,json; print(json.load(sys.stdin)['jwt'])" > "$here/.state/lemmy.token" 2>/dev/null || true
+	site lemmy.test -s -o /dev/null -X PUT https://lemmy.test:6443/api/v4/site -H "Authorization: Bearer $(cat "$here/.state/lemmy.token")" \
+		-H 'Content-Type: application/json' -d '{"registration_mode":"open","email_verification_required":false,"captcha_enabled":false}'
 	echo "lemmy: https://lemmy.test:6443"
 }

tools/pasture/peers/mastodon.sh

@@ -18,6 +18,7 @@ ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=pasturedeterministickey0000000000
 ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=pasturederivationsalt000000000000
 ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=pastureprimarykey0000000000000000
 ALLOWED_PRIVATE_ADDRESSES=10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
+TRUSTED_PROXY_IP=$subnet
 SSL_CERT_FILE=/pasture/ca/bundle.pem
 SKIP_POST_DEPLOYMENT_MIGRATIONS=false
 WEB_CONCURRENCY=0

tools/pasture/peers/shared.sh

@@ -11,3 +11,12 @@ shared_redis_up() {
 	podman container exists pasture-redis && return 0
 	podman run -d --replace --name pasture-redis --network $net --network-alias redis docker.io/library/redis:7-alpine >/dev/null
 }
+
+# pg_db <name> [extension...]: a database of the shared Postgres for one peer, with the extensions it needs
+pg_db() {
+	local name=$1; shift
+	podman exec pasture-postgres sh -c "psql -U pasture -tc \"select 1 from pg_database where datname='$name'\" | grep -q 1 || createdb -U pasture $name"
+	for ext in "$@"; do
+		podman exec pasture-postgres psql -U pasture -d "$name" -qc "create extension if not exists \"$ext\";" >/dev/null
+	done
+}

tools/pasture/run.sh

@@ -1,8 +1,10 @@
 #!/usr/bin/env bash
 # A private test fediverse on one podman network: PrivaPub (privapub.test) and the peers asked for, behind one Caddy
 # whose internal CA every side is told to accept (its root is copied to .ca/root.crt). From the workstation: PrivaPub's
-# API at http://127.0.0.1:6971, every site at https://<name>.test:6443 (curl --resolve <name>.test:6443:127.0.0.1 -k).
-# usage: tools/pasture/run.sh up [peer...] | down | logs <name> | ps        peers: gts (default), mastodon, misskey, sharkey, akkoma, lemmy
+# API at http://127.0.0.1:6971 ($PASTURE_PORT), every site at https://<name>.test:6443 (curl --resolve <name>.test:6443:127.0.0.1 -k).
+# usage: tools/pasture/run.sh up [peer...] | add <peer...> | down [--purge] | logs <name> | ps | stats
+#   up starts everything afresh (PrivaPub republished, Mongo empty); add starts more peers next to a running pasture.
+#   peers: gts (default), mastodon, misskey, sharkey, akkoma, lemmy, decepub
 set -euo pipefail
 . "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/pasture.sh"
 
@@ -16,10 +18,22 @@ up)
 		. "$here/peers/$peer.sh"
 		"${peer}_up"
 	done
-	echo "privapub: http://127.0.0.1:6971, https://privapub.test:6443"
+	echo "privapub: http://127.0.0.1:$port, https://privapub.test:6443"
 	;;
-down) pasture_down ;;
+add)
+	shift
+	podman container exists pasture-privapub || { echo "the pasture is not up: run.sh up first" >&2; exit 1; }
+	caddy_current || caddy_up
+	podman exec pasture-caddy caddy reload --config /etc/caddy/Caddyfile >/dev/null 2>&1 || true
+	for peer in "$@"; do
+		[ -f "$here/peers/$peer.sh" ] || { echo "no such peer: $peer" >&2; exit 1; }
+		. "$here/peers/$peer.sh"
+		"${peer}_up"
+	done
+	;;
+down) pasture_down "${2:-}" ;;
 logs) podman logs --tail 200 "pasture-${2:-privapub}" ;;
 ps) podman ps --filter name=pasture- --format '{{.Names}}\t{{.Status}}' ;;
-*) echo "usage: $0 up [peer...] | down | logs <name> | ps" >&2; exit 2 ;;
+stats) pasture_stats ;;
+*) echo "usage: $0 up [peer...] | add <peer...> | down [--purge] | logs <name> | ps | stats" >&2; exit 2 ;;
 esac

tools/pasture/town.sh

@@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+# The town: a fake community seeded across every pasture peer, then checked for coherence (see town/town.py).
+# usage: tools/pasture/town.sh <command> [args]      commands: selftest, seed, check, report, backlog, gaps, drive, stored, seen
+exec python3 "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/town/town.py" "$@"

tools/pasture/town/backlog.py

@@ -0,0 +1,86 @@
+"""docs/INTEROP-BACKLOG.md from the latest town results, the bash scenarios and decePub's e2e cells: new failures (to fix
+or to record in gaps.json), known gaps by phase, known gaps now passing (close them), and the scenarios' failures.
+
+  town.sh backlog [run] [--write]"""
+import json
+import os
+import sys
+import time
+from collections import defaultdict
+
+import check
+import seed
+
+DOCS = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))), "docs")
+
+
+def load_jsonl(path):
+	if not os.path.exists(path):
+		return []
+	with open(path) as f:
+		return [json.loads(line) for line in f if line.strip()]
+
+
+def main(args):
+	name = next((a for a in args if not a.startswith("--")), "village")
+	run_dir = seed.run_dir(name)
+	with open(os.path.join(run_dir, "results.json")) as f:
+		result = json.load(f)
+	gaps = {g["id"]: g for g in check.load_gaps()}
+	client = check.classify([dict(c) for c in load_jsonl(os.path.join(seed.OUT, "client.jsonl"))], list(gaps.values()))
+	scenarios = load_jsonl(os.path.join(seed.OUT, "scenarios.jsonl"))
+	checks = result["checks"] + client
+
+	new = defaultdict(list)
+	known = defaultdict(list)
+	closing = defaultdict(list)
+	for c in checks:
+		if c["status"] == "fail":
+			new[c["cell"]].append(c)
+		elif c["status"] == "xfail":
+			known[c["gap"]].append(c)
+		elif c["status"] == "xpass":
+			closing[c["gap"]].append(c)
+
+	lines = ["# Interop backlog", "",
+			 f"Generated by `tools/pasture/town.sh backlog --write` from `{os.path.basename(run_dir)}` on "
+			 f"{time.strftime('%Y-%m-%d', time.gmtime())}. Do not edit by hand: fix the cause, or record a known gap in "
+			 "`tools/pasture/town/gaps.json`, and generate it again.", "",
+			 f"Checks: {sum(1 for c in checks if c['status'] == 'pass')} pass, {sum(1 for c in checks if c['status'] == 'fail')} "
+			 f"fail, {sum(1 for c in checks if c['status'] == 'xfail')} known gaps, "
+			 f"{sum(1 for c in checks if c['status'] == 'xpass')} known gaps now passing.", ""]
+	lines += ["## New failures", ""]
+	if not new:
+		lines.append("None.")
+	for cell, cs in sorted(new.items(), key=lambda kv: -len(kv[1])):
+		ex = cs[0]
+		got = " ".join(str(ex.get("got")).replace("`", "'").split())[:140]
+		lines.append(f"- **`{cell}`**: {len(cs)} failing. Example: {ex.get('ref') or ''} expected `{ex.get('expect')}`, "
+					 f"got `{got}`" + (f" ({ex['how']})" if ex.get("how") else "") + ".")
+	lines += ["", "## Known gaps still failing", ""]
+	if not known:
+		lines.append("None.")
+	for gid, cs in sorted(known.items()):
+		g = gaps.get(gid, {})
+		lines.append(f"- **{gid}** ({g.get('kind')}, {g.get('phase')}): {g.get('title')}. {len(cs)} checks. "
+					 + (f"Code: `{g['code']}`." if g.get("code") else ""))
+	lines += ["", "## Known gaps now passing (close them)", ""]
+	if not closing:
+		lines.append("None.")
+	for gid, cs in sorted(closing.items()):
+		lines.append(f"- **{gid}**: {gaps.get(gid, {}).get('title')}. {len(cs)} checks pass.")
+	lines += ["", "## Scenario failures", ""]
+	bad = [s for s in scenarios if s["status"] == "fail"]
+	lines += [f"- {s['peer']}: {s['check']}" for s in bad] or ["None."]
+	text = "\n".join(lines) + "\n"
+	if "--write" in args:
+		with open(os.path.join(DOCS, "INTEROP-BACKLOG.md"), "w") as f:
+			f.write(text)
+		print(os.path.join(DOCS, "INTEROP-BACKLOG.md"))
+	else:
+		print(text)
+	return 0
+
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))

tools/pasture/town/check.py

@@ -0,0 +1,470 @@
+"""The coherence sweep: what every server should hold, show and count, from the plan and the ledger, against what each
+one does hold (its database) and show (its API, as one account). Writes out/<run>/results.json.
+
+Each check is a cell `feature|origin|observer|direction`:
+  in       a peer's object as PrivaPub holds it        out      PrivaPub's object as a peer holds it
+  via      a peer's object on another peer, through PrivaPub (a community it hosts)
+  control  a peer's object on another peer, PrivaPub not involved (tells a PrivaPub bug from a peer quirk)
+  local    an object on its own server
+Known gaps (gaps.json) turn a failure into xfail, and a pass of a known gap into XPASS."""
+import json
+import os
+import re
+import sys
+import time
+from collections import defaultdict
+
+import gen
+import state
+from core import podman
+from dialects import driver
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+HEARTS = {"❤", "❤️", "♥", "♥️"}
+# what each platform keeps when it is sent something: Lemmy keeps only what lives in communities, and private messages
+RECEIVES = {
+	"lemmy": {"community", "direct"},
+}
+# Misskey and its forks drop a reply whose parent they cannot fetch ("Error in inReplyTo ... 404"): a reply to a
+# followers-only post or a DM they do not hold never lands there, whoever sent it (docs/INTEROP.md, Misskey)
+DROPS_ORPHAN_REPLIES = {"misskey", "sharkey"}
+
+
+class World:
+	"""The plan and the ledger together: accounts, follows that held, groups and their members, objects with URIs."""
+
+	def __init__(self, run_dir):
+		with open(os.path.join(run_dir, "spec.json")) as f:
+			self.spec = json.load(f)
+		snap = os.path.join(run_dir, "plan.json")
+		if os.path.exists(snap):
+			with open(snap) as f:
+				self.planner = gen.Stored(json.load(f))
+		else:
+			self.planner = gen.Planner(self.spec)
+			self.planner.plan()
+		self.facts = []
+		with open(os.path.join(run_dir, "ledger.jsonl")) as f:
+			self.facts = [json.loads(line) for line in f if line.strip()]
+		self.ok_steps = {f["n"] for f in self.facts if f.get("type") not in ("error", "unsupported") and "n" in f}
+		self.failed = {f["n"]: f for f in self.facts if f.get("type") in ("error", "unsupported")}
+		self.objects = {}
+		for f in self.facts:
+			if f.get("type") == "object":
+				o = dict(self.planner.objects[f["ref"]])
+				o.update(uri=f["uri"], origin=f["origin"], fetched_on=f.get("fetched_on", {}))
+				self.objects[f["ref"]] = o
+		self.deleted = {f["ref"] for f in self.facts if f.get("type") == "mutation" and f["verb"] == "delete"}
+		self.edited = {f["ref"] for f in self.facts if f.get("type") == "mutation" and f["verb"] == "edit"}
+		self.follows = self._follows()
+		self.members = self._members()
+		self.interactions = [f for f in self.facts if f.get("type") == "interaction"]
+
+	def _follows(self):
+		"""A follow holds once the follow went out and the target either needs no approval or gave it. A follow of an
+		unlocked remote account answers "requested" until its Accept arrives, so the plan's lock flag decides."""
+		accepted = set()
+		for f in self.facts:
+			if f.get("type") != "relation":
+				continue
+			if f["verb"] == "follow" and (f["state"] == "accepted" or not self.planner.accounts[f["target"]]["locked"]):
+				accepted.add((f["actor"], f["target"]))
+			elif f["verb"] == "accept":
+				accepted.add((f["target"], f["actor"]))
+		edges = defaultdict(set)
+		for follower, target in accepted:
+			edges[target].add(follower)
+		return edges
+
+	def _members(self):
+		members = defaultdict(set)
+		joined = {}
+		for f in self.facts:
+			if f.get("type") != "relation":
+				continue
+			if f["verb"] == "join":
+				grp = self.planner.groups[f["group"]]
+				if grp["kind"] == "circle" and not f["actor"].startswith("privapub/"):
+					joined[(f["group"], f["actor"])] = True  # a remote member counts once the owner approves
+				else:
+					members[f["group"]].add(f["actor"])
+			elif f["verb"] == "approve":
+				members[f["group"]].add(f["member"])
+		for ref, grp in self.planner.groups.items():
+			members[ref].add(grp["owner"])
+		return members
+
+	def platform(self, key):
+		return key.split("/")[0]
+
+	def recipients(self, o):
+		"""The accounts an object was addressed to, as the ledger says the graph turned out."""
+		v = o["visibility"]
+		who = set(o.get("mentions") or ())
+		if v in ("public", "unlisted", "followers"):
+			who |= self.follows.get(o["author"], set())
+			if o.get("parent") and o["parent"] in self.objects:
+				who.add(self.objects[o["parent"]]["author"])
+			if o.get("quote") and o["quote"] in self.objects:
+				who.add(self.objects[o["quote"]]["author"])
+		elif v == "circle":
+			who |= self.members[o["group"]]
+		elif v == "community" or (o.get("group") and self.planner.groups.get(o["group"], {}).get("kind") == "community"):
+			grp = self.planner.groups[o["group"]]
+			who |= self.members[o["group"]] | {grp["owner"]}
+			if o.get("parent") and o["parent"] in self.objects:
+				who.add(self.objects[o["parent"]]["author"])
+		return who - {o["author"]}
+
+	def kind_for(self, o):
+		if o["visibility"] == "community" or (o.get("group") and self.planner.groups.get(o["group"], {}).get("kind") == "community"):
+			return "community"
+		return o["visibility"]
+
+
+def direction(origin, observer, through_privapub=False):
+	if origin == observer:
+		return "local"
+	if observer == "privapub":
+		return "in"
+	if origin == "privapub":
+		return "out"
+	return "via" if through_privapub else "control"
+
+
+class Sweep:
+	def __init__(self, world, platforms):
+		self.w = world
+		self.platforms = platforms
+		self.checks = []
+
+	def add(self, feature, origin, observer, ok, ref=None, expect=None, got=None, through=False, how=None):
+		self.checks.append({"cell": f"{feature}|{origin}|{observer}|{direction(origin, observer, through)}",
+							"feature": feature, "origin": origin, "observer": observer,
+							"direction": direction(origin, observer, through), "ok": bool(ok), "ref": ref,
+							"expect": expect, "got": got, "how": how})
+
+	def stored_all(self):
+		uris = [o["uri"] for o in self.w.objects.values()]
+		out = {}
+		for p in self.platforms:
+			try:
+				out[p] = driver(p).stored(uris)
+			except Exception as e:
+				print(f"  {p}: cannot read its database: {e!r}", file=sys.stderr)
+				out[p] = {}
+		return out
+
+	def run(self, deadline):
+		started = time.time()
+		while True:
+			stored = self.stored_all()
+			self.checks = []
+			self.delivery(stored)
+			self.counts(stored)
+			self.threads(stored)
+			self.lifecycle(stored)
+			failing = [c for c in self.checks if not c["ok"] and c["feature"].startswith(("deliver", "count", "edit", "delete", "thread"))]
+			if not failing or time.time() - started > deadline:
+				break
+			print(f"  {len(failing)} checks still failing; sweeping again in 20s", flush=True)
+			time.sleep(20)
+		self.visibility(stored)
+		self.graph()
+		self.privacy(stored)
+		return self.checks
+
+	# -- who holds what
+	def delivery(self, stored):
+		for ref, o in self.w.objects.items():
+			if ref in self.w.deleted:
+				continue
+			kind = self.w.kind_for(o)
+			recipients = self.w.recipients(o)
+			want = {self.w.platform(k) for k in recipients} - {o["origin"]}
+			through = kind == "community" and self.w.planner.groups[o["group"]]["host"] == "privapub"
+			for p in self.platforms:
+				if p == o["origin"]:
+					continue
+				row = stored.get(p, {}).get(o["uri"])
+				held = bool(row and row.exists and not row.deleted)
+				receives = RECEIVES.get(p)
+				parent = self.w.objects.get(o.get("parent") or "")
+				orphan = p in DROPS_ORPHAN_REPLIES and parent is not None \
+					and not (stored.get(p, {}).get(parent["uri"]) and stored[p][parent["uri"]].exists) \
+					and parent["visibility"] not in ("public", "unlisted")
+				if orphan:
+					continue
+				if p in want and (receives is None or kind in receives):
+					fetched = p in o.get("fetched_on", {})
+					self.add(f"deliver.{kind}", o["origin"], p, held, ref, "held", "held" if held else "missing", through,
+							 how="fetched by the seeder" if fetched else None)
+				elif kind in ("followers", "direct", "circle", "located") and p not in want:
+					self.add(f"confine.{kind}", o["origin"], p, not held, ref, "absent", "held" if held else "absent", through)
+
+	# -- counts on the object's own server, and PrivaPub's exact counts
+	def counts(self, stored):
+		likes, reacts, boosts, votes = defaultdict(int), defaultdict(lambda: defaultdict(int)), defaultdict(int), defaultdict(lambda: defaultdict(int))
+		for f in self.w.interactions:
+			ref = f["ref"]
+			if f["verb"] == "like":
+				likes[ref] += 1
+			elif f["verb"] == "react":
+				# Misskey and its forks send a reaction as a Like carrying the emoji, and PrivaPub keeps a heart there as a
+				# like (Reactions.IsHeart); an EmojiReact keeps its heart as a reaction, written ❤️
+				if f["emoji"] in HEARTS and self.w.platform(f["actor"]) in ("misskey", "sharkey"):
+					likes[ref] += 1
+				else:
+					reacts[ref]["❤️" if f["emoji"] in HEARTS else f["emoji"]] += 1
+			elif f["verb"] == "boost":
+				boosts[ref] += 1
+			elif f["verb"] == "vote":
+				for c in f["choices"]:
+					votes[ref][c] += 1
+		for ref, o in self.w.objects.items():
+			if ref in self.w.deleted:
+				continue
+			origin = o["origin"]
+			row = stored.get(origin, {}).get(o["uri"])
+			if not row or not row.exists:
+				continue
+			v = self.w.kind_for(o)
+			if origin == "privapub":
+				self.add(f"count.like.{v}", "privapub", "privapub", row.likes == likes[ref], ref, likes[ref], row.likes)
+				want_reacts = {e: n for e, n in reacts[ref].items()}
+				got_reacts = {e: n for e, n in (row.reactions or {}).items()}
+				if want_reacts or got_reacts:
+					self.add(f"count.react.{v}", "privapub", "privapub", got_reacts == want_reacts, ref, want_reacts, got_reacts)
+				if v in ("public", "unlisted"):
+					self.add(f"count.boost.{v}", "privapub", "privapub", row.boosts == boosts[ref], ref, boosts[ref], row.boosts)
+				if votes[ref] or row.votes:
+					want = [votes[ref].get(i, 0) for i in range(len(row.votes or []))]
+					self.add(f"count.vote.{v}", "privapub", "privapub", row.votes == want, ref, want, row.votes)
+			elif origin != "lemmy":
+				# elsewhere: at least the plain likes that every platform sends as Like
+				plain = sum(1 for f in self.w.interactions if f["ref"] == ref and f["verb"] == "like")
+				if plain:
+					total = (row.likes or 0)
+					self.add(f"count.like.{v}", "*", origin, total >= plain, ref, f">={plain}", total)
+				if boosts[ref]:
+					self.add(f"count.boost.{v}", "*", origin, (row.boosts or 0) >= boosts[ref], ref, f">={boosts[ref]}", row.boosts)
+				if votes[ref]:
+					want = sum(votes[ref].values())
+					got = sum(row.votes or [])
+					self.add(f"count.vote.{v}", "*", origin, got >= want, ref, f">={want}", got)
+
+	# -- threads: each copy of a reply names its parent
+	def threads(self, stored):
+		for ref, o in self.w.objects.items():
+			if not o.get("parent") or ref in self.w.deleted or o["parent"] not in self.w.objects:
+				continue
+			parent_uri = self.w.objects[o["parent"]]["uri"]
+			for p in self.platforms:
+				row = stored.get(p, {}).get(o["uri"])
+				if not row or not row.exists or row.deleted or p == "lemmy":
+					continue
+				# a server links a reply to a parent it holds; one it never received (a followers-only post of
+				# someone nobody there follows) leaves the reply unlinked, rightly
+				parent = stored.get(p, {}).get(parent_uri)
+				if not parent or not parent.exists:
+					continue
+				self.add("thread.parent", o["origin"], p, row.parent_uri == parent_uri, ref, parent_uri, row.parent_uri)
+
+	def audience_servers(self, o):
+		return {self.w.platform(k) for k in self.w.recipients(o)} | {o["origin"]}
+
+	# -- edits and deletes reach every copy in the object's audience (a server holding it through a boost or a fetch
+	# is sent no Update: Mastodon and the others address edits to the post's own audience only)
+	def lifecycle(self, stored):
+		for ref in self.w.edited - self.w.deleted:
+			o = self.w.objects.get(ref)
+			if not o:
+				continue
+			audience = self.audience_servers(o)
+			for p in self.platforms:
+				row = stored.get(p, {}).get(o["uri"])
+				if not row or not row.exists or row.deleted or p in ("misskey",) or p not in audience:
+					continue
+				text = re.sub("<[^>]+>", "", row.text or "")
+				self.add("edit.text", o["origin"], p, "(edited)" in text, ref, "(edited)", text[-60:])
+		for ref in self.w.deleted:
+			o = self.w.objects.get(ref)
+			if not o:
+				continue
+			audience = self.audience_servers(o)
+			for p in self.platforms:
+				if p not in audience:
+					continue
+				row = stored.get(p, {}).get(o["uri"])
+				gone = not row or not row.exists or row.deleted
+				self.add("delete.gone", o["origin"], p, gone, ref, "gone", "gone" if gone else "held")
+
+	# -- what accounts see through their APIs
+	def visibility(self, stored):
+		sessions = {f"{s.account.platform}/{s.account.username}": s for s in state.sessions()}
+		by_platform = defaultdict(list)
+		for ref, o in self.w.objects.items():
+			if ref in self.w.deleted or o["visibility"] in ("public", "unlisted", "community"):
+				continue
+			recipients = self.w.recipients(o)
+			for p in self.platforms:
+				if p == "lemmy":
+					continue
+				row = stored.get(p, {}).get(o["uri"])
+				if not row or not row.exists or row.deleted:
+					continue
+				here = [k for k in self.w.planner.accounts if self.w.platform(k) == p and k != o["author"]]
+				inside = sorted(k for k in here if k in recipients)[:2]
+				outside = sorted(k for k in here if k not in recipients)[:2]
+				for k in inside:
+					by_platform[p].append((ref, o, k, True))
+				for k in outside:
+					by_platform[p].append((ref, o, k, False))
+		for p, items in by_platform.items():
+			d = driver(p)
+			for ref, o, k, should in items:
+				s = sessions.get(k)
+				if s is None:
+					continue
+				try:
+					seen = d.seen(s, [o["uri"]])[o["uri"]]
+				except Exception as e:
+					seen = None
+				feature = f"{'see' if should else 'hide'}.{o['visibility']}"
+				self.add(feature, o["origin"], p, seen is should, ref, should, seen, how=k)
+
+	# -- the graph as each side reports it
+	def graph(self):
+		sessions = {f"{s.account.platform}/{s.account.username}": s for s in state.sessions()}
+		for target, followers in self.w.follows.items():
+			for follower in followers:
+				fp = self.w.platform(follower)
+				s = sessions.get(follower)
+				if s is None:
+					continue
+				try:
+					rel = driver(fp).relationship(s, gen.Planner.acct(target))
+					ok = rel.get("following")
+				except Exception as e:
+					ok, rel = False, repr(e)[:100]
+				self.add("graph.following", self.w.platform(target), fp, ok, None, True, ok, how=f"{follower} -> {target}")
+
+	# -- privacy rows
+	def privacy(self, stored):
+		personas = [k for k in self.w.planner.accounts if k.startswith("privapub/")]
+		# PV1 sibling keys differ, PV2 published is a whole day at most two weeks before now
+		from core.http import client
+		http = client()
+		keys, published = {}, {}
+		for k in personas:
+			name = k.split("/")[1]
+			r = http.get(f"https://privapub.test/peasants/{name}", headers={"Accept": "application/activity+json"})
+			doc = r.json() or {}
+			keys[k] = (doc.get("publicKey") or {}).get("publicKeyPem")
+			published[k] = doc.get("published")
+		roots = defaultdict(list)
+		for k in personas:
+			roots[self.w.planner.accounts[k]["root"]].append(k)
+		for root, sibs in roots.items():
+			pems = [keys[k] for k in sibs]
+			self.add("privacy.sibling-keys", "privapub", "privapub", all(pems) and len(set(pems)) == len(pems), None,
+					 "distinct", f"{len(set(pems))} of {len(pems)}")
+			for k in sibs:
+				p = published[k] or ""
+				self.add("privacy.published-day", "privapub", "privapub", p.endswith("T00:00:00Z"), None, "midnight", p, how=k)
+		# PV4 canaries: no root login or password anywhere on a peer
+		canaries = {a["root"] for a in self.w.planner.accounts.values() if a.get("root")} | \
+				   {a["password"] for a in self.w.planner.accounts.values() if a.get("root")}
+		for p in self.platforms:
+			if p == "privapub":
+				continue
+			found = canary_hits(p, canaries)
+			self.add("privacy.canary", "privapub", p, not found, None, "no root login or password", found or "none")
+		# PV7 located posts never left
+		for ref, o in self.w.objects.items():
+			if o["visibility"] != "located":
+				continue
+			held = [p for p in self.platforms if p != "privapub" and stored.get(p, {}).get(o["uri"]) and stored[p][o["uri"]].exists]
+			jobs = podman.mongo(f"db.Job.countDocuments({{Kind: 0, Payload: /{re.escape(o['uri'].rsplit('/', 1)[1])}/}})") or 0
+			self.add("privacy.located-stays", "privapub", "*", not held and not jobs, ref, "nowhere, no delivery",
+					 {"held": held, "jobs": jobs})
+
+
+def canary_hits(platform, canaries):
+	"""Which canaries a peer's database holds anywhere (a full dump, searched as text)."""
+	db = {"mastodon": "mastodon", "misskey": "misskey", "sharkey": "sharkey", "akkoma": "akkoma", "lemmy": "lemmy"}.get(platform)
+	if db:
+		dump = podman.run("exec", "pasture-postgres", "pg_dump", "-U", "pasture", "--data-only", db, timeout=600)
+	elif platform == "gts":
+		dump = podman.run("exec", "pasture-gts", "sh", "-c", "cat /gotosocial/storage/sqlite.db /gotosocial/storage/sqlite.db-wal 2>/dev/null | strings", check=False)
+	else:
+		return []
+	return sorted(c for c in canaries if c in dump)
+
+
+def load_gaps():
+	try:
+		with open(os.path.join(HERE, "gaps.json")) as f:
+			return json.load(f)
+	except FileNotFoundError:
+		return []
+
+
+def classify(checks, gaps):
+	for c in checks:
+		gap = next((g for g in gaps if g.get("status") != "closed" and _matches(g["match"], c)), None)
+		if gap:
+			c["gap"] = gap["id"]
+			c["status"] = "xpass" if c["ok"] else "xfail"
+		else:
+			c["status"] = "pass" if c["ok"] else "fail"
+	return checks
+
+
+def _matches(match, c):
+	for k, pattern in match.items():
+		if not re.fullmatch(pattern, str(c.get(k, ""))):
+			return False
+	return True
+
+
+def summarise(checks):
+	cells = {}
+	for c in checks:
+		cell = cells.setdefault(c["cell"], {"n": 0, "pass": 0, "fail": 0, "xfail": 0, "xpass": 0, "examples": []})
+		cell["n"] += 1
+		cell[c["status"]] += 1
+		if c["status"] in ("fail", "xpass") and len(cell["examples"]) < 3:
+			cell["examples"].append({k: c[k] for k in ("ref", "expect", "got", "how") if c.get(k) is not None})
+	return cells
+
+
+def main(args):
+	import seed
+	name = next((a for a in args if not a.startswith("--")), "village")
+	run_dir = seed.run_dir(name)
+	world = World(run_dir)
+	platforms = [p for p in gen.HOSTS if p in world.spec["peers"]]
+	deadline = world.spec.get("time", {}).get("deadlineSeconds", 240)
+	for a in args:
+		if a.startswith("--deadline="):
+			deadline = int(a.split("=", 1)[1])
+	print(f"checking {run_dir}: {len(world.objects)} objects on {len(platforms)} servers")
+	checks = classify(Sweep(world, platforms).run(deadline), load_gaps())
+	cells = summarise(checks)
+	from collections import Counter
+	totals = Counter(c["status"] for c in checks)
+	result = {"run": run_dir, "when": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()), "totals": totals,
+			  "checks": checks, "cells": cells}
+	with open(os.path.join(run_dir, "results.json"), "w") as f:
+		json.dump(result, f, indent=1, default=str, ensure_ascii=False)
+	print(json.dumps(totals))
+	failing = sorted((c, v) for c, v in cells.items() if v["fail"])
+	for cell, v in failing[:60]:
+		print(f"  FAIL {cell}: {v['fail']}/{v['n']} e.g. {v['examples'][:1]}")
+	strict = "--strict" in args
+	return 1 if totals.get("fail") or (strict and totals.get("xpass")) else 0
+
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))

tools/pasture/town/core/http.py

@@ -0,0 +1,219 @@
+"""HTTP to the pasture: every https://<name>.test site through Caddy on 127.0.0.1:6443, verified against Caddy's own CA
+(.ca/root.crt), with one kept-alive connection per site and thread. 429s wait for the server's reset; connection drops
+are retried. Every request is timed into `Client.timings` for the load mode."""
+import http.client
+import json as jsonlib
+import os
+import socket
+import ssl
+import threading
+import time
+import urllib.parse
+import uuid
+
+CADDY = ("127.0.0.1", 6443)
+
+
+class HttpError(Exception):
+	def __init__(self, method, url, status, body):
+		super().__init__(f"{method} {url} -> {status}: {body[:300]!r}")
+		self.status = status
+		self.body = body
+
+
+class Response:
+	def __init__(self, status, headers, body, url):
+		self.status = status
+		self.headers = headers
+		self.body = body
+		self.url = url
+
+	@property
+	def ok(self):
+		return 200 <= self.status < 300
+
+	@property
+	def text(self):
+		return self.body.decode("utf-8", "replace")
+
+	def json(self):
+		if not self.body:
+			return None
+		try:
+			return jsonlib.loads(self.body)
+		except ValueError:
+			return None
+
+	def header(self, name):
+		for k, v in self.headers:
+			if k.lower() == name.lower():
+				return v
+		return None
+
+
+class _CaddyConnection(http.client.HTTPSConnection):
+	"""Connects to Caddy's port but speaks TLS (SNI and certificate check) as the named site."""
+
+	def __init__(self, site, context, timeout):
+		super().__init__(site, 443, context=context, timeout=timeout)
+		self._site = site
+		self._ctx = context
+
+	def connect(self):
+		sock = socket.create_connection(CADDY, self.timeout)
+		self.sock = self._ctx.wrap_socket(sock, server_hostname=self._site)
+
+
+class Client:
+	def __init__(self, ca_file, timeout=60):
+		self.ctx = ssl.create_default_context(cafile=ca_file)
+		self.timeout = timeout
+		self.local = threading.local()
+		self.timings = []  # (method, site, path template, status, ms)
+		self.lock = threading.Lock()
+		self.record = False
+
+	def _connection(self, scheme, netloc):
+		conns = getattr(self.local, "conns", None)
+		if conns is None:
+			conns = self.local.conns = {}
+		key = (scheme, netloc)
+		conn = conns.get(key)
+		if conn is None:
+			if scheme == "https":
+				conn = _CaddyConnection(netloc.split(":")[0], self.ctx, self.timeout)
+			else:
+				host, _, port = netloc.partition(":")
+				conn = http.client.HTTPConnection(host, int(port or 80), timeout=self.timeout)
+			conns[key] = conn
+		return conn
+
+	def _drop(self, scheme, netloc):
+		conn = self.local.conns.pop((scheme, netloc), None)
+		if conn is not None:
+			conn.close()
+
+	def request(self, method, url, *, headers=None, json=None, form=None, data=None, files=None, params=None,
+				ok=None, attempts=4, wait_429=True, template=None):
+		"""Sends one request. `json` is a JSON body, `form` urlencoded pairs (a dict or a list of pairs, lists expand to
+		key[]=v as Rails expects), `files` a multipart body ({name: (filename, bytes, type)} plus `form` fields).
+		`ok` is a set of accepted statuses: anything else raises HttpError."""
+		parts = urllib.parse.urlsplit(url)
+		path = parts.path or "/"
+		query = parts.query
+		if params:
+			extra = urllib.parse.urlencode(_pairs(params), doseq=True)
+			query = f"{query}&{extra}" if query else extra
+		target = path + (f"?{query}" if query else "")
+		hdrs = {"Accept": "application/json", "User-Agent": "pasture-town/1"}
+		body = None
+		if json is not None:
+			body = jsonlib.dumps(json).encode()
+			hdrs["Content-Type"] = "application/json"
+		elif files is not None:
+			boundary = uuid.uuid4().hex
+			body = _multipart(boundary, _pairs(form or {}), files)
+			hdrs["Content-Type"] = f"multipart/form-data; boundary={boundary}"
+		elif form is not None:
+			body = urllib.parse.urlencode(_pairs(form), doseq=True).encode()
+			hdrs["Content-Type"] = "application/x-www-form-urlencoded"
+		elif data is not None:
+			body = data
+		if headers:
+			hdrs.update(headers)
+		last = None
+		for attempt in range(attempts):
+			conn = self._connection(parts.scheme, parts.netloc)
+			started = time.monotonic()
+			try:
+				conn.request(method, target, body=body, headers=hdrs)
+				raw = conn.getresponse()
+				payload = raw.read()
+				resp = Response(raw.status, raw.getheaders(), payload, url)
+			except (ConnectionError, http.client.HTTPException, socket.timeout, ssl.SSLError, OSError) as e:
+				self._drop(parts.scheme, parts.netloc)
+				last = e
+				time.sleep(0.5 * (attempt + 1))
+				continue
+			ms = (time.monotonic() - started) * 1000
+			if self.record:
+				with self.lock:
+					self.timings.append((method, parts.netloc, template or path, resp.status, ms))
+			if resp.status == 429 and wait_429 and attempt < attempts - 1:
+				time.sleep(_retry_after(resp))
+				continue
+			if resp.status in (502, 503, 504) and attempt < attempts - 1:
+				time.sleep(1 + attempt)
+				continue
+			if ok is not None and resp.status not in ok:
+				raise HttpError(method, url, resp.status, resp.text)
+			return resp
+		raise HttpError(method, url, 0, repr(last))
+
+	def get(self, url, **kw):
+		return self.request("GET", url, **kw)
+
+	def post(self, url, **kw):
+		return self.request("POST", url, **kw)
+
+
+def _pairs(values):
+	"""A dict or list of pairs, with list values as Rails' key[]=v (unless the key already ends in [])."""
+	items = values.items() if isinstance(values, dict) else values
+	out = []
+	for k, v in items:
+		if v is None:
+			continue
+		if isinstance(v, (list, tuple)):
+			key = k if k.endswith("[]") else f"{k}[]"
+			out.extend((key, _scalar(x)) for x in v)
+		else:
+			out.append((k, _scalar(v)))
+	return out
+
+
+def _scalar(v):
+	if v is True:
+		return "true"
+	if v is False:
+		return "false"
+	return str(v)
+
+
+def _multipart(boundary, fields, files):
+	chunks = []
+	for k, v in fields:
+		chunks.append(f'--{boundary}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
+	for name, (filename, content, ctype) in files.items():
+		chunks.append(f'--{boundary}\r\nContent-Disposition: form-data; name="{name}"; filename="{filename}"\r\n'
+					  f"Content-Type: {ctype}\r\n\r\n".encode() + content + b"\r\n")
+	chunks.append(f"--{boundary}--\r\n".encode())
+	return b"".join(chunks)
+
+
+def _retry_after(resp):
+	value = resp.header("Retry-After")
+	if value and value.isdigit():
+		return min(int(value), 60)
+	reset = resp.header("X-RateLimit-Reset")
+	if reset:
+		try:
+			from email.utils import parsedate_to_datetime
+			import datetime
+			when = datetime.datetime.fromisoformat(reset.replace("Z", "+00:00")) if "T" in reset \
+				else parsedate_to_datetime(reset)
+			return max(1, min(60, (when - datetime.datetime.now(datetime.timezone.utc)).total_seconds()))
+		except (ValueError, TypeError):
+			pass
+	return 5
+
+
+_client = None
+
+
+def client():
+	global _client
+	if _client is None:
+		here = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+		_client = Client(os.path.join(here, ".ca", "root.crt"))
+	return _client

tools/pasture/town/core/media.py

@@ -0,0 +1,67 @@
+"""Media made from a seed, so two runs upload the same bytes: identicon PNGs, a sine-tone WAV, and the committed
+media/tiny.mp4 and media/tiny.ogg (made once with ffmpeg; the command is in media/SOURCES.md)."""
+import hashlib
+import math
+import os
+import struct
+import zlib
+
+HERE = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+def png(seed, size=96, cells=6):
+	"""A symmetric identicon: a colour and a mirrored grid from the SHA-256 of the seed."""
+	h = hashlib.sha256(str(seed).encode()).digest()
+	fg = (h[0], h[1], h[2])
+	bg = (240, 240, 236) if sum(fg) < 380 else (34, 34, 40)
+	half = (cells + 1) // 2
+	bits = int.from_bytes(h[3:11], "big")
+	grid = [[False] * cells for _ in range(cells)]
+	for y in range(cells):
+		for x in range(half):
+			on = bool(bits & 1)
+			bits >>= 1
+			grid[y][x] = grid[y][cells - 1 - x] = on
+	cell = size // cells
+	rows = []
+	for py in range(cell * cells):
+		row = bytearray(b"\x00")
+		for px in range(cell * cells):
+			row += bytes(fg if grid[py // cell][px // cell] else bg)
+		rows.append(bytes(row))
+	w = hgt = cell * cells
+	return _png_bytes(w, hgt, b"".join(rows))
+
+
+def _png_bytes(w, h, raw):
+	def chunk(t, c):
+		return struct.pack(">I", len(c)) + t + c + struct.pack(">I", zlib.crc32(t + c) & 0xffffffff)
+	return (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
+			+ chunk(b"IDAT", zlib.compress(raw, 9)) + chunk(b"IEND", b""))
+
+
+def wav(seed, seconds=1.0, rate=8000):
+	"""A mono 8-bit tone whose pitch comes from the seed."""
+	h = hashlib.sha256(str(seed).encode()).digest()
+	freq = 220 + h[0] * 2
+	n = int(seconds * rate)
+	samples = bytes(int(128 + 90 * math.sin(2 * math.pi * freq * i / rate)) for i in range(n))
+	header = b"RIFF" + struct.pack("<I", 36 + n) + b"WAVEfmt " + struct.pack("<IHHIIHH", 16, 1, 1, rate, rate, 1, 8) \
+		+ b"data" + struct.pack("<I", n)
+	return header + samples
+
+
+def file(name):
+	with open(os.path.join(HERE, "media", name), "rb") as f:
+		return f.read()
+
+
+def upload(kind, seed):
+	"""(filename, bytes, content type) for an upload of kind image, audio or video."""
+	if kind == "image":
+		return (f"town-{seed}.png", png(seed), "image/png")
+	if kind == "audio":
+		return (f"town-{seed}.ogg", file("tiny.ogg"), "audio/ogg")
+	if kind == "video":
+		return (f"town-{seed}.mp4", file("tiny.mp4"), "video/mp4")
+	raise ValueError(kind)

tools/pasture/town/core/podman.py

@@ -0,0 +1,99 @@
+"""The pasture's containers from the workstation: commands, Postgres rows as JSON, Mongo documents as JSON, a copy of
+GoToSocial's sqlite. Only reads go through here; nothing a peer does is faked in its database."""
+import json
+import os
+import shutil
+import sqlite3
+import subprocess
+import tempfile
+
+
+def run(*args, input=None, check=True, timeout=300):
+	proc = subprocess.run(["podman", *args], input=input, capture_output=True, timeout=timeout,
+						  text=isinstance(input, str) or input is None)
+	if check and proc.returncode != 0:
+		raise RuntimeError(f"podman {' '.join(args[:3])}… failed: {proc.stderr.strip()[:500]}")
+	return proc.stdout
+
+
+def exec_(container, *cmd, input=None, check=True, timeout=300, workdir=None):
+	args = ["exec"]
+	if input is not None:
+		args.append("-i")
+	if workdir:
+		args += ["-w", workdir]
+	return run(*args, container, *cmd, input=input, check=check, timeout=timeout)
+
+
+def exists(container):
+	return subprocess.run(["podman", "container", "exists", container]).returncode == 0
+
+
+def running():
+	out = run("ps", "--format", "{{.Names}}")
+	return [n for n in out.split() if n.startswith("pasture-")]
+
+
+def psql(db, sql, *params):
+	"""Rows of `sql` as a list of dicts. Parameters are bound by psql (:'p1', :'p2'...), never pasted into the SQL."""
+	args = ["exec", "-i", "pasture-postgres", "psql", "-U", "pasture", "-d", db, "-tAX", "-v", "ON_ERROR_STOP=1"]
+	for i, p in enumerate(params, 1):
+		args += ["-v", f"p{i}={p}"]
+	wrapped = f"select coalesce(json_agg(t), '[]'::json) from ({sql.rstrip().rstrip(';')}) t;\n"
+	out = run(*args, input=wrapped)
+	return json.loads(out.strip() or "[]")
+
+
+def psql_value(db, sql, *params):
+	rows = psql(db, sql, *params)
+	if not rows:
+		return None
+	return next(iter(rows[0].values()))
+
+
+def mongo(js, db="PrivaPub"):
+	"""The JSON value of a mongosh expression, e.g. `db.Post.find({...}).toArray()`."""
+	script = f"print(EJSON.stringify(({js}), {{relaxed: true}}))"
+	out = run("exec", "pasture-mongo", "mongosh", "--quiet", db, "--eval", script, timeout=120)
+	return json.loads(out.strip().splitlines()[-1]) if out.strip() else None
+
+
+class SqliteCopy:
+	"""A consistent copy of a container's sqlite database (with its WAL), opened read-only on the workstation."""
+
+	def __init__(self, container, path):
+		self.dir = tempfile.mkdtemp(prefix="town-sqlite-")
+		local = os.path.join(self.dir, os.path.basename(path))
+		run("cp", f"{container}:{path}", local)
+		for suffix in ("-wal", "-shm"):
+			subprocess.run(["podman", "cp", f"{container}:{path}{suffix}", local + suffix], capture_output=True)
+		self.db = sqlite3.connect(local)
+		self.db.row_factory = sqlite3.Row
+
+	def rows(self, sql, *params):
+		return [dict(r) for r in self.db.execute(sql, params)]
+
+	def close(self):
+		self.db.close()
+		shutil.rmtree(self.dir, ignore_errors=True)
+
+	def __enter__(self):
+		return self
+
+	def __exit__(self, *exc):
+		self.close()
+
+
+def logs(container, since=None):
+	args = ["logs"]
+	if since:
+		args += ["--since", since]
+	return run(*args, container, check=False)
+
+
+def stats():
+	out = run("stats", "--no-stream", "--format", "json", *running(), check=False)
+	try:
+		return json.loads(out)
+	except ValueError:
+		return []

tools/pasture/town/core/rng.py

@@ -0,0 +1,93 @@
+"""A seeded random generator that is the same on every Python: xoshiro256** seeded through splitmix64 from a SHA-256
+of the seed and a name. `sub(name)` gives an independent stream, so changing how many posts are drawn never reshuffles
+the follow graph."""
+import hashlib
+
+MASK = (1 << 64) - 1
+
+
+def _rotl(x, k):
+	return ((x << k) | (x >> (64 - k))) & MASK
+
+
+def _splitmix(state):
+	while True:
+		state = (state + 0x9E3779B97F4A7C15) & MASK
+		z = state
+		z = ((z ^ (z >> 30)) * 0xBF58476D1CE4E5B9) & MASK
+		z = ((z ^ (z >> 27)) * 0x94D049BB133111EB) & MASK
+		yield z ^ (z >> 31)
+
+
+class Rng:
+	def __init__(self, seed, name="root"):
+		self.seed = seed
+		self.name = name
+		digest = hashlib.sha256(f"{seed}/{name}".encode()).digest()
+		mix = _splitmix(int.from_bytes(digest[:8], "little"))
+		self.s = [next(mix) for _ in range(4)]
+
+	def sub(self, name):
+		return Rng(self.seed, f"{self.name}/{name}")
+
+	def next64(self):
+		s = self.s
+		result = (_rotl((s[1] * 5) & MASK, 7) * 9) & MASK
+		t = (s[1] << 17) & MASK
+		s[2] ^= s[0]
+		s[3] ^= s[1]
+		s[1] ^= s[2]
+		s[0] ^= s[3]
+		s[2] ^= t
+		s[3] = _rotl(s[3], 45)
+		return result
+
+	def random(self):
+		return (self.next64() >> 11) / float(1 << 53)
+
+	def below(self, n):
+		"""An integer in [0, n)."""
+		if n <= 0:
+			raise ValueError("below() needs n > 0")
+		limit = MASK - (MASK % n)
+		while True:
+			x = self.next64()
+			if x < limit:
+				return x % n
+
+	def between(self, lo, hi):
+		"""An integer in [lo, hi]."""
+		return lo + self.below(hi - lo + 1)
+
+	def chance(self, p):
+		return self.random() < p
+
+	def choice(self, seq):
+		return seq[self.below(len(seq))]
+
+	def weighted(self, weights):
+		"""A key of {key: weight}, keys taken in sorted order so the draw does not depend on dict order."""
+		keys = sorted(k for k, w in weights.items() if w > 0)
+		total = sum(weights[k] for k in keys)
+		x = self.random() * total
+		for k in keys:
+			x -= weights[k]
+			if x < 0:
+				return k
+		return keys[-1]
+
+	def shuffle(self, items):
+		items = list(items)
+		for i in range(len(items) - 1, 0, -1):
+			j = self.below(i + 1)
+			items[i], items[j] = items[j], items[i]
+		return items
+
+	def sample(self, seq, k):
+		return self.shuffle(seq)[:max(0, min(k, len(seq)))]
+
+	def span(self, pair):
+		"""A value drawn from [lo, hi] given as a two-item list, or the value itself."""
+		if isinstance(pair, (list, tuple)):
+			return self.between(pair[0], pair[1])
+		return pair

tools/pasture/town/dialects/__init__.py

@@ -0,0 +1,28 @@
+"""The platform registry: name -> driver class and its site."""
+from dialects.akkoma import Akkoma
+from dialects.gts import Gts
+from dialects.hollo import Hollo
+from dialects.lemmy_api import LemmyApi
+from dialects.mastodon import Mastodon
+from dialects.misskey_api import Misskey, Sharkey
+from dialects.privapub import PrivaPub
+
+DRIVERS = {
+	"privapub": (PrivaPub, "privapub.test"),
+	"gts": (Gts, "gts.test"),
+	"mastodon": (Mastodon, "mastodon.test"),
+	"misskey": (Misskey, "misskey.test"),
+	"sharkey": (Sharkey, "sharkey.test"),
+	"akkoma": (Akkoma, "akkoma.test"),
+	"lemmy": (LemmyApi, "lemmy.test"),
+	"hollo": (Hollo, "hollo.test"),
+}
+
+_made = {}
+
+
+def driver(platform):
+	if platform not in _made:
+		cls, host = DRIVERS[platform]
+		_made[platform] = cls(host)
+	return _made[platform]

tools/pasture/town/dialects/akkoma.py

@@ -0,0 +1,94 @@
+"""Akkoma 3.20 (and Pleroma, which shares all of this): accounts made with pleroma_ctl, tokens through the password
+grant, objects read from its Postgres database (`akkoma`). Its Linkify never takes @name@host.test for a mention, so
+mentions are addressed with Pleroma's `to[]` as well. Status ids are the Create activity's FlakeId in base62."""
+import urllib.parse
+
+from core import podman
+from dialects.base import Stored
+from dialects.mastodon_api import MastodonApi
+
+PUBLIC = "https://www.w3.org/ns/activitystreams#Public"
+B62 = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
+
+
+def flake(uuid):
+	n = int(uuid.replace("-", ""), 16)
+	out = ""
+	while n:
+		n, r = divmod(n, 62)
+		out = B62[r] + out
+	return out or "0"
+
+
+class Akkoma(MastodonApi):
+	platform = "akkoma"
+	caps = MastodonApi.caps | {"react", "quote"}
+	container = "pasture-akkoma"
+	ctl = "/opt/akkoma/bin/pleroma_ctl"
+	db = "akkoma"
+
+	def provision(self, accounts):
+		existing = {r["nickname"] for r in podman.psql(self.db, "select nickname from users where local")}
+		for a in accounts:
+			if a.username in existing:
+				continue
+			# pleroma_ctl passes its arguments on unquoted: no value may hold a space
+			podman.exec_(self.container, self.ctl, "user", "new", a.username, f"{a.username}@{self.host}",
+						 "--password", a.password, "--name", a.username, "--assume-yes")
+		app = self.http.post(self.base + "/api/v1/apps", ok={200}, form={
+			"client_name": "pasture-town", "redirect_uris": "urn:ietf:wg:oauth:2.0:oob", "scopes": "read write follow"}).json()
+		sessions = []
+		for a in accounts:
+			token = self.http.post(self.base + "/oauth/token", ok={200}, form={
+				"grant_type": "password", "username": a.username, "password": a.password, "client_id": app["client_id"],
+				"client_secret": app["client_secret"], "scope": "read write follow"}).json()["access_token"]
+			sessions.append(self.session_from_token(a, token))
+		return sessions
+
+	def status_form(self, s, spec):
+		form = super().status_form(s, spec)
+		if spec.mentions:
+			# Pleroma's to[] takes a local user by bare nickname: name@own-host addresses nobody
+			form["to"] = [a.split("@")[0] if a.endswith("@" + self.host) else a for a in spec.mentions]
+		return form
+
+	def react(self, s, uri, emoji):
+		sid = self.local_status_id(s, uri) or self.resolve(s, uri)
+		self.api(s, "PUT", f"/api/v1/pleroma/statuses/{sid}/reactions/{urllib.parse.quote(emoji)}", ok={200})
+
+	def _rows(self, uris):
+		if not uris:
+			return {}
+		rows = podman.psql(self.db, """
+			select o.data->>'id' as uri, o.data as data, a.id::text as act
+			from objects o
+			left join activities a on a.data->>'type' = 'Create'
+				and coalesce(a.data->'object'->>'id', a.data->>'object') = o.data->>'id'
+			where o.data->>'id' = any(string_to_array(:'p1', ' '))""", " ".join(uris))
+		out = {}
+		for r in rows:
+			d = r["data"]
+			deleted = d.get("type") == "Tombstone"
+			to, cc = _list(d.get("to")), _list(d.get("cc"))
+			if PUBLIC in to:
+				vis = "public"
+			elif PUBLIC in cc:
+				vis = "unlisted"
+			elif any(x.endswith("/followers") for x in to + cc):
+				vis = "followers"
+			else:
+				vis = "direct"
+			options = d.get("oneOf") or d.get("anyOf")
+			votes = [((o.get("replies") or {}).get("totalItems") or 0) for o in options] if options else None
+			reactions = {e[0]: len(e[1]) for e in d.get("reactions") or [] if len(e) >= 2}
+			out[r["uri"]] = Stored(True, deleted, flake(r["act"]) if r["act"] else None, vis, d.get("content"),
+								   d.get("summary") or None, bool(d.get("formerRepresentations")), d.get("inReplyTo"),
+								   d.get("like_count") or 0, d.get("announcement_count") or 0, d.get("repliesCount") or 0,
+								   votes, reactions, d)
+		return out
+
+
+def _list(v):
+	if v is None:
+		return []
+	return v if isinstance(v, list) else [v]

tools/pasture/town/dialects/base.py

@@ -0,0 +1,212 @@
+"""What every peer driver offers. A driver acts as one account of its platform through that platform's own API, and reads
+back what the platform holds without ever making it fetch: `stored` asks its database, `seen` its API as one account.
+Anything a platform cannot do raises Unsupported, and the planner never asks for it."""
+from dataclasses import dataclass, field
+from typing import Optional
+
+from core.http import client
+
+
+class Unsupported(Exception):
+	def __init__(self, platform, what):
+		super().__init__(f"{platform} cannot {what}")
+		self.platform = platform
+		self.what = what
+
+
+@dataclass
+class Account:
+	platform: str
+	username: str
+	password: str
+	name: str = None
+	bio: str = None
+	fields: list = field(default_factory=list)  # [(name, value)]
+	locked: bool = False
+	bot: bool = False
+	lang: str = "en"
+	avatar_seed: str = None
+	header_seed: str = None
+	root: str = None  # PrivaPub only: the root login that owns this persona
+
+	def acct(self, host):
+		return f"{self.username}@{host}"
+
+
+@dataclass
+class Session:
+	account: Account
+	token: str
+	local_id: str  # the account's id in its own server's API
+	actor_uri: str  # its ActivityPub id
+
+
+@dataclass
+class PostSpec:
+	text: str
+	kind: str = "note"  # note | article | page | event | link | video | audio | image
+	visibility: str = "public"  # public | unlisted | followers | direct | circle | community | located
+	cw: str = None
+	mentions: list = field(default_factory=list)  # accts: "name@host"
+	tags: list = field(default_factory=list)
+	media: list = field(default_factory=list)  # [{"kind": "image|audio|video", "seed": str, "alt": str, "sensitive": bool}]
+	poll: dict = None  # {"options": [...], "multiple": bool, "expires_in": seconds}
+	quote_uri: str = None
+	reply_to_uri: str = None
+	title: str = None
+	link: str = None
+	event: dict = None  # {"start", "end", "place"}
+	language: str = None
+	group: str = None  # community or circle reference (driver-specific id)
+	location: dict = None  # PrivaPub located posts: {"lat", "lng", "range_km"}
+
+
+@dataclass
+class Made:
+	uri: str
+	local_id: str
+	url: Optional[str] = None
+
+
+@dataclass
+class Stored:
+	"""One object as a server holds it, read from its database."""
+	exists: bool
+	deleted: bool = False
+	local_id: str = None
+	visibility: str = None
+	text: str = None
+	cw: str = None
+	edited: bool = False
+	parent_uri: str = None
+	likes: int = None
+	boosts: int = None
+	replies: int = None
+	votes: list = None
+	reactions: dict = None
+	raw: dict = None
+
+
+class Driver:
+	"""Base for every platform. `host` is its site name (gts.test); `api` its base URL through Caddy."""
+	platform = "base"
+	caps = frozenset()
+
+	def __init__(self, host):
+		self.host = host
+		self.base = f"https://{host}"
+		self.http = client()
+
+	# -- accounts
+	def provision(self, accounts):
+		"""Creates the accounts (idempotently) and returns a Session for each, in order."""
+		raise Unsupported(self.platform, "provision accounts")
+
+	def update_profile(self, s, account):
+		raise Unsupported(self.platform, "update profiles")
+
+	def lookup(self, s, acct):
+		"""The local id of account `acct` (name@host) as `s` sees it, resolving it through WebFinger if needed."""
+		raise Unsupported(self.platform, "look up accounts")
+
+	# -- the graph
+	def follow(self, s, acct):
+		"""Follows acct; returns "accepted" or "requested"."""
+		raise Unsupported(self.platform, "follow")
+
+	def unfollow(self, s, acct):
+		raise Unsupported(self.platform, "unfollow")
+
+	def pending(self, s):
+		"""The accts waiting for `s` to accept their follow."""
+		raise Unsupported(self.platform, "list follow requests")
+
+	def accept(self, s, acct):
+		raise Unsupported(self.platform, "accept follow requests")
+
+	def reject(self, s, acct):
+		raise Unsupported(self.platform, "reject follow requests")
+
+	def relationship(self, s, acct):
+		"""{following, followed_by, requested, blocking, muting}"""
+		raise Unsupported(self.platform, "read relationships")
+
+	def block(self, s, acct):
+		raise Unsupported(self.platform, "block")
+
+	def unblock(self, s, acct):
+		raise Unsupported(self.platform, "unblock")
+
+	def mute(self, s, acct):
+		raise Unsupported(self.platform, "mute")
+
+	def report(self, s, acct, uris, comment):
+		raise Unsupported(self.platform, "report")
+
+	# -- content
+	def post(self, s, spec):
+		raise Unsupported(self.platform, "post")
+
+	def edit(self, s, uri, spec):
+		raise Unsupported(self.platform, "edit")
+
+	def delete(self, s, uri):
+		raise Unsupported(self.platform, "delete")
+
+	def like(self, s, uri):
+		raise Unsupported(self.platform, "like")
+
+	def unlike(self, s, uri):
+		raise Unsupported(self.platform, "unlike")
+
+	def boost(self, s, uri):
+		raise Unsupported(self.platform, "boost")
+
+	def unboost(self, s, uri):
+		raise Unsupported(self.platform, "unboost")
+
+	def react(self, s, uri, emoji):
+		raise Unsupported(self.platform, "react")
+
+	def vote(self, s, uri, choices):
+		raise Unsupported(self.platform, "vote")
+
+	def bookmark(self, s, uri):
+		raise Unsupported(self.platform, "bookmark")
+
+	# -- reading back, never fetching
+	def local_status_id(self, s, uri):
+		"""This server's API id for the object `uri`, or None if it does not hold it. Never fetches."""
+		raise Unsupported(self.platform, "find statuses by URI")
+
+	def resolve(self, s, uri):
+		"""This server's API id for `uri`, fetching it if it must (a deliberate fetch, recorded by the caller)."""
+		raise Unsupported(self.platform, "resolve statuses")
+
+	def stored(self, uris):
+		"""{uri: Stored} for every uri, from the database."""
+		raise Unsupported(self.platform, "read its database")
+
+	def seen(self, s, uris):
+		"""{uri: bool}: whether `s` can see each object through the API."""
+		out = {}
+		for uri in uris:
+			sid = self.local_status_id(s, uri)
+			out[uri] = sid is not None and self.visible(s, sid)
+		return out
+
+	def visible(self, s, local_id):
+		raise Unsupported(self.platform, "read statuses")
+
+	def notifications(self, s):
+		"""[{type, acct, uri}] newest first, normalised to Mastodon's types."""
+		raise Unsupported(self.platform, "read notifications")
+
+	def actor_uri(self, username):
+		"""The ActivityPub id of a local account, as this server's WebFinger names it."""
+		r = self.http.get(f"{self.base}/.well-known/webfinger", params={"resource": f"acct:{username}@{self.host}"},
+						  headers={"Accept": "application/jrd+json"})
+		for link in (r.json() or {}).get("links", []) if r.ok else []:
+			if link.get("rel") == "self" and "json" in (link.get("type") or ""):
+				return link["href"]
+		return None

tools/pasture/town/dialects/gts.py

@@ -0,0 +1,103 @@
+"""GoToSocial 0.22: accounts made with its admin CLI (then one restart, as the pasture does), tokens through the OAuth
+code flow with a signed-in cookie (it has no password grant), objects read from a copy of its sqlite database."""
+import json
+import re
+import urllib.parse
+
+from core import podman
+from dialects.base import Stored
+from dialects.mastodon_api import MastodonApi
+
+# gtsmodel.Visibility since 0.20: stored as small integers
+VIS = {2: "public", 3: "unlisted", 4: "followers", 5: "mutuals", 6: "direct"}
+OOB = "urn:ietf:wg:oauth:2.0:oob"
+
+
+class Gts(MastodonApi):
+	platform = "gts"
+
+	def provision(self, accounts):
+		created = False
+		existing = {r["username"] for r in self._sqlite("select username from accounts where domain is null")}
+		for a in accounts:
+			if a.username in existing:
+				continue
+			podman.exec_("pasture-gts", "/gotosocial/gotosocial", "admin", "account", "create", "--username", a.username,
+						 "--email", f"{a.username}@gts.test", "--password", a.password)
+			podman.exec_("pasture-gts", "/gotosocial/gotosocial", "admin", "account", "confirm", "--username", a.username)
+			created = True
+		if created:
+			# the running server keeps what it has cached about accounts; the pasture restarts it after the CLI too
+			podman.run("restart", "pasture-gts")
+			self._wait()
+		return [self.session_from_token(a, self._token(a)) for a in accounts]
+
+	def _wait(self):
+		import time
+		for _ in range(90):
+			try:
+				if self.http.get(self.base + "/api/v1/instance").status == 200:
+					return
+			except Exception:
+				pass
+			time.sleep(1)
+		raise TimeoutError("GoToSocial did not come back")
+
+	def _token(self, a):
+		app = self.http.post(self.base + "/api/v1/apps", ok={200}, form={
+			"client_name": "pasture-town", "redirect_uris": OOB, "scopes": "read write follow"}).json()
+		cookies = {}
+
+		def send(method, path, **kw):
+			# the sign-in pages are HTML: asked for JSON (the client's default) they answer 406 and set no session
+			headers = {"Accept": "text/html,*/*"}
+			if cookies:
+				headers["Cookie"] = "; ".join(f"{k}={v}" for k, v in cookies.items())
+			r = self.http.request(method, self.base + path, headers=headers, **kw)
+			for k, v in r.headers:
+				if k.lower() == "set-cookie":
+					name, _, value = v.split(";")[0].partition("=")
+					cookies[name.strip()] = value.strip()
+			return r
+
+		query = urllib.parse.urlencode({"client_id": app["client_id"], "redirect_uri": OOB, "response_type": "code",
+										"scope": "read write follow"})
+		send("GET", f"/oauth/authorize?{query}")
+		send("POST", "/auth/sign_in", form={"username": f"{a.username}@gts.test", "password": a.password})
+		r = send("POST", "/oauth/authorize")
+		location = r.header("Location") or ""
+		code = urllib.parse.parse_qs(urllib.parse.urlsplit(location).query).get("code", [None])[0]
+		if code is None:
+			found = re.search(r"code=([A-Za-z0-9_\-]+)", r.text)
+			code = found.group(1) if found else None
+		if code is None:
+			raise RuntimeError(f"GoToSocial gave {a.username} no authorization code ({r.status})")
+		token = self.http.post(self.base + "/oauth/token", ok={200}, form={
+			"grant_type": "authorization_code", "code": code, "client_id": app["client_id"],
+			"client_secret": app["client_secret"], "redirect_uri": OOB, "scope": "read write follow"}).json()
+		return token["access_token"]
+
+	def _sqlite(self, sql, *params):
+		with podman.SqliteCopy("pasture-gts", "/gotosocial/storage/sqlite.db") as db:
+			return db.rows(sql, *params)
+
+	def _rows(self, uris):
+		if not uris:
+			return {}
+		marks = ",".join("?" * len(uris))
+		rows = self._sqlite(f"""
+			select s.id as local_id, s.uri, s.visibility, s.flags, s.text, s.content, s.content_warning as cw,
+				s.edited_at is not null as edited, s.in_reply_to_uri as parent_uri,
+				(select count(*) from status_faves f where f.status_id = s.id) as likes,
+				(select count(*) from statuses b where b.boost_of_id = s.id) as boosts,
+				(select count(*) from statuses r where r.in_reply_to_id = s.id) as replies,
+				(select votes from polls p where p.status_id = s.id) as votes
+			from statuses s where s.boost_of_id is null and s.uri in ({marks})""", *uris)
+		out = {}
+		for r in rows:
+			votes = json.loads(r["votes"]) if r["votes"] else None
+			# 0.22 soft-deletes: flag 2 marks a deleted status (its statuses_deleted_idx), 8 a local one
+			out[r["uri"]] = Stored(True, bool(r["flags"] & 2), r["local_id"], VIS.get(r["visibility"], str(r["visibility"])),
+								   r["text"] or r["content"], r["cw"] or None, bool(r["edited"]), r["parent_uri"],
+								   r["likes"], r["boosts"], r["replies"], votes, None, r)
+		return out

tools/pasture/town/dialects/hollo.py

@@ -0,0 +1,124 @@
+"""Hollo 0.9 (Fedify): one login (made by peers/hollo.sh) owning every town account, each its own actor, the way a PrivaPub
+root owns its personas. Accounts are made through its /accounts form and tokens through its OAuth consent page, which
+asks the signed-in owner which account the token is for. Its forms check only that Origin is its own (Hono's csrf())."""
+import html
+import re
+import urllib.parse
+
+from core import podman
+from dialects.base import Stored
+from dialects.mastodon_api import MastodonApi
+
+OOB = "urn:ietf:wg:oauth:2.0:oob"
+OWNER = ("owner@hollo.test", "Hollo-Pasture-Pass-1")
+VIS = {"public": "public", "unlisted": "unlisted", "private": "followers", "direct": "direct"}
+
+
+class Hollo(MastodonApi):
+	platform = "hollo"
+	caps = MastodonApi.caps | {"quote", "react"}
+
+	def __init__(self, host):
+		super().__init__(host)
+		self._cookies = {}
+
+	def _form(self, method, path, form=None):
+		headers = {"Accept": "text/html,*/*", "Origin": self.base}
+		if self._cookies:
+			headers["Cookie"] = "; ".join(f"{k}={v}" for k, v in self._cookies.items())
+		r = self.http.request(method, self.base + path, headers=headers, form=form)
+		for k, v in r.headers:
+			if k.lower() == "set-cookie":
+				name, _, value = v.split(";")[0].partition("=")
+				self._cookies[name.strip()] = value.strip()
+		return r
+
+	def _login(self):
+		if "login" not in self._cookies:
+			self._form("POST", "/login", {"email": OWNER[0], "password": OWNER[1]})
+
+	def provision(self, accounts):
+		self._login()
+		existing = {r["handle"].split("@")[1] for r in podman.psql("hollo", "select handle from accounts where handle like '@%'")
+					if r["handle"].count("@") >= 2}
+		for a in accounts:
+			if a.username in existing:
+				continue
+			self._form("POST", "/accounts", {"username": a.username, "name": a.name or a.username, "bio": a.bio or "",
+											 "discoverable": "on", "language": a.lang or "en", "visibility": "public", "themeColor": "azure",
+											 **({"protected": "on"} if a.locked else {})})
+		app = self.http.post(self.base + "/api/v1/apps", ok={200}, form={
+			"client_name": "pasture-town", "redirect_uris": OOB, "scopes": "read write follow"}).json()
+		return [self.session_from_token(a, self._token(app, a.username)) for a in accounts]
+
+	def _token(self, app, username):
+		query = urllib.parse.urlencode({"client_id": app["client_id"], "redirect_uri": OOB, "response_type": "code",
+										"scope": "read write follow"})
+		page = self._form("GET", f"/oauth/authorize?{query}").text
+		form = dict((k, html.unescape(v)) for k, v in re.findall(r'<input type="hidden" name="([^"]+)" value="([^"]*)"', page))
+		account = None
+		for block in re.findall(r'<label[^>]*>(.*?)</label>', page, re.S):
+			if f"@{username}@" in block:
+				found = re.search(r'name="account_id"[^>]*value="([^"]+)"|value="([^"]+)"[^>]*name="account_id"', block)
+				if found:
+					account = found.group(1) or found.group(2)
+		if account is None:
+			raise RuntimeError(f"Hollo's consent page offers no account @{username}")
+		form.update({"account_id": account, "decision": "allow"})
+		r = self._form("POST", "/oauth/authorize", form)
+		location = r.header("Location") or ""
+		code = urllib.parse.parse_qs(urllib.parse.urlsplit(location).query).get("code", [None])[0]
+		if code is None:
+			found = re.search(r'<code[^>]*>([^<]+)</code>', r.text)
+			code = found.group(1).strip() if found else None
+		if code is None:
+			raise RuntimeError(f"Hollo gave @{username} no authorization code ({r.status})")
+		return self.http.post(self.base + "/oauth/token", ok={200}, form={
+			"grant_type": "authorization_code", "code": code, "client_id": app["client_id"],
+			"client_secret": app["client_secret"], "redirect_uri": OOB}).json()["access_token"]
+
+	def update_profile(self, s, account):
+		# Hollo has no update_credentials for fields and images beyond its own forms; name, bio and lock are enough here
+		form = {"display_name": account.name, "note": account.bio, "locked": account.locked}
+		self.api(s, "PATCH", "/api/v1/accounts/update_credentials", ok={200}, form=form)
+
+	def post(self, s, spec):
+		"""Hollo reads a status as JSON (its poll is a nested object a form cannot carry)."""
+		from dialects.base import Made, Unsupported
+		if spec.kind not in ("note", "image", "video", "audio"):
+			raise Unsupported(self.platform, f"post a {spec.kind}")
+		form = self.status_form(s, spec)
+		body = {k: v for k, v in form.items() if not k.startswith("poll[")}
+		if spec.poll:
+			body["poll"] = {"options": spec.poll["options"], "expires_in": spec.poll.get("expires_in", 86400),
+							"multiple": bool(spec.poll.get("multiple"))}
+		status = self.api_json(s, "POST", "/api/v1/statuses", json=body)
+		return Made(status["uri"], status["id"], status.get("url"))
+
+	def vote(self, s, uri, choices):
+		sid = self.local_status_id(s, uri) or self.resolve(s, uri)
+		poll = (self.api_json(s, "GET", f"/api/v1/statuses/{sid}").get("poll") or {})
+		if not poll:
+			raise LookupError(f"{self.host} shows no poll on {uri}")
+		self.api(s, "POST", f"/api/v1/polls/{poll['id']}/votes", ok={200}, json={"choices": choices})
+
+	def react(self, s, uri, emoji):
+		sid = self.local_status_id(s, uri) or self.resolve(s, uri)
+		self.api(s, "PUT", f"/api/v1/statuses/{sid}/emoji_reactions/{urllib.parse.quote(emoji)}", ok={200})
+
+	def _rows(self, uris):
+		if not uris:
+			return {}
+		rows = podman.psql("hollo", """
+			select p.id::text as local_id, p.iri as uri, p.visibility::text as visibility, p.content_html as text,
+				p.summary as cw, p.updated > p.published as edited, r.iri as parent_uri,
+				(select count(*) from likes l where l.post_id = p.id) as likes,
+				p.shares_count as boosts, p.replies_count as replies,
+				(select array_agg(o.votes_count order by o.index) from poll_options o where o.poll_id = p.poll_id) as votes,
+				(select json_object_agg(e.emoji, e.n) from (select emoji, count(*) n from reactions x where x.post_id = p.id
+					group by emoji) e) as reactions
+			from posts p left join posts r on r.id = p.reply_target_id
+			where p.sharing_id is null and p.iri = any(string_to_array(:'p1', ' '))""", " ".join(uris))
+		return {r["uri"]: Stored(True, False, r["local_id"], VIS.get(r["visibility"], r["visibility"]), r["text"], r["cw"],
+								 bool(r["edited"]), r["parent_uri"], r["likes"], r["boosts"], r["replies"], r["votes"], r["reactions"] or {}, r)
+				for r in rows}

tools/pasture/town/dialects/lemmy_api.py

@@ -0,0 +1,211 @@
+"""Lemmy 1.0 (API v4; PieFed's Lemmy-compatible API later): threads in communities, comments, votes, private
+messages. Its admin is made by `setup`; the town opens registration without email or captcha and registers everyone
+else. Objects are read from `post`, `comment` and `private_message` by `ap_id`."""
+import os
+
+from core import podman
+from core.http import HttpError
+from dialects.base import Driver, Made, Session, Stored, Unsupported
+
+
+class LemmyApi(Driver):
+	platform = "lemmy"
+	caps = frozenset({"thread", "comment", "upvote", "downvote", "dm", "delete", "community", "edit", "report",
+					  "block"})
+	db = "lemmy"
+	prefix = "/api/v4"
+
+	def __init__(self, host):
+		super().__init__(host)
+		self._ids = {}
+
+	def update_profile(self, s, account):
+		self.lm(s, "PUT", "account/settings/save", {"display_name": account.name, "bio": account.bio, "bot_account": account.bot})
+
+	def lm(self, s, method, path, body=None, params=None, ok=(200,)):
+		token = s.token if isinstance(s, Session) else s
+		headers = {"Authorization": f"Bearer {token}"} if token else {}
+		r = self.http.request(method, f"{self.base}{self.prefix}/{path}", headers=headers, json=body, params=params,
+							  template=f"{self.prefix}/{path}")
+		if ok and r.status not in ok:
+			raise HttpError(method, path, r.status, r.text)
+		return r.json()
+
+	def admin_token(self):
+		here = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+		with open(os.path.join(here, ".state", "lemmy.token")) as f:
+			return f.read().strip()
+
+	def provision(self, accounts):
+		admin = self.admin_token()
+		self.lm(admin, "PUT", "site", {"registration_mode": "open", "email_verification_required": False,
+									   "captcha_enabled": False})
+		sessions = []
+		for a in accounts:
+			r = self.http.post(f"{self.base}{self.prefix}/account/auth/login",
+							   json={"username_or_email": a.username, "password": a.password})
+			jwt = (r.json() or {}).get("jwt") if r.ok else None
+			if not jwt:
+				jwt = self.lm(None, "POST", "account/auth/register", {
+					"username": a.username, "password": a.password, "password_verify": a.password,
+					"show_nsfw": False})["jwt"]
+			me = self.lm(jwt, "GET", "account")
+			person = (me.get("local_user_view") or me.get("my_user", {}).get("local_user_view") or {}).get("person") or me.get("person")
+			sessions.append(Session(a, jwt, str(person["id"]), person["ap_id"]))
+		return sessions
+
+	def warm(self, s, accts, seconds=120):
+		"""Lemmy starts its send worker for a server at the newest activity and never sends what it queued for that server
+		before then (Accepts and Follows included), so the town makes it meet every server first and waits for the
+		workers. Lemmy only knows a server once it has resolved an account there."""
+		import time
+		hosts = set()
+		for acct in accts:
+			try:
+				self.lookup(s, acct)
+				hosts.add(acct.split("@")[1])
+			except Exception:
+				pass
+		deadline = time.time() + seconds
+		while time.time() < deadline:
+			ready = {r["domain"] for r in podman.psql(self.db, """
+				select i.domain from federation_queue_state q join instance i on i.id = q.instance_id""")}
+			if hosts <= ready:
+				return sorted(hosts)
+			time.sleep(5)
+		return sorted(ready & hosts)
+
+	# -- communities and people
+	def community(self, s, name, title):
+		return self.lm(s, "POST", "community", {"name": name, "title": title})["community_view"]["community"]
+
+	def resolve_community(self, s, ref):
+		"""ref: !name@host"""
+		return self.lm(s, "GET", "resolve_object", params={"q": ref})["community"]["id"]
+
+	def follow_community(self, s, community_id):
+		self.lm(s, "POST", "community/follow", {"community_id": community_id, "follow": True})
+
+	def lookup(self, s, acct):
+		key = (s.token, acct)
+		if key not in self._ids:
+			self._ids[key] = self.lm(s, "GET", "resolve_object", params={"q": f"@{acct}"})["person"]["id"]
+		return self._ids[key]
+
+	def follow(self, s, acct):
+		raise Unsupported(self.platform, "follow people")
+
+	def block(self, s, acct):
+		self.lm(s, "POST", "account/block/person", {"person_id": self.lookup(s, acct), "block": True})
+
+	def unblock(self, s, acct):
+		self.lm(s, "POST", "account/block/person", {"person_id": self.lookup(s, acct), "block": False})
+
+	# -- content
+	def post(self, s, spec):
+		if spec.visibility == "direct":
+			if len(spec.mentions) != 1:
+				raise Unsupported(self.platform, "message several people at once")
+			pm = self.lm(s, "POST", "private_message", {"content": spec.text, "recipient_id": self.lookup(s, spec.mentions[0])})
+			pm = pm.get("private_message_view", pm)["private_message"]
+			return Made(pm["ap_id"], str(pm["id"]))
+		if spec.reply_to_uri:
+			parent = self._find(s, spec.reply_to_uri)
+			body = {"content": spec.text, "post_id": parent["post_id"]}
+			if parent["kind"] == "comment":
+				body["parent_id"] = parent["id"]
+			c = self.lm(s, "POST", "comment", body)["comment_view"]["comment"]
+			return Made(c["ap_id"], f"c{c['id']}")
+		if spec.group is None:
+			raise Unsupported(self.platform, "post outside a community")
+		body = {"name": spec.title or spec.text[:80], "body": spec.text, "community_id": spec.group}
+		if spec.link:
+			body["url"] = spec.link
+		p = self.lm(s, "POST", "post", body)["post_view"]["post"]
+		return Made(p["ap_id"], f"p{p['id']}")
+
+	def _find(self, s, uri):
+		row = self._rows([uri]).get(uri)
+		if row is None:
+			found = self.lm(s, "GET", "resolve_object", params={"q": uri})
+			row = self._rows([uri]).get(uri)
+			if row is None:
+				raise LookupError(f"{self.host} cannot resolve {uri}: {found}")
+		return row.raw
+
+	def edit(self, s, uri, spec):
+		row = self._find(s, uri)
+		if row["kind"] == "comment":
+			self.lm(s, "PUT", "comment", {"comment_id": row["id"], "content": spec.text})
+		else:
+			self.lm(s, "PUT", "post", {"post_id": row["id"], "body": spec.text})
+
+	def delete(self, s, uri):
+		row = self._find(s, uri)
+		# 1.0 deletes with DELETE on the object's own route (0.19 had POST .../delete)
+		if row["kind"] == "comment":
+			self.lm(s, "DELETE", "comment", {"comment_id": row["id"], "deleted": True})
+		elif row["kind"] == "post":
+			self.lm(s, "DELETE", "post", {"post_id": row["id"], "deleted": True})
+		else:
+			self.lm(s, "DELETE", "private_message", {"private_message_id": row["id"], "deleted": True})
+
+	def _vote(self, s, uri, up):
+		row = self._find(s, uri)
+		if row["kind"] == "comment":
+			self.lm(s, "POST", "comment/like", {"comment_id": row["id"], "is_upvote": up})
+		else:
+			self.lm(s, "POST", "post/like", {"post_id": row["id"], "is_upvote": up})
+
+	def like(self, s, uri):
+		self._vote(s, uri, True)
+
+	def downvote(self, s, uri):
+		self._vote(s, uri, False)
+
+	def report(self, s, acct, uris, comment):
+		for uri in uris:
+			row = self._find(s, uri)
+			path = "comment/report" if row["kind"] == "comment" else "post/report"
+			key = "comment_id" if row["kind"] == "comment" else "post_id"
+			self.lm(s, "POST", path, {key: row["id"], "reason": comment})
+
+	# -- reading back
+	def local_status_id(self, s, uri):
+		row = self._rows([uri]).get(uri)
+		return row.local_id if row and not row.deleted else None
+
+	def stored(self, uris):
+		rows = self._rows(uris)
+		return {u: rows.get(u) or Stored(False) for u in uris}
+
+	def seen(self, s, uris):
+		return {u: self.local_status_id(s, u) is not None for u in uris}
+
+	def actor_uri(self, username):
+		return f"{self.base}/u/{username}"
+
+	def _rows(self, uris):
+		if not uris:
+			return {}
+		rows = podman.psql(self.db, """
+			select 'post' as kind, p.id, p.id as post_id, p.ap_id, p.deleted or p.removed as deleted, p.body as text,
+				p.name as title, p.upvotes, p.downvotes, p.comments as replies, p.updated_at is not null as edited,
+				null::text as parent_uri
+			from post p where p.ap_id = any(string_to_array(:'p1', ' '))
+			union all
+			select 'comment', c.id, c.post_id, c.ap_id, c.deleted or c.removed, c.content, null, c.upvotes, c.downvotes,
+				c.child_count, c.updated_at is not null,
+				coalesce((select pc.ap_id from comment pc where pc.id::text = split_part(c.path::text, '.', nlevel(c.path) - 1)
+					and nlevel(c.path) > 2), (select pp.ap_id from post pp where pp.id = c.post_id))
+			from comment c where c.ap_id = any(string_to_array(:'p1', ' '))
+			union all
+			select 'pm', m.id, null, m.ap_id, m.deleted or m.removed, m.content, null, 0, 0, 0, m.updated_at is not null, null
+			from private_message m where m.ap_id = any(string_to_array(:'p1', ' '))""", " ".join(uris))
+		out = {}
+		for r in rows:
+			prefix = {"post": "p", "comment": "c", "pm": "m"}[r["kind"]]
+			out[r["ap_id"]] = Stored(True, bool(r["deleted"]), f"{prefix}{r['id']}", "public" if r["kind"] != "pm" else "direct",
+									 r["text"], None, bool(r["edited"]), r["parent_uri"], r["upvotes"], None, r["replies"],
+									 None, {"down": r["downvotes"]}, r)
+		return out

tools/pasture/town/dialects/mastodon.py

@@ -0,0 +1,57 @@
+"""Mastodon 4.7: accounts and tokens made in one `rails runner` (it has no password grant, and tootctl boots Rails once
+per account), objects read from its Postgres database (`mastodon`)."""
+import json
+
+from core import podman
+from dialects.base import Stored
+from dialects.mastodon_api import MastodonApi
+
+VIS = {0: "public", 1: "unlisted", 2: "followers", 3: "direct", 4: "limited"}
+
+PROVISION = r'''
+app = Doorkeeper::Application.find_or_create_by!(name: "pasture-town") { |a| a.redirect_uri = "urn:ietf:wg:oauth:2.0:oob"; a.scopes = "read write follow" }
+out = {}
+JSON.parse(STDIN.read).each do |u|
+  account = Account.find_local(u["username"])
+  if account.nil?
+    account = Account.new(username: u["username"])
+    user = User.new(email: "#{u["username"]}@mastodon.test", password: u["password"], agreement: true, approved: true,
+                    confirmed_at: Time.now.utc, account: account)
+    user.save!
+    account.reload
+  end
+  account.user.approve! unless account.user.approved?
+  account.user.confirm unless account.user.confirmed?
+  token = Doorkeeper::AccessToken.find_or_create_for(application: app, resource_owner: account.user, scopes: Doorkeeper::OAuth::Scopes.from_string("read write follow"), expires_in: nil, use_refresh_token: false)
+  out[u["username"]] = token.token
+end
+puts "TOWN" + out.to_json
+'''
+
+
+class Mastodon(MastodonApi):
+	platform = "mastodon"
+	caps = MastodonApi.caps | {"quote"}
+
+	def provision(self, accounts):
+		payload = json.dumps([{"username": a.username, "password": a.password} for a in accounts])
+		out = podman.exec_("pasture-mastodon", "bin/rails", "runner", PROVISION, input=payload, timeout=600)
+		tokens = json.loads(next(line[4:] for line in out.splitlines() if line.startswith("TOWN")))
+		return [self.session_from_token(a, tokens[a.username]) for a in accounts]
+
+	def _rows(self, uris):
+		if not uris:
+			return {}
+		rows = podman.psql("mastodon", """
+			select s.id::text as local_id, s.uri, s.deleted_at is not null as deleted, s.visibility, s.text,
+				s.spoiler_text as cw, s.edited_at is not null as edited, p.uri as parent_uri,
+				coalesce(st.favourites_count, 0) as likes, coalesce(st.reblogs_count, 0) as boosts,
+				coalesce(st.replies_count, 0) as replies, pl.cached_tallies as votes
+			from statuses s
+			left join statuses p on p.id = s.in_reply_to_id
+			left join status_stats st on st.status_id = s.id
+			left join polls pl on pl.status_id = s.id
+			where s.reblog_of_id is null and s.uri = any(string_to_array(:'p1', ' '))""", " ".join(uris))
+		return {r["uri"]: Stored(True, r["deleted"], r["local_id"], VIS.get(r["visibility"]), r["text"], r["cw"] or None,
+								 r["edited"], r["parent_uri"], r["likes"], r["boosts"], r["replies"], r["votes"], None, r)
+				for r in rows}

tools/pasture/town/dialects/mastodon_api.py

@@ -0,0 +1,252 @@
+"""Everything that speaks the Mastodon client API: Mastodon, GoToSocial, Akkoma, and later Pleroma, Iceshrimp.NET, Hollo,
+snac, Friendica, Pixelfed, Mitra. Subclasses provide accounts and tokens (each platform makes them its own way) and
+`_rows(uris)`, their database's view of a set of objects."""
+import time
+
+from core import media
+from core.http import HttpError
+from dialects.base import Driver, Made, Session, Stored, Unsupported
+
+VISIBILITY = {"public": "public", "unlisted": "unlisted", "followers": "private", "direct": "direct"}
+
+
+class MastodonApi(Driver):
+	platform = "mastodon-api"
+	caps = frozenset({"post", "reply", "cw", "media", "poll", "like", "boost", "bookmark", "follow", "block", "mute",
+					  "report", "dm", "delete", "edit", "vote", "profile"})
+	media_endpoint = "/api/v2/media"
+	max_media = 4
+
+	def __init__(self, host):
+		super().__init__(host)
+		self._ids = {}  # (session token, acct) -> local account id
+
+	# -- plumbing
+	def api(self, s, method, path, ok=None, **kw):
+		headers = kw.pop("headers", {})
+		if s is not None:
+			headers["Authorization"] = f"Bearer {s.token}"
+		return self.http.request(method, self.base + path, headers=headers, ok=ok, template=path.split("?")[0], **kw)
+
+	def api_json(self, s, method, path, **kw):
+		return self.api(s, method, path, ok={200, 201, 202}, **kw).json()
+
+	def session_from_token(self, account, token):
+		me = self.api_json(None, "GET", "/api/v1/accounts/verify_credentials", headers={"Authorization": f"Bearer {token}"})
+		actor = self.actor_uri(account.username) or me.get("url")
+		return Session(account, token, me["id"], actor)
+
+	# -- profiles
+	def update_profile(self, s, account):
+		form = {"display_name": account.name, "note": account.bio, "locked": account.locked, "bot": account.bot}
+		for i, (k, v) in enumerate(account.fields[:4]):
+			form[f"fields_attributes[{i}][name]"] = k
+			form[f"fields_attributes[{i}][value]"] = v
+		files = {}
+		if account.avatar_seed:
+			files["avatar"] = media.upload("image", account.avatar_seed)
+		if account.header_seed:
+			files["header"] = media.upload("image", account.header_seed)
+		self.api(s, "PATCH", "/api/v1/accounts/update_credentials", ok={200}, form=form, files=files or None)
+
+	# -- accounts as this server knows them
+	def lookup(self, s, acct):
+		key = (s.token, acct)
+		if key in self._ids:
+			return self._ids[key]
+		found = None
+		r = self.api(s, "GET", "/api/v1/accounts/lookup", params={"acct": acct})
+		if r.ok and r.json():
+			found = r.json()["id"]
+		if found is None:
+			r = self.api(s, "GET", "/api/v2/search", params={"q": f"@{acct}", "resolve": "true", "type": "accounts"})
+			for a in (r.json() or {}).get("accounts", []) if r.ok else []:
+				if _same_acct(a.get("acct"), acct, self.host):
+					found = a["id"]
+					break
+		if found is None:
+			raise LookupError(f"{self.host} cannot find {acct}")
+		self._ids[key] = found
+		return found
+
+	def follow(self, s, acct):
+		rel = self.api_json(s, "POST", f"/api/v1/accounts/{self.lookup(s, acct)}/follow")
+		return "accepted" if rel.get("following") else "requested"
+
+	def unfollow(self, s, acct):
+		self.api(s, "POST", f"/api/v1/accounts/{self.lookup(s, acct)}/unfollow", ok={200})
+
+	def pending(self, s):
+		r = self.api(s, "GET", "/api/v1/follow_requests", params={"limit": 80}, ok={200})
+		return [_full_acct(a["acct"], self.host) for a in r.json() or []]
+
+	def accept(self, s, acct):
+		self.api(s, "POST", f"/api/v1/follow_requests/{self.lookup(s, acct)}/authorize", ok={200})
+
+	def reject(self, s, acct):
+		self.api(s, "POST", f"/api/v1/follow_requests/{self.lookup(s, acct)}/reject", ok={200})
+
+	def relationship(self, s, acct):
+		rels = self.api_json(s, "GET", "/api/v1/accounts/relationships", params={"id": [self.lookup(s, acct)]})
+		r = rels[0] if rels else {}
+		return {k: bool(r.get(k)) for k in ("following", "followed_by", "requested", "blocking", "muting", "blocked_by")}
+
+	def block(self, s, acct):
+		self.api(s, "POST", f"/api/v1/accounts/{self.lookup(s, acct)}/block", ok={200})
+
+	def unblock(self, s, acct):
+		self.api(s, "POST", f"/api/v1/accounts/{self.lookup(s, acct)}/unblock", ok={200})
+
+	def mute(self, s, acct):
+		self.api(s, "POST", f"/api/v1/accounts/{self.lookup(s, acct)}/mute", ok={200})
+
+	def report(self, s, acct, uris, comment):
+		ids = [i for i in (self.local_status_id(s, u) for u in uris) if i]
+		self.api(s, "POST", "/api/v1/reports", ok={200}, form={"account_id": self.lookup(s, acct), "status_ids": ids,
+																 "comment": comment, "forward": True})
+
+	# -- content
+	def upload(self, s, item):
+		filename, content, ctype = media.upload(item["kind"], item["seed"])
+		r = self.api(s, "POST", self.media_endpoint, files={"file": (filename, content, ctype)},
+					 form={"description": item.get("alt")})
+		if r.status == 404 and self.media_endpoint != "/api/v1/media":
+			r = self.api(s, "POST", "/api/v1/media", files={"file": (filename, content, ctype)},
+						 form={"description": item.get("alt")})
+		if not r.ok:
+			raise HttpError("POST", self.media_endpoint, r.status, r.text)
+		attachment = r.json()
+		for _ in range(60):
+			if attachment.get("url"):
+				break
+			time.sleep(1)
+			attachment = self.api_json(s, "GET", f"/api/v1/media/{attachment['id']}")
+		return attachment["id"]
+
+	def status_form(self, s, spec):
+		text = spec.text
+		for acct in spec.mentions:
+			if f"@{acct}" not in text:
+				text = f"@{acct} {text}"
+		for tag in spec.tags:
+			if f"#{tag}" not in text:
+				text = f"{text} #{tag}"
+		visibility = VISIBILITY.get(spec.visibility)
+		if visibility is None:
+			raise Unsupported(self.platform, f"post with visibility {spec.visibility}")
+		form = {"status": text, "visibility": visibility, "language": spec.language}
+		if spec.cw:
+			form["spoiler_text"] = spec.cw
+			form["sensitive"] = True
+		if spec.media:
+			form["media_ids"] = [self.upload(s, m) for m in spec.media[:self.max_media]]
+			if any(m.get("sensitive") for m in spec.media):
+				form["sensitive"] = True
+		if spec.poll:
+			form["poll[options]"] = spec.poll["options"]
+			form["poll[expires_in]"] = spec.poll.get("expires_in", 86400)
+			form["poll[multiple]"] = bool(spec.poll.get("multiple"))
+		if spec.reply_to_uri:
+			form["in_reply_to_id"] = self.local_status_id(s, spec.reply_to_uri) or self.resolve(s, spec.reply_to_uri)
+		if spec.quote_uri:
+			if "quote" not in self.caps:
+				raise Unsupported(self.platform, "quote")
+			form["quoted_status_id"] = self.local_status_id(s, spec.quote_uri) or self.resolve(s, spec.quote_uri)
+		return form
+
+	def post(self, s, spec):
+		if spec.kind not in ("note", "image", "video", "audio"):
+			raise Unsupported(self.platform, f"post a {spec.kind}")
+		status = self.api_json(s, "POST", "/api/v1/statuses", form=self.status_form(s, spec))
+		return Made(status["uri"], status["id"], status.get("url"))
+
+	def edit(self, s, uri, spec):
+		sid = self._own(s, uri)
+		text = spec.text
+		for acct in spec.mentions:
+			if f"@{acct}" not in text:
+				text = f"@{acct} {text}"
+		form = {"status": text}
+		if spec.cw:
+			form["spoiler_text"] = spec.cw
+		self.api(s, "PUT", f"/api/v1/statuses/{sid}", ok={200}, form=form)
+
+	def delete(self, s, uri):
+		self.api(s, "DELETE", f"/api/v1/statuses/{self._own(s, uri)}", ok={200})
+
+	def _act(self, s, uri, action):
+		sid = self.local_status_id(s, uri) or self.resolve(s, uri)
+		self.api(s, "POST", f"/api/v1/statuses/{sid}/{action}", ok={200})
+
+	def like(self, s, uri):
+		self._act(s, uri, "favourite")
+
+	def unlike(self, s, uri):
+		self._act(s, uri, "unfavourite")
+
+	def boost(self, s, uri):
+		self._act(s, uri, "reblog")
+
+	def unboost(self, s, uri):
+		self._act(s, uri, "unreblog")
+
+	def bookmark(self, s, uri):
+		self._act(s, uri, "bookmark")
+
+	def vote(self, s, uri, choices):
+		sid = self.local_status_id(s, uri) or self.resolve(s, uri)
+		status = self.api_json(s, "GET", f"/api/v1/statuses/{sid}")
+		poll = status.get("poll")
+		if not poll:
+			raise LookupError(f"{self.host} shows no poll on {uri}")
+		self.api(s, "POST", f"/api/v1/polls/{poll['id']}/votes", ok={200}, form={"choices": choices})
+
+	def _own(self, s, uri):
+		sid = self.local_status_id(s, uri)
+		if sid is None:
+			raise LookupError(f"{self.host} does not hold {uri}")
+		return sid
+
+	# -- reading back
+	def resolve(self, s, uri):
+		r = self.api(s, "GET", "/api/v2/search", params={"q": uri, "resolve": "true", "type": "statuses"})
+		for st in (r.json() or {}).get("statuses", []) if r.ok else []:
+			if st.get("uri") == uri or st.get("url") == uri:
+				return st["id"]
+		raise LookupError(f"{self.host} cannot resolve {uri}")
+
+	def local_status_id(self, s, uri):
+		row = self._rows([uri]).get(uri)
+		return row.local_id if row and row.exists and not row.deleted else None
+
+	def stored(self, uris):
+		rows = self._rows(uris)
+		return {u: rows.get(u) or Stored(False) for u in uris}
+
+	def visible(self, s, local_id):
+		return self.api(s, "GET", f"/api/v1/statuses/{local_id}").status == 200
+
+	def status(self, s, local_id):
+		r = self.api(s, "GET", f"/api/v1/statuses/{local_id}")
+		return r.json() if r.ok else None
+
+	def notifications(self, s):
+		r = self.api(s, "GET", "/api/v1/notifications", params={"limit": 80}, ok={200})
+		out = []
+		for n in r.json() or []:
+			out.append({"type": n["type"], "acct": _full_acct((n.get("account") or {}).get("acct", ""), self.host),
+						"uri": (n.get("status") or {}).get("uri")})
+		return out
+
+	def _rows(self, uris):
+		raise Unsupported(self.platform, "read its database")
+
+
+def _full_acct(acct, host):
+	return acct if "@" in acct else f"{acct}@{host}"
+
+
+def _same_acct(found, wanted, host):
+	if not found:
+		return False
+	return _full_acct(found, host).lower() == wanted.lower()

tools/pasture/town/dialects/misskey_api.py

@@ -0,0 +1,273 @@
+"""Misskey and its forks (Sharkey, later CherryPick, Iceshrimp's Misskey API): every call is POST /api/<endpoint> with
+the token as `i`. Accounts are made by the admin made at setup (`admin/accounts/create`); objects are read from the
+`note` table. A local note has no `uri` there: its id is the last part of https://<host>/notes/<id>."""
+from core import media, podman
+from core.http import HttpError
+from dialects.base import Driver, Made, Session, Stored, Unsupported
+import os
+
+VISIBILITY = {"public": "public", "unlisted": "home", "followers": "followers", "direct": "specified"}
+VIS = {"public": "public", "home": "unlisted", "followers": "followers", "specified": "direct"}
+LIKE = "❤"
+
+
+class MisskeyApi(Driver):
+	platform = "misskey"
+	caps = frozenset({"post", "reply", "cw", "media", "poll", "like", "react", "boost", "bookmark", "follow", "block",
+					  "mute", "report", "dm", "delete", "vote", "quote", "profile"})
+	db = "misskey"
+
+	def __init__(self, host, db=None):
+		super().__init__(host)
+		self.db = db or host.split(".")[0]
+		self._ids = {}
+
+	def mk(self, s_or_token, endpoint, body=None, ok=None):
+		body = dict(body or {})
+		token = s_or_token.token if isinstance(s_or_token, Session) else s_or_token
+		if token:
+			body["i"] = token
+		r = self.http.post(f"{self.base}/api/{endpoint}", json=body, template=f"/api/{endpoint}")
+		if ok is not False and r.status not in (200, 204):
+			raise HttpError("POST", endpoint, r.status, r.text)
+		return r.json() if r.body else None
+
+	def admin_token(self):
+		here = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+		with open(os.path.join(here, ".state", f"{self.db}.token")) as f:
+			return f.read().strip()
+
+	def provision(self, accounts):
+		admin = self.admin_token()
+		sessions = []
+		existing = {r["username"] for r in podman.psql(self.db, 'select username from "user" where host is null')}
+		for a in accounts:
+			if a.username in existing:
+				r = self.http.post(f"{self.base}/api/signin-flow", json={"username": a.username, "password": a.password})
+				token = (r.json() or {}).get("i")
+				if not token:
+					raise RuntimeError(f"{self.host}: cannot sign {a.username} in ({r.status} {r.text[:200]})")
+			else:
+				token = self.mk(admin, "admin/accounts/create", {"username": a.username, "password": a.password})["token"]
+			me = self.mk(token, "i")
+			sessions.append(Session(a, token, me["id"], f"{self.base}/users/{me['id']}"))
+		return sessions
+
+	def update_profile(self, s, account):
+		body = {"name": account.name, "description": account.bio, "isLocked": account.locked, "isBot": account.bot,
+				"fields": [{"name": k, "value": v} for k, v in account.fields[:4]]}
+		if account.avatar_seed:
+			body["avatarId"] = self._upload(s, {"kind": "image", "seed": account.avatar_seed, "alt": None})
+		if account.header_seed:
+			body["bannerId"] = self._upload(s, {"kind": "image", "seed": account.header_seed, "alt": None})
+		self.mk(s, "i/update", body)
+
+	def _upload(self, s, item):
+		filename, content, ctype = media.upload(item["kind"], item["seed"])
+		form = {"i": s.token, "isSensitive": bool(item.get("sensitive"))}
+		if item.get("alt"):
+			form["comment"] = item["alt"]
+		r = self.http.post(f"{self.base}/api/drive/files/create", files={"file": (filename, content, ctype)}, form=form, ok={200})
+		return r.json()["id"]
+
+	# -- accounts
+	def lookup(self, s, acct):
+		key = (s.token, acct)
+		if key not in self._ids:
+			name, _, host = acct.partition("@")
+			body = {"username": name}
+			if host and host != self.host:
+				body["host"] = host
+			self._ids[key] = self.mk(s, "users/show", body)["id"]
+		return self._ids[key]
+
+	def follow(self, s, acct):
+		self.mk(s, "following/create", {"userId": self.lookup(s, acct)}, ok=False)
+		rel = self.relationship(s, acct)
+		return "accepted" if rel["following"] else "requested"
+
+	def unfollow(self, s, acct):
+		self.mk(s, "following/delete", {"userId": self.lookup(s, acct)})
+
+	def pending(self, s):
+		out = []
+		for req in self.mk(s, "following/requests/list", {"limit": 100}) or []:
+			u = req["follower"]
+			out.append(f"{u['username']}@{u.get('host') or self.host}")
+		return out
+
+	def accept(self, s, acct):
+		self.mk(s, "following/requests/accept", {"userId": self.lookup(s, acct)})
+
+	def reject(self, s, acct):
+		self.mk(s, "following/requests/reject", {"userId": self.lookup(s, acct)})
+
+	def relationship(self, s, acct):
+		r = self.mk(s, "users/relation", {"userId": self.lookup(s, acct)})
+		r = r[0] if isinstance(r, list) else r
+		return {"following": r.get("isFollowing"), "followed_by": r.get("isFollowed"),
+				"requested": r.get("hasPendingFollowRequestFromYou"), "blocking": r.get("isBlocking"),
+				"muting": r.get("isMuted"), "blocked_by": r.get("isBlocked")}
+
+	def block(self, s, acct):
+		self.mk(s, "blocking/create", {"userId": self.lookup(s, acct)})
+
+	def unblock(self, s, acct):
+		self.mk(s, "blocking/delete", {"userId": self.lookup(s, acct)})
+
+	def mute(self, s, acct):
+		self.mk(s, "mute/create", {"userId": self.lookup(s, acct)})
+
+	def report(self, s, acct, uris, comment):
+		self.mk(s, "users/report-abuse", {"userId": self.lookup(s, acct), "comment": comment})
+
+	# -- content
+	def post(self, s, spec):
+		if spec.kind not in ("note", "image", "video", "audio"):
+			raise Unsupported(self.platform, f"post a {spec.kind}")
+		visibility = VISIBILITY.get(spec.visibility)
+		if visibility is None:
+			raise Unsupported(self.platform, f"post with visibility {spec.visibility}")
+		text = spec.text
+		for acct in spec.mentions:
+			if f"@{acct}" not in text:
+				text = f"@{acct} {text}"
+		for tag in spec.tags:
+			if f"#{tag}" not in text:
+				text = f"{text} #{tag}"
+		body = {"text": text, "visibility": visibility}
+		if spec.cw:
+			body["cw"] = spec.cw
+		if visibility == "specified":
+			body["visibleUserIds"] = [self.lookup(s, a) for a in spec.mentions]
+		if spec.media:
+			body["fileIds"] = [self._upload(s, m) for m in spec.media[:16]]
+		if spec.poll:
+			body["poll"] = {"choices": spec.poll["options"], "multiple": bool(spec.poll.get("multiple")),
+							"expiredAfter": spec.poll.get("expires_in", 86400) * 1000}
+		if spec.reply_to_uri:
+			body["replyId"] = self.local_status_id(s, spec.reply_to_uri) or self.resolve(s, spec.reply_to_uri)
+		if spec.quote_uri:
+			body["renoteId"] = self.local_status_id(s, spec.quote_uri) or self.resolve(s, spec.quote_uri)
+		note = self.mk(s, "notes/create", body)["createdNote"]
+		return Made(note.get("uri") or f"{self.base}/notes/{note['id']}", note["id"], note.get("url"))
+
+	def edit(self, s, uri, spec):
+		if "edit" not in self.caps:
+			raise Unsupported(self.platform, "edit")
+		text = spec.text
+		for acct in spec.mentions:
+			if f"@{acct}" not in text:
+				text = f"@{acct} {text}"
+		body = {"editId": self._own(uri), "text": text, "cw": spec.cw}
+		self.mk(s, "notes/edit", body)
+
+	def delete(self, s, uri):
+		self.mk(s, "notes/delete", {"noteId": self._own(uri)})
+
+	def _note(self, s, uri):
+		return self.local_status_id(s, uri) or self.resolve(s, uri)
+
+	def like(self, s, uri):
+		self.mk(s, "notes/reactions/create", {"noteId": self._note(s, uri), "reaction": LIKE})
+
+	def unlike(self, s, uri):
+		self.mk(s, "notes/reactions/delete", {"noteId": self._note(s, uri)})
+
+	def react(self, s, uri, emoji):
+		self.mk(s, "notes/reactions/create", {"noteId": self._note(s, uri), "reaction": emoji})
+
+	def boost(self, s, uri):
+		self.mk(s, "notes/create", {"renoteId": self._note(s, uri), "visibility": "public"})
+
+	def unboost(self, s, uri):
+		self.mk(s, "notes/unrenote", {"noteId": self._note(s, uri)})
+
+	def bookmark(self, s, uri):
+		self.mk(s, "notes/favorites/create", {"noteId": self._note(s, uri)})
+
+	def vote(self, s, uri, choices):
+		nid = self._note(s, uri)
+		for c in choices:
+			self.mk(s, "notes/polls/vote", {"noteId": nid, "choice": c})
+
+	def _own(self, uri):
+		nid = self.local_status_id(None, uri)
+		if nid is None:
+			raise LookupError(f"{self.host} does not hold {uri}")
+		return nid
+
+	# -- reading back
+	def resolve(self, s, uri):
+		r = self.mk(s, "ap/show", {"uri": uri}, ok=False)
+		if r and r.get("type") == "Note":
+			return r["object"]["id"]
+		raise LookupError(f"{self.host} cannot resolve {uri}")
+
+	def local_status_id(self, s, uri):
+		row = self._rows([uri]).get(uri)
+		return row.local_id if row else None
+
+	def stored(self, uris):
+		rows = self._rows(uris)
+		return {u: rows.get(u) or Stored(False) for u in uris}
+
+	def seen(self, s, uris):
+		out = {}
+		for uri in uris:
+			nid = self.local_status_id(s, uri)
+			if nid is None:
+				out[uri] = False
+				continue
+			r = self.http.post(f"{self.base}/api/notes/show", json={"i": s.token, "noteId": nid})
+			note = r.json() if r.ok else None
+			out[uri] = bool(note) and not note.get("isHidden")
+		return out
+
+	def notifications(self, s):
+		kinds = {"reaction": "favourite", "renote": "reblog", "reply": "mention", "mention": "mention",
+				 "follow": "follow", "receiveFollowRequest": "follow_request", "quote": "quote", "pollEnded": "poll"}
+		out = []
+		for n in self.mk(s, "i/notifications", {"limit": 100}) or []:
+			u = n.get("user") or {}
+			note = n.get("note") or {}
+			out.append({"type": kinds.get(n["type"], n["type"]), "acct": f"{u.get('username')}@{u.get('host') or self.host}",
+						"uri": note.get("uri") or (f"{self.base}/notes/{note['id']}" if note.get("id") else None)})
+		return out
+
+	def actor_uri(self, username):
+		row = podman.psql(self.db, 'select id from "user" where host is null and username = :\'p1\'', username)
+		return f"{self.base}/users/{row[0]['id']}" if row else None
+
+	def _rows(self, uris):
+		if not uris:
+			return {}
+		local = {u: u.rsplit("/", 1)[1] for u in uris if u.startswith(f"{self.base}/notes/")}
+		edited = "n.\"updatedAt\" is not null" if self.platform == "sharkey" else "false"
+		rows = podman.psql(self.db, f"""
+			select n.id, n.uri, n.visibility, n.text, n.cw, {edited} as edited, n."renoteCount" as boosts,
+				n."repliesCount" as replies, n.reactions, p.votes, coalesce(r.uri, case when r.id is not null
+					then 'https://{self.host}/notes/' || r.id end) as parent_uri
+			from note n left join poll p on p."noteId" = n.id left join note r on r.id = n."replyId"
+			where (n."renoteId" is null or n.text is not null)
+				and (n.uri = any(string_to_array(:'p1', ' ')) or n.id = any(string_to_array(:'p2', ' ')))""",
+						   " ".join(uris), " ".join(local.values()) or "-")
+		out = {}
+		for r in rows:
+			uri = r["uri"] or f"{self.base}/notes/{r['id']}"
+			if uri not in uris:
+				continue
+			reactions = r["reactions"] or {}
+			out[uri] = Stored(True, False, r["id"], VIS.get(r["visibility"], r["visibility"]), r["text"], r["cw"],
+							  bool(r["edited"]), r["parent_uri"], sum(reactions.values()), r["boosts"], r["replies"],
+							  r["votes"], reactions, r)
+		return out
+
+
+class Misskey(MisskeyApi):
+	platform = "misskey"
+
+
+class Sharkey(MisskeyApi):
+	platform = "sharkey"
+	caps = MisskeyApi.caps | {"edit"}

tools/pasture/town/dialects/privapub.py

@@ -0,0 +1,133 @@
+"""PrivaPub: roots signed up on /clientapi, personas made under them, and each persona's Mastodon token exchanged for the
+root's JWT exactly as decePubClient does (RFC 8693, client `decepub`). Everything a Mastodon client can say goes
+through the Mastodon API; circles, communities and located posts through /clientapi. Objects are read from Mongo."""
+from core import podman
+from dialects.base import Made, Session, Stored, Unsupported
+from dialects.mastodon_api import MastodonApi
+
+VIS = {0: "public", 1: "unlisted", 2: "followers", 3: "direct", 4: "circle", 5: "located",
+	   "Public": "public", "Unlisted": "unlisted", "FollowersOnly": "followers", "Direct": "direct", "Circle": "circle",
+	   "LocalGeo": "located"}
+
+
+class PrivaPub(MastodonApi):
+	platform = "privapub"
+	caps = MastodonApi.caps | {"react", "quote", "circle", "community", "located"}
+
+	def __init__(self, host="privapub.test"):
+		super().__init__(host)
+		self._jwts = {}
+
+	# -- roots and personas
+	def jwt(self, root, password):
+		if root in self._jwts:
+			return self._jwts[root]
+		body = {"userName": root, "password": password}
+		r = self.http.post(self.base + "/clientapi/user/signup", json=body)
+		token = (r.json() or {}).get("token") if r.ok else None
+		if not token:
+			token = self.http.post(self.base + "/clientapi/user/login", json=body, ok={200}).json()["token"]
+		self._jwts[root] = token
+		return token
+
+	def personas(self, root, password):
+		jwt = self.jwt(root, password)
+		r = self.http.get(self.base + "/clientapi/avatar/private/list", headers={"Authorization": f"Bearer {jwt}"}, ok={200})
+		return {p["userName"]: p for p in r.json() or []}
+
+	def provision(self, accounts):
+		"""Each account is a persona; `account.root` names its root, whose password is `account.password`."""
+		sessions = []
+		for root in dict.fromkeys(a.root for a in accounts):
+			mine = [a for a in accounts if a.root == root]
+			password = mine[0].password
+			jwt = self.jwt(root, password)
+			existing = self.personas(root, password)
+			for a in mine:
+				if a.username not in existing:
+					self.http.post(self.base + "/clientapi/avatar/private/insert", ok={200, 201},
+								   headers={"Authorization": f"Bearer {jwt}"},
+								   json={"userName": a.username, "name": a.name or a.username, "biography": a.bio or a.name or a.username,
+										 "fields": dict(a.fields)})
+			existing = self.personas(root, password)
+			for a in mine:
+				token = self.exchange(jwt, existing[a.username]["id"])
+				sessions.append(self.session_from_token(a, token))
+		by_name = {s.account.username: s for s in sessions}
+		return [by_name[a.username] for a in accounts]
+
+	def exchange(self, jwt, avatar_id):
+		return self.http.post(self.base + "/oauth/token", ok={200}, form={
+			"grant_type": "urn:ietf:params:oauth:grant-type:token-exchange", "client_id": "decepub",
+			"subject_token": jwt, "subject_token_type": "urn:ietf:params:oauth:token-type:jwt",
+			"avatar_id": avatar_id, "scope": "read write follow"}).json()["access_token"]
+
+	def actor_uri(self, username):
+		return f"https://{self.host}/peasants/{username}"
+
+	# -- groups: communities (FEP-1b12) and circles
+	def group(self, s, username, name, community, policy="followers", approve=False):
+		jwt = self._jwts[s.account.root]
+		r = self.http.post(self.base + "/clientapi/group/insert", headers={"Authorization": f"Bearer {jwt}"}, ok={200, 201},
+						   json={"avatarId": s.local_id, "userName": username, "name": name, "description": name,
+								 "isCommunity": community, "postingPolicy": policy, "isDiscoverable": community,
+								 "manuallyApprovesMembers": approve})
+		return r.json()
+
+	# -- content
+	def post(self, s, spec):
+		if spec.visibility in ("circle", "community", "located"):
+			return self._client_post(s, spec)
+		return super().post(s, spec)
+
+	def _client_post(self, s, spec):
+		jwt = self._jwts[s.account.root]
+		body = {"avatarId": s.local_id, "text": spec.text, "title": spec.title, "hasContentWarning": bool(spec.cw),
+				"spoilerText": spec.cw}
+		if spec.visibility == "located":
+			loc = spec.location
+			body.update({"latitude": loc["lat"], "longitude": loc["lng"], "rangeKm": loc.get("range_km", 5)})
+		else:
+			body["groupId"] = spec.group
+		if spec.reply_to_uri:
+			body["answeringToPostId"] = self.local_status_id(s, spec.reply_to_uri)
+		r = self.http.post(self.base + "/clientapi/post/insert", headers={"Authorization": f"Bearer {jwt}"},
+						   ok={200, 201}, json=body).json()
+		post_id = r.get("id") if isinstance(r, dict) else None
+		if not post_id:
+			raise RuntimeError(f"/clientapi/post/insert answered {r!r}")
+		row = podman.mongo(f"db.Post.findOne({{_id: ObjectId('{post_id}')}}, {{ObjectURI: 1}})")
+		return Made(row["ObjectURI"], post_id, None)
+
+	def react(self, s, uri, emoji):
+		import urllib.parse
+		sid = self.local_status_id(s, uri) or self.resolve(s, uri)
+		self.api(s, "PUT", f"/api/v1/pleroma/statuses/{sid}/reactions/{urllib.parse.quote(emoji)}", ok={200})
+
+	# -- reading back
+	def _rows(self, uris):
+		if not uris:
+			return {}
+		import json
+		docs = podman.mongo(f"db.Post.find({{ObjectURI: {{$in: {json.dumps(list(uris))}}}}}, "
+							"{ObjectURI: 1, Visibility: 1, Text: 1, ContentHtml: 1, SpoilerText: 1, EditedAt: 1, "
+							"InReplyToURI: 1, FavouritesCount: 1, ReblogsCount: 1, RepliesCount: 1, DownvotesCount: 1, "
+							"Poll: 1, DeletedAt: 1, ReblogOfPostId: 1, AuthorGone: 1, GroupId: 1, ConversationId: 1}).toArray()") or []
+		ids = [d["_id"]["$oid"] if isinstance(d["_id"], dict) else d["_id"] for d in docs]
+		reactions = {}
+		if ids:
+			for g in podman.mongo(f"db.Reaction.aggregate([{{$match: {{PostId: {{$in: {json.dumps(ids)}}}}}}}, "
+								  "{$group: {_id: {p: '$PostId', e: '$Emoji'}, n: {$sum: 1}}}]).toArray()") or []:
+				reactions.setdefault(g["_id"]["p"], {})[g["_id"]["e"]] = g["n"]
+		out = {}
+		for d in docs:
+			if d.get("ReblogOfPostId"):
+				continue
+			pid = d["_id"]["$oid"] if isinstance(d["_id"], dict) else d["_id"]
+			poll = d.get("Poll") or {}
+			votes = [o.get("Votes", o.get("VotesCount", 0)) for o in poll.get("Options", [])] if poll else None
+			out[d["ObjectURI"]] = Stored(True, bool(d.get("DeletedAt")), pid, VIS.get(d.get("Visibility"), str(d.get("Visibility"))),
+										 d.get("Text") or d.get("ContentHtml"), d.get("SpoilerText") or None,
+										 bool(d.get("EditedAt")), d.get("InReplyToURI"), d.get("FavouritesCount", 0),
+										 d.get("ReblogsCount", 0), d.get("RepliesCount", 0), votes, reactions.get(pid, {}), d)
+		return out

tools/pasture/town/expect.py

@@ -0,0 +1,63 @@
+"""What the town's objects should look like to a reader, for decePub's e2e tests: for every object PrivaPub holds and some
+town persona may see, its ref, URI, PrivaPub id, origin, kind, visibility, the text marker, content warning, media alt
+texts, poll options, quoted and parent URIs, and the personas that may see it.
+
+  town.sh expect [run] [--persona <username>]"""
+import json
+import sys
+
+import check
+import seed
+from dialects import driver
+
+
+def edited_cw(world, ref, args):
+	"""The warning an object shows now: an edit made before plans kept warnings replaced the post without one."""
+	edit = next((s for s in world.planner.steps if s["verb"] == "edit" and s["args"]["target"] == ref), None)
+	if edit is None or ref not in world.edited:
+		return args.get("cw")
+	return edit["args"].get("cw")
+
+
+def objects(world):
+	steps = {s["ref"]: s for s in world.planner.steps if s.get("ref") and s["verb"] == "post"}
+	held = driver("privapub").stored([o["uri"] for o in world.objects.values()])
+	personas = [k for k in world.planner.accounts if k.startswith("privapub/")]
+	out = []
+	for ref, o in sorted(world.objects.items(), key=lambda kv: int(kv[0][1:])):
+		row = held.get(o["uri"])
+		if not row or not row.exists or row.deleted or ref in world.deleted:
+			continue
+		recipients = world.recipients(o)
+		readers = [p.split("/")[1] for p in personas
+				   if p == o["author"] or p in recipients or o["visibility"] in ("public", "unlisted", "community")]
+		if not readers:
+			continue
+		args = steps[ref]["args"]
+		out.append({
+			"ref": ref, "uri": o["uri"], "id": row.local_id, "origin": o["origin"], "author": o["author"],
+			"kind": o["kind"], "visibility": o["visibility"], "marker": f"({ref})", "cw": edited_cw(world, ref, args),
+			"alts": [m["alt"] for m in args.get("media", [])], "poll": (args.get("poll") or {}).get("options"),
+			"quote": world.objects.get(o.get("quote") or "", {}).get("uri"),
+			"parent": world.objects.get(o.get("parent") or "", {}).get("uri"),
+			"edited": ref in world.edited, "title": args.get("title"), "readers": readers,
+			"followers_of_author": sorted(p.split("/")[1] for p in world.follows.get(o["author"], set()) if p.startswith("privapub/")),
+		})
+	return out
+
+
+def main(args):
+	name = next((a for a in args if not a.startswith("--") and not a.startswith("privapub/")), "village")
+	persona = None
+	if "--persona" in args:
+		persona = args[args.index("--persona") + 1].split("/")[-1]
+	world = check.World(seed.run_dir(name))
+	items = objects(world)
+	if persona:
+		items = [i for i in items if persona in i["readers"]]
+	print(json.dumps(items, ensure_ascii=False))
+	return 0
+
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))

tools/pasture/town/gaps.json

@@ -0,0 +1,47 @@
+[
+	{
+		"id": "G-0001",
+		"title": "Likes, reactions, votes and downvotes from followers on followers-only posts, and from members on circle posts, are dropped",
+		"match": {
+			"feature": "count\\.(like|react|vote)\\.(followers|circle)",
+			"observer": "privapub"
+		},
+		"kind": "server",
+		"phase": "now",
+		"code": "PrivaPub/Federation/Inbox/Handlers/LikeHandler.cs (MaySee)",
+		"opened": "2026-10-04",
+		"status": "closed",
+		"fixed_in": "128ff89",
+		"closed": "2026-10-04",
+		"note": "19 checks passed on the fixed PrivaPub (village, second run)"
+	},
+	{
+		"id": "G-0002",
+		"title": "A remote Block is not enforced: the blocked persona still sees and reaches the blocker",
+		"match": {
+			"feature": "block\\.enforced",
+			"observer": "privapub"
+		},
+		"kind": "server",
+		"phase": "P7",
+		"doc": "docs/INTEROP.md (Mastodon gaps)",
+		"opened": "2026-10-01",
+		"status": "closed",
+		"closed": "2026-10-04",
+		"fixed_in": "d405269",
+		"note": "checked by the Mastodon scenario (blocked_by)"
+	},
+	{
+		"id": "G-0003",
+		"title": "Votes and moderation that a Lemmy community relays inside Announce (Like, Dislike, Undo, Delete, Block) are dropped",
+		"match": {
+			"feature": "count\\.like\\.community",
+			"observer": "privapub"
+		},
+		"kind": "server",
+		"phase": "P7",
+		"code": "PrivaPub/Federation/Inbox/Handlers/AnnounceHandler.cs:203",
+		"opened": "2026-10-01",
+		"status": "open"
+	}
+]

tools/pasture/town/gen.py

@@ -0,0 +1,535 @@
+"""The plan: a fake community drawn from a spec and its seed, the same on every run (tests/test_gen.py pins its hash).
+
+A plan is a list of steps, each {n, phase, round, actor, verb, args, ref}. Accounts are named "<platform>/<username>",
+objects by a ref ("p12"); the seeder turns refs into URIs as it goes. Phases run in order:
+  accounts   every account, its profile, PrivaPub's roots and personas
+  groups     PrivaPub circles and communities, Lemmy communities
+  graph      follows (requests to locked accounts accepted, left pending or rejected), circle and community joins
+  content    top-level posts, then reply rounds, each waiting for the round before to arrive
+  interact   likes, boosts, reactions, votes, bookmarks
+  mutate     edits, deletes, blocks, mutes, reports
+The planner only asks a platform for what its driver can do (`caps`) and only lets an account touch what it could see.
+"""
+import hashlib
+import json
+import os
+import sys
+
+from core.rng import Rng
+
+GENERATOR_VERSION = 1
+HOSTS = {"privapub": "privapub.test", "gts": "gts.test", "mastodon": "mastodon.test", "misskey": "misskey.test",
+		 "sharkey": "sharkey.test", "akkoma": "akkoma.test", "lemmy": "lemmy.test", "hollo": "hollo.test"}
+SHORT = {"privapub": "pp", "gts": "gt", "mastodon": "ms", "misskey": "mk", "sharkey": "sk", "akkoma": "ak", "lemmy": "lm",
+		 "hollo": "ho"}
+MICRO = ("privapub", "gts", "mastodon", "misskey", "sharkey", "akkoma", "hollo")
+CAPS = {
+	"privapub": {"post", "reply", "cw", "media", "poll", "like", "boost", "bookmark", "follow", "block", "mute", "report",
+				 "dm", "delete", "edit", "vote", "react", "quote", "profile", "circle", "community", "located"},
+	"gts": {"post", "reply", "cw", "media", "poll", "like", "boost", "bookmark", "follow", "block", "mute", "report",
+			"dm", "delete", "edit", "vote", "profile"},
+	"mastodon": {"post", "reply", "cw", "media", "poll", "like", "boost", "bookmark", "follow", "block", "mute", "report",
+				 "dm", "delete", "edit", "vote", "quote", "profile"},
+	"misskey": {"post", "reply", "cw", "media", "poll", "like", "react", "boost", "bookmark", "follow", "block", "mute",
+				"report", "dm", "delete", "vote", "quote", "profile"},
+	"sharkey": {"post", "reply", "cw", "media", "poll", "like", "react", "boost", "bookmark", "follow", "block", "mute",
+				"report", "dm", "delete", "vote", "quote", "profile", "edit"},
+	"akkoma": {"post", "reply", "cw", "media", "poll", "like", "boost", "bookmark", "follow", "block", "mute", "report",
+			   "dm", "delete", "edit", "vote", "react", "quote", "profile"},
+	"lemmy": {"thread", "comment", "upvote", "downvote", "dm", "delete", "community", "edit", "block"},
+	"hollo": {"post", "reply", "cw", "media", "poll", "like", "boost", "bookmark", "follow", "block", "mute", "report",
+			  "dm", "delete", "edit", "vote", "quote", "react", "profile"},
+}
+
+FIRST = ["ada", "bo", "cleo", "dario", "elif", "femi", "gaia", "hiro", "ines", "jun", "kai", "lia", "milo", "nadia",
+		 "omar", "pia", "quinn", "rosa", "sami", "tea", "umbe", "vera", "wren", "xavi", "yuki", "zoe", "aldo", "bea",
+		 "cosimo", "dora", "enzo", "flor", "gigi", "hana", "ivo", "jade", "kira", "luca", "mara", "nico"]
+WORDS = {
+	"en": "the garden river morning bread market neighbour bicycle library rain coffee window festival map bridge "
+		  "lantern harvest tide mountain postcard quiet bright small warm early late again together tomorrow".split(),
+	"it": "il giardino fiume mattina pane mercato vicino bicicletta biblioteca pioggia caffè finestra festa mappa ponte "
+		  "lanterna raccolto marea montagna cartolina tranquillo luminoso piccolo caldo presto tardi insieme domani".split(),
+	"ja": "庭 川 朝 パン 市場 隣人 自転車 図書館 雨 コーヒー 窓 祭り 地図 橋 提灯 収穫 山 葉書 静か 明るい".split(),
+	"ar": "حديقة نهر صباح خبز سوق جار دراجة مكتبة مطر قهوة نافذة مهرجان خريطة جسر فانوس حصاد جبل هادئ".split(),
+}
+EMOJI = ["🌱", "🚲", "☕", "🌧", "🎉", "📚", "🌉", "🍞", "🗺", "🏔"]
+TAGS = ["town", "pasture", "garden", "market", "rain", "bikes", "books", "festival"]
+REACTIONS = ["🎉", "👍", "❤", "🔥", "😂"]
+PLACES = [(45.4642, 9.19, "Milano"), (45.07, 7.6869, "Torino"), (41.9028, 12.4964, "Roma"), (48.8566, 2.3522, "Paris"),
+		  (52.52, 13.405, "Berlin"), (35.6762, 139.6503, "Tokyo")]
+
+
+def load(path):
+	if not os.path.isabs(path) and not os.path.exists(path):
+		path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "specs", path if path.endswith(".json") else f"{path}.json")
+	with open(path) as f:
+		return json.load(f)
+
+
+class Planner:
+	def __init__(self, spec):
+		self.spec = spec
+		self.rng = Rng(spec["seed"], f"{spec['name']}/v{GENERATOR_VERSION}")
+		self.steps = []
+		self.accounts = {}  # key -> account dict
+		self.follows = {}  # follower key -> set of followee keys (accepted)
+		self.objects = {}  # ref -> object dict
+		self.groups = {}  # ref -> group dict
+		self.n = 0
+
+	def step(self, phase, actor, verb, args, ref=None, round_=0):
+		self.n += 1
+		s = {"n": self.n, "phase": phase, "round": round_, "actor": actor, "verb": verb, "args": args}
+		if ref:
+			s["ref"] = ref
+		self.steps.append(s)
+		return s
+
+	@staticmethod
+	def acct(key):
+		platform, user = key.split("/", 1)
+		return f"{user}@{HOSTS[platform]}"
+
+	def text(self, rng, lang, n_words, ref):
+		words = WORDS.get(lang, WORDS["en"])
+		body = " ".join(rng.choice(words) for _ in range(n_words))
+		if rng.chance(0.3):
+			body += " " + rng.choice(EMOJI)
+		return f"{body} ({ref})"
+
+	# -- accounts
+	def plan_accounts(self):
+		rng = self.rng.sub("accounts")
+		prof = self.spec.get("profiles", {})
+		langs = prof.get("langs", {"en": 70, "it": 15, "ja": 8, "ar": 7})
+		used = set()
+		for platform, conf in self.spec["peers"].items():
+			if platform == "privapub":
+				for r in range(conf.get("roots", 1)):
+					# root logins and passwords are canaries: neither may ever appear on another server or in a log line
+					canary = hashlib.sha256(f"{self.spec['seed']}/root{r}".encode()).hexdigest()
+					root = f"rootcanary{r + 1:02d}{canary[:8]}"
+					password = f"Canary-{canary[8:28]}-9!"
+					for _ in range(rng.span(conf.get("personas", [2, 3]))):
+						self._account(rng, platform, used, prof, langs, root=root, password=password)
+			else:
+				for _ in range(conf.get("accounts", 3)):
+					self._account(rng, platform, used, prof, langs)
+
+	def _account(self, rng, platform, used, prof, langs, root=None, password=None):
+		while True:
+			name = f"{rng.choice(FIRST)}_{SHORT[platform]}{rng.between(1, 99):02d}"
+			if name not in used:
+				used.add(name)
+				break
+		key = f"{platform}/{name}"
+		lang = rng.weighted(langs)
+		locked = platform != "lemmy" and rng.chance(prof.get("locked", 0.2))
+		account = {
+			"platform": platform, "username": name,
+			"password": password or f"Town-{hashlib.sha256(name.encode()).hexdigest()[:12]}-7!",
+			"name": name.split("_")[0].capitalize() + f" of {platform}", "lang": lang,
+			"bio": self.text(rng, lang, rng.span(prof.get("bioWords", [5, 20])), f"bio-{name}"),
+			"fields": [[f"field{i}", rng.choice(WORDS["en"])] for i in range(rng.span(prof.get("fields", [0, 3])))],
+			"locked": locked, "bot": rng.chance(prof.get("bot", 0.05)),
+			"avatar_seed": f"avatar-{name}" if rng.chance(prof.get("avatar", 0.9)) else None,
+			"header_seed": f"header-{name}" if rng.chance(prof.get("header", 0.5)) else None,
+			"root": root,
+		}
+		self.accounts[key] = account
+		self.step("accounts", key, "account", account)
+
+	def keys(self, platform=None, micro=False):
+		return [k for k, a in self.accounts.items()
+				if (platform is None or a["platform"] == platform) and (not micro or a["platform"] in MICRO)]
+
+	# -- groups
+	def plan_groups(self):
+		rng = self.rng.sub("groups")
+		conf = self.spec["peers"]
+		personas = self.keys("privapub")
+		for i in range(conf.get("privapub", {}).get("circles", 0)):
+			owner = rng.choice(personas)
+			ref = f"circle{i + 1}"
+			self.groups[ref] = {"kind": "circle", "owner": owner, "members": {owner}, "name": f"{ref}_{rng.between(100, 999)}"}
+			self.step("groups", owner, "group", {"kind": "circle", "name": self.groups[ref]["name"]}, ref)
+		for i in range(conf.get("privapub", {}).get("communities", 0)):
+			owner = rng.choice(personas)
+			ref = f"ppcomm{i + 1}"
+			self.groups[ref] = {"kind": "community", "host": "privapub", "owner": owner, "members": set(),
+								"name": f"square{i + 1}_{rng.between(100, 999)}"}
+			self.step("groups", owner, "group", {"kind": "community", "name": self.groups[ref]["name"]}, ref)
+		for i in range(conf.get("lemmy", {}).get("communities", 0)):
+			lemmings = self.keys("lemmy")
+			if not lemmings:
+				break
+			owner = rng.choice(lemmings)
+			ref = f"lmcomm{i + 1}"
+			self.groups[ref] = {"kind": "community", "host": "lemmy", "owner": owner, "members": {owner},
+								"name": f"plaza{i + 1}_{rng.between(100, 999)}"}
+			self.step("groups", owner, "group", {"kind": "community", "name": self.groups[ref]["name"]}, ref)
+
+	# -- the graph
+	def plan_graph(self):
+		rng = self.rng.sub("graph")
+		g = self.spec.get("graph", {})
+		micro = self.keys(micro=True)
+		popularity = {k: 1 + rng.below(5) for k in micro}
+		for follower in micro:
+			n = rng.span(g.get("follows", [3, 8]))
+			candidates = [k for k in micro if k != follower]
+			weights = {k: popularity[k] * (3 if k.split("/")[0] == "privapub" or follower.startswith("privapub/") else 1)
+					   for k in candidates}
+			chosen = set()
+			while len(chosen) < min(n, len(candidates)):
+				chosen.add(rng.weighted({k: w for k, w in weights.items() if k not in chosen}))
+			for target in sorted(chosen):
+				self._follow(rng, g, follower, target)
+			# mutuals: some of those followed follow back
+			for target in sorted(chosen):
+				if rng.chance(g.get("mutual", 0.4)) and follower not in self.follows.get(target, set()):
+					self._follow(rng, g, target, follower)
+		# circles: members are local siblings' neighbours and remote followers of the owner
+		for ref, grp in sorted(self.groups.items()):
+			if grp["kind"] == "circle":
+				pool = sorted(k for k in micro if k != grp["owner"] and k.split("/")[0] in ("privapub", "gts", "mastodon", "akkoma"))
+				for member in rng.sample(pool, min(len(pool), self.spec.get("circleMembers", 4))):
+					grp["members"].add(member)
+					self.step("graph", member, "join", {"group": ref})
+					self.step("graph", grp["owner"], "approve", {"group": ref, "member": member})
+			elif grp["host"] == "privapub":
+				pool = sorted(k for k in self.accounts if k != grp["owner"])
+				for member in rng.sample(pool, min(len(pool), self.spec.get("communityMembers", 6))):
+					grp["members"].add(member)
+					self.step("graph", member, "join", {"group": ref})
+			else:
+				pool = sorted(k for k in self.accounts if k != grp["owner"] and self.accounts[k]["platform"] in ("privapub", "lemmy", "mastodon", "gts"))
+				for member in rng.sample(pool, min(len(pool), self.spec.get("communityMembers", 6))):
+					grp["members"].add(member)
+					self.step("graph", member, "join", {"group": ref})
+
+	def _follow(self, rng, g, follower, target):
+		self.step("graph", follower, "follow", {"target": target})
+		if self.accounts[target]["locked"]:
+			fate = rng.weighted({"accept": 1 - g.get("pending", 0.1) - g.get("rejected", 0.05),
+								 "pending": g.get("pending", 0.1), "reject": g.get("rejected", 0.05)})
+			if fate == "accept":
+				self.step("graph", target, "accept", {"target": follower})
+			elif fate == "reject":
+				self.step("graph", target, "reject", {"target": follower})
+				return
+			else:
+				return
+		self.follows.setdefault(follower, set()).add(target)
+
+	def followers(self, key):
+		return {f for f, targets in self.follows.items() if key in targets}
+
+	def can_see(self, viewer, obj):
+		if viewer == obj["author"]:
+			return True
+		v = obj["visibility"]
+		if v in ("public", "unlisted"):
+			return True
+		if v == "followers":
+			return viewer in obj["followers"] or viewer in obj["mentions"]
+		if v == "direct":
+			return viewer in obj["mentions"]
+		if v == "circle":
+			return viewer in self.groups[obj["group"]]["members"]
+		if v == "community":
+			return True
+		return False
+
+	# -- content
+	def plan_content(self):
+		rng = self.rng.sub("content")
+		c = self.spec.get("content", {})
+		mix = c.get("mix", {"text": 40, "cw": 8, "tags": 10, "mention": 10, "image": 8, "poll": 6, "quote": 5})
+		vis = c.get("visibility", {"public": 55, "unlisted": 15, "followers": 15, "direct": 10, "circle": 5})
+		micro = self.keys(micro=True)
+		top = []
+		for i in range(c.get("posts", 60)):
+			author = rng.choice(micro)
+			platform = author.split("/")[0]
+			obj = self._post(rng, f"p{len(self.objects) + 1}", author, platform, mix, vis, c, top)
+			if obj:
+				top.append(obj)
+		# community posts: Lemmy threads, posts into the PrivaPub community
+		for ref, grp in sorted(self.groups.items()):
+			if grp["kind"] != "community":
+				continue
+			posters = sorted(m for m in grp["members"] | {grp["owner"]}
+							 if (grp["host"] == "lemmy" and m.startswith("lemmy/")) or (grp["host"] == "privapub" and m.startswith(("privapub/", "lemmy/"))))
+			for _ in range(c.get("communityPosts", 4)):
+				if not posters:
+					break
+				author = rng.choice(posters)
+				oref = f"p{len(self.objects) + 1}"
+				obj = {"ref": oref, "author": author, "visibility": "community", "group": ref, "mentions": set(),
+					   "followers": set(), "kind": "thread", "round": 0}
+				self.objects[oref] = obj
+				lang = self.accounts[author]["lang"]
+				self.step("content", author, "post", {"visibility": "community", "group": ref,
+													   "title": f"{rng.choice(WORDS['en']).capitalize()} thread {oref}",
+													   "text": self.text(rng, lang, rng.between(8, 30), oref)}, oref)
+				top.append(obj)
+		# reply rounds
+		rounds = c.get("replyRounds", 3)
+		previous = top
+		for rnd in range(1, rounds + 1):
+			current = []
+			for parent in previous:
+				if not rng.chance(c.get("replies", 0.45) if rnd == 1 else c.get("deeperReplies", 0.35)):
+					continue
+				repliers = [k for k in self.accounts if k != parent["author"] and self.can_see(k, parent)
+							and self._can_reply(k, parent)]
+				if not repliers:
+					continue
+				for _ in range(1 if rnd > 1 else rng.between(1, 2)):
+					replier = rng.choice(repliers)
+					obj = self._reply(rng, replier, parent, rnd)
+					if obj:
+						current.append(obj)
+			previous = current
+
+	def _can_reply(self, key, parent):
+		platform = key.split("/")[0]
+		if parent["visibility"] == "community":
+			grp = self.groups[parent["group"]]
+			return key in grp["members"] or key == grp["owner"]
+		if platform == "lemmy":
+			return False
+		if parent["visibility"] == "circle":
+			return platform == "privapub" or key in self.groups[parent["group"]]["members"]
+		if parent["visibility"] == "located":
+			return False
+		return "reply" in CAPS[platform]
+
+	def _post(self, rng, ref, author, platform, mix, vis, c, top):
+		caps = CAPS[platform]
+		vweights = {k: w for k, w in vis.items() if k in ("public", "unlisted", "followers", "direct")
+					or (k in ("circle", "located") and k in caps)}
+		visibility = rng.weighted(vweights)
+		circles = []
+		if visibility == "circle":
+			circles = [r for r, g in sorted(self.groups.items()) if g["kind"] == "circle" and g["owner"] == author]
+			if not circles:
+				visibility = "followers"
+		# circle and located posts go through /clientapi, which takes text, a warning and a title only
+		plain = visibility in ("circle", "located")
+		kind = rng.weighted({k: w for k, w in mix.items()
+							 if (k in ("text", "tags", "mention") or k in caps or (k == "image" and "media" in caps))
+							 and not (k == "quote" and not top)
+							 and not (plain and k in ("image", "poll", "quote", "mention", "tags"))})
+		lang = self.accounts[author]["lang"]
+		args = {"visibility": visibility, "text": self.text(rng, lang, rng.between(4, 40), ref), "language": lang}
+		obj = {"ref": ref, "author": author, "visibility": visibility, "mentions": set(), "kind": kind, "round": 0,
+			   "followers": set(self.followers(author)), "group": None}
+		if visibility == "circle":
+			obj["group"] = circles[0]
+			args["group"] = circles[0]
+		if visibility == "located":
+			lat, lng, place = rng.choice(PLACES)
+			args["location"] = {"lat": lat, "lng": lng, "range_km": rng.choice([2, 5, 20])}
+			args["text"] += f" near {place}"
+		if visibility == "direct" or kind == "mention":
+			others = [k for k in self.keys(micro=True) if k != author]
+			count = rng.between(1, 3) if visibility == "direct" else 1
+			obj["mentions"] = set(rng.sample(others, count))
+			args["mentions"] = sorted(obj["mentions"])
+		if kind == "cw":
+			args["cw"] = f"{rng.choice(WORDS['en'])} {rng.choice(['spoilers', 'food', 'weather', 'politics'])}"
+			obj["cw"] = args["cw"]
+		elif kind == "tags":
+			args["tags"] = rng.sample(TAGS, rng.between(1, 3))
+		elif kind == "image":
+			args["media"] = [{"kind": "image", "seed": f"{ref}-{i}", "alt": f"picture {i + 1} of {ref}",
+							  "sensitive": rng.chance(0.15)} for i in range(rng.between(1, 3))]
+		elif kind == "poll":
+			opts = rng.sample(WORDS["en"], rng.between(2, 4))
+			args["poll"] = {"options": opts, "multiple": rng.chance(0.3), "expires_in": 86400}
+			obj["options"] = len(opts)
+		elif kind == "quote":
+			candidates = [o for o in top if o["visibility"] in ("public", "unlisted") and o["kind"] != "quote"]
+			if not candidates or "quote" not in caps:
+				kind = obj["kind"] = "text"
+			else:
+				target = rng.choice(candidates)
+				args["quote"] = target["ref"]
+				obj["quote"] = target["ref"]
+		self.objects[ref] = obj
+		self.step("content", author, "post", args, ref)
+		return obj
+
+	def _reply(self, rng, replier, parent, rnd):
+		ref = f"p{len(self.objects) + 1}"
+		platform = replier.split("/")[0]
+		if parent["visibility"] == "community" and platform == "lemmy":
+			kind = "comment"
+		elif platform == "lemmy":
+			return None
+		else:
+			kind = "reply"
+		visibility = parent["visibility"] if parent["visibility"] in ("direct", "followers", "circle", "community") else \
+			rng.weighted({"public": 70, "unlisted": 30})
+		if visibility == "circle" and platform != "privapub":
+			visibility = "direct"  # a remote member answers a circle post as the post's own audience
+		mentions = set()
+		if visibility == "direct":
+			mentions = (parent["mentions"] | {parent["author"]}) - {replier}
+		elif visibility in ("public", "unlisted", "followers") and parent["author"].split("/")[0] in MICRO \
+				and parent["author"] != replier:
+			mentions = {parent["author"]}  # clients mention the account replied to, and servers deliver by mention
+		lang = self.accounts[replier]["lang"]
+		args = {"visibility": visibility, "text": self.text(rng, lang, rng.between(3, 25), ref), "reply_to": parent["ref"],
+				"language": lang}
+		if mentions:
+			args["mentions"] = sorted(mentions)
+		if visibility == "circle":
+			args["group"] = parent["group"]
+		if visibility == "community":
+			grp = self.groups[parent["group"]]
+			if platform == "privapub" and grp["host"] == "privapub":
+				args["group"] = parent["group"]
+			elif platform != "lemmy":
+				args["visibility"] = "public"  # a microblog reply to a community thread is a public reply the group relays
+		obj = {"ref": ref, "author": replier, "visibility": visibility, "mentions": mentions, "kind": kind,
+			   "round": rnd, "parent": parent["ref"], "followers": set(self.followers(replier)), "group": parent.get("group")}
+		self.objects[ref] = obj
+		self.step("content", replier, "post", args, ref, rnd)
+		return obj
+
+	# -- interactions
+	def plan_interactions(self):
+		rng = self.rng.sub("interact")
+		conf = self.spec.get("interactions", {})
+		for ref, obj in sorted(self.objects.items(), key=lambda kv: int(kv[0][1:])):
+			if obj["visibility"] == "located":
+				continue
+			viewers = [k for k in self.accounts if k != obj["author"] and self.can_see(k, obj)
+					   and self._can_interact(k, obj)]
+			if not viewers:
+				continue
+			for k in rng.sample(viewers, rng.span(conf.get("likes", [0, 4]))):
+				platform = k.split("/")[0]
+				if platform == "lemmy":
+					self.step("interact", k, "like" if rng.chance(0.75) else "downvote", {"target": ref})
+				elif "react" in CAPS[platform] and rng.chance(conf.get("react", 0.3)):
+					self.step("interact", k, "react", {"target": ref, "emoji": rng.choice(REACTIONS)})
+				else:
+					self.step("interact", k, "like", {"target": ref})
+			if obj["visibility"] in ("public", "unlisted"):
+				for k in rng.sample([v for v in viewers if "boost" in CAPS[v.split("/")[0]]], rng.span(conf.get("boosts", [0, 2]))):
+					self.step("interact", k, "boost", {"target": ref})
+			if obj.get("options"):
+				for k in rng.sample([v for v in viewers if "vote" in CAPS[v.split("/")[0]]],
+									round(len(viewers) * conf.get("vote", 0.6))):
+					self.step("interact", k, "vote", {"target": ref, "choices": [rng.below(obj["options"])]})
+			if rng.chance(conf.get("bookmark", 0.1)):
+				k = rng.choice([v for v in viewers if "bookmark" in CAPS[v.split("/")[0]]] or [None])
+				if k:
+					self.step("interact", k, "bookmark", {"target": ref})
+
+	def _can_interact(self, key, obj):
+		platform = key.split("/")[0]
+		if obj["visibility"] == "community":
+			grp = self.groups[obj["group"]]
+			return key in grp["members"] or key == grp["owner"]
+		if platform == "lemmy":
+			return False
+		if obj["visibility"] == "circle":
+			return key in self.groups[obj["group"]]["members"]
+		return True
+
+	# -- mutations
+	def plan_mutations(self):
+		rng = self.rng.sub("mutate")
+		m = self.spec.get("mutations", {})
+		for ref, obj in sorted(self.objects.items(), key=lambda kv: int(kv[0][1:])):
+			platform = obj["author"].split("/")[0]
+			if "edit" in CAPS[platform] and rng.chance(m.get("edit", 0.08)) and obj["visibility"] not in ("circle", "community", "located"):
+				lang = self.accounts[obj["author"]]["lang"]
+				# an edit keeps the names the post had, as a client's edit box does; dropping them changes who it is for
+				self.step("mutate", obj["author"], "edit", {"target": ref, "text": self.text(rng, lang, 6, ref) + " (edited)",
+															 "mentions": sorted(obj["mentions"]), "cw": obj.get("cw")})
+				obj["edited"] = True
+		for ref, obj in sorted(self.objects.items(), key=lambda kv: int(kv[0][1:])):
+			if obj["round"] > 0 and rng.chance(m.get("delete", 0.04)) and obj["visibility"] not in ("circle", "located"):
+				self.step("mutate", obj["author"], "delete", {"target": ref})
+				obj["deleted"] = True
+		micro = self.keys(micro=True)
+		for verb, count in (("block", m.get("blocks", 2)), ("mute", m.get("mutes", 2))):
+			for _ in range(count):
+				a, b = rng.sample(micro, 2)
+				self.step("mutate", a, verb, {"target": b})
+		for _ in range(m.get("reports", 1)):
+			reporter = rng.choice(micro)
+			theirs = [o for o in self.objects.values() if o["author"] != reporter and o["visibility"] == "public"
+					  and o["author"].split("/")[0] in MICRO]
+			if theirs:
+				o = rng.choice(theirs)
+				self.step("mutate", reporter, "report", {"target": o["author"], "refs": [o["ref"]], "comment": "town report"})
+
+	def plan(self):
+		self.plan_accounts()
+		self.plan_groups()
+		self.plan_graph()
+		self.plan_content()
+		self.plan_interactions()
+		self.plan_mutations()
+		return self.steps
+
+
+def plan(spec):
+	return Planner(spec).plan()
+
+
+def snapshot(planner):
+	"""The planner's world as JSON (sets as sorted lists), stored with a run so later generator changes never re-read it."""
+	def plain(v):
+		if isinstance(v, set):
+			return sorted(v)
+		if isinstance(v, dict):
+			return {k: plain(x) for k, x in v.items()}
+		if isinstance(v, list):
+			return [plain(x) for x in v]
+		return v
+	return {"version": GENERATOR_VERSION, "steps": planner.steps, "accounts": planner.accounts,
+			"objects": plain(planner.objects), "groups": plain(planner.groups), "follows": plain(planner.follows)}
+
+
+class Stored(Planner):
+	"""A Planner rebuilt from a run's plan.json instead of from the spec."""
+
+	def __init__(self, snap):
+		self.spec = None
+		self.steps = snap["steps"]
+		self.accounts = snap["accounts"]
+		self.objects = {r: {k: (set(v) if k in ("mentions", "followers") else v) for k, v in o.items()} for r, o in snap["objects"].items()}
+		self.groups = {r: {k: (set(v) if k == "members" else v) for k, v in g.items()} for r, g in snap["groups"].items()}
+		self.follows = {k: set(v) for k, v in snap["follows"].items()}
+
+	def plan(self):
+		return self.steps
+
+
+def digest(steps):
+	return hashlib.sha256(json.dumps(steps, sort_keys=True, ensure_ascii=False).encode()).hexdigest()
+
+
+def main(args):
+	spec = load(args[0] if args else "village")
+	steps = plan(spec)
+	if "--summary" in args:
+		from collections import Counter
+		print(json.dumps({"steps": len(steps), "sha": digest(steps),
+						  "by": dict(Counter(f"{s['phase']}:{s['verb']}" for s in steps))}, indent=1))
+		return 0
+	for s in steps:
+		print(json.dumps(s, ensure_ascii=False))
+	return 0
+
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))

tools/pasture/town/media/SOURCES.md

@@ -0,0 +1,13 @@
+# Town media
+
+Made once on the workstation with ffmpeg 7 from its own test sources (no third-party footage or sound):
+
+```sh
+ffmpeg -f lavfi -i "testsrc=size=160x120:rate=10:duration=2" -f lavfi -i "sine=frequency=440:duration=2" \
+	-c:v libx264 -pix_fmt yuv420p -profile:v baseline -c:a aac -b:a 32k -movflags +faststart -shortest \
+	-fflags +bitexact -flags:v +bitexact -flags:a +bitexact -map_metadata -1 tiny.mp4
+ffmpeg -f lavfi -i "sine=frequency=330:duration=2" -c:a libvorbis -q:a 0 \
+	-fflags +bitexact -flags:a +bitexact -map_metadata -1 tiny.ogg
+```
+
+Images are drawn by `core/media.py` from a seed, so they need no file here.

tools/pasture/town/report.py

@@ -0,0 +1,161 @@
+"""out/<run>/report.html: one self-contained page (no script, fake pasture data only, no tokens) with the run's totals,
+the feature × server matrix, every failing cell with its examples, and the bash scenarios' checks. Written so it can be
+published as it is."""
+import html
+import json
+import os
+import sys
+from collections import Counter, defaultdict
+
+import gen
+
+DIRS = ["in", "out", "via", "control", "local"]
+DIR_LABEL = {"in": "into PrivaPub", "out": "from PrivaPub", "via": "through PrivaPub", "control": "peer to peer",
+			 "local": "on its own server"}
+GROUPS = [("deliver", "Delivery"), ("confine", "Confinement"), ("see", "Seen by the audience"),
+		  ("hide", "Hidden from outsiders"), ("count", "Counts"), ("thread", "Threads"), ("edit", "Edits"),
+		  ("delete", "Deletes"), ("graph", "Follows"), ("privacy", "Privacy")]
+STATUS_ORDER = ["fail", "xpass", "xfail", "pass"]
+
+CSS = """
+/* layout: one column; a sticky summary, then the matrix in its own horizontal scroller, then failures and scenarios */
+:root {
+	--bg: #f6f7f6; --panel: #ffffff; --fg: #1d2421; --muted: #5c6863; --line: #d9dfdc;
+	--pass: #2f7d57; --fail: #c2412d; --xfail: #a77a14; --xpass: #6a4fc0; --none: #e4e9e6;
+	--sans: "IBM Plex Sans", system-ui, sans-serif; --mono: "IBM Plex Mono", ui-monospace, monospace;
+}
+@media (prefers-color-scheme: dark) { :root:not([data-theme="light"]) {
+	--bg: #121715; --panel: #1a211e; --fg: #e3e9e6; --muted: #98a59f; --line: #2c3632;
+	--pass: #4fb184; --fail: #ec6c57; --xfail: #d6a940; --xpass: #9d86ef; --none: #26302c; color-scheme: dark } }
+:root[data-theme="dark"] {
+	--bg: #121715; --panel: #1a211e; --fg: #e3e9e6; --muted: #98a59f; --line: #2c3632;
+	--pass: #4fb184; --fail: #ec6c57; --xfail: #d6a940; --xpass: #9d86ef; --none: #26302c; color-scheme: dark }
+body { background: var(--bg); color: var(--fg); font: 15px/1.5 var(--sans); margin: 0; }
+main { max-width: 1180px; margin: 0 auto; padding-inline: 16px; padding-block: 24px 48px; display: grid; gap: 28px; }
+h1 { font-size: 1.6rem; margin: 0; text-wrap: balance; }
+h2 { font-size: 1.15rem; margin: 0 0 10px; text-wrap: balance; }
+p { margin: 0; max-width: 70ch; color: var(--muted); }
+.meta { font: 13px var(--mono); color: var(--muted); overflow-wrap: anywhere; }
+.totals { display: flex; flex-wrap: wrap; gap: 8px 18px; font-variant-numeric: tabular-nums; }
+.total { display: flex; align-items: baseline; gap: 6px; }
+.total b { font: 600 1.35rem var(--mono); }
+.total span { text-transform: uppercase; letter-spacing: .06em; font-size: 12px; color: var(--muted); }
+.legend { display: flex; flex-wrap: wrap; gap: 6px 16px; font-size: 13px; color: var(--muted); }
+.legend i, .m { display: inline-block; width: 12px; height: 12px; border-radius: 2px; vertical-align: -1px; }
+.s-pass { background: var(--pass); } .s-fail { background: var(--fail); } .s-xfail { background: var(--xfail); }
+.s-xpass { background: var(--xpass); } .s-none { background: var(--none); }
+.scroll { overflow-x: auto; background: var(--panel); border: 1px solid var(--line); border-radius: 6px; }
+table { border-collapse: collapse; width: 100%; font-variant-numeric: tabular-nums; }
+th, td { padding: 6px 10px; border-bottom: 1px solid var(--line); text-align: left; vertical-align: top; }
+thead th { font-size: 12px; text-transform: uppercase; letter-spacing: .06em; color: var(--muted); white-space: nowrap; }
+tbody th { font: 13px var(--mono); white-space: nowrap; }
+tr.group th { font: 600 12px var(--sans); text-transform: uppercase; letter-spacing: .08em; color: var(--muted);
+	background: var(--bg); }
+.cell { display: flex; gap: 3px; flex-wrap: nowrap; }
+.cell .m { width: 14px; height: 14px; }
+.fails { display: grid; gap: 10px; }
+details { background: var(--panel); border: 1px solid var(--line); border-radius: 6px; padding: 8px 12px; }
+details[open] { padding-bottom: 12px; }
+summary { cursor: pointer; font: 13px var(--mono); overflow-wrap: anywhere; }
+summary:focus-visible { outline: 2px solid var(--xpass); outline-offset: 2px; }
+pre { font: 12px/1.45 var(--mono); white-space: pre-wrap; overflow-wrap: anywhere; margin: 8px 0 0; color: var(--muted); }
+.tag { font: 12px var(--mono); padding: 1px 6px; border-radius: 3px; color: var(--bg); }
+"""
+
+
+def cell_marks(cells, feature, observer):
+	parts = []
+	for d in DIRS:
+		agg = Counter()
+		for key, v in cells.items():
+			f, origin, obs, direction = key.split("|")
+			if f == feature and obs == observer and direction == d:
+				for s in STATUS_ORDER:
+					agg[s] += v[s]
+		if not agg:
+			continue
+		worst = next(s for s in STATUS_ORDER if agg[s])
+		title = f"{DIR_LABEL[d]}: " + ", ".join(f"{agg[s]} {s}" for s in STATUS_ORDER if agg[s])
+		parts.append(f'<i class="m s-{worst}" title="{html.escape(title)}"></i>')
+	return "".join(parts) or '<i class="m s-none" title="not checked"></i>'
+
+
+def render(result, scenarios, title):
+	cells = result["cells"]
+	totals = Counter(result["totals"])
+	observers = [p for p in gen.HOSTS if any(k.split("|")[2] == p for k in cells)] + \
+				(["*"] if any(k.split("|")[2] == "*" for k in cells) else [])
+	features = sorted({k.split("|")[0] for k in cells})
+	out = [f"<title>{html.escape(title)}</title>",
+		   '<link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=IBM+Plex+Mono:wght@400;600&family=IBM+Plex+Sans:wght@400;600&display=swap">',
+		   f"<style>{CSS}</style>", "<main>"]
+	out.append(f"<header style=\"display:grid;gap:10px\"><h1>{html.escape(title)}</h1>"
+			   f"<p>Every object the town's accounts made, checked on every server it should and should not have reached, "
+			   f"through each server's database and API. A cell's squares are the directions it was checked in: "
+			   f"into PrivaPub, from PrivaPub, through PrivaPub, peer to peer, and on the author's own server.</p>"
+			   f"<div class=\"meta\">{html.escape(result['run'])} · checked {html.escape(result['when'])}</div></header>")
+	out.append('<section class="totals" aria-label="Totals">' + "".join(
+		f'<div class="total"><i class="m s-{s}"></i><b>{totals.get(s, 0)}</b><span>{s}</span></div>' for s in STATUS_ORDER) + "</section>")
+	out.append('<div class="legend"><span><i class="s-pass"></i> pass</span><span><i class="s-fail"></i> fail</span>'
+			   '<span><i class="s-xfail"></i> known gap, still failing</span><span><i class="s-xpass"></i> known gap, now passing</span>'
+			   '<span><i class="s-none"></i> not checked</span></div>')
+	# the matrix
+	rows = ["<thead><tr><th>feature</th>" + "".join(f"<th>{html.escape(o)}</th>" for o in observers) + "</tr></thead><tbody>"]
+	for prefix, label in GROUPS:
+		group = [f for f in features if f.split(".")[0] == prefix]
+		if not group:
+			continue
+		rows.append(f'<tr class="group"><th colspan="{len(observers) + 1}">{label}</th></tr>')
+		for f in group:
+			rows.append(f"<tr><th>{html.escape(f)}</th>" + "".join(
+				f'<td><span class="cell">{cell_marks(cells, f, o)}</span></td>' for o in observers) + "</tr>")
+	rows.append("</tbody>")
+	out.append('<section><h2>Matrix</h2><div class="scroll"><table>' + "".join(rows) + "</table></div></section>")
+	# failures
+	bad = sorted(((k, v) for k, v in cells.items() if v["fail"] or v["xpass"]), key=lambda kv: (-kv[1]["fail"], kv[0]))
+	items = []
+	for key, v in bad:
+		f, origin, obs, d = key.split("|")
+		status = "fail" if v["fail"] else "xpass"
+		examples = "\n".join(json.dumps(e, ensure_ascii=False, default=str) for e in v["examples"])
+		items.append(f'<details><summary><span class="tag s-{status}">{v[status]}/{v["n"]}</span> {html.escape(f)} · '
+					 f'{html.escape(origin)} → {html.escape(obs)} · {html.escape(DIR_LABEL.get(d, d))}</summary>'
+					 f"<pre>{html.escape(examples)}</pre></details>")
+	out.append(f'<section><h2>Failing cells ({len(bad)})</h2><div class="fails">' + ("".join(items) or "<p>None.</p>") + "</div></section>")
+	# bash scenarios
+	if scenarios:
+		per = defaultdict(Counter)
+		for s in scenarios:
+			per[s["peer"]][s["status"]] += 1
+		srows = "".join(f"<tr><th>{html.escape(p)}</th>" + "".join(f"<td>{c.get(s, 0)}</td>" for s in ("pass", "fail", "xfail")) + "</tr>"
+						for p, c in per.items())
+		failed = "".join(f"<li>{html.escape(s['peer'])}: {html.escape(s['check'])} ({s['status']})</li>"
+						 for s in scenarios if s["status"] != "pass")
+		out.append('<section><h2>Scenarios</h2><p>The pasture\'s scripted checks, one peer at a time.</p>'
+				   '<div class="scroll"><table><thead><tr><th>peer</th><th>pass</th><th>fail</th><th>known gap</th></tr></thead>'
+				   f"<tbody>{srows}</tbody></table></div><ul>{failed}</ul></section>")
+	out.append("</main>")
+	return "\n".join(out)
+
+
+def main(args):
+	import seed
+	name = next((a for a in args if not a.startswith("--")), "village")
+	run_dir = seed.run_dir(name)
+	with open(os.path.join(run_dir, "results.json")) as f:
+		result = json.load(f)
+	scenarios = []
+	path = os.path.join(seed.OUT, "scenarios.jsonl")
+	if os.path.exists(path):
+		with open(path) as f:
+			scenarios = [json.loads(line) for line in f if line.strip()]
+	page = render(result, scenarios, "PrivaPub Town Matrix")
+	target = os.path.join(run_dir, "report.html")
+	with open(target, "w") as f:
+		f.write(page)
+	print(target)
+	return 0
+
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))

tools/pasture/town/seed.py

@@ -0,0 +1,404 @@
+"""Replays a plan against the pasture and writes what happened to out/<run>/ledger.jsonl, one fact per line:
+accounts made, groups made, follows and their state, objects with their URIs, interactions, mutations, errors.
+
+A reply waits for its parent to arrive on the replier's server (the round before had time to settle); if it never
+arrives the parent is fetched and the ledger says `arrived_by: fetch` for that server, so a delivery that failed shows
+up as a cell in the report rather than as a broken thread. An error never stops the seeding: it is a ledger line."""
+import json
+import os
+import sys
+import threading
+import time
+import traceback
+from concurrent.futures import ThreadPoolExecutor
+
+import gen
+import state
+from core import podman
+from dialects import driver
+from dialects.base import Account, PostSpec, Unsupported
+
+OUT = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "out")
+
+
+class Ledger:
+	def __init__(self, run_dir):
+		self.path = os.path.join(run_dir, "ledger.jsonl")
+		self.lock = threading.Lock()
+		self.facts = []
+		if os.path.exists(self.path):
+			with open(self.path) as f:
+				self.facts = [json.loads(line) for line in f if line.strip()]
+
+	def add(self, **fact):
+		fact["t"] = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
+		with self.lock:
+			self.facts.append(fact)
+			with open(self.path, "a") as f:
+				f.write(json.dumps(fact, ensure_ascii=False, default=str) + "\n")
+
+	def done(self):
+		return {f["n"] for f in self.facts if "n" in f and f.get("type") != "error"}
+
+
+class Seeder:
+	def __init__(self, spec, run_dir, only=None):
+		self.spec = spec
+		snap_path = os.path.join(run_dir, "plan.json")
+		if os.path.exists(snap_path):
+			with open(snap_path) as f:
+				self.steps = json.load(f)["steps"]
+		else:
+			planner = gen.Planner(spec)
+			planner.plan()
+			self.steps = planner.steps
+			with open(snap_path, "w") as f:
+				json.dump(gen.snapshot(planner), f, ensure_ascii=False)
+		self.ledger = Ledger(run_dir)
+		self.only = only
+		self.uris = {}  # ref -> uri
+		self.origin = {}  # ref -> platform
+		self.groups = {}  # ref -> {kind, host, id, code, handle, ...}
+		self.timing = spec.get("time", {})
+		for f in self.ledger.facts:
+			if f.get("type") == "object":
+				self.uris[f["ref"]] = f["uri"]
+				self.origin[f["ref"]] = f["origin"]
+			elif f.get("type") == "group":
+				self.groups[f["ref"]] = f["group"]
+
+	def acct(self, key):
+		return gen.Planner.acct(key)
+
+	def session(self, key):
+		platform, user = key.split("/", 1)
+		return state.session(platform, user)
+
+	def run(self):
+		done = self.ledger.done()
+		phases = ["accounts", "groups", "graph", "content", "interact", "mutate"]
+		for phase in phases:
+			steps = [s for s in self.steps if s["phase"] == phase and s["n"] not in done]
+			if self.only and phase not in self.only:
+				continue
+			if not steps:
+				continue
+			started = time.time()
+			print(f"{phase}: {len(steps)} steps", flush=True)
+			if phase == "accounts":
+				self.accounts(steps)
+			elif phase == "graph" and any(k.startswith("lemmy/") for k in self.accounts_planned()):
+				self.warm_lemmy()
+				self.parallel(steps, ordered=True)
+			elif phase == "content":
+				# follows, joins and their Accepts travel through queues: posts made before they land reach nobody new
+				time.sleep(self.timing.get("graphSettleSeconds", 30))
+				rounds = sorted({s["round"] for s in steps})
+				for r in rounds:
+					if r > 0:
+						time.sleep(self.timing.get("roundSettleSeconds", 20))
+					self.parallel([s for s in steps if s["round"] == r])
+			else:
+				if phase in ("interact", "mutate"):
+					time.sleep(self.timing.get("roundSettleSeconds", 20))
+				self.parallel(steps, ordered=phase in ("graph", "groups"))
+			print(f"{phase}: done in {time.time() - started:.0f}s", flush=True)
+		return 0
+
+	def accounts_planned(self):
+		return [s["actor"] for s in self.steps if s["verb"] == "account"]
+
+	def warm_lemmy(self):
+		"""One account on every other server, looked up from Lemmy, so its send workers exist before the first Follow."""
+		keys = self.accounts_planned()
+		first = {}
+		for k in keys:
+			first.setdefault(k.split("/")[0], k)
+		lemmy = next(k for k in keys if k.startswith("lemmy/"))
+		try:
+			ready = driver("lemmy").warm(self.session(lemmy), [self.acct(k) for p, k in first.items() if p != "lemmy"])
+			self.ledger.add(type="warmup", platform="lemmy", ready=ready)
+		except Exception as e:
+			self.ledger.add(type="error", verb="warmup", actor=lemmy, error=repr(e)[:400])
+
+	def parallel(self, steps, ordered=False):
+		"""Steps of one actor run in plan order; different actors run side by side (four per platform at most)."""
+		if ordered:
+			# the graph's accept and approve steps need the follow before them, whoever made it
+			for s in steps:
+				self.do(s)
+			return
+		by_actor = {}
+		for s in steps:
+			by_actor.setdefault(s["actor"], []).append(s)
+		with ThreadPoolExecutor(max_workers=12) as pool:
+			for actor_steps in by_actor.values():
+				pool.submit(lambda ss: [self.do(s) for s in ss], actor_steps)
+
+	def do(self, s):
+		try:
+			getattr(self, f"v_{s['verb']}")(s)
+		except Exception as e:
+			kind = "unsupported" if isinstance(e, Unsupported) else "error"
+			self.ledger.add(type=kind, n=s["n"], verb=s["verb"], actor=s["actor"], ref=s.get("ref"), error=repr(e)[:600])
+			if kind == "error" and os.environ.get("TOWN_DEBUG"):
+				traceback.print_exc()
+
+	# -- accounts
+	def accounts(self, steps):
+		by_platform = {}
+		for s in steps:
+			by_platform.setdefault(s["args"]["platform"], []).append(s)
+		for platform, ss in by_platform.items():
+			accounts = [Account(**{k: v for k, v in s["args"].items()}) for s in ss]
+			for a in accounts:
+				a.fields = [tuple(f) for f in a.fields]
+			try:
+				sessions = state.provision(platform, accounts)
+			except Exception as e:
+				for s in ss:
+					self.ledger.add(type="error", n=s["n"], verb="account", actor=s["actor"], error=repr(e)[:600])
+				traceback.print_exc()
+				continue
+			d = driver(platform)
+
+			def profile(pair):
+				s, sess = pair
+				try:
+					d.update_profile(sess, sess.account)
+					self.ledger.add(type="account", n=s["n"], key=s["actor"], actor_uri=sess.actor_uri, local_id=sess.local_id,
+									profile=True)
+				except Unsupported:
+					self.ledger.add(type="account", n=s["n"], key=s["actor"], actor_uri=sess.actor_uri, local_id=sess.local_id,
+									profile=False)
+				except Exception as e:
+					# the account exists; only its profile is missing, which the checker can tell from profile=False
+					self.ledger.add(type="account", n=s["n"], key=s["actor"], actor_uri=sess.actor_uri, local_id=sess.local_id,
+									profile=False, profile_error=repr(e)[:300])
+			with ThreadPoolExecutor(max_workers=6) as pool:
+				list(pool.map(profile, zip(ss, sessions)))
+
+	# -- groups
+	def v_group(self, s):
+		platform = s["actor"].split("/")[0]
+		sess = self.session(s["actor"])
+		kind, name = s["args"]["kind"], s["args"]["name"]
+		if platform == "privapub":
+			g = driver("privapub").group(sess, name, name.replace("_", " ").title(), community=kind == "community")
+			group = {"kind": kind, "host": "privapub", "id": g["id"], "code": g.get("invitationCode"),
+					 "handle": f"{name}@privapub.test", "owner": s["actor"]}
+		else:
+			c = driver("lemmy").community(sess, name, name.replace("_", " ").title())
+			group = {"kind": kind, "host": "lemmy", "id": c["id"], "handle": f"{name}@lemmy.test", "owner": s["actor"],
+					 "ap_id": c["ap_id"]}
+		self.groups[s["ref"]] = group
+		self.ledger.add(type="group", n=s["n"], ref=s["ref"], group=group)
+
+	# -- the graph
+	def v_follow(self, s):
+		platform = s["actor"].split("/")[0]
+		outcome = driver(platform).follow(self.session(s["actor"]), self.acct(s["args"]["target"]))
+		self.ledger.add(type="relation", n=s["n"], verb="follow", actor=s["actor"], target=s["args"]["target"], state=outcome)
+
+	def _wait_pending(self, s):
+		platform = s["actor"].split("/")[0]
+		d, sess, follower = driver(platform), self.session(s["actor"]), self.acct(s["args"]["target"])
+		for _ in range(30):
+			if follower.lower() in (p.lower() for p in d.pending(sess)):
+				return d, sess, follower
+			time.sleep(1)
+		raise TimeoutError(f"{follower}'s follow request never reached {s['actor']}")
+
+	def v_accept(self, s):
+		d, sess, follower = self._wait_pending(s)
+		d.accept(sess, follower)
+		self.ledger.add(type="relation", n=s["n"], verb="accept", actor=s["actor"], target=s["args"]["target"], state="accepted")
+
+	def v_reject(self, s):
+		d, sess, follower = self._wait_pending(s)
+		d.reject(sess, follower)
+		self.ledger.add(type="relation", n=s["n"], verb="reject", actor=s["actor"], target=s["args"]["target"], state="rejected")
+
+	def v_join(self, s):
+		group = self.groups[s["args"]["group"]]
+		platform = s["actor"].split("/")[0]
+		sess = self.session(s["actor"])
+		how = None
+		if platform == "privapub" and group["host"] == "privapub":
+			pp = driver("privapub")
+			pp.http.post(pp.base + "/clientapi/group/join", ok={200, 201, 204},
+						 headers={"Authorization": f"Bearer {pp.jwt(sess.account.root, sess.account.password)}"},
+						 json={"avatarId": sess.local_id, "invitationCode": group["code"]})
+			how = "invitation"
+		elif platform == "lemmy":
+			lm = driver("lemmy")
+			cid = group["id"] if group["host"] == "lemmy" else lm.resolve_community(sess, "!" + group["handle"])
+			lm.follow_community(sess, cid)
+			how = "community-follow"
+		else:
+			how = driver(platform).follow(sess, group["handle"])
+		self.ledger.add(type="relation", n=s["n"], verb="join", actor=s["actor"], group=s["args"]["group"], state=how)
+
+	def v_approve(self, s):
+		group = self.groups[s["args"]["group"]]
+		member = s["args"]["member"]
+		if member.startswith("privapub/"):
+			self.ledger.add(type="relation", n=s["n"], verb="approve", actor=s["actor"], group=s["args"]["group"],
+							member=member, state="local")
+			return
+		member_uri = self.session(member).actor_uri
+		for _ in range(30):
+			found = podman.mongo(f"db.Follower.findOne({{LocalActorId: '{group['id']}', IsAccepted: false}}, {{ActorURI: 1}})")
+			pending = podman.mongo(f"db.Follower.find({{LocalActorId: '{group['id']}', IsAccepted: false}}, {{ActorURI: 1}}).toArray()") or []
+			if any(p.get("ActorURI") == member_uri for p in pending):
+				break
+			time.sleep(1)
+		else:
+			raise TimeoutError(f"{member}'s request to join {s['args']['group']} never arrived")
+		sess = self.session(s["actor"])
+		pp = driver("privapub")
+		pp.http.post(pp.base + "/clientapi/group/approve", ok={200, 201, 204},
+					 headers={"Authorization": f"Bearer {pp.jwt(sess.account.root, sess.account.password)}"},
+					 json={"avatarId": sess.local_id, "groupId": group["id"], "memberActorURI": member_uri})
+		self.ledger.add(type="relation", n=s["n"], verb="approve", actor=s["actor"], group=s["args"]["group"],
+						member=member, state="accepted")
+
+	# -- content
+	def arrived(self, platform, sess, ref, wait_seconds):
+		"""The local id of `ref` on the actor's server once it is there; None (and a fetch to come) if it never comes."""
+		uri = self.uris[ref]
+		if self.origin.get(ref) == platform:
+			return True
+		d = driver(platform)
+		deadline = time.time() + wait_seconds
+		while True:
+			try:
+				if d.local_status_id(sess, uri):
+					return True
+			except Exception:
+				pass
+			if time.time() >= deadline:
+				return False
+			time.sleep(2)
+
+	def v_post(self, s):
+		platform = s["actor"].split("/")[0]
+		sess = self.session(s["actor"])
+		a = dict(s["args"])
+		fetched = []
+		spec = PostSpec(text=a["text"], visibility=a["visibility"], cw=a.get("cw"), tags=a.get("tags", []),
+						media=a.get("media", []), poll=a.get("poll"), language=a.get("language"), title=a.get("title"),
+						location=a.get("location"), mentions=[self.acct(k) for k in a.get("mentions", [])])
+		for field, key in (("reply_to_uri", "reply_to"), ("quote_uri", "quote")):
+			if a.get(key):
+				if a[key] not in self.uris:
+					raise LookupError(f"{a[key]} was never made")
+				if not self.arrived(platform, sess, a[key], self.timing.get("arrivalSeconds", 20)):
+					fetched.append(a[key])
+				setattr(spec, field, self.uris[a[key]])
+		if a.get("group"):
+			group = self.groups[a["group"]]
+			if platform == "lemmy":
+				lm = driver("lemmy")
+				spec.group = group["id"] if group["host"] == "lemmy" else lm.resolve_community(sess, "!" + group["handle"])
+				if spec.visibility == "community":
+					spec.visibility = "public"
+			else:
+				spec.group = group["id"]
+		made = driver(platform).post(sess, spec)
+		self.uris[s["ref"]] = made.uri
+		self.origin[s["ref"]] = platform
+		self.ledger.add(type="object", n=s["n"], ref=s["ref"], origin=platform, author=s["actor"], uri=made.uri,
+						local_id=made.local_id, fetched_on={platform: fetched} if fetched else {})
+
+	def _target(self, s):
+		platform = s["actor"].split("/")[0]
+		ref = s["args"]["target"]
+		if ref not in self.uris:
+			raise LookupError(f"{ref} was never made")
+		sess = self.session(s["actor"])
+		arrived = self.arrived(platform, sess, ref, self.timing.get("arrivalSeconds", 20))
+		return platform, sess, self.uris[ref], arrived
+
+	def _interaction(self, s, call):
+		platform, sess, uri, arrived = self._target(s)
+		call(driver(platform), sess, uri)
+		self.ledger.add(type="interaction", n=s["n"], verb=s["verb"], actor=s["actor"], ref=s["args"]["target"],
+						arrived_by="delivery" if arrived else "fetch", **{k: v for k, v in s["args"].items() if k != "target"})
+
+	def v_like(self, s):
+		self._interaction(s, lambda d, sess, uri: d.like(sess, uri))
+
+	def v_downvote(self, s):
+		self._interaction(s, lambda d, sess, uri: d.downvote(sess, uri))
+
+	def v_boost(self, s):
+		self._interaction(s, lambda d, sess, uri: d.boost(sess, uri))
+
+	def v_bookmark(self, s):
+		self._interaction(s, lambda d, sess, uri: d.bookmark(sess, uri))
+
+	def v_react(self, s):
+		self._interaction(s, lambda d, sess, uri: d.react(sess, uri, s["args"]["emoji"]))
+
+	def v_vote(self, s):
+		self._interaction(s, lambda d, sess, uri: d.vote(sess, uri, s["args"]["choices"]))
+
+	# -- mutations
+	def v_edit(self, s):
+		platform = s["actor"].split("/")[0]
+		spec = PostSpec(text=s["args"]["text"], cw=s["args"].get("cw"), mentions=[self.acct(k) for k in s["args"].get("mentions", [])])
+		driver(platform).edit(self.session(s["actor"]), self.uris[s["args"]["target"]], spec)
+		self.ledger.add(type="mutation", n=s["n"], verb="edit", actor=s["actor"], ref=s["args"]["target"], text=s["args"]["text"])
+
+	def v_delete(self, s):
+		platform = s["actor"].split("/")[0]
+		driver(platform).delete(self.session(s["actor"]), self.uris[s["args"]["target"]])
+		self.ledger.add(type="mutation", n=s["n"], verb="delete", actor=s["actor"], ref=s["args"]["target"])
+
+	def v_block(self, s):
+		platform = s["actor"].split("/")[0]
+		driver(platform).block(self.session(s["actor"]), self.acct(s["args"]["target"]))
+		self.ledger.add(type="relation", n=s["n"], verb="block", actor=s["actor"], target=s["args"]["target"], state="blocking")
+
+	def v_mute(self, s):
+		platform = s["actor"].split("/")[0]
+		driver(platform).mute(self.session(s["actor"]), self.acct(s["args"]["target"]))
+		self.ledger.add(type="relation", n=s["n"], verb="mute", actor=s["actor"], target=s["args"]["target"], state="muting")
+
+	def v_report(self, s):
+		platform = s["actor"].split("/")[0]
+		uris = [self.uris[r] for r in s["args"]["refs"] if r in self.uris]
+		driver(platform).report(self.session(s["actor"]), self.acct(s["args"]["target"]), uris, s["args"]["comment"])
+		self.ledger.add(type="report", n=s["n"], actor=s["actor"], target=s["args"]["target"], refs=s["args"]["refs"])
+
+
+def run_dir(name, new=False):
+	os.makedirs(OUT, exist_ok=True)
+	latest = os.path.join(OUT, f"{name}.latest")
+	if not new and os.path.exists(latest):
+		with open(latest) as f:
+			return f.read().strip()
+	d = os.path.join(OUT, f"{name}-{time.strftime('%Y%m%d-%H%M%S')}")
+	os.makedirs(d, exist_ok=True)
+	with open(latest, "w") as f:
+		f.write(d)
+	return d
+
+
+def main(args):
+	spec = gen.load(args[0] if args else "village")
+	resume = "--resume" in args
+	only = None
+	for a in args:
+		if a.startswith("--only="):
+			only = a.split("=", 1)[1].split(",")
+	d = run_dir(spec["name"], new=not resume and not only)
+	with open(os.path.join(d, "spec.json"), "w") as f:
+		json.dump(spec, f, indent=1)
+	print(f"run: {d}")
+	return Seeder(spec, d, only).run()
+
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]))

tools/pasture/town/selftest.py

@@ -0,0 +1,154 @@
+"""Each driver against its own server, before any federation is trusted: two accounts of the platform make, like,
+boost, reply to, vote on, edit and delete objects, and the database view (`stored`) must agree with what was done and
+with the API. A failing selftest means the driver is wrong, not the federation."""
+import time
+import traceback
+
+from dialects import driver
+from dialects.base import Account, PostSpec, Unsupported
+import state
+
+PASSWORD = "Town-Selftest-Pass-7!"
+
+
+def wait(fn, tries=20, delay=1.0):
+	for _ in range(tries):
+		value = fn()
+		if value:
+			return value
+		time.sleep(delay)
+	return fn()
+
+
+class Result:
+	def __init__(self, peer):
+		self.peer = peer
+		self.passed = 0
+		self.failed = 0
+		self.skipped = 0
+
+	def check(self, cond, what):
+		if cond:
+			self.passed += 1
+			print(f"  ok   {what}")
+		else:
+			self.failed += 1
+			print(f"  FAIL {what}")
+
+	def skip(self, what):
+		self.skipped += 1
+		print(f"  n/a  {what}")
+
+
+def accounts(peer):
+	root = "st_root" if peer == "privapub" else None
+	# Mastodon reserves names that contain its own, so the platform goes in as its two-letter code
+	import gen
+	short = gen.SHORT.get(peer, peer[:2])
+	return [Account(peer, f"st_a_{short}", PASSWORD, name="Selftest A", bio="selftest", root=root,
+					fields=[("site", "pasture")], avatar_seed=f"st-a-{peer}"),
+			Account(peer, f"st_b_{short}", PASSWORD, name="Selftest B", root=root)]
+
+
+def run(peers):
+	total_failed = 0
+	for peer in peers:
+		print(peer)
+		r = Result(peer)
+		try:
+			if peer == "lemmy":
+				lemmy(r)
+			else:
+				microblog(peer, r)
+		except Exception as e:
+			r.check(False, f"selftest stopped: {e!r}")
+			traceback.print_exc()
+		print(f"  {r.passed} passed, {r.failed} failed, {r.skipped} not applicable")
+		total_failed += r.failed
+	return 1 if total_failed else 0
+
+
+def microblog(peer, r):
+	d = driver(peer)
+	a, b = state.provision(peer, accounts(peer))
+	r.check(a.token and b.token, "both accounts signed in")
+	try:
+		d.update_profile(a, a.account)
+		r.check(True, "profile updated (name, bio, fields, avatar)")
+	except Unsupported as e:
+		r.skip(str(e))
+
+	made = d.post(a, PostSpec(text=f"selftest post on {peer} (st1)", tags=["townselftest"]))
+	row = wait(lambda: d.stored([made.uri])[made.uri].exists and d.stored([made.uri])[made.uri])
+	r.check(row and row.exists, f"the post is in the database under its uri {made.uri}")
+	r.check(row and str(row.local_id) == str(made.local_id), f"database id {row and row.local_id} is the API id {made.local_id}")
+	r.check(row and row.visibility == "public", f"stored visibility public (got {row and row.visibility})")
+
+	d.like(b, made.uri)
+	r.check(wait(lambda: d.stored([made.uri])[made.uri].likes == 1), "a like counts once")
+	try:
+		d.boost(b, made.uri)
+		r.check(wait(lambda: d.stored([made.uri])[made.uri].boosts == 1), "a boost counts once")
+	except Unsupported as e:
+		r.skip(str(e))
+
+	reply = d.post(b, PostSpec(text=f"selftest reply on {peer} (st2)", reply_to_uri=made.uri))
+	rrow = wait(lambda: d.stored([reply.uri])[reply.uri].exists and d.stored([reply.uri])[reply.uri])
+	r.check(rrow and rrow.parent_uri == made.uri, f"the reply names its parent (got {rrow and rrow.parent_uri})")
+	r.check(wait(lambda: (d.stored([made.uri])[made.uri].replies or 0) >= 1), "the parent counts the reply")
+
+	poll = d.post(a, PostSpec(text=f"selftest poll on {peer} (st3)", poll={"options": ["tea", "coffee"], "expires_in": 86400}))
+	d.vote(b, poll.uri, [1])
+	r.check(wait(lambda: (d.stored([poll.uri])[poll.uri].votes or [None, None])[1] == 1),
+			f"a vote counts on the second option (got {d.stored([poll.uri])[poll.uri].votes})")
+
+	private = d.post(a, PostSpec(text=f"selftest followers-only on {peer} (st4)", visibility="followers"))
+	prow = d.stored([private.uri])[private.uri]
+	r.check(prow.visibility == "followers", f"stored visibility followers (got {prow.visibility})")
+	seen = d.seen(b, [made.uri, private.uri])
+	r.check(seen[made.uri], "a non-follower sees the public post")
+	r.check(not seen[private.uri], "a non-follower does not see the followers-only post")
+
+	try:
+		d.edit(a, made.uri, PostSpec(text=f"selftest post on {peer}, edited (st1)"))
+		r.check(wait(lambda: d.stored([made.uri])[made.uri].edited), "the edit is recorded")
+	except Unsupported as e:
+		r.skip(str(e))
+
+	d.delete(b, reply.uri)
+	gone = wait(lambda: (lambda x: not x.exists or x.deleted)(d.stored([reply.uri])[reply.uri]))
+	r.check(gone, "a deleted reply is gone or tombstoned")
+
+	try:
+		d.react(b, made.uri, "🎉")
+		r.check(wait(lambda: (d.stored([made.uri])[made.uri].reactions or {}).get("🎉", 0) >= 1), "a reaction is stored")
+	except Unsupported as e:
+		r.skip(str(e))
+
+
+def lemmy(r):
+	d = driver("lemmy")
+	a, b = state.provision("lemmy", accounts("lemmy"))
+	r.check(a.token and b.token, "both accounts signed in")
+	name = f"stc{int(time.time())}"
+	c = d.community(a, name, "Selftest community")
+	thread = d.post(a, PostSpec(text="selftest thread body (st1)", title="Selftest thread", group=c["id"]))
+	row = d.stored([thread.uri])[thread.uri]
+	r.check(row.exists and row.local_id == thread.local_id, f"the thread is in the database ({row.local_id})")
+	d.like(b, thread.uri)
+	r.check(wait(lambda: d.stored([thread.uri])[thread.uri].likes >= 2), "an upvote counts (with the author's own)")
+	comment = d.post(b, PostSpec(text="selftest comment (st2)", reply_to_uri=thread.uri))
+	crow = d.stored([comment.uri])[comment.uri]
+	r.check(crow.parent_uri == thread.uri, f"the comment names its thread (got {crow.parent_uri})")
+	reply = d.post(a, PostSpec(text="selftest comment reply (st3)", reply_to_uri=comment.uri))
+	rrow = d.stored([reply.uri])[reply.uri]
+	r.check(rrow.parent_uri == comment.uri, f"a nested comment names its parent comment (got {rrow.parent_uri})")
+	d.downvote(a, comment.uri)
+	r.check(wait(lambda: (d.stored([comment.uri])[comment.uri].reactions or {}).get("down") == 1), "a downvote counts")
+	d.edit(a, thread.uri, PostSpec(text="selftest thread body, edited (st1)"))
+	r.check(wait(lambda: d.stored([thread.uri])[thread.uri].edited), "the edit is recorded")
+	pm = d.post(a, PostSpec(text="selftest private message (st4)", visibility="direct",
+							mentions=[f"{b.account.username}@lemmy.test"]))
+	r.check(d.stored([pm.uri])[pm.uri].exists, "a private message is in the database")
+	d.delete(a, reply.uri)
+	r.check(wait(lambda: d.stored([reply.uri])[reply.uri].deleted), "a deleted comment is marked deleted")

tools/pasture/town/specs/village.json

@@ -0,0 +1,31 @@
+{
+	"schema": "pasture-town/1",
+	"name": "village",
+	"seed": 20261004,
+	"peers": {
+		"privapub": {"roots": 2, "personas": [2, 3], "circles": 1, "communities": 1},
+		"gts": {"accounts": 3},
+		"mastodon": {"accounts": 4},
+		"misskey": {"accounts": 3},
+		"sharkey": {"accounts": 2},
+		"akkoma": {"accounts": 3},
+		"lemmy": {"accounts": 3, "communities": 1}
+	},
+	"profiles": {"locked": 0.2, "bot": 0.05, "fields": [0, 3], "avatar": 0.9, "header": 0.5, "bioWords": [5, 20],
+		"langs": {"en": 70, "it": 15, "ja": 8, "ar": 7}},
+	"graph": {"follows": [3, 7], "mutual": 0.4, "pending": 0.1, "rejected": 0.05},
+	"circleMembers": 4,
+	"communityMembers": 6,
+	"content": {
+		"posts": 80,
+		"communityPosts": 3,
+		"mix": {"text": 40, "cw": 8, "tags": 10, "mention": 10, "image": 8, "poll": 6, "quote": 5},
+		"visibility": {"public": 50, "unlisted": 12, "followers": 18, "direct": 10, "circle": 6, "located": 4},
+		"replyRounds": 3,
+		"replies": 0.45,
+		"deeperReplies": 0.35
+	},
+	"interactions": {"likes": [0, 4], "boosts": [0, 2], "react": 0.3, "vote": 0.6, "bookmark": 0.1},
+	"mutations": {"edit": 0.08, "delete": 0.04, "blocks": 2, "mutes": 2, "reports": 1},
+	"time": {"roundSettleSeconds": 20, "settleSeconds": 60, "deadlineSeconds": 240}
+}

tools/pasture/town/state.py

@@ -0,0 +1,67 @@
+"""What the town keeps between commands in .state/town/ (removed with the pasture): every account it made and the
+session (token, local id, actor) each one signs in with, so a second command never makes or signs in anyone again."""
+import json
+import os
+import threading
+
+from dialects import driver
+from dialects.base import Account, Session
+
+ROOT = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), ".state", "town")
+_lock = threading.Lock()
+
+
+def _path(name):
+	os.makedirs(ROOT, exist_ok=True)
+	return os.path.join(ROOT, name)
+
+
+def load(name, default):
+	try:
+		with open(_path(name)) as f:
+			return json.load(f)
+	except (FileNotFoundError, ValueError):
+		return default
+
+
+def save(name, value):
+	with _lock:
+		tmp = _path(name) + ".tmp"
+		with open(tmp, "w") as f:
+			json.dump(value, f, indent=1, default=str)
+		os.replace(tmp, _path(name))
+
+
+def remember(sessions):
+	stored = load("sessions.json", {})
+	for s in sessions:
+		stored[f"{s.account.platform}/{s.account.username}"] = {
+			"account": s.account.__dict__, "token": s.token, "local_id": s.local_id, "actor_uri": s.actor_uri}
+	save("sessions.json", stored)
+
+
+def session(platform, username):
+	stored = load("sessions.json", {}).get(f"{platform}/{username}")
+	if stored is None:
+		raise KeyError(f"the town has no account {username} on {platform}")
+	return Session(Account(**stored["account"]), stored["token"], stored["local_id"], stored["actor_uri"])
+
+
+def sessions(platform=None):
+	out = []
+	for key, stored in load("sessions.json", {}).items():
+		if platform and not key.startswith(platform + "/"):
+			continue
+		out.append(Session(Account(**stored["account"]), stored["token"], stored["local_id"], stored["actor_uri"]))
+	return out
+
+
+def provision(platform, accounts):
+	"""Sessions for `accounts`, making and signing in only those the town does not already have."""
+	known = {s.account.username: s for s in sessions(platform)}
+	missing = [a for a in accounts if a.username not in known]
+	if missing:
+		made = driver(platform).provision(missing)
+		remember(made)
+		known.update({s.account.username: s for s in made})
+	return [known[a.username] for a in accounts]

tools/pasture/town/tests/test_town.py

@@ -0,0 +1,79 @@
+"""The town's own logic, with no pasture: python3 -m unittest discover -s tools/pasture/town/tests"""
+import os
+import sys
+import unittest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import check  # noqa: E402
+import gen  # noqa: E402
+from core.rng import Rng  # noqa: E402
+
+
+class RngTests(unittest.TestCase):
+	def test_the_same_seed_and_name_give_the_same_stream(self):
+		a, b = Rng(7, "x"), Rng(7, "x")
+		self.assertEqual([a.next64() for _ in range(5)], [b.next64() for _ in range(5)])
+
+	def test_substreams_do_not_disturb_each_other(self):
+		root = Rng(7)
+		before = Rng(7).sub("graph").next64()
+		root.sub("content").next64()
+		self.assertEqual(before, root.sub("graph").next64())
+
+	def test_below_stays_in_range(self):
+		r = Rng(1)
+		self.assertTrue(all(0 <= r.below(3) < 3 for _ in range(1000)))
+
+
+class PlanTests(unittest.TestCase):
+	def test_a_spec_always_plans_the_same(self):
+		spec = gen.load("village")
+		self.assertEqual(gen.digest(gen.plan(spec)), gen.digest(gen.plan(spec)))
+
+	def test_every_post_names_its_marker_and_every_reply_a_parent_planned_before_it(self):
+		planner = gen.Planner(gen.load("village"))
+		steps = planner.plan()
+		made = set()
+		for s in steps:
+			if s["verb"] != "post":
+				continue
+			self.assertIn(f"({s['ref']})", s["args"]["text"])
+			if s["args"].get("reply_to"):
+				self.assertIn(s["args"]["reply_to"], made)
+			made.add(s["ref"])
+
+	def test_lemmy_accounts_never_follow_people_and_platforms_get_only_what_they_can_do(self):
+		steps = gen.plan(gen.load("village"))
+		for s in steps:
+			platform = s["actor"].split("/")[0]
+			if s["verb"] == "follow":
+				self.assertNotEqual("lemmy", platform)
+			if s["verb"] in ("react", "edit", "vote", "boost"):
+				self.assertIn({"react": "react", "edit": "edit", "vote": "vote", "boost": "boost"}[s["verb"]], gen.CAPS[platform])
+
+	def test_a_stored_plan_rebuilds_the_same_world(self):
+		planner = gen.Planner(gen.load("village"))
+		planner.plan()
+		stored = gen.Stored(gen.snapshot(planner))
+		self.assertEqual(planner.objects.keys(), stored.objects.keys())
+		self.assertEqual(planner.objects["p1"]["mentions"], stored.objects["p1"]["mentions"])
+
+
+class GapTests(unittest.TestCase):
+	def test_a_known_gap_turns_failures_into_xfail_and_passes_into_xpass(self):
+		gaps = [{"id": "G-9", "match": {"feature": r"count\.like\..*", "observer": "privapub"}, "status": "open"}]
+		checks = check.classify([
+			{"feature": "count.like.followers", "observer": "privapub", "ok": False},
+			{"feature": "count.like.public", "observer": "privapub", "ok": True},
+			{"feature": "count.like.public", "observer": "gts", "ok": False},
+		], gaps)
+		self.assertEqual(["xfail", "xpass", "fail"], [c["status"] for c in checks])
+
+	def test_a_closed_gap_matches_nothing(self):
+		gaps = [{"id": "G-9", "match": {"feature": ".*"}, "status": "closed"}]
+		self.assertEqual("fail", check.classify([{"feature": "x", "ok": False}], gaps)[0]["status"])
+
+
+if __name__ == "__main__":
+	unittest.main()

tools/pasture/town/town.py

@@ -0,0 +1,62 @@
+"""The town's command line. Run through tools/pasture/town.sh.
+
+  selftest <peer>...            each driver against its own server: accounts, posts, likes, replies, polls, edits,
+                                deletes, and that its database view agrees with its API
+  drive <peer> <user> <verb> '<json>'   one action as one town account, printed as JSON
+  stored <peer> <uri>...        what a peer holds of each object (from its database, never fetching)
+  seen <peer> <user> <uri>...   what one account can see of each object
+  plan <spec>                   the deterministic plan of a spec, printed as JSON lines
+  seed <spec>                   replays a spec's plan against the pasture, writing out/<run>/ledger.jsonl
+  check [run]                   sweeps every peer for what the ledger expects
+  report [run]                  out/<run>/report.html
+  backlog [run] [--write]       docs/INTEROP-BACKLOG.md from the latest results
+"""
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, HERE)
+
+from dialects import DRIVERS, driver  # noqa: E402
+from dialects.base import Account, PostSpec  # noqa: E402
+import state  # noqa: E402
+
+
+def main(argv):
+	if not argv or argv[0] in ("-h", "--help", "help"):
+		print(__doc__)
+		return 0
+	cmd, args = argv[0], argv[1:]
+	if cmd == "selftest":
+		import selftest
+		return selftest.run(args or [p for p in DRIVERS])
+	if cmd == "drive":
+		peer, user, verb = args[0], args[1], args[2]
+		params = json.loads(args[3]) if len(args) > 3 else {}
+		s = state.session(peer, user)
+		d = driver(peer)
+		if verb == "post":
+			made = d.post(s, PostSpec(**params))
+			print(json.dumps(made.__dict__))
+		else:
+			result = getattr(d, verb)(s, **params)
+			print(json.dumps(result if not hasattr(result, "__dict__") else result.__dict__, default=str))
+		return 0
+	if cmd == "stored":
+		rows = driver(args[0]).stored(args[1:])
+		print(json.dumps({u: {k: v for k, v in r.__dict__.items() if k != "raw"} for u, r in rows.items()}, default=str))
+		return 0
+	if cmd == "seen":
+		s = state.session(args[0], args[1])
+		print(json.dumps(driver(args[0]).seen(s, args[2:])))
+		return 0
+	if cmd in ("plan", "seed", "check", "report", "backlog", "gaps", "expect", "load"):
+		module = __import__(cmd if cmd not in ("plan",) else "gen")
+		return module.main(args)
+	print(f"unknown command {cmd}", file=sys.stderr)
+	return 2
+
+
+if __name__ == "__main__":
+	sys.exit(main(sys.argv[1:]) or 0)