SocialPub/docs/ROADMAP.md

# PrivaPub roadmap

Written 2026-10-01 from the original 2023 code, the decePubClient UI, a federation gap audit of commit `075c222`, research on .NET ActivityPub libraries, and the owner's decisions. Each phase ends in a tagged deploy plus verification; tick phases off here as they land.

## Status

- [x] P0 Security baseline: v1.1.0, deployed and verified 2026-10-01
- [x] P1.1 Infrastructure and content model: v1.2.0, deployed and verified 2026-10-01
- [x] P1.2 Social graph and timelines: v1.3.0, deployed and verified 2026-10-01
- [x] P2 Mastodon client API: v1.4.0, deployed 2026-10-01; OAuth and the anonymous API verified on production, the signed-in API verified locally (no real-client login on production yet)
- [x] P3 Social features: v1.5.0, deployed 2026-10-01; media, proxy, blocks, mutes, bookmarks, pins and reports verified by tests and locally (no upload on production yet)
- [x] P4 Groups and privacy features: v1.6.0, deployed 2026-10-01; communities, circles and local-only located posts verified by tests. v1.6.1 adds the pasture (`tools/pasture/`): live interop with GoToSocial 0.22.1 passes all 25 checks, three runs in a row. Lemmy and a live Mastodon circle member are not run yet; the pasture has GoToSocial only
- [x] P5 Lose nothing: v1.7.0 to v1.9.1, deployed 2026-10-01; parsing, typed details, provenance, downvotes, tombstones, federated blocks, all checked live against GoToSocial. Book reviews and forum threads keep their raw form only (typed in P7 and P8)
- [x] P6 Emoji, polls, quotes, reactions, cards, players: v1.10.0 to v1.15.0, deployed 2026-10-01; polls, link cards, ranged video streaming and quote policies checked live against GoToSocial
- [ ] T/M Test sweep, the full pasture and the interaction ledger (see "Sweep and statistics" below). T1–T4: v1.15.1, deployed and verified 2026-10-03 (CI runs all tests on a throwaway mongod; nothing answers 500). M1–M6: v1.16.0, deployed and verified 2026-10-03 (the interaction ledger records every inbox answer, handler verdict, delivery attempt and outbound request; the GoToSocial pasture passes 33/33 with no names in the statistics). T5–T9, T11 and M7–M11: v1.17.0, 2026-10-03:
  - 639 tests, now over HTTP too;
  - daily rollups, every touched server described and located, the admin statistics API, the opt-in crawler and `/stargazing`;
  - the pasture as plugins: GoToSocial 37/37, Mastodon 49 plus 2 expected failures.

  T10 and T12–T14: v1.17.1, deployed and verified 2026-10-03. The pasture runs all six peers in one pass:
  - GoToSocial 54 plus 1 expected failure;
  - Mastodon 49 plus 2;
  - Misskey 35, Sharkey 40, Akkoma 44;
  - Lemmy 20 plus 3.

  The expected failures are circle posts on GoToSocial and Mastodon, inbound Block, and Lemmy's relayed votes and
  removals (P7). It found three bugs, all fixed:
  - quick GoToSocial edits were lost;
  - followers-only posts arrived as DMs on Pleroma and Akkoma;
  - Akkoma's open polls showed as ended and refused votes.

  Still open: installing the geolocation timer on Max (`deploy/max/setup.sh`, as root), and the smoke persona's
  `PRIVAPUB_SMOKE_TOKEN` secret, without which the deploy skips the signed-in half of its Mastodon API check.
- [ ] P7 Threads, communities, moderation, the social graph
- [ ] P8 Signatures, discovery, the long tail

## Intent

It is a self-hosted, Pleroma-like microblogging server for privacy-minded individuals and small communities. It
federates with Mastodon, Pleroma/Akkoma, Misskey and GoToSocial (`AvatarServer` enum). Its distinguishing idea is
**one private login (RootUser) owning several public personas (Avatars), each a separate actor with its own keys**.
The rename SocialPub → PrivaPub happened in the commit that added the private avatar service.

Its privacy additions on top of the usual fediverse features:
- **Location-ranged posts:** `Post.Location` and `RangeKm` (5 km default), currently unused.
- **Contacts a persona chooses to share:** `Avatar.SharedPersonalContacts`, plus a root contact book (`ContactItem`).
- **Private notes:** about other accounts (`RootUserNote`) and on one's own avatar (`PersonalNote`).
- **Invitation-gated groups and DM groups.**
- **Signup without email:** email is only for password recovery.

The client (decePubClient) is a near copy of **Pleroma-FE plus Pleroma's admin panel**: a visibility picker, a
subject line, Plain/HTML/Markdown content types, media with alt text, boosts and likes, threads, mutes and blocks, data
import/export, and admin sections for users, reports, emoji, MRF policies and uploads. Today it shows mock data and
calls nothing.

Leftovers of the collAnon template, not intent:
- the discussion and confrontation resources;
- collAnon's invitation flow;
- QR scanning;
- the "collAnon support" mail sender.

The finishing pass reinterpreted the owner's never-written `Group` and `DmGroup` as an ActivityPub Group actor with an
invitation code and a direct-message conversation. The owner confirmed it on 2026-10-01, with groups split into communities
and circles (see Owner decisions).

**Status at HEAD:**

| Status | Features |
|---|---|
| Works | Accounts and JWT, several avatars, per-avatar actors and keys, posts, replies, CW, delete, DMs, groups and invitations, WebFinger, NodeInfo, signed inbox for Follow/Undo/Create/Delete/Update, delivery queue |
| Missing | Following remote accounts (no outgoing Follow), home/local/federated timelines, notifications, likes, boosts, visibility other than public, media, polls, edits by the author, profile Update federation, mutes, blocks, reports (Flag), Move, custom emoji, the location and contact features, admin and moderation, and a client that talks to the server |

**Persona separation is breached in three places today:**
- invitation signup names the avatar after the root username;
- logs pair root IDs with IP addresses;
- NodeInfo counts avatars as users.

## Where it stood

**Works:**
- WebFinger, including `acct:` for groups
- NodeInfo 2.0
- Person, Group and Application actors with SPKI keys
- Outbound cavage signatures (every fetch is signed, so authorized-fetch servers work)
- Inbound cavage verification, with key refetch and a key-owner check
- Shared inbox, durable delivery with backoff
- Follow with auto-Accept or manual approval for groups, and Undo
- Inbound public, group and DM Creates
- Delete and Update restricted to the author
- DMs both ways, with Mention tags

**Critical (security):**
- **S1, actor/key cache poisoning:** `RemoteActorService.Upsert` and `GetActorByKeyId` accept any document under its
  own `id`, with no origin, `publicKey.id` or `owner` checks. Any remote actor can be impersonated.
- **S2:** no check that an object's `id` host matches its actor's.
- **S3, SSRF:** only string checks. Names resolving to private addresses, redirects and rebinding all get through, and
  a request can trigger it before authentication.
- **S4:** unbounded fetch size, and 500s on unexpected content.
- **S5:** remote HTML is stored raw, with no format flag.
- **S6:** loose signature freshness.
- **S8:** DM conversation injection.
- **S9:** "private" groups federate as Public.
- **S14:** the "admin" username grants admin, and Swagger is open in production.

**Blockers:**
- **O1:** no outgoing Follow, so no home timeline.
- **O2, O3:** replies and mentions aren't delivered to their targets.
- **I1:** inbound replies and mentions are stored but invisible, and there are no notifications.
- **G1:** groups Announce the object URI instead of the activity, so Lemmy and other FEP-1b12 software see empty
  communities.
- **K7:** no media, timelines, likes or boosts for local users.

**Important:**
- **Data:** no indexes or unique constraints, upsert races (K1-K4).
- **Inbox:** processing runs synchronously with no idempotency store (I9, I10).
- **Delivery:** one serial worker, and a single poisoned row can stall the queue (L1-L3).
- **Objects:**
  - Edits and profile updates don't federate (O7).
  - No 410 Tombstones (A7).
  - Content warning without a title (O5).
  - Titles are lost on Mastodon (O6).
  - Visibility modes are missing (O4).
- **Interactions:** likes, boosts and reports are dropped (I2, I3).
- **Media:** inbound attachments are dropped, and there's no media proxy (I6, S13).
- **Actor profile:**
  - Actor `url` serves JSON to browsers (A1).
  - No `attachment` fields, Move or `alsoKnownAs` (A2, A3).
  - No locked or undiscoverable accounts (A4).
- **Signatures:** inbound RFC 9421 and Content-Digest (H1, H2).

## Owner decisions (2026-10-01)

| Question | Decision |
|---|---|
| Client interface | **Mastodon client API**, so Tusky, Elk, Phanpy, Ivory and the official apps work. Each avatar is its own Mastodon account: at OAuth authorize, the logged-in RootUser picks the avatar the token is for. PrivaPub-only features (avatars, groups, contacts, range posts) stay on `/clientapi`. Moving decePubClient onto the Mastodon API is out of scope. |
| Personas | **Unlinkable to other users and servers.** The admin can still see the link in the database. No root IDs next to IPs in logs, NodeInfo counts nothing that links personas, blocks, mutes and notifications are per avatar, and invitation signup no longer names the avatar after the root username. |
| Location-ranged posts | **Local only, never federated.** Shown to local users within the radius, with coordinates rounded on storage. |
| Groups | **Per group, two kinds.** A *community* federates per FEP-1b12, Lemmy-compatible: it Announces the activity, uses `audience`, and accepts posts from non-followers. A *circle* is invitation-only: posts are addressed to the members collection, and objects are served only to signed requests from members. |

### Owner decisions on what PrivaPub reveals (2026-10-01, from `docs/INTEROP.md` §6)

| Question | Decision |
|---|---|
| Link previews | **The server fetches the linked page itself.** It does this only for public posts, after a short random delay, once per link for the whole server (a shared cache, so a fetch never points at one persona), and when the post arrives, never when someone reads it. Cards built from the post's own data are used first. The client never contacts the site. |
| Blocks | **Federated.** A persona's block is sent to the blocked account's server as `Block`, and an unblock as `Undo{Block}`. Their server enforces it too, and they can learn they were blocked. This replaces the earlier "blocks never federate" rule. |
| Website authorship (`attributionDomains`, `fediverse:creator`) | **Off.** It is never emitted, not even as a per-persona option, because it would publicly tie a persona to a website. |
| PeerTube views | **Never sent.** The remote video file is still downloaded through our proxy when it is played; that cannot be avoided without pre-downloading video. |
| Bluesky bridging (Bridgy Fed) | **Allowed per persona**, with a clear warning that the posts become far more widely copied. Leaving the bridge must work, through a federated `Block` sent to the bridge. Persona creation dates are moved back by a random number of days, so personas created the same day no longer share a date. |
| Reactions and votes | **Public, with a one-time notice.** The client tells each persona once, before its first reaction or vote, that these are public and visible as that persona. |
| Who may quote a persona (default) | **Anyone, automatically**, as on Mastodon, for public and unlisted posts only. Each persona can change its default (followers only, or nobody) and each post can be changed; a granted quote can be revoked. Followers-only posts and DMs can never be quoted. |
| Quote permission address | `/peasants/{name}/parrot-licences/{id}` |

### Owner decisions on statistics (2026-10-03)

| Question | Decision |
|---|---|
| Fediverse statistics | **Recorded now, published later as per-server aggregates only.** The following are each kept for 90 days as an event that names the remote *server*, never a remote account or a local persona: every inbox answer, every processed activity, every delivery attempt and every outbound request. Each day folds into per-server counters kept indefinitely, plus weekly server snapshots. A future public page, `/stargazing`, shows per-server aggregates for educational use, never per account and never per persona. |
| Remote accounts in statistics | **Never stored.** Distinct accounts per server per day are counted with a keyed hash. Its key is made for that day, kept only until the day's rollup and then destroyed, so the hashes can be neither reversed nor linked across days. |
| Local side in statistics | **Only the kind of local actor** (person, group, application), **and only on public and unlisted traffic.** DMs, followers-only and circle traffic are one "private" class, never broken out per server in public. Circles are never named, whether as a kind or as a reason. Fetches of our own documents are counted per day, never per server. |
| Reading-driven traffic | **Counted per day, never logged per event:** the media proxy, lookups a client asks for, and the client API per endpoint group (admin only). Client app names are not recorded. |
| Describing servers | **Every server we exchange activities with is described weekly**, from its NodeInfo (including the user counts it publishes) and its Mastodon instance API, never its contact account. These requests are unsigned, because they are not ActivityPub documents. Never on read. |
| Server locations | **City and network (ASN) from the offline DB-IP Lite databases** (CC BY 4.0, attributed), downloaded monthly outside the app. The location comes from the address we connected to; an inbound sender's address is never recorded, and no address is stored. In public: city and network only for servers reporting at least 10 users and not behind a CDN; the country otherwise; only the CDN's name for CDN-fronted servers. The admin sees everything. |
| Crawler | **Off by default** (`Statistics:Crawler:Enabled`). When on, it identifies as `PrivaPub-Stargazer/<version> (+https://privapub.thepra.dev/stargazing)`, where `/stargazing` explains it and how to opt out. It honours robots.txt (an unreachable robots.txt means "keep out") and domain blocks. It visits one server a minute, each at most weekly, and at most 5000 servers. It reads only robots.txt, NodeInfo, the instance API and the peers list, never accounts, posts or directories. Crawled servers stay marked as crawled. |
| A remote account deletes itself | **Its posts are kept but hidden everywhere** (`Post.AuthorGone`): from timelines, profiles, search and lookups by id. Its follows and timeline rows go, as before. |
| Signed-in smoke check in production | **An undiscoverable persona**, whose read-only token is the Gitea secret `PRIVAPUB_SMOKE_TOKEN`; the deploy checks `verify_credentials`, home and notifications with it. The owner creates both. |

## Libraries (researched; no maintained .NET ActivityPub library exists, so Letterbook and Iceshrimp.NET both wrote their own)

| Area | Choice |
|---|---|
| AS2 / ActivityPub model | Own thin layer on `System.Text.Json.Nodes`: inbound helpers for single-or-array values, id-or-object, `type` arrays and the three Public forms; outbound builders with Mastodon's `@context`. No JSON-LD processing; fetch from the origin instead of verifying LD signatures. |
| HTTP signatures | Keep own draft-cavage (sign and verify). Add **NSign 1.2.5** (`NSign.AspNetCore`) to verify inbound RFC 9421 and Content-Digest. Always answer a bad signature with 401, never 500. |
| HTML | **HtmlSanitizer 9.x** with Mastodon's allowlist (tags `p br span a del pre code em strong b i u ul ol li blockquote`, attributes `href rel class`, microformat classes, a scheme allowlist, forced `rel=nofollow noopener noreferrer`) |
| Markdown | **Markdig 1.4**: `DisableHtml`, autolinks, custom inline parsers for `@user@domain` and `#tag` emitting Mastodon's h-card and hashtag markup |
| Media | **NetVips** (+NetVips.Native) for images: autorotate, strip EXIF/GPS, thumbnails. **Blurhash.Core**, **FFMpegCore** for video. Own `IMediaStore` (local disk now). Not ImageSharp 4, whose license key is enforced at build. |
| OAuth | **OpenIddict 7.7 + OpenIddict.MongoDb**: authorization code with PKCE, client_credentials, `oob` redirect, dynamic apps from `POST /api/v1/apps`, reference tokens |
| Jobs | Own Mongo job collection for both inbox processing and delivery: `FindOneAndUpdate` leases, a `Channel` wake-up, per-host circuit breaker, Mastodon's backoff (`n⁴+15+jitter`, 16 tries). No MassTransit (commercial from v9, no Mongo transport). |
| Tests | New xUnit project with fixture JSON captured from Mastodon, GoToSocial, Misskey, Lemmy and Akkoma; interop checked with Fediverse Pasture, verify.funfedi.dev and activitypub.academy |

**FEPs, now:** FEP-f1d5 (NodeInfo 2.0 and 2.1), FEP-67ff (FEDERATION.md), FEP-8fcf, FEP-5feb `indexable`,
FEP-7628 Move, FEP-2c59, FEP-044f quotes (parse, plus `interactionPolicy`), FEP-1b12 groups.

**FEPs, later:** FEP-521a, FEP-8b32, FEP-e232, relays (FEP-ae0c).

**FEPs, skipped:** FEP-c390, FEP-ef61, FEP-fb2a, FEP-844e.

## Route names (deliberate; part of the project's character)

The odd names are the owner's and stay. Every new actor-scoped route follows the same theme. The names marked *new*
were agreed on 2026-10-01.

| Thing | Route | Status |
|---|---|---|
| Actor | `/peasants/{name}` | kept (`/users/{name}` 301-redirects here) |
| Inbox | `/peasants/{name}/mouth` | kept |
| Outbox | `/peasants/{name}/anus` | kept |
| Shared inbox | `/human-centipede` (and `/peasants/{name}/human-centipede`) | kept |
| Token refresh (client API) | `/clientapi/user/sniff/again` | kept |
| Followers | `/peasants/{name}/groupies` | *new*, replaces `/followers` in P1 |
| Following | `/peasants/{name}/stalking` | *new*, replaces `/following` in P1 |
| Notes (objects) | `/peasants/{name}/scribbles/{id}` | *new*, replaces `/posts/{id}` in P1 |
| Activities | `/peasants/{name}/grunts/{id}` | *new*, replaces `/activities/{id}` in P1 |
| Replies, likes, shares collections | `/scribbles/{id}/gossip`, `/drool`, `/echoes` | *new* |
| Featured (pins) | `/peasants/{name}/trophies` | *new* |
| Featured tags | `/peasants/{name}/tattoos` | *new* |
| Group members | `/peasants/{group}/flock` | *new* |
| Group moderators | `/peasants/{group}/wardens` | *new* |
| DM conversation context | `/peasants/{name}/whispers/{id}` | *new*, replaces `/conversations/{id}` |
| Quote permission (FEP-044f stamp) | `/peasants/{name}/parrot-licences/{id}` | chosen by the owner, 2026-10-01 |

Routes outside the actor namespace stay conventional, because other software looks for them by name:
- `/.well-known/*` and `/nodeinfo/*`;
- the Mastodon API (`/api/v1/*`, `/oauth/*`);
- the human pages `/@{name}` and `/@{name}/{id}`;
- `/media/*`.

Once a P1 deploy has federated a name, it is frozen.

## Target structure (moved incrementally; entity class names and URI shapes never change)

```
PrivaPub/
  Infrastructure/  Http/ (IpRangeGuard, SafeHttpHandlerFactory, FederationHttp, BoundedReader), Jobs/ (JobQueue, JobWorker,
                   Backoff, HostCircuitBreaker), Ids/ (PrivacyIdGenerator, PostIds), Data/ (Indexes, Migrations/_NNN_*),
                   RateLimiting, ErrorHandling
  Federation/      Signing/ (DraftCavage ← Services/Federation/HttpSignatures.cs, SignedFetchAuthorizer, Rfc9421Verifier)
                   Actors/ (LocalActorService, Keys, RemoteActorService, WebFingerClient, ActorParser)
                   Objects/ (Origin, ObjectFetcher, NoteParser, Addressing, ContentSanitizer)
                   Inbox/ (InboxReceiver, InboxProcessor, Handlers/{Follow,Accept,Reject,Undo,Create,Update,Delete,Like,Announce,Flag,Move}Handler)
                   Outbox/ (AudienceResolver, OutboxPublisher, GroupDistributor)  Rendering/ (ApContext, Actor/Note/Activity/CollectionRenderer)
                   Controllers/ (Peasants, WellKnown, NodeInfo, Users)
  Domain/          AvatarContext, Statuses/StatusService (← Services/PostsService.cs), Timelines/FanoutService, Notifications/,
                   Relationships/, Groups/GroupService (← GroupUsersService.cs), Content/ (MarkdownRenderer, PlainTextRenderer,
                   MentionTagParsers), Privacy/VisibilityPolicy (single CanSee), Media/ (P3), Geo/ (P4)
  Api/             ClientApi/Controllers (← Controllers/ClientToServer, routes unchanged), Mastodon/{Controllers,Entities,Mappers,Auth,Infrastructure}
  Web/Pages/       Profile, Status (P1); OAuth/Login, ChooseAvatar, Oob (P2)
```

The first refactor commit is a pure move with namespaces only. Logic changes follow in separate commits.

## Phases (each one ends in a tagged deploy plus verification)

### P0 Security baseline
- **S1, actor verification:** `RemoteActorService`.
  - `GetActor` accepts a document only if its `id` equals the final URL, on the same origin.
  - `GetActorByKeyId` resolves the owner, then the strict actor, and requires `publicKey.id == keyId` and
    `owner == id`.
  - Upserts become atomic on a unique `ActorURI`.
  - Key refetch is limited to once per 5 minutes.
- **S3 and S4, safe outbound HTTP:**
  - A `SocketsHttpHandler.ConnectCallback` resolves DNS and rejects loopback, private, link-local, CGNAT and ULA
    addresses, plus NAT64, 6to4, Teredo and IPv4-mapped forms. It then connects to the vetted IP.
  - Redirects are manual (at most 3, each re-checked).
  - Responses are capped at 1 MB with a content-type check and a 15 s timeout, with a negative cache for failing URLs.
  - Exceptions map to ProblemDetails: 400 or 401, never 500.
- **S2, origin checks:** `Origin.Same`. An activity's or object's `id` must share the actor's origin, and
  cross-origin embedded objects are refetched.
- **S5, sanitising:** `ContentSanitizer` with HtmlSanitizer. `Post.ContentHtml` and `ContentFormat` are added, and
  migration `_002` sanitises stored remote content.
- **S6, signature freshness:** `(request-target)` and `host` are required, plus `date` or `(created)` and `digest`.
  The window is −1 h to +15 min. `(expires)` is enforced, and the raw request target is used.
- **S8, DM injection:** a DM joins a conversation by `context` only if the author is already a member; otherwise it
  is matched on `DmGroup.ParticipantsKey`.
- **S9, groups:** a `Group.Kind` field is added. Existing groups become Circles (migration `_003`), and Circle posts
  stay local-only until P4.
- **S14:**
  - Delete the `"admin"` signup branch and add the CLI `admin promote`.
  - Serve Swagger only in Development.
  - Return generic error messages (no `ex.Message`).
- **Privacy:**
  - Drop the IP, User-Agent and root-id log lines in `RootUserController`.
  - Invitation signup takes its own `AvatarUserName`.
  - All fetches are signed by the instance actor.
- **Indexes and races:** `Infrastructure/Data/Indexes.cs`, with migration `_001` deduplicating first.
  - Unique: `ObjectURI`, `ForeignAvatar.ActorURI`, the Follower triple, RootToAvatar, and `ReservedName` (one
    username space across avatars and groups).
  - Plain: `PublicKeyId`, `(GroupUserId, _id)`, `(GroupId, _id)`.
- **Rate limiting** on login, signup, invitation and inbox (per key host).
- **Cleanup:** delete `Services/ActivityPubClient.cs` and `Models/Post/PostBoost.cs`.
- **New `PrivaPub.Tests`** (xUnit v3): IpRangeGuard table, signature fixtures (a real Mastodon request plus tampering
  cases), key-poisoning, origin rules, XSS corpus, DM injection.
- **CI:** `build.yml` runs the tests. `deploy.yml` runs the tests and a `mongodump` before swapping.

### P1 Interop foundations (two deploys)

**P1.1, infrastructure and content model:**
- **Mongo job queue** (`Job`, `RemoteInstance`):
  - Leases via `UpdateAndGet`, `Channel` wake-up, a reaper for expired leases.
  - Workers: delivery 8 (at most 2 per host), inbox 2.
  - Backoff `n⁴+15+jitter`, 16 tries. Non-transient errors go straight to Dead.
  - Per-host circuit breaker. Dedupe on `activity.id` and `activityId|inbox`.
  - TTL on finished jobs.
  - Migration `_004` moves pending `Delivery` rows into jobs.
- **Async inbox:** `InboxReceiver` verifies and enqueues, then answers 202 (a duplicate also gets 202).
  `InboxProcessor` dispatches to the handlers split out of `InboxService`.
- **Merge DmPost into Post** (migration `_005`; ids kept; `Visibility=Direct`, `ConversationId`). Post gains:
  - `Visibility {Public, Unlisted, FollowersOnly, Direct, Circle, LocalGeo}`
  - `AuthorAccountId`, `To`, `Cc`, `ActivityURI`, `Url`, `ContextURI`
  - `InReplyToURI`, `InReplyToAccountId`, `ReblogOfPostId`
  - `SpoilerText`, `Language`, `Mentions[]`, `Tags[]`, `Media[]`
  - counters, `Revisions[]`, `DeletedAt`

  `/clientapi/dm/*` moves onto Post, and every read goes through `VisibilityPolicy.CanSee`.
- **Parsing inbound content:**
  - `NoteParser` handles Note, Article, Page and Question. It reads `content`, then `contentMap`, then
    `_misskey_content`; also `summary`, `name`, `sensitive`, the `tag` array, `attachment` (as remote media),
    `inReplyTo`, `context`, `audience`, `updated`, and the `quote` fields (stored as a URI).
  - `Addressing` classifies Public, Unlisted, FollowersOnly or Direct. Followers-only is detected by the actor's
    stored `followers` URL exactly, not by a path suffix.
- **Rendering local text:**
  - `MarkdownRenderer` (Markdig) and `PlainTextRenderer` (for Mastodon API input).
  - Mentions resolved through WebFinger become an h-card plus a `Mention` tag; hashtags become a `Hashtag` tag.
- **Renderers:**
  - `ApContext`: Mastodon's `@context` (toot, schema, `indexable`, `blurhash`, `focalPoint`, `featured`,
    `alsoKnownAs`, `movedTo`, `webfinger`).
  - Actor and note `url` point at HTML pages; fields become `attachment` PropertyValue; the `webfinger` property is
    added; `published` is truncated to the day.
  - A title becomes `name` and is also prepended to the content as bold text.
  - A CW without a title gets a localised "Content warning" summary.
  - Routes are renamed to the themed set (`/groupies`, `/stalking`, `/scribbles/{id}`, `/grunts/{id}`,
    `/whispers/{id}`), activity URIs resolve, and the outbox gets a `first` page.
- **Web:**
  - Razor pages `/@{user}` and `/@{user}/{id}`, with content negotiation; `/peasants/{u}` with `text/html`
    redirects there.
  - WebFinger `profile-page` points at the HTML page.
  - NodeInfo 2.1 (avatars counted as users, `openRegistrations` from config) and `FEDERATION.md`.
- **Moderation:** `DomainBlock` (Silence, Suspend, RejectMedia), applied in the receiver, the fetcher and delivery.
- **IDs:** register `PrivacyIdGenerator` for Avatar and Group (day-truncated timestamp plus random bytes). Remote post
  ids are ObjectIds generated from their `published` time.

**P1.2, social graph and timelines:**
- **Outgoing Follow** (`Following` entity, plus Accept, Reject and outbound Undo). A local target is handled in-process.
- **`AudienceResolver` and `OutboxPublisher`:**

  | Visibility | to | cc |
  |---|---|---|
  | Public | Public | followers + mentions |
  | Unlisted | followers | Public + mentions |
  | Followers-only | followers | mentions |
  | Direct | mentions | (none) |

  Replies are also delivered to the parent's author, and group posts to the group. Mentions are delivered to personal
  inboxes.
- **Inbound Create:**
  - The interested set is followers of the author, plus addressed or mentioned local avatars, plus the local parent's
    author, plus group members. If nobody local is interested, the post is dropped.
  - Store the post (idempotent on ObjectURI), then fan out to the **`TimelineEntry`** collection
    (`{AvatarId, PostId, AuthorAccountId, ReblogOfPostId}`, unique `(AvatarId, PostId)`, paged by PostId).
  - Mastodon's reply, mute and reblog rules apply at write time.
  - Mention notifications are written.
- **`Notification`** entity: `{AvatarId, Type, FromAccountId, PostId, DedupeKey}`, unique on `DedupeKey`.
- **Edits:** outbound `Update{Note}` to the same To/Cc, with revisions kept. Profile edits send `Update{Person}`.
- **Deletes:** soft delete, a Delete sent to the stored audience, and the object then answers **410** with a
  Tombstone (a deleted actor answers 410 too).
- **Inbound Like and Announce:** a `Favourite` entity, reblog rows with the original refetched, Undo of both,
  counters and notifications.
- **Threads:** a `FetchAncestors` backfill job, depth at most 10.
- **Test endpoints:** `/clientapi/timeline/home`, `/clientapi/follow` and `/clientapi/notifications`, usable before P2.

### P2 Mastodon client API
- **OAuth:** OpenIddict 7.7 + OpenIddict.MongoDb.
  - Storage is `IMongoDatabase` registered as `DB.Default.Database()`.
  - Endpoints: `/oauth/authorize`, `/oauth/token`, `/oauth/revoke` and `/.well-known/oauth-authorization-server`.
  - Flows: authorization code (PKCE optional) and client_credentials.
  - Scopes: `read`, `write`, `follow`, `push` and the granular ones.
  - Tokens are non-expiring reference tokens. The signing and encryption keys are persistent, in `/etc/privapub/oidc-*.pem`.
  - The `oob` code page is supported, as is `force_login`.
  - **Authorize UX:** log in with the root password (a short-lived cookie limited to `/oauth`), choose an avatar, consent.
  - **The token's `sub` is the avatar id, and no root claim is ever in the principal.** A banned or deleted root
    invalidates its tokens.
  - `POST /api/v1/apps` creates OpenIddict applications dynamically and prunes unused ones.
- **Plumbing:**
  - `MastodonJson`: snake_case, explicit nulls and empty arrays (Tusky breaks on missing fields).
  - A binder that merges Rails-style query, form and JSON parameters.
  - `{"error"}` responses.
  - `Link` paging on `max_id`, `since_id` and `min_id`; CORS exposes `Link`.
  - `Idempotency-Key` honoured on posting statuses.
- **Endpoints:**
  - **Instance:** v1 and v2, advertising version `4.2.0 (compatible; PrivaPub)`.
  - **Accounts:** `verify_credentials`, `update_credentials`, `:id`, `:id/statuses`, `lookup`, `relationships`,
    `search`, followers and following, follow and unfollow, `follow_requests`.
  - **Statuses:** CRUD, edit, `context`, `history`, `source`, favourite and reblog plus their undos (outbound Like and
    Announce), `reblogged_by`, `favourited_by`.
  - **Timelines and the rest:** home, public, tag; notifications; markers; conversations; `/api/v2/search` with
    `resolve`.
  - **Stubs:** `custom_emojis`, filters, lists, announcements, trends, suggestions, `followed_tags`, preferences.
- **Mapping:**
  - Avatar, ForeignAvatar and Group all map to Account (`group: true` for groups). `acct` uses a WebFinger-verified
    handle; `created_at` is truncated to the day; avatar and header always have a placeholder URL.
  - Post maps to Status. Reblogs wrap the original. Per-viewer flags are loaded in batches.
  - LocalGeo never appears in the Mastodon API.
- **New Avatar settings:** `IsLocked`, `IsDiscoverable`, `IsIndexable`, `IsBot`, `HideCollections`,
  `DefaultVisibility`, `DefaultSensitive`, `DefaultLanguage`.

### P3 Social features
- **Media pipeline:** `MediaAttachment` entity plus `/api/v1/media` and `/api/v2/media`.
  - NetVips: autorotate, strip all metadata, 4096 px cap, 640 px thumbnail.
  - Blurhash.Core.
  - FFMpegCore: remux only, with `-map_metadata -1`.
  - Stored in `/var/lib/privapub/media`, outside the web root that deploys replace. nginx serves `/media/` with
    `nosniff` and a strict CSP, and allows larger uploads only on the media endpoints.
  - A job deletes unattached uploads.
- **Remote media proxy:** `/media/proxy/{hmac}/{url}`, going through the safe HTTP handler, with a ~5 GB disk LRU. The
  API and avatars use proxied URLs, so clients never contact remote hosts.
- **Outbound attachments:** `mediaType`, alt text, `blurhash`, `focalPoint` and dimensions. Profile avatar and header
  images are sent as `Update{Person}`.
- **Bookmarks and pins:** pins are served as the `featured` collection at `/peasants/{name}/trophies`, with featured
  tags at `/tattoos`.
- **Blocks and mutes:** per avatar. **Block is not federated**: it sends Reject or Undo Follow instead and drops the
  blocked actor's traffic. Domain blocks per account as well.
- **Reports:** inbound Flag becomes a `Report`. Outbound Flag is **sent by the instance actor**, so the reporting
  persona isn't revealed. Moderator endpoints on `/clientapi`.
- **Locked accounts:** the follow-request flow.

### P4 Groups and privacy features
- **Group model:** `Group.Kind`, `PostingPolicy`, `Rules`; `GroupMember.State`; new collections for
  moderators (`/wardens`), members (`/flock`) and featured (`/trophies`).
- **Communities (FEP-1b12):**
  - `GroupDistributor` announces the full activity (Create, Update, Delete, Like or Undo) with `audience` to the
    group's followers. New posts additionally get `Announce(object)`, so Mastodon shows them.
  - Top-level posts render as `Page` with a title; comments as `Note`.
  - Posting from non-followers is allowed according to `PostingPolicy`.
  - Inbound Announces from remote groups are refetched from their origin.
  - A Mastodon client posts into a group by mentioning `@group@host`.
- **Circles:**
  - The actor is undiscoverable, with manual approval and invitation-only joining.
  - Posts are addressed `to: [members, group]` and delivered to members' **personal** inboxes, never with Announce.
  - `SignedFetchAuthorizer` serves circle objects only to a signed request from a member, or from the instance actor
    of a member's server. Anything else gets 404.
  - `FEDERATION.md` documents the Mastodon limitation: Mastodon members' replies reach only whoever they mention.
- **Local-only location posts:**
  - `Post.Location` becomes a GeoJSON point rounded to 2 decimals (about 1 km), with `RangeKm` clamped to 1–50 and a
    `2dsphere` index. Visibility is LocalGeo.
  - `/clientapi/post/nearby` runs `$geoNear` filtered by each post's own radius. The viewer's position is never stored.
  - The audience is always empty; ActivityPub endpoints and the Mastodon API never expose these posts.
- **Persona hardening:** optionally re-key existing avatar ids; an optional `SecureMode` (signed GETs required); no
  suggestions or directories that could relate sibling avatars.

### P5 and beyond: rich content and the rest of the fediverse

Rewritten on 2026-10-01 from the research in `docs/INTEROP.md`, which holds the per-platform evidence and the
priorities. The goal is a future client that shows and links every kind of fediverse content in one place, with a
details view of where each object came from. So the server keeps everything it receives: typed where it understands
it, raw where it doesn't.

#### P5 Lose nothing (wire tolerance and full objects)
- **Cheap fixes that were wrong** (done in v1.7.0):
  - `summary` is a content warning only on a `Note`/`Question`, or when `sensitive` is set; elsewhere it is an excerpt
    (INTEROP W2). Remote titles now show in the Mastodon API for non-Note objects.
  - Persona usernames match Mastodon's and Misskey's pattern (groups already did).
  - Every term we emit is defined in our JSON-LD context.
  - `Vary: Accept`.
  - Not done on purpose: see "Deliberately not done" below.
- **Owner decisions that were small** (done in v1.7.0): blocks federate (`Block`, `Undo{Block}`); a persona's and a
  group's `published` is a random day up to two weeks before its creation (migration `_007`).
- **Parsing every shape** (done in v1.8.0, `Federation/Objects/ObjectShapes.cs`):
  - `url`, `icon`, `image`, `attachment` and `attributedTo` as a value, an object or an array (W1).
  - A Markdown `content` and `source` (W3).
  - Inferring a missing `mediaType` (W4).
  - Every thumbnail location (W5).
  - Language from `@context` (W11); alt text from `summary` (W12).
  - A null `id`, a 200 `Tombstone`, a `Delete` before its `Create` (W9, W13).
- **Typed payloads** (done in v1.8.0 for link, video, audio, event and the cover image; review and thread are still
  open): `Post.ObjectType` plus `Link`, `Video`, `Audio`, `Event`, `CoverURL` and `Source` (INTEROP §4.1). The Mastodon
  API view: the title above the body, an event's "when · where" line, a card made from the object without fetching,
  attachments with their thumbnail and duration. Video and audio playback through the proxy is P6.
- **Raw capture for the details view** (done in v1.8.0): the object as received, how it arrived, the signature scheme and
  key, received versus `published`, the extensions detected, and the origin's software (§4.3). Exposed at
  `/api/privapub/v1/statuses/:id/provenance` and `/api/privapub/v1/instances/:host`. Checked live against GoToSocial.
- **Routing by object type** (done in v1.9.0):
  - `Accept`/`Reject` act only on a Follow we sent and ignore any other object (W6); routing to RSVPs and interaction
    approvals comes with those features.
  - `ChatMessage` in as a direct message.
  - `Dislike` and its `Undo` in a downvote ledger, shown with favourites as `privapub.votes`.
  - A `Join` answered with `Ignore` until RSVP exists.
  - Friendica's thread-`Follow` is ignored without an error.
  - A deleted object's id is kept for 90 days (W9), and its `ObjectRecord` is removed with the post.
- **Typed details reach clients** (done in v1.9.0) as `Status.privapub`.
- **Delivery:** a 503 with `Retry-After` is waited out like a 429 and no longer counts against the host (done in v1.9.0).
  We answer 503 with `Retry-After` ourselves when a sender's key cannot be fetched for a temporary reason (v1.9.1),
  so Mastodon 4.7 retries instead of switching to RFC 9421.
- **Deliberately not done: dereferenceable `Follow`, `Like`, `Block`, `Accept`, `Reject` and `Undo` ids.** Serving them
  would publish who follows, likes and blocks whom, which our collections deliberately hide. Every one of them is sent
  with its object embedded, which is all Misskey needs. `Create` and `Announce` ids do dereference.

#### P6 What people see: emoji, polls, quotes, reactions, cards, players
- **Custom emoji** on posts, names, fields and poll options, proxied (done in v1.10.0), together with fuller remote
  profiles: header, fields, locked, published, moved-to, indexable, memorial, image descriptions.
- **Polls** in and out, with the Misskey, Pleroma and PieFed vote shapes. A count refresh is never an edit (done in
  v1.10.0; checked live both ways against GoToSocial).
- **Quotes (FEP-044f):**
  - read every key; verify `QuoteAuthorization` on all of its fields; handle revocation (done in v1.13.0);
  - personas quote others: `quoted_status_id`, `QuoteRequest`, `Accept{result}` verified, `Update` with
    `quoteAuthorization`; legacy quotes for posts that state no policy (done in v1.13.0); `api_versions.mastodon = 7`;
  - be quotable (done in v1.15.0): `canQuote` on our posts (owner's default: anyone, automatically), `QuoteRequest`
    answered with a parrot-licence or `Reject`, licences served and revoked; personas set `source[quote_policy]`, posts
    `quote_approval_policy`; `POST /statuses/:id/quotes/:quoting_id/revoke`, `PUT /statuses/:id/interaction_policy`.
- **Emoji reactions** in all three inbound forms, plus outbound `EmojiReact`, exposed as `emoji_reactions` (done in
  v1.11.0; Pleroma's `/api/v1/pleroma/statuses/:id/reactions` endpoints and `pleroma:emoji_reaction` notifications).
- **Link cards** (done in v1.12.0):
  - from the object, or from FEP-8967 `preview`, without fetching;
  - otherwise the server reads the page (owner decision 1): public posts only, 0–60 s after arrival, cached per address
    for 7 days across the whole server, `Federation:FetchLinkPreviews` to switch it off. Checked live against a
    GoToSocial profile page.
- **Media** (done in v1.14.0 except where noted):
  - video playback through the proxy: range requests streamed from the origin, the poster, a playable attachment picked
    from MP4 or fragmented MP4 files (HLS playlists are not rewritten; not needed while PeerTube publishes the files);
  - audio attachments;
  - JPEG/PNG for Pixelfed: uploads were already re-encoded to JPEG, PNG or GIF;
  - an article reader view: the data is in `Status.privapub` (title, excerpt, cover, full HTML in `content`); the view
    itself belongs to the client.

#### P7 Threads, communities and the social graph
- **Thread backfill:**
  - read in order: `contextHistory`, then `context` (paged, with ETag), then `replies`;
  - group by the root post;
  - publish our own `context` and a paged `replies`.
- **Lemmy, PieFed and Mbin:**
  - the moderation set: removals, locks, bans, featured, moderators, `Warn`, `Resolve`;
  - votes in and out; link posts; flairs; `Feed` actors; community polls; post `Move`;
  - the outbound shape Lemmy requires;
  - communities we host announce to the author's own instance too;
  - Flags from a `Service`-typed reporter actor.
- **GoToSocial interaction policies:** stored and shown; send `ReplyRequest`/`LikeRequest` where approval is
  needed; handle `Accept{result}`.
- **Accounts and follows:**
  - inbound `Move` with Mastodon's checks;
  - re-run WebFinger on a rename;
  - inbound `Block`, plus `Add`/`Remove` of pins;
  - FEP-8fcf followers sync;
  - `indexable`/`discoverable`/`searchableBy`;
  - edit history from `formerRepresentations`;
  - PeerTube reply rules and `ApproveReply`.
- **Events:** structured RSVP (`Join`/`Leave` with stable ids).

#### P8 Signatures, discovery and the long tail
- **Signatures:**
  - RFC 9421 inbound (RSA and Ed25519, Content-Digest);
  - outbound double-knock, remembered per host;
  - `publicKey` arrays and FEP-521a Multikey;
  - FEP-8b32 proof verification;
  - `hs2019` with SHA-512.
- **Discovery:**
  - a relay client for both relay styles;
  - instance actor discovery (FEP-d556, FEP-2677);
  - `implements` (FEP-844e).
- **Mastodon API:** streaming WebSocket, Web Push, grouped notifications. Then advertise an honest version.
- **Backfill:** an author's outbox after following them.
- **Long tail:**
  - MFM rendering data;
  - Misskey actor extras;
  - book reviews (`relatedWith`, `rating`);
  - Funkwhale and Castopod metadata;
  - Pixelfed `place` (display only, never re-federated);
  - Bluesky bridging per persona.

### Sweep and statistics (T and M, planned 2026-10-03, between P6 and P7)

Two tracks, interleaved so that the test host exists before the ledger, and the ledger starts collecting early, because
statistics gain value with every day recorded.

| Step | Content | Tag |
|---|---|---|
| T1–T4 | Tests stop sharing state they don't own. CI runs every test against a throwaway mongod. The whole server runs under test (`WebApplicationFactory`). Nothing answers 500. | v1.15.1 |
| M1–M6 | The interaction ledger: inbox answers, handler verdicts, delivery attempts, outbound requests, served and client traffic. Provenance fixes (fetched records, stored extensions, group-wrapped Update/Delete). | v1.16.0 |
| T5–T8 | Coverage sweep over HTTP: OAuth, `/clientapi`, the Mastodon API, federation GETs, inbox gaps, jobs, migrations, pages. | v1.17.0 |
| M7–M10 | Daily rollups (`InstanceDay`, `ServerDay`). Every touched server described weekly (NodeInfo usage, instance API, snapshots). Geolocation (DB-IP Lite city and ASN). Admin statistics API under `/clientapi/admin/statistics`. | v1.17.0 (planned v1.18.0) |
| T9–T14 | The pasture as plugins. GoToSocial gaps, then Mastodon, Misskey/Sharkey, Akkoma and Lemmy 1.0, each run also checking that peer's statistics. | v1.17.0, v1.17.1 |
| M11 | The opt-in crawler (`PrivaPub-Stargazer`) and the `/stargazing` explainer. | v1.17.0 (planned v1.19.0) |

Later, and not part of these steps: the public `/stargazing` statistics. It is anonymous, cached and rate-limited, and
covers per-server software, self-published counts, location as projected by the rule above, availability buckets and
the public activity mix. Per-server activity is shown only as order-of-magnitude buckets, and only for servers that
report at least 10 users; smaller ones fold into one "small servers" aggregate.

### Cut or deferred (deliberately)
- **Cut:**
  - Link-preview cards *fetched from the linked page* by default. Cards built from the object, or from FEP-8967
    `preview` data, are in P6; fetching the page is an owner decision (INTEROP §6.1).
  - Translation, trends, directory and lists (filters stay as stubs).
  - Scheduled posts.
  - The Mastodon admin API: moderation stays on `/clientapi`.
  - `POST /api/v1/accounts` registration: avatars are created through `/clientapi`.
  - Local custom emoji.
  - S3 storage.
  - JSON-LD and LD-signature processing. FEP-8b32 proofs need neither (JCS), so they are in P8.
- **Deferred:** video transcoding (remux only for now).
- **Out of scope:** moving decePubClient onto the Mastodon API.

## Verification (per phase)

- **Unit tests** (`PrivaPub.Tests`) on every phase. Fixture JSON is captured per peer into
  `Fixtures/{mastodon,gotosocial,misskey,akkoma,lemmy}/`. Renderers are checked against golden JSON. Inbound
  scenarios run as integration tests on a throwaway mongod.
- **Interop:** **Fediverse Pasture** runs on the workstation with podman: Mastodon, GoToSocial, Misskey/Sharkey,
  Akkoma, plus Lemmy for P4. As built, `tools/pasture/run.sh` runs PrivaPub and GoToSocial behind one Caddy on a
  podman network, with `AllowPrivateNetworks`, `AllowPlainHttp` and `AcceptAnyCertificate`; startup refuses all three in
  Production. `tools/pasture/interop.sh` drives the matrix for GoToSocial; the other peers are still to be added. The
  matrix, per peer:
  - follow both ways;
  - public, unlisted, followers-only and direct posts both ways;
  - a reply landing in the remote thread;
  - a mention producing a notification;
  - edit, and delete answering 410;
  - like and boost counters;
  - CW and title rendering on Mastodon;
  - Pasture's odd-payload inputs always answered 202 or 4xx, never 500.

  Also run verify.funfedi.dev against `/peasants/privapub` and one avatar.
- **P0:** `curl` the inbox with junk and get 400/401. Resolving `127.0.0.1.nip.io` or a host with an `::1` record is
  refused. Swagger answers 404 in production.
- **P1:** 500 deliveries queued for a dead host don't delay deliveries to live hosts.
- **P2:** Tusky, Elk, Phanpy and Ivory each log in with two avatars of one root as separate accounts. Home paginates
  both ways; post, reply, DM, follow-by-search, notifications, CW, and edit or delete-and-redraft all work.
  `tools/smoke/mastodon-api.sh` runs after every deploy. A test asserts that no response for avatar A contains avatar
  B's id.
- **P3:**
  - `exiftool` shows no GPS data on an uploaded original.
  - Elk renders the blurhash.
  - Phanpy's network tab shows no requests to remote hosts.
  - Like and boost round-trip with Mastodon and GtS, and a block cuts the follow both ways.
  - A Flag reaches the Mastodon admin UI.
- **P4:**
  - Lemmy follows a community; posts and comments flow both ways; votes show; moderator deletes propagate.
  - A Mastodon member of a circle gets a limited post. An unsigned or non-member GET answers 404.
  - Two local avatars, 3 km and 8 km away, see a 5 km post correctly, and the post creates no delivery jobs.
- **Every deploy:** `/build.json` reports the commit. `privapub.thepra.dev/peasants/privapub` and NodeInfo answer.

## Risks and sequencing

- Unique indexes fail on existing duplicates, so the dedupe migration runs first. Run a `mongodump` before every
  tagged deploy from P0 on.
- Merge DmPost into Post and register `PrivacyIdGenerator` **before P2** exposes ids to clients.
- The publish grows by about 20 MB (OpenIddict, NetVips.Native, FFMpegCore). Keep `deploy.yml`'s asserted file list
  current, and add ffmpeg in `deploy/max/setup.sh`.
- Pasture's private-network and plain-HTTP allowances stay out of `appsettings.Production.json`, and startup asserts
  they're off.
- Scale: this is several weeks of work. Each phase is a separate tagged release, and I report after each one.