thepra/SocialPub
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
830385ea2b96dd14a5af200ad559019042190e7a
SocialPub/CLAUDE.md
T
thepraandClaude Opus 5.5 dc63fa57b8 A login's storage is counted, and can have a quota 
MediaAttachment.Size was stored and never summed: nobody, the administrator included, could tell what media took,
and nothing bounded it. Counted.Media and MediaOfRoot sum what is kept (not trashed). A login sees what its personas'
uploads and pictures take at /clientapi/user/storage (ViewStorage), with its quota when the server sets one;
Media:QuotaBytesPerRoot (0, no quota, by default) refuses an upload or a picture over it with 422, whichever persona
sends it; the statistics overview gains the media totals, the proxy cache and the trash (ViewMediaTotals). Tests: a
login's storage counts what its personas keep and forgets what is trashed; two uploads from two personas of one login
are refused together past the quota.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
2026-10-07 11:03:25 +02:00

858 lines
76 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# CLAUDE.md
Guidance for working in this repository. `docs/ROADMAP.md` holds the owner's decisions and the phased plan to full
ActivityPub interop; read it before changing anything federation-, privacy- or API-shaped. `docs/INTEROP.md` is the
per-platform evidence behind phases P5 to P8: what each peer sends, what it expects, and what PrivaPub still drops.
Check the platform's section there before writing a parser or a renderer, and add to it whatever you learn.
## What this is
**PrivaPub** (repo name SocialPub) is a self-hosted ActivityPub server in C#, live at https://privapub.thepra.dev.
It is meant as a Pleroma-like microblogging server for privacy-minded people and small communities, federating with
Mastodon, GoToSocial, Pleroma/Akkoma, Misskey and Lemmy.
Its defining idea: **one private login owns several public personas.**
- **`RootUser` is the private login.** Username, password, optional email (used only for recovery), policies.
- **`Avatar` is a public persona.** Each one is an ActivityPub actor with its own RSA keys at `/peasants/{username}`.
Root and avatar are linked only through `RootToAvatar`.
- **Personas must stay unlinkable** to other users and servers; only the instance admin can see the link.
- **`Group` is an ActivityPub Group actor.** It has members, an invitation code and an optional password. Groups are
becoming two kinds: a public **community** (FEP-1b12, Lemmy-compatible) and a private, invitation-only **circle**.
- **`DmGroup` is a direct-message conversation.**
- **`privapub` is the instance actor** (type Application). It signs fetches no persona should be tied to.
- **`privapub_reports` is the reporter** (type Service, `LocalActorKind.Reporter`). It carries reports to Lemmy, which
takes none from an Application, and to PieFed and Mbin, and names nobody. Both are server actors
(`LocalActor.IsServerActor`): never followed, mentioned or shown as accounts.
Privacy features in the model:
- location-ranged posts (`Post.Location`, `RangeKm`), to become local-only and never federated;
- contacts a persona can share (`Avatar.SharedPersonalContacts`);
- private notes (`RootUserNote`, `PersonalNote`).
The intended client is the Mastodon client API (Tusky, Elk, Phanpy, Ivory), where each avatar logs in as its own
account. The private `/clientapi` covers what Mastodon can't express. `thepra/decePubClient`
(https://decepub.thepra.dev) is the owner's own Pleroma-FE-like PWA. It signs in once on `/clientapi` and exchanges that
JWT for each persona's Mastodon token (`PersonaExchange`), then uses both APIs.
## Route names are deliberate
The odd names are the owner's and part of the project's character. **Never "fix" them to conventional ones.** A new
actor-scoped route gets a name in the same spirit, agreed with the owner, and a name is frozen once it has federated.
| Thing | Route |
|---|---|
| Actor | `/peasants/{name}` (`/users/{name}` 301-redirects here; browsers are sent to `/@{name}`) |
| Inbox | `/peasants/{name}/mouth` |
| Outbox | `/peasants/{name}/anus` (`?page=true[&max_id=]` for pages) |
| Shared inbox | `/human-centipede` (also `/peasants/{name}/human-centipede`) |
| Followers | `/peasants/{name}/groupies` |
| Following | `/peasants/{name}/stalking` |
| Notes | `/peasants/{name}/scribbles/{id}` |
| Activities | `/peasants/{name}/grunts/{id}` (`create-{postId}` resolves) |
| DM context | `/peasants/{name}/whispers/{id}` |
| Quote permission (FEP-044f `QuoteAuthorization`) | `/peasants/{name}/parrot-licences/{id}` |
| Token refresh | `/clientapi/user/sniff/again` |
Agreed for later phases: `/gossip`, `/drool`, `/echoes` (replies, likes, shares), `/trophies` (featured), `/tattoos`
(featured tags), `/flock` and `/wardens` (group members and moderators).
Routes other software looks up by name stay conventional:
- `/.well-known/*` and `/nodeinfo/*`;
- `/api/v1/*` and `/oauth/*`;
- `/@{name}` pages;
- `/media/*`.
## Repo map
```
PrivaPub.sln
PrivaPub/ ASP.NET Core Web API, net10.0
Program.cs host, Mongo init (GUIDs Standard), pipeline, /build.json
Middleware/SocialPubConfigurations.cs every DI registration (auth, federation, services, swagger, CORS)
Controllers/ClientToServer/ /clientapi/*: RootUser (signup/login/invitations/recovery), PrivateAvatar,
Group, Post (posts + DMs), Admin, Data
Infrastructure/
Http/ FederationHttp + SafeHttpHandlerFactory + IpRangeGuard: the only way out
Jobs/ JobQueue (leases), JobWorker, Backoff, HostCircuitBreaker
Ids/ PrivacyIds (day-only ids for personas and groups, arrival-ordered ids for remote posts)
Data/ Indexes (created at start), EntityMaps.Warm, Migrations/_NNN_*.cs
Cli/ AdminCommands (`PrivaPub admin promote|demote <root>`)
RateLimiting.cs accounts (per client address) and inbox (per sending origin) policies
Federation/
Controllers/ PeasantsController (actor, outbox, followers, following, posts, inboxes),
WellKnownController (webfinger, nodeinfo), UsersController (redirect)
Actors/ LocalActorService (LocalActor, Keys, ReservedName), RemoteActorService
(authoritative fetch, key verification, WebFinger), ActorDocument (parser)
Objects/ Origin, ActivityJson, NoteParser, Addressing, ContentSanitizer
Moderation/ DomainBlocks (suspend / silence / reject media)
Signing/ HttpSignatures (draft-cavage sign/verify), MessageSignatures (RFC 9421 verify),
RequestSignature (whichever a request carries), IntegrityProofs (FEP-8b32
eddsa-jcs-2022 by a persona's Ed25519 key, FEP-521a Multikeys), Jcs (RFC 8785)
Inbox/ InboxReceiver (verify, queue, 202) → InboxProcessor (job) → Handlers/{Follow,Accept,Reject,
Undo,Create,Update,Delete,Like,Announce}; RemotePosts (build, fetch parents, FetchAncestors);
RemoteReplies (FetchReplies: a thread's `context`, else `replies` two levels down);
Forwarded (what a thread's server passes on, believed as far as the origin vouches);
ReplyRelay (third-party replies to a persona's posts passed on to its followers)
Outbox/ OutboxPublisher (who a post goes to), DeliveryService (queues jobs) + DeliveryJobHandler
Rendering/ ActivityPubRenderer (Mastodon @context, actors, notes, collections)
Domain/
Content/ ContentRenderer (Markdown or plain text → HTML with h-card mentions and hashtags)
Social/ FollowService (local in-process, remote Follow/Accept), Notifications
Timelines/ Fanout (TimelineEntry rows, Mastodon's home rules), TimelineService
Privacy/ VisibilityPolicy (IsPublic expression, CanSee)
Relationships/ RelationshipService (blocks, mutes, account domain blocks; Hidden), ReportService
Media/ MediaService (libvips, ffmpeg remux, blurhash), MediaProxy, MediaJanitor
Domain/Statuses/ StatusService: publish, edit, remove, favourite, reblog, for a persona (both client APIs use it);
InteractionApprovals: GoToSocial's canReply/canLike/canAnnounce, judged, asked and answered;
Participations: a persona's Join and Leave of a remote event, and the organiser's answer
Api/Mastodon/
Auth/ OpenIddict setup (keys in Mongo), MastodonScopes, TokenController, OAuthPruner
Infrastructure/ MastodonController (avatar context, scopes, errors, Link), MastodonParams, MastodonJson, Page
Entities/ Mappers/ Mastodon entities; MastodonMapper (Account, Status), AccountSearch
Controllers/ apps, instance, accounts, statuses, timelines, notifications, search, stubs
Web/Pages/ Razor: /@{user}, /@{user}/{id} (public posts only, strict CSP, noindex);
OAuth/: /oauth/login (root password), /oauth/authorize (choose persona, consent)
Services/ RootUsersService, GroupUsersService, PostsService, AppConfigurationService, …
Models/ Mongo entities: User/, Group/, Post/, Federation/, Jobs/, AppConfiguration
StaticServices/ DbEntities (Find<T> accessors), AuthTokenManager (JWT), PasswordHasher
Data/InitDb.cs first-run seeding (languages)
PrivaPub.ClientModels/ DTOs + validation resources shared with clients
PrivaPub.Tests/ xUnit v3; Support/ has a fake two-origin peer and a throwaway-database fixture
deploy/ nginx vhost, systemd units (privapub, privapub-mongod), max/setup.sh
.gitea/workflows/ build.yml (push) · deploy.yml (tag v*)
docs/ROADMAP.md decisions + phased plan
```
The roadmap moves code towards `Infrastructure/`, `Federation/`, `Domain/`, `Api/{ClientApi,Mastodon}` and `Web/`,
one pure-move commit at a time.
## Commands
```bash
dotnet build PrivaPub.sln -c Release
dotnet test PrivaPub.sln # unit tests; integration tests skip
PRIVAPUB_TEST_MONGOD=1 dotnet test PrivaPub.sln # all of them, against mongod on 127.0.0.1:27017
# (PRIVAPUB_TEST_MONGO overrides; a fresh database per run, dropped after)
tools/ci/with-test-mongod.sh dotnet test PrivaPub.sln # all of them, on a throwaway mongod, as CI runs them
cd PrivaPub && ASPNETCORE_ENVIRONMENT=Development \
Kestrel__Endpoints__Http__Url=http://127.0.0.1:6970 Kestrel__Endpoints__Http__Protocols=Http1AndHttp2 \
AppConfiguration__BackendBaseAddress=http://127.0.0.1:6970 MongoSettings__Database=PrivaPubTest \
dotnet run # needs a local mongod on 27017; swagger at /swagger
```
Development config (`appsettings.Development.json`) binds HTTPS 7195 with HTTP/2 only; the overrides above make it
curl-able. Outbound fetches only go to https DNS names resolving to public addresses; a test network (Pasture) sets
`Federation__AllowPrivateNetworks=true`, `Federation__AllowPlainHttp=true` and `Federation__AcceptAnyCertificate=true`,
which startup refuses in Production.
The admin CLI runs with every service built but nothing started (no Kestrel, no hosted services), so it can run next to
the live service. Sign-up is closed in production (invitations only), so the first login is made here; promoting is how
an admin is made (signing up as "admin" grants nothing):
```bash
cd /var/www/privapub.thepra.dev
echo '<password>' | ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin create-root <login> --admin # password on stdin
ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin promote|demote <root>
ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin smoke <persona> # the deploy's: prints "deploy-smoke <new password>"
```
The deploy runs them as `build-runner`, which owns the published files, reads `appsettings.Production.json` through
group www-data and reaches the private mongod; `sudo -u www-data` works too.
## Federation invariants
1. **Every outbound request goes through `IFederationHttp`.** Its handler resolves the name itself and connects only
to public addresses; redirects are followed by hand (three at most, each re-checked); bodies are capped at 1 MB;
only JSON media types are read; a refused URL is not asked again for five minutes. Never create another
`HttpClient` for federation. The only other outbound traffic is SMTP and `GeoUpdater`'s monthly DB-IP Lite download
(its own `geo` client, a fixed HTTPS host, size-capped, the file checked before it is swapped in).
2. **Every fetch is signed by the instance actor** (`privapub`), never by a persona; deliveries are signed by the acting
avatar or group. Both are draft-cavage rsa-sha256 over `(request-target) host date` (+ `digest` on bodies). Inbound,
an RFC 9421 signature (`Signature-Input`) is verified too, its target against our public address. A persona's own
activity going to a relay also carries a FEP-8b32 proof by its Ed25519 key (`Avatar.SigningKey`, one per persona,
never shared), and nothing else does: Mitra refuses a proof by a key it has not read. A forwarded activity is taken
as it came only when its proof verifies with a Multikey of its actor's.
3. **A remote document is believed only from its own address.** `RemoteActorService.FetchObject` requires the
document's `id` to be the URL it was served from (a same-origin alias is followed once). A key is accepted only if
the actor lists it, its `owner` is the actor and it shares the actor's origin.
4. **Inbound inboxes verify everything before acting:**
- the signature covers `(request-target)`, `host`, `digest` and `date` or `(created)`;
- the Digest matches the body and the date is at most an hour old and fifteen minutes ahead;
- the signature verifies against the key owner's key, and the activity's `actor` is the key owner, or else it was
forwarded (`Forwarded`: a thread's server passing on what happens in it, as Mastodon and Friendica do). A forwarded
activity proves nothing about its author: answered 202, a Create or Update is taken as its object reads at the
actor's origin now, a Delete of a public or unlisted copy once that origin answers 404 or 410
(`RemoteActorService.IsGone`), anything else let go. Forwarded copies have their own dedupe key, so a copy that
failed never hides the author's own delivery;
- an activity is queued once per id, but an id that comes back carrying another type, actor or object is a second
activity, not a copy (Friendica's ids are `uniqid()`, which two of its processes can share), and is queued apart;
- the activity's `id`, and any object it creates, updates or deletes, is on the actor's origin; a cross-origin
object is refetched from its own origin. One attributed to another account of the actor's server (Mobilizon's
organiser and the group's event) is taken as that server has it, and deleted once it answers 404 or 410.
5. **Status codes:**
- bad or missing signature: **401**;
- malformed or forged body: **400**;
- accepted: **202**;
- over the rate limit: **429**;
- **never 500** from `/peasants` or an inbox. Peers retry or "double-knock" based on these codes.
6. **Actor documents:**
- served as `application/activity+json`;
- `publicKey` is an SPKI PEM at `{actor}#main-key`, with `owner` equal to the actor's id;
- the shared inbox is advertised in `endpoints.sharedInbox`.
7. **Remote HTML is sanitized before it is stored** (`ContentSanitizer`); `Post.ContentHtml` is what is shown,
`ContentFormat` says what `Text` holds. Remote names are plain text.
8. **Circles federate to members only.** A circle is an undiscoverable Group actor that takes follow requests (the owner
approves); its posts are addressed to the circle and its `/flock`, delivered to members' personal inboxes, never
announced. Each member's copy also names that member in `cc` (`OutboxPublisher.Naming`, owner decision
2026-10-04), because Mastodon and GoToSocial keep a post only when it names one of their accounts; Create, every
Update and the Delete all go through `OutboxPublisher.Publish` for that. A reply to a circle post stays in the circle,
and a circle post never asks a non-member for a quote. Circles never appear in search, lookups, mentions or profile
pages.
**A post that is not public is served only to a signed request from someone it was for** (`SignedFetchAuthorizer.MayRead`):
a follower or an addressed account for followers-only, an addressed account for a DM, a member for a circle, or the
instance actor of a server where one of them lives; 404 to anyone else, 410 to them once it is deleted. A circle
refetch names the requesting member, or the members on the requesting server. A DM's `context`
(`/peasants/{name}/whispers/{id}`) lists the conversation's posts for its participants. Mastodon deletes its copy
when a refetch answers 404, which is why this matters.
**Communities** are FEP-1b12 groups: `GroupDistributor` announces the whole activity (plus the object for new posts,
for Mastodon), top-level posts are `Page`s with a `name`, posting follows `Group.PostingPolicy`.
Located posts (`LocalGeo`) are the only local-only posts. The Mastodon API looks past them everywhere, homes
included (the fan-out gives them no `TimelineEntry`, not even their author's): they are read through
`/clientapi/post/nearby` and deleted through `/clientapi/post/delete`.
9. **A DM joins a conversation only by `DmGroup.ParticipantsKey`**, the exact set of its participants; a remote
`context` decides nothing. DMs are `Post`s with `Visibility = Direct` and a `ConversationId` (`DmPost` is legacy).
`DmGroup.LastPostId` pages `/api/v1/conversations`, and `ConversationState` keeps each persona's read and removed
marks (`Domain/Statuses/ConversationStates.cs`; writing in a conversation reads it).
10. **Nothing slow happens inside a request.** Deliveries and inbox processing are `Job`s (`Infrastructure/Jobs`):
leased, retried on Mastodon's curve, at most two per host, paused per host by `RemoteInstance`. The inbox answers
202 once it has verified and queued; a handler must be idempotent (unique `ObjectURI`, job `DedupeKey`). A lease
(2 minutes, stamped with its own owner) is renewed every third of its length while the handler runs, so a long job
runs once; a lease found taken cancels the handler, and `Finish` only counts for the lease it was given.
11. **Every number about posts and users goes through `Domain/Privacy/Counted`** (owner decision 2026-10-04: one
answer everywhere): a persona's `statuses_count`, its outbox `totalItems`, NodeInfo `localPosts` and the instance
`status_count` all count Mastodon's way (not deleted, not a DM, boosts included, circle and located posts too);
users exclude personas of banned or deleted roots; `replies_count` counts only public and unlisted replies; a
remote account that deletes itself takes its likes, votes, reactions, boosts and replies out of every count
(`GoneActors`). Follower and following counts are public, their members never are (`hide_collections` is always
true).
12. **Every "may anyone see this" goes through `VisibilityPolicy.IsPublic`;** a persona-specific read uses `CanSee`, and
any other read of stored posts filters by `IsShown`. All three hide deleted posts and the posts of a remote account
that deleted itself (`Post.AuthorGone`: kept, hidden everywhere, owner decision).
13. **Remote content is stored only when someone here asked for it:** a persona follows the author, is addressed or
mentioned, it replies to a local post, or it is addressed to a community the author follows; a public parent is
fetched as context. Followers-only is detected by the author's stored `followers` URL.
14. **Home timelines are written, not computed:** every stored or created post goes through `Fanout.Distribute`, and
every delete removes its `TimelineEntry` rows. Local deletes are soft (content cleared, 410 Tombstone).
15. **What PrivaPub passes on is exactly what it received** (owner decision 2026-10-05). A public or unlisted reply from
another server to a persona's public, unlisted or followers-only post, and its Update and Delete, go to the
persona's followers as the activity arrived (`Arrival.Raw`, `Federation/Inbox/ReplyRelay.cs`), signed by the
persona for the transport only: never to the replier's own server, never for a local-only or group post. Only an
activity its own actor delivered carries `Raw`, so a forwarded one is never passed on again.
16. **An `Accept` or `Reject` is routed by what it answers:** our Follow (`FollowService`, any `-again-N` resend), an
interaction request (`InteractionApprovals`, with the author's authorization read back from its origin) or a
persona's `Join` (`Participations`, by its `/grunts/join-<id>` id), each only from the origin of what it answers.
17. **A server's software is for display, with two exceptions** (owner decisions 2026-10-05 and 2026-10-06): a direct
message to one account on a server whose NodeInfo names Lemmy before 1.0 or Mbin goes as a `ChatMessage`
(`StatusService.TakesOnlyChatMessages`), as one to an account that writes to us that way does; and a report of a
post in a community on a server whose NodeInfo is in `ReportService.ServiceReportTakers` leaves in Lemmy's shape,
from the reporter, one `Flag` per post. A server joins that set only once the pasture shows it keeps such a report
with its reason. Nothing else may branch on `RemoteInstance.Software`.
## Mastodon client API invariants
1. **A token is one persona.** Its subject is the avatar id; the root id lives only in the fifteen-minute `/oauth`
cookie used while choosing the persona, and never in a token, an authorization or a response.
- **The first-party client exchanges the root JWT for one persona's token** (RFC 8693 on `/oauth/token`,
`subject_token_type` `urn:ietf:params:oauth:token-type:jwt`, `avatar_id`; `Api/Mastodon/Auth/PersonaExchange.cs`).
Only the seeded public application (`ClientApi:FirstPartyClientId`, default `decepub`) holds the grant.
`RootJwtSubjectToken` validates the JWT through `RootJwt`, the same rules as JwtBearer (signature, lifetime, ban,
deletion, session stamp). The issued token is the same kind as the authorization code flow's, so revocation and
`RootSessions` cover it.
2. `/api/*` authenticates with OpenIddict validation, everything else with the old JWT (`PrivaPub` policy scheme).
Every `/api` request re-checks that the persona's root is neither banned nor deleted (`MastodonController`).
3. Read parameters through `Params` (query, form and JSON merged Rails-style), never MVC binding. A value type read
from a conditional must say `(int?)null`, not `default`: that bug once made every list one item long.
4. Answer with `Json(...)` (snake_case, explicit nulls) or `Error(status, message)`; page lists with `Page` and `Link`.
5. Unsupported features answer empty lists or 422 with a message, never 404 or 500, so clients degrade. Any `/api`
error that leaves without a body (a challenge, a 404 from routing, a 429) gets Mastodon's `{"error": ...}` from
`UseMastodonErrorBodies`. `NeverFiveHundredTests` walks every route with junk ids, anonymously, as a persona and
with a junk token.
6. Advertise `4.2.0 (compatible; PrivaPub)` until streaming and Web Push exist too; `api_versions.mastodon = 7` stays,
because clients show quotes from it, so everything a 4.3+ client calls because of it must answer: grouped
notifications (`/api/v2/notifications`, its unread count, groups, accounts and dismiss) and the notification policy.
The policy is real (`Domain/Social/NotificationPolicies.cs`, applied in `Notifications.Add`): accepted by default,
it filters (into a `NotificationRequest` per account, out of every list and count unless `include_filtered`) or drops
what strangers, accounts that do not follow the persona (or only for three days), accounts newer than 30 days,
unasked private mentions and silenced accounts send; accepting a request lets that account in for good
(`NotificationPermission`).
7. **What a Mastodon `Status` cannot say goes in `Status.privapub`** (`PrivaPubStatus`): object type, title, excerpt,
cover, the author's source, link, video, audio and event details, a remote post's own place (Pixelfed's `location`,
never federated again), and up/down votes. Every media URL in it goes
through the proxy; only page links (`link.url`, an event's online link) point at the remote site, because following
one is the reader's choice.
## Media invariants
1. **No upload keeps its metadata.** Images are re-encoded by libvips with `keep=none`; audio and video are remuxed with
`-map_metadata -1`. `MediaProcessingTests` checks EXIF and XMP are gone.
2. **An image is checked before it is decoded.**
- **Loaders:** only libvips' JPEG, PNG, GIF, WebP and HEIF loaders ever run on an upload (`MediaService`'s static
constructor blocks every other loader), so an SVG, PDF or TIFF claiming another type never loads.
- **Size:** the header alone tells width × height (× frames for a GIF), refused above `Media:MaxPixels` (40 MP) or
`Media:MaxFrames`.
- **Then:** the image is shrunk on load to `MaxImageSide`, turned by its orientation, and its colours brought into
sRGB (`thumbnail`).
- **GIFs:** an animated GIF becomes a looping silent H.264 mp4 typed `gifv`, as on Mastodon (ffmpeg with libx264;
`PostMedia.Kind` says so), and a still GIF is an image.
- **Types:** HEIC and HEIF are not accepted, since the bundled libvips has no HEVC decoder.
- **Load:** processing runs `Media:Concurrency` at a time, and uploads have their own rate limit (`uploads`, per
credential).
3. **Audio and video are processed off the request when the client allows it.**
- **v2:** `POST /api/v2/media` stores them as sent in `media-incoming` (beside the root, never served) and answers
202 with no url, while a `ProcessMedia` job (one at a time, its lease renewed) does the work. Until then
`GET /api/v1/media/:id` answers 206, and 422 with the reason if it failed. Media still processing can't be posted.
- **v1:** processes before answering.
- **ffmpeg** reads only the stored file: `-protocol_whitelist file`, and the input format is forced from the probe.
Data and subtitle tracks are dropped (`-dn -sn`, which iPhone MOVs need).
- **Remux or transcode:** a video browsers play as it is (H.264, VP8, VP9 or AV1, within `Media:MaxVideoPixels` and
`MaxFrameRate`) is remuxed; anything else is transcoded to H.264 that fits, as Mastodon does.
- **Limits:** longer than `Media:MaxSeconds` is refused.
- **Saving:** nothing is saved until everything succeeded; outputs move into place rather than being read into
memory, and every temporary file goes. FLAC is served as `audio/flac`, and the unit runs with `PrivateTmp`.
4. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
sibling `media-proxy` and the trash the sibling `media-trash`, neither of which `/media/files` serves.
5. **A file lives exactly as long as something holds it.** Every upload is a `MediaAttachment` row, profile pictures
too (`Kind` avatar or header, `ProfileOfAvatarId`). Deleting a post, an edit leaving media out, a replaced picture,
a dropped scheduled post, a removed root, and an upload never posted for a day each trash theirs
(`IMediaService.Trash`):
- the row gets `TrashedAt` in one conditional update, so a row attached meanwhile is left alone;
- its files move into `media-trash` at once, so `/media/files` stops serving them;
- `MediaJanitor` deletes them a day later (`TrashGrace`).
What a login's media take (every persona's, trashed ones no longer) is `Counted.MediaOfRoot`, shown to the login at
`/clientapi/user/storage`; `Media:QuotaBytesPerRoot` (0, none, by default) refuses an upload over it with 422; the
administrator's statistics overview has the totals, the proxy cache and the trash.
Nothing is deleted because it looks unused. `PrivaPub admin media audit [--fix]` compares disk and database: with
`--fix` (as www-data) it gives pictures shown from before their rows a row, and trashes media of deleted posts or
personas, rows whose files are missing, and files nothing holds.
6. **A client never contacts a remote server for media:** every remote URL the API returns goes through
`IMediaProxy.Wrap`, an HMAC-signed `/media/proxy/` URL fetched by `IFederationHttp.GetMedia`.
7. **The proxy serves three ways:**
- **Cached:** a file already cached is served from disk, ranges included. It is opened before the answer, so a trim
that deletes it meanwhile breaks nothing.
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached:
- once for everyone asking for it at the same time;
- streamed into a `.part` file beside its place and renamed there, never held in memory;
- with its host in its `.type` sidecar, so that blocking a server can purge it.
- **Streamed:** a ranged request, or anything too big to cache, is streamed from the origin with the range passed on,
and never cached. That is how remote video plays. A file found too big is remembered for an hour, so it isn't
fetched twice; one that failed is remembered for five minutes.
Also:
- The cache's size is counted as it grows and trimmed as soon as it passes `ProxyCacheBytes`; the janitor
recounts hourly.
- Browsers may cache only a success.
- Nothing of a suspended server, or of one whose media are rejected, is proxied (avatars, emoji, covers and link
cards included, since they all go through it), and blocking one purges its cache.
- The `proxy` rate limit counts per client address.
nginx has a `/media/proxy/` location with `proxy_buffering off` and a 600 s read timeout for those streams.
8. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made
every status, timeline and Note holding its post fail to serialise; migration `_016` removed the ones stored before.
9. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
up to 720p that carries both sound and picture, including PeerTube's fragmented files inside an HLS entry. HLS
playlists themselves are not rewritten.
## Privacy invariants
- **No root id in federation output, NodeInfo or logs, no IP next to an identity in logs, and no `ex.Message` to a
client** ("Something went wrong." instead).
- **Sign-in and recovery never tell which logins or addresses exist** (owner decision 2026-10-04): every failed sign-in
gets "That username and password do not match." after the same hashing (`RootUsersService.Decoy`, a constant-time
comparison), and only a correct password learns of a ban. Every recovery request answers the same sentence and
queues a `SendRecovery` job, found or not; `RecoveryJob` sends the email and keeps only a SHA-256 of the code, for an
hour. A recovered password ends every session of the root (`RootSessions`: a new `SessionStamp`, which every `/clientapi`
JWT carries, and OpenIddict revocation for each persona).
- **Deleting a root deletes everything public it had** (`RootRemoval`, from the admin route or `/clientapi/user/delete`
with the password): its sessions end, each persona and each group it owns sends `Delete{Actor}` to followers, members
and the accounts it follows, the personas' posts are emptied, and `/peasants/{name}`, its inbox and WebFinger answer
410 (`LocalActorService.Gone`) while the names stay reserved. The root keeps only a `deleted-{id}` name.
- **Never derive a public name from the root username.** Invitation sign-up takes `AvatarUserName` and refuses one
equal to the login.
- **One username space:** personas, groups and the instance reserve their name in `ReservedName` (unique index) before
they are saved; `LocalActorService.TryReserveUserName` is the only way to claim one.
- **Usernames are `Constants.UserNameRegex`** (`^[a-z0-9_]+$`): the intersection of what Mastodon and Misskey accept. A
name outside it creates a persona nobody on those servers can reach.
- **An actor's `published` is `PublishedOn`**, a random whole day up to two weeks before creation, so personas made the
same day do not share a date. New persona and group ids carry that same day (`GenerateNewID`), because local clients
see ids. Never emit `CreatedAt`.
- **Per-avatar state stays per avatar:** blocks, mutes, notifications, follows. Nothing may relate sibling avatars.
- **Blocks federate** (owner decision, 2026-10-01): a block is sent as `Block` from the blocking avatar, an unblock as
`Undo{Block}`. Reports still leave as `Flag` from the server's own actors (the instance actor, or the reporter for
communities on Lemmy, PieFed and Mbin), never from the reporting avatar.
- **What PrivaPub reveals is the owner's call.** Previews, blocks, website authorship, views, bridging and reactions were
decided in `docs/ROADMAP.md` ("Owner decisions on what PrivaPub reveals"). Anything new that tells another server
something about an avatar gets the same treatment: ask, then record it there.
- **Location-ranged posts never federate.**
- **Statistics name servers, never people** (owner decisions on statistics, 2026-10-03). Every interaction is recorded
through `IInteractionLedger` (`Infrastructure/Statistics`) as an `InteractionEvent` (90 days), but an event never
holds a root, persona or group id, an activity id, an inbox URL, a remote actor URI or a sender IP:
- Distinct remote accounts are counted with `ActorHash`, an HMAC keyed by that day's `InteractionSalt`, which the
day's rollup deletes.
- `LocalKind` survives `Interactions.Sanitize` only on public and unlisted traffic, so a circle can show neither by
presence nor by absence, and no reason code names one.
- Traffic a reader causes (the media proxy, lookups, the client API, fetches of our own documents) is only counted
per day (`Count`, `CountServer`), never logged per event.
- A host claimed by an unverified sender is kept only if it is already a `RemoteInstance`.
- `Record` never blocks and never throws: it writes to a bounded channel, and a full channel drops and counts.
## Data
- MongoDB through **MongoDB.Entities 25.1**, instance API: `DB.Default.Find<T>()`, `.SaveAsync(e)`, `.Update<T>()`,
`.DeleteAsync<T>()`, `.CountAsync<T>()`. `DbEntities` wraps the finds.
- **Pass no cancellation token to `DeleteAsync` outside a transaction;** v25 throws if you do.
- **Collection name = class name, so never rename an entity class.**
- `Entity.ID` is a 24-character lowercase hex string; `GenerateNewID()` returns `object`, so cast it.
- New fields must be additive: a deploy rollback restores the binary, not the database.
- **Every stored remote object has an `ObjectRecord`** (raw JSON up to 256 KB plus its hash; delivered or fetched; the
activity, inbox, key and signed headers; received time; up to 10 later revisions). Write it right after the post's
save: `CreateHandler` (delivered) and `RemotePosts.StoreContext` (fetched) do, and `UpdateHandler` appends a
revision. Delivery details reach them through `Arrival.Current`, which `InboxProcessor` sets for the handler's
duration. A fetched record has no signature, key or `@context` of the activity that caused the fetch (its activity
fields name that trigger), and every record stores its `Extensions` and `ContextNamespaces` for statistics. The raw form lives outside `Post` so timelines never load it. Read through
`/api/privapub/v1/statuses/:id/provenance` and `/api/privapub/v1/instances/:host` (or `?host[]=`, up to 40; this
server answers too).
- **A server's place leaves only through `PublicGeo.Project`:** the city, coordinates (0.1°) and network it was located
to, whatever its size (there is no user threshold), and only the CDN's name for a CDN-fronted one. This server's own
place is its host's address located once a day (`SelfLocation`), or `Statistics:Geo:Self` when the owner sets it.
- **CDNs are found three ways, best first** (`Infrastructure/Geo/CdnCatalog.cs`): the ranges a CDN publishes
(`CdnUpdater` downloads them daily into `CdnRangeSet`, keeping the old set when a download or parse fails), the CDN's
fingerprint in the responses PrivaPub already gets (`EdgeHintsHandler` on the federation client, memory only), and
a network that carries nothing but a CDN. Never list an ASN shared with plain hosting (AWS, Google, Microsoft,
DataPacket). A server's `Geo` records the CDN, its domain and how it was found.
- **Servers through time** (`Domain/Statistics/ServerPlaces.cs`, from the weekly `RemoteInstanceSnapshot`s):
`/api/privapub/v1/instances/:host/history`, a CDN-fronted server's `geo.before_cdn`, and the CDNs grouped by their
own domain (`/api/privapub/v1/cdns`, `/cdns/:domain` with weekly joined/left).
- **Remote objects are parsed for every shape in `ObjectShapes`:** `url`/`icon`/`image` as a value, an object or an
array; Markdown `content`; missing `mediaType`s inferred; thumbnails from any of their five places; and the typed
`Link`, `Video`, `Audio` and `Event` details. A Mastodon API card is built from those, never by fetching the linked page
(that fetch is the owner's decision 1, for P6).
- **An `Update` is an edit only when its `updated` is newer than ours**, or for a first edit no older and with the text
actually changed, since GoToSocial's whole-second timestamps make a quick edit carry `updated == published` (`RemoteEdits.IsEdit`; plain and community-wrapped
Updates both go through `RemoteEdits.Apply`, and deletes through `RemoteDeletes.Remove`). Otherwise it refreshes
the poll, video, audio and event details and nothing else, and leaves no revision: Mastodon and Misskey refresh poll
counts with bare Updates.
- **A poll vote is a `Note` with a `name`, an `inReplyTo` that is a poll we hold, and no content.** `CreateHandler`
hands it to `PollService.Receive` before anything else, so it never becomes a reply. Our votes on other servers' polls
go only to the poll's author, without `published` (PieFed counts a vote only then). A `closed` in the future is the
poll's end, not its closing: Akkoma sends an open poll's end only there (`ObjectShapes`).
- **Link previews follow owner decision 1** (`Domain/Content/LinkPreviews.cs`): only public posts, queued on arrival with
0–60 s of jitter, one cached `LinkPreview` per address for the whole server, never fetched when someone reads.
`LinkPreviews.Wanted` is called where posts are saved (`CreateHandler`, `StoreContext`, `StatusService.Publish`).
- **Quote states come from `QuoteService.Resolve`:**
- with a stamp: accepted only if `Verified` (fetched, on the quoted author's origin, naming both posts exactly);
- a FEP-044f `quote` without a stamp: pending;
- older keys only: accepted when the quoted post is public.
`QuotesCount` moves with the accepted state, never with the raw key. When a persona quotes, `QuotePermission` decides:
asking first for posts that state a policy, quoting at once for posts that state none, refusing otherwise. Only `quote`
of a post that asks for consent puts `quote` in our JSON; older-key quotes leave it out, as Sharkey learned they must.
- **Our quote policy is `ActivityPubRenderer.QuotableBy`:** the post's `LocalQuotePolicy`, falling back to the persona's
`Settings.QuotePolicy` (default `public`, owner decision), and always `nobody` for anything but public and unlisted. The
same rule writes `canQuote`, answers `QuoteRequest`s (`QuoteService.ReceiveRequest`) and fills `quote_approval`, so they
cannot disagree. A persona quoting another persona gets a parrot-licence too, so other servers see an approved quote.
- **A server is described on arrival, never on read.** The first record from a host enqueues `DescribeInstance` (its
NodeInfo, at most once a week, into `RemoteInstance`), so opening the details view tells nobody anything. A
description that failed frees its week, so the server's next arrival (an hour on at the soonest) asks again.
- **Post ids are the timeline order, so they follow arrival, not `published`.** `PrivacyIds.Arrived` gives a remote post
published within the last hour (or in the future) a fresh `ObjectId`, which sorts after every post already stored,
and only backfill keeps a `published`-derived id. With `published` ids a reply arriving in the same second could sort
under the post it answers, and a late arrival landed behind a client's `since_id` and was never seen. `created_at`
comes from `CreationDate`, never from the id.
- **The same ordering problem exists on the other side, so `published` carries milliseconds**
(`ActivityPubRenderer.Timestamp`). GoToSocial (ULIDs) and Mastodon (Snowflakes) derive a remote status's id from
`published` at millisecond resolution. With whole seconds, two of our posts from the same second sorted at random
there.
- **Startup order:** `EntityMaps.Warm()` (every entity's class map, one at a time; two mapped at once throw "An item
with the same key has already been added" and stay broken), then `MigrateAsync` (`Infrastructure/Data/Migrations`,
`_NNN_` order, each runs once), then `Indexes.Create()`. A new entity needs nothing; a new unique index needs a
dedupe migration before it.
- Production runs its **own mongod** on 127.0.0.1:27022 (unit `privapub-mongod`, no auth, data in
`/var/lib/privapub/mongo`). The box's shared mongod needs credentials nobody here has.
## Code style
- Tabs, block-scoped namespaces, Allman braces.
- New services use `readonly _camel` fields and constructor injection. Older services use PascalCase fields; leave
them.
- Services return `WebResult` (`PrivaPub.ClientModels/WebResult.cs`): `result.Invalidate(localizer[...], status)`.
Controllers turn it into a status code.
- Localised strings go through `IStringLocalizer<GenericRes>`.
- Every `Display`/`ErrorMessage` resource key must exist in `FieldsNameResource`/`ErrorsResource`, including the
`Designer.cs`, which the CLI build doesn't regenerate. A missing key throws at validation time.
- **`cond ? value : default` with a value-type branch is the type's default, not null:** `false`, `0`, or year one
stored as an edit date on every remote post. Write `(T?)null`. It has shipped four times (`MastodonParams.Bool/Int`,
`NoteParser.Int`, `NoteParser.Time`).
- ActivityPub output is built with `System.Text.Json.Nodes` in `ActivityPubRenderer`, not typed models. Inbound
documents are read through `ActivityJson.Id`/`Value`, which handle string, object and array.
- Libraries chosen for the roadmap: HtmlSanitizer, Markdig (`DisableHtml`), NSign (RFC 9421 inbound), OpenIddict +
OpenIddict.MongoDb, NetVips, Blurhash.Core, FFMpegCore. No ImageSharp (licence key enforced), no MassTransit.
## Testing
`PrivaPub.Tests` (xUnit v3). Unit tests need nothing; tests marked `Category=Integration` need a mongod and skip
without `PRIVAPUB_TEST_MONGOD=1`. CI (`build.yml` and `deploy.yml`) runs all of them through
`tools/ci/with-test-mongod.sh`, which starts a throwaway mongod on a random localhost port and deletes it afterwards.
The box's own mongods are production, so `MongoFixture` refuses port 27022, a data directory under `/var/lib/privapub`,
and in CI anything but the wrapper's mongod. With `PRIVAPUB_TEST_REQUIRE_MONGOD=1`, which the wrapper sets, a missing
mongod fails the run instead of silently skipping half the tests.
- `Support/Peer` is an in-process HTTP server answering on two origins (`127.0.0.1` and `localhost`), so origin rules
can be tested; `Support/RemoteActor` signs real deliveries with its own key.
- Inbox scenarios go through `InboxReceiver.Receive` with a signed request, not through the private handlers.
- `Support/Host/PrivaPubHost` is the whole server under test (`WebApplicationFactory<Program>`, environment `Testing`,
configured only through `UseSetting`, on the fixture's database). `PrivaPubHost.Shared()` boots it once per run;
`SecureModeHost` is the same with `Federation:SecureMode`. Background workers are removed, so a test runs the jobs it
queued with `host.Run(j => ...)` or `host.RunInbox(activityId)`. `Accounts` signs a root up, adds personas, and gets a
Mastodon token through the real `/oauth` code flow; `RemoteActor.SignedPost`/`SignedGet` sign `HttpRequestMessage`s
for the real `/peasants` routes. Each client gets its own `X-Test-Client` address, so rate limits don't collide.
- All test classes share one database and run in parallel, so a test touches only rows it made:
- random names and GUIDs;
- `Harness.Outgoing` sees only deliveries queued since that harness started, because Peer ports are reused;
- a worker gets a scoped `new JobQueue(j => ...)` so it never leases another test's jobs;
- domain blocks are set with `DomainBlocks.Load`, never written to the database.
A test that must change something database-wide (drop indexes, run a migration over every post, let deliveries to
`localhost` fail and trip its breaker) goes in `[Xunit.Collection(nameof(Exclusive))]`, which runs alone, and
cleans up after itself. Pure logic belongs in an unconditional unit test, not in a Mongo-gated class.
Beyond the tests, verify by building, running locally, and exercising:
- the client API (sign up, create an avatar, a group, a post);
- the ActivityPub endpoints with curl and `Accept: application/activity+json`.
Interop is checked against real servers, starting with the workstation's own pasture:
```bash
DOTNET=~/.dotnet/dotnet tools/pasture/run.sh up [gts mastodon ...] # podman: PrivaPub + the named peers (default gts) + Mongo, behind Caddy
tools/pasture/interop.sh [gts mastodon ...] # each peer's scenario, then what PrivaPub's statistics saw of it
tools/pasture/run.sh down # removes every pasture container and volume
```
- **A change to what PrivaPub sends** (the `@context`, an actor's properties, an activity's shape, a delivery's headers)
is run against every peer's scenario before it is committed, not only the peers it was written for. One reader
stricter than the rest fails on it alone: the 2026-10-06 `proof` term passed every peer but Smithereen, whose JSON-LD
1.0 reader refused every document carrying it.
- **Layout:** `lib/pasture.sh` (network, Caddy, Mongo, PrivaPub), `peers/<name>.sh` (`<name>_up`, plus `peers/shared.sh`
for the Postgres and Redis several peers share), `lib/interop.sh` (`ok`, `ko`, `xf` for a check expected to fail until
a later phase, `privapub_token <persona>`, `stats_check <host> <software>`), `scenarios/<name>.sh`. Each peer talks to
its own PrivaPub persona under one root, so OAuth's persona choice is exercised too. Images are pinned.
- Caddy's CA lives in the `pasture-caddy-data` volume and is copied to `tools/pasture/.ca/root.crt` (and `bundle.pem`
with the system roots) for peers that must trust it instead of skipping verification.
- **All sites on one podman network, one Caddy in front.** `privapub.test`, `gts.test`, `mastodon.test` and the other
peers are network aliases of the Caddy container, which serves them all with its internal CA (`tls internal`).
PrivaPub accepts any certificate (`appsettings.Pasture.json`) and GoToSocial is told to skip verification
(`GTS_HTTP_CLIENT_TLS_INSECURE_SKIP_VERIFY`); Mastodon trusts the copied CA through `SSL_CERT_FILE`. Peers fetch over
https only, so plain http between them is not an option.
- From the workstation, PrivaPub's API is `http://127.0.0.1:6971`. A peer is reached as `https://<name>.test:6443` with
`curl -k --resolve <name>.test:6443:127.0.0.1`, because sign-in cookies are bound to the host name.
- **GoToSocial (0.22.1):**
- Its cached home timeline can stop taking new posts after its first read, its owner's own included, while a
`min_id` query shows them all. So a delivery is checked by looking the object up by URI with `resolve=false`
(`on_gts`), which answers from GoToSocial's database and never fetches from us. A home-timeline check there proves
nothing, in either direction. A deleted status still turns up in that search as a "deleted status" stub, so a
delete is checked as a 404 on `/api/v1/statuses/{id}`.
- It creates its accounts locked, so the scenario approves alice's request through `/api/v1/follow_requests`, which
also checks our pending (`requested`) state and the manual Accept.
- 64 checks: discovery and follows both ways; posts and CW; a reply and its notification; likes and boosts both
ways; DMs both ways and off public timelines; polls both ways; quote policy; link cards; edits and deletes both
ways; communities and circles; locked personas; interaction policies (a reply and a like asked for and approved
through `/api/v1/interaction_requests`, a boost refused); unfollow; block and unblock; statistics.
- It files a circle post as a direct message and shows it only to the accounts it mentions, so like a DM it is
checked in the member's `/api/v1/conversations`, never by URI.
- **Mastodon (4.7.3):** web and sidekiq on the shared Postgres and Redis, `ALLOWED_PRIVATE_ADDRESSES` for the network.
Its token comes from `rails runner` (no password grant). Without Elasticsearch its status search finds nothing, so
deliveries are checked through `/api/v1/accounts/:id/statuses` of the sender as Mastodon knows them, or
`Status.exists?` through `rails runner`. Its actors are numbered (`/ap/users/<id>`), so look URIs up rather than
build them. Circle posts and a followers-only post survive its signed refetch (`ActivityPub::FetchRemoteStatusService`
through `rails runner`). Its block of a persona, and the unblock that ends the run, are checked through our
`blocked_by`; the run starts with that unblock too, since Mastodon rejects every follow from an account it blocks.
- **Misskey (2026.10.0):** one container on the shared Postgres and Redis. A new Misskey federates with nobody
(`federation: none`) until `admin/update-meta` says `all`, which `misskey_up` does. Its API is `POST /api/<endpoint>`
with the token as `i` (`mk` in the scenario); `users/relation` answers a list, `users/notes` leaves replies out unless
asked, and a user has one reaction per note. Never verify a delivery with `ap/show`: it fetches. 35 checks.
- **Sharkey (2025.4.7):** `peers/sharkey.sh` is the Misskey peer under another name, image, database, Redis db and
home (`/sharkey/.config`); `scenarios/sharkey.sh` runs Misskey's scenario with `MISSKEY_NAME=sharkey`, then follows
again (Misskey's ends unfollowed and blocked) and checks edits both ways and the FEP-e232 quote tag. 40 checks.
- **Akkoma (3.20.1):** no official image, so `images/akkoma` installs the OTP release (pinned by checksum; the "stable"
zip moves, and a moved one fails the build) and `akkoma_up` builds it once. Its HTTP clients read the CA bundles
shipped in the release (CAStore, certifi), so the entrypoint appends Caddy's CA there. `pleroma_ctl` passes its
arguments on unquoted: no value may contain a space. Its Linkify never takes `@user@host.test` for a mention, so the
scenario addresses alice with Pleroma's `to[]`; its API never reports a remote blocker as `blocked_by`, so the block is
read from its `user_relationships`. `akkoma_token <user>` signs one of its users in. 45 checks.
- **Lemmy (1.0.0-beta.2):** the backend alone on the shared Postgres, its admin made by `setup` in the generated
`config.hjson`. It trusts Caddy's CA through `SSL_CERT_FILE` and reaches the network through
`DANGER_FEDERATION_ALLOW_LOCAL_IP=1`; 0.19 cannot join (its rustls trusts only its bundled roots). Its API is
`/api/v4/<path>` with a bearer token (`lm` in the scenario), and `sort` values are lowercase. It logs no refused
activity at `warn` (`LEMMY_LOG` sets `RUST_LOG`); the reason is in the 400's body. It answers our community's echo of
its own activity and every bare `Announce{object}` 400 by design, and the echo is still needed (see
`docs/INTEROP.md`, Lemmy). A new Lemmy never sends what it queued for a server before its send worker for that
server started, so the scenario waits for that worker (`lm_worker`) before its first follow, and it sends what it
queued every 30 seconds, so a vote or a moderator's act takes up to a minute. Its lock, unlock and unban need a
`reason`, or it refuses them and federates nothing. 29 checks, among them a moderator's lock, ban and removal.
- **PieFed (1.7.17):** dockurr's image of the release, its web app (with PieFed's own cron, `CRON=true`, for its send
queue) and a Celery worker on the shared Postgres and Redis (dbs 10 and 11), sharing the `pasture-piefed-media`
volume. httpx trusts only certifi's bundle, so the pasture's is mounted over it. `flask init-db` reads its admin
(pfuser) from stdin and drops every table it finds, so it runs once, after the web app's migrations ("Starting
Gunicorn"). Its API is Lemmy's v3 under `/api/alpha` with a JWT (`pf` in the scenario); `resolve_object` answers a
view (`community.community.id`). It sends a community's announces to the Application at a peer's root, assuming
`/inbox` when there is none, and keeps serving a thread its moderator removed. `scenarios/piefed.sh`, 29 checks:
communities both ways, threads with titles, comments, votes up and down both ways, a community poll and alice's vote,
private messages both ways, a moderator's lock, unlock and removal, the unfollow, statistics.
- **Mbin (1.10.1):** its own image (FrankenPHP serving plain HTTP behind Caddy, `SERVER_NAME=:80`) and a messenger
worker (`mbin_worker`, consuming again whenever it stops: once, the shared Postgres restarted under it and Mbin sent
nothing for a day), on the shared Postgres and Redis (db 12) with a RabbitMQ of its own (its transports carry AMQP options; it
runs as its own user on its own volume, or it cannot read the `.erlang.cookie` it wrote as root). The pasture's bundle
is mounted over the system one; its API's rate limits (two threads every six minutes) are raised by a copy of its
`rate_limiter.yaml`. Its admin mbuser is made by its console; `peers/mbin_token.py` gets mbuser's OAuth token through
the authorization-code flow (login form, consent), since a client-credentials client acts as a bot that may not vote.
Mbin names what it makes during a request after the request's host, so every call says `Host: mbin.test`, never
the workstation's port. Its access token lasts an hour, and the scenario gets a new one when it has expired. Its API
never starts a conversation with an account elsewhere, so mbuser answers in the thread alice began; Mbin addresses
that answer to alice's profile page. `scenarios/mbin.sh`, 26 checks: magazines both ways, threads with titles, a Note
to a magazine as a microblog post, comments both ways, favourites and upvotes both ways, private messages both ways,
a moderator's lock, unlock and removal, the unfollow, statistics.
- **NodeBB (4.16.1):** the official image on the pasture's Mongo (database `nodebb`), set up once by its automated
setup (`SETUP` with `NODEBB_*` variables, admin nbuser) into the `pasture-nodebb-config` volume, trusting the CA
through `NODE_EXTRA_CA_CERTS`; its image runs `npm install` at every start. Its API (`/api/v3`) takes a bearer token
that `nodebb_settle` writes where NodeBB keeps its tokens. A remote account's uid there is its actor's URL, but its
API follows one only when named by its handle; a chat room is made, then written in. `scenarios/nodebb.sh`, 23
checks: a category followed and its topic as a titled thread, replies both ways, nbuser following alice, votes both
ways, an edit and a deletion, a chat both ways, the unfollow, statistics.
- **Lemmy 0.19 (0.19.20), as `lemmy19.test`:** what most of the threadiverse runs. Its release trusts only the roots its
rustls bundles, so `images/lemmy19` builds the tag with reqwest's `rustls-tls-native-roots` added (about ten minutes
the first time) and the pasture's bundle is mounted as the system's. Its config names the database as `uri` (1.0:
`connection`) and its API is v3 (`/api/v3`, `sort=New`, `resolve_object` answering views). It takes private messages
only as `ChatMessage`, which PrivaPub sends it, a first message too (invariant 17). `scenarios/lemmy19.sh`, 30
checks.
- **Threads (`scenarios/threads.sh`, needs mastodon):** mastouser follows alice, carol answers alice's public post,
and opening the thread on Mastodon finds carol's reply through alice's `replies` (the scenario makes Mastodon's copy
look ten minutes old: Mastodon reads a thread's replies only five minutes after it first holds the post). 5 checks.
- **Followers synchronisation (`scenarios/followsync.sh`, needs mastodon):** sent: PrivaPub forgets mastouser's follow
of alice (mongo), alice's followers-only post naming mastouser makes Mastodon read her roll-call and drop it; then
Mastodon forgets a follow (rails) and alice's next followers-only post makes it send the `Undo` PrivaPub applies.
Received: alice and bob follow mastouser; Mastodon forgets alice's follow and mastouser's followers-only post makes
PrivaPub end it, then PrivaPub forgets it and the next makes PrivaPub undo it there. Follows are checked in Mastodon's
database: its API caches relationships for a day. 15 checks.
- **Pins (`scenarios/pins.sh`, needs mastodon):** a Mastodon account pins and unpins while alice follows it, alice pins
and unpins while it follows her, a fresh account's earlier pin shows once PrivaPub resolves it, and following it brings
its earlier posts (its outbox). 9 checks.
- **Vernissage (1.43.0):** its image on SQLite in the `pasture-vernissage` volume, its queues on the shared Redis's
database 0 (it federates nothing without them; every other database is taken), its seeded admin (admin / admin)
logged in by the scenario. `scenarios/vernissage.sh`, 19 checks.
- **BookWyrm (0.9.3):** its image, migrated, on the shared Postgres (database `bookwyrm`) and Redis (databases 14
and 15), gunicorn and one Celery worker (federation runs there). It has no client API: bwuser (named
`bwuser@bookwyrm.test`, as its signup names users), a book and every status come from its Django shell
(`bookwyrm_shell`, `-v 0`). `scenarios/bookwyrm.sh`, 20 checks.
- **Ghost (6.67) with its ActivityPub service (1.2.14):** Ghost and the service on the shared MySQL (databases `ghost`
and `activitypub`, the service's migrations run first), routed by Caddy as Ghost's own proxy does. The owner comes
from Ghost's setup API (a password that repeats the site's name is refused), staff device verification is off, and
Ghost starts again once so it sets up its publishing webhooks. Calls to the service say `Host: ghost.test`, or it
cannot find the site's keys. `scenarios/ghost.sh`, 22 checks.
- **Owncast (0.3.0):** its image, its data in the `pasture-owncast` volume, federation turned on through its admin
API (`owncast_admin`, its default admin / abc123) with the server's address and the actor's name.
`scenarios/owncast.sh`, 13 checks.
- **WriteFreely (0.17.2):** built from its release tarball with `openssl` beside it (`images/writefreely`, it makes
each blog's keys with the command), on SQLite in the `pasture-writefreely` volume, which the scenario reads from the
host. wfuser and its blog come from its CLI; its API takes a login. `scenarios/writefreely.sh`, 13 checks.
- **snac2 (2.95):** built from its tag (`images/snac2`, the project publishes no image), its data in the
`pasture-snac` volume from `snac init` and `snac adduser`; the password comes from `snac resetpwd` and the token from
its login form and a code grant. Its API lags behind what it has taken in, so `scenarios/snac.sh` reads its files
(`snac_stored`, `snac_in_timeline`). 27 checks.
- **Mitra (5.9.1):** its image on the shared Postgres (database `mitra`), as `mitra.test`, with the pasture's bundle as
the system's roots; mitrauser comes from its CLI and its token from a password grant. Its search resolves an account
elsewhere only without `type`. `scenarios/mitra.sh`, 28 checks.
- **Relays (`peers/relay.sh` Activity-Relay 2.0.9 as `relay.test`, on the shared Redis's database 13;
`peers/aoderelay.sh` aode-relay 0.3.129 as `aoderelay.test`):** `appsettings.Pasture.json` names both in
`Federation:Relays`, so PrivaPub subscribes a minute after it starts. `scenarios/relay.sh` has Mastodon subscribe to
each in turn, checks that a post of an account nobody here follows reaches the federated timeline (forwarded by one,
announced by the other), that alice's public post goes to the relays (and through both to Mastodon: Activity-Relay's
forward on its FEP-8b32 proof, after the scenario makes Mastodon read alice's keys anew) and nothing less public, and
has Mastodon leave again, so the town sees no relayed posts. 16 checks.
- **Castopod (1.15.5, `peers/castopod.sh`):** the official image on the shared MySQL (database `castopod`, file cache,
`CP_DISABLE_HTTPS`), its database and superadmin (admin@castopod.test) from `spark`, its REST API switched on in the
`.env` the image rewrites at each start, `spark fediverse:broadcast` looping in the container. The podcast `@pod` is
made and published through its admin pages (`cp_web`, CSRF on each form). `scenarios/castopod.sh`, 15 checks; the
episode's sound is made with ffmpeg on the workstation.
- **Ktistec (3.13.0, `peers/ktistec.sh`):** built from its tag by `images/ktistec` (its own Dockerfile, pinned: a static
Crystal server, its assets with node), SQLite in the `pasture-ktistec` volume, the pasture bundle mounted as
`/etc/ssl/cert.pem`. The server's name and ktuser come from its setup API (every field required, `summary` too);
`ktistec_token` signs in on `/sessions`. `scenarios/ktistec.sh`, 27 checks: its Mastodon API where it has the route,
its outbox API (`kn`) for follows, polls (`poll-duration` is seconds from now), edits, deletions and quotes, and a
copy of its SQLite database (`kt_sql`) for what it holds.
- **Forte (26.9.10, `peers/forte.sh`):** built from its tag by `images/forte` (composer on Hubzilla's PHP image, an nginx
routing through `index.php?req=`), on the shared MySQL (database `forte`), its `.htconfig.php` written to
`.state/forte`, a cron sidecar (`src/Daemon/Run.php Cron`). The pasture bundle is mounted over `library/cacert.pem`,
the bundle its curl uses. ftuser (administrator) and ftfriend are made through its PHP (`forte_eval`: `Account::create`,
`Channel::create` with the role `social` and `autoperms`, or every connection waits). Connecting from the command line
runs the notifier itself (`ft_connect`). `scenarios/forte.sh`.
- **Hubzilla (11.4.1, `peers/hubzilla.sh`):** the hlhd image (php-fpm and nginx on 8080) on the shared MySQL
(database `hubzilla`, its schema loaded from the image), `.htconfig.php` written to `.state/hubzilla` and mounted, a
cron sidecar running `Zotlabs/Daemon/Master.php Cron` each minute. Its addons `pubcrawl` (ActivityPub) and `statistics`
(NodeInfo) are installed with `util/addons`; hzuser (the hub's administrator) and hzfriend are made through its own PHP
(`hz_eval`), as accounts written the way its registration queue would and channels with the role `public` (the older
role names grant connections nothing), the ActivityPub app installed as each channel. Its API (`api/z/1.0`) posts and
edits with HTTP Basic (email and password); liking and dropping go through its web pages signed in (`hz_web`).
`scenarios/hubzilla.sh`, 20 checks and one known gap (G-0010, its unfollow).
- **Smithereen (1.0.3):** its image on the shared MySQL (database `smithereen`, its schema from the image's commit),
with imgproxy and a file server behind Caddy as `smithereen.test` (`/i` and `/s`), trusting the CA through a JDK
store with it added (`JAVA_TOOL_OPTIONS`). MySQL takes its stored functions only with
`log_bin_trust_function_creators`, and a first start that failed there leaves the database without triggers (drop it
and start again). Accounts come from its signup form (opened, with a short description, which its NodeInfo needs) and
are renamed in its database; its API is VKontakte's (`/api/method/<name>?v=1.0`), with a password grant for a local
application the peer script inserts. `scenarios/smithereen.sh`, 40 checks, walls both ways (alice's profile is
told again first, so Smithereen reads her `sm:wall`; `servers.features & 1` says it took PrivaPub for a server with
walls). Its scenario texts avoid apostrophes: `p_home_has` puts them inside a Python string.
- **Load (`load.sh`, needs the `flood` peer):** `flood` (`flood/flood.cs`, published once into `.flood`) answers as
twenty fake servers and sends signed activities at a set rate; `load.sh --rate=N --seconds=N` measures the answers,
the queue's wait and processing times, its drain, and a persona's home timeline meanwhile, and keeps each run in
`out/load/`. `docs/LOAD.md` has the method and the runs.
- **Account moves (`scenarios/moves.sh`, needs gts):** two fresh GoToSocial accounts made by its admin CLI; alice follows
the old one, the new one names it as an alias (`/api/v1/accounts/alias`), the old one moves (`/api/v1/accounts/move`),
and PrivaPub shows it `moved` and moves alice's follow to the new one, which (locked by GoToSocial) approves it.
8 checks.
- **Hollo (0.9.19):** Fedify's microblog server on the shared Postgres, set up through its web form (which
checks `Origin` against `Host`, so the request names `hollo.test` without the port). It needs a 44-character
`SECRET_KEY`, a media directory and a `themeColor`; statuses and votes go as JSON. Town only, no scenario.
- **PeerTube (8.3.1):** the official image on the shared Postgres and Redis (db 4), configured through `PEERTUBE_*`
variables, trusting Caddy's CA through `NODE_EXTRA_CA_CERTS`; `peertube_settle` turns transcoding off (a test video
is served as uploaded) and keeps root's token in `.state/peertube.token`. It wants a bare `Host: peertube.test`.
Its list of comments held for review (`/api/v1/users/me/videos/comments?isHeldForReview=true`) answers 500 in 8.3.1;
the video's owner sees them in the video's threads (`heldForReview`). `scenarios/peertube.sh`, 28 checks.
- **Pleroma (2.10.2):** `images/pleroma` installs the OTP release, pinned by checksum, as Akkoma's does; it also needs
libvips, and `instance gen` asks about deduplicating uploads. Its federation runs on hackney, which trusts only
certifi's compiled-in roots, so the entrypoint points `:pleroma, :http, adapter` at the system CA bundle. It answers
an inbox POST at once and verifies in a worker (`oban_jobs` shows what it refused and why). Town only, no scenario.
- **Iceshrimp.NET (2026.1.2-beta):** on the shared Postgres with AuthorizedFetch on and open registrations, trusting
Caddy's CA through `SSL_CERT_FILE`. Accounts come from its own `/api/iceshrimp/auth/register` (its login cuts the
connection short for a name it does not know), Mastodon API tokens from its OAuth form, which prints the out-of-band
code in the page. Its `jobs` table shows what it queued for whom. Town only, no scenario.
- **Pixelfed (0.14.4):** serversideup's FrankenPHP image on the shared Postgres (database pixelfed) and Redis (dbs 5 and
6), with Horizon and the scheduler as sidecars of the same image sharing its storage volume, and Caddy's CA mounted
as its system bundle. `pixelfed_settle` makes `caption` nullable (its PostgreSQL migration never runs, so every remote
boost failed), creates Passport's keys and restarts the web server so FrankenPHP reads them, numbers the OAuth
clients from a million (Passport refuses a token whose user id equals its client's id), and imports its cities.
Tokens are personal access tokens made through tinker (`App\Models\User`). Every top-level post needs a picture.
`scenarios/pixelfed.sh`, 26 checks; the town's driver and `specs/pixelfed-pair.json`.
- **WordPress (7.1.2, ActivityPub plugin 9.3.1):** the official image on a shared MySQL 8.4 (`shared_mysql_up`, ready
only when it answers over TCP: its first start runs a server without networking), installed and configured by wp-cli,
its CA bundle (`wp-includes/certificates/ca-bundle.crt`) given Caddy's root, and WP-Cron run every five seconds by a
sidecar (`DISABLE_WP_CRON`), since the plugin federates from it. Its automatic updates are off
(`AUTOMATIC_UPDATER_DISABLED`): the sidecar ran one (6 to 7.1.2) whose new CA bundle dropped Caddy's root, and
WordPress reached nobody after it. Authors are actors; the REST API takes application passwords.
`scenarios/wordpress.sh`, 17 checks.
- **Friendica (2026.05):** the official image on the shared MySQL (database friendica) and Redis (db 7, for its cache,
locks and sessions), installed by its autoinstall, with the image's worker daemon (`cron.sh`) as a sidecar sharing
its files. `friendica_settle` repairs what the install leaves: it names the system user (uid 0), or Friendica never
makes the system account that signs its fetches; it declares the daemon running (`worker_daemon_mode` in
`key-value`), or the web container, which cannot see the sidecar's pid, never wakes it and everything waits for its
five-minute cron; it makes the log file and turns logging on. `friendica_user` saves each account through the API
with `locked=0` (a number: "true" reads as 0), which makes it a "soapbox" page that takes followers without asking and
follows nobody back, and reads its outbox once: a Follow that reaches a user Friendica has not cached yet makes it
fetch that user from itself, signed, and checking the signature recurses for about five minutes. Its API takes HTTP
Basic (`nick:password`); an edit through it never federates, so the scenario edits in the web editor.
`scenarios/friendica.sh`, 25 checks; the town's driver and `specs/friendica-pair.json`.
- **Mobilizon (5.2.4):** kaihuri's image of the release on a PostGIS of its own (`shared_postgis_up`: it needs the
extension), with the pasture's bundle mounted over certifi's and castore's CA files (hackney trusts only those) and
the system store, and geocoding pointed at a closed local port. `mobilizon_ctl users.new` makes mzuser (it splits
its arguments on spaces); the API is GraphQL (`/api`), signed in by the `login` mutation. An event made through it
without options has its comments closed. `scenarios/mobilizon.sh`, 27 checks: a group and its events (dates, place),
alice joining an event (Mobilizon's participant row and its Accept) and leaving it, comments both ways, the
organiser's edit, closed comments and deletes, a group post, the unfollow.
- **Gancio (1.28.2):** cisti's image on sqlite in the `pasture-gancio-data` volume, whose `config.json` is written before
the first start (without it Gancio waits in its setup wizard), trusting Caddy's CA through `NODE_EXTRA_CA_CERTS`.
`gancio users create` makes gcadmin; `gancio settings set enable_resources true` keeps fediverse replies. One
Application actor, `relay`, publishes the agenda. Its API takes a token from an OAuth password grant
(`/oauth/login`, client_id `self`); an event is a multipart POST (`--form-string` for text starting with `<`).
`scenarios/gancio.sh`, 17 checks.
- **Funkwhale (2.0.11):** the API (gunicorn), a Celery worker with its beat and the front's nginx, sharing the
`pasture-funkwhale-data` volume, on the shared Postgres (database funkwhale) and Redis (dbs 8 and 9); Python's requests
trusts the pasture's bundle through `REQUESTS_CA_BUNDLE`. `funkwhale-manage fw users create` makes fwuser and an OAuth
access token is made in its Django shell. Its API is `/api/v1` and `/api/v2` (uploads); a track is a fresh tone made
by ffmpeg in its container, since it skips a file it already has. `scenarios/funkwhale.sh`, 16 checks.
- **SecureMode:** `PRIVAPUB_ENV="Federation__SecureMode=true" run.sh up …`, as production runs. A check of what an
unsigned reader sees uses `unserved` and `gone_unsigned` (`lib/interop.sh`), which expect 401 when SecureMode is on and
404 or 410 when it is off.
- **Crawler:** `PRIVAPUB_ENV="Statistics__Crawler__Enabled=true Statistics__Crawler__Seeds__0=mastodon.test"
run.sh up mastodon`, then `interop.sh crawler`. `PRIVAPUB_ENV` passes any setting to the PrivaPub container.
- `run.sh up` replaces every container, Mongo included, so each run starts clean. To keep the data, republish into
`tools/pasture/.publish` and `podman restart pasture-privapub`; that is how a migration is tried on dirty data.
- `run.sh add <peer>` starts more peers next to a running pasture; `run.sh down` removes every pasture container with
its volumes (`--purge` drops Caddy's CA too); `run.sh stats` shows each container's memory.
- **The network looks public** (`PASTURE_SUBNET`, default `11.42.0.0/24`), so peers that refuse private addresses with no
switch (Pixelfed, Mbin, WordPress, Discourse, Fedify) federate. Mastodon then needs `TRUSTED_PROXY_IP` for Caddy, or it
answers every request with an IP spoofing error.
- **`PASTURE_PORT`** moves PrivaPub's workstation port off 6971 when something else holds it; `lib/interop.sh` follows.
- Every `ok`/`ko`/`xf` is also a line of `out/scenarios.jsonl`, which the town's report shows.
#### The town (`tools/pasture/town.sh`, `tools/pasture/town/`)
A fake community seeded across every running peer, then checked for coherence: what each server holds (read from its
database, never by making it fetch), what each account sees through its API, counts, threads, edits, deletes, follows
and the privacy rows. Python 3 standard library only, tabs.
```bash
tools/pasture/town.sh selftest [peer...] # each driver against its own server; run after changing a driver or a pin
tools/pasture/town.sh seed village # specs/village.json: 23 accounts on seven servers, about 11 minutes
tools/pasture/town.sh check village # out/<run>/results.json; --deadline=0 re-sweeps an old run at once
tools/pasture/town.sh report village # out/<run>/report.html, publishable (fake data, no tokens)
tools/pasture/town.sh report village hollo-pair iceshrimp-pair pleroma-pair pixelfed-pair friendica-pair # one matrix
tools/pasture/town.sh backlog --write # docs/INTEROP-BACKLOG.md
```
- **Drivers** (`dialects/`): one class per platform on a dialect base (`mastodon_api`, `misskey_api`, `lemmy_api`,
`privapub`). `stored()` reads the database (shared Postgres through `psql`, the shared MySQL through `mysql_json`,
GoToSocial's sqlite copied out, Mongo);
`seen()` asks the API as one account. A driver that cannot do something raises `Unsupported`, and the planner never
asks for it (`gen.CAPS`).
- **The plan** (`gen.py`) is a pure function of the spec and its seed; each run keeps its own `plan.json`, `ledger.jsonl`
(what happened, one fact per line) and `results.json`. `seed --resume` redoes only failed steps.
- **Cells** are `feature|origin|observer|direction` (`in`, `out`, `via`, `control`, `local`). A known gap is a
`gaps.json` entry: matching failures are `xfail`, matching passes `xpass` (close the gap).
- **What the peers do on purpose**, so the checker expects it: Misskey and Sharkey drop a reply whose parent they cannot
fetch; Lemmy keeps only community content and private messages; servers send edits and deletes to the post's own
audience only; a Misskey heart reaction arrives as a Like.
- Peers' own limits are lifted for the town: GoToSocial `GTS_ADVANCED_RATE_LIMIT_REQUESTS=0`, Lemmy's
`local_site_rate_limit`. Mastodon reserves usernames that contain `mastodon`.
- decePubClient's e2e tests (`decePubClient/tests/e2e/run.sh`) publish the client with the Pasture environment, serve
it as `https://decepub.test` (`peers/decepub.sh`) and write their cells to `out/client.jsonl`.
After the pasture, verify.funfedi.dev and the owner's GoToSocial at social.arasaka.software. **Ask before acting from
the owner's GoToSocial account.**
## Deploy
- **CI/CD:** push to `master` runs `build.yml` (build + tests) on the instance-wide `build` runner. A `v*` tag runs
`deploy.yml`: tests, self-contained linux-x64 publish, snapshot and `mongodump` to `/var/backups/privapub.thepra.dev`
(never the statistics salt: `InteractionSalt` is excluded and the dump is checked for it),
stop → rsync → start, a `127.0.0.1:6970/build.json` health loop with rollback, then public checks (actor, NodeInfo,
Swagger 404, inbox junk 400, unsigned 401) and `tools/smoke/mastodon-api.sh` (app registration, client credentials,
discovery, instance, public timeline). Then it checks that what production should be running is running:
- NodeInfo and the instance API agree on registrations (closed, invitations only);
- `/stargazing` says the crawler is on;
- **@thepra signs in**: `PrivaPub admin smoke thepra` gives the root `deploy-smoke` a new password, `tools/smoke/oauth.sh`
runs the real OAuth flow (the pasture's `privapub_token` uses the same script), the signed-in API is checked, the
persona must be undiscoverable, and the token is revoked;
- the DB-IP Lite databases, which the server fetches itself (`GeoUpdater`), exist and are at most 40 days old.
- **The box:** Max (`nuvola.xyz`). Unit `privapub` runs as www-data from `/var/www/privapub.thepra.dev` with
`ASPNETCORE_ENVIRONMENT=Production`.
- **One-time root setup:** `deploy/max/setup.sh`, run through `../arasaka.software/tools/max/run.sh` (directories, the
runner's sudoers line, the units, nginx and the certificate). Nothing that runs later needs it again.
- **Config:** `appsettings.Production.json` is committed and deployed, **secrets included, by the owner's convention**
(the same as arasaka.software). A hand edit on the box is lost at the next deploy.
- **Secrets drift:** `AppConfigurationService` copies `AppConfiguration` into Mongo on first boot and reads the stored
copy afterwards, so changing those keys needs a Mongo update as well.
- **Logs:** `journalctl -u privapub`, plus the `logs` database on the private mongod.
Reference in new issue View Git Blame Copy Permalink