Votes out. A persona's downvote is a Dislike (POST /api/v1/statuses/:id/downvote and /undownvote, the viewer's own as privapub.votes.downvoted). One vote a post: a changed vote is sent as the new vote alone, as Lemmy sends one, since an Undo of the old one beside it could arrive after it and take the new one away; Undo goes only when a vote is taken back. A vote on a post in a remote community goes to the community, which counts it, and to the author only when on another server: one copy a server, since PieFed drops a second copy of an activity it has just seen. Mbin keeps a favourite apart from a vote, so there a favourite stays after a switch to a downvote. Feeds followed (owner decision 2026-10-07: through the server). Following a feed (Lemmy's multi-community, PieFed's feed) keeps a FeedSubscription for the persona and nothing else; the new Service privapub_feeds (LocalActorKind.Reader, reserved by migration _017) follows every community of the feeds read here, reconciled when a persona follows or leaves one, when a feed's list is read again (kept as it was when it cannot be read) and every six hours. Its Following rows carry FollowerKind, so nobody's home gets what it brings in and its unanswered follows are sent again as its own. The persona reads GET /api/v1/timelines/feed/:id (the feed's threads, ours included) and lists its feeds at GET /api/v1/feeds. Checked in the pasture, every scenario: 873 pass. The 14 failures are Hubzilla's (identical on the previous commit: Hubzilla no longer answers a follow in this pasture since its restore) and two activities Smithereen never sent; followsync passes once the pasture's restore is older than the 14-day pause. Live: alice's downvotes count as downvotes on Lemmy 1.0 and PieFed, replacing her upvote; Lemmy 1.0 and PieFed take privapub_feeds' follows and their threads reach the feed timelines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
949 lines
86 KiB
Markdown
949 lines
86 KiB
Markdown
# CLAUDE.md
|
||
|
||
Guidance for working in this repository. `docs/ROADMAP.md` holds the owner's decisions and the phased plan to full
|
||
ActivityPub interop; read it before changing anything federation-, privacy- or API-shaped. `docs/INTEROP.md` is the
|
||
per-platform evidence behind phases P5 to P8: what each peer sends, what it expects, and what PrivaPub still drops.
|
||
Check the platform's section there before writing a parser or a renderer, and add to it whatever you learn.
|
||
|
||
## What this is
|
||
|
||
**PrivaPub** (repo name SocialPub) is a self-hosted ActivityPub server in C#, live at https://privapub.thepra.dev.
|
||
It is meant as a Pleroma-like microblogging server for privacy-minded people and small communities, federating with
|
||
Mastodon, GoToSocial, Pleroma/Akkoma, Misskey and Lemmy.
|
||
|
||
Its defining idea: **one private login owns several public personas.**
|
||
|
||
- **`RootUser` is the private login.** Username, password, optional email (used only for recovery), policies.
|
||
- **`Avatar` is a public persona.** Each one is an ActivityPub actor with its own RSA keys at `/peasants/{username}`.
|
||
Root and avatar are linked only through `RootToAvatar`.
|
||
- **Personas must stay unlinkable** to other users and servers; only the instance admin can see the link.
|
||
- **`Group` is an ActivityPub Group actor.** It has members, an invitation code and an optional password. Groups are
|
||
becoming two kinds: a public **community** (FEP-1b12, Lemmy-compatible) and a private, invitation-only **circle**.
|
||
- **`DmGroup` is a direct-message conversation.**
|
||
- **`privapub` is the instance actor** (type Application). It signs fetches no persona should be tied to.
|
||
- **`privapub_reports` is the reporter** (type Service, `LocalActorKind.Reporter`). It carries reports to Lemmy, which
|
||
takes none from an Application, and to PieFed and Mbin, and names nobody.
|
||
- **`privapub_feeds` is the reader of feeds** (type Service, `LocalActorKind.Reader`). When a persona follows a feed
|
||
(Lemmy's multi-community, PieFed's feed), it follows the feed's communities in the server's name
|
||
(`Domain/Social/FeedFollows.cs`, owner decision 2026-10-07); its `Following` rows carry `FollowerKind`, and nobody's
|
||
home gets what it brings in. All three are server actors (`LocalActor.IsServerActor`): never followed, mentioned or
|
||
shown as accounts.
|
||
|
||
Privacy features in the model:
|
||
- location-ranged posts (`Post.Location`, `RangeKm`), to become local-only and never federated;
|
||
- contacts a persona can share (`Avatar.SharedPersonalContacts`);
|
||
- private notes (`RootUserNote`, `PersonalNote`).
|
||
|
||
The intended client is the Mastodon client API (Tusky, Elk, Phanpy, Ivory), where each avatar logs in as its own
|
||
account. The private `/clientapi` covers what Mastodon can't express. `thepra/decePubClient`
|
||
(https://decepub.thepra.dev) is the owner's own Pleroma-FE-like PWA. It signs in once on `/clientapi` and exchanges that
|
||
JWT for each persona's Mastodon token (`PersonaExchange`), then uses both APIs.
|
||
|
||
## Route names are deliberate
|
||
|
||
The odd names are the owner's and part of the project's character. **Never "fix" them to conventional ones.** A new
|
||
actor-scoped route gets a name in the same spirit, agreed with the owner, and a name is frozen once it has federated.
|
||
|
||
| Thing | Route |
|
||
|---|---|
|
||
| Actor | `/peasants/{name}` (`/users/{name}` 301-redirects here; browsers are sent to `/@{name}`) |
|
||
| Inbox | `/peasants/{name}/mouth` |
|
||
| Outbox | `/peasants/{name}/anus` (`?page=true[&max_id=]` for pages) |
|
||
| Shared inbox | `/human-centipede` (also `/peasants/{name}/human-centipede`) |
|
||
| Followers | `/peasants/{name}/groupies` |
|
||
| Following | `/peasants/{name}/stalking` |
|
||
| Notes | `/peasants/{name}/scribbles/{id}` |
|
||
| Activities | `/peasants/{name}/grunts/{id}` (`create-{postId}` resolves) |
|
||
| DM context | `/peasants/{name}/whispers/{id}` |
|
||
| Quote permission (FEP-044f `QuoteAuthorization`) | `/peasants/{name}/parrot-licences/{id}` |
|
||
| Token refresh | `/clientapi/user/sniff/again` |
|
||
|
||
Agreed for later phases: `/gossip`, `/drool`, `/echoes` (replies, likes, shares), `/trophies` (featured), `/tattoos`
|
||
(featured tags), `/flock` and `/wardens` (group members and moderators).
|
||
|
||
Routes other software looks up by name stay conventional:
|
||
- `/.well-known/*` and `/nodeinfo/*`;
|
||
- `/api/v1/*` and `/oauth/*`;
|
||
- `/@{name}` pages;
|
||
- `/media/*`.
|
||
|
||
## Repo map
|
||
|
||
```
|
||
PrivaPub.sln
|
||
PrivaPub/ ASP.NET Core Web API, net10.0
|
||
Program.cs host, Mongo init (GUIDs Standard), pipeline, /build.json
|
||
Middleware/SocialPubConfigurations.cs every DI registration (auth, federation, services, swagger, CORS)
|
||
Controllers/ClientToServer/ /clientapi/*: RootUser (signup/login/invitations/recovery), PrivateAvatar,
|
||
Group, Post (posts + DMs), Admin, Data
|
||
Infrastructure/
|
||
Http/ FederationHttp + SafeHttpHandlerFactory + IpRangeGuard: the only way out
|
||
Jobs/ JobQueue (leases), JobWorker, Backoff, HostCircuitBreaker
|
||
Ids/ PrivacyIds (day-only ids for personas and groups, arrival-ordered ids for remote posts)
|
||
Data/ Indexes (created at start), EntityMaps.Warm, Migrations/_NNN_*.cs
|
||
Cli/ AdminCommands (`PrivaPub admin promote|demote <root>`)
|
||
RateLimiting.cs accounts (per client address) and inbox (per sending origin) policies
|
||
Federation/
|
||
Controllers/ PeasantsController (actor, outbox, followers, following, posts, inboxes),
|
||
WellKnownController (webfinger, nodeinfo), UsersController (redirect)
|
||
Actors/ LocalActorService (LocalActor, Keys, ReservedName), RemoteActorService
|
||
(authoritative fetch, key verification, WebFinger), ActorDocument (parser)
|
||
Objects/ Origin, ActivityJson, NoteParser, Addressing, ContentSanitizer
|
||
Moderation/ DomainBlocks (suspend / silence / reject media)
|
||
Signing/ HttpSignatures (draft-cavage sign/verify), MessageSignatures (RFC 9421 verify),
|
||
RequestSignature (whichever a request carries), IntegrityProofs (FEP-8b32
|
||
eddsa-jcs-2022 by a persona's Ed25519 key, FEP-521a Multikeys), Jcs (RFC 8785)
|
||
Inbox/ InboxReceiver (verify, queue, 202) → InboxProcessor (job) → Handlers/{Follow,Accept,Reject,
|
||
Undo,Create,Update,Delete,Like,Announce}; RemotePosts (build, fetch parents, FetchAncestors);
|
||
RemoteReplies (FetchReplies: a thread's `context`, else `replies` two levels down);
|
||
Forwarded (what a thread's server passes on, believed as far as the origin vouches);
|
||
ReplyRelay (third-party replies to a persona's posts passed on to its followers)
|
||
Outbox/ OutboxPublisher (who a post goes to), DeliveryService (queues jobs) + DeliveryJobHandler
|
||
Rendering/ ActivityPubRenderer (Mastodon @context, actors, notes, collections)
|
||
Domain/
|
||
Content/ ContentRenderer (Markdown or plain text → HTML with h-card mentions and hashtags)
|
||
Social/ FollowService (local in-process, remote Follow/Accept), Notifications
|
||
Timelines/ Fanout (TimelineEntry rows, Mastodon's home rules), TimelineService
|
||
Privacy/ VisibilityPolicy (IsPublic expression, CanSee)
|
||
Relationships/ RelationshipService (blocks, mutes, account domain blocks; Hidden), ReportService
|
||
Media/ MediaService (libvips, ffmpeg remux, blurhash), MediaProxy, MediaJanitor
|
||
Domain/Statuses/ StatusService: publish, edit, remove, favourite, reblog, for a persona (both client APIs use it);
|
||
InteractionApprovals: GoToSocial's canReply/canLike/canAnnounce, judged, asked and answered;
|
||
Participations: a persona's Join and Leave of a remote event, and the organiser's answer
|
||
Api/Mastodon/
|
||
Auth/ OpenIddict setup (keys in Mongo), MastodonScopes, TokenController, OAuthPruner
|
||
Infrastructure/ MastodonController (avatar context, scopes, errors, Link), MastodonParams, MastodonJson, Page
|
||
Entities/ Mappers/ Mastodon entities; MastodonMapper (Account, Status), AccountSearch
|
||
Controllers/ apps, instance, accounts, statuses, timelines, notifications, search, stubs
|
||
Web/Pages/ Razor: /@{user}, /@{user}/{id} (public posts only, strict CSP, noindex);
|
||
OAuth/: /oauth/login (root password), /oauth/authorize (choose persona, consent)
|
||
Services/ RootUsersService, GroupUsersService, PostsService, AppConfigurationService, …
|
||
Models/ Mongo entities: User/, Group/, Post/, Federation/, Jobs/, AppConfiguration
|
||
StaticServices/ DbEntities (Find<T> accessors), AuthTokenManager (JWT), PasswordHasher
|
||
Data/InitDb.cs first-run seeding (languages)
|
||
PrivaPub.ClientModels/ DTOs + validation resources shared with clients
|
||
PrivaPub.Tests/ xUnit v3; Support/ has a fake two-origin peer and a throwaway-database fixture
|
||
deploy/ nginx vhost, systemd units (privapub, privapub-mongod), max/setup.sh
|
||
.gitea/workflows/ build.yml (push) · deploy.yml (tag v*)
|
||
docs/ROADMAP.md decisions + phased plan
|
||
```
|
||
|
||
The roadmap moves code towards `Infrastructure/`, `Federation/`, `Domain/`, `Api/{ClientApi,Mastodon}` and `Web/`,
|
||
one pure-move commit at a time.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
dotnet build PrivaPub.sln -c Release
|
||
dotnet test PrivaPub.sln # unit tests; integration tests skip
|
||
PRIVAPUB_TEST_MONGOD=1 dotnet test PrivaPub.sln # all of them, against mongod on 127.0.0.1:27017
|
||
# (PRIVAPUB_TEST_MONGO overrides; a fresh database per run, dropped after)
|
||
tools/ci/with-test-mongod.sh dotnet test PrivaPub.sln # all of them, on a throwaway mongod, as CI runs them
|
||
cd PrivaPub && ASPNETCORE_ENVIRONMENT=Development \
|
||
Kestrel__Endpoints__Http__Url=http://127.0.0.1:6970 Kestrel__Endpoints__Http__Protocols=Http1AndHttp2 \
|
||
AppConfiguration__BackendBaseAddress=http://127.0.0.1:6970 MongoSettings__Database=PrivaPubTest \
|
||
dotnet run # needs a local mongod on 27017; swagger at /swagger
|
||
```
|
||
|
||
Development config (`appsettings.Development.json`) binds HTTPS 7195 with HTTP/2 only; the overrides above make it
|
||
curl-able. Outbound fetches only go to https DNS names resolving to public addresses; a test network (Pasture) sets
|
||
`Federation__AllowPrivateNetworks=true`, `Federation__AllowPlainHttp=true` and `Federation__AcceptAnyCertificate=true`,
|
||
which startup refuses in Production.
|
||
|
||
The admin CLI runs with every service built but nothing started (no Kestrel, no hosted services), so it can run next to
|
||
the live service. Sign-up is closed in production (invitations only), so the first login is made here; promoting is how
|
||
an admin is made (signing up as "admin" grants nothing):
|
||
|
||
```bash
|
||
cd /var/www/privapub.thepra.dev
|
||
echo '<password>' | ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin create-root <login> --admin # password on stdin
|
||
ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin promote|demote <root>
|
||
ASPNETCORE_ENVIRONMENT=Production ./PrivaPub admin smoke <persona> # the deploy's: prints "deploy-smoke <new password>"
|
||
```
|
||
|
||
The deploy runs them as `build-runner`, which owns the published files, reads `appsettings.Production.json` through
|
||
group www-data and reaches the private mongod; `sudo -u www-data` works too.
|
||
|
||
## Federation invariants
|
||
|
||
1. **Every outbound request goes through `IFederationHttp`.** Its handler resolves the name itself and connects only
|
||
to public addresses; redirects are followed by hand (three at most, each re-checked); bodies are capped at 1 MB;
|
||
only JSON media types are read; a refused URL is not asked again for five minutes. Never create another
|
||
`HttpClient` for federation. The only other outbound traffic is SMTP and `GeoUpdater`'s monthly DB-IP Lite download
|
||
(its own `geo` client, a fixed HTTPS host, size-capped, the file checked before it is swapped in).
|
||
2. **Every fetch is signed by the instance actor** (`privapub`), never by a persona; deliveries are signed by the acting
|
||
avatar or group. Both are draft-cavage rsa-sha256 over `(request-target) host date` (+ `digest` on bodies). Inbound,
|
||
an RFC 9421 signature (`Signature-Input`) is verified too, its target against our public address. A persona's own
|
||
activity going to a relay also carries a FEP-8b32 proof by its Ed25519 key (`Avatar.SigningKey`, one per persona,
|
||
never shared), and nothing else does: Mitra refuses a proof by a key it has not read. A forwarded activity is taken
|
||
as it came only when its proof verifies with a Multikey of its actor's.
|
||
3. **A remote document is believed only from its own address.** `RemoteActorService.FetchObject` requires the
|
||
document's `id` to be the URL it was served from (a same-origin alias is followed once). A key is accepted only if
|
||
the actor lists it, its `owner` is the actor and it shares the actor's origin.
|
||
4. **Inbound inboxes verify everything before acting:**
|
||
- the signature covers `(request-target)`, `host`, `digest` and `date` or `(created)`;
|
||
- the Digest matches the body and the date is at most an hour old and fifteen minutes ahead;
|
||
- the signature verifies against the key owner's key, and the activity's `actor` is the key owner, or else it was
|
||
forwarded (`Forwarded`: a thread's server passing on what happens in it, as Mastodon and Friendica do). A forwarded
|
||
activity proves nothing about its author: answered 202, a Create or Update is taken as its object reads at the
|
||
actor's origin now, a Delete of a public or unlisted copy once that origin answers 404 or 410
|
||
(`RemoteActorService.IsGone`), anything else let go. Forwarded copies have their own dedupe key, so a copy that
|
||
failed never hides the author's own delivery;
|
||
- an activity is queued once per id, but an id that comes back carrying another type, actor or object is a second
|
||
activity, not a copy (Friendica's ids are `uniqid()`, which two of its processes can share), and is queued apart;
|
||
- the activity's `id`, and any object it creates, updates or deletes, is on the actor's origin; a cross-origin
|
||
object is refetched from its own origin. One attributed to another account of the actor's server (Mobilizon's
|
||
organiser and the group's event) is taken as that server has it, and deleted once it answers 404 or 410.
|
||
5. **Status codes:**
|
||
- bad or missing signature: **401**;
|
||
- malformed or forged body: **400**;
|
||
- accepted: **202**;
|
||
- over the rate limit: **429**;
|
||
- **never 500** from `/peasants` or an inbox. Peers retry or "double-knock" based on these codes.
|
||
6. **Actor documents:**
|
||
- served as `application/activity+json`;
|
||
- `publicKey` is an SPKI PEM at `{actor}#main-key`, with `owner` equal to the actor's id;
|
||
- the shared inbox is advertised in `endpoints.sharedInbox`.
|
||
7. **Remote HTML is sanitized before it is stored** (`ContentSanitizer`); `Post.ContentHtml` is what is shown,
|
||
`ContentFormat` says what `Text` holds. Remote names are plain text.
|
||
8. **Circles federate to members only.** A circle is an undiscoverable Group actor that takes follow requests (the owner
|
||
approves); its posts are addressed to the circle and its `/flock`, delivered to members' personal inboxes, never
|
||
announced. Each member's copy also names that member in `cc` (`OutboxPublisher.Naming`, owner decision
|
||
2026-10-04), because Mastodon and GoToSocial keep a post only when it names one of their accounts; Create, every
|
||
Update and the Delete all go through `OutboxPublisher.Publish` for that. A reply to a circle post stays in the circle,
|
||
and a circle post never asks a non-member for a quote. Circles never appear in search, lookups, mentions or profile
|
||
pages.
|
||
**A post that is not public is served only to a signed request from someone it was for** (`SignedFetchAuthorizer.MayRead`):
|
||
a follower or an addressed account for followers-only, an addressed account for a DM, a member for a circle, or the
|
||
instance actor of a server where one of them lives; 404 to anyone else, 410 to them once it is deleted. A circle
|
||
refetch names the requesting member, or the members on the requesting server. A DM's `context`
|
||
(`/peasants/{name}/whispers/{id}`) lists the conversation's posts for its participants. Mastodon deletes its copy
|
||
when a refetch answers 404, which is why this matters.
|
||
**Communities** are FEP-1b12 groups: `GroupDistributor` announces the whole activity (plus the object for new posts,
|
||
for Mastodon), top-level posts are `Page`s with a `name`, posting follows `Group.PostingPolicy`.
|
||
Located posts (`LocalGeo`) are the only local-only posts. The Mastodon API looks past them everywhere, homes
|
||
included (the fan-out gives them no `TimelineEntry`, not even their author's): they are read through
|
||
`/clientapi/post/nearby` and deleted through `/clientapi/post/delete`.
|
||
9. **A DM joins a conversation only by `DmGroup.ParticipantsKey`**, the exact set of its participants; a remote
|
||
`context` decides nothing. DMs are `Post`s with `Visibility = Direct` and a `ConversationId` (`DmPost` is legacy).
|
||
`DmGroup.LastPostId` pages `/api/v1/conversations`, and `ConversationState` keeps each persona's read and removed
|
||
marks (`Domain/Statuses/ConversationStates.cs`; writing in a conversation reads it).
|
||
10. **Nothing slow happens inside a request.** Deliveries and inbox processing are `Job`s (`Infrastructure/Jobs`):
|
||
leased, retried on Mastodon's curve, at most two per host, paused per host by `RemoteInstance`. The inbox answers
|
||
202 once it has verified and queued; a handler must be idempotent (unique `ObjectURI`, job `DedupeKey`). A lease
|
||
(2 minutes, stamped with its own owner) is renewed every third of its length while the handler runs, so a long job
|
||
runs once; a lease found taken cancels the handler, and `Finish` only counts for the lease it was given.
|
||
11. **Every number about posts and users goes through `Domain/Privacy/Counted`** (owner decision 2026-10-04: one
|
||
answer everywhere): a persona's `statuses_count`, its outbox `totalItems`, NodeInfo `localPosts` and the instance
|
||
`status_count` all count Mastodon's way (not deleted, not a DM, boosts included, circle and located posts too);
|
||
users exclude personas of banned or deleted roots; `replies_count` counts only public and unlisted replies; a
|
||
remote account that deletes itself takes its likes, votes, reactions, boosts and replies out of every count
|
||
(`GoneActors`). Follower and following counts are public, their members never are (`hide_collections` is always
|
||
true).
|
||
12. **Every "may anyone see this" goes through `VisibilityPolicy.IsPublic`;** a persona-specific read uses `CanSee`, and
|
||
any other read of stored posts filters by `IsShown`. All three hide deleted posts and the posts of a remote account
|
||
that deleted itself (`Post.AuthorGone`: kept, hidden everywhere, owner decision).
|
||
13. **Remote content is stored only when someone here asked for it:** a persona follows the author, is addressed or
|
||
mentioned, it replies to a local post, or it is addressed to a community the author follows; a public parent is
|
||
fetched as context. Followers-only is detected by the author's stored `followers` URL.
|
||
14. **Home timelines are written, not computed:** every stored or created post goes through `Fanout.Distribute`, and
|
||
every delete removes its `TimelineEntry` rows. Local deletes are soft (content cleared, 410 Tombstone).
|
||
15. **What PrivaPub passes on is exactly what it received** (owner decision 2026-10-05). A public or unlisted reply from
|
||
another server to a persona's public, unlisted or followers-only post, and its Update and Delete, go to the
|
||
persona's followers as the activity arrived (`Arrival.Raw`, `Federation/Inbox/ReplyRelay.cs`), signed by the
|
||
persona for the transport only: never to the replier's own server, never for a local-only or group post. Only an
|
||
activity its own actor delivered carries `Raw`, so a forwarded one is never passed on again.
|
||
16. **An `Accept` or `Reject` is routed by what it answers:** our Follow (`FollowService`, any `-again-N` resend), an
|
||
interaction request (`InteractionApprovals`, with the author's authorization read back from its origin) or a
|
||
persona's `Join` (`Participations`, by its `/grunts/join-<id>` id), each only from the origin of what it answers.
|
||
17. **A server's software is for display, with two exceptions** (owner decisions 2026-10-05 and 2026-10-06): a direct
|
||
message to one account on a server whose NodeInfo names Lemmy before 1.0 or Mbin goes as a `ChatMessage`
|
||
(`StatusService.TakesOnlyChatMessages`), as one to an account that writes to us that way does; and a report of a
|
||
post in a community on a server whose NodeInfo is in `ReportService.ServiceReportTakers` leaves in Lemmy's shape,
|
||
from the reporter, one `Flag` per post. A server joins that set only once the pasture shows it keeps such a report
|
||
with its reason. Nothing else may branch on `RemoteInstance.Software`.
|
||
|
||
## Mastodon client API invariants
|
||
|
||
1. **A token is one persona.** Its subject is the avatar id; the root id lives only in the fifteen-minute `/oauth`
|
||
cookie used while choosing the persona, and never in a token, an authorization or a response.
|
||
- **The first-party client exchanges the root JWT for one persona's token** (RFC 8693 on `/oauth/token`,
|
||
`subject_token_type` `urn:ietf:params:oauth:token-type:jwt`, `avatar_id`; `Api/Mastodon/Auth/PersonaExchange.cs`).
|
||
Only the seeded public application (`ClientApi:FirstPartyClientId`, default `decepub`) holds the grant.
|
||
`RootJwtSubjectToken` validates the JWT through `RootJwt`, the same rules as JwtBearer (signature, lifetime, ban,
|
||
deletion, session stamp). The issued token is the same kind as the authorization code flow's, so revocation and
|
||
`RootSessions` cover it.
|
||
2. `/api/*` authenticates with OpenIddict validation, everything else with the old JWT (`PrivaPub` policy scheme).
|
||
Every `/api` request re-checks that the persona's root is neither banned nor deleted (`MastodonController`).
|
||
3. Read parameters through `Params` (query, form and JSON merged Rails-style), never MVC binding. A value type read
|
||
from a conditional must say `(int?)null`, not `default`: that bug once made every list one item long.
|
||
4. Answer with `Json(...)` (snake_case, explicit nulls) or `Error(status, message)`; page lists with `Page` and `Link`.
|
||
5. Unsupported features answer empty lists or 422 with a message, never 404 or 500, so clients degrade. Any `/api`
|
||
error that leaves without a body (a challenge, a 404 from routing, a 429) gets Mastodon's `{"error": ...}` from
|
||
`UseMastodonErrorBodies`. `NeverFiveHundredTests` walks every route with junk ids, anonymously, as a persona and
|
||
with a junk token.
|
||
6. Advertise `4.2.0 (compatible; PrivaPub)` until streaming and Web Push exist too; `api_versions.mastodon = 7` stays,
|
||
because clients show quotes from it, so everything a 4.3+ client calls because of it must answer: grouped
|
||
notifications (`/api/v2/notifications`, its unread count, groups, accounts and dismiss) and the notification policy.
|
||
The policy is real (`Domain/Social/NotificationPolicies.cs`, applied in `Notifications.Add`): accepted by default,
|
||
it filters (into a `NotificationRequest` per account, out of every list and count unless `include_filtered`) or drops
|
||
what strangers, accounts that do not follow the persona (or only for three days), accounts newer than 30 days,
|
||
unasked private mentions and silenced accounts send; accepting a request lets that account in for good
|
||
(`NotificationPermission`).
|
||
7. **What a Mastodon `Status` cannot say goes in `Status.privapub`** (`PrivaPubStatus`): object type, title, excerpt,
|
||
cover, the author's source, link, video, audio and event details, a remote post's own place (Pixelfed's `location`,
|
||
never federated again), up/down votes and the viewer's own downvote (`votes.downvoted`, set through PrivaPub's
|
||
`POST /api/v1/statuses/:id/downvote` and `/undownvote`; a favourite is the upvote), and its community's flairs
|
||
(`ObjectShapes.Flairs`, both dialects). Every media URL in it goes
|
||
through the proxy; only page links (`link.url`, an event's online link) point at the remote site, because following
|
||
one is the reader's choice.
|
||
|
||
## Media invariants
|
||
|
||
1. **No upload keeps its metadata.** Images are re-encoded by libvips with `keep=none`; audio and video are remuxed with
|
||
`-map_metadata -1`. `MediaProcessingTests` checks EXIF and XMP are gone.
|
||
2. **An image is checked before it is decoded.**
|
||
- **Loaders:** only libvips' JPEG, PNG, GIF, WebP and HEIF loaders ever run on an upload (`MediaService`'s static
|
||
constructor blocks every other loader), so an SVG, PDF or TIFF claiming another type never loads.
|
||
- **Size:** the header alone tells width × height (× frames for a GIF), refused above `Media:MaxPixels` (40 MP) or
|
||
`Media:MaxFrames`.
|
||
- **Then:** the image is shrunk on load to `MaxImageSide`, turned by its orientation, and its colours brought into
|
||
sRGB (`thumbnail`).
|
||
- **GIFs:** an animated GIF becomes a looping silent H.264 mp4 typed `gifv`, as on Mastodon (ffmpeg with libx264;
|
||
`PostMedia.Kind` says so), and a still GIF is an image.
|
||
- **Types:** HEIC and HEIF are not accepted, since the bundled libvips has no HEVC decoder.
|
||
- **Load:** processing runs `Media:Concurrency` at a time, and uploads have their own rate limit (`uploads`, per
|
||
credential).
|
||
3. **Audio and video are processed off the request when the client allows it.**
|
||
- **v2:** `POST /api/v2/media` stores them as sent in `media-incoming` (beside the root, never served) and answers
|
||
202 with no url, while a `ProcessMedia` job (one at a time, its lease renewed) does the work. Until then
|
||
`GET /api/v1/media/:id` answers 206, and 422 with the reason if it failed. Media still processing can't be posted.
|
||
- **v1:** processes before answering.
|
||
- **ffmpeg** reads only the stored file: `-protocol_whitelist file`, and the input format is forced from the probe.
|
||
Data and subtitle tracks are dropped (`-dn -sn`, which iPhone MOVs need).
|
||
- **Remux or transcode:** a video browsers play as it is (H.264, VP8, VP9 or AV1, within `Media:MaxVideoPixels` and
|
||
`MaxFrameRate`) is remuxed; anything else is transcoded to H.264 that fits, as Mastodon does.
|
||
- **Limits:** longer than `Media:MaxSeconds` is refused.
|
||
- **Saving:** nothing is saved until everything succeeded; outputs move into place rather than being read into
|
||
memory, and every temporary file goes. FLAC is served as `audio/flac`, and the unit runs with `PrivateTmp`.
|
||
4. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
|
||
sibling `media-proxy` and the trash the sibling `media-trash`, neither of which `/media/files` serves.
|
||
5. **A file lives exactly as long as something holds it.** Every upload is a `MediaAttachment` row, profile pictures
|
||
too (`Kind` avatar or header, `ProfileOfAvatarId`). Deleting a post, an edit leaving media out, a replaced picture,
|
||
a dropped scheduled post, a removed root, and an upload never posted for a day each trash theirs
|
||
(`IMediaService.Trash`):
|
||
- the row gets `TrashedAt` in one conditional update, so a row attached meanwhile is left alone;
|
||
- its files move into `media-trash` at once, so `/media/files` stops serving them;
|
||
- `MediaJanitor` deletes them a day later (`TrashGrace`).
|
||
|
||
What a login's media take (every persona's, trashed ones no longer) is `Counted.MediaOfRoot`, shown to the login at
|
||
`/clientapi/user/storage`; `Media:QuotaBytesPerRoot` (0, none, by default) refuses an upload over it with 422; the
|
||
administrator's statistics overview has the totals, the proxy cache and the trash.
|
||
|
||
Nothing is deleted because it looks unused. `PrivaPub admin media audit [--fix]` compares disk and database: with
|
||
`--fix` (as www-data) it gives pictures shown from before their rows a row, and trashes media of deleted posts or
|
||
personas, rows whose files are missing, and files nothing holds.
|
||
6. **A client never contacts a remote server for media:** every remote URL the API returns goes through
|
||
`IMediaProxy.Wrap`, an HMAC-signed `/media/proxy/` URL fetched by `IFederationHttp.GetMedia`.
|
||
7. **The proxy serves three ways:**
|
||
- **Cached:** a file already cached is served from disk, ranges included. It is opened before the answer, so a trim
|
||
that deletes it meanwhile breaks nothing.
|
||
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached:
|
||
- once for everyone asking for it at the same time;
|
||
- streamed into a `.part` file beside its place and renamed there, never held in memory;
|
||
- with its host in its `.type` sidecar, so that blocking a server can purge it.
|
||
- **Streamed:** a ranged request, or anything too big to cache, is streamed from the origin with the range passed on,
|
||
and never cached. That is how remote video plays. A file found too big is remembered for an hour, so it isn't
|
||
fetched twice; one that failed is remembered for five minutes.
|
||
|
||
Also:
|
||
- The cache's size is counted as it grows and trimmed as soon as it passes `ProxyCacheBytes`; the janitor
|
||
recounts hourly.
|
||
- Browsers may cache only a success.
|
||
- Nothing of a suspended server, or of one whose media are rejected, is proxied (avatars, emoji, covers and link
|
||
cards included, since they all go through it), and blocking one purges its cache.
|
||
- The `proxy` rate limit counts per client address.
|
||
|
||
nginx has a `/media/proxy/` location with `proxy_buffering off` and a 600 s read timeout for those streams.
|
||
8. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made
|
||
every status, timeline and Note holding its post fail to serialise; migration `_016` removed the ones stored before.
|
||
9. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
|
||
up to 720p that carries both sound and picture, including PeerTube's fragmented files inside an HLS entry. HLS
|
||
playlists themselves are not rewritten.
|
||
|
||
## Privacy invariants
|
||
|
||
- **No root id in federation output, NodeInfo or logs, no IP next to an identity in logs, and no `ex.Message` to a
|
||
client** ("Something went wrong." instead).
|
||
- **Sign-in and recovery never tell which logins or addresses exist** (owner decision 2026-10-04): every failed sign-in
|
||
gets "That username and password do not match." after the same hashing (`RootUsersService.Decoy`, a constant-time
|
||
comparison), and only a correct password learns of a ban. Every recovery request answers the same sentence and
|
||
queues a `SendRecovery` job, found or not; `RecoveryJob` sends the email and keeps only a SHA-256 of the code, for an
|
||
hour. A recovered password ends every session of the root (`RootSessions`: a new `SessionStamp`, which every `/clientapi`
|
||
JWT carries, and OpenIddict revocation for each persona).
|
||
- **Deleting a root deletes everything public it had** (`RootRemoval`, from the admin route or `/clientapi/user/delete`
|
||
with the password): its sessions end, each persona and each group it owns sends `Delete{Actor}` to followers, members
|
||
and the accounts it follows, the personas' posts are emptied, and `/peasants/{name}`, its inbox and WebFinger answer
|
||
410 (`LocalActorService.Gone`) while the names stay reserved. The root keeps only a `deleted-{id}` name.
|
||
- **Never derive a public name from the root username.** Invitation sign-up takes `AvatarUserName` and refuses one
|
||
equal to the login.
|
||
- **One username space:** personas, groups and the instance reserve their name in `ReservedName` (unique index) before
|
||
they are saved; `LocalActorService.TryReserveUserName` is the only way to claim one.
|
||
- **Usernames are `Constants.UserNameRegex`** (`^[a-z0-9_]+$`): the intersection of what Mastodon and Misskey accept. A
|
||
name outside it creates a persona nobody on those servers can reach.
|
||
- **An actor's `published` is `PublishedOn`**, a random whole day up to two weeks before creation, so personas made the
|
||
same day do not share a date. New persona and group ids carry that same day (`GenerateNewID`), because local clients
|
||
see ids. Never emit `CreatedAt`.
|
||
- **Per-avatar state stays per avatar:** blocks, mutes, notifications, follows. Nothing may relate sibling avatars.
|
||
- **Blocks federate** (owner decision, 2026-10-01): a block is sent as `Block` from the blocking avatar, an unblock as
|
||
`Undo{Block}`. Reports still leave as `Flag` from the server's own actors (the instance actor, or the reporter for
|
||
communities on Lemmy, PieFed and Mbin), never from the reporting avatar.
|
||
- **What PrivaPub reveals is the owner's call.** Previews, blocks, website authorship, views, bridging and reactions were
|
||
decided in `docs/ROADMAP.md` ("Owner decisions on what PrivaPub reveals"). Anything new that tells another server
|
||
something about an avatar gets the same treatment: ask, then record it there.
|
||
- **Location-ranged posts never federate.**
|
||
- **Statistics name servers, never people** (owner decisions on statistics, 2026-10-03). Every interaction is recorded
|
||
through `IInteractionLedger` (`Infrastructure/Statistics`) as an `InteractionEvent` (90 days), but an event never
|
||
holds a root, persona or group id, an activity id, an inbox URL, a remote actor URI or a sender IP:
|
||
- Distinct remote accounts are counted with `ActorHash`, an HMAC keyed by that day's `InteractionSalt`, which the
|
||
day's rollup deletes.
|
||
- `LocalKind` survives `Interactions.Sanitize` only on public and unlisted traffic, so a circle can show neither by
|
||
presence nor by absence, and no reason code names one.
|
||
- Traffic a reader causes (the media proxy, lookups, the client API, fetches of our own documents) is only counted
|
||
per day (`Count`, `CountServer`), never logged per event.
|
||
- A host claimed by an unverified sender is kept only if it is already a `RemoteInstance`.
|
||
- `Record` never blocks and never throws: it writes to a bounded channel, and a full channel drops and counts.
|
||
|
||
## Data
|
||
|
||
- MongoDB through **MongoDB.Entities 25.1**, instance API: `DB.Default.Find<T>()`, `.SaveAsync(e)`, `.Update<T>()`,
|
||
`.DeleteAsync<T>()`, `.CountAsync<T>()`. `DbEntities` wraps the finds.
|
||
- **Pass no cancellation token to `DeleteAsync` outside a transaction;** v25 throws if you do.
|
||
- **Collection name = class name, so never rename an entity class.**
|
||
- `Entity.ID` is a 24-character lowercase hex string; `GenerateNewID()` returns `object`, so cast it.
|
||
- New fields must be additive: a deploy rollback restores the binary, not the database.
|
||
- **Every stored remote object has an `ObjectRecord`** (raw JSON up to 256 KB plus its hash; delivered or fetched; the
|
||
activity, inbox, key and signed headers; received time; up to 10 later revisions). Write it right after the post's
|
||
save: `CreateHandler` (delivered) and `RemotePosts.StoreContext` (fetched) do, and `UpdateHandler` appends a
|
||
revision. Delivery details reach them through `Arrival.Current`, which `InboxProcessor` sets for the handler's
|
||
duration. A fetched record has no signature, key or `@context` of the activity that caused the fetch (its activity
|
||
fields name that trigger), and every record stores its `Extensions` and `ContextNamespaces` for statistics. The raw form lives outside `Post` so timelines never load it. Read through
|
||
`/api/privapub/v1/statuses/:id/provenance` and `/api/privapub/v1/instances/:host` (or `?host[]=`, up to 40; this
|
||
server answers too).
|
||
- **A server's place leaves only through `PublicGeo.Project`:** the city, coordinates (0.1°) and network it was located
|
||
to, whatever its size (there is no user threshold), and only the CDN's name for a CDN-fronted one. This server's own
|
||
place is its host's address located once a day (`SelfLocation`), or `Statistics:Geo:Self` when the owner sets it.
|
||
- **CDNs are found three ways, best first** (`Infrastructure/Geo/CdnCatalog.cs`): the ranges a CDN publishes
|
||
(`CdnUpdater` downloads them daily into `CdnRangeSet`, keeping the old set when a download or parse fails), the CDN's
|
||
fingerprint in the responses PrivaPub already gets (`EdgeHintsHandler` on the federation client, memory only), and
|
||
a network that carries nothing but a CDN. Never list an ASN shared with plain hosting (AWS, Google, Microsoft,
|
||
DataPacket). A server's `Geo` records the CDN, its domain and how it was found.
|
||
- **Servers through time** (`Domain/Statistics/ServerPlaces.cs`, from the weekly `RemoteInstanceSnapshot`s):
|
||
`/api/privapub/v1/instances/:host/history`, a CDN-fronted server's `geo.before_cdn`, and the CDNs grouped by their
|
||
own domain (`/api/privapub/v1/cdns`, `/cdns/:domain` with weekly joined/left).
|
||
- **Remote objects are parsed for every shape in `ObjectShapes`:** `url`/`icon`/`image` as a value, an object or an
|
||
array; Markdown `content`; missing `mediaType`s inferred; thumbnails from any of their five places; and the typed
|
||
`Link`, `Video`, `Audio` and `Event` details. A Mastodon API card is built from those, never by fetching the linked page
|
||
(that fetch is the owner's decision 1, for P6).
|
||
- **An `Update` is an edit only when its `updated` is newer than ours**, or for a first edit no older and with the text
|
||
actually changed, since GoToSocial's whole-second timestamps make a quick edit carry `updated == published` (`RemoteEdits.IsEdit`; plain and community-wrapped
|
||
Updates both go through `RemoteEdits.Apply`, and deletes through `RemoteDeletes.Remove`). Otherwise it refreshes
|
||
the poll, video, audio and event details and nothing else, and leaves no revision: Mastodon and Misskey refresh poll
|
||
counts with bare Updates.
|
||
- **A poll vote is a `Note` with a `name`, an `inReplyTo` that is a poll we hold, and no content.** `CreateHandler`
|
||
hands it to `PollService.Receive` before anything else, so it never becomes a reply. Our votes on other servers' polls
|
||
go only to the poll's author, without `published` (PieFed counts a vote only then). A `closed` in the future is the
|
||
poll's end, not its closing: Akkoma sends an open poll's end only there (`ObjectShapes`).
|
||
- **Link previews follow owner decision 1** (`Domain/Content/LinkPreviews.cs`): only public posts, queued on arrival with
|
||
0–60 s of jitter, one cached `LinkPreview` per address for the whole server, never fetched when someone reads.
|
||
`LinkPreviews.Wanted` is called where posts are saved (`CreateHandler`, `StoreContext`, `StatusService.Publish`).
|
||
- **Quote states come from `QuoteService.Resolve`:**
|
||
- with a stamp: accepted only if `Verified` (fetched, on the quoted author's origin, naming both posts exactly);
|
||
- a FEP-044f `quote` without a stamp: pending;
|
||
- older keys only: accepted when the quoted post is public.
|
||
|
||
`QuotesCount` moves with the accepted state, never with the raw key. When a persona quotes, `QuotePermission` decides:
|
||
asking first for posts that state a policy, quoting at once for posts that state none, refusing otherwise. Only `quote`
|
||
of a post that asks for consent puts `quote` in our JSON; older-key quotes leave it out, as Sharkey learned they must.
|
||
- **Our quote policy is `ActivityPubRenderer.QuotableBy`:** the post's `LocalQuotePolicy`, falling back to the persona's
|
||
`Settings.QuotePolicy` (default `public`, owner decision), and always `nobody` for anything but public and unlisted. The
|
||
same rule writes `canQuote`, answers `QuoteRequest`s (`QuoteService.ReceiveRequest`) and fills `quote_approval`, so they
|
||
cannot disagree. A persona quoting another persona gets a parrot-licence too, so other servers see an approved quote.
|
||
- **A server is described on arrival, never on read.** The first record from a host enqueues `DescribeInstance` (its
|
||
NodeInfo, at most once a week, into `RemoteInstance`), so opening the details view tells nobody anything. A
|
||
description that failed frees its week, so the server's next arrival (an hour on at the soonest) asks again.
|
||
- **Post ids are the timeline order, so they follow arrival, not `published`.** `PrivacyIds.Arrived` gives a remote post
|
||
published within the last hour (or in the future) a fresh `ObjectId`, which sorts after every post already stored,
|
||
and only backfill keeps a `published`-derived id. With `published` ids a reply arriving in the same second could sort
|
||
under the post it answers, and a late arrival landed behind a client's `since_id` and was never seen. `created_at`
|
||
comes from `CreationDate`, never from the id.
|
||
- **The same ordering problem exists on the other side, so `published` carries milliseconds**
|
||
(`ActivityPubRenderer.Timestamp`). GoToSocial (ULIDs) and Mastodon (Snowflakes) derive a remote status's id from
|
||
`published` at millisecond resolution. With whole seconds, two of our posts from the same second sorted at random
|
||
there.
|
||
- **Startup order:** `EntityMaps.Warm()` (every entity's class map, one at a time; two mapped at once throw "An item
|
||
with the same key has already been added" and stay broken), then `MigrateAsync` (`Infrastructure/Data/Migrations`,
|
||
`_NNN_` order, each runs once), then `Indexes.Create()`. A new entity needs nothing; a new unique index needs a
|
||
dedupe migration before it.
|
||
- Production runs its **own mongod** on 127.0.0.1:27022 (unit `privapub-mongod`, no auth, data in
|
||
`/var/lib/privapub/mongo`). The box's shared mongod needs credentials nobody here has. It is a one-member replica set
|
||
(`--replSet rs0`, a 990 MB oplog; owner decision 2026-10-07), so a backup reads every collection at one instant;
|
||
`setup.sh` converts it once (PrivaPub stopped, mongod restarted, `rs.initiate`), and every connection string says
|
||
`directConnection=true`, which also works against a standalone. The pasture's mongo is one too.
|
||
- **Backups** (`Infrastructure/Backup/`; owner decisions 2026-10-07: the whole server, plain files protected by their
|
||
permissions, run from the CLI, nightly and from the administrator's page). Each backup is a directory
|
||
`<yyyyMMdd-HHmmss>-<kind>` under `Backups:Root` (`/var/lib/privapub/backups`, www-data 2770, files 0640), written as
|
||
`.partial` and renamed once whole:
|
||
- `manifest.json` (`ArchiveManifest`): host, build, newest migration, whether it was read at one instant, each
|
||
collection's count, size, sha256 and indexes, what was left out and why, and the media list;
|
||
- `db/<collection>.jsonl.gz`: each document as one line of canonical Extended JSON, read raw, so every BSON type comes
|
||
back byte for byte; on a replica set every collection is read in one snapshot session
|
||
(`minSnapshotHistoryWindowInSeconds` is raised to an hour only while it reads);
|
||
- `media/`: hard links to the files of untrashed `MediaAttachment` rows (no disk spent; a deleted file stays until the
|
||
backup rotates out), copies where a link can't be made. `--db-only` lists them instead.
|
||
|
||
**Never in a backup** (`ServerBackup.Excluded`): `InteractionSalt` (owner rule), `Job` and `Delivery` (work in flight),
|
||
`EmailRecovery`, `openiddict.tokens` and `.authorizations` (sessions), `MaintenanceLock`, and `AppConfiguration`
|
||
(its SMTP password; it is made again from appsettings at boot). One backup or restore runs at a time
|
||
(`MaintenanceLock`, a heartbeat document; a holder silent for 2 minutes is taken over), and the media janitor purges
|
||
nothing meanwhile. `BackupScheduler` backs up at `Backups:NightlyAt` (03:30 UTC), or at once when it missed the night;
|
||
rotation keeps 7 daily and 4 weekly nightly backups, 3 pre-deploy, 3 pre-restore, and manual and uploaded ones until
|
||
deleted. CLI: `PrivaPub admin backup [--kind manual|pre-deploy] [--db-only]`, `admin backups`, `admin backup verify
|
||
<id>`; these run before migrations, so the deploy's backup is of the database as the live build left it.
|
||
- **Restores** (`ServerRestore`, `ProtectiveMerge`; owner decision 2026-10-07). Asking (`PrivaPub admin restore <id>`, or
|
||
the administrator's page) checks the backup (same host, a format and newest migration this build reads, every hash)
|
||
and writes `restore.json` in the backups' root; the running service sees it within seconds (`RestoreWatcher`) and
|
||
stops, and systemd starts it again. `MaintenanceGate` (in `Program`, right after the build, before migrations, indexes
|
||
and hosted services) carries it out:
|
||
1. a pre-restore backup P, taken once (a retry reuses it, never a half-restored database);
|
||
2. every collection the backup holds dropped and imported raw with its indexes, every other one dropped, except what a
|
||
backup never holds, which stays as it is;
|
||
3. media files the live directory lacks linked back from the backup (or moved from the trash); a restore deletes no file;
|
||
4. **the protective merge from P: a restore never undoes a protective act.** Followers and follows are P's; blocks,
|
||
mutes, domain blocks, being blocked, reserved names, `DeletedObject`, reports, filters and OAuth applications are
|
||
the union, P's row winning; roots, personas, groups, posts and remote accounts deleted since are deleted again, with
|
||
what their deletion takes away; P's moderation of a persona and a root's password, e-mail, ban and policies stay;
|
||
roots, personas and groups made since become tombstones (deleted, names kept), local posts made since answer 410,
|
||
media made or trashed since go to the trash;
|
||
5. every session ends (a new `SessionStamp` for each root, persona tokens deleted), circuits close, and a
|
||
`RestoreRecord` (never in a backup) tells what happened (`admin restore --status`).
|
||
|
||
Each attempt redoes everything. Refused before anything changed, it is abandoned and recorded, and the server boots as
|
||
it was; failed midway, the process exits 1 and the next start tries again; after 3 failures it exits 75, which the
|
||
unit's `RestartPreventExitStatus` leaves down for someone to look. While `restore.json` exists, commands exit 75 (but
|
||
`admin restore --status`) and the deploy refuses to run. For `RestoreRecord.FollowersGrace` (14 days) after a restore,
|
||
FEP-8fcf rests: no `Collection-Synchronization` header goes out, and a follow only the remote remembers is adopted,
|
||
not undone.
|
||
- **The administrator's page** (`BackupController`, `/clientapi/admin/backups`; owner decision 2026-10-07, which approved
|
||
these endpoints in production): the list (with what runs, the restore waiting, the last restore's report, logins
|
||
deleted since each backup), back up now (202; the page polls), delete, and:
|
||
- **download** for the password: a ticket good for ten minutes in the link's path (`/download/{ticket}`, anonymous, so a
|
||
browser saves it to disk), the backup as one tar (`BackupTar`: its exact length known first, written from the
|
||
files, `Range` honoured);
|
||
- **upload** in pieces of at most 32 MB (`TransferStore`): each `PUT ?offset=` must start where the upload stands, else
|
||
409 with what was received, so a broken upload resumes; `finish` reads it into a backup (`kind` uploaded), refusing
|
||
anything but plain files under one backup's folder, a path leaving it, or more than the disk holds;
|
||
- **restore** for the password and the server's host typed out.
|
||
|
||
Every call checks the administrator against the database, not only the token. nginx streams downloads for an hour and
|
||
takes upload pieces unbuffered (`deploy/nginx`, applied by `setup.sh`).
|
||
- **A persona's archive** (`Domain/Portability/`, `PersonaArchiveController` at `/clientapi/persona/{avatarId}/archive`;
|
||
owner decision 2026-10-07), for the persona's own root only, never a banned one. State per persona in `PersonaArchive`
|
||
(never in a backup), files in `<backups>/.personas/<avatarId>/`, a week.
|
||
- **Export** (`ExportArchiveJob`): Mastodon's account-archive layout, so other servers' importers read it: `actor.json`
|
||
(its public key only), `outbox.json` (its own posts as Creates, addressed as delivered, but circles' and located
|
||
posts; its boosts as Announces), `media_attachments/files/` (what the attachments' urls name), avatar and header,
|
||
`likes.json`, `bookmarks.json` and Mastodon's CSV files; then `privapub/`: filters, followed hashtags, the
|
||
notification policy, pins, scheduled and located posts, `archive.json`. Nothing of its root, its siblings, its keys,
|
||
anyone's token. Downloaded through a ten-minute ticket in the link's path.
|
||
- **Import** (`ImportArchiveJob`, the parts the root picks, with progress, stoppable): uploaded in pieces like a backup,
|
||
read through `SafeArchive`, which refuses links, paths leaving it, names twice, too many or too large files, a file
|
||
compressed over 100:1, and streams the outbox an item at a time (none over 1 MB). Only the archive actor's own
|
||
Creates become posts; boosts, direct and circle posts are counted. **Imported posts are delivered to no one**, put in
|
||
no home and notify nobody (an exception next to located posts), yet show on the profile, the outbox, hashtags and
|
||
search. Back home (same actor) a post keeps its id and address, and one already here, even deleted, or tombstoned,
|
||
stays as it is; from another actor it gets an id of its date, its address here and `ImportedFromURI` (unique per
|
||
persona), so importing twice changes nothing. Replies and self-quotes inside the archive point at their copies; polls
|
||
come closed without votes; mentions stay links; HTML is sanitized; media go through `MediaService` and the quota.
|
||
The other parts go through the existing services: follows asked again (never on a blocked server), blocks, mutes,
|
||
blocked servers, lists (members only if followed), bookmarks (by signed fetch), filters, followed hashtags, the
|
||
notification policy, pins (no `Add` delivered), the profile (announced as Settings would). Located and scheduled
|
||
posts (PrivaPub's archives) and likes (each tells its author) only when asked. Followers are never imported.
|
||
|
||
## Code style
|
||
|
||
- Tabs, block-scoped namespaces, Allman braces.
|
||
- New services use `readonly _camel` fields and constructor injection. Older services use PascalCase fields; leave
|
||
them.
|
||
- Services return `WebResult` (`PrivaPub.ClientModels/WebResult.cs`): `result.Invalidate(localizer[...], status)`.
|
||
Controllers turn it into a status code.
|
||
- Localised strings go through `IStringLocalizer<GenericRes>`.
|
||
- Every `Display`/`ErrorMessage` resource key must exist in `FieldsNameResource`/`ErrorsResource`, including the
|
||
`Designer.cs`, which the CLI build doesn't regenerate. A missing key throws at validation time.
|
||
- **`cond ? value : default` with a value-type branch is the type's default, not null:** `false`, `0`, or year one
|
||
stored as an edit date on every remote post. Write `(T?)null`. It has shipped four times (`MastodonParams.Bool/Int`,
|
||
`NoteParser.Int`, `NoteParser.Time`).
|
||
- ActivityPub output is built with `System.Text.Json.Nodes` in `ActivityPubRenderer`, not typed models. Inbound
|
||
documents are read through `ActivityJson.Id`/`Value`, which handle string, object and array.
|
||
- Libraries chosen for the roadmap: HtmlSanitizer, Markdig (`DisableHtml`), NSign (RFC 9421 inbound), OpenIddict +
|
||
OpenIddict.MongoDb, NetVips, Blurhash.Core, FFMpegCore. No ImageSharp (licence key enforced), no MassTransit.
|
||
|
||
## Testing
|
||
|
||
`PrivaPub.Tests` (xUnit v3). Unit tests need nothing; tests marked `Category=Integration` need a mongod and skip
|
||
without `PRIVAPUB_TEST_MONGOD=1`. CI (`build.yml` and `deploy.yml`) runs all of them through
|
||
`tools/ci/with-test-mongod.sh`, which starts a throwaway mongod on a random localhost port and deletes it afterwards.
|
||
Like production's, it is a one-member replica set (`rs0`, reached with `directConnection=true`), so backups' snapshot
|
||
reads are what the tests exercise (`TopologyTests`).
|
||
The box's own mongods are production, so `MongoFixture` refuses port 27022, a data directory under `/var/lib/privapub`,
|
||
and in CI anything but the wrapper's mongod. With `PRIVAPUB_TEST_REQUIRE_MONGOD=1`, which the wrapper sets, a missing
|
||
mongod fails the run instead of silently skipping half the tests.
|
||
|
||
- `Support/Peer` is an in-process HTTP server answering on two origins (`127.0.0.1` and `localhost`), so origin rules
|
||
can be tested; `Support/RemoteActor` signs real deliveries with its own key.
|
||
- Inbox scenarios go through `InboxReceiver.Receive` with a signed request, not through the private handlers.
|
||
- `Support/Host/PrivaPubHost` is the whole server under test (`WebApplicationFactory<Program>`, environment `Testing`,
|
||
configured only through `UseSetting`, on the fixture's database). `PrivaPubHost.Shared()` boots it once per run;
|
||
`SecureModeHost` is the same with `Federation:SecureMode`. Background workers are removed, so a test runs the jobs it
|
||
queued with `host.Run(j => ...)` or `host.RunInbox(activityId)`. `Accounts` signs a root up, adds personas, and gets a
|
||
Mastodon token through the real `/oauth` code flow; `RemoteActor.SignedPost`/`SignedGet` sign `HttpRequestMessage`s
|
||
for the real `/peasants` routes. Each client gets its own `X-Test-Client` address, so rate limits don't collide.
|
||
- All test classes share one database and run in parallel, so a test touches only rows it made:
|
||
- random names and GUIDs;
|
||
- `Harness.Outgoing` sees only deliveries queued since that harness started, because Peer ports are reused;
|
||
- a worker gets a scoped `new JobQueue(j => ...)` so it never leases another test's jobs;
|
||
- domain blocks are set with `DomainBlocks.Load`, never written to the database.
|
||
|
||
A test that must change something database-wide (drop indexes, run a migration over every post, let deliveries to
|
||
`localhost` fail and trip its breaker) goes in `[Xunit.Collection(nameof(Exclusive))]`, which runs alone, and
|
||
cleans up after itself. Pure logic belongs in an unconditional unit test, not in a Mongo-gated class.
|
||
|
||
Beyond the tests, verify by building, running locally, and exercising:
|
||
- the client API (sign up, create an avatar, a group, a post);
|
||
- the ActivityPub endpoints with curl and `Accept: application/activity+json`.
|
||
|
||
Interop is checked against real servers, starting with the workstation's own pasture:
|
||
|
||
```bash
|
||
DOTNET=~/.dotnet/dotnet tools/pasture/run.sh up [gts mastodon ...] # podman: PrivaPub + the named peers (default gts) + Mongo, behind Caddy
|
||
tools/pasture/interop.sh [gts mastodon ...] # each peer's scenario, then what PrivaPub's statistics saw of it
|
||
tools/pasture/run.sh down # removes every pasture container and volume
|
||
```
|
||
|
||
- **A change to what PrivaPub sends** (the `@context`, an actor's properties, an activity's shape, a delivery's headers)
|
||
is run against every peer's scenario before it is committed, not only the peers it was written for. One reader
|
||
stricter than the rest fails on it alone: the 2026-10-06 `proof` term passed every peer but Smithereen, whose JSON-LD
|
||
1.0 reader refused every document carrying it.
|
||
- **Layout:** `lib/pasture.sh` (network, Caddy, Mongo, PrivaPub), `peers/<name>.sh` (`<name>_up`, plus `peers/shared.sh`
|
||
for the Postgres and Redis several peers share), `lib/interop.sh` (`ok`, `ko`, `xf` for a check expected to fail until
|
||
a later phase, `privapub_token <persona>`, `stats_check <host> <software>`), `scenarios/<name>.sh`. Each peer talks to
|
||
its own PrivaPub persona under one root, so OAuth's persona choice is exercised too. Images are pinned.
|
||
- Caddy's CA lives in the `pasture-caddy-data` volume and is copied to `tools/pasture/.ca/root.crt` (and `bundle.pem`
|
||
with the system roots) for peers that must trust it instead of skipping verification.
|
||
|
||
- **All sites on one podman network, one Caddy in front.** `privapub.test`, `gts.test`, `mastodon.test` and the other
|
||
peers are network aliases of the Caddy container, which serves them all with its internal CA (`tls internal`).
|
||
PrivaPub accepts any certificate (`appsettings.Pasture.json`) and GoToSocial is told to skip verification
|
||
(`GTS_HTTP_CLIENT_TLS_INSECURE_SKIP_VERIFY`); Mastodon trusts the copied CA through `SSL_CERT_FILE`. Peers fetch over
|
||
https only, so plain http between them is not an option.
|
||
- From the workstation, PrivaPub's API is `http://127.0.0.1:6971`. A peer is reached as `https://<name>.test:6443` with
|
||
`curl -k --resolve <name>.test:6443:127.0.0.1`, because sign-in cookies are bound to the host name.
|
||
- **GoToSocial (0.22.1):**
|
||
- Its cached home timeline can stop taking new posts after its first read, its owner's own included, while a
|
||
`min_id` query shows them all. So a delivery is checked by looking the object up by URI with `resolve=false`
|
||
(`on_gts`), which answers from GoToSocial's database and never fetches from us. A home-timeline check there proves
|
||
nothing, in either direction. A deleted status still turns up in that search as a "deleted status" stub, so a
|
||
delete is checked as a 404 on `/api/v1/statuses/{id}`.
|
||
- It creates its accounts locked, so the scenario approves alice's request through `/api/v1/follow_requests`, which
|
||
also checks our pending (`requested`) state and the manual Accept.
|
||
- 64 checks: discovery and follows both ways; posts and CW; a reply and its notification; likes and boosts both
|
||
ways; DMs both ways and off public timelines; polls both ways; quote policy; link cards; edits and deletes both
|
||
ways; communities and circles; locked personas; interaction policies (a reply and a like asked for and approved
|
||
through `/api/v1/interaction_requests`, a boost refused); unfollow; block and unblock; statistics.
|
||
- It files a circle post as a direct message and shows it only to the accounts it mentions, so like a DM it is
|
||
checked in the member's `/api/v1/conversations`, never by URI.
|
||
- **Mastodon (4.7.3):** web and sidekiq on the shared Postgres and Redis, `ALLOWED_PRIVATE_ADDRESSES` for the network.
|
||
Its token comes from `rails runner` (no password grant). Without Elasticsearch its status search finds nothing, so
|
||
deliveries are checked through `/api/v1/accounts/:id/statuses` of the sender as Mastodon knows them, or
|
||
`Status.exists?` through `rails runner`. Its actors are numbered (`/ap/users/<id>`), so look URIs up rather than
|
||
build them. Circle posts and a followers-only post survive its signed refetch (`ActivityPub::FetchRemoteStatusService`
|
||
through `rails runner`). Its block of a persona, and the unblock that ends the run, are checked through our
|
||
`blocked_by`; the run starts with that unblock too, since Mastodon rejects every follow from an account it blocks.
|
||
- **Misskey (2026.10.0):** one container on the shared Postgres and Redis. A new Misskey federates with nobody
|
||
(`federation: none`) until `admin/update-meta` says `all`, which `misskey_up` does. Its API is `POST /api/<endpoint>`
|
||
with the token as `i` (`mk` in the scenario); `users/relation` answers a list, `users/notes` leaves replies out unless
|
||
asked, and a user has one reaction per note. Never verify a delivery with `ap/show`: it fetches. 35 checks.
|
||
- **Sharkey (2025.4.7):** `peers/sharkey.sh` is the Misskey peer under another name, image, database, Redis db and
|
||
home (`/sharkey/.config`); `scenarios/sharkey.sh` runs Misskey's scenario with `MISSKEY_NAME=sharkey`, then follows
|
||
again (Misskey's ends unfollowed and blocked) and checks edits both ways and the FEP-e232 quote tag. 40 checks.
|
||
- **Akkoma (3.20.1):** no official image, so `images/akkoma` installs the OTP release (pinned by checksum; the "stable"
|
||
zip moves, and a moved one fails the build) and `akkoma_up` builds it once. Its HTTP clients read the CA bundles
|
||
shipped in the release (CAStore, certifi), so the entrypoint appends Caddy's CA there. `pleroma_ctl` passes its
|
||
arguments on unquoted: no value may contain a space. Its Linkify never takes `@user@host.test` for a mention, so the
|
||
scenario addresses alice with Pleroma's `to[]`; its API never reports a remote blocker as `blocked_by`, so the block is
|
||
read from its `user_relationships`. `akkoma_token <user>` signs one of its users in. 45 checks.
|
||
- **Lemmy (1.0.0-beta.2):** the backend alone on the shared Postgres, its admin made by `setup` in the generated
|
||
`config.hjson`. It trusts Caddy's CA through `SSL_CERT_FILE` and reaches the network through
|
||
`DANGER_FEDERATION_ALLOW_LOCAL_IP=1`; 0.19 cannot join (its rustls trusts only its bundled roots). Its API is
|
||
`/api/v4/<path>` with a bearer token (`lm` in the scenario), and `sort` values are lowercase. It logs no refused
|
||
activity at `warn` (`LEMMY_LOG` sets `RUST_LOG`); the reason is in the 400's body. It answers our community's echo of
|
||
its own activity and every bare `Announce{object}` 400 by design, and the echo is still needed (see
|
||
`docs/INTEROP.md`, Lemmy). A new Lemmy never sends what it queued for a server before its send worker for that
|
||
server started, so the scenario waits for that worker (`lm_worker`) before its first follow, and it sends what it
|
||
queued every 30 seconds, so a vote or a moderator's act takes up to a minute. Its lock, unlock and unban need a
|
||
`reason`, or it refuses them and federates nothing. 29 checks, among them a moderator's lock, ban and removal.
|
||
- **PieFed (1.7.17):** dockurr's image of the release, its web app (with PieFed's own cron, `CRON=true`, for its send
|
||
queue) and a Celery worker on the shared Postgres and Redis (dbs 10 and 11), sharing the `pasture-piefed-media`
|
||
volume. httpx trusts only certifi's bundle, so the pasture's is mounted over it. `flask init-db` reads its admin
|
||
(pfuser) from stdin and drops every table it finds, so it runs once, after the web app's migrations ("Starting
|
||
Gunicorn"). Its API is Lemmy's v3 under `/api/alpha` with a JWT (`pf` in the scenario); `resolve_object` answers a
|
||
view (`community.community.id`). It sends a community's announces to the Application at a peer's root, assuming
|
||
`/inbox` when there is none, and keeps serving a thread its moderator removed. `scenarios/piefed.sh`, 29 checks:
|
||
communities both ways, threads with titles, comments, votes up and down both ways, a community poll and alice's vote,
|
||
private messages both ways, a moderator's lock, unlock and removal, the unfollow, statistics.
|
||
- **Mbin (1.10.1):** its own image (FrankenPHP serving plain HTTP behind Caddy, `SERVER_NAME=:80`) and a messenger
|
||
worker (`mbin_worker`, consuming again whenever it stops: once, the shared Postgres restarted under it and Mbin sent
|
||
nothing for a day), on the shared Postgres and Redis (db 12) with a RabbitMQ of its own (its transports carry AMQP options; it
|
||
runs as its own user on its own volume, or it cannot read the `.erlang.cookie` it wrote as root). The pasture's bundle
|
||
is mounted over the system one; its API's rate limits (two threads every six minutes) are raised by a copy of its
|
||
`rate_limiter.yaml`. Its admin mbuser is made by its console; `peers/mbin_token.py` gets mbuser's OAuth token through
|
||
the authorization-code flow (login form, consent), since a client-credentials client acts as a bot that may not vote.
|
||
Mbin names what it makes during a request after the request's host, so every call says `Host: mbin.test`, never
|
||
the workstation's port. Its access token lasts an hour, and the scenario gets a new one when it has expired. Its API
|
||
never starts a conversation with an account elsewhere, so mbuser answers in the thread alice began; Mbin addresses
|
||
that answer to alice's profile page. `scenarios/mbin.sh`, 26 checks: magazines both ways, threads with titles, a Note
|
||
to a magazine as a microblog post, comments both ways, favourites and upvotes both ways, private messages both ways,
|
||
a moderator's lock, unlock and removal, the unfollow, statistics.
|
||
- **NodeBB (4.16.1):** the official image on the pasture's Mongo (database `nodebb`), set up once by its automated
|
||
setup (`SETUP` with `NODEBB_*` variables, admin nbuser) into the `pasture-nodebb-config` volume, trusting the CA
|
||
through `NODE_EXTRA_CA_CERTS`; its image runs `npm install` at every start. Its API (`/api/v3`) takes a bearer token
|
||
that `nodebb_settle` writes where NodeBB keeps its tokens. A remote account's uid there is its actor's URL, but its
|
||
API follows one only when named by its handle; a chat room is made, then written in. `scenarios/nodebb.sh`, 23
|
||
checks: a category followed and its topic as a titled thread, replies both ways, nbuser following alice, votes both
|
||
ways, an edit and a deletion, a chat both ways, the unfollow, statistics.
|
||
- **Lemmy 0.19 (0.19.20), as `lemmy19.test`:** what most of the threadiverse runs. Its release trusts only the roots its
|
||
rustls bundles, so `images/lemmy19` builds the tag with reqwest's `rustls-tls-native-roots` added (about ten minutes
|
||
the first time) and the pasture's bundle is mounted as the system's. Its config names the database as `uri` (1.0:
|
||
`connection`) and its API is v3 (`/api/v3`, `sort=New`, `resolve_object` answering views). It takes private messages
|
||
only as `ChatMessage`, which PrivaPub sends it, a first message too (invariant 17). `scenarios/lemmy19.sh`, 30
|
||
checks.
|
||
- **Threads (`scenarios/threads.sh`, needs mastodon):** mastouser follows alice, carol answers alice's public post,
|
||
and opening the thread on Mastodon finds carol's reply through alice's `replies` (the scenario makes Mastodon's copy
|
||
look ten minutes old: Mastodon reads a thread's replies only five minutes after it first holds the post). 5 checks.
|
||
- **Followers synchronisation (`scenarios/followsync.sh`, needs mastodon):** sent: PrivaPub forgets mastouser's follow
|
||
of alice (mongo), alice's followers-only post naming mastouser makes Mastodon read her roll-call and drop it; then
|
||
Mastodon forgets a follow (rails) and alice's next followers-only post makes it send the `Undo` PrivaPub applies.
|
||
Received: alice and bob follow mastouser; Mastodon forgets alice's follow and mastouser's followers-only post makes
|
||
PrivaPub end it, then PrivaPub forgets it and the next makes PrivaPub undo it there. Follows are checked in Mastodon's
|
||
database: its API caches relationships for a day. 15 checks.
|
||
- **Pins (`scenarios/pins.sh`, needs mastodon):** a Mastodon account pins and unpins while alice follows it, alice pins
|
||
and unpins while it follows her, a fresh account's earlier pin shows once PrivaPub resolves it, and following it brings
|
||
its earlier posts (its outbox). 9 checks.
|
||
- **Vernissage (1.43.0):** its image on SQLite in the `pasture-vernissage` volume, its queues on the shared Redis's
|
||
database 0 (it federates nothing without them; every other database is taken), its seeded admin (admin / admin)
|
||
logged in by the scenario. `scenarios/vernissage.sh`, 19 checks.
|
||
- **BookWyrm (0.9.3):** its image, migrated, on the shared Postgres (database `bookwyrm`) and Redis (databases 14
|
||
and 15), gunicorn and one Celery worker (federation runs there). It has no client API: bwuser (named
|
||
`bwuser@bookwyrm.test`, as its signup names users), a book and every status come from its Django shell
|
||
(`bookwyrm_shell`, `-v 0`). `scenarios/bookwyrm.sh`, 20 checks.
|
||
- **Ghost (6.67) with its ActivityPub service (1.2.14):** Ghost and the service on the shared MySQL (databases `ghost`
|
||
and `activitypub`, the service's migrations run first), routed by Caddy as Ghost's own proxy does. The owner comes
|
||
from Ghost's setup API (a password that repeats the site's name is refused), staff device verification is off, and
|
||
Ghost starts again once so it sets up its publishing webhooks. Calls to the service say `Host: ghost.test`, or it
|
||
cannot find the site's keys. `scenarios/ghost.sh`, 22 checks.
|
||
- **Owncast (0.3.0):** its image, its data in the `pasture-owncast` volume, federation turned on through its admin
|
||
API (`owncast_admin`, its default admin / abc123) with the server's address and the actor's name.
|
||
`scenarios/owncast.sh`, 13 checks.
|
||
- **WriteFreely (0.17.2):** built from its release tarball with `openssl` beside it (`images/writefreely`, it makes
|
||
each blog's keys with the command), on SQLite in the `pasture-writefreely` volume, which the scenario reads from the
|
||
host. wfuser and its blog come from its CLI; its API takes a login. `scenarios/writefreely.sh`, 13 checks.
|
||
- **snac2 (2.95):** built from its tag (`images/snac2`, the project publishes no image), its data in the
|
||
`pasture-snac` volume from `snac init` and `snac adduser`; the password comes from `snac resetpwd` and the token from
|
||
its login form and a code grant. Its API lags behind what it has taken in, so `scenarios/snac.sh` reads its files
|
||
(`snac_stored`, `snac_in_timeline`). 27 checks.
|
||
- **Mitra (5.9.1):** its image on the shared Postgres (database `mitra`), as `mitra.test`, with the pasture's bundle as
|
||
the system's roots; mitrauser comes from its CLI and its token from a password grant. Its search resolves an account
|
||
elsewhere only without `type`. `scenarios/mitra.sh`, 28 checks.
|
||
- **Relays (`peers/relay.sh` Activity-Relay 2.0.9 as `relay.test`, on the shared Redis's database 13;
|
||
`peers/aoderelay.sh` aode-relay 0.3.129 as `aoderelay.test`):** `appsettings.Pasture.json` names both in
|
||
`Federation:Relays`, so PrivaPub subscribes a minute after it starts. `scenarios/relay.sh` has Mastodon subscribe to
|
||
each in turn, checks that a post of an account nobody here follows reaches the federated timeline (forwarded by one,
|
||
announced by the other), that alice's public post goes to the relays (and through both to Mastodon: Activity-Relay's
|
||
forward on its FEP-8b32 proof, after the scenario makes Mastodon read alice's keys anew) and nothing less public, and
|
||
has Mastodon leave again, so the town sees no relayed posts. 16 checks.
|
||
- **Castopod (1.15.5, `peers/castopod.sh`):** the official image on the shared MySQL (database `castopod`, file cache,
|
||
`CP_DISABLE_HTTPS`), its database and superadmin (admin@castopod.test) from `spark`, its REST API switched on in the
|
||
`.env` the image rewrites at each start, `spark fediverse:broadcast` looping in the container. The podcast `@pod` is
|
||
made and published through its admin pages (`cp_web`, CSRF on each form). `scenarios/castopod.sh`, 15 checks; the
|
||
episode's sound is made with ffmpeg on the workstation.
|
||
- **Ktistec (3.13.0, `peers/ktistec.sh`):** built from its tag by `images/ktistec` (its own Dockerfile, pinned: a static
|
||
Crystal server, its assets with node), SQLite in the `pasture-ktistec` volume, the pasture bundle mounted as
|
||
`/etc/ssl/cert.pem`. The server's name and ktuser come from its setup API (every field required, `summary` too);
|
||
`ktistec_token` signs in on `/sessions`. `scenarios/ktistec.sh`, 27 checks: its Mastodon API where it has the route,
|
||
its outbox API (`kn`) for follows, polls (`poll-duration` is seconds from now), edits, deletions and quotes, and a
|
||
copy of its SQLite database (`kt_sql`) for what it holds.
|
||
- **Forte (26.9.10, `peers/forte.sh`):** built from its tag by `images/forte` (composer on Hubzilla's PHP image, an nginx
|
||
routing through `index.php?req=`), on the shared MySQL (database `forte`), its `.htconfig.php` written to
|
||
`.state/forte`, a cron sidecar (`src/Daemon/Run.php Cron`). The pasture bundle is mounted over `library/cacert.pem`,
|
||
the bundle its curl uses. ftuser (administrator) and ftfriend are made through its PHP (`forte_eval`: `Account::create`,
|
||
`Channel::create` with the role `social` and `autoperms`, or every connection waits). Connecting from the command line
|
||
runs the notifier itself (`ft_connect`). `scenarios/forte.sh`.
|
||
- **Hubzilla (11.4.1, `peers/hubzilla.sh`):** the hlhd image (php-fpm and nginx on 8080) on the shared MySQL
|
||
(database `hubzilla`, its schema loaded from the image), `.htconfig.php` written to `.state/hubzilla` and mounted, a
|
||
cron sidecar running `Zotlabs/Daemon/Master.php Cron` each minute. Its addons `pubcrawl` (ActivityPub) and `statistics`
|
||
(NodeInfo) are installed with `util/addons`; hzuser (the hub's administrator) and hzfriend are made through its own PHP
|
||
(`hz_eval`), as accounts written the way its registration queue would and channels with the role `public` (the older
|
||
role names grant connections nothing), the ActivityPub app installed as each channel. Its API (`api/z/1.0`) posts and
|
||
edits with HTTP Basic (email and password); liking and dropping go through its web pages signed in (`hz_web`).
|
||
`scenarios/hubzilla.sh`, 20 checks and one known gap (G-0010, its unfollow).
|
||
- **Smithereen (1.0.3):** its image on the shared MySQL (database `smithereen`, its schema from the image's commit),
|
||
with imgproxy and a file server behind Caddy as `smithereen.test` (`/i` and `/s`), trusting the CA through a JDK
|
||
store with it added (`JAVA_TOOL_OPTIONS`). MySQL takes its stored functions only with
|
||
`log_bin_trust_function_creators`, and a first start that failed there leaves the database without triggers (drop it
|
||
and start again). Accounts come from its signup form (opened, with a short description, which its NodeInfo needs) and
|
||
are renamed in its database; its API is VKontakte's (`/api/method/<name>?v=1.0`), with a password grant for a local
|
||
application the peer script inserts. `scenarios/smithereen.sh`, 40 checks, walls both ways (alice's profile is
|
||
told again first, so Smithereen reads her `sm:wall`; `servers.features & 1` says it took PrivaPub for a server with
|
||
walls). Its scenario texts avoid apostrophes: `p_home_has` puts them inside a Python string.
|
||
- **Load (`load.sh`, needs the `flood` peer):** `flood` (`flood/flood.cs`, published once into `.flood`) answers as
|
||
twenty fake servers and sends signed activities at a set rate; `load.sh --rate=N --seconds=N` measures the answers,
|
||
the queue's wait and processing times, its drain, and a persona's home timeline meanwhile, and keeps each run in
|
||
`out/load/`. `docs/LOAD.md` has the method and the runs.
|
||
- **Account moves (`scenarios/moves.sh`, needs gts):** two fresh GoToSocial accounts made by its admin CLI; alice follows
|
||
the old one, the new one names it as an alias (`/api/v1/accounts/alias`), the old one moves (`/api/v1/accounts/move`),
|
||
and PrivaPub shows it `moved` and moves alice's follow to the new one, which (locked by GoToSocial) approves it.
|
||
8 checks.
|
||
- **Hollo (0.9.19):** Fedify's microblog server on the shared Postgres, set up through its web form (which
|
||
checks `Origin` against `Host`, so the request names `hollo.test` without the port). It needs a 44-character
|
||
`SECRET_KEY`, a media directory and a `themeColor`; statuses and votes go as JSON. Town only, no scenario.
|
||
- **PeerTube (8.3.1):** the official image on the shared Postgres and Redis (db 4), configured through `PEERTUBE_*`
|
||
variables, trusting Caddy's CA through `NODE_EXTRA_CA_CERTS`; `peertube_settle` turns transcoding off (a test video
|
||
is served as uploaded) and keeps root's token in `.state/peertube.token`. It wants a bare `Host: peertube.test`.
|
||
Its list of comments held for review (`/api/v1/users/me/videos/comments?isHeldForReview=true`) answers 500 in 8.3.1;
|
||
the video's owner sees them in the video's threads (`heldForReview`). `scenarios/peertube.sh`, 28 checks.
|
||
- **Pleroma (2.10.2):** `images/pleroma` installs the OTP release, pinned by checksum, as Akkoma's does; it also needs
|
||
libvips, and `instance gen` asks about deduplicating uploads. Its federation runs on hackney, which trusts only
|
||
certifi's compiled-in roots, so the entrypoint points `:pleroma, :http, adapter` at the system CA bundle. It answers
|
||
an inbox POST at once and verifies in a worker (`oban_jobs` shows what it refused and why). Town only, no scenario.
|
||
- **Iceshrimp.NET (2026.1.2-beta):** on the shared Postgres with AuthorizedFetch on and open registrations, trusting
|
||
Caddy's CA through `SSL_CERT_FILE`. Accounts come from its own `/api/iceshrimp/auth/register` (its login cuts the
|
||
connection short for a name it does not know), Mastodon API tokens from its OAuth form, which prints the out-of-band
|
||
code in the page. Its `jobs` table shows what it queued for whom. Town only, no scenario.
|
||
- **Pixelfed (0.14.4):** serversideup's FrankenPHP image on the shared Postgres (database pixelfed) and Redis (dbs 5 and
|
||
6), with Horizon and the scheduler as sidecars of the same image sharing its storage volume, and Caddy's CA mounted
|
||
as its system bundle. `pixelfed_settle` makes `caption` nullable (its PostgreSQL migration never runs, so every remote
|
||
boost failed), creates Passport's keys and restarts the web server so FrankenPHP reads them, numbers the OAuth
|
||
clients from a million (Passport refuses a token whose user id equals its client's id), and imports its cities.
|
||
Tokens are personal access tokens made through tinker (`App\Models\User`). Every top-level post needs a picture.
|
||
`scenarios/pixelfed.sh`, 26 checks; the town's driver and `specs/pixelfed-pair.json`.
|
||
- **WordPress (7.1.2, ActivityPub plugin 9.3.1):** the official image on a shared MySQL 8.4 (`shared_mysql_up`, ready
|
||
only when it answers over TCP: its first start runs a server without networking), installed and configured by wp-cli,
|
||
its CA bundle (`wp-includes/certificates/ca-bundle.crt`) given Caddy's root, and WP-Cron run every five seconds by a
|
||
sidecar (`DISABLE_WP_CRON`), since the plugin federates from it. Its automatic updates are off
|
||
(`AUTOMATIC_UPDATER_DISABLED`): the sidecar ran one (6 to 7.1.2) whose new CA bundle dropped Caddy's root, and
|
||
WordPress reached nobody after it. Authors are actors; the REST API takes application passwords.
|
||
`scenarios/wordpress.sh`, 17 checks.
|
||
- **Friendica (2026.05):** the official image on the shared MySQL (database friendica) and Redis (db 7, for its cache,
|
||
locks and sessions), installed by its autoinstall, with the image's worker daemon (`cron.sh`) as a sidecar sharing
|
||
its files. `friendica_settle` repairs what the install leaves: it names the system user (uid 0), or Friendica never
|
||
makes the system account that signs its fetches; it declares the daemon running (`worker_daemon_mode` in
|
||
`key-value`), or the web container, which cannot see the sidecar's pid, never wakes it and everything waits for its
|
||
five-minute cron; it makes the log file and turns logging on. `friendica_user` saves each account through the API
|
||
with `locked=0` (a number: "true" reads as 0), which makes it a "soapbox" page that takes followers without asking and
|
||
follows nobody back, and reads its outbox once: a Follow that reaches a user Friendica has not cached yet makes it
|
||
fetch that user from itself, signed, and checking the signature recurses for about five minutes. Its API takes HTTP
|
||
Basic (`nick:password`); an edit through it never federates, so the scenario edits in the web editor.
|
||
`scenarios/friendica.sh`, 25 checks; the town's driver and `specs/friendica-pair.json`.
|
||
- **Mobilizon (5.2.4):** kaihuri's image of the release on a PostGIS of its own (`shared_postgis_up`: it needs the
|
||
extension), with the pasture's bundle mounted over certifi's and castore's CA files (hackney trusts only those) and
|
||
the system store, and geocoding pointed at a closed local port. `mobilizon_ctl users.new` makes mzuser (it splits
|
||
its arguments on spaces); the API is GraphQL (`/api`), signed in by the `login` mutation. An event made through it
|
||
without options has its comments closed. `scenarios/mobilizon.sh`, 27 checks: a group and its events (dates, place),
|
||
alice joining an event (Mobilizon's participant row and its Accept) and leaving it, comments both ways, the
|
||
organiser's edit, closed comments and deletes, a group post, the unfollow.
|
||
- **Gancio (1.28.2):** cisti's image on sqlite in the `pasture-gancio-data` volume, whose `config.json` is written before
|
||
the first start (without it Gancio waits in its setup wizard), trusting Caddy's CA through `NODE_EXTRA_CA_CERTS`.
|
||
`gancio users create` makes gcadmin; `gancio settings set enable_resources true` keeps fediverse replies. One
|
||
Application actor, `relay`, publishes the agenda. Its API takes a token from an OAuth password grant
|
||
(`/oauth/login`, client_id `self`); an event is a multipart POST (`--form-string` for text starting with `<`).
|
||
`scenarios/gancio.sh`, 17 checks.
|
||
- **Funkwhale (2.0.11):** the API (gunicorn), a Celery worker with its beat and the front's nginx, sharing the
|
||
`pasture-funkwhale-data` volume, on the shared Postgres (database funkwhale) and Redis (dbs 8 and 9); Python's requests
|
||
trusts the pasture's bundle through `REQUESTS_CA_BUNDLE`. `funkwhale-manage fw users create` makes fwuser and an OAuth
|
||
access token is made in its Django shell. Its API is `/api/v1` and `/api/v2` (uploads); a track is a fresh tone made
|
||
by ffmpeg in its container, since it skips a file it already has. `scenarios/funkwhale.sh`, 16 checks.
|
||
- **SecureMode:** `PRIVAPUB_ENV="Federation__SecureMode=true" run.sh up …`, as production runs. A check of what an
|
||
unsigned reader sees uses `unserved` and `gone_unsigned` (`lib/interop.sh`), which expect 401 when SecureMode is on and
|
||
404 or 410 when it is off.
|
||
- **Crawler:** `PRIVAPUB_ENV="Statistics__Crawler__Enabled=true Statistics__Crawler__Seeds__0=mastodon.test"
|
||
run.sh up mastodon`, then `interop.sh crawler`. `PRIVAPUB_ENV` passes any setting to the PrivaPub container.
|
||
- `run.sh up` replaces every container, Mongo included, so each run starts clean. To keep the data, republish into
|
||
`tools/pasture/.publish` and `podman restart pasture-privapub`; that is how a migration is tried on dirty data.
|
||
- `run.sh add <peer>` starts more peers next to a running pasture; `run.sh down` removes every pasture container with
|
||
its volumes (`--purge` drops Caddy's CA too); `run.sh stats` shows each container's memory.
|
||
- **The network looks public** (`PASTURE_SUBNET`, default `11.42.0.0/24`), so peers that refuse private addresses with no
|
||
switch (Pixelfed, Mbin, WordPress, Discourse, Fedify) federate. Mastodon then needs `TRUSTED_PROXY_IP` for Caddy, or it
|
||
answers every request with an IP spoofing error.
|
||
- **`PASTURE_PORT`** moves PrivaPub's workstation port off 6971 when something else holds it; `lib/interop.sh` follows.
|
||
- Every `ok`/`ko`/`xf` is also a line of `out/scenarios.jsonl`, which the town's report shows.
|
||
|
||
#### The town (`tools/pasture/town.sh`, `tools/pasture/town/`)
|
||
|
||
A fake community seeded across every running peer, then checked for coherence: what each server holds (read from its
|
||
database, never by making it fetch), what each account sees through its API, counts, threads, edits, deletes, follows
|
||
and the privacy rows. Python 3 standard library only, tabs.
|
||
|
||
```bash
|
||
tools/pasture/town.sh selftest [peer...] # each driver against its own server; run after changing a driver or a pin
|
||
tools/pasture/town.sh seed village # specs/village.json: 23 accounts on seven servers, about 11 minutes
|
||
tools/pasture/town.sh check village # out/<run>/results.json; --deadline=0 re-sweeps an old run at once
|
||
tools/pasture/town.sh report village # out/<run>/report.html, publishable (fake data, no tokens)
|
||
tools/pasture/town.sh report village hollo-pair iceshrimp-pair pleroma-pair pixelfed-pair friendica-pair # one matrix
|
||
tools/pasture/town.sh backlog --write # docs/INTEROP-BACKLOG.md
|
||
```
|
||
|
||
- **Drivers** (`dialects/`): one class per platform on a dialect base (`mastodon_api`, `misskey_api`, `lemmy_api`,
|
||
`privapub`). `stored()` reads the database (shared Postgres through `psql`, the shared MySQL through `mysql_json`,
|
||
GoToSocial's sqlite copied out, Mongo);
|
||
`seen()` asks the API as one account. A driver that cannot do something raises `Unsupported`, and the planner never
|
||
asks for it (`gen.CAPS`).
|
||
- **The plan** (`gen.py`) is a pure function of the spec and its seed; each run keeps its own `plan.json`, `ledger.jsonl`
|
||
(what happened, one fact per line) and `results.json`. `seed --resume` redoes only failed steps.
|
||
- **Cells** are `feature|origin|observer|direction` (`in`, `out`, `via`, `control`, `local`). A known gap is a
|
||
`gaps.json` entry: matching failures are `xfail`, matching passes `xpass` (close the gap).
|
||
- **What the peers do on purpose**, so the checker expects it: Misskey and Sharkey drop a reply whose parent they cannot
|
||
fetch; Lemmy keeps only community content and private messages; servers send edits and deletes to the post's own
|
||
audience only; a Misskey heart reaction arrives as a Like.
|
||
- Peers' own limits are lifted for the town: GoToSocial `GTS_ADVANCED_RATE_LIMIT_REQUESTS=0`, Lemmy's
|
||
`local_site_rate_limit`. Mastodon reserves usernames that contain `mastodon`.
|
||
- decePubClient's e2e tests (`decePubClient/tests/e2e/run.sh`) publish the client with the Pasture environment, serve
|
||
it as `https://decepub.test` (`peers/decepub.sh`) and write their cells to `out/client.jsonl`.
|
||
|
||
After the pasture, verify.funfedi.dev and the owner's GoToSocial at social.arasaka.software. **Ask before acting from
|
||
the owner's GoToSocial account.**
|
||
|
||
## Deploy
|
||
|
||
- **CI/CD:** push to `master` runs `build.yml` (build + tests) on the instance-wide `build` runner. A `v*` tag runs
|
||
`deploy.yml`: tests, self-contained linux-x64 publish, a snapshot of the site to `/var/backups/privapub.thepra.dev`,
|
||
the server's own pre-deploy backup (`PrivaPub admin backup --kind pre-deploy --db-only`, run from the new build before
|
||
its migrations, verified, and checked to hold no statistics salt; no deploy while `restore.json` waits),
|
||
stop → rsync → start, a `127.0.0.1:6970/build.json` health loop with rollback, then public checks (actor, NodeInfo,
|
||
Swagger 404, inbox junk 400, unsigned 401) and `tools/smoke/mastodon-api.sh` (app registration, client credentials,
|
||
discovery, instance, public timeline). Then it checks that what production should be running is running:
|
||
- NodeInfo and the instance API agree on registrations (closed, invitations only);
|
||
- `/stargazing` says the crawler is on;
|
||
- **@thepra signs in**: `PrivaPub admin smoke thepra` gives the root `deploy-smoke` a new password, `tools/smoke/oauth.sh`
|
||
runs the real OAuth flow (the pasture's `privapub_token` uses the same script), the signed-in API is checked, the
|
||
persona must be undiscoverable, and the token is revoked;
|
||
- the DB-IP Lite databases, which the server fetches itself (`GeoUpdater`), exist and are at most 40 days old.
|
||
- **The box:** Max (`nuvola.xyz`). Unit `privapub` runs as www-data from `/var/www/privapub.thepra.dev` with
|
||
`ASPNETCORE_ENVIRONMENT=Production`.
|
||
- **One-time root setup:** `deploy/max/setup.sh`, run through `../arasaka.software/tools/max/run.sh` (directories, the
|
||
runner's sudoers line, the units, nginx and the certificate). Nothing that runs later needs it again.
|
||
- **Config:** `appsettings.Production.json` is committed and deployed, **secrets included, by the owner's convention**
|
||
(the same as arasaka.software). A hand edit on the box is lost at the next deploy.
|
||
- **Secrets drift:** `AppConfigurationService` copies `AppConfiguration` into Mongo on first boot and reads the stored
|
||
copy afterwards, so changing those keys needs a Mongo update as well.
|
||
- **Logs:** `journalctl -u privapub`, plus the `logs` database on the private mongod.
|