12bb75809f495f9a385ceb92929934ff003bc695
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  thepraandClaude Opus 5.5
|
12bb75809f |
The server backs itself up: every collection byte for byte, the media linked
A backup is a directory <stamp>-<kind> under Backups:Root (/var/lib/privapub/backups, 2770, files 0640), written as .partial and renamed once whole: a manifest (host, build, newest migration, each collection's count, size, sha256 and indexes, what was left out and why, the media list), each collection as gzipped canonical Extended JSON read raw, and hard links to the files of untrashed media rows (copies where a link can't be made). On a replica set every collection is read in one snapshot session. Never in a backup: the statistics salt, jobs, recovery codes, sessions, the maintenance lock and the configuration's copy with its SMTP password. One backup or restore at a time (MaintenanceLock, a heartbeat document), and the janitor purges nothing meanwhile. BackupScheduler backs up nightly at 03:30 UTC (or at once after missing a night); rotation keeps 7 daily, 4 weekly, 3 pre-deploy and 3 pre-restore backups. CLI: admin backup [--kind] [--db-only], admin backups, admin backup verify; these run before migrations, so the deploy's own pre-deploy backup, which replaces mongodump, is of the database as the live build left it. EntityMaps.Warm runs once under a lock, since test hosts now boot side by side. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
3ca29603ed |
The media proxy is bounded
Anyone could mint signed proxy URLs (a remote account changes its icon, an anonymous lookup returns the URL), and each anonymous request held up to 40 MB in memory; the cache grew without bound between hourly trims; two clients asking for the same new file downloaded it twice and wrote over each other in place, so a reader could get half a file with a 7-day cache header; a file over the limit was downloaded twice on every request; a failed fetch, a 404, was cached by browsers for a week; cached media of a server suspended later were still served, and RejectMedia skipped avatars, emoji, covers, video variants, link cards and remote edits; /clientapi/group/members returned remote pictures raw. Now a download is shared by everyone asking at once, streamed into a .part file and renamed into place (FederationHttp.DownloadMedia copies bounded, never into memory), at most eight at a time; a file too big to cache is remembered for an hour and only streamed, a failure for five minutes; the cache's size is counted as it grows and trimmed as soon as it passes the cap; a cached file is opened before it is answered; browsers may cache only a success; nothing of a suspended server, or of one whose media are rejected, is proxied (everything remote a client sees goes through the proxy, so that covers every kind), and blocking one purges its cache; the proxy has its own rate limit per client address; group members' pictures are proxied; the proxy's key is loaded once, the oldest if two were made. This changes what PrivaPub serves its clients, not what it sends to other servers. Tests: clients asking at once share one download, a failure isn't cached by browsers, an over-limit file is fetched three times for two requests instead of four, a blocked server's media are refused and its cache purged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
e4f9d0b61e |
An image is checked before it is decoded
An upload went straight to libvips: whatever loader recognised the bytes ran (an SVG sent as image/png was rasterised), nothing bounded how many pixels it would decode to (a small PNG could decode to gigabytes, three times over), a GIF was loaded frame by frame and never resized, and all of it ran inside the request with nothing limiting how many at once. A GIF was typed gifv but stayed a .gif, which a gifv player can't play; its metadata was kept; colours lost their ICC profile without being converted; HEIC was advertised but the bundled libvips can't decode it. Now: - only libvips' JPEG, PNG, GIF, WebP and HEIF loaders ever run on an upload (every other loader is blocked); - the header alone says how big an image would decode, refused above Media:MaxPixels (40 MP) or MaxFrames; - a still image is shrunk on load, turned by its orientation and brought into sRGB (thumbnail), then written without metadata, a profile picture the same way; - an animated GIF becomes a looping silent H.264 mp4 typed gifv, as on Mastodon (PostMedia.Kind keeps it a gifv), and a remote GIF is an image; - processing runs Media:Concurrency at a time, and uploads have their own rate limit per credential; - HEIC and HEIF are no longer offered. Tests: only the upload formats load, the header tells the size, an SVG posing as a PNG and an image too large are refused before decoding, an animated GIF becomes a gifv and a still one an image, HEIC isn't advertised. The media scenarios against the pasture (GoToSocial, Mastodon, Misskey, Akkoma, Pixelfed, Smithereen, Vernissage, Castopod, PeerTube) pass: 329 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
01808fa644 |
Follow requests nobody answers are sent again
A Follow was resent only when the persona followed again. Lemmy 1.0 sends nothing it queued for a server before it started sending there, so the Accept of a community follow made on first contact was lost for good: the village's persona stayed "requested" a day while Lemmy listed her as a follower, and every post the community announced was refused as not followed. A request still unanswered is now sent again, the same activity, after 15 minutes, an hour, 6 hours, a day, two and four days (FollowResender, every 15 minutes); a server that holds the follow answers the copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
436f7da464 |
CDNs found by themselves, and servers followed through time
PrivaPub now finds CDNs three ways, best first: the address ranges the CDNs publish (Cloudflare, Fastly, Amazon CloudFront, Bunny, Gcore, Imperva), downloaded daily by CdnUpdater and kept in CdnRangeSet; the CDN's fingerprint in the responses it already gets from a server (EdgeHintsHandler on the federation client); and the networks that carry only a CDN. The fixed ASN list is gone; ASNs shared with plain hosting (AWS, DataPacket) no longer hide a server. A server's Geo records the CDN, its domain and how it was found, and weekly snapshots now keep the city and coordinates too. Servers through time (ServerPlaces): /instances/:host/history lists a server's weekly snapshots, a CDN-fronted server's geo names the CDN's domain and where the server was before it (before_cdn), and /api/privapub/v1/cdns and /cdns/:domain group servers by CDN with week by week who joined and who left. Owner decisions recorded in ROADMAP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
5f56681c01 |
Everything on, phase 1: geolocation fetches itself, the deploy signs in as @thepra, the crawler is on, sign-up by invitation
Owner decisions (2026-10-04, recorded in docs/ROADMAP.md): production runs everything that is built, and nothing waits
on a person running a command.
- Geolocation updates itself. GeoUpdater, a hosted service, checks daily whether each DB-IP Lite database was built this
month. If not, it fetches this month's, or last month's early in the month. It installs a file only once it opens as
the right kind of database, then swaps it in atomically, and the locator reloads at once. Lookups now run under the
lock, so a reload can no longer dispose a reader mid-lookup. The systemd timer, its script and their setup.sh lines
are gone: the root step they needed never happened, and none is needed now. /stargazing names the database in use.
- The admin CLI runs after the app is built, with every service and nothing started.
- `create-root <login> [--admin]` takes the password on stdin; it is how the first login is made while sign-up is
closed.
- `smoke <persona>` keeps the root `deploy-smoke` and an undiscoverable persona, and gives the root a new password
on every run.
- The deploy signs in as @thepra. It runs the CLI, gets a token through the real OAuth flow (tools/smoke/oauth.sh,
moved out of the pasture's privapub_token, which now uses it), checks the signed-in API and that @thepra is
undiscoverable, then revokes the token. PRIVAPUB_SMOKE_TOKEN is gone.
- The deploy also fails when:
- NodeInfo and the instance API disagree about registrations;
- /stargazing does not say the crawler is on;
- the geolocation databases are missing or more than 40 days old.
- The crawler is on in production, seeded with ten large servers of different kinds. FEDERATION.md now describes it
and how to opt out.
- One registrations switch (Registrations:Mode, default Invitations; Open in tests and the pasture). It is read by
open sign-up (403 when closed), NodeInfo `openRegistrations`, and v1 and v2 of the instance API, so they can no longer
disagree. Before, NodeInfo said open and the instance API said closed. Group invitations always work, so
invites_enabled is true.
- A persona edit through /clientapi no longer resets what the Mastodon API set (discoverable, locked, quote policy…):
the theme is merged into the settings instead of replacing them.
650 tests pass. The deploy's smoke step was rehearsed against the pasture's PrivaPub.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
|
||
|
thepraandClaude Opus 5.5
|
fc5bb9511f |
T6: the Mastodon API over HTTP
88 integration tests drive the Mastodon client API through the whole server
(PrivaPubHost), with remote actors on an in-process Peer and deliveries read
from the job queue. Helpers live in Support/Host/MastodonHelpers.cs.
Coverage:
- Accounts: verify_credentials (no root id or login name); update_credentials
with indexed and array fields_attributes (form and JSON), source[*],
quote_policy, locked/bot, avatar and header uploads resized and stripped of
EXIF and XMP; lookup (local, @domain, remote; a circle, the instance actor
and a circle's id answer 404); search with and without resolve (only a
signed-in persona resolves, the Peer is untouched otherwise), by post and
actor address, hashtags, undiscoverable personas; account statuses with
pinned, exclude_replies, exclude_reblogs, only_media, tagged and Link paging
both ways; followers-only posts for followers (local and remote authors);
community accounts; followers/following only to their owner; follow (open,
locked, remote Follow delivery), unfollow and Undo; follow requests from
local and remote followers answered with the original Follow;
remove_from_followers; blocks with Reject and Block/Undo deliveries; mutes
with duration and the notifications choice, never federated; domain blocks;
relationships with junk ids; a banned login's tokens; reports forwarded as a
Flag from the instance actor only; account stub routes.
- Statuses: each visibility's to/cc as delivered; CW as summary; replies to
local and remote posts (mention, inReplyTo, the author's inbox); polls and
votes (local, and remote votes only to the author without published); media
attached only by its owner; quotes, the quotes list and revocation; edit
history, source and the Update delivery; delete for redraft, the 410
Tombstone and the Delete delivery; favourite/reblog counts with Like,
Announce and their Undos; favourited_by, reblogged_by; bookmarks; pins;
interaction_policy matching canQuote in the note and in the Update;
strangers get 404 for followers-only and direct posts; a located post is
unreachable by id for anyone else on every route; statuses?id[];
Idempotency-Key; scopes; deleting a reblog.
- Timelines: home paging with max_id, since_id and min_id; public local and
remote; tag (anonymous); list stub; favourites; conversations and read;
markers; notifications with types[], exclude_types[], account_id, paging,
get, dismiss, clear and unread_count.
- Instance: v1 and v2 (4.2.0 (compatible; PrivaPub)), peers, activity, rules,
extended_description, apps and every stub route.
- Media: v1 and v2 uploads, owner-only GET and PUT, 422 for unsupported or
unreadable files, video and audio made with ffmpeg lavfi sources and checked
with ffprobe; every remote media address goes through the proxy; the proxy
refuses unsigned URLs, streams ranges as 206 without caching, caches whole
downloads and serves them with ranges, and streams anything over
Media:MaxProxiedBytes (a SmallProxyHost) without caching.
- Provenance of local, delivered (signature) and fetched (instance actor,
no signature, the trigger as activity) posts, visibility of provenance,
instance descriptions; reading any of them makes no outbound request.
Pleroma reactions with EmojiReact and Undo deliveries, and local reaction
notifications.
Bugs fixed:
- remove_from_followers deleted the Follower row but never told a remote
follower. It now sends Reject{Follow} with the stored Follow id, through
RelationshipService.RemoveFollower, which Block now shares.
- VisibilityPolicy.CanSee refused followers-only posts to accepted followers,
so a post in their home timeline answered 404 to GET, context, favourite and
reply. Followers of the author (local or remote) may now see them.
- Account statuses of a remote account hid followers-only posts from
personas that follow it.
- exclude_replies dropped the author's own threads; like Mastodon it now
drops only replies to other accounts.
- A community account's statuses were always empty: they are now the posts
addressed to the community.
- Pinning someone else's visible post answered 404; it answers 422 like
Mastodon.
- GET /api/v1/notifications/:id answered 200 with null when the notification's
post was gone; it answers 404.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
|
||
|
thepraandClaude Opus 5.5
|
31d614efd4 |
tests: the federation surface's group helper is FederatedGroup, so it no longer clashes with the client API's
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2 |
||
|
thepraandClaude Opus 5.5
|
2cfea7b60c |
T7: the federation surface over HTTP
Tests through the real routes of PeasantsController, WellKnownController and UsersController, on the whole server under test (34 new tests): - FederationGetTests: the actor document (activity+json, SPKI key at #main-key owned by the actor, sharedInbox, published = PublishedOn's day, no creation date, no root); ld+json; browsers sent to /@name; Vary: Accept; /users 301; the outbox's totalItems and ?page=true&max_id paging across the 20-item boundary, boosts as Announces, and no followers-only, direct, located, federated-copy or deleted post; /groupies and /stalking naming nobody; /trophies (public pins, newest first) and /tattoos; /scribbles (public and unlisted 200, browsers redirected, followers-only, direct and located 404, deleted 410 Tombstone); a circle post only for a signed member or its instance actor; a circle's /groupies, /flock and /wardens only for members; /grunts create- and announce- ids; /parrot-licences 200, revoked 410, wrong author 404; secure mode's 401 for every unsigned GET but the instance actor's. - WellKnownTests: WebFinger by acct:, @-prefixed, bare, upper-case and actor URI; other domains, unknown names, a root's login name and no resource; the instance actor, a community, a circle (answered: current behaviour); NodeInfo links, 2.0 and 2.1 naming no root, unknown versions 404; usage counting only public, unlisted, non-boost local posts (Exclusive). - InboxRouteTests: all three inboxes accept a signed delivery and refuse junk (400), unsigned (401), a bad Digest, a two-hour-old Date, a signature for another host and a swapped body (401); an unknown persona's /mouth is 404; ld+json with the ActivityStreams profile is accepted; a signer whose actor answers 503 gets 503 with Retry-After; the 301st unsigned POST from one address is 429 while a signed server from it is not. - PersonaSeparationHttpTests: with a sibling persona and its community on the same login, every GET under /api (filled with the persona's ids) plus search, lookup and relationships, and a crawl of everything federation publishes about the persona (actor, outbox pages, collections, scribbles, grunts, WebFinger, NodeInfo, /@ pages), never name the sibling, its community or the login. Fixed: - A circle's /groupies told anyone how many followers (members) it has, while its /flock and /wardens were already for members only; it now answers 404 to anyone but a signed member or a member's instance actor. - WebFinger answered 404 to a bare user@domain or @user@domain resource, which Mastodon, GoToSocial and Pleroma all accept; it now treats them as acct: (noted in docs/INTEROP.md). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2 |
||
|
thepraandClaude Opus 5.5
|
c5e4934ba6 |
T5: OAuth and the client API over HTTP
96 integration tests through PrivaPubHost, the real pipeline end to end:
- OAuth: the token's subject is the persona, and neither the token response,
verify_credentials nor the stored token entries name the root. A wrong password shows
an error and sets no login cookie; a login without the antiforgery token is a 400; the
return address never leaves the site; deny answers access_denied with no code; another
root's persona re-renders the choice with no code; a banned root is sent back to the
login and a code issued before the ban buys no token; force_login asks again; a code
works once and its reuse revokes the token it bought; password and refresh_token
grants are refused; a client_credentials token gets 401 on user routes; a read-only
token gets 403 with a Mastodon error on POST /api/v1/statuses; follow covers
read:follows; revoke works; the login and authorize pages send their CSP and no-store;
the 11th /oauth/login from one address in a minute is a 429.
- Accounts: sign-up, duplicates in any case, invalid models answer 400 with a message,
login and logout, recovery email, settings, password change, invitation sign-up and
login (refusing a persona named after the login), recovery without an email, through
an unreachable mail server, with a wrong and with a valid code, token refresh, the 11th
sign-up from one address, expired, garbage and foreign-key JWTs.
- Personas: a rootId in the body is ignored, the username regex and reserved names hold,
personas and groups share ReservedName, an update delivers Update{Person} to followers,
PublishedOn and the id's day fall within two weeks before creation, the list holds only
one's own personas, another root's persona cannot be updated.
- Groups: communities and circles are created, joining takes the code and the password,
members leave and owners cannot, a remote follow request becomes a member only on
approval, a circle never shows in lookup, account by id, v2 search or /@name, and its
/flock and /wardens answer 404 unsigned and to non-members, 200 to a member's signed GET.
- Moderation (Exclusive, it suspends localhost): ban, unban and remove; non-admins get
403; reports are listed without the reporter and resolved; domain blocks are inserted,
listed and deleted, bad domains refused, and a suspended server's delivery is answered
202 and kept nowhere; the data endpoints.
- AdminCommands: exit codes 0, 1 and 2 and the resulting policies.
Fixed:
- A banned or removed root kept using /clientapi with its JWT until it expired: only /api
re-checked the root. JwtEvents.TokenValidated now loads the root and fails the request
when it is banned or deleted, and takes the policy claims from the database, so a
demoted admin loses admin at once (and a promoted one gains it).
- The 401 and 403 bodies JwtEvents writes were PascalCase while every other /clientapi
answer is camelCase; they now use the web defaults.
- /clientapi/user/sniff/again (token refresh) answered an empty 200; it now answers a
fresh JwtUser, like login.
- Password recovery answered SMTP reply codes as HTTP statuses (421, 454, 554, and 550 for
an invalid address); a mail server failure is now 503 and an invalid address 400.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
|
||
|
thepraandClaude Opus 5.5
|
fc15f6356d |
M7: daily rollups
A RollupDay job folds each finished day's InteractionEvents into one InstanceDay per server:
- Counters for the admin, keyed channel:activity:object:outcome:reason, plus signature
schemes, audiences, local kinds and features;
- PublicCounters, everything a public page may ever read:
- inbound and outbound activities from an allowlist, on public, unlisted or unaddressed
traffic only, with the outcome collapsed (accepted or dropped, delivered or failed);
- no reasons, no Flag or Block;
- features of public objects;
- health:ok or health:failed from reachability (a 4xx means the server answered);
- latency, wait and byte histograms, and the number of distinct accounts from that day's
hashes.
Re-running a day replaces it and keeps the live Reads counters. The day's salt is then
deleted, so its hashes can never be recomputed, and the next day is queued.
StatisticsSchedule plans today's rollup every hour and catches up any of the last seven
days that have events but no rollup. Waits round up into their bucket.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
|
||
|
thepraandClaude Opus 5.5
|
5e34517e73 |
T3: the whole server under test
PrivaPubHost is a WebApplicationFactory<Program> on the fixture's database, configured only through UseSetting (visible before Build, unlike ConfigureAppConfiguration). It drops the background workers so tests run the jobs they queue (Jobs.Run, RunInbox), gives each client its own address for the rate limiter, and has a SecureMode variant. Accounts signs up roots, adds personas and gets Mastodon tokens through the real /oauth code flow. RemoteActor signs HttpRequestMessages for the real /peasants routes; Peer records bodies and headers and serves files with ranges and text pages. Program registers the Guid serializer with TryRegisterSerializer, so a second host in one process starts; the fixture runs the migrations in production's order before any test. HostBootTests: the host shares the fixture database; the harness handles exactly the activities the server registers; every job kind has one handler; every controller and page model can be made; the service graph validates with ValidateOnBuild and ValidateScopes; Swagger is 404 outside Development; a persona's token never names its root; a signed DM through the real /mouth route is queued, processed and stored. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2 |