8fecf7de2dc4edfd2709e1a3e402526cce4c0457
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  thepraandClaude Opus 5.5
|
3ca29603ed |
The media proxy is bounded
Anyone could mint signed proxy URLs (a remote account changes its icon, an anonymous lookup returns the URL), and each anonymous request held up to 40 MB in memory; the cache grew without bound between hourly trims; two clients asking for the same new file downloaded it twice and wrote over each other in place, so a reader could get half a file with a 7-day cache header; a file over the limit was downloaded twice on every request; a failed fetch, a 404, was cached by browsers for a week; cached media of a server suspended later were still served, and RejectMedia skipped avatars, emoji, covers, video variants, link cards and remote edits; /clientapi/group/members returned remote pictures raw. Now a download is shared by everyone asking at once, streamed into a .part file and renamed into place (FederationHttp.DownloadMedia copies bounded, never into memory), at most eight at a time; a file too big to cache is remembered for an hour and only streamed, a failure for five minutes; the cache's size is counted as it grows and trimmed as soon as it passes the cap; a cached file is opened before it is answered; browsers may cache only a success; nothing of a suspended server, or of one whose media are rejected, is proxied (everything remote a client sees goes through the proxy, so that covers every kind), and blocking one purges its cache; the proxy has its own rate limit per client address; group members' pictures are proxied; the proxy's key is loaded once, the oldest if two were made. This changes what PrivaPub serves its clients, not what it sends to other servers. Tests: clients asking at once share one download, a failure isn't cached by browsers, an over-limit file is fetched three times for two requests instead of four, a blocked server's media are refused and its cache purged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
8e59145826 |
Audio and video are processed off the request
Every audio and video upload was probed and remuxed inside the request, whatever its length; ffmpeg would read any protocol and probe any format; `-map 0` kept the data tracks iPhones add, which mp4 refuses; nothing was ever transcoded, so HEVC or MPEG-4 Part 2 reached browsers that can't play them, and the advertised video_matrix_limit and frame rate limit were never applied; the output was read whole into memory, the video was saved before its poster could fail, and the poster's frame leaked in /tmp. FLAC uploads were served as 404. Now an upload sent to /api/v2/media is stored as sent in media-incoming (beside the media root, never served) and answered with 202 and no url, while a ProcessMedia job, one at a time, does the work; GET /api/v1/media/:id answers 206 until it is ready, or 422 with why, and media still processing can't be posted. v1 processes before answering. ffmpeg reads only that file (protocol whitelist, format forced from the probe) and drops data and subtitle tracks. A video browsers play as it is (H.264, VP8, VP9, AV1 within Media:MaxVideoPixels and MaxFrameRate) is remuxed, anything else transcoded to H.264 that fits, as Mastodon does; longer than Media:MaxSeconds is refused. Outputs move into place only once everything succeeded, every temporary file goes, durations are kept, FLAC is served as audio/flac, and the unit gets PrivateTmp. The instance API advertises the limits that are now applied. No pasture scenario uploads audio or video through PrivaPub, so the sweep could not see this. MastodonMediaTests: v2 answers 202 then the job makes it playable (and an unreadable file 422 once processed), media still processing can't be posted, MPEG-4 Part 2 becomes H.264, a video over the limit is made smaller, FLAC is served. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
e4f9d0b61e |
An image is checked before it is decoded
An upload went straight to libvips: whatever loader recognised the bytes ran (an SVG sent as image/png was rasterised), nothing bounded how many pixels it would decode to (a small PNG could decode to gigabytes, three times over), a GIF was loaded frame by frame and never resized, and all of it ran inside the request with nothing limiting how many at once. A GIF was typed gifv but stayed a .gif, which a gifv player can't play; its metadata was kept; colours lost their ICC profile without being converted; HEIC was advertised but the bundled libvips can't decode it. Now: - only libvips' JPEG, PNG, GIF, WebP and HEIF loaders ever run on an upload (every other loader is blocked); - the header alone says how big an image would decode, refused above Media:MaxPixels (40 MP) or MaxFrames; - a still image is shrunk on load, turned by its orientation and brought into sRGB (thumbnail), then written without metadata, a profile picture the same way; - an animated GIF becomes a looping silent H.264 mp4 typed gifv, as on Mastodon (PostMedia.Kind keeps it a gifv), and a remote GIF is an image; - processing runs Media:Concurrency at a time, and uploads have their own rate limit per credential; - HEIC and HEIF are no longer offered. Tests: only the upload formats load, the header tells the size, an SVG posing as a PNG and an image too large are refused before decoding, an animated GIF becomes a gifv and a still one an image, HEIC isn't advertised. The media scenarios against the pasture (GoToSocial, Mastodon, Misskey, Akkoma, Pixelfed, Smithereen, Vernissage, Castopod, PeerTube) pass: 329 checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
bb680e8cb8 |
A file lives exactly as long as something holds it
Deleting a post, editing media out, replacing an avatar or a header, and removing a whole root deleted no file: every one stayed on disk and publicly served from /media/files with a year-long immutable cache, its row orphaned. Now every upload is a row, profile pictures too (Kind avatar or header, ProfileOfAvatarId), and each of those acts trashes what it held, as does an upload never posted for a day and a dropped scheduled post. A trashed row is marked in one conditional update (an upload attached meanwhile is left alone), its files move into media-trash, beside the media root and outside what /media/files serves, and the janitor deletes them a day later. Nothing is deleted for looking unused. Along the way: a profile picture that isn't an image, or can't be read, answers 422 instead of being silently ignored with a 200; a removed root's scheduled posts are dropped, so nothing of it publishes later; media rows get indexes (they had none), and the janitor's first pass comes five minutes after boot instead of an hour. `PrivaPub admin media audit [--fix]` compares the disk with the database. With --fix (as www-data) it gives the pictures personas show today a row, and trashes media of deleted posts or personas, rows whose files are missing, and files nothing holds: the leftovers of every deletion until now. MediaLifecycleTests covers each act, that the trash is never served, and the audit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
52d201eee9 |
A focal point is two finite numbers
float.TryParse takes "NaN" and "Infinity", and Math.Clamp keeps a NaN, so `focus=NaN,NaN` on an upload or a media PUT was stored as given. Once the media was posted, every status and timeline response holding the post, and the Note and its delivery, failed to serialise, for every viewer. FocalPoint.Parse takes finite numbers only (anything else is ignored, as an unreadable focus was); the mapper and the renderer leave out a focus that isn't sound, and migration _016 removes the ones stored before, from the uploads and from the copy each post keeps. What PrivaPub sends is unchanged for any focus that could be sent before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
cafd2e6db3 |
Scheduled posts: written now, published at their time
POST /api/v1/statuses with scheduled_at no longer posts at once: the post is kept as asked (at least five minutes ahead; 300 waiting, 25 a day, as Mastodon allows), its media kept from the janitor, and a PublishScheduled job publishes it at its time as the persona. scheduled_statuses lists, moves and drops them; a moved post's old job finds it not due. Idempotency-Key holds for scheduling too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw |
||
|
thepraandClaude Opus 5.5
|
8f25bf056d |
Everything on, phase 4a: one answer everywhere for counts, search, collections and the instance API
Owner decision 2026-10-04: fix the mismatches and every other mismatch of the same kind.
- One counting rule (Domain/Privacy/Counted), Mastodon's. It is used for a persona's statuses_count, its outbox
totalItems, NodeInfo localPosts and the instance status_count, which used to count four different things. It
counts every post that is neither deleted nor a DM, boosts included, and circle and located posts too (owner
decision). A group's count includes its remote members' posts.
- Users. Personas of banned or deleted roots no longer count, and are not found in search. NodeInfo now gives
activeMonth and activeHalfyear, and the v2 instance gives active_month instead of a constant 0.
- replies_count counts only public and unlisted replies, so it no longer tells anyone that a private reply exists.
Migration _012 recounts it.
- A remote account that deletes itself takes everything out of every count (GoneActors): its likes, downvotes,
reactions and poll votes go and their counters come back, as do its boosts', replies' and quotes' counts, and its
notifications. Lookups, account lists, search and favourited_by no longer show it. Migration _012 applies this to
accounts already gone.
- Deleting a post also deletes its pins and the local boosts of it.
- /stalking gives the same total as following_count. Members are still never listed, and hide_collections is now
always true, since the setting never did anything.
- Joining a community by invitation is following it, so /flock and /groupies agree; leaving unfollows.
- Search. Anyone may search, as on Mastodon; resolve and offset need a sign-in, offset pages, and deleted accounts
are never found.
- notifications/unread_count counts what the list shows, and the owner's follower and following lists page with
Link.
- The instance API advertises what is enforced:
- max_characters, now enforced with a 422;
- max_pinned_statuses = MaxPins;
- the media types and limits MediaService and MediaOptions accept;
- PollService's limits;
- the configured languages;
- no streaming URL until streaming exists.
domain_count counts the servers we have exchanged with; which ones stays unpublished (peers is empty).
- Routes Mastodon answers now answer instead of 404:
- directory, tags/{name}, timelines/link and identity_proofs;
- instance/languages, translation_languages, domain_blocks and privacy_policy;
- the v1 and v2 notification policy, and notification requests.
Also, from phase 3: a recovered password ends /clientapi sessions through a per-root SessionStamp claim instead of
comparing the JWT's whole-second nbf with the change time. That comparison let a token issued in the same second
survive, which made a test flaky.
671 tests pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
|
||
|
thepraandClaude Opus 5.5
|
2645dea26f |
T8: inbox gaps, jobs, migrations and pages; a deleted remote account's posts are hidden
Owner decision (2026-10-03, "A remote account deletes itself"): its posts are kept but
hidden everywhere.
- Post.AuthorGone (additive bool). DeleteHandler's actor-delete branch sets it on every
post whose ActorURI is the actor (one update-many), besides dropping its follows and
timeline rows as before. RemotePosts.Build sets it on a post stored later for an
account already marked Deleted.
- One rule in VisibilityPolicy: IsShown (not deleted, author not gone), IsPublic and
CanSee exclude AuthorGone, plus Shown(post) for loaded posts.
- Lookups by id answer 404 through CanSee (statuses/:id and every sub-route, context,
bookmarks, favourites, polls, reactions, search); provenance, account statuses,
home/public/tag timelines, notifications, conversations, reblogged_by, the clientapi
home and post/DM lists, a community's outbox and our Announces filter on IsShown or
IsPublic; the Mastodon mapper never renders a hidden post or a boost of one.
Tests (30 new):
- AuthorGoneTests: the rule, the handler (posts kept, boosts included, follows and rows
gone), a post fetched after the delete, and 20 Mastodon/ActivityPub lookups over HTTP
seen before and hidden after.
- InboxGapTests: actor Update refresh (name, sanitised summary, key rotation in place
and to a new key id) even with an older `updated`; Undo{Follow} by activity id and by
object; Reject of our QuoteRequest (and a stranger's ignored); group-wrapped
Announce{Like} and Announce{Undo{Like}}; a locked persona's pending follow,
FollowRequest notification, and Decide accepting and rejecting with the original Follow.
- JobHandlerTests: AncestorsJobHandler up to its depth limit; PollRefreshJob and
PollCloseJob (local and remote polls); InstanceDescriber from a peer's NodeInfo and
the weekly dedupe through ObjectRecords; LinkPreviews for public posts only;
DeliveryJobHandler outcomes (2xx, 404/410, 429/503 with Retry-After in seconds and as
a date, 5xx) and a signature and Digest the peer can verify; MediaJanitor.Sweep;
OAuthPruner.Prune.
- MigrationTests: _003, _004, _006 and _007 on seeded rows.
- PublicPagesTests: /@user and /@user/{id} (visibility, junk ids, exact CSP,
Referrer-Policy and nosniff), circle 404, community page, the instance actor,
ActivityPub redirects, and markup escaped in posts, titles and bios.
Production changes besides the rule:
- LinkPreviews.Handle re-checks that a post is still shown and public (the rule
Wanted applies) before fetching anything; before, only enqueueing checked it.
- The legacy /clientapi post and DM lists no longer return soft-deleted posts.
- MediaJanitor.Sweep and OAuthPruner.Prune are the loop bodies, now public and tested.
- InstanceDescriber.Address: a protected virtual identity seam so a test can point
the https NodeInfo addresses at a plain-http peer; production behaviour unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
|
||
|
thepraandClaude Opus 5.5
|
f9658a8e7a |
P6: remote video and audio play through the proxy
- The media proxy streams ranged requests from the origin (passing the range on, never caching), downloads and caches whole files otherwise, and serves cached files with range support. A PeerTube video is never fetched whole for one viewer, and clients still never contact the remote host. - PeerTube's fragmented MP4 files inside an HLS entry are read as variants, so HLS-only instances play too. - A remote Video or Audio post becomes one playable Mastodon attachment: the best MP4 up to 720p that carries both sound and picture, with its poster and duration; the card is kept only when nothing is playable. - nginx: /media/proxy/ with proxy_buffering off and a 600 s read timeout (applied on the box, with a backup). Checked live: a GoToSocial image through the proxy answers 206 with exactly the asked range when streamed, 200 when cached, and 206 with the right Content-Range from the cache. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB |
||
|
thepraandClaude Opus 5.5
|
4a713f3fb6 |
Media: uploads stripped of metadata, attachments both ways, a remote media proxy
- /api/v1/media and /api/v2/media (and GET/PUT /api/v1/media/:id):
images go through libvips (NetVips, its native build bundled):
autorotated, every kind of metadata dropped (EXIF, GPS, XMP, IPTC,
comments), capped at 4096 px, with a 640 px preview and a blurhash
(own encoder, the reference algorithm); animated GIFs are re-encoded;
video and audio are remuxed by ffmpeg with -map_metadata -1, never
re-encoded, and a video gets a still preview. Files get random names
under /var/lib/privapub/media, outside the web root deploys replace, and
are served at /media/files with nosniff and a sandbox CSP.
- media_ids on create and edit (four at most, the persona's own, each used
once); notes carry them as Document attachments with alt text, blurhash,
focalPoint and size; inbound attachments were already kept.
- avatar and header uploads in update_credentials, cropped to 400x400 and
1500x500, federated with Update{Person}.
- Remote media reaches clients only through /media/proxy/{hmac}/{url},
fetched by the guarded client (no SVG, 40 MB cap) and cached outside the
served root, trimmed to 5 GB; foreign avatars and headers use it too, so
a client never contacts another server.
- MediaJanitor deletes uploads left unattached for a day.
- nginx accepts 100 MB bodies on the upload endpoints only (applied on Max).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
|