Audio and video are processed off the request

Every audio and video upload was probed and remuxed inside the request, whatever its length; ffmpeg would read any
protocol and probe any format; `-map 0` kept the data tracks iPhones add, which mp4 refuses; nothing was ever
transcoded, so HEVC or MPEG-4 Part 2 reached browsers that can't play them, and the advertised video_matrix_limit
and frame rate limit were never applied; the output was read whole into memory, the video was saved before its
poster could fail, and the poster's frame leaked in /tmp. FLAC uploads were served as 404.

Now an upload sent to /api/v2/media is stored as sent in media-incoming (beside the media root, never served) and
answered with 202 and no url, while a ProcessMedia job, one at a time, does the work; GET /api/v1/media/:id answers 206
until it is ready, or 422 with why, and media still processing can't be posted. v1 processes before answering.
ffmpeg reads only that file (protocol whitelist, format forced from the probe) and drops data and subtitle tracks. A
video browsers play as it is (H.264, VP8, VP9, AV1 within Media:MaxVideoPixels and MaxFrameRate) is remuxed, anything
else transcoded to H.264 that fits, as Mastodon does; longer than Media:MaxSeconds is refused. Outputs move into place
only once everything succeeded, every temporary file goes, durations are kept, FLAC is served as audio/flac, and the
unit gets PrivateTmp. The instance API advertises the limits that are now applied.

No pasture scenario uploads audio or video through PrivaPub, so the sweep could not see this. MastodonMediaTests: v2
answers 202 then the job makes it playable (and an unreadable file 422 once processed), media still processing can't
be posted, MPEG-4 Part 2 becomes H.264, a video over the limit is made smaller, FLAC is served.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
This commit is contained in:
thepraandClaude Opus 5.5 committed 2026-10-07 10:48:56 +02:00
1 parent e4f9d0b61e
commit 8e59145826
14 files changed
+281 -48

No files matched your search

+18 -6
View File
@@ -308,9 +308,21 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
- **Types:** HEIC and HEIF are not accepted, since the bundled libvips has no HEVC decoder.
- **Load:** processing runs `Media:Concurrency` at a time, and uploads have their own rate limit (`uploads`, per
credential).
3. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
3. **Audio and video are processed off the request when the client allows it.**
- **v2:** `POST /api/v2/media` stores them as sent in `media-incoming` (beside the root, never served) and answers
202 with no url, while a `ProcessMedia` job (one at a time, its lease renewed) does the work. Until then
`GET /api/v1/media/:id` answers 206, and 422 with the reason if it failed. Media still processing can't be posted.
- **v1:** processes before answering.
- **ffmpeg** reads only the stored file: `-protocol_whitelist file`, and the input format is forced from the probe.
Data and subtitle tracks are dropped (`-dn -sn`, which iPhone MOVs need).
- **Remux or transcode:** a video browsers play as it is (H.264, VP8, VP9 or AV1, within `Media:MaxVideoPixels` and
`MaxFrameRate`) is remuxed; anything else is transcoded to H.264 that fits, as Mastodon does.
- **Limits:** longer than `Media:MaxSeconds` is refused.
- **Saving:** nothing is saved until everything succeeded; outputs move into place rather than being read into
memory, and every temporary file goes. FLAC is served as `audio/flac`, and the unit runs with `PrivateTmp`.
4. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
sibling `media-proxy` and the trash the sibling `media-trash`, neither of which `/media/files` serves.
4. **A file lives exactly as long as something holds it.** Every upload is a `MediaAttachment` row, profile pictures
5. **A file lives exactly as long as something holds it.** Every upload is a `MediaAttachment` row, profile pictures
too (`Kind` avatar or header, `ProfileOfAvatarId`). Deleting a post, an edit leaving media out, a replaced picture,
a dropped scheduled post, a removed root, and an upload never posted for a day each trash theirs
(`IMediaService.Trash`):
@@ -321,18 +333,18 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
Nothing is deleted because it looks unused. `PrivaPub admin media audit [--fix]` compares disk and database: with
`--fix` (as www-data) it gives pictures shown from before their rows a row, and trashes media of deleted posts or
personas, rows whose files are missing, and files nothing holds.
5. **A client never contacts a remote server for media:** every remote URL the API returns goes through
6. **A client never contacts a remote server for media:** every remote URL the API returns goes through
`IMediaProxy.Wrap`, an HMAC-signed `/media/proxy/` URL fetched by `IFederationHttp.GetMedia`.
6. **The proxy serves three ways:**
7. **The proxy serves three ways:**
- **Cached:** a file already cached is served from disk, ranges included.
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached.
- **Streamed:** a ranged request, or anything too big to cache, is streamed from the origin with the range passed on,
and never cached. That is how remote video plays.
nginx has a `/media/proxy/` location with `proxy_buffering off` and a 600 s read timeout for those streams.
7. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made
8. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made
every status, timeline and Note holding its post fail to serialise; migration `_016` removed the ones stored before.
8. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
9. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
up to 720p that carries both sound and picture, including PeerTube's fragmented files inside an HLS entry. HLS
playlists themselves are not rewritten.