A file lives exactly as long as something holds it
Deleting a post, editing media out, replacing an avatar or a header, and removing a whole root deleted no file: every one stayed on disk and publicly served from /media/files with a year-long immutable cache, its row orphaned. Now every upload is a row, profile pictures too (Kind avatar or header, ProfileOfAvatarId), and each of those acts trashes what it held, as does an upload never posted for a day and a dropped scheduled post. A trashed row is marked in one conditional update (an upload attached meanwhile is left alone), its files move into media-trash, beside the media root and outside what /media/files serves, and the janitor deletes them a day later. Nothing is deleted for looking unused. Along the way: a profile picture that isn't an image, or can't be read, answers 422 instead of being silently ignored with a 200; a removed root's scheduled posts are dropped, so nothing of it publishes later; media rows get indexes (they had none), and the janitor's first pass comes five minutes after boot instead of an hour. `PrivaPub admin media audit [--fix]` compares the disk with the database. With --fix (as www-data) it gives the pictures personas show today a row, and trashes media of deleted posts or personas, rows whose files are missing, and files nothing holds: the leftovers of every deletion until now. MediaLifecycleTests covers each act, that the trash is never served, and the audit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
This commit is contained in:
1 parent
52d201eee9
commit
bb680e8cb8
15 files changed
+462
-36
No files matched your search
@@ -297,19 +297,30 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
|
||||
1. **No upload keeps its metadata.** Images are re-encoded by libvips with `keep=none`; audio and video are remuxed with
|
||||
`-map_metadata -1`. `MediaProcessingTests` checks EXIF and XMP are gone.
|
||||
2. Files live under `Media:Root` (`/var/lib/privapub/media`), never in the published directory; the proxy cache is the
|
||||
sibling `media-proxy`, which `/media/files` does not serve.
|
||||
3. **A client never contacts a remote server for media:** every remote URL the API returns goes through
|
||||
sibling `media-proxy` and the trash the sibling `media-trash`, neither of which `/media/files` serves.
|
||||
3. **A file lives exactly as long as something holds it.** Every upload is a `MediaAttachment` row, profile pictures
|
||||
too (`Kind` avatar or header, `ProfileOfAvatarId`). Deleting a post, an edit leaving media out, a replaced picture,
|
||||
a dropped scheduled post, a removed root, and an upload never posted for a day each trash theirs
|
||||
(`IMediaService.Trash`):
|
||||
- the row gets `TrashedAt` in one conditional update, so a row attached meanwhile is left alone;
|
||||
- its files move into `media-trash` at once, so `/media/files` stops serving them;
|
||||
- `MediaJanitor` deletes them a day later (`TrashGrace`).
|
||||
|
||||
Nothing is deleted because it looks unused. `PrivaPub admin media audit [--fix]` compares disk and database: with
|
||||
`--fix` (as www-data) it gives pictures shown from before their rows a row, and trashes media of deleted posts or
|
||||
personas, rows whose files are missing, and files nothing holds.
|
||||
4. **A client never contacts a remote server for media:** every remote URL the API returns goes through
|
||||
`IMediaProxy.Wrap`, an HMAC-signed `/media/proxy/` URL fetched by `IFederationHttp.GetMedia`.
|
||||
4. **The proxy serves three ways:**
|
||||
5. **The proxy serves three ways:**
|
||||
- **Cached:** a file already cached is served from disk, ranges included.
|
||||
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached.
|
||||
- **Streamed:** a ranged request, or anything too big to cache, is streamed from the origin with the range passed on,
|
||||
and never cached. That is how remote video plays.
|
||||
|
||||
nginx has a `/media/proxy/` location with `proxy_buffering off` and a 600 s read timeout for those streams.
|
||||
5. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made
|
||||
6. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made
|
||||
every status, timeline and Note holding its post fail to serialise; migration `_016` removed the ones stored before.
|
||||
6. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
|
||||
7. **Remote video and audio become one playable attachment** in the Mastodon API (`MastodonMapper.Playable`): the best MP4
|
||||
up to 720p that carries both sound and picture, including PeerTube's fragmented files inside an HLS entry. HLS
|
||||
playlists themselves are not rewritten.
|
||||
|
||||
|
||||
Reference in new issue
Block a user