The media proxy is bounded

Anyone could mint signed proxy URLs (a remote account changes its icon, an anonymous lookup returns the URL), and each
anonymous request held up to 40 MB in memory; the cache grew without bound between hourly trims; two clients asking
for the same new file downloaded it twice and wrote over each other in place, so a reader could get half a file with a
7-day cache header; a file over the limit was downloaded twice on every request; a failed fetch, a 404, was cached by
browsers for a week; cached media of a server suspended later were still served, and RejectMedia skipped avatars,
emoji, covers, video variants, link cards and remote edits; /clientapi/group/members returned remote pictures raw.

Now a download is shared by everyone asking at once, streamed into a .part file and renamed into place
(FederationHttp.DownloadMedia copies bounded, never into memory), at most eight at a time; a file too big to cache is
remembered for an hour and only streamed, a failure for five minutes; the cache's size is counted as it grows and
trimmed as soon as it passes the cap; a cached file is opened before it is answered; browsers may cache only a
success; nothing of a suspended server, or of one whose media are rejected, is proxied (everything remote a client
sees goes through the proxy, so that covers every kind), and blocking one purges its cache; the proxy has its own rate
limit per client address; group members' pictures are proxied; the proxy's key is loaded once, the oldest if two
were made. This changes what PrivaPub serves its clients, not what it sends to other servers.

Tests: clients asking at once share one download, a failure isn't cached by browsers, an over-limit file is fetched
three times for two requests instead of four, a blocked server's media are refused and its cache purged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
This commit is contained in:
thepraandClaude Opus 5.5 committed 2026-10-07 10:54:33 +02:00
1 parent 8e59145826
commit 3ca29603ed
11 files changed
+399 -73

No files matched your search

+16 -3
View File
@@ -336,10 +336,23 @@ group www-data and reaches the private mongod; `sudo -u www-data` works too.
6. **A client never contacts a remote server for media:** every remote URL the API returns goes through
`IMediaProxy.Wrap`, an HMAC-signed `/media/proxy/` URL fetched by `IFederationHttp.GetMedia`.
7. **The proxy serves three ways:**
- **Cached:** a file already cached is served from disk, ranges included.
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached.
- **Cached:** a file already cached is served from disk, ranges included. It is opened before the answer, so a trim
that deletes it meanwhile breaks nothing.
- **Downloaded:** a request without a `Range` is downloaded whole, up to `Media:MaxProxiedBytes`, then cached:
- once for everyone asking for it at the same time;
- streamed into a `.part` file beside its place and renamed there, never held in memory;
- with its host in its `.type` sidecar, so that blocking a server can purge it.
- **Streamed:** a ranged request, or anything too big to cache, is streamed from the origin with the range passed on,
and never cached. That is how remote video plays.
and never cached. That is how remote video plays. A file found too big is remembered for an hour, so it isn't
fetched twice; one that failed is remembered for five minutes.
Also:
- The cache's size is counted as it grows and trimmed as soon as it passes `ProxyCacheBytes`; the janitor
recounts hourly.
- Browsers may cache only a success.
- Nothing of a suspended server, or of one whose media are rejected, is proxied (avatars, emoji, covers and link
cards included, since they all go through it), and blocking one purges its cache.
- The `proxy` rate limit counts per client address.
nginx has a `/media/proxy/` location with `proxy_buffering off` and a 600 s read timeout for those streams.
8. **A focal point is two finite numbers** within -1..1 (`FocalPoint.Parse`); anything else is ignored. A stored NaN made