PrivaPub now finds CDNs three ways, best first: the address ranges the
CDNs publish (Cloudflare, Fastly, Amazon CloudFront, Bunny, Gcore,
Imperva), downloaded daily by CdnUpdater and kept in CdnRangeSet; the
CDN's fingerprint in the responses it already gets from a server
(EdgeHintsHandler on the federation client); and the networks that carry
only a CDN. The fixed ASN list is gone; ASNs shared with plain hosting
(AWS, DataPacket) no longer hide a server. A server's Geo records the
CDN, its domain and how it was found, and weekly snapshots now keep the
city and coordinates too.
Servers through time (ServerPlaces): /instances/:host/history lists a
server's weekly snapshots, a CDN-fronted server's geo names the CDN's
domain and where the server was before it (before_cdn), and
/api/privapub/v1/cdns and /cdns/:domain group servers by CDN with week
by week who joined and who left. Owner decisions recorded in ROADMAP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
Off by default (Statistics:Crawler:Enabled), as the owner decided. When it is on:
- CrawlPlan runs hourly, from StatisticsSchedule. It inserts the configured seeds and
queues up to HostsPerHour servers whose last visit is older than RevisitDays, are not
paused by the breaker and are not domain-blocked, spread across the hour.
- CrawlInstance visits one server at a time as
"PrivaPub-Stargazer/<ref> (+<base>/stargazing)":
- it reads robots.txt (RFC 9309: its own group first, then PrivaPub, then *; longest
rule wins; a 4xx allows everything; a 5xx or no answer keeps it out);
- it describes servers that only crawling ever found, through
InstanceDescriber.Describe with robots.txt as the path filter;
- it reads /api/v1/instance/peers through the new GetStringArray, which keeps what it
read from the first 1 MB instead of refusing a large list;
- it adds the names a server could ever be reached at as "crawled": DNS only,
punycode, no addresses, ports or hidden services, and the reserved test names only on
a test network. Never more than MaxNewHostsPerCrawl per visit or MaxHosts in all, and
never over a touched server.
- It reads nothing but robots.txt, NodeInfo, the instance API and the peers list.
IFederationHttp.GetText serves robots.txt, and HttpScope.Crawl carries the
User-Agent.
- /stargazing explains all this and how to keep the crawler out, says whether it is on,
and credits DB-IP. GET /clientapi/admin/statistics/crawler shows the frontier.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
- IConnectedAddresses remembers which address each host's last connection reached, set in
SafeHttpHandlerFactory's connect callback. That is once per pooled connection, with no
second DNS lookup, and the address is never stored.
- IGeoLocator / DbIpLocator reads the offline DB-IP Lite city and ASN databases (MaxMind
.mmdb, via MaxMind.Db). It maps memory, swaps to new files within ten minutes, rounds
coordinates to one decimal, never looks up a private address, and answers nothing
when the files are missing. CdnNetworks names the CDNs whose edge addresses say nothing
about where a server is.
- deploy/max/geo-update.sh fetches this or last month's databases, checks them and swaps
them in atomically. The privapub-geo timer runs it monthly as www-data, and setup.sh
installs the directory, the script, the units and a first download.
Describing servers will use these in M8.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
- HttpScope (AsyncLocal) tags each outbound request with a purpose and a trigger:
- purpose is set by the caller: actor, key, object, webfinger, context, nodeinfo;
- trigger is set by the job kind, by "verify" during inbox verification, or defaults
to "request".
- FederationHttp records every JSON, media and stream fetch: status, time, bytes, hops,
and an outcome of ok, refused or failed, with a reason: disallowed, remembered,
bad-redirect, too-many-redirects, content-type, too-large, bad-json, private-address,
timeout, network, or the status. A fetch a reader caused (trigger "request") is only
counted per server per day.
- Link previews record a 'preview' event: card, no-card or failed.
- The media proxy counts cache hits.
- TrafficMeter counts the client API per endpoint group, method and status class. It
counts our served documents (actor, outbox, collection, object, activity, licence,
webfinger, nodeinfo) by kind, status and whether signed, per day and never per server,
and never names a circle's collections.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
- The media proxy streams ranged requests from the origin (passing the range on, never caching), downloads and caches
whole files otherwise, and serves cached files with range support. A PeerTube video is never fetched whole for one
viewer, and clients still never contact the remote host.
- PeerTube's fragmented MP4 files inside an HLS entry are read as variants, so HLS-only instances play too.
- A remote Video or Audio post becomes one playable Mastodon attachment: the best MP4 up to 720p that carries both
sound and picture, with its poster and duration; the card is kept only when nothing is playable.
- nginx: /media/proxy/ with proxy_buffering off and a 600 s read timeout (applied on the box, with a backup).
Checked live: a GoToSocial image through the proxy answers 206 with exactly the asked range when streamed, 200 when
cached, and 206 with the right Content-Range from the cache.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
- A public post that links to a page gets a preview card: from the post's own data first (FEP-8967 Link preview, the
object's image, title and summary); otherwise the server reads the page once, 0-60 s after the post arrives, never
when someone reads it. OpenGraph and Twitter tags give title, description and image; one LinkPreview per address
is cached for 7 days and shared by the whole server, so a fetch never points at a persona.
- Lemmy link posts, which carry no title or description, get them filled in.
- The page fetch uses the guarded client (public addresses, three redirects, HTML only, first 512 KB).
- Federation:FetchLinkPreviews switches page fetching off.
- Local public posts get cards too.
Checked live: a link to a GoToSocial profile page becomes a card with its title, description and proxied image.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
- When a sender's key cannot be fetched because its server timed out or answered 5xx, the inbox answers 503 with
Retry-After: 300 instead of 401, so Mastodon 4.7 retries rather than switching to RFC 9421 signatures we do not
verify yet. The fetcher's failure cache now remembers whether a failure was temporary.
- Follow, Like, Block, Accept, Reject and Undo ids are deliberately not dereferenceable: serving them would publish
who follows, likes and blocks whom. They are always sent with their object embedded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
tools/pasture/run.sh starts PrivaPub, GoToSocial and Mongo on one podman network behind Caddy's internal CA, and
interop.sh drives both through their own client APIs: follows (one to a locked account), posts, CW, replies, likes,
boosts, DMs, edits, deletes and unfollow. All 25 checks pass, three fresh runs in a row.
- Federation:AcceptAnyCertificate joins the two test-network switches; startup refuses all three in Production.
- WebFinger falls back to http only when AllowPlainHttp is on.
- A bootstrap logger, so a failure before the host is built is no longer silent.
- P4 is ticked in the roadmap, with what has not been run live (Lemmy, a Mastodon circle member).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
Communities:
- a post addressed to a community (to, cc or audience) is accepted
according to its posting policy - followers, anyone, or moderators -
and GroupDistributor announces the whole activity with `audience` to
the community's followers, plus the object for new posts so Mastodon
shows them; updates and deletes of community content are announced too;
- top-level posts are Pages with a name (the title, or a headline from
the text); /flock counts members, /wardens lists moderators;
- a Mastodon client posts into a community by mentioning it, or into a
remote group, which sets `audience`;
- an Announce of an activity from a remote group a persona follows (Lemmy)
is followed through: the object is fetched from its own origin, kept with
its AudienceURI, and fanned out to the group's local followers; updates
are applied in place and deletes checked against the origin.
Circles stop being local-only: an undiscoverable Group actor whose follows
are all requests the owner approves; posts addressed to the circle and its
/flock and delivered to members' own inboxes, never announced, never
public; a remote member's post into the circle is accepted from members
only. SignedFetchAuthorizer serves circle posts and collections only to a
signed request from a member or a member server's instance actor - 404 for
anyone else. Circles never surface in search, lookups, mentions, account
ids or profile pages.
Federation:SecureMode requires a valid signature on every GET under
/peasants except the instance actor. Group forms take a posting policy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
- /api/v1/media and /api/v2/media (and GET/PUT /api/v1/media/:id):
images go through libvips (NetVips, its native build bundled):
autorotated, every kind of metadata dropped (EXIF, GPS, XMP, IPTC,
comments), capped at 4096 px, with a 640 px preview and a blurhash
(own encoder, the reference algorithm); animated GIFs are re-encoded;
video and audio are remuxed by ffmpeg with -map_metadata -1, never
re-encoded, and a video gets a still preview. Files get random names
under /var/lib/privapub/media, outside the web root deploys replace, and
are served at /media/files with nosniff and a sandbox CSP.
- media_ids on create and edit (four at most, the persona's own, each used
once); notes carry them as Document attachments with alt text, blurhash,
focalPoint and size; inbound attachments were already kept.
- avatar and header uploads in update_credentials, cropped to 400x400 and
1500x500, federated with Update{Person}.
- Remote media reaches clients only through /media/proxy/{hmac}/{url},
fetched by the guarded client (no SVG, 40 MB cap) and cached outside the
served root, trimmed to 5 GB; foreign avatars and headers use it too, so
a client never contacts another server.
- MediaJanitor deletes uploads left unattached for a day.
- nginx accepts 100 MB bodies on the upload endpoints only (applied on Max).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
DomainBlock (domain, severity, reject-media, public and private comment)
covers the domain and its subdomains. Admins manage them under
/clientapi/admin/domainblocks/{list,insert,delete}; the set is kept in
memory, reloaded on every change and at most five minutes stale.
A suspended domain is refused by FederationHttp.IsAllowed, so nothing is
fetched from it and no job delivers to it, and the inbox drops its
activities with a 202 before fetching any key. Reject-media strips the
attachments of posts from that domain. Silence is recorded for the
timelines and notifications that arrive in P1.2.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
Infrastructure/Http adds the client the roadmap's S3 and S4 ask for:
- the connect callback resolves the name itself and refuses loopback,
private, link-local, CGNAT, documentation, multicast, ULA, NAT64, 6to4,
Teredo and IPv4-mapped/compatible forms, then connects to the vetted
address, so DNS rebinding cannot swap it afterwards;
- redirects are followed by hand, at most three, each one re-checked;
- bodies are capped at 1 MB after decompression, only JSON media types are
read, every request has a 15 s budget, and a refused URL is not asked
again for five minutes.
Actor and WebFinger fetches and inbox deliveries all use it. Test networks
can switch on Federation:AllowPrivateNetworks/AllowPlainHttp; startup
refuses both in Production.
PrivaPub.Tests (xUnit v3) starts with the address table and the fetcher's
limits against an in-process peer; build.yml and deploy.yml run it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
thepraandClaude Opus 5.5