# PrivaPub under load

How PrivaPub holds up when the fediverse sends it a lot at once, measured in the pasture with `tools/pasture/load.sh`
against the `flood` peer, and what each run changed. Numbers are from the workstation (32 cores, PrivaPub and Mongo
8 in podman), so they compare runs with each other; they do not promise a production figure.

## The method

- **The crowd:** `flood` (`tools/pasture/flood/flood.cs`, a .NET file-based app) answers as twenty servers,
  `flood1..20.test`, with ten actors each: their actor documents and keys, WebFinger, NodeInfo, and inboxes that accept
  any Follow with a signed Accept. `flood run` sends signed activities from random actors to PrivaPub's shared inbox at
  a set rate: `Create`s of public notes that mention a persona, `Like`s of the personas' posts, `Follow`s of them.
- **The audience:** `load.sh` makes the personas `load0..N` under the pasture root; each follows its own share of the
  crowd (so a note reaches some homes and not others) and posts twice for the crowd to like.
- **What is measured:**
  - the crowd's view: what PrivaPub answered (202, or 429 when an origin is over its limit) and how fast;
  - for every activity PrivaPub took, how long it waited in the job queue and how long processing took (its
    `InteractionEvent`s);
  - the queue's peak of due jobs, and how long it took to drain once the crowd stopped;
  - a persona's home timeline (`/api/v1/timelines/home?limit=40`), read four times a second throughout.
- Each run is printed and kept in `tools/pasture/out/load/`.

`tools/pasture/load.sh --rate=300 --seconds=60 --personas=20 --follows=50` is the reference run below. PrivaPub lets
each sending origin burst 300 deliveries and earn back 5 a second (`RateLimits:Inbox*`), so twenty origins at 300 a
second leave about 200 a second through and are answered 429 for the rest, by design.

## Runs (2026-10-05)

| Run | Posts held | Queue wait p50 / p95 | Processing p50 / p95 | Queue peak, drain | Home p95 |
|---|---|---|---|---|---|
| 100/s, 2 workers | 17 000 | 12 / 17 ms | 11 / 15 ms | 4, 0 s | 18 ms |
| 300/s, 2 workers | 23 000 | 29 / 47 s | 17 / 23 ms | 5 104, 49 s | 27 ms |
| 300/s, 8 workers | 29 000 | 38 / 60 s | 81 / 123 ms | 5 786, 63 s | 119 ms |
| 300/s, 8 workers, unique indexes fixed | 45 000 | 6 / 9 ms | 6 / 8 ms | 0, 0 s | 14 ms |
| 600/s, origin limits lifted | 70 000 | 7 / 80 ms | 6 / 9 ms | 0, 0 s | 17 ms |
| 1000/s, origin limits lifted | 110 000 | 3.1 / 4.1 s | 7 / 11 ms | 3 933, 6 s | 18 ms |

What the runs found:
- **Two inbox workers** kept up with about 110 activities a second. `Federation:InboxConcurrency` and
  `Federation:DeliveryConcurrency` (default 8) now set them.
- **More workers made it worse**, because every lookup by a unique key scanned its whole collection: those indexes
  were partial on `$type: "string"`, which MongoDB never uses for an equality lookup. Every post by `ObjectURI` (each
  arriving `Create` checks for a duplicate), every actor by `ActorURI`, deleted objects, domain blocks, remote
  instances and the rest. They are partial on `$gt: ""` now (`Indexes.Unique`), which an equality on a string implies,
  and MongoDB.Entities rebuilt them in place at the next start.
- The indexes the plan listed as missing were added on the way (a post's boosts and replies, a persona's boosts, who
  follows an actor, a timeline's rows by author, a post's likes and pins).
- With both fixed, eight workers process about 900 activities a second here, each in under 10 ms, and the home
  timeline does not notice. Above that the queue grows and drains as soon as the burst ends.