A lease lasted two minutes and was never renewed, so the reaper gave any longer job to a second worker while the first
still ran it, and both finished it. The worker now renews the lease every third of its length while the handler runs;
a lease found taken (reaped and leased again) cancels the handler. Each lease carries its own owner stamp, since every
worker of a process shares one name, and Finish only counts for the lease it was given. Media processing and persona
archives will run longer than two minutes. JobQueueTests: a job three times its lease runs once with the reaper finding
nothing, and a stolen lease stops its handler and drops its outcome.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
Friendica's activity ids are uniqid(): a short prefix and the microsecond. Two of its processes answering two follows at
once gave both Accepts one id, and PrivaPub, queueing each inbox activity once per id, dropped the second as a copy:
that follow stayed pending on our side while Friendica counted the persona as a follower (seen in the town's Friendica
pair). An id that comes back carrying another type, actor or object is now queued apart; a true copy is still dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LsXgEaXee4GCU1hwYgPJXw
- JobQueue takes an optional scope, so a test's worker leases and reaps only its own jobs.
- The dead-host delivery test runs alone (Exclusive), on its own jobs, and cleans up the
breaker rows it trips; the breaker has tests of its own on unique hosts.
- Index and migration tests run alone: they drop indexes and rewrite every post.
- DomainBlocks.Load replaces reflection and a database-wide block in tests.
- Harness.Outgoing sees only deliveries queued since the harness started: Peer ports are
reused within a run, which made the circle test flaky.
- Two pure-logic tests leave Mongo-gated classes, so CI runs them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ELjqpznMFMNrJoJUj6K5p2
The single serial DeliveryWorker is replaced by Infrastructure/Jobs:
- Job rows are leased with one FindOneAndUpdate (oldest RunAt first, a
two-minute lease) and a reaper returns expired leases every 30 s;
- enqueueing wakes the workers, which otherwise poll every five seconds;
- delivery runs eight at a time with at most two per host, so a slow or
dead server holds two slots, not the queue;
- a failure waits n^4 + 15 + jitter seconds (Mastodon's curve) for up to
16 attempts; a 4xx other than 408/429 is final, a 429 honours
Retry-After;
- RemoteInstance is a per-host circuit breaker: ten consecutive failures
quarantine a host for an hour, doubling to a week, and its jobs wait
without spending attempts;
- a delivery is queued once per activity and inbox (unique DedupeKey), and
finished jobs expire after seven days (TTL on FinishedAt).
Migration _004 moves pending Delivery rows into jobs and marks them
abandoned, so a rollback to the old worker cannot send them twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
thepraandClaude Opus 5.5