Deliveries run on a Mongo job queue with leases, backoff and per-host limits

The single serial DeliveryWorker is replaced by Infrastructure/Jobs:
- Job rows are leased with one FindOneAndUpdate (oldest RunAt first, a
  two-minute lease) and a reaper returns expired leases every 30 s;
- enqueueing wakes the workers, which otherwise poll every five seconds;
- delivery runs eight at a time with at most two per host, so a slow or
  dead server holds two slots, not the queue;
- a failure waits n^4 + 15 + jitter seconds (Mastodon's curve) for up to
  16 attempts; a 4xx other than 408/429 is final, a 429 honours
  Retry-After;
- RemoteInstance is a per-host circuit breaker: ten consecutive failures
  quarantine a host for an hour, doubling to a week, and its jobs wait
  without spending attempts;
- a delivery is queued once per activity and inbox (unique DedupeKey), and
  finished jobs expire after seven days (TTL on FinishedAt).

Migration _004 moves pending Delivery rows into jobs and marks them
abandoned, so a rollback to the old worker cannot send them twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB

PrivaPub/Infrastructure/Data/Indexes.cs

@@ -4,6 +4,7 @@ using MongoDB.Entities;
 
 using PrivaPub.Models.Federation;
 using PrivaPub.Models.Group;
+using PrivaPub.Models.Jobs;
 using PrivaPub.Models.Post;
 using PrivaPub.Models.User;
 
@@ -48,6 +49,15 @@ namespace PrivaPub.Infrastructure.Data
 			await Plain<DmGroup>(token, g => g.ParticipantsKey);
 
 			await Plain<Delivery>(token, d => d.DeliveredAt, d => d.AbandonedAt, d => d.NextAttemptAt);
+
+			await Plain<Job>(token, j => j.Kind, j => j.State, j => j.RunAt);
+			await Plain<Job>(token, j => j.State, j => j.LeasedUntil);
+			await Unique<Job>(j => j.DedupeKey, Builders<Job>.Filter.Type(j => j.DedupeKey, BsonType.String), token);
+			await DB.Default.Index<Job>()
+				.Key(j => j.FinishedAt, KeyType.Ascending)
+				.Option(o => o.ExpireAfter = TimeSpan.FromDays(7))
+				.CreateAsync(token);
+			await Unique<RemoteInstance>(i => i.Host, Builders<RemoteInstance>.Filter.Type(i => i.Host, BsonType.String), token);
 		}
 
 		static async Task Unique<T>(System.Linq.Expressions.Expression<Func<T, object>> key, FilterDefinition<T> partial,