The single serial DeliveryWorker is replaced by Infrastructure/Jobs: - Job rows are leased with one FindOneAndUpdate (oldest RunAt first, a two-minute lease) and a reaper returns expired leases every 30 s; - enqueueing wakes the workers, which otherwise poll every five seconds; - delivery runs eight at a time with at most two per host, so a slow or dead server holds two slots, not the queue; - a failure waits n^4 + 15 + jitter seconds (Mastodon's curve) for up to 16 attempts; a 4xx other than 408/429 is final, a 429 honours Retry-After; - RemoteInstance is a per-host circuit breaker: ten consecutive failures quarantine a host for an hour, doubling to a week, and its jobs wait without spending attempts; - a delivery is queued once per activity and inbox (unique DedupeKey), and finished jobs expire after seven days (TTL on FinishedAt). Migration _004 moves pending Delivery rows into jobs and marks them abandoned, so a rollback to the old worker cannot send them twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012CzABvBkbcFqoHdmi8b9WB
14 lines
447 B
C#
14 lines
447 B
C#
namespace PrivaPub.Infrastructure.Jobs
|
|
{
|
|
public static class Backoff
|
|
{
|
|
public static TimeSpan After(int attempt) =>
|
|
TimeSpan.FromSeconds(Math.Pow(attempt, 4) + 15 + Random.Shared.Next(10) * (attempt + 1));
|
|
|
|
public static TimeSpan HostQuarantine(int consecutiveFailures, int threshold) =>
|
|
consecutiveFailures < threshold
|
|
? TimeSpan.Zero
|
|
: TimeSpan.FromHours(Math.Min(Math.Pow(2, consecutiveFailures - threshold), 24 * 7));
|
|
}
|
|
}
|