Distributed Systems
TaskForge
Coordinate background work through short PostgreSQL transactions, renewable ownership, and a durable record of every attempt.
Case study · Summer 2026
Measured results
Local Docker benchmarks
- speedup
- 15.84×
From 1 to 16 workers on synthetic 50 ms waits
- no-op tasks/sec
- ≈1,284
Median processing throughput at 4 workers; a separate no-op workload
I designed and built TaskForge independently during summer 2026, before the school year began. My work spans the full system: the API and console, PostgreSQL task and attempt model, Go workers and schedulers, observability, and benchmark harness.
- Authority
- PostgreSQL tasks + attempt history
- Execution
- Go workers · one handler per process
- Recovery
- Go schedulers · leases + retries
System architecture
Background work must outlive an HTTP request and remain inspectable when a process fails. TaskForge separates admission, execution, and lifecycle maintenance around one durable database.
PostgreSQL coordinates independent processes
01 / Admission
React console → FastAPI
The console uses a same-origin Nginx API proxy. FastAPI validates submissions and reads task, worker, and attempt history.
Submit / inspect / cancel ↔ PostgreSQL ↓
02 / Authority
PostgreSQL tasks + numbered attempts
Committed rows hold priority, schedule, owner, lease, current state, outcomes, and the history of each execution attempt.
Claim / renew / finalize ↔ workers · Recover / promote ↔ schedulers ↓
03 / Execution & maintenance
Go workers + Go schedulers
Each worker runs one handler at a time, with independent heartbeat and lease-renewal loops. Schedulers recover expired ownership and promote due retries; replicas coordinate through row locks.
Submission & idempotency
Admit a logical task
FastAPI validates task type, payload, queue, priority, total-attempt budget, and optional schedule. A committed task starts QUEUED with no attempts. An optional idempotency key makes uncertain client retries safe for admission.
Arbitrate matching replays
A global partial unique index reserves the key. A SHA-256 fingerprint covers the canonical request, including queue, priority, budget, and normalized schedule. Concurrent inserts resolve to the stored task: matching fingerprints return its current state; different requests with the same key conflict.
Retain admission identity
Idempotency keys remain associated with their stored task records. Replaying a completed task does not create new work.
Competing workers
Workers select due QUEUED tasks with attempts remaining, ordered by descending priority, then creation time and ID. A short transaction uses FOR UPDATE SKIP LOCKED so replicas can claim different rows without a central dispatcher.
Worker A locks task X
It sets RUNNING, assigns its owner and lease, increments the attempt number, and inserts the matching RUNNING attempt. These changes commit together.
Worker B skips X
While X is locked, B can claim another eligible row or return no work. If A rolls back, X stays QUEUED without that attempt and becomes available to a later poll.
Execute after commit
The handler runs outside the claim transaction. Completion locks the task again and checks owner, attempt number, RUNNING state, and valid lease before committing task and attempt outcomes together.
Workers prioritize eligible tasks and skip locked rows so other work can proceed.
Success & retry walkthrough
Successful attempt
QUEUED → RUNNING
Claim commits ownership and attempt N. The worker then invokes its allowlisted handler.
Renew while executing
An independent loop extends task ownership. Process heartbeats report liveness separately.
RUNNING → SUCCEEDED
Guarded completion persists task result and SUCCEEDED attempt output together, then clears ownership.
Retryable failure → success
Attempt N → FAILED · task → RETRYING
A typed retryable error records the failed attempt and schedules the same logical task, if its total-attempt budget remains.
RETRYING → QUEUED
The scheduler promotes it once due. Promotion creates no attempt; a later worker claim creates N + 1.
Next attempt → SUCCEEDED or FAILED
Success finalizes normally. Ordinary errors are terminal; a retryable error at the last allowed attempt makes the task FAILED.
Default backoff starts at 2 seconds, doubles by failed-attempt number, applies 20% jitter, and caps at 300 seconds. The default budget is three total attempts, including the first claim and crash replacements. scheduled_at is earliest eligibility, not an exact start-time promise; promotion, polling, and contention add delay.
Logical task identity remains stable across retries and recovery. Numbered attempts retain RUNNING, SUCCEEDED, FAILED, or ABANDONED history. A failed attempt and a failed logical task are different outcomes.
Lease renewal & recovery
Lease and heartbeat answer different questions
Defaults are a 30-second task lease renewed every 10 seconds, and a process heartbeat every 5 seconds. A heartbeat shows recent communication; it neither proves useful progress nor revokes a task. Ownership comparisons use database time.
Expired ownership triggers recovery
A crash stops renewals but causes no immediate state transition. Schedulers lock expired RUNNING tasks and matching attempts, mark attempts ABANDONED, and requeue within the remaining budget—or mark the logical task FAILED when exhausted. Crash recovery requeues directly rather than applying retry backoff.
Guard completion with ownership
The owner and attempt number reject stale database completion after replacement.
Retries and crash recovery operate within the configured attempt budget; execution can be repeated.
Benchmark methodology & results
Benchmarks compare worker configurations across independently reset trials. Persisted task and attempt counts are reconciled with Prometheus metrics, excluding 100 warmup tasks.
The no-op workload uses 5,000 no-op tasks per trial; the synthetic-wait workload uses 1,000 synthetic 50 ms waits per trial. Both test 1, 4, 8, and 16 worker processes across three independently reset blocks: 12 trials each. The runs contain 60,000 and 12,000 tasks respectively.
Measurements use synthetic workloads in local Docker; throughput depends on handler work and available resources.
Recorded environment: Apple M4 Pro · 12 logical CPUs · 24 GiB RAM · local Docker.
Processing tasks/sec = logical tasks / (latest attempt finish − earliest attempt start)
Charts show medians of per-trial processing throughput. Speedup is the ratio of aggregate medians; parallel efficiency is speedup divided by worker count. Each chart has its own zero-based scale.
No-op coordination
median processing tasks/sec
- 1 worker779.748 tasks/sec
- 4 workers1284.015 tasks/sec
- 8 workers1279.605 tasks/sec
- 16 workers1214.429 tasks/sec
Synthetic 50 ms waits
median processing tasks/sec
- 1 worker18.764 tasks/sec
- 4 workers74.668 tasks/sec
- 8 workers149.442 tasks/sec
- 16 workers297.264 tasks/sec
- Fail-once retries
- 3,000 tasks · 6,000 attempts
Three trials; 10 workers and 3 schedulers; fixed 100 ms retry/promotion configuration. Exactly FAILED → SUCCEEDED histories, with zero duplicate identities or stranded leases in these runs.
- Hard-kill recovery
- 30 abandoned attempts replaced
Three trials of 1,000 synthetic 500 ms waits; 20 workers, 3 schedulers, 10 killed owners per trial, and 5-second leases. Every captured abandoned attempt had exactly one later successful replacement, with zero duplicate recovery-state transitions in these runs.
Median-trial p95 recovery lag was 36.681 ms from lease expiration to the recovery timestamp recorded in the SQL path. Median kill-to-final-drain was 25.271239 seconds.
Design decisions & tradeoffs
Short transactions and renewable ownership
Task state and attempt history must agree without holding locks through a handler. Use short claim/completion transactions and renewable task leases.
- Benefit
- Task and attempt updates commit together while handler work runs outside transactions.
- Accepted limitation
- Database ownership does not deduplicate external effects; handlers need destination idempotency for those effects.
One PostgreSQL authority
Admission, ownership, outcomes, and history share durable state. PostgreSQL is the authority; workers poll it for eligible tasks.
- Benefit
- Database transactions coordinate independent workers and schedulers.
- Accepted limitation
- Polling and shared writes create database pressure as concurrency rises.
Priority with bounded attempts
Urgent work and repeated failures need explicit scheduling limits. Claim by descending priority and cap each task’s total attempts.
- Benefit
- Claim order favors higher-priority eligible tasks; the budget bounds retries and crash replacements.
- Accepted limitation
- Sustained high-priority load can delay lower-priority tasks; the policy does not guarantee fairness.
Repository
Explore the API, Go workers and schedulers, SQL migrations, console, and benchmark harness on GitHub.
By Aansh Singh · Solo project · Summer 2026