ADR-002 — Postgres as the source of truth for jobs#

Context#

The async job service needs a durable record of every job's lifecycle state, plus a fast way for workers to discover new work. Two options were on the table:

1. Redis-only queue (BRPOPLPUSH-style) where the queue itself is the system of record.
2. Postgres as the authoritative store, with Redis used only to signal "there is work to do."

Workers can crash, leases can expire, and the queue can drop messages under failure. We need a model where none of those events can lose or corrupt job truth.

Decision#

PostgreSQL is the authoritative source of job truth. Redis is transport only.

Every job's lifecycle state (queued, processing, done, failed, enqueue_failed), ownership (owned_by, claimed_at), fencing token (lease_version), retry counter, result, and error live in Postgres. Redis carries only the job id as a wake-up signal.

The API writes the job row to Postgres first and commits, then enqueues the id into Redis. If the Redis enqueue fails, the row is parked in enqueue_failed — durable, recoverable, never lost.

Workers treat Redis as a hint. They claim work by atomically transitioning queued → processing in Postgres and bumping lease_version. Final results are accepted only when the worker's in-memory (lease_version, owned_by, status=processing) still matches the DB.

Consequences#

Positive

• Redis loss does not lose jobs. The DB row survives; the reaper re-enqueues anything stuck in processing.
• Duplicate queue delivery is safe by construction — the second claim attempt fails the atomic status transition.
• Stale workers cannot overwrite newer truth. The fencing predicate rejects any write whose lease_version no longer matches.
• Recovery is a first-class, observable operation (reaper sweeps, lease_expired_total, retry_distribution) rather than opaque broker behavior.

Negative

• Every claim and every final write costs a Postgres round-trip. Throughput is bounded by DB write capacity, not Redis throughput.
• Two systems must stay roughly in sync. The enqueue_failed state exists specifically to handle the window where the DB commit succeeds but the Redis enqueue does not.
• A pure Redis design would be simpler to operate; we accept the extra moving part because it removes a whole class of truth-loss failures.

ADR-003 — Reaper cadence vs lease TTL#

Context#

A job can sit in processing for two reasons: a worker is legitimately running it, or a worker crashed mid-execution. The reaper sweeps Postgres for jobs whose claimed_at exceeds a timeout, bumps lease_version, clears ownership, and re-enqueues them.

Two knobs control how long recovery takes:

• Processing timeout — how long a job may stay in processing before the reaper considers it abandoned.
• Reaper interval — how often the reaper sweeps.

The external SLO is: 95% of jobs reach a terminal state (done or failed) within 25 seconds. We need values that respect that budget while not falsely reaping jobs that are still healthy.

Decision#

• Processing timeout: 15 seconds
• Reaper interval: 5 seconds

Recovery requires a Postgres row lock, a status flip back to queued, a lease_version bump, an ownership clear, and a Redis re-enqueue after the DB commit.

Consequences#

Worst-case modeled recovery path

worker compute        5s
timeout detection    15s
reaper interval       5s
                  ------
total upper bound    25s

This is the budget the 25s external SLO is derived from. A single worker crash on a single job still lands inside the SLO window.

Positive

• Recovery latency is bounded and predictable, not an emergent broker property.
• The reaper interval is short enough that the timeout dominates — adding workers does not shift the recovery profile.
• Fencing (ADR-002) makes the reaper safe even if it acts on a job that was about to finish: the late write is rejected.

Negative

• A 15s timeout means a healthy job that legitimately takes longer than 15s will be reaped and rerun. Today's compute is bounded well below this; once real workloads exceed 10s p99, this ADR needs to be revisited (longer timeout, or a heartbeat-extension path).
• A 5s reaper interval is a fixed background load on Postgres regardless of queue depth.

Job Lifecycle Specification v1#

Purpose#

Define authoritative lifecycle rules for async job execution so API, worker, and recovery logic share the same state model.

Source of Truth#

PostgreSQL is the authoritative source of job truth. Redis only transports work signals.

Job States#

queued#

Job exists durably in PostgreSQL and has been accepted for worker pickup.

processing#

A worker has atomically claimed ownership of the job and holds an active lease.

done#

Worker completed execution and persisted final result successfully.

failed#

Execution ended without meeting success condition, or retry budget exhausted.

enqueue_failed#

Job row exists but Redis enqueue did not succeed.

Required Columns#

• id
• status
• result
• error
• created_at
• updated_at
• claimed_at
• owned_by
• lease_version
• retry_count

Ownership Rule#

A worker may begin execution only after atomically changing status from queued to processing.

Lease Rule#

A processing job is considered abandoned if claimed_at exceeds lease duration.

Retry Rule#

Abandoned jobs return to queued with retry_count incremented.

Retry Limit#

If retry_count exceeds maximum allowed retries, state transitions to failed.

Fencing Rule#

Each ownership claim increments lease_version. A worker may persist final result only if lease_version matches current database value.

Idempotency Rule#

Before execution, worker must verify that status is not already done.

Async Job Service v1 — Reliability Contract#

1. Service Purpose#

This service accepts asynchronous jobs through an API, persists them durably in the database, signals them through Redis for worker execution, and guarantees eventual transition into a terminal state (done or failed) under bounded retry and recovery policies.

The database is the durable source of truth. Redis acts only as transport for work discovery.

---

2. Lifecycle States#

The service currently supports the following job states:

• enqueue_failed → Job persisted in database but queue signal failed.
• queued → Job available for worker claim.
• processing → Worker owns valid execution authority.
• done → Job completed successfully and result is durable.
• failed → Retry budget exhausted and terminal failure reached.

Legal Transitions#

• enqueue_failed → queued
• queued → processing
• processing → done
• processing → queued (retry or reaper recovery)
• processing → failed

Terminal states:

• done
• failed

---

3. Authority Model#

Worker authority is enforced using three fields:

• lease_version
• owned_by
• status

A worker may write state only if all three match current database truth.

Authority Predicate#

A write is valid only when:

• lease_version matches
• owned_by matches worker id
• status = processing

This prevents stale workers from writing results after timeout recovery or ownership reassignment.

---

4. Retry Policy#

The service uses bounded retries.

Retry Budget#

• Maximum retries: 3

Retry Flow#

On worker failure:

• retry_count += 1
• error = latest failure
• result = null

If retry budget remains:

• status = queued
• lease_version += 1
• owned_by = null
• claimed_at = null

If retry budget exhausted:

• status = failed
• owned_by = null
• claimed_at = null

Redis enqueue occurs only after DB commit.

---

5. Recovery Model#

A reaper process periodically repairs unfinished work.

Timeout Rules#

• Processing timeout: 15 seconds
• Reaper interval: 5 seconds

Recovery Logic#

If a job remains in processing beyond timeout:

• lock row
• set status = queued
• increment lease_version
• clear ownership
• clear claim timestamp
• commit
• enqueue again

This restores claimability while preserving stale-write safety.

---

6. Latency SLO#

Internal Design Upper Bound#

Worst modeled recovery path:

• worker compute: 5s
• timeout detection: 15s
• reaper interval: 5s

Total upper bound:

• 25 seconds

Initial External SLO#

95% of jobs must reach terminal state within 25 seconds.

Terminal states include:

• done
• failed

---

7. Reliability Interpretation#

Latency SLO and success rate are independent.

A failed job may satisfy latency SLO if terminal state is reached within budget.

Failure still impacts reliability metrics separately.

---

8. Known Current Limitations#

• Redis duplicate delivery tolerated but not suppressed.
• Queue visibility is repaired by reaper, not transactionally guaranteed.
• Historical executor identity is not preserved separately from active ownership.
• Logs are required for full failure chronology.

---

9. Current System Truth#

• PostgreSQL = authority
• Redis = discovery
• Worker = execution
• Reaper = authority repair

Failure Matrix v1#

Failure	Detected By	State Transition
Redis enqueue fails	API	queued → enqueue_failed
Worker crashes during processing	lease recovery	processing → queued
Lease expires beyond retry limit	recovery process	processing → failed
Fence mismatch on final write	worker	stale result discarded
Final DB write fails	lease recovery	processing → queued
Retry limit exceeded	recovery process	queued → failed