04 / Architecture

Event-driven job processing

A client should be able to submit a job quickly and find out how it ended, even if a message arrives twice or a worker fails.

This example illustrates how Palash approaches asynchronous work such as report generation. It describes a reference design, not a deployment or measured performance.

Requirements and assumptions

  • Clients can submit work, poll its status and retrieve the result without holding an HTTP connection open.
  • Once accepted, a job must remain recoverable after a queue outage or worker crash. Retries must not produce duplicate visible results.
  • Support bursty, independent jobs in a single region. Assume no cross-job ordering, no hard real-time deadline and no cross-region failover.

Component flow

Normal processing path
  1. Intake API
  2. Job store
  3. Outbox relay
  4. Job queue
  5. Workers
Intake API

Accept the job and return a status URL.

Data flow, step by step

  1. Intake API. The API authenticates and authorizes submissions, checks quotas, and validates the job type and input reference. It returns 202 and a status URL only after the database commits. Reusing an idempotency key within the same tenant returns the existing job for an identical request. A different request with that key is rejected.
  2. Job store. A relational database saves job status and an outbox event in one transaction. The status endpoint reads from that database and checks tenant ownership. The outbox event carries an event ID, a job ID, and a schema version. Inputs and results stay out of the message.
  3. Outbox relay. The relay reads pending outbox rows and marks them sent only after the broker confirms publication. A crash between those steps can lead to a duplicate send. If the broker is down, pending rows stay available for retry.
  4. Job queue. The durable queue delivers at least once and does not guarantee global ordering. Transient failures retry with capped exponential backoff and jitter. When attempts run out, the message goes to a dead-letter queue for inspection. A reconciler checks stalled jobs and records terminal failures so they do not appear to run forever.
  5. Workers. A worker conditionally takes an expiring claim and reads the input through an authorized reference. Only the current claim can commit the final outcome. The worker stores the result durably, records its reference in the job database, and then acknowledges the message. If a finished job is delivered again, the worker acknowledges it without rerunning it.

Clients poll the API for status and can retrieve results they are authorized to see. Workers record outcomes in the job store. Messages that run out of retries leave the normal path for the dead-letter queue.

Operating the system

  • Reliability and recovery

    Long jobs renew their claims; abandoned claims expire. Each attempt writes to its own result key, but clients see only the committed reference. Lifecycle rules remove orphaned objects. External side effects need a stable idempotency key or reconciliation. Operators watch the oldest pending outbox row, queue age, stuck jobs, and dead-letter arrivals. After fixing the failure, they can replay affected jobs using the original IDs and a controlled status transition.

  • Security and isolation

    The API, relay, and workers each have a separate least-privilege identity. Every status or result read checks tenant ownership. Inputs have size and type limits and use server-issued object references. Transport and storage are encrypted, queue messages and logs omit payloads and credentials, and privileged replays and result access are audited.

  • Scaling and backpressure

    Queue age and backlog guide worker scaling, but the database and downstream services set the concurrency ceiling. Submission and concurrency limits keep one tenant from taking over the fleet. If slow jobs regularly delay short ones, separate queues can give each workload its own cap.

  • Cost and tradeoffs

    Compute time, broker operations, database writes, and retained bytes show what each completed job costs. Capped retries, result expiry, and tuned polling intervals keep extra work in check. A database-only queue is simpler at low volume. Adding an outbox and broker costs extra writes and operational work, but separates burst buffering from worker scaling. At-least-once delivery avoids promising exactly-once processing end to end; idempotency and recovery logic handle repeat deliveries.