Building Queue Systems with Claude Code: Practical Async Processing Guide

When Claude Code helps you build a web app, it is tempting to put every action inside the request handler. A contact form sends email before returning. An upload endpoint resizes images before responding. A billing webhook calls the payment provider, updates the database, sends a receipt, and posts to the CRM in one long path. That works in a demo, but production traffic adds timeouts, duplicate requests, provider rate limits, deploy restarts, and partial failures.

A queue system gives those slow or fragile tasks a safer place to run. The API records the request and publishes a job. A separate worker receives that job, performs the side effect, acknowledges success, retries temporary failures, and sends repeated failures to a dead-letter queue. Claude Code can generate the code quickly, but only if the prompt describes the actual contract: producer, consumer, message payload, visibility timeout, retries, dead-letter queue, idempotency, backpressure, and monitoring.

This guide uses dependency-free Node.js examples, so you can copy the snippets and run them locally without Redis, AWS, or RabbitMQ. In production you may choose SQS, RabbitMQ, BullMQ, or another broker, but the operational rules are the same: do not assume exactly-once delivery, do not hide secrets in payloads, do not retry forever, and make failed work visible.

Queue System Map

A queue is not just a speed trick. It decouples the user-facing request from slow work, protects external services from spikes, gives failed jobs a place to wait, and lets operators see whether the system is healthy.

flowchart LR
  A["Producer<br/>API, cron, webhook"] --> B["Queue<br/>message payload"]
  B --> C["Consumer<br/>worker process"]
  C --> D["External service<br/>mail, image, billing"]
  C -- "retryable failure" --> B
  C -- "poison message" --> E["DLQ<br/>manual review"]
  C --> F["Metrics<br/>logs and alerts"]

Here is the vocabulary in practical terms:

Term	Plain meaning	Design decision
Producer	The code that enqueues work	Payload shape, validation, priority, dedupe key
Consumer	The worker that receives work	Concurrency, timeout, retry behavior
Message payload	The data the worker reads	IDs, type, schema version, no secrets
Visibility timeout	How long one worker owns a job before it can be redelivered	Slightly longer than p95 processing time
Retry	Re-running temporary failures	Max attempts, backoff, jitter, failure reason
DLQ	Dead-letter queue for jobs that should stop retrying	Owner, alert, replay rules
Idempotency	Making a repeated job produce only one business result	Unique key, processed-job table, provider idempotency key
Backpressure	Slowing intake when workers cannot keep up	Concurrency limit, rate limit, queue-depth gate
Monitoring	Evidence that the queue is healthy or stuck	Depth, oldest job age, active count, fail rate, DLQ count

If you include this table in a Claude Code prompt, the generated implementation usually becomes much less toy-like. It also gives reviewers concrete criteria instead of asking whether the queue “looks okay.”

Use Cases

The first common use case is email delivery. Welcome messages, password resets, invoice reminders, and support replies should not block the request path. Store the business event, enqueue a small payload, and let a worker call the provider. For related implementation details, see email automation with Claude Code and the SendGrid email guide. The queue payload should contain a deliveryId, templateId, and userId, not a provider API key or the full email body.

The second use case is image or video processing. Uploads often need thumbnails, WebP conversion, virus scanning, subtitles, or preview clips. These jobs can be CPU-heavy and slow. A queue lets the API return “accepted” quickly while workers process the asset in controlled batches. The main risk is unbounded concurrency: if every upload spawns unlimited image jobs, the server can exhaust CPU, memory, or storage bandwidth.

The third use case is billing retries. Payment providers and card networks can fail temporarily. A retry queue can recover from those failures, but it must be finite. Infinite retries can double-charge customers, hammer a recovering provider, and hide a real schema or permission problem. Billing jobs need idempotency keys, exponential backoff, a DLQ, and a manual review path.

The fourth use case is lead enrichment and report generation. After a form submission, you may enrich company data, write to a CRM, generate a sales report, and notify Slack. None of that has to happen before the user sees the thank-you page. For the broader event model, read event-driven architecture; for visibility, connect it with logging and monitoring; for payload safety, use the guardrails in security best practices.

Example 1: Dependency-Free In-Memory Queue

This first script demonstrates producers, consumers, message payloads, visibility timeout, and backpressure in one file. Save it as queue-basic-demo.mjs and run node queue-basic-demo.mjs. It is intentionally in-memory, so it is not a production broker, but it makes the queue lifecycle easy to inspect.

// queue-basic-demo.mjs
let nextJobId = 1;

class InMemoryQueue {
  constructor({ visibilityTimeoutMs = 800, maxInFlight = 2 } = {}) {
    this.visibilityTimeoutMs = visibilityTimeoutMs;
    this.maxInFlight = maxInFlight;
    this.ready = [];
    this.inFlight = new Map();
  }

  enqueue(type, payload) {
    const job = {
      id: `job-${nextJobId++}`,
      type,
      payload,
      attempts: 0,
      visibleAt: 0,
      lockedBy: null,
    };
    this.ready.push(job);
    return job.id;
  }

  receive(workerId) {
    this.requeueExpired();

    if (this.inFlight.size >= this.maxInFlight) {
      return null;
    }

    const job = this.ready.shift();
    if (!job) return null;

    job.attempts += 1;
    job.lockedBy = workerId;
    job.visibleAt = Date.now() + this.visibilityTimeoutMs;
    this.inFlight.set(job.id, job);

    return {
      id: job.id,
      type: job.type,
      payload: job.payload,
      attempts: job.attempts,
    };
  }

  ack(jobId) {
    this.inFlight.delete(jobId);
  }

  requeueExpired(now = Date.now()) {
    for (const [jobId, job] of this.inFlight.entries()) {
      if (job.visibleAt <= now) {
        this.inFlight.delete(jobId);
        job.lockedBy = null;
        this.ready.push(job);
      }
    }
  }

  stats() {
    this.requeueExpired();
    return {
      ready: this.ready.length,
      inFlight: this.inFlight.size,
    };
  }
}

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

function produce(queue) {
  queue.enqueue("email.send", {
    deliveryId: "mail-1001",
    templateId: "welcome",
    userId: "user-42",
  });
  queue.enqueue("image.resize", {
    assetId: "asset-9001",
    sizes: [320, 768, 1280],
  });
  queue.enqueue("report.generate", {
    reportId: "weekly-2026-06-02",
    accountId: "acct-7",
  });
}

async function consume(queue, workerId) {
  for (let step = 0; step < 8; step += 1) {
    const job = queue.receive(workerId);

    if (!job) {
      console.log(`${workerId}: no job or backpressure`, queue.stats());
      await sleep(120);
      continue;
    }

    console.log(`${workerId}: started ${job.id}`, job.payload);
    await sleep(job.type === "image.resize" ? 300 : 90);
    queue.ack(job.id);
    console.log(`${workerId}: acked ${job.id}`, queue.stats());
  }
}

async function main() {
  const queue = new InMemoryQueue({
    visibilityTimeoutMs: 500,
    maxInFlight: 2,
  });

  produce(queue);
  await Promise.all([consume(queue, "worker-a"), consume(queue, "worker-b")]);
  console.log("final stats", queue.stats());
}

void main();

In a real deployment, the ready array is replaced by a durable service such as SQS, RabbitMQ, or Redis. The important model is the same: a job can be ready, in flight, acknowledged, or returned to the queue after its visibility timeout expires.

Example 2: Idempotency Guard for Workers

Most queues provide at-least-once delivery. That means a job can be delivered more than once. If the worker sends an email twice, charges a card twice, or creates two CRM records, the bug is in the application design, not in the queue service. The following script shows a minimal idempotency guard.

// idempotent-worker-demo.mjs
const idempotencyStore = new Map();
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function withIdempotency(key, work) {
  const current = idempotencyStore.get(key);

  if (current?.status === "done") {
    return { skipped: true, result: current.result };
  }

  if (current?.status === "processing") {
    return { skipped: true, reason: "already processing" };
  }

  idempotencyStore.set(key, { status: "processing" });

  try {
    const result = await work();
    idempotencyStore.set(key, { status: "done", result });
    return { skipped: false, result };
  } catch (error) {
    idempotencyStore.delete(key);
    throw error;
  }
}

async function fakeSendEmail(payload) {
  await sleep(50);
  return {
    providerMessageId: `sg_${payload.deliveryId}`,
    sentToUserId: payload.userId,
  };
}

async function handleEmailJob(job) {
  const key = job.payload.idempotencyKey;
  if (!key) throw new Error("missing idempotencyKey");

  return withIdempotency(key, () => fakeSendEmail(job.payload));
}

async function main() {
  const original = {
    id: "job-1",
    payload: {
      idempotencyKey: "email:welcome:user-42",
      deliveryId: "mail-1001",
      userId: "user-42",
    },
  };

  console.log(await handleEmailJob(original));
  console.log(await handleEmailJob({ ...original, id: "job-1-redelivery" }));
}

void main();

In production, replace the Map with a database unique constraint, Redis SETNX, or a provider-specific idempotency feature. Ask Claude Code to mark work as done only after the external side effect succeeds, release the lock on failure, and avoid placing API keys, access tokens, or full message bodies in the job payload.

Example 3: Retries and Dead-Letter Queue

Retries are useful for temporary network problems. They are not a cure for invalid payloads, deleted users, permission errors, or missing provider configuration. A poison message is a job that will keep failing until a human fixes the cause. If poison messages keep returning to the main queue, workers waste capacity and hide the real incident.

// retry-dlq-demo.mjs
let nextRetryJobId = 1;

class RetryQueue {
  constructor({ maxAttempts = 3 } = {}) {
    this.maxAttempts = maxAttempts;
    this.ready = [];
    this.delayed = [];
    this.dead = [];
    this.completed = [];
  }

  enqueue(payload) {
    this.ready.push({
      id: `retry-job-${nextRetryJobId++}`,
      payload,
      attempts: 0,
      runAt: Date.now(),
      lastError: null,
    });
  }

  moveReadyJobs(now = Date.now()) {
    const stillDelayed = [];
    for (const job of this.delayed) {
      if (job.runAt <= now) {
        this.ready.push(job);
      } else {
        stillDelayed.push(job);
      }
    }
    this.delayed = stillDelayed;
  }

  retryOrDeadLetter(job, error) {
    job.lastError = error.message;

    if (job.attempts >= this.maxAttempts) {
      this.dead.push(job);
      return;
    }

    const delayMs = 50 * 2 ** (job.attempts - 1);
    job.runAt = Date.now() + delayMs;
    this.delayed.push(job);
  }

  async drain(handler) {
    let idleRounds = 0;

    while (this.ready.length > 0 || this.delayed.length > 0) {
      this.moveReadyJobs();
      const job = this.ready.shift();

      if (!job) {
        idleRounds += 1;
        if (idleRounds > 100) throw new Error("drain timeout");
        await sleep(20);
        continue;
      }

      idleRounds = 0;
      job.attempts += 1;

      try {
        const result = await handler(job);
        this.completed.push({ id: job.id, result });
      } catch (error) {
        this.retryOrDeadLetter(job, error);
      }
    }

    return {
      completed: this.completed.length,
      dead: this.dead.length,
    };
  }
}

const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

async function handler(job) {
  if (job.payload.kind === "poison") {
    throw new Error("invalid payload schema");
  }

  if (job.payload.kind === "flaky" && job.attempts < 2) {
    throw new Error("temporary provider timeout");
  }

  return `processed ${job.payload.kind}`;
}

async function main() {
  const queue = new RetryQueue({ maxAttempts: 3 });
  queue.enqueue({ kind: "normal" });
  queue.enqueue({ kind: "flaky" });
  queue.enqueue({ kind: "poison" });

  console.log(await queue.drain(handler));
  console.log(
    "dead letters",
    queue.dead.map((job) => ({
      id: job.id,
      attempts: job.attempts,
      lastError: job.lastError,
      payload: job.payload,
    }))
  );
}

void main();

A DLQ is useful only when someone owns it. Add an alert, capture the failure reason, decide who reviews the message, and document when replay is allowed. Otherwise the DLQ becomes a quieter version of data loss.

Operational Checklist

• Include jobId, type, schemaVersion, business ID, and idempotency key in the payload.
• Do not put API keys, OAuth tokens, card data, long free-form text, or full email bodies in the payload.
• Validate payloads before enqueueing so broken jobs do not enter the system.
• Set visibility timeout slightly above p95 processing time; split jobs that are too long.
• Define max attempts, backoff, jitter, and DLQ rules before production.
• Size worker concurrency from database connections, provider rate limits, CPU, and memory.
• Monitor queue depth, oldest job age, active count, failure rate, DLQ count, and p95 processing time.
• Write a runbook for DLQ review, replay, deletion, and customer communication.
• Assume duplicate delivery for email, billing, points, and CRM writes.
• Ask Claude Code to test failure paths, not only the happy path.

The visibility timeout deserves extra attention. If it is too short, a slow job can be delivered to another worker while the first worker is still running. If it is too long, a crashed worker keeps the job invisible for too long. Measure real processing time, use p95 as a starting point, and break long video or report jobs into smaller steps.

Claude Code Prompt Pattern

Give Claude Code the failure contract, not just the library name. A useful prompt looks like this:

Add an email delivery queue to this repository. The API should save the request, then enqueue only deliveryId and templateId. The worker must use an idempotency key to prevent duplicate sends, retry temporary provider errors up to 3 times with exponential backoff, and move repeated failures to a DLQ table. Do not put API keys, email bodies, or personal data in the queue payload. Expose queue depth, oldest job age, failure rate, and DLQ count through logs or metrics. Add tests for duplicate delivery, poison messages, and visibility timeout behavior.

This kind of prompt gives the agent enough structure to produce reviewable code. It also prevents the common mistake of accepting a short demo that has no retry policy, no idempotency, and no operational evidence.

Production Choices and Official Docs

If your infrastructure is AWS-first, start with the Amazon SQS Developer Guide. Standard queues work well for high throughput; FIFO queues help when ordering and deduplication matter, with different limits and design constraints.

If you need routing patterns, exchanges, pub/sub, and more control over messaging topology, read the RabbitMQ documentation. If your Node.js stack already uses Redis and you want delayed jobs, repeatable jobs, and worker ergonomics, BullMQ documentation is the relevant starting point.

Choose the tool after you define the contract. Payload shape, idempotency storage, retry rules, DLQ ownership, metrics, permissions, and cost matter more than the package name. The dependency-free examples in this article are meant to make those decisions visible before you bind the design to a managed service.

Pitfalls

Duplicate processing is the first trap. Queues usually promise at-least-once delivery, not exactly-once business effects. Network loss after a successful side effect, worker restarts, and visibility timeout expiry can all redeliver a job. Put the idempotency guard where the business effect happens.

Poison messages are the second trap. A malformed payload, deleted user, bad permission, or obsolete schema will not become valid after ten retries. Validate early, record the failure reason, move the job to a DLQ, and provide a controlled replay path after the root cause is fixed.

Infinite retry loops are the third trap. During a provider outage, immediate retry can multiply traffic and delay recovery. Use finite attempts, exponential backoff, jitter, and backpressure on producers when queue depth or oldest job age crosses a threshold.

Secrets in payloads are the fourth trap. Queues are copied into logs, dashboards, DLQs, dumps, and support tooling. Keep payloads as references, then let the worker fetch sensitive data from an authorized store at processing time.

Training and Consulting CTA

Queue systems look simple in code and difficult in production. ClaudeCodeLab can help teams turn this checklist into a repeatable review process: Claude Code prompts, CLAUDE.md rules, payload schemas, DLQ runbooks, monitoring metrics, and CI checks. For team rollout, use Claude Code training and implementation consulting. For solo work, paste the checklist into your pull request template and make every queue change answer those questions.

Summary

A job queue is production infrastructure, not a background convenience. It controls slow work, isolates failure, prevents duplicate business effects, limits concurrency, and gives operators evidence. When you ask Claude Code to implement a queue, specify producers, consumers, payloads, visibility timeout, retries, DLQs, idempotency, backpressure, and monitoring from the first prompt.

Masa’s hands-on result: I ran the three Node.js examples locally without any external service and verified the basic queue flow, duplicate-delivery guard, and poison-message DLQ behavior. The idempotency example was especially useful as a prompt artifact: when the same email job was redelivered, the log showed that the second run reused the stored result instead of sending again.