The Async Decoupling Pattern for Scalable Batch Processing
Batch processing architecture has a clean pattern that scales elegantly: decouple batch systems asynchronously from everything else. When you get this right, your real-time system stays stable regardless of batch volume, and you never need elaborate job scheduling to avoid infrastructure strain.
Why Async Decoupling Matters
Consider a scenario: you have 100 ML jobs that spin up daily. They all finish around the same time, and each one tries to call back into your real-time system synchronously.
Without proper decoupling, your real-time system gets overwhelmed. The queue backs up. Latency spikes. Maybe something crashes. Those latency spikes matter more than teams realize: Amazon found that every 100ms of latency costs roughly 1% in sales.
One workaround is staggering jobs: run customer A at 7:00 AM, customer B at 7:05, customer C at 7:10. But now you have a scheduling problem layered on top of your architecture. You're treating symptoms, not causes. This mirrors a pattern I see across organizations: operational pain usually traces back to upstream architectural decisions, not the symptoms you're fighting.
The better approach is building proper queuing into your batch architecture from the start. When batch systems are asynchronously decoupled from the rest of your infrastructure, batch workload spikes stay isolated.
What Batch Processing Actually Requires
This pattern assumes your work units are truly independent. If they're not, the first step is refactoring to make them so.
A proper batch system has four components:
- Fan-out at the top. Something triggers the batch run: a cron job, a schedule, or an event. That trigger calls a single service whose only job is creating work payloads. It runs a few for loops, pulls all customers, pulls all job types, generates a payload for each permutation, and shoves them into a queue.
- Independent, consistently sized work units. Each payload should be small and independent. Customer A's daily classification has nothing to do with Customer B's weekly segmentation. They can run in parallel without coordination.
- Elastic consumers. Scale up workers to consume from the queue. They pull work, process it, and when the queue empties, they scale down. The queue absorbs the burst, not your infrastructure. Modern cloud platforms make this straightforward: AWS Lambda with SQS starts at 5 concurrent invocations and can scale up by 300 per minute to a maximum of 1,250 concurrent executions per event source mapping.
- Async decoupling at the bottom. When jobs finish, they don't call back synchronously into your real-time system. They drop results onto another queue. A separate consumer process pulls from that results queue and feeds them into the rest of your system at whatever pace it can handle. This requires idempotent consumers since queue delivery is at-least-once, not exactly-once.
The real-time system never sees a burst. Results pool up, the consumer chews through them, and everything flows smoothly. If results sit in the queue for an extra 10 minutes, who cares? The batch job took 30 minutes anyway. Another 10 is noise. This pattern delivers real results: one retail platform reduced checkout latency by 40% by offloading inventory updates to asynchronous processing.
The For Loops Principle
This is where teams leave performance on the table. The more dimensions of independence your work has, the more parallelism you can achieve.
Say your batch jobs have three dimensions: customer, job type (classification, segmentation, use case analysis), and time period (daily, weekly, monthly). If all three dimensions are independent (and they usually are), you can have three nested for loops:
for each customer:
for each job_type:
for each period:
create_payload(customer, job_type, period)Instead of 10 payloads (one per customer), you might have 10 customers × 3 job types × 3 periods = 90 payloads. Smaller work units. More parallelism. Better queue throughput. And horizontal scaling is linear: doubling queue consumers doubles throughput, making capacity planning predictable.
The key constraint is that the work must be independent. Customer A's daily classification cannot depend on Customer B's weekly segmentation. If there's coupling, you can't nest those loops. Your batch system's parallelism ceiling is determined by how many dimensions of independence exist in your workload.
More independence means more for loops. More for loops means more payloads. More payloads means smaller, faster work units that can scale horizontally.
The Ideal State
When batch processing is done right, your real-time system runs 24/7 with roughly the same capacity. Sensors send the same data every minute. Work coming in is consistent. Work going through is consistent.
Batch is different. It's massive for a block of time, then quiet. You fire everything off at 7:00 AM, the queue fills up instantly, you see message lag as consumers catch up, and then it drains. No staggering required. No elaborate scheduling. No overwhelming your synchronous systems.
The queue absorbs the burst. The async consumer on the output side absorbs the results burst. The real-time system never knows batch ran.
This design pattern aligns with broader architectural thinking about how to structure systems for scale. Thinking about your architecture like a building helps surface these trade-offs clearly: you're deciding whether to build weak foundations and expensive application workarounds, or strong infrastructure that other layers can trust.
Implementation Checklist
When designing or refactoring a batch system, recognize this is infrastructure-level thinking, not feature work. The patterns here affect how your entire organization scales. Understanding where a problem sits in your architecture helps gauge what kind of thinking it requires.
- Map your work dimensions. What are the independent axes? Customers, job types, time periods, regions, tenants? Each independent dimension can become a for loop.
- Verify independence. Can customer A's job run without waiting for customer B's job? Can the daily run happen independent of the weekly run? If yes, they can be separate payloads.
- Add input queuing. The cron trigger should do nothing except create payloads and push them to a queue. Keep this service simple. It's a fan-out mechanism, not a processing engine.
- Scale consumers elastically. Workers should pull from the queue, process one payload, complete, and repeat. Autoscale based on queue depth.
- Make the output async. Results go to a results queue, not to a synchronous endpoint. A separate consumer drains results into the rest of your system at its own pace.
- Measure queue lag, not job timing. Success isn't "all jobs completed by 8:00 AM." Success is "queue drained, results propagated, no downstream impact."
- Monitor queue health. Track queue depth, consumer lag, and message age. Alert on growing backlogs before they become incidents.
The Payoff
A properly decoupled batch system gives you predictable infrastructure costs, eliminates job staggering complexity, enables horizontal scalability without rearchitecture, and isolates the blast radius when something fails.
Your real-time system stays stable. Your batch system scales with your business. And when you add a new customer, you add one more payload to the queue, not a new staggered cron job at 7:47 AM.
Stop staggering jobs. Build queues.
Related Content
