Home / Rank #5

Batch APIs

Typical savingsFlat 50% on most providers
EffortLow — if your workload tolerates async

OpenAI, Anthropic, and Google all offer batch endpoints at roughly 50% off in exchange for asynchronous processing (typically completed well within 24 hours, often much faster). Any workload that is not user-facing-realtime — enrichment, classification, embeddings backfills, evals, report generation — is leaving money on the table if it runs through the synchronous API.

How to do it

  1. Audit which jobs are actually latency-sensitive; most pipelines are not.
  2. Move offline jobs to the provider batch endpoint (JSONL in, JSONL out).
  3. Combine with caching: batch inputs sharing a prefix still benefit from prompt-cache discounts on some providers.

Frequently asked questions

How fast do batches complete?

Providers guarantee a 24-hour window but typically finish in minutes to a few hours depending on load. Design for the guarantee, enjoy the typical case.

Tools for this method

Open-source LLM gateway

LiteLLM

The default self-hosted gateway: one OpenAI-compatible proxy across 100+ providers with budgets, caching, routing, fallbacks, and …

Next method: #6 Output length control