If your monthly OpenAI invoice is climbing faster than your user base, you are likely falling into the "synchronous trap." Many engineering teams treat Large Language Models (LLMs) like standard REST APIs—expecting an immediate response for every request.
While low latency is non-negotiable for chatbots, it is financially wasteful for background tasks. If you are running classification, sentiment analysis, translation, or synthetic data generation, paying for on-demand compute is an architectural inefficiency.
OpenAI’s Batch API offers a strict 50% discount on token costs in exchange for a flexible completion window (up to 24 hours). This guide provides a production-grade implementation strategy using Python and Node.js to migrate heavy workloads to the Batch API.
The Root Cause: Why Synchronous Requests Command a Premium
To understand the savings, you must understand the infrastructure overhead. When you make a standard chat completion request (POST /v1/chat/completions), you demand immediate GPU availability.
OpenAI must reserve massive compute capacity to handle peak burst traffic without queuing. You are paying a premium for latency assurance and availability.
However, tasks like bulk categorizing 100,000 support tickets do not require millisecond latency. By switching to the Batch API, you allow OpenAI to schedule your workload during periods of lower GPU utilization. In return, they pass the efficiency savings to you—cutting costs by half and significantly increasing your rate limits.
The Architecture: The Asynchronous Batch Lifecycle
Migrating to the Batch API changes your application flow from a synchronous Request/Response cycle to an asynchronous pipeline:
- Serialization: Convert requests into a specific JSONL (JSON Lines) format.
- Upload: Send the file to OpenAI storage.
- Execution: Trigger the batch job.
- Retrieval: Download the results once processing is complete.
We will use Python for the data serialization (due to its dominance in data engineering) and Node.js for the backend service that manages the API lifecycle.
Step 1: Preparing the Data (Python)
The Batch API requires a strictly formatted .jsonl file. Each line must be a JSON object containing a custom_id (essential for mapping results back to your database) and the request body.
Here is a robust Python script to transform a dataset into the required format.
import json
import uuid
# Sample data representing your internal database records
raw_data = [
{"id": "ticket_8821", "text": "I cannot reset my password on the dashboard."},
{"id": "ticket_8822", "text": "The API latency is too high in the EU region."},
{"id": "ticket_8823", "text": "Billing invoice for March is incorrect."}
]
def create_batch_file(data, output_filename="batch_input.jsonl"):
with open(output_filename, 'w') as f:
for entry in data:
# Construct the request object
request_obj = {
"custom_id": entry["id"],
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": "gpt-4o",
"messages": [
{"role": "system", "content": "Classify this ticket: Bug, Feature, or Billing."},
{"role": "user", "content": entry["text"]}
],
"max_tokens": 50
}
}
# Write as a single line JSON string
f.write(json.dumps(request_obj) + '\n')
if __name__ == "__main__":
create_batch_file(raw_data)
print("✅ JSONL file generated successfully.")
Key Technical Detail: The custom_id is arbitrary but critical. When the batch finishes, the output file will be unordered. This ID is the only way to join the LLM output back to your original data source.
Step 2: Uploading and Creating the Batch (Node.js)
Once the file is ready, your backend service needs to upload it and instruct OpenAI to start processing. We use Node.js here, assuming this runs within a typical microservice environment.
Ensure you have the latest SDK: npm install openai
import fs from 'node:fs';
import OpenAI from 'openai';
const openai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY,
});
async function triggerBatchJob(filePath) {
try {
// 1. Upload the file with purpose="batch"
console.log("📤 Uploading file...");
const file = await openai.files.create({
file: fs.createReadStream(filePath),
purpose: "batch",
});
console.log(`✅ File uploaded. ID: ${file.id}`);
// 2. Create the Batch Job
console.log("🚀 Initializing Batch Process...");
const batch = await openai.batches.create({
input_file_id: file.id,
endpoint: "/v1/chat/completions",
completion_window: "24h", // Currently the only supported window
metadata: {
description: "Support ticket classification nightly job"
}
});
console.log(`🎉 Batch created successfully!`);
console.log(`Batch ID: ${batch.id}`);
console.log(`Status: ${batch.status}`);
return batch.id;
} catch (error) {
console.error("❌ Error creating batch:", error);
throw error;
}
}
// Execute
triggerBatchJob("./batch_input.jsonl");
Step 3: Handling Retrieval and Completion
The Batch API is asynchronous. You should not keep a connection open. In a production environment, you should generally rely on polling via a Cron job or a dedicated worker.
Here is a Node.js function to check status and download results.
async function checkAndDownloadBatch(batchId) {
const batch = await openai.batches.retrieve(batchId);
console.log(`Current Status: ${batch.status}`);
if (batch.status === 'completed' && batch.output_file_id) {
console.log("⬇️ Downloading results...");
const fileResponse = await openai.files.content(batch.output_file_id);
const fileContents = await fileResponse.text();
// Process the results
const results = fileContents.split('\n')
.filter(line => line.trim() !== '')
.map(line => JSON.parse(line));
// Example of handling the response
results.forEach(row => {
const ticketId = row.custom_id;
const classification = row.response.body.choices[0].message.content;
console.log(`Ticket ${ticketId}: ${classification}`);
// TODO: Update your database here
});
} else if (batch.status === 'failed') {
console.error("Batch failed:", batch.errors);
} else {
console.log("Batch is still processing. Try again later.");
}
}
Deep Dive: Handling Failures and Edge Cases
Batch processing introduces complexity regarding error handling that doesn't exist in synchronous calls.
The "Double Error" Scenario
There are two layers of failure in a batch process:
- Batch Failure: The entire job fails (e.g., the input file was malformed). The
batch.statuswill befailed. - Request Failure: The batch succeeds, but specific rows fail (e.g., context window exceeded for one specific prompt).
In the Request Failure scenario, the batch status will be completed, but the error_file_id field in the batch object will be populated. You must write logic to check both the output_file_id (for successes) and error_file_id (for failures) to ensure data integrity.
Rate Limits
The Batch API operates on a separate rate limit tier. While standard GPT-4o tiers might limit you to 30,000 TPM (Tokens Per Minute), the Batch API often allows for significantly higher throughput because the load is spread out over time. This makes it ideal for backfilling databases or processing historical logs.
The 24-Hour Window
The completion_window: "24h" is an SLA (Service Level Agreement). In practice, many small batches complete in minutes or hours, but you must architect your system assuming it could take the full 24 hours. Do not use this for features where a user is waiting for a notification.
Financial Impact
For heavy users, the math is compelling.
- Scenario: Processing 1M documents with GPT-4o.
- Avg Tokens: 1,000 input / 200 output per doc.
- Standard Cost: ~$5,000 (approximate blended rate).
- Batch Cost: ~$2,500.
For a startup or enterprise running continuous classification pipelines, this is the difference between a sustainable feature and a money pit.
Conclusion
The OpenAI Batch API is not just a cost-saving measure; it is a pattern for building scalable, resilient AI systems. By decoupling ingestion from inference, you reduce system coupling, avoid rate-limit throttling, and cut your operational expenses by exactly 50%.
Start by identifying your background workers—cron jobs, data pipelines, and nightly reports—and migrate them to the batch workflow today. Your CFO will thank you.