Introducing the Batch API: Simpler and more efficient for large-scale workloads

TL;DR – We’re excited to introduce our Batch API—an asynchronous endpoint built to efficiently handle large volumes of requests. It streamlines workflows by removing the complexities of managing many synchronous requests, such as threading and retries. Compared to OpenAI, our Batch API offers a shorter 12-hour completion window, ideal for overnight runs and maximizing daytime productivity, along with higher throughput: up to 1 GB file sizes, 100K inputs per batch, and 1B tokens per organization. It’s also offered at a 33% cost savings, making it a scalable and cost-effective option.

Not every workload requires real-time processing—for example, offline vectorizing of large corpora for semantic search or running large-scale evaluations. To support these use cases, we’re launching our Batch API, an asynchronous endpoint designed to efficiently process high volumes of requests. Compared to our synchronous API, the Batch API:

Simplifies large-scale workflows by eliminating the need to manage queues, retries, threading, or rate limits. Our Batch API offers a 12-hour completion window—half of market alternatives, including OpenAI. This is great for overnight runs, maximizing daytime productivity.
Offers higher throughput and limits, supporting up to 1 GB file sizes (5x larger than OpenAI’s 200 MB), 100K inputs per batch (2x OpenAI’s 50K limit), and 1B tokens per organization.
Delivers cost savings of 33%, making it a cost-effective option at scale.

How it works

The Batch API introduces a new Files API for managing input and output files. To run a batch job, first create a JSONL file containing your requests and upload it via the Files API. Then, submit the batch job and receive the results within 12 hours. Once the job is complete, you can download the output JSONL file through the same Files API.

# Create a JSONL with your requests:
{"custom_id":"task-1","body":{"input":["Hello World"]}}
{"custom_id":"task-2","body":{"input":["How are you?"]}}

# Create a JSONL with your requests:
{"custom_id":"task-1","body":{"input":["Hello World"]}}
{"custom_id":"task-2","body":{"input":["How are you?"]}}

Getting started

To get started, check out the documentation. The Batch API is fully compatible with the OpenAI Batch API, making it easy to migrate from OpenAI embedding models to Voyage AI. If you’re already using the OpenAI SDK, you only need to update a few lines of code to start using Voyage AI models–here’s an example notebook to show you how!