Architecture

LambdaDB separates request handling, writes, queries, and index building from durable storage. Gateway, Writer, Query Executor, and Builder provide compute capacity for different parts of the workload. Documents, vectors, and indexes persist independently of those compute resources. Last updated: October 19, 2026 Tap or click to enlarge. Stacked cards represent variable compute capacity. Each compute component can expand or contract independently within its configured capacity limits. Builder jobs run as indexing work becomes available and exit when that work is complete.

​

Components

​

Gateway

Gateway is the regional entry point for API requests. It validates project API keys, enforces access and rate limits, and handles collection and data-versioning management requests. It sends document writes and branch queries or fetches with consistentRead: true to Writer. Other read requests go to the Query Executor’s coordination service.

​

Writer

Writer durably records document changes in a write-ahead log (WAL) before acknowledging them. It also maintains a local view of eligible changes that have not yet been incorporated into a committed snapshot. For consistent reads, Writer coordinates that pending view with the committed data searched by Query Executors. A consistent read can also run before a new collection has its first committed snapshot.

​

Query Executor

Query Executor searches committed snapshots. Its coordination service resolves the requested branch, tag, or alias, distributes work across the relevant internal shards, and merges the results. For consistent reads, Writer coordinates the query and Query Executors search the committed portion. Query Executors read indexes through an S3-native key-value storage engine with memory and local disk caches. Apache Lucene handles full-text search and filtering. Dense vector retrieval uses a dedicated vector index with a structure optimized for S3-backed storage. This index uses the same key-value storage engine as the Lucene indexes.

​

Builder

Builder consumes WAL changes, builds indexes, and publishes immutable snapshots. It advances the branch head after the new snapshot is ready, making those changes available to ordinary reads. Builder capacity is provided by on-demand jobs, separately from request-serving capacity. Builder uses the same storage engine to publish new index roots and compact stored index data. The engine runs within Builder and Query Executor.

​

Durable storage

S3 stores documents, vectors, indexes, and the WAL. The S3-backed WAL implements the OSWALD design. The WAL records accepted changes; the indexes provide efficient search over committed snapshots. Compute-local caches and pending-write indexes can be reconstructed from durable storage. The metadata store uses a PostgreSQL-compatible relational database to hold collection schemas, branch and tag references, aliases, snapshot references, and coordination state. It tracks which data is current and how work is coordinated; documents and vector indexes remain in S3.

​

Storage organization

Each collection branch’s snapshot reference in PostgreSQL points to its current committed collection snapshot in S3. The snapshot records its partitions, each with an index storage root and shard metadata for query execution. The partition’s storage root contains both Lucene index data and the vector index. Tap or click to enlarge. Snapshot metadata and new index objects use the collection branch’s prefix, with partition metadata organized under partition-specific subprefixes. Snapshots can reuse existing index objects through references. The diagram shows logical references, rather than the physical prefix layout. The WAL records pending changes separately; Builder incorporates those changes into new snapshots.

​

Write and indexing flow

1. Gateway authenticates a document write and routes it to Writer for the selected branch.
2. Writer commits the change to the durable WAL before returning a successful response. A successful write does not mean a new snapshot is already available.
3. Builder processes pending changes and writes the resulting indexes and snapshot data to S3.
4. Builder publishes the snapshot through the metadata store and advances the branch head. Ordinary reads can then see the indexed changes.

Branches advance to new snapshots as data is indexed. Tags pin immutable snapshots, and aliases route reads to a named branch or tag.

​

Read consistency

By default, reads use the selected committed snapshot, so newly acknowledged writes may not be visible immediately. For a branch query or fetch with consistentRead: true, Writer combines the committed snapshot with eligible pending changes, including updates and deletes. Consistent reads require a direct branch selector; tags and aliases do not support them. Pending bulk imports remain invisible until Builder commits them to a snapshot. A consistent read can return 429 when its pending payload exceeds the supported limit. See Consistent reads. Query coordination merges and deduplicates candidates before returning results. When requested, native reranking evaluates globally merged candidates before the final size limit. On a partitioned collection, partitionFilter selects the physical partitions to search. Without it, all partitions are in scope. Internal sharding distributes work within that scope. LambdaDB automatically manages shard counts and placement to balance workloads and use compute capacity efficiently.