Bulk upsert data

Besides the upsert operation which has 6MB maximum payload size limit, LambdaDB also supports bulk upsert operation to insert or update multiple documents up to 200MB at once.

Bulk upsert is not supported for collections that contain managed embedding vector fields. Use the regular upsert or update flow so LambdaDB can generate embeddings from the configured source text fields.

Recommended: One-step bulk upsert

The easiest way to bulk upsert is to use the SDK’s one-step method. The client uploads your documents to the presigned URL and completes the bulk upsert for you (up to 200MB). Document field names are preserved as supplied in every SDK. The examples use dense_vector and sparse_vector; configure indexes with those same names to search these values.

from lambdadb import LambdaDB

with LambdaDB(
    project_api_key="YOUR_API_KEY",
    base_url="YOUR_BASE_URL",
    project_name="YOUR_PROJECT_NAME",
) as client:
    coll = client.collection("my_collection")
    docs = [
        {"id": "bulk_1", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is the vector lake for AI memory ... ", "dense_vector": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], "sparse_vector": {"LambdaDB": 0.83, "is": 0.1, "a": 0.1, "AI": 0.7}},
        {"id": "bulk_2", "url": "https://en.wikipedia.org/wiki/Winamp", "title": "Winamp", "text": "Winamp is a media player for Windows, macOS and Android ...", "dense_vector": [1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0]},
    ]
    coll.docs.bulk_upsert_docs(docs=docs)

Python: LambdaDB supports context manager usage. __enter__ returns the client, and __exit__ calls client.close() (closing the SDK-owned HTTP client) and makes the client unusable after the with block. If you don’t use with, call client.close() when you’re done. If you pass a custom client=/async_client=, you own closing it.

import { LambdaDBClient } from "@functional-systems/lambdadb";

const client = new LambdaDBClient({
  projectApiKey: "YOUR_API_KEY",
  baseUrl: "YOUR_BASE_URL",
  projectName: "YOUR_PROJECT_NAME",
});
const collection = client.collection("my_collection");
const docs = [
  { id: "bulk_1", url: "https://en.wikipedia.org/wiki/LambdaDB", title: "LambdaDB", text: "LambdaDB is the vector lake for AI memory ... ", dense_vector: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], sparse_vector: { LambdaDB: 0.83, is: 0.1, a: 0.1, AI: 0.7 } },
  { id: "bulk_2", url: "https://en.wikipedia.org/wiki/Winamp", title: "Winamp", text: "Winamp is a media player for Windows, macOS and Android ...", dense_vector: [1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0] },
];
await collection.docs.bulkUpsertDocs({ docs });

package main

import (
  "context"
  "log"
  lambdadb "github.com/lambdadb/go-lambdadb"
)

func main() {
  ctx := context.Background()
  client := lambdadb.New(
    lambdadb.WithBaseURL("YOUR_BASE_URL"),
    lambdadb.WithProjectName("YOUR_PROJECT_NAME"),
    lambdadb.WithAPIKey("YOUR_API_KEY"),
  )
  coll := client.Collection("my_collection")
  docs := []map[string]interface{}{
    {"id": "bulk_1", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is the vector lake for AI memory ... ", "dense_vector": []float64{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}},
    {"id": "bulk_2", "url": "https://en.wikipedia.org/wiki/Winamp", "title": "Winamp", "text": "Winamp is a media player for Windows, macOS and Android ...", "dense_vector": []float64{1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0}},
  }
  _, err := coll.Docs().BulkUpsertDocuments(ctx, lambdadb.UpsertDocsInput{Docs: docs})
  if err != nil {
    log.Fatal(err)
  }
}

After a successful request you’ll receive a message such as "Upsert request is accepted". Documents are processed asynchronously and become searchable after indexing completes.

Two-step: Get presigned URL, then upload and complete

If you need to control the upload yourself (e.g. from a different process or storage), use this flow: (1) get bulk upsert info, (2) upload the payload with every returned signed header, then (3) call the bulk-upsert API with the objectKey. Both REST control calls accept a branch. Use the same branch for the upload-URL request and the completion request; omitting it in both places writes to main. The uploaded object is bound to that branch’s write stream. Do not reuse its key for another branch or after deleting and recreating the original branch. The REST completion request’s type field is optional; the examples use "application/json". The server validates the uploaded object’s Content-Type, not this request field. Setting type does not replace sending the required Content-Type header during upload.

Step 1: Get bulk upsert information

from lambdadb import LambdaDB

with LambdaDB(
    project_api_key="YOUR_API_KEY",
    base_url="YOUR_BASE_URL",
    project_name="YOUR_PROJECT_NAME",
) as client:
    coll = client.collection("my_collection")
    get_bulk_upsert = coll.docs.get_bulk_upsert()
    # get_bulk_upsert.url, get_bulk_upsert.type, get_bulk_upsert.headers,
    # get_bulk_upsert.object_key, get_bulk_upsert.size_limit_bytes

import { LambdaDBClient } from "@functional-systems/lambdadb";

const client = new LambdaDBClient({
  projectApiKey: "YOUR_API_KEY",
  baseUrl: "YOUR_BASE_URL",
  projectName: "YOUR_PROJECT_NAME",
});
const info = await client.collection("my_collection").docs.getBulkUpsert();
// info.url, info.type, info.headers, info.objectKey, info.sizeLimitBytes

info, err := client.Collection("my_collection").Docs().GetBulkUpsertInfo(ctx)
if err != nil {
  log.Fatal(err)
}
// info.URL, info.Type, info.Headers, info.ObjectKey, info.SizeLimitBytes

curl -i -X GET \
  "$BASE_URL/projects/$PROJECT_NAME/collections/{collectionName}/docs/bulk-upsert?branch=candidate" \
  -H "x-api-key: $LAMBDADB_PROJECT_API_KEY"

Response:

{
  "url": "<string>",
  "type": "application/json",
  "httpMethod": "PUT",
  "objectKey": "<string>",
  "sizeLimitBytes": 209715200,
  "headers": {
    "If-None-Match": "*"
  }
}

Step 2: Upload to presigned URL and call bulk-upsert

Upload the document list as JSON to the url. Set Content-Type to the returned type and include every entry in the returned headers map, then call the bulk-upsert API with the objectKey from step 1. Do not drop, rename, or recompute signed headers. The Python tab below is a complete alternative to its Step 1 snippet: it obtains a fresh upload URL, uploads, and completes the import inside one client context. Run it on its own so the client remains open through completion. The TypeScript, Go, and cURL tabs continue from Step 1. For the manual Python upload, also install requests with pip install requests.

import json
import requests
from lambdadb import LambdaDB

docs = [
    {"id": "bulk_33201222", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is the vector lake for AI memory ... ", "dense_vector": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], "sparse_vector": {"LambdaDB": 0.83, "is": 0.1, "a": 0.1, "AI": 0.7}},
]
with LambdaDB(
    project_api_key="YOUR_API_KEY",
    base_url="YOUR_BASE_URL",
    project_name="YOUR_PROJECT_NAME",
) as client:
    coll = client.collection("my_collection")
    get_bulk_upsert = coll.docs.get_bulk_upsert()
    upload_headers = {"Content-Type": get_bulk_upsert.type.value, **get_bulk_upsert.headers}
    # Upload to presigned URL
    resp = requests.put(
        get_bulk_upsert.url,
        data=json.dumps({"docs": docs}),
        headers=upload_headers,
    )
    resp.raise_for_status()
    # Complete bulk upsert while the client is still open
    coll.docs.bulk_upsert(object_key=get_bulk_upsert.object_key)

const docs = [
  { id: "bulk_33201222", url: "https://en.wikipedia.org/wiki/LambdaDB", title: "LambdaDB", text: "LambdaDB is the vector lake for AI memory ... ", dense_vector: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], sparse_vector: { LambdaDB: 0.83, is: 0.1, a: 0.1, AI: 0.7 } },
];
// Upload to presigned URL
const uploadRes = await fetch(info.url, {
  method: "PUT",
  headers: {
    "Content-Type": info.type,
    ...info.headers,
  },
  body: JSON.stringify({ docs }),
});
if (!uploadRes.ok) throw new Error(`Upload failed: ${uploadRes.status}`);
// Complete bulk upsert
await client.collection("my_collection").docs.bulkUpsert({ objectKey: info.objectKey });

// Assume bytes, encoding/json, net/http are imported
docs := []map[string]interface{}{
  {"id": "bulk_33201222", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is the vector lake for AI memory ... ", "dense_vector": []float64{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}},
}
payload, _ := json.Marshal(map[string]interface{}{"docs": docs})
req, _ := http.NewRequestWithContext(ctx, "PUT", info.URL, bytes.NewReader(payload))
req.Header.Set("Content-Type", string(*info.Type))
for name, value := range info.Headers {
  req.Header.Set(name, value)
}
uploadRes, err := http.DefaultClient.Do(req)
if err != nil {
  log.Fatal(err)
}
uploadRes.Body.Close()
if uploadRes.StatusCode < 200 || uploadRes.StatusCode >= 300 {
  log.Fatalf("upload failed: %s", uploadRes.Status)
}
// Complete bulk upsert
_, err = client.Collection("my_collection").Docs().BulkUpsert(ctx, lambdadb.BulkUpsertInput{ObjectKey: info.ObjectKey})
if err != nil {
  log.Fatal(err)
}

# 1. Upload to the presigned URL from step 1
curl --fail -X PUT "<provided_url>" \
  -H 'Content-Type: application/json' \
  -H 'If-None-Match: *' \
  -d '{"docs": [{"id": "bulk_33201222", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is the vector lake for AI memory ... ", "dense_vector": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]}]}' &&

# 2. Complete bulk upsert with the objectKey from step 1
curl -i -X POST \
  "$BASE_URL/projects/$PROJECT_NAME/collections/{collectionName}/docs/bulk-upsert" \
  -H 'content-type: application/json' \
  -H "x-api-key: $LAMBDADB_PROJECT_API_KEY" \
  -d '{
    "objectKey": "<objectKey from step 1>",
    "type": "application/json",
    "branch": "candidate"
  }'

The cURL headers show the current type and headers response values. Always set Content-Type from type and use the exact signed header names and values returned for that upload URL.

Upload retries

The signed upload is create-only (If-None-Match: *). Uploading again to an object that already exists can return 412 from object storage. This is not a LambdaDB JSON error. Acquire a new URL for a new upload attempt rather than blindly repeating the same PUT. If the upload or completion outcome is uncertain, verify the prior outcome before automatically submitting another import. For browser uploads, the target deployment’s CORS configuration must allow the returned signed headers. Use the returned httpMethod and enforce sizeLimitBytes.

Response

After successful bulk upsert initiation you’ll receive:

{
  "message": "Bulk upsert request is accepted"
}

Bulk upsert operations are processed asynchronously in the background. Consequently, newly uploaded documents may not be immediately available for search or fetch requests, even if consistentRead (Python: consistent_read) is set to true. The documents will become available only after the indexing process is fully complete.

Get started

Migrations

Collections

Data versioning

Documents

Search

Manage Costs

Recommended: One-step bulk upsert

Two-step: Get presigned URL, then upload and complete

Step 1: Get bulk upsert information

Step 2: Upload to presigned URL and call bulk-upsert

Upload retries

Response

​Recommended: One-step bulk upsert

​Two-step: Get presigned URL, then upload and complete

​Step 1: Get bulk upsert information

​Step 2: Upload to presigned URL and call bulk-upsert

​Upload retries

​Response

Recommended: One-step bulk upsert

Two-step: Get presigned URL, then upload and complete

Step 1: Get bulk upsert information

Step 2: Upload to presigned URL and call bulk-upsert

Upload retries

Response