Bulk upsert data

Besides the upsert operation which has 6MB maximum payload size limit, LambdaDB also supports bulk upsert operation to insert or update multiple documents up to 200MB at once.

Bulk upsert is not supported for collections that contain managed embedding vector fields. Use the regular upsert or update flow so LambdaDB can generate embeddings from the configured source text fields.

Recommended: One-step bulk upsert

The easiest way to bulk upsert is to use the SDK’s one-step method. The client uploads your documents to the presigned URL and completes the bulk upsert for you (up to 200MB).

from lambdadb import LambdaDB

with LambdaDB(
    project_api_key="YOUR_API_KEY",
    base_url="YOUR_BASE_URL",
    project_name="YOUR_PROJECT_NAME",
) as client:
    coll = client.collection("my_collection")
    docs = [
        {"id": "bulk_1", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is an AI-native database ... ", "dense_vector": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], "sparse_vector": {"LambdaDB": 0.83, "is": 0.1, "a": 0.1, "AI": 0.7}},
        {"id": "bulk_2", "url": "https://en.wikipedia.org/wiki/Winamp", "title": "Winamp", "text": "Winamp is a media player for Windows, macOS and Android ...", "dense_vector": [1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0]},
    ]
    coll.docs.bulk_upsert_docs(docs=docs)

Python: LambdaDB supports context manager usage. __enter__ returns the client, and __exit__ calls client.close() (closing the SDK-owned HTTP client) and makes the client unusable after the with block. If you don’t use with, call client.close() when you’re done. If you pass a custom client=/async_client=, you own closing it.

import { LambdaDBClient } from "@functional-systems/lambdadb";

const client = new LambdaDBClient({
  projectApiKey: "YOUR_API_KEY",
  baseUrl: "YOUR_BASE_URL",
  projectName: "YOUR_PROJECT_NAME",
});
const collection = client.collection("my_collection");
const docs = [
  { id: "bulk_1", url: "https://en.wikipedia.org/wiki/LambdaDB", title: "LambdaDB", text: "LambdaDB is an AI-native database ... ", denseVector: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], sparseVector: { LambdaDB: 0.83, is: 0.1, a: 0.1, AI: 0.7 } },
  { id: "bulk_2", url: "https://en.wikipedia.org/wiki/Winamp", title: "Winamp", text: "Winamp is a media player for Windows, macOS and Android ...", denseVector: [1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0] },
];
await collection.docs.bulkUpsertDocs({ docs });

package main

import (
  "context"
  "log"
  lambdadb "github.com/lambdadb/go-lambdadb"
)

func main() {
  ctx := context.Background()
  client := lambdadb.New(
    lambdadb.WithBaseURL("YOUR_BASE_URL"),
    lambdadb.WithProjectName("YOUR_PROJECT_NAME"),
    lambdadb.WithAPIKey("YOUR_API_KEY"),
  )
  coll := client.Collection("my_collection")
  docs := []map[string]interface{}{
    {"id": "bulk_1", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is an AI-native database ... ", "dense_vector": []float64{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}},
    {"id": "bulk_2", "url": "https://en.wikipedia.org/wiki/Winamp", "title": "Winamp", "text": "Winamp is a media player for Windows, macOS and Android ...", "dense_vector": []float64{1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0}},
  }
  _, err := coll.Docs().BulkUpsertDocuments(ctx, docs)
  if err != nil {
    log.Fatal(err)
  }
}

After a successful request you’ll receive a message such as "Upsert request is accepted". Documents are processed asynchronously and become searchable after indexing completes.

Two-step: Get presigned URL, then upload and complete

If you need to control the upload yourself (e.g. from a different process or storage), use this flow: (1) get bulk upsert info (presigned URL and objectKey), (2) upload the payload to the presigned URL, then (3) call the bulk-upsert API with the objectKey.

Step 1: Get bulk upsert information

from lambdadb import LambdaDB

with LambdaDB(
    project_api_key="YOUR_API_KEY",
    base_url="YOUR_BASE_URL",
    project_name="YOUR_PROJECT_NAME",
) as client:
    coll = client.collection("my_collection")
    get_bulk_upsert = coll.docs.get_bulk_upsert()
    # get_bulk_upsert.url, get_bulk_upsert.object_key, get_bulk_upsert.size_limit_bytes

import { LambdaDBClient } from "@functional-systems/lambdadb";

const client = new LambdaDBClient({
  projectApiKey: "YOUR_API_KEY",
  baseUrl: "YOUR_BASE_URL",
  projectName: "YOUR_PROJECT_NAME",
});
const info = await client.collection("my_collection").docs.getBulkUpsert();
// info.url, info.objectKey, info.sizeLimitBytes

info, err := client.Collection("my_collection").Docs().GetBulkUpsertInfo(ctx)
if err != nil {
  log.Fatal(err)
}
// info.URL, info.ObjectKey, info.SizeLimitBytes

curl -i -X GET \
  "$BASE_URL/projects/$PROJECT_NAME/collections/{collectionName}/docs/bulk-upsert" \
  -H 'x-api-key: $LAMBDADB_PROJECT_API_KEY'

Response:

{
  "url": "<string>",
  "type": "application/json",
  "httpMethod": "PUT",
  "objectKey": "<string>",
  "sizeLimitBytes": 209715200
}

Step 2: Upload to presigned URL and call bulk-upsert

Upload the document list as JSON to the url (PUT), then call the bulk-upsert API with the objectKey from step 1.

import json
import requests

docs = [
    {"id": "bulk_33201222", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is an AI-native database ... ", "dense_vector": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], "sparse_vector": {"LambdaDB": 0.83, "is": 0.1, "a": 0.1, "AI": 0.7}},
]
# Upload to presigned URL
resp = requests.put(
    get_bulk_upsert.url,
    data=json.dumps({"docs": docs}),
    headers={"Content-Type": "application/json"},
)
resp.raise_for_status()
# Complete bulk upsert
coll.docs.bulk_upsert(object_key=get_bulk_upsert.object_key)

const docs = [
  { id: "bulk_33201222", url: "https://en.wikipedia.org/wiki/LambdaDB", title: "LambdaDB", text: "LambdaDB is an AI-native database ... ", denseVector: [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], sparseVector: { LambdaDB: 0.83, is: 0.1, a: 0.1, AI: 0.7 } },
];
// Upload to presigned URL
const uploadRes = await fetch(info.url, {
  method: "PUT",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ docs }),
});
if (!uploadRes.ok) throw new Error(`Upload failed: ${uploadRes.status}`);
// Complete bulk upsert
await client.collection("my_collection").docs.bulkUpsert({ objectKey: info.objectKey });

// Assume bytes, encoding/json, net/http are imported
docs := []map[string]interface{}{
  {"id": "bulk_33201222", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is an AI-native database ... ", "dense_vector": []float64{0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}},
}
payload, _ := json.Marshal(map[string]interface{}{"docs": docs})
req, _ := http.NewRequestWithContext(ctx, "PUT", info.URL, bytes.NewReader(payload))
req.Header.Set("Content-Type", "application/json")
uploadRes, err := http.DefaultClient.Do(req)
if err != nil {
  log.Fatal(err)
}
uploadRes.Body.Close()
if uploadRes.StatusCode < 200 || uploadRes.StatusCode >= 300 {
  log.Fatalf("upload failed: %s", uploadRes.Status)
}
// Complete bulk upsert
_, err = client.Collection("my_collection").Docs().BulkUpsert(ctx, lambdadb.BulkUpsertDocsInput{ObjectKey: info.ObjectKey})
if err != nil {
  log.Fatal(err)
}

# 1. Upload to the presigned URL from step 1
curl -X PUT "<provided_url>" \
  -H 'Content-Type: application/json' \
  -d '{"docs": [{"id": "bulk_33201222", "url": "https://en.wikipedia.org/wiki/LambdaDB", "title": "LambdaDB", "text": "LambdaDB is an AI-native database ... ", "dense_vector": [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]}]}'

# 2. Complete bulk upsert with the objectKey from step 1
curl -i -X POST \
  "$BASE_URL/projects/$PROJECT_NAME/collections/{collectionName}/docs/bulk-upsert" \
  -H 'content-type: application/json' \
  -H 'x-api-key: $LAMBDADB_PROJECT_API_KEY' \
  -d '{"objectKey": "<objectKey from step 1>"}'

Response

After successful bulk upsert initiation you’ll receive:

{
  "message": "Bulk upsert request is accepted"
}

Bulk upsert operations are processed asynchronously in the background. Consequently, newly uploaded documents may not be immediately available for search or fetch requests, even if consistentRead (Python: consistent_read) is set to true. The documents will become available only after the indexing process is fully complete.

Get started

Migrations

Collections

Documents

Search

Manage Costs

Recommended: One-step bulk upsert

Two-step: Get presigned URL, then upload and complete

Step 1: Get bulk upsert information

Step 2: Upload to presigned URL and call bulk-upsert

Response

​Recommended: One-step bulk upsert

​Two-step: Get presigned URL, then upload and complete

​Step 1: Get bulk upsert information

​Step 2: Upload to presigned URL and call bulk-upsert

​Response

Recommended: One-step bulk upsert

Two-step: Get presigned URL, then upload and complete

Step 1: Get bulk upsert information

Step 2: Upload to presigned URL and call bulk-upsert

Response