Meilisearch Indexing Pipeline: From an HTTP Write to a Committed LMDB Index

Every write in Meilisearch (adding documents, updating settings, swapping indexes) is asynchronous. A POST /indexes/{index_uid}/documents call does not touch the search index inline: it streams the request body to a file on disk, appends one durable task to a queue, and returns a task id. A background scheduler picks the task up later, groups it with compatible neighbors, and runs the milli update pipeline that actually rewrites the on-disk databases. That split is what lets Meilisearch absorb bursts of writes without blocking search, and it is why the IndexScheduler is the single most depended-upon structure in the codebase. This page traces one document from the HTTP handler to a committed index, using the module graph generated by Symvanta.

The pipeline has two halves that never share a thread. The front half is cheap and synchronous with the request: validate, persist the payload, enqueue. The back half runs inside the scheduler's own loop: it opens a write transaction against LMDB, extracts field and word data from the changed documents, and commits roaring-bitmap postings. Everything the search side later reads (the ranking rules deep dive covers that side) is written here.

The moving parts

• document_addition
• register
• tick
• create_next_batch
• autobatch
• apply_index_operation
• index
• extract_all
• write_to_db
• CboRoaringBitmapCodec

document_addition is the actix handler behind both POST (replace) and PUT (update) on the documents route: it validates the content type, streams the body to an update file, and hands off. register is the queue's front door: it turns the request into a Task (a KindWithContent describing the operation) and persists it in enqueued state. tick is the scheduler loop, create_next_batch and autobatch decide how many enqueued tasks to fold into one transaction, and apply_index_operation is the bridge from a scheduler batch into milli. The index function is milli's update entry point; it fans out into extract_all (which turns documents into word and facet deltas) and write_to_db (which drains those deltas into LMDB). The postings themselves are encoded by CboRoaringBitmapCodec, the compressed-bitmap codec that stores which document ids match each term or facet value. Index (the LMDB environment) and FieldsIdsMap (the string-to-u16 field map) are the two on-disk structures both halves of the pipeline share.

How it works

Here is the canonical path for a document write, from the HTTP handler through the enqueued task to a committed index.

1. document_addition is the shared handler for POST /indexes/{index_uid}/documents (replace) and its PUT sibling (update). It checks the content type, then streams the raw request body to an on-disk update file via copy_body_to_file and create_update_file rather than parsing every document inline, so the request thread does almost no work.
2. register appends the write to the queue. The handler builds a KindWithContent::DocumentAdditionOrUpdate (index uid, primary key, method, and the update-file uuid), and register persists it as a Task in enqueued state and returns its id immediately. The HTTP response is sent here, long before any indexing happens.
3. tick is the scheduler loop, running on its own thread. On each turn it opens a read transaction, asks create_next_batch for the next unit of work, processes it, and writes the resulting task states back to the queue.
4. create_next_batch groups compatible enqueued tasks into one Batch using autobatch. Consecutive document operations on the same index coalesce into a single IndexOperation, so a burst of POST calls is applied in one write transaction instead of one per request.
5. apply_index_operation is reached from process_batch (call site line 179). For a document operation it opens the index write transaction, reads back the current FieldsIdsMap, builds a milli IndexOperations from the batched update files, and turns it into the DocumentChanges the indexer consumes.
6. index is milli's update entry point. It takes those DocumentChanges, spins up the extractor thread pool and a bounded bbqueue channel, and runs extraction and writing in parallel: producers extract, a consumer writes, so a large payload never has to fit in memory at once.
7. extract_all runs the extractors over the changed documents, producing the field distribution, a WordDelta (words added, modified, and deleted), and facet-field deltas, and streams the resulting key-value writes into the channel.
8. write_to_db drains that channel on a dedicated writer thread and commits the postings into LMDB, encoding each document set with CboRoaringBitmapCodec. post_process then rebuilds the prefix and word-FST databases from the deltas, and the write transaction commits: the document is now searchable.

Where it connects

The scheduler cluster (Task Management System, hub Result) is the most depended-upon module on the Meilisearch module map: it calls into the request and error plumbing 387 times and into the on-disk FieldId cluster 305 times, because every task it runs ends by reading and writing index state through field identifiers. The indexer half lives in the Indexing and Retrieval cluster (hub FieldId, 1803 symbols), whose CboRoaringBitmapCodec and Index are the on-disk data model the query path later reads from. The heaviest single edge on the whole map, 731 calls from the extraction cluster into FieldId, is this pipeline: extraction is a loop that resolves document fields to field ids and writes their postings.

The output of this pipeline is exactly the input to the query side. extract_all writes the word and facet postings; the ranking rules deep dive shows bucket_sort walking those same postings to order results. The one structure both halves share is Index: the write path commits into it under a write transaction, and every search opens a read transaction against it, which is why the scheduler exposes its own read_txn helper next to the index-level one.

By the numbers

The Task Management System cluster is 1869 symbols, the largest of Meilisearch's 52 detected modules. The milli indexer subtree (crates/milli/src/update/new/indexer) is 15 files and 328 symbols. The two functions where a batched write becomes committed on-disk state are large: apply_index_operation spans lines 43 through 574, and milli's index spans lines 68 through 246. The whole pipeline sits in a graph with zero dependency cycles (modularity Q=0.77), so the write path and the read path stay cleanly separated even though they meet at the same LMDB Index.