Meilisearch Indexing Pipeline, Write to Index

Every write in Meilisearch (adding documents, updating settings, swapping indexes) is asynchronous. A POST /indexes/{index_uid}/documents call does not touch the search index inline: it streams the request body to a file on disk, appends one durable task to a queue, and returns a task id. A background scheduler picks the task up later, groups it with compatible neighbors, and runs the milli update pipeline that actually rewrites the on-disk databases. That split is what lets Meilisearch absorb bursts of writes without blocking search, and IndexScheduler is the structure both halves meet at: the HTTP layer holds it as shared state, and the scheduler loop drives it. This page traces one document from the HTTP handler to a committed index, using the module graph generated by Symvanta.

The pipeline has two halves that never share a thread. The front half is cheap and synchronous with the request: validate, persist the payload, enqueue. The back half runs inside the scheduler's own loop: it opens a write transaction against LMDB, extracts field and word data from the changed documents, and commits roaring-bitmap postings. Everything the search side later reads (the ranking rules trace covers that side) is written here.

The moving parts

• document_addition
• register
• tick
• create_next_batch
• autobatch
• apply_index_operation
• index
• extract_all
• write_to_db
• CboRoaringBitmapCodec

document_addition is the actix handler behind both POST (replace) and PUT (update) on the documents route: it validates the content type, streams the body to an update file, and hands off. register is the queue's front door: it turns the request into a Task (a KindWithContent describing the operation) and persists it in enqueued state. tick is the scheduler loop, create_next_batch and autobatch decide how many enqueued tasks to fold into one transaction, and apply_index_operation is the bridge from a scheduler batch into milli. The index function is milli's update entry point; it fans out into extract_all (which turns documents into word and facet deltas) and write_to_db (which drains those deltas into LMDB). The postings themselves are encoded by CboRoaringBitmapCodec, the compressed-bitmap codec that stores which document ids match each term or facet value. Index (the LMDB environment) and FieldsIdsMap (the string-to-u16 field map) are the two on-disk structures both halves of the pipeline share.

How it works

Here is the canonical path for a document write, from the HTTP handler through the enqueued task to a committed index.

1. document_addition is the shared handler for POST /indexes/{index_uid}/documents (replace) and its PUT sibling (update). It checks the content type, then streams the raw request body to an on-disk update file via copy_body_to_file and create_update_file rather than parsing every document inline, so the request thread does almost no work.
2. The register family appends the write to the queue. The handler builds a KindWithContent::DocumentAdditionOrUpdate (index uid, primary key, method, and the update-file uuid) and passes it to register_with_custom_metadata (documents.rs:1633), the sibling that also carries the request's custom metadata; both it and register are thin wrappers over register_with_custom_metadata_and_network, which persists the Task in enqueued state and returns its id immediately. The HTTP response is sent here, long before any indexing happens.
3. tick is the scheduler loop, running on its own thread. On each turn it opens a read transaction, asks create_next_batch for the next unit of work, processes it, and writes the resulting task states back to the queue.
4. create_next_batch picks the next unit of work, and for ordinary document writes it delegates to create_next_batch_unprioritized, which is what runs autobatch over the enqueued tasks. Consecutive document operations on the same index coalesce into a single IndexOperation, so a burst of POST calls is applied in one write transaction instead of one per request.
5. apply_index_operation is reached from process_batch (call site line 183). For a document operation it opens the index write transaction, reads back the current FieldsIdsMap, builds a milli IndexOperations from the batched update files, and turns it into the DocumentChanges the indexer consumes.
6. index is milli's update entry point. It takes those DocumentChanges, spins up the extractor thread pool and a bounded bbqueue channel, and runs extraction and writing in parallel: producers extract, a consumer writes, so a large payload never has to fit in memory at once.
7. extract_all runs the extractors over the changed documents, producing the field distribution, a WordDelta (words added, modified, and deleted), and facet-field deltas, and streams the resulting key-value writes into the channel.
8. write_to_db drains that channel on a dedicated writer thread and puts each buffer straight into its LMDB database. The CboRoaringBitmapCodec encoding already happened on the producer side, in the extractor caches and the merger that filled the channel. post_process then rebuilds the prefix and word-FST databases from the deltas, and the write transaction commits: the document is now searchable.

Where it connects

The scheduler and the query engine share one cluster on the Meilisearch module map: Scheduler Runtime and Query Execution (hub Result, 1413 symbols), which holds tick, register, create_next_batch, and process_batch next to milli's execute_search and bucket_sort. It calls into the Index Storage and Write Pipeline cluster 253 times and into On-disk Codecs and Field Ids 128 times, because every task it runs ends by reading and writing index state through field identifiers. The indexer half lives in Index Storage and Write Pipeline (hub Index, 1369 symbols), where milli's index, extract_all, and write_to_db sit beside the LMDB Index they commit into. Its 447 calls back into the scheduler cluster are the heaviest edge on the whole map, and its 301 calls into On-disk Codecs and Field Ids are the second heaviest: extraction is a loop that resolves document fields to field ids and writes their postings. CboRoaringBitmapCodec itself sits one cluster over, in Query Term Interning and Bitmaps (hub Interned, 591 symbols), alongside the interner and bitmap types the query path reads.

The output of this pipeline is exactly the input to the query side. extract_all writes the word and facet postings; the ranking rules trace shows bucket_sort walking those same postings to order results. The one structure both halves share is Index: the write path commits into it under a write transaction, and every search opens a read transaction against it, which is why the scheduler exposes its own read_txn helper next to the index-level one.

By the numbers

The Scheduler Runtime and Query Execution cluster is 1413 symbols, the second largest of Meilisearch's 61 detected modules, behind the Meilisearch HTTP API cluster at 1728. The milli indexer subtree (crates/milli/src/update/new/indexer) is 15 files and 327 symbols. The two functions where a batched write becomes committed on-disk state are large: apply_index_operation spans lines 45 through 581, and milli's index spans lines 68 through 246. The whole pipeline sits in a graph with zero dependency cycles (modularity Q=0.78), so the write path and the read path stay cleanly separated even though they meet at the same LMDB Index.