Architecture

Reflex is a trigram-based full-text code search engine written in Rust, with symbol detection via Tree-sitter. This page describes the internal architecture for contributors and users who want to understand how it works.

System overview

CLI / MCP / HTTP API

Query Engine

Trigram

Index

Content

Store

Symbol

Cache + Parser

Cache Manager

meta.db

SQLite (WAL)

trigrams.bin

varint, mmap

content.bin

mmap

Core components

Trigram Indexer

Extracts every three-byte sequence from each file and builds an inverted index: trigram → [(file, line)]. At query time, the trigrams of the search term are intersected to find the few lines that can possibly match, so only those lines are verified.

Content Store

A single memory-mapped file holding the contents of every indexed file, with a fixed-width file table for O(1) lookup by file ID. Memory mapping means the OS handles paging: only the regions a query touches are loaded into RAM.

Symbol Cache and Parser

rfx index extracts imports with Tree-sitter while it builds the trigram index, then spawns a detached background pass that parses every file once and stores its symbols in meta.db. The pass runs one combined Tree-sitter query per language per file and writes the symbol lists zstd-compressed. rfx index status shows its progress.

Symbol queries read from this cache. A file the pass has not reached yet, or one that changed since, is parsed at query time with the same code, so --symbols works before the pass finishes. Cache entries are keyed by content hash, so a changed file is never served stale symbols.

Files with no parser (the text, lock, and generated tiers, and Swift) are indexed for full-text search only.

Cache Manager

Manages the .reflex/ directory:

File	Purpose
meta.db	SQLite database (WAL mode): file metadata and fingerprints, statistics, branch tracking, dependency graph, symbol cache
trigrams.bin	Trigram inverted index (memory-mapped)
content.bin	Full file contents (memory-mapped)
config.toml	Index settings
index.lock	OS advisory lock held for the whole rfx index run
indexing.lock, indexing.status	Lock and progress file for the background symbol pass

meta.db-wal / meta.db-shm (SQLite WAL files) and a trigram_temp/ directory (partial trigram batches on large trees) may also appear while Reflex is running.

Indexing pipeline

1. Walk — traverse the tree with ripgrep’s defaults: respect .gitignore, .ignore, .rgignore and [index] exclude, skip dot-directories and binary files (a NUL byte anywhere). Every remaining file is indexed; code gets a language, everything else joins the text, lock, or generated tier
2. Incremental check — skip files whose content is unchanged since the last run
3. Read, hash, extract — a thread pool reads each file, hashes it with blake3, extracts imports with Tree-sitter, and extracts trigram postings
4. Build — batches of up to 5,000 files or 48 MiB are built into trigram shards in parallel; on large trees partial batches go to trigram_temp/ and are merged by byte copy, so the output is identical whatever the batch boundaries
5. Record — file metadata is written in one transaction; imports are resolved against an in-memory path table and written through a single prepared-statement transaction
6. Write — trigrams.bin and content.bin are written to a .tmp file, synced, and renamed into place, so readers only ever see a complete file
7. Symbols — the detached background pass fills the symbol cache

On a Kubernetes checkout (27,448 files, 245 MB, 16 cores), a from-scratch rfx index takes 7.7 s and the background symbol pass 3.4 s.

Query pipeline

Full-text search

1. Extract trigrams from the pattern
2. Look up each trigram’s posting list in the memory-mapped directory (binary search)
3. Intersect the posting lists with a linear sorted merge, cheapest first; stop early once the next list is far larger than the surviving candidates
4. Verify only the candidate lines, in parallel, against the file contents in content.bin
5. Apply filters (language, file, glob, exclude, lock/generated tiers), group, and sort

In parallel with the search, a separate thread checks freshness: it compares the working tree against the per-file fingerprints (size, mtime_ns, blake3 hash) stored at index time and sets status / can_trust_results on the response.

When a limit is set, verification stops once the page is full, and the total is reported as a sampled estimate (total_is_exact: false, approx_total). Count mode (--count) builds no result objects at all and always reports an exact total.

Symbol search

1. Use the trigram index to find candidate files for the pattern
2. Read their symbols from the symbol cache, parsing only cache misses
3. Match symbol names against the pattern (and --kind, if given)

Regex search

1. Extract literal substrings of 3+ characters from the regex
2. Use the literals for trigram narrowing. Literals under (?i) are looked up under every case variant, so case-insensitive regexes also use the index
3. Verify only the candidate lines the literals name, in parallel
4. If no usable literal exists (for example \w+_id), fall back to scanning every line, with a warning

--timing --json reports which path a query took: index_path is "trigram" or "scan".

Long-running servers

rfx mcp and rfx serve keep the index open across calls (memory maps, path table, query thread pool) and reopen it only when the files change on disk or after re-indexing.

Data formats

All integers are little-endian; posting lists use variable-length integers (varints).

trigrams.bin (format V4)

• Header (32 bytes): RFTG magic, version, trigram count, file count, offset of the paths section
• Directory: one 16-byte entry per trigram (trigram, data offset, compressed size), sorted by trigram and binary-searched directly in the memory map
• Posting lists: one posting per distinct trigram per line, grouped into per-file blocks: varint(file_id delta), varint(line count), then one varint(line delta) per line. No byte offsets are stored
• Paths section: the indexed file paths, validated on open but decoded only on demand

The index is 0.9–1.4x the size of the corpus it covers for code (prose-heavy trees run larger, since nearly every trigram on a prose line is distinct).

content.bin (format V2)

• Header (32 bytes): RFCT magic, version, file count, offset of the file index
• File contents: concatenated
• File index: a fixed-width table of 28-byte entries (content offset, length, path position, path length), read in place at index_offset + file_id × 28, followed by the path blob

Opening the store is O(1): no table is decoded up front.

meta.db

SQLite database in WAL mode (REFLEX_SQLITE_JOURNAL=delete opts out, for network filesystems):

Table	Purpose
files	Path, language, line count, and the freshness fingerprint (size, mtime_ns, hash, dirty_at_index)
statistics	Aggregate stats, writer version, cache schema hash, last compaction time
config	Index configuration snapshot
branches / file_branches	Git-aware tracking: which content hash each file had on each branch
file_dependencies	Imports per file, resolved to indexed files where possible
file_exports	Re-exports (e.g. JavaScript/TypeScript barrel files)
symbols	Background symbol cache, keyed by file and content hash, zstd-compressed

A schema hash computed at build time is stored in statistics, and a mismatch triggers a full rebuild. statistics also records the rfx version that wrote the cache: rfx index refuses a cache written by a different release (1.7.2 or later) unless you pass --force (which deletes .reflex/, config.toml included) or set REFLEX_ALLOW_SCHEMA_REBUILD=1.

Performance optimizations

Memory-mapped I/O

Both trigrams.bin and content.bin are accessed via memmap2. Benefits:

• No explicit loading — the OS pages data on demand
• Shared across processes
• Opening the index is constant time; a query touches only the posting lists and file entries it needs

Compact posting lists

Per-line postings with varint deltas that restart at every file block keep trigrams.bin near the size of the source itself. On a synthetic 30 MB corpus, trigrams.bin is about 30 MB.

blake3 hashing

blake3 fingerprints each file’s content for incremental indexing, freshness checks, and symbol-cache keys. Freshness checks compare size and mtime_ns first and hash only on a mismatch, so a touch costs one hash.

Parallelism

• Indexing uses 80% of available cores, capped at 32
• Query verification runs on a pool sized the same way
• The background symbol pass uses 50% of cores, capped at 32

[performance] parallel_threads and symbol_threads in .reflex/config.toml override these (REFLEX_SYMBOL_THREADS also overrides the symbol pass).

Technology stack

Crate	Purpose
tree-sitter	Parsing for symbol and import extraction, AST queries
memmap2	Memory-mapped file I/O
rusqlite	SQLite for metadata, dependencies, and the symbol cache
blake3	Fast content hashing
zstd	Symbol cache compression
regex	Regex search and verification
memchr	Fast byte scanning
ignore	.gitignore-aware file walking
globset	Glob filters
rayon	Parallel indexing and verification
notify	File watching (rfx watch)
clap	CLI argument parsing
axum	HTTP API server
tokio	Async runtime
reqwest	LLM provider requests (rfx ask, rfx pulse)
ratatui	Interactive TUI
serde_json	JSON output

Design principles

1. Performance first — every design choice prioritizes query speed
2. Completeness over precision — find every occurrence, let the user filter
3. Simplicity over features — do fewer things well
4. Determinism — same query, same results, every time
5. Extensibility — adding a language means adding one parser file

References

• Russ Cox — Regular Expression Matching with a Trigram Index
• Zoekt — trigram code search
• Sourcegraph — code intelligence platform
• ripgrep — regex search tool

Next steps

• Contributing — development setup and code organization
• CLI Commands — full command reference
• Supported Languages — parser details per language