rifflux (Yxbh)

rifflux

Rifflux is a local/offline retrieval engine for markdown and file content with hybrid search:

• lexical search via SQLite FTS5/BM25
• semantic search via local embeddings
• score fusion via Reciprocal Rank Fusion (RRF)
• MCP tool surface for agent workflows

Architecture docs

• Rifflux architecture, indexing flow, and MCP-vs-grep comparison
• Fresh quantitative benchmark: MCP indexed search vs grep-style scan
• Embedding backend decision record (why ONNX, alternatives, trade-offs)

Status

Core retrieval engine with deterministic chunking, incremental indexing, hybrid search (lexical + semantic + RRF), background indexing with retry, file watching, and MCP tool surface.

Quick start

1. Create environment and install:

- python -m venv .venv - .venv\\Scripts\\activate - pip install -e .[dev]

1. Build index with your own script using rifflux.indexing.indexer.Indexer
2. Run MCP server:

- python -m rifflux.mcp.server

Local CLI helpers

• Reindex markdown under current folder:

- python scripts/reindex.py --path . - rifflux-reindex --path .

• Query hybrid search:

- python scripts/query.py "cache ttl" --mode hybrid --top-k 5 - rifflux-query "cache ttl" --mode hybrid --top-k 5

• Rebuild DB after schema changes (delete DB + force reindex):

- python scripts/rebuild.py --path . --db .tmp/rifflux/rifflux.db - rifflux-rebuild --path . --db .tmp/rifflux/rifflux.db

• Benchmark indexing/search with github/awesome-copilot sample corpus:

- python scripts/benchmark_awesome_copilot.py --runs 3 --query-runs 5 - python scripts/benchmark_awesome_copilot.py --refresh-repo --output .tmp/benchmarks/awesome-copilot.json

• Compare benchmark runs (baseline vs current):

- python scripts/compare_benchmarks.py .tmp/benchmarks/baseline.json .tmp/benchmarks/current.json - python scripts/compare_benchmarks.py .tmp/benchmarks/baseline.json .tmp/benchmarks/current.json --max-index-regression-pct 10 --max-search-regression-pct 15

• Generate markdown benchmark report from baseline + current JSON (one command):

- python scripts/generate_benchmark_report.py - python scripts/generate_benchmark_report.py --baseline .tmp/benchmarks/awesome-copilot-smoke.json --current .tmp/benchmarks/awesome-copilot-report.json --output .tmp/benchmarks/awesome-copilot-perf-report.md

• Inspect MCP list_tools metadata (descriptions + schemas):

- python scripts/inspect_mcp_tools.py --pretty - python scripts/inspect_mcp_tools.py --pretty --output .tmp/mcp-tools.json

Embedding backend toggle

Rifflux embedding behavior is controlled by RIFFLUX_* environment variables.

• Canonical prefix: RIFFLUX_*
• RIFFLUX_EMBEDDING_BACKEND=auto|onnx|hash (default auto)
• RIFFLUX_EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
• RIFFLUX_EMBEDDING_DIM=384
• RIFFLUX_DB_PATH=.tmp/riflux/rifflux.db
• RIFFLUX_INDEX_INCLUDE_GLOBS=*.md (comma-separated)
• RIFFLUX_INDEX_EXCLUDE_GLOBS=.git/*,.venv/*,**/__pycache__/*,**/.pytest_cache/*,**/.ruff_cache/*,**/node_modules/* (comma-separated)
• RIFFLUX_AUTO_REINDEX_ON_SEARCH=0|1 (default 0)
• RIFFLUX_AUTO_REINDEX_PATHS=. (comma-separated paths)
• RIFFLUX_AUTO_REINDEX_MIN_INTERVAL_SECONDS=2.0
• RIFFLUX_FILE_WATCHER=0|1 (default 1)
• RIFFLUX_FILE_WATCHER_PATHS= (comma-separated directories to watch; required when watcher is enabled)
• RIFFLUX_FILE_WATCHER_DEBOUNCE_MS=500 (minimum ms between FS event batches)

Environment variables reference

Example configurations

Minimal deterministic local setup (hash backend):

RIFFLUX_EMBEDDING_BACKEND=hash
RIFFLUX_DB_PATH=.tmp/riflux/rifflux.db
RIFFLUX_LOG_LEVEL=INFO

Higher-quality semantic setup (ONNX-focused):

RIFFLUX_EMBEDDING_BACKEND=onnx
RIFFLUX_EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
RIFFLUX_DB_PATH=.tmp/riflux/rifflux.db
RIFFLUX_LOG_LEVEL=INFO

Auto-refresh + watcher setup for active docs workspace:

RIFFLUX_EMBEDDING_BACKEND=auto
RIFFLUX_AUTO_REINDEX_ON_SEARCH=1
RIFFLUX_AUTO_REINDEX_PATHS=docs,notes
RIFFLUX_FILE_WATCHER=1
RIFFLUX_FILE_WATCHER_PATHS=docs,notes
RIFFLUX_FILE_WATCHER_DEBOUNCE_MS=500
RIFFLUX_LOG_LEVEL=DEBUG

Behavior:

• hash: deterministic local hash embedder only
• onnx: ONNX-capable embedder path via optional dependency; falls back to hash if unavailable
• auto: try ONNX path first, then hash fallback
• include/exclude globs are applied during MCP reindex operations
• optional live refresh: when RIFFLUX_AUTO_REINDEX_ON_SEARCH=1, searches trigger incremental reindex over RIFFLUX_AUTO_REINDEX_PATHS (throttled by RIFFLUX_AUTO_REINDEX_MIN_INTERVAL_SECONDS)

Embedding model choice

Default recommendation:

• RIFFLUX_EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
• RIFFLUX_EMBEDDING_BACKEND=auto

Backend selection:

• auto: best general default for mixed/dev environments.
• onnx: use when semantic quality is the priority and setup is controlled.
• hash: use for deterministic CI or minimal-setup workflows.

When changing models:

• Keep RIFFLUX_EMBEDDING_DIM aligned to the model output dimension (default 384).
• Reindex after changing RIFFLUX_EMBEDDING_MODEL to keep index/query vectors consistent.

For deeper rationale and trade-offs, see docs/embedding-backend-decision.md.

File watcher:

• When RIFFLUX_FILE_WATCHER=1 and RIFFLUX_FILE_WATCHER_PATHS is set, Riflux monitors those paths and auto-submits background reindex jobs.
• The watcher uses watchfiles (Rust-backed, cross-platform). Install with pip install -e .[watch] or pip install -e .[dev].
• Only files matching RIFFLUX_INDEX_INCLUDE_GLOBS (and not excluded) trigger reindex jobs.
• The watcher auto-restarts on transient OS errors (up to 5 consecutive crashes with exponential backoff).
• The watcher starts lazily on the first search call, not at server startup.

Schema-change policy:

• On DB schema changes, use a full rebuild of the target DB file (delete/recreate + reindex).
• Incremental reindex is for content changes only, not structural migrations of existing DB files.

Default DB location:

• If RIFFLUX_DB_PATH is not set, Rifflux uses .tmp/rifflux/rifflux.db.
• The .tmp/ folder is git-ignored by default.

Running as an MCP server

Rifflux MCP defaults are environment-variable driven.

• Preferred configuration surface: environment variables (especially when launched by MCP hosts).
• If RIFFLUX_DB_PATH is omitted, DB files are created under .tmp/rifflux/.
• Relative paths are resolved from the MCP server process working directory.
• For local/manual launches, python -m rifflux.mcp.server also accepts repeatable --watch-path args.

Example environment setup:

• RIFFLUX_DB_PATH=.tmp/rifflux/rifflux.db
• RIFFLUX_EMBEDDING_BACKEND=auto
• RIFFLUX_AUTO_REINDEX_ON_SEARCH=0

Tip:

• Use an absolute RIFFLUX_DB_PATH if your MCP host runs from a different working directory than expected.

CLI watcher-path override example (repeatable):

python -m rifflux.mcp.server --db .tmp/rifflux/rifflux.db --watch-path docs --watch-path notes

When one or more --watch-path values are provided, Rifflux enables file watching for those paths for that process.

To enable ONNX-capable backend support:

• pip install -e .[onnx]

Background indexing and resilience

Reindex jobs submitted via background: true or triggered by auto-reindex / file watcher run in a single sequential background thread.

• Retry on transient errors: If a job fails with a transient SQLite error (database is locked, database is busy), it is retried up to 3 times with exponential backoff (1s, 2s, 4s). Non-transient errors fail immediately.
• Graceful shutdown: On process exit (including VS Code killing the MCP server), atexit cleanup stops the file watcher, cancels queued jobs, and waits for any running job to finish.
• Job status: index_status returns all background job details including retries count and crash_restarts from the file watcher.

Layout

• src/rifflux/indexing: markdown chunking + incremental indexing
• src/rifflux/retrieval: lexical, semantic, RRF, orchestrated search
• src/rifflux/db: SQLite schema and storage operations
• src/rifflux/mcp: FastMCP server and tools

MCP reindex tool arguments

reindex supports both a single input location and multiple input locations.

• Single input location (backward compatible):

{
   "path": "./docs",
   "force": false
}

• Multiple input locations:

{
   "paths": ["./docs", "./notes", "./knowledge-base"],
   "force": true,
   "prune_missing": true
}

Behavior notes:

• If paths is provided and non-empty, it is used.
• If paths is omitted, path is used.
• If both are omitted, indexing defaults to the server current working directory.
• Reindex is progressive: unchanged files are skipped, changed files are re-chunked/re-embedded.
• prune_missing (default true) controls whether missing files are pruned (deleted_files).
• Git fingerprint metadata is stored when a scanned path is inside a Git worktree.

MCP tool schema hints

Rifflux MCP tools expose descriptions and argument metadata so clients (including VS Code) can present richer tool guidance.

• search_rifflux

- query: natural-language search query - top_k: integer, minimum 1, maximum 100, default 10 - mode: enum lexical | semantic | hybrid (default hybrid)

• get_chunk

- chunk_id: stable chunk identifier

• get_file

- path: indexed source file path

• index_status

- no arguments

• reindex

- path: optional single source location - paths: optional list of source locations - force: optional boolean rebuild flag - prune_missing: optional boolean stale-file prune toggle (default true) - response includes deleted_files and git_fingerprint

Debug logging

Rifflux emits structured debug logs via Python logging under these loggers:

• rifflux.mcp.tools — tool call entry/exit with wall-clock timing, embedder resolution, schema init
• rifflux.indexing — file scan counts, per-file skip/index decisions, chunk+embed timing
• rifflux.retrieval — lexical/semantic/embed phase durations and hit counts

Set RIFFLUX_LOG_LEVEL to control verbosity:

• RIFFLUX_LOG_LEVEL=DEBUG — full timing and decision traces (recommended for diagnosing slow tool calls)
• RIFFLUX_LOG_LEVEL=INFO — high-level summaries only
• RIFFLUX_LOG_LEVEL=WARNING — default, silent unless something is wrong

Example output at DEBUG level:

10:42:01 rifflux.mcp.tools DEBUG search_rifflux start query='cache ttl' top_k=3 mode=hybrid
10:42:01 rifflux.retrieval DEBUG search phases: lexical=0.001s (2 hits) embed=0.000s semantic=0.003s (5 hits)
10:42:01 rifflux.mcp.tools DEBUG search_rifflux done in 0.005s count=3

For VS Code MCP server usage, add to .vscode/mcp.json env:

"RIFFLUX_LOG_LEVEL": "DEBUG"

For CLI usage:

RIFFLUX_LOG_LEVEL=DEBUG rifflux-query "cache ttl" --mode hybrid

Troubleshooting

• If search or reindex fails with SQL errors like no such column: vec or FTS mismatch errors, rebuild the DB schema and reindex:

- rifflux-rebuild --path . --db .tmp/rifflux/rifflux.db

• If background reindex jobs fail with database is locked, they are automatically retried (up to 3 times). Check index_status for job details and retry counts.
• If the file watcher stops unexpectedly, it auto-restarts with backoff. After 5 consecutive crashes it gives up — check logs at RIFFLUX_LOG_LEVEL=DEBUG.