Rag Multimodal

具有代理工具调用的多模式RAG

使用以下命令从摄入的文档中检索相关信息 基于多模态嵌入的语义搜索 （带文本和图像的PDF）和工具 模型上下文协议（MCP）两阶段检索：使用单向量嵌入进行ANN搜索，然后使用多向量嵌入进行重新排序。

Agentic Tool System (MCP + Internal Services Integration)

该系统使用统一的工具注册表，该注册表结合了：

1. 外部MCP工具：来自外部MCP服务器的工具（stdio通信）请参阅MCP_server_docs/

- 通过包裹 MCPToolAdapter 匹配内部工具界面 - 示例：网络搜索、URL获取（来自 mcp_server_docs/)

1. 内部服务工具：在后端直接调用函数

- 实施 BaseTool 直接接口 - 例子： retrieve_documents （语义文档搜索）

这两种工具类型都在中注册 ToolRegistry 并且看起来与LLM完全相同 AgentOrchestrator.

目录

.
├── backend/                      
│   ├── api/                          
│   │   ├── routes/                   
│   │   └── schemas/                  
│   ├── core/                         
│   │   ├── config.py                 
│   │   ├── exceptions.py             
│   │   └── startup.py                
│   ├── domain/                       # Business logic
│   │   ├── agentic/                  # Agent orchestration, mcp client, tools
│   │   ├── rag/                      # Chunking, embedding, retrieval
│   │   └── evaluation/               # Retrieval evaluation
│   ├── services/                     # Orchestrates domain logic
│   ├── storage/                      
│   │   ├── document_sql_store.py     # PostgreSQL (documents & chunks metadata)
│   │   ├── single_vector_store.py    # ChromaDB (single-vector embeddings)
│   │   ├── multi_vector_store.py     # Mmap numpy (multi-vector embeddings)
│   │   └── file_store.py             # Filesystem
│   └── data/                        
│       ├── documents/                # Uploaded PDFs
│       ├── chunks/                   # Uploaded PDFs chunked by page
│       ├── single_vector_db/         
│       └── multi_vector_db/          
├── frontend/                        # React frontend (Vite)
├── mcp_server_docs/                 # External MCP server (stdio)
├── docker-compose.yml               # Currently just Postgres Service
└── Makefile                        

快速开始

后端：

# Backend
# Runs on `http://localhost:8000` 
# Auto-starts MCP server as it's set to stdio for transport
make dev-back

# Frontend - WIP
# Runs on `http://localhost:3000`
make dev-front

应用程序编程接口

后端运行后，API文档可在以下位置获得：

• Swagger用户界面: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Tech Stack

后端

前端

Query Flow

当用户发送问题时。..

Inspect Data (DEV)

PostgreSQL检查

brew install --cask dbeaver-community      
# If permission errors, fix permissions once, then retry 
sudo chown -R $(whoami) /usr/local/var/homebrew

______________________________________________________________________

用于单向量和多向量存储的Python REPL

cd backend && uv run python

import chromadb
from pathlib import Path
import pickle
import numpy as np
import json
from pprint import pprint

# ---------------- ChromaDB ----------------

client = chromadb.PersistentClient(
    path=str(Path("data/single_vector_db"))
)

collection = client.get_collection("embeddings")
print(f"\nCollection: {collection.name}. Total vectors: {collection.count()}\n")

# Get sample data (must explicitly include embeddings)
sample = collection.get(
    limit=1,
    include=['embeddings', 'metadatas', 'documents']
    # Note: ChromaDB's get() doesn't return embeddings/metadatas/documents by default
    # (for performance - embeddings can be large). Must explicitly include them.
)

# Investigate metadata

print(f"\nSample ID: {sample.get('ids')[0]}\n")
print("Metadata:")
pprint(sample.get('metadatas', []), indent=4, width=100)

# Investigate embedding
# Embedding first 5 dimensions

embeddings = sample.get('embeddings')
print(f"  Shape: {len(embeddings[0])} dimensions")
print(f"  First 5 values: {embeddings[0][:5]}")

# Query random vector
query_vec = embeddings[0]  
results = collection.query(
    query_embeddings=[query_vec],  
    n_results=2,
    include=['embeddings', 'metadatas', 'documents', 'distances']  
)

# Query results
pprint(results, indent=2, width=100)

# ---------------- mmap files ----------------

index_path = Path("data/multi_vector_db/multi_vector_index.pkl")
with open(index_path, "rb") as f:
    index = pickle.load(f)

print(f"\nTotal chunks: {len(index)}\n")
print("\nFirst 2 chunk IDs:", list(index.keys())[:2])
print()

# First chunk details

first_id = list(index.keys())[0]
emb = index[first_id]
print(f"\nChunk {first_id}:")

vec = np.array(emb)
print(f"\nVector shape: {vec.shape}, dtype: {vec.dtype}")

References

• 主控程序https://modelcontextprotocol.io/docs/develop/build-server#python
• 在claude桌面中启用mcp工具：

  {
    "mcpServers": {
      "documentation": {
        "command": "/Users/jamessukanto/.local/bin/uv",
        "args": [
          "--directory",
          "/Users/jamessukanto/Desktop/codes/exps/mcp/mcp_server_docs",
          "run",
          "main.py"
        ]
      }
    }
  }