MCP Vs Rag

A unified research framework for comparing RAG and MCP approaches in question-answering systems

   

______________________________________________________________________

概述

该项目实现了统一的LangGraph工作流 并行RAG和MCP分支 它们同时执行并将结果合并到 单个JSON输出 用于直接比较。

主要特点

建筑

                    User Prompt
                         │
          ┌──────────────┴──────────────┐
          ↓                             ↓
    RAG Branch                     MCP Branch
    (Retrieve → Generate → Eval)   (Search → Generate → Eval)
          ↓                             ↓
          └──────────────┬──────────────┘
                         ↓
                  Merge Results
                         ↓
              Single Unified JSON
              (Contains both RAG and MCP results)

______________________________________________________________________

项目结构

mcp-vs-rag/
├── config/
│   └── docker-compose.yml      # Qdrant configuration
├── src/
│   ├── rag/                    # RAG implementations (Naive & Hybrid)
│   ├── mcp/                    # MCP client for tool integration
│   ├── models/                 # LLM factory (GPT-5, Gemini 2.5 Pro)
│   ├── evaluation/             # RAGAS evaluator
│   └── workflow/               # LangGraph workflow orchestration
├── data/
│   ├── knowledge_base/         # RAG data (chunks, embeddings, originals)
│   └── outputs/                # Experiment results (JSON files)
├── requirements.txt
├── .env.example
├── run_experiment.py           # Main experiment runner
├── README.md                   # This file
└── copilot-instructions.md     # Development guide

______________________________________________________________________

快速开始

1.安装依赖项

pip install -r requirements.txt

2.配置环境

cp .env.example .env

编辑 .env 使用API密钥：

• OPENAI_API_KEY --GPT-5
• GOOGLE_API_KEY --适用于Gemini 2.5 Pro
• TAVILY_API_KEY --Tavily MCP工具

3.启动Qdrant数据库

cd config
docker-compose up -d

4.运行实验

python run_experiment.py

这将执行实验 所有问题 从地面实况数据集中，运行 8种配置 对于每个问题：

命令行选项

结果保存到 data/outputs/ 作为两个主要的JSON文件。

______________________________________________________________________

输出文件

当运行多个问题的实验时，系统会生成 只有两个主文件:

______________________________________________________________________

输出格式

每个实验产生一个 统一JSON文件 包含RAG和MCP结果：

View JSON Structure

{
  "execution_id": "uuid",
  "timestamp": "2025-11-03T10:30:00",
  "configuration": {
    "model": "gpt-5",
    "rag_type": "naive",
    "mcp_server": "tavily"
  },
  "prompt": "What are the latest AI developments?",
  
  "rag_results": {
    "retrieved_context": ["Context from Qdrant..."],
    "generated_answer": "Based on knowledge base...",
    "ragas_metrics": {
      "answer_relevancy": 0.87,
      "faithfulness": 0.92
    }
  },
  
  "mcp_results": {
    "retrieved_context": ["Web search from Tavily..."],
    "generated_answer": "Based on web sources...",
    "ragas_metrics": {
      "answer_relevancy": 0.82,
      "faithfulness": 0.89
    }
  }
}

______________________________________________________________________

配置

实验配置

系统运行中定义的8种组合 run_experiment.py:

View Configurations

CONFIGURATIONS = [
    {"model": "gpt-5", "rag_type": "naive", "mcp_server": "tavily"},
    {"model": "gpt-5", "rag_type": "naive", "mcp_server": "duckduckgo"},
    {"model": "gpt-5", "rag_type": "hybrid", "mcp_server": "tavily"},
    {"model": "gpt-5", "rag_type": "hybrid", "mcp_server": "duckduckgo"},
    {"model": "gemini-2.5-pro", "rag_type": "naive", "mcp_server": "tavily"},
    {"model": "gemini-2.5-pro", "rag_type": "naive", "mcp_server": "duckduckgo"},
    {"model": "gemini-2.5-pro", "rag_type": "hybrid", "mcp_server": "tavily"},
    {"model": "gemini-2.5-pro", "rag_type": "hybrid", "mcp_server": "duckduckgo"},
]

定制

修改 run_experiment.py 致：

• 更改测试提示
• 添加/删除配置
• 调整输出格式
• 修改评估指标

______________________________________________________________________

技术栈

______________________________________________________________________

分析与比较

运行实验后，您可以分析：

______________________________________________________________________

发展

有关详细的开发说明，请参阅 copilot-instructions.md.

代码输入点

from src.workflow.main_workflow import execute_unified_workflow

result = await execute_unified_workflow(
    prompt="Your question here",
    model_name="gpt-5",
    rag_type="naive",
    mcp_server="tavily"
)

运行测试

View Test Example

python -c "
import asyncio
from src.workflow.main_workflow import execute_unified_workflow

async def test():
    result = await execute_unified_workflow(
        prompt='What is quantum computing?',
        model_name='gpt-5',
        rag_type='naive',
        mcp_server='tavily'
    )
    print(result)

asyncio.run(test())
"

______________________________________________________________________

需求

MCP服务器

# Tavily
docker run -i --rm -e TAVILY_API_KEY mcp/tavily

# DuckDuckGo
docker run -i --rm mcp/duckduckgo

______________________________________________________________________

贡献

1. 克隆该仓库
2. 创建要素分支
3. 进行更改
4. 彻底测试
5. 提交拉取请求

______________________________________________________________________

许可证

MIT许可证

______________________________________________________________________

报告Bug · 请求功能