Ds Wiki Transformer

形式诊断原理（PFD）

用于研究数据集的图一致性引擎

自动分析研究数据集的结构连贯性、维度和形式基础。

给PFD一个结构化的数据集——代谢网络、分类法、电网、化学数据库、知识图——它会产生：

• 一级报告 --内部结构连贯性、Fisher有效维数（D_eff）、状态分类、交互式网络可视化
• 二级报告 --跨数据集桥梁分析：该数据集与正式参考知识图的锚定程度
• PFD评分 --综合0.0-1.0的质量分数，并进行充分推理（无黑盒判断）

状态：活跃的研究工具。麻省理工学院许可。建筑稳定；已构建3A阶段（索赔提取+信道解析）。

______________________________________________________________________

管道概述

Your Dataset (CSV / JSON / MATPOWER / custom)
    │
[Step 1] INGEST         → entries + links → rrp_*.db (SQLite)
[Step 2] BUILD GRAPH    → NetworkX internal graph
[Step 3] TIER-1         → D_eff, coherence score, regime distribution
    │                      → Tier-1 HTML report + D3.js network viz
    │
[Step 4] BUILD BRIDGE   → RRP nodes ↔ DS Wiki nodes (semantic similarity)
[Step 5] TIER-2         → Bridge quality, anchor distribution, domain coverage
    │                      → Tier-2 HTML report
    │
[Step 6] PFD SCORE      → 0.0–1.0 combined verdict

______________________________________________________________________

已验证的数据集（截至2026年3月14日）

______________________________________________________________________

快速开始

# 1. Clone + setup (creates venv, installs deps, verifies data)
git clone https://github.com/IanD25/ds-wiki-transformer.git
cd ds-wiki-transformer
bash setup.sh

# 2. Activate environment
source .venv/bin/activate        # Mac/Linux
.venv\Scripts\activate           # Windows

# 3. Run a full two-tier PFD report
pfd report --rrp data/rrp/ecoli_core/rrp_ecoli_core.db

# 4. Or use the script directly
python scripts/run_fisher_suite.py --mode internal_rrp \
    --rrp data/rrp/ecoli_core/rrp_ecoli_core.db

# 5. Verify (450 tests)
python -m pytest tests/ -v --tb=short

替代方案：pip安装

git clone https://github.com/IanD25/ds-wiki-transformer.git
cd ds-wiki-transformer
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pfd --help

Windows（ShadowPC）： 在Python命令前加上前缀 PYTHONUTF8=1 处理DS Wiki条目中的Unicode数学符号。

______________________________________________________________________

文档

______________________________________________________________________

关键命令

# ── Using the pfd CLI (after pip install -e .) ─────────────────────
pfd report --rrp data/rrp/ecoli_core/rrp_ecoli_core.db    # Full PFD report
pfd internal --rrp data/rrp/ecoli_core/rrp_ecoli_core.db   # Tier-1 only
pfd bridge --rrp data/rrp/ecoli_core/rrp_ecoli_core.db     # Tier-2 only
pfd wiki                                                     # DS Wiki health check
pfd node --node-id CHEM5                                     # Single node deep-dive
pfd extract "We find that k = 3.11 at 90% confidence"       # Extract claims
pfd resolve "entropy increases with dimension"               # Resolve to DS Wiki

# ── Or using scripts directly ──────────────────────────────────────
python scripts/run_fisher_suite.py --mode report \
    --rrp data/rrp/ecoli_core/rrp_ecoli_core.db \
    --db data/ds_wiki.db

______________________________________________________________________

需求

• Python 3.11+（在3.11、3.12、3.13上测试）
• ~500MB磁盘（所有数据已提交——克隆后无需重建）
• CPU、CUDA（RTX 2000+）或Apple Silicon——自动检测

核心依赖关系 （自动安装）：

sentence-transformers  # BGE embeddings (auto-downloaded from HuggingFace)
chromadb               # semantic vector indexing
numpy, plotly, matplotlib, networkx  # numerics + visualization

可选：

pip install -e ".[mcp]"    # fastmcp — MCP server for Claude tool access
pip install -e ".[power]"  # pandapower — IEEE power grid parsing
pip install -e ".[dev]"    # all of the above + pytest

______________________________________________________________________

项目结构

ds-wiki-transformer/
├── CLAUDE.md                      ← LLM context (auto-loaded)
├── README.md                      ← This file
├── MASTER_SUMMARY.md              ← Full technical reference
├── USER_GUIDE.md                  ← User-facing interpretation guide
├── docs/
│   ├── FISHER_PIPELINE_REDESIGN.md
│   ├── ARCHITECTURE_DECISIONS.md
│   ├── PFD_PROJECT_FOUNDATIONAL_PLAN.md
│   ├── TIER1_VALIDATION_REPORT.md
│   └── archive/                   ← Completed specs (historical)
├── src/
│   ├── analysis/
│   │   ├── fisher_diagnostics.py  ← Core FIM math + graph metrics
│   │   ├── fisher_report.py       ← Report generator
│   │   └── fisher_bridge_filter.py
│   ├── ingestion/
│   │   ├── parsers/               ← ecoli, zoo, periodic_table, ieee, opera, ccbh
│   │   ├── passes/                ← entity_catalog_pass.py
│   │   └── cross_universe_query.py
│   └── viz/
│       ├── tier1_dashboard.py     ← Coherence/regime charts + D3.js network
│       ├── tier1_report.py        ← Tier-1 HTML report
│       └── tier2_report.py        ← Tier-2 HTML report
├── scripts/
│   ├── run_fisher_suite.py        ← Main CLI (6 modes)
│   └── run_entity_catalog_pass.py
├── data/
│   ├── ds_wiki.db                 ← Reference knowledge graph
│   ├── chroma_db/                 ← ChromaDB vector index (bge-large 1024-dim)
│   ├── wiki_history.db            ← Embedding history snapshots
│   ├── reports/                   ← Fisher Suite HTML reports
│   ├── viz/                       ← Visualization outputs
│   └── rrp/                       ← Dataset bundles
└── tests/                         ← 450 tests