goldencheck (Benzsevern)

动了。 该回购已转移到 benzsevern/goldenmatch monorepo packages/python/goldencheck (and packages/typescript/goldencheck)/。此回购已存档；monorepo出现了新的发展。

金支票

数据验证，从数据中发现规则，这样您就不必编写它们。 建造于 本·塞文.

 ](https://www.npmjs.com/package/goldencheck)   ](https://pepy.tech/project/goldencheck) ](https://www.npmjs.com/package/goldencheck)  ](https://nodejs.org)     

每个竞争对手都让你先写规则。GoldenCheck翻转它： 先验证，遵守你关心的规则。

为什么选择GoldenCheck？

安装

pip install goldencheck

借助LLM boost支持：

pip install goldencheck[llm]

具有深度分析和基线支持（scipy、numpy）：

pip install goldencheck[baseline]

使用语义类型推理作为基线（句子转换）：

pip install goldencheck[baseline,semantic]

JavaScript/TypeScript

npm install goldencheck

边缘安全核心 （浏览器、Cloudflare Workers、Vercel Edge）：

import { scanData, TabularData } from "goldencheck/core";

Node.js （文件读取、CLI、MCP）：

import { readFile, scanData } from "goldencheck/node";

快速开始

# Scan a file — discovers issues, launches interactive TUI
goldencheck data.csv

# CLI-only output (no TUI)
goldencheck data.csv --no-tui

# With LLM enhancement (requires API key)
goldencheck data.csv --llm-boost --no-tui

# Validate against saved rules (for CI/pipelines)
goldencheck validate data.csv

# JSON output for CI integration
goldencheck data.csv --no-tui --json

# Learn baseline (one-time, deep analysis)
goldencheck baseline data.csv

# Scan with drift detection (fast, uses saved baseline)
goldencheck scan new_data.csv

TypeScript快速入门

// Scan an array of records (edge-safe — works anywhere)
import { scanData, TabularData, Severity } from "goldencheck";

const data = new TabularData([
  { id: 1, email: "alice@example.com", age: 30, status: "active" },
  { id: 2, email: "bob@test.com", age: -5, status: "inactive" },
  { id: 3, email: "not-an-email", age: 25, status: "active" },
]);

const { findings, profile } = scanData(data);
for (const f of findings) {
  console.log(`[${f.severity === Severity.ERROR ? "ERROR" : "WARNING"}] ${f.column}: ${f.message}`);
}

// Scan a CSV file (Node.js)
import { readFile, scanData, applyConfidenceDowngrade, healthScore } from "goldencheck/node";

const data = readFile("data.csv");
const result = scanData(data, { domain: "healthcare" });
const findings = applyConfidenceDowngrade(result.findings, false);

// Health score
const byCol = {};
for (const f of findings) {
  if (f.severity >= 2) {
    byCol[f.column] ??= { errors: 0, warnings: 0 };
    byCol[f.column][f.severity === 3 ? "errors" : "warnings"]++;
  }
}
const { grade, points } = healthScore(byCol);
console.log(`Health: ${grade} (${points}/100)`);

// Validate against pinned rules
import { readFile, scanData, validateConfig, validateData } from "goldencheck/node";
import { readFileSync } from "node:fs";
import YAML from "yaml";

const config = validateConfig(YAML.parse(readFileSync("goldencheck.yml", "utf-8")));
const data = readFile("data.csv");
const findings = validateData(data, config);

// Create baseline and detect drift
import { readFile, createBaseline, serializeBaseline, scanData } from "goldencheck/node";
import { runDriftChecks, deserializeBaseline } from "goldencheck";
import { writeFileSync, readFileSync } from "node:fs";

// Learn baseline
const data = readFile("reference.csv");
const baseline = createBaseline(data);
writeFileSync("baseline.json", serializeBaseline(baseline));

// Later: detect drift
const newData = readFile("production.csv");
const saved = deserializeBaseline(readFileSync("baseline.json", "utf-8"));
const driftFindings = runDriftChecks(newData, saved);

// LLM-enhanced scanning (edge-safe)
import { scanData, TabularData, callLlm, parseLlmResponse, mergeLlmFindings, buildSampleBlocks } from "goldencheck";

const data = new TabularData(records);
const result = scanData(data, { returnSample: true });
const blocks = buildSampleBlocks(result.sample, result.findings);
const { text } = await callLlm("anthropic", JSON.stringify(blocks));
const llmResponse = parseLlmResponse(text);
if (llmResponse) {
  const enhanced = mergeLlmFindings(result.findings, llmResponse);
}

运作原理

1. SCAN     →  goldencheck data.csv
                GoldenCheck profiles your data and discovers what "healthy" looks like

2. REVIEW   →  Interactive TUI shows findings sorted by severity
                Each finding has: description, affected rows, sample values

3. PIN      →  Press Space to promote findings into permanent rules
                Dismiss false positives — they won't come back

4. EXPORT   →  Press F2 to save rules to goldencheck.yml
                Human-readable YAML with your pinned rules

5. VALIDATE →  goldencheck validate data.csv
                Enforce rules in CI with exit codes (0 = pass, 1 = fail)

它检测到什么

柱级配置文件

横柱轮廓仪

基线深度剖面和漂移检测

跑 goldencheck baseline 一旦建立健康数据的统计档案。在每次后续扫描中，GoldenCheck都会将新数据与保存的基线进行比较，并报告13种检查类型的偏差：

基线是使用6种技术构建的：统计分析器（分布、本福德定律、熵）、约束挖掘器（函数依赖、时间顺序）、语义类型推断器（嵌入+关键字）、相关性分析器（Pearson、Cramér's V）、模式语法诱导器和置信度先验构建器。

域包

通过域特定类型定义提高检测精度：

goldencheck scan data.csv --domain healthcare   # NPI, ICD, insurance, patient types
goldencheck scan data.csv --domain finance      # accounts, routing, CUSIP, transactions
goldencheck scan data.csv --domain ecommerce    # SKUs, orders, tracking, products

域包添加了语义类型，减少了误报，提高了行业特定数据的分类。

架构差异

比较数据文件的两个版本：

goldencheck diff data.csv                  # compare against git HEAD
goldencheck diff old.csv new.csv           # compare two files
goldencheck diff data.csv --ref main       # compare against a branch

自动修正

应用自动修复程序来清理数据：

goldencheck fix data.csv                          # safe: trim, normalize, fix encoding
goldencheck fix data.csv --mode moderate          # + standardize case
goldencheck fix data.csv --mode aggressive --force # + coerce types
goldencheck fix data.csv --dry-run                # preview changes

观看模式

持续监控目录的数据质量：

goldencheck watch data/ --interval 30        # re-scan every 30s
goldencheck watch data/ --exit-on error      # CI mode: fail on first error

REST API

将GoldenCheck作为微服务运行：

goldencheck serve --port 8000

# Scan via file upload
curl -X POST http://localhost:8000/scan --data-binary @data.csv

# Scan via URL
curl -X POST http://localhost:8000/scan/url -d '{"url": "https://example.com/data.csv"}'

数据库扫描

直接扫描表格——无需CSV导出：

pip install goldencheck[db]
goldencheck scan-db "postgresql://user:pass@host/db" --table orders
goldencheck scan-db "snowflake://..." --query "SELECT * FROM orders WHERE date > '2024-01-01'"

计划运行

使用webhook通知进行类似Cron的调度：

goldencheck schedule data/*.csv --interval hourly --webhook https://hooks.slack.com/...
goldencheck schedule data/*.csv --interval daily --notify-on grade-drop

LLM提升

添加 --llm-boost 利用LLM智能增强分析器的发现。LLM会收到您数据的代表性样本，并：

1. 查找分析人员遗漏的问题 --语义理解（例如，姓名栏中的“12345”）
2. 升级严重性 --知道“应该需要电子邮件”，即使分析器只说“信息”
3. 发现关系 --确定列之间的时间顺序，如 signup_date 和 last_login
4. 降低误报率 --混合电话格式很常见，不是错误

# Using OpenAI
export OPENAI_API_KEY=sk-...
goldencheck data.csv --llm-boost --llm-provider openai --no-tui

# Using Anthropic
export ANTHROPIC_API_KEY=sk-ant-...
goldencheck data.csv --llm-boost --no-tui

成本： 每次扫描约0.01美元（一次API调用，带有代表性样本，而不是每次扫描）。

预算控制：

export GOLDENCHECK_LLM_BUDGET=0.50  # max spend per scan in USD

配置（goldencheck.yml）

version: 1

settings:
  sample_size: 100000
  fail_on: error

columns:
  email:
    type: string
    required: true
    format: email
    unique: true

  age:
    type: integer
    range: [0, 120]

  status:
    type: string
    enum: [active, inactive, pending, closed]

relations:
  - type: temporal_order
    columns: [start_date, end_date]

ignore:
  - column: notes
    check: nullability

此文件中只显示固定规则，而不是每个发现。这 ignore 该列表可防止被驳回的调查结果再次出现。

CLI 参考

旗帜

TypeScript命令行界面

npx goldencheck-js scan data.csv --json
npx goldencheck-js scan data.csv --domain healthcare
npx goldencheck-js health-score data.csv
npx goldencheck-js profile data.csv
npx goldencheck-js validate data.csv --config goldencheck.yml
npx goldencheck-js baseline data.csv --output baseline.json
npx goldencheck-js fix data.csv --mode safe
npx goldencheck-js diff old.csv new.csv
npx goldencheck-js demo

TypeScript架构

goldencheck (npm)
├── goldencheck/core    # Edge-safe: browsers, Workers, Edge Runtime
│   ├── types           # Finding, Severity, DatasetProfile, Config types
│   ├── data            # TabularData — zero-dep columnar abstraction
│   ├── profilers       # 10 column profilers + 4 relation profilers
│   ├── semantic        # Type classifier, suppression, 3 domain packs
│   ├── engine          # Scanner, confidence, validator, triage, differ, fixer
│   ├── baseline        # Statistical profiling, constraints, correlation, patterns
│   ├── drift           # 13 drift checks against saved baseline
│   ├── llm             # Anthropic + OpenAI via fetch(), merger, budget
│   ├── agent           # Strategy, handoff, review queue
│   └── reporters       # JSON, CI
└── goldencheck/node    # Node.js >= 20
    ├── reader          # CSV, Parquet (via nodejs-polars)
    ├── mcp             # MCP server (7 tools)
    ├── a2a             # Agent-to-Agent HTTP server
    ├── tui             # ANSI terminal output
    ├── db-scanner      # Postgres, MySQL, SQLite
    └── watcher         # Directory polling

基准测试

速度

DQBench v1.0--头对头

GoldenCheck的零配置发现优于所有竞争对手，即使他们有手写的规则。

自己运行基准测试：

pip install dqbench goldencheck
dqbench run goldencheck

检测精度

在自定义基准测试中，对9个类别的341个植入数据质量问题进行了测试。

v0.2.0的改进：少数错误类型检测、范围分析器链、更广泛的时间启发式和置信度评分将分析器的召回率从87%提高到100%。

Raha基准数据集

技术栈

可选： Anthropic SDK / OpenAI SDK LLM Boost| MCP-SDK 对于MCP服务器| SciPy + 数值Python 用于深度基线分析([baseline]) | 句子变换器 用于基线语义类型推理([semantic])

Types/Node.js

MCP服务器（克劳德桌面）

GoldenCheck包括一个用于Claude Desktop集成的MCP服务器：

pip install goldencheck[mcp]

添加到您的Claude桌面配置(claude_desktop_config.json):

{
  "mcpServers": {
    "goldencheck": {
      "command": "goldencheck",
      "args": ["mcp-serve"]
    }
  }
}

可用工具：

远程MCP服务器

GoldenCheck可作为托管MCP服务器在 铁匠铺 --无需安装任何东西即可从任何MCP客户端连接。

克劳德桌面/克劳德代码：

{
  "mcpServers": {
    "goldencheck": {
      "url": "https://goldencheck-mcp-production.up.railway.app/mcp/"
    }
  }
}

本地服务器：

pip install goldencheck[mcp]
goldencheck mcp-serve

19种可用工具：扫描文件、验证规则、配置文件列、健康评分数据集、自动配置验证、解释发现、比较域、建议修复。

Jupyter/Colab

GoldenCheck在Jupyter笔记本中呈现丰富的HTML：

from goldencheck.engine.scanner import scan_file
from goldencheck.engine.confidence import apply_confidence_downgrade
from goldencheck.notebook import ScanResult

findings, profile = scan_file("data.csv")
findings = apply_confidence_downgrade(findings, llm_boost=False)

# Rich HTML display in notebooks
ScanResult(findings=findings, profile=profile)


API快速参考

python

import goldencheck

# Scan a CSV for quality issues
findings = goldencheck.scan_file("data.csv")
for f in findings:
    print(f"[{f.severity}] {f.column}: {f.check} — {f.message}")

# Create baseline and detect drift
from goldencheck import create_baseline, scan_file
baseline = create_baseline("data.csv")
baseline.save("goldencheck_baseline.yaml")
findings, profile = scan_file("data.csv", baseline="goldencheck_baseline.yaml")

# Health score
score = goldencheck.health_score("data.csv")
print(score)  # e.g. "B (78/100)"

TypeScript

import { scanData, TabularData, Severity } from "goldencheck";

// Scan records (edge-safe)
const data = new TabularData(records);
const { findings, profile } = scanData(data);
for (const f of findings) {
  console.log(`[${f.severity === Severity.ERROR ? "ERROR" : "WARNING"}] ${f.column}: ${f.message}`);
}

import { readFile, scanData, applyConfidenceDowngrade, healthScore } from "goldencheck/node";

// Scan a CSV file (Node.js)
const data = readFile("data.csv");
const result = scanData(data, { domain: "healthcare" });
const findings = applyConfidenceDowngrade(result.findings, false);

// Health score
const byCol = {};
for (const f of findings) {
  if (f.severity >= 2) {
    byCol[f.column] ??= { errors: 0, warnings: 0 };
    byCol[f.column][f.severity === 3 ? "errors" : "warnings"]++;
  }
}
const { grade, points } = healthScore(byCol);
console.log(`Health: ${grade} (${points}/100)`);

import { readFile, createBaseline, serializeBaseline } from "goldencheck/node";
import { runDriftChecks, deserializeBaseline } from "goldencheck";
import { writeFileSync, readFileSync } from "node:fs";

// Create baseline and detect drift
const data = readFile("reference.csv");
const baseline = createBaseline(data);
writeFileSync("baseline.json", serializeBaseline(baseline));

const newData = readFile("production.csv");
const saved = deserializeBaseline(readFileSync("baseline.json", "utf-8"));
const driftFindings = runDriftChecks(newData, saved);

贡献

看 贡献.md 用于开发设置和指南。

作者

本·塞文

许可证

麻省理工学院——见 许可证

______________________________________________________________________

黄金套房的一部分

配套项目：

• dbt-goldencheck --数据验证作为dbt测试。
• 金饰类型 --社区贡献的域类型包。
• 戈尔登切克行动 --GitHub CI行动，附带公关评论。