冰山湖屋
本地首个数据湖屋，配备Apache Iceberg存储、Vortex柱状格式和通过MCP访问LLM。
建筑
┌─────────────────────────────────────────────────────────────────┐
│ LLM (Claude) │
│ "Query my expenses..." │
└─────────────────────────┬───────────────────────────────────────┘
│ MCP Protocol
▼
┌─────────────────────────────────────────────────────────────────┐
│ MCP Server (lakehouse) │
│ Tools: query, insert, update, delete, upsert, convert, ... │
└─────────────────────────┬───────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ DuckDB │
│ (in-memory, Iceberg + Vortex extensions) │
└───────────────┬─────────────────────────┬───────────────────────┘
│ PyIceberg │ Arrow bridge
▼ ▼
┌──────────────────────────┐ ┌────────────────────────────────────┐
│ Iceberg Tables │ │ Vortex Files │
│ ~/.lakehouse/warehouse/ │ │ (exported .vortex files) │
│ ├── expenses/ │ │ Faster reads, smaller files │
│ ├── health/ │ └────────────────────────────────────┘
│ └── notes/ │
└──────────────────────────┘快速开始
# Install dependencies
cd iceberg-lakehouse
uv sync
# Initialize lakehouse (creates catalog + sample tables)
uv run lakehouse init --with-sample-data
# Query via CLI
uv run lakehouse query "SELECT * FROM expenses LIMIT 10"
# Start MCP server (for Claude Desktop)
uv run lakehouse serve特性
- 冰山储存：全表版本控制、时间旅行、模式演变
- 涡流格式：柱状格式，文件小37-76%，读取速度快1.3-2.8倍
- DuckDB查询：使用SQL进行快速分析查询
- LLM访问：通过MCP进行自然语言查询（18个工具）
- 本地优先：所有数据都保留在您的机器上
- 完整CRUD：插入、更新、删除、追加销售、批处理操作
- 时间旅行：查询数据的任何历史快照
- 模式演进：添加、删除、重命名列而不重写数据
- 格式转换：在Parquet和Vortex格式之间转换
- 可配置格式：全局和每表格式首选项
CLI命令
# Data operations
lakehouse query "SELECT * FROM expenses WHERE amount > 100"
lakehouse query "SELECT * FROM expenses" --as-of 2025-12-01T00:00:00 --table-name expenses
lakehouse ingest data.csv expenses --format csv
# Table management
lakehouse tables # List all tables
lakehouse describe expenses # Show table schema
lakehouse snapshots expenses # List snapshots
lakehouse rollback expenses --snapshot-id 12345
lakehouse expire expenses --retain-last 5
# Schema evolution
lakehouse alter expenses add-column tags string
lakehouse alter expenses drop-column tags
lakehouse alter expenses rename-column desc description
# Batch operations
lakehouse batch '[{"action":"insert","table_name":"expenses","rows":[{"id":10,"amount":50}]}]'
lakehouse upsert expenses id '[{"id":1,"amount":90}]'
lakehouse delete expenses "id = 5" --force
# Vortex format
lakehouse convert data.parquet --to vortex
lakehouse convert data.vortex --to parquet
lakehouse convert-table expenses -o ./exports --compact
lakehouse query-vortex data.vortex "SELECT * FROM data"
# Configuration
lakehouse config show
lakehouse config set-format vortex
lakehouse config set-format parquet --table expenses
lakehouse alter expenses set-property write.format.default vortex
# Table properties
lakehouse alter expenses set-property write.format.default vortex
lakehouse alter expenses get-property write.format.default
lakehouse alter expenses remove-property write.format.default
# Benchmarks
lakehouse benchmark --rows 1000,10000,100000
lakehouse benchmark -o docs/benchmarks.mdMCP工具
MCP服务器公开了18个LLM访问工具：
| 工具 | 说明 |
|---|---|
query | 执行SQL查询（支持时间旅行） |
list_tables | 列出可用表格 |
describe_table | 获取表架构 |
insert | 插入行 |
update | 更新与筛选器匹配的行 |
delete | 删除与筛选器匹配的行 |
upsert | 插入或更新密钥匹配 |
alter_table | 添加、删除、重命名列 |
batch | 执行多个操作 |
rollback | 回滚到上一个快照 |
expire_snapshots | 清理旧快照 |
list_snapshots | 列出可用快照 |
refresh | 刷新表数据 |
convert_format | 将表格导出到Vortex |
query_vortex | 直接查询Vortex文件 |
get_format_config | 获取格式配置 |
set_format_config | 设置格式首选项 |
set_table_property | 设置冰山表属性 |
Claude桌面配置
增添 ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"lakehouse": {
"command": "uv",
"args": ["--directory", "/path/to/iceberg-lakehouse", "run", "lakehouse", "serve"]
}
}
}项目结构
iceberg-lakehouse/
├── pyproject.toml # Dependencies (uv)
├── src/lakehouse/
│ ├── __init__.py
│ ├── cli.py # CLI commands (Click)
│ ├── server.py # MCP server (18 tools)
│ ├── catalog.py # Iceberg catalog + CRUD operations
│ ├── query.py # DuckDB query engine + Vortex integration
│ ├── config.py # Format configuration (TOML)
│ ├── vortex_io.py # Vortex I/O and conversion utilities
│ └── _vortex_compat.py # Substrait compatibility shim
├── benchmarks/
│ └── format_comparison.py # Parquet vs Vortex benchmarks
├── docs/
│ ├── vortex.md # Vortex format guide
│ ├── format-comparison.md # When to use Parquet vs Vortex
│ ├── migration.md # Data migration guide
│ ├── benchmarks.md # Benchmark results
│ └── vortex-research.md # Vortex research notes
├── examples/
│ ├── vortex_basic.py # Basic Vortex usage
│ ├── migrate_to_vortex.py # Migration example
│ └── mixed_format.py # Mixed format queries
└── tests/ # 172 tests文档
- Vortex格式指南 --如何将Vortex与湖屋结合使用
- 格式比较 --何时使用Parquet vs Vortex
- 迁移指南 --转换现有数据
- 基准结果 --性能比较
路线图
- \[x\] 第一阶段：基本MCP服务器+冰山读取
- \[x\] 第2阶段：编写支持（插入、更新、删除、追加、批处理、模式演化、时间旅行、快照）
- \[x\] 第3阶段：Vortex数据格式集成