Use this skill for work involving DataFusion, Arrow schemas, SQL planning/execution, and table providers.
Implementation guidance:
- Use existing schema registry helpers for Arrow schema construction; memoize schemas where supported.
- Implement or extend TableProvider with correct schema, statistics, and scan behavior.
- Keep SQL handling in kalamdb-core/sql/executor and route through handler modules.
- Use DataFusion’s logical plan for validation; avoid manual SQL parsing unless required.
- Keep table/provider creation cheap; cache shared providers if appropriate.
- Ensure column types map correctly to Arrow types and are consistent across writes and reads.
Best practices:
- Respect DataFusion’s async execution model; avoid blocking IO in scan/exec paths.
- Prefer predicate pushdown where the provider supports it.
- Align system tables with kalamdb-commons models and constants.
Pitfalls:
- Mismatched schema ordering or nullability between writer and provider.
- Unbounded in-memory collection during scans.
- Creating new providers per request when a shared instance is intended.