去-objstore

Go的统一对象存储和文件系统抽象库。

](https://golang.org) ![codecov](https://codecov.io/gh/jeremyhahn/go-objstore) ![Security](#test-coverage) ![AGPL-3.0 License](LICENSE-AGPL-3.txt) ![Commercial License](LICENSE-COMMERCIAL.md)

特性

外观模式：用于所有存储操作的集中式、安全的API
多后端支持：同时使用多个存储后端
输入验证：内置防止注射攻击的保护
跨所有存储后端的统一API
支持加密的存储后端之间的复制和同步
可插拔适配器，用于自定义日志记录和身份验证
具有目录操作的完整文件系统接口
自动删除和存档的生命周期策略
多种服务器接口：gRPC、REST、QUIC/HTTP3和MCP
具有灵活配置选项的CLI工具
用于嵌入C/C++应用程序的C API
TLS/mTLS支持安全通信

快速开始

安装

go get github.com/jeremyhahn/go-objstore

基本用途（立面图案-推荐）

package main

import (
    "bytes"
    "context"
    "fmt"
    "io"

    "github.com/jeremyhahn/go-objstore/pkg/common"
    "github.com/jeremyhahn/go-objstore/pkg/factory"
    "github.com/jeremyhahn/go-objstore/pkg/objstore"
)

func main() {
    // Create storage backends
    local, _ := factory.NewStorage("local", map[string]string{
        "path": "/tmp/my-storage",
    })

    // Initialize facade (do this once at app startup)
    objstore.Initialize(&objstore.FacadeConfig{
        Backends: map[string]common.Storage{
            "local": local,
        },
        DefaultBackend: "local",
    })
    defer objstore.Reset()

    // Store data
    data := []byte("Hello, World!")
    objstore.Put("greeting.txt", bytes.NewReader(data))

    // Retrieve data
    reader, _ := objstore.Get("greeting.txt")
    defer reader.Close()

    content, _ := io.ReadAll(reader)
    fmt.Println(string(content))  // Output: Hello, World!

    // Delete data
    objstore.Delete("greeting.txt")
}

直接存储访问（传统）

为了向后兼容，您仍然可以使用直接存储访问：

// Create a storage backend
storage, err := factory.NewStorage("local", map[string]string{
    "path": "/tmp/my-storage",
})
if err != nil {
    panic(err)
}

// Use storage directly
storage.Put("greeting.txt", bytes.NewReader(data))

注：建议将facade模式用于新代码，因为它提供了集中验证、多后端支持和增强的安全性。

支持的后端

后端	类型	用例
本地	存储	开发、测试、本地存档
S3	存储	AWS对象存储，高可用性
MinIO	存储	自托管S3兼容对象存储
GCS	存储	谷歌云对象存储
Azure Blob	存储	Microsoft Azure对象存储
Glacier	仅存档	AWS长期冷存储
Azure存档	仅存档	Azure长期冷存储

后端配置

本地存储

storage, _ := factory.NewStorage("local", map[string]string{
    "path": "/var/data/storage",
})

亚马逊S3

storage, _ := factory.NewStorage("s3", map[string]string{
    "region": "us-east-1",
    "bucket": "my-bucket",
    // Optional: for custom endpoints (MinIO, LocalStack)
    "endpoint":       "http://localhost:9000",
    "forcePathStyle": "true",
    "accessKey":      "minioadmin",
    "secretKey":      "minioadmin",
})

MinIO

storage, _ := factory.NewStorage("minio", map[string]string{
    "bucket":    "my-bucket",
    "endpoint":  "http://localhost:9000",
    "accessKey": "minioadmin",
    "secretKey": "minioadmin",
    // Optional: defaults to "us-east-1"
    "region":    "us-east-1",
})

谷歌云存储

storage, _ := factory.NewStorage("gcs", map[string]string{
    "bucket": "my-gcs-bucket",
})

Azure Blob存储

storage, _ := factory.NewStorage("azure", map[string]string{
    "accountName":   "myaccount",
    "accountKey":    "base64key==",
    "containerName": "mycontainer",
})

高级功能

立面图案（推荐）

facade模式提供了一个集中、安全的API，用于处理多个存储后端。它可以防止泄漏的抽象，并确保所有入口点的一致验证。

益处

多后端支持：同时使用多个存储后端
后端路由：使用 backend:key 针对特定后端的语法
自动验证：内置防止路径遍历、注入攻击和格式错误输入的保护
集中式API：所有存储操作的单一入口点
安全：山宁泰错误消息防止信息泄露

多后端示例

import (
    "github.com/jeremyhahn/go-objstore/pkg/common"
    "github.com/jeremyhahn/go-objstore/pkg/factory"
    "github.com/jeremyhahn/go-objstore/pkg/objstore"
)

// Create multiple storage backends
local, _ := factory.NewStorage("local", map[string]string{
    "path": "/tmp/local-storage",
})

s3, _ := factory.NewStorage("s3", map[string]string{
    "bucket": "my-bucket",
    "region": "us-east-1",
})

// Initialize facade once at application startup
objstore.Initialize(&objstore.FacadeConfig{
    Backends: map[string]common.Storage{
        "local": local,
        "s3":    s3,
    },
    DefaultBackend: "local",
})
defer objstore.Reset()

// Use default backend
objstore.Put("file.txt", data)

// Target specific backend
objstore.PutWithContext(ctx, "s3:backups/file.txt", data)
objstore.PutWithContext(ctx, "local:cache/temp.dat", data)

// Get from specific backend
reader, _ := objstore.GetWithContext(ctx, "s3:backups/file.txt")

// List all available backends
backends := objstore.Backends()  // ["local", "s3"]

安全功能

facade会自动验证所有输入以防止攻击：

// These all fail with validation errors
objstore.Put("../../../etc/passwd", data)        // Path traversal blocked
objstore.Put("/etc/passwd", data)                // Absolute path blocked
objstore.Put("file\x00.txt", data)              // Null byte blocked
objstore.Put("file\n.txt", data)                // Control character blocked
objstore.PutWithContext(ctx, "INVALID:key", data) // Invalid backend name blocked

有关详细的迁移指南和示例，请参阅 docs/facade-migration.md.

文件系统接口

将对象存储与熟悉的文件系统操作一起使用：

import "github.com/jeremyhahn/go-objstore/pkg/storagefs"

fs := storagefs.New(storage)

// Create directories
fs.MkdirAll("docs/2024", 0755)

// Create and write to a file
file, _ := fs.Create("docs/readme.txt")
file.WriteString("Hello from StorageFS!")
file.Close()

// List directory contents
dir, _ := fs.Open("docs")
defer dir.Close()

entries, _ := dir.Readdir(-1)
for _, entry := range entries {
    fmt.Printf("%s (dir: %v, size: %d bytes)\n",
        entry.Name(), entry.IsDir(), entry.Size())
}

// Read directory names only
dir2, _ := fs.Open("docs")
names, _ := dir2.Readdirnames(-1)
for _, name := range names {
    fmt.Println(name)
}

生命周期策略

自动化数据保留和归档：

import (
    "time"
    "github.com/jeremyhahn/go-objstore/pkg/common"
)

// Delete old logs after 30 days
deletePolicy := common.LifecyclePolicy{
    ID:        "cleanup-old-logs",
    Prefix:    "logs/",
    Action:    "delete",
    Retention: 30 * 24 * time.Hour,
}
storage.AddPolicy(deletePolicy)

// Archive data to Glacier after 90 days
glacier, _ := factory.NewArchiver("glacier", map[string]string{
    "vaultName": "long-term-archive",
    "region":    "us-east-1",
})

archivePolicy := common.LifecyclePolicy{
    ID:          "archive-old-data",
    Prefix:      "data/",
    Action:      "archive",
    Destination: glacier,
    Retention:   90 * 24 * time.Hour,
}
storage.AddPolicy(archivePolicy)

上下文支持

所有操作都支持取消和超时上下文：

import "context"

ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()

// Context-aware operations
err := storage.PutWithContext(ctx, "key", data)
reader, err := storage.GetWithContext(ctx, "key")
err = storage.DeleteWithContext(ctx, "key")

元数据支持

存储和检索自定义元数据：

metadata := &common.Metadata{
    ContentType:     "application/json",
    ContentEncoding: "utf-8",
    Custom: map[string]string{
        "author":  "john-doe",
        "version": "1.0",
    },
}

// Put with metadata
storage.PutWithMetadata(ctx, "data.json", reader, metadata)

// Get metadata
meta, _ := storage.GetMetadata(ctx, "data.json")
fmt.Println(meta.Custom["author"])  // Output: john-doe

// Update metadata
meta.Custom["version"] = "2.0"
storage.UpdateMetadata(ctx, "data.json", meta)

带分页的列表

高效地列出大型目录：

opts := &common.ListOptions{
    Prefix:     "logs/2024/",
    MaxResults: 100,
    Delimiter:  "/",
}

result, _ := storage.ListWithOptions(ctx, opts)
for _, obj := range result.Objects {
    fmt.Printf("%s (%d bytes)\n", obj.Key, obj.Metadata.Size)
}

// Get next page
if result.Truncated {
    opts.ContinueFrom = result.NextToken
    nextPage, _ := storage.ListWithOptions(ctx, opts)
}

C API

在C/C++应用程序中嵌入go-objstore：

#include "libobjstore.h"

int main(void) {
    // Create storage
    char *keys[] = {"path"};
    char *values[] = {"/tmp/storage"};
    int handle = ObjstoreNewStorage("local", keys, values, 1);

    // Store data
    char *data = "Hello from C!";
    ObjstorePut(handle, "test.txt", data, strlen(data));

    // Retrieve data
    char buffer[256];
    int len = ObjstoreGet(handle, "test.txt", buffer, 256);
    buffer[len] = '\0';
    printf("%s\n", buffer);

    // Cleanup
    ObjstoreDelete(handle, "test.txt");
    ObjstoreClose(handle);
    return 0;
}

构建说明：

# Build the shared library
make lib

# Compile your C program
gcc -o myapp myapp.c -L./bin -lobjstore -lpthread -ldl

# Run with library path
LD_LIBRARY_PATH=./bin ./myapp

文档

完整的文档可在 docs/ 目录。

建筑

配置

用法

额外资源

例子

示例代码可在示例/ 目录：

C客户端 -使用C应用程序
StorageFS -文件系统接口示例

项目结构

go-objstore/
├── pkg/                        # Core packages
│   ├── factory/               # Backend factory
│   ├── common/                # Shared interfaces and types
│   ├── local/                 # Local filesystem backend
│   ├── s3/                    # Amazon S3 backend
│   ├── gcs/                   # Google Cloud Storage backend
│   ├── azure/                 # Azure Blob Storage backend
│   ├── glacier/               # AWS Glacier archiver
│   ├── azurearchive/          # Azure Archive archiver
│   ├── storagefs/             # Filesystem abstraction
│   ├── cli/                   # CLI commands and config
│   └── server/                # Server implementations
│       ├── grpc/              # gRPC server
│       ├── rest/              # REST API server
│       ├── quic/              # QUIC/HTTP3 server
│       └── mcp/               # MCP server
├── cmd/
│   ├── objstore/              # CLI binary
│   ├── objstore-server/       # All-in-one multi-protocol server
│   ├── objstore-grpc-server/  # Individual gRPC server
│   ├── objstore-rest-server/  # Individual REST server
│   ├── objstore-quic-server/  # Individual QUIC/HTTP3 server
│   ├── objstore-mcp-server/   # Individual MCP server
│   └── objstorelib/           # C API shared library
├── api/                       # API definitions
│   ├── proto/                 # Protocol buffers for gRPC
│   ├── openapi/               # OpenAPI specs for REST
│   └── mcp/                   # MCP server configuration
├── examples/                   # Usage examples
├── test/integration/          # Integration tests
└── docs/                      # Documentation

发展

先决条件

达到1.23或更高
Docker（用于集成测试）
制造

建筑

# Install dependencies
make deps

# Build the library
make build

# Build CLI tool
make build-cli

# Build server
make build-server

# Build C shared library
make lib

测试

# Run unit tests (fast, in-memory)
make test

# Run all integration tests (backends + CLI)
make integration-test

# Run ALL integration tests including servers
make integration-test-all

# Run specific backend tests
make integration-test-local
make integration-test-s3
make integration-test-azure
make integration-test-gcs
make integration-test-minio
make integration-test-factory

# Run CLI integration tests
make integration-test-cli

# Run server integration tests (gRPC, REST, QUIC, MCP)
make test-servers

# Generate coverage report
make coverage-report

# Check per-package coverage (highlights packages under 90%)
make coverage-check

测试覆盖率

单元测试运行迅速，无需外部依赖。集成测试对所有服务器和后端使用基于Docker的模拟器。CLI集成测试会自动生成CLI二进制文件（如果不存在）。使用gosec和govulncheck进行安全扫描。有关详细的覆盖率统计信息，请参阅上面的徽章或运行 make coverage-report.

建筑

存储接口

所有后端都实现了一个通用的存储接口：

type Storage interface {
    // Basic operations
    Put(key string, data io.Reader) error
    Get(key string) (io.ReadCloser, error)
    Delete(key string) error
    List(prefix string) ([]string, error)

    // Context-aware operations
    PutWithContext(ctx context.Context, key string, data io.Reader) error
    GetWithContext(ctx context.Context, key string) (io.ReadCloser, error)
    DeleteWithContext(ctx context.Context, key string) error
    ListWithContext(ctx context.Context, prefix string) ([]string, error)

    // Metadata operations
    PutWithMetadata(ctx context.Context, key string, data io.Reader, metadata *Metadata) error
    GetMetadata(ctx context.Context, key string) (*Metadata, error)
    UpdateMetadata(ctx context.Context, key string, metadata *Metadata) error

    // Advanced operations
    Exists(ctx context.Context, key string) (bool, error)
    ListWithOptions(ctx context.Context, opts *ListOptions) (*ListResult, error)
    Archive(key string, destination Archiver) error

    // Lifecycle management
    AddPolicy(policy LifecyclePolicy) error
    RemovePolicy(id string) error
    GetPolicies() ([]LifecyclePolicy, error)
}

工厂模式

工厂模式提供了一种创建后端的统一方法：

storage, err := factory.NewStorage(backendType, config)
archiver, err := factory.NewArchiver(archiverType, config)

这抽象了后端特定的初始化，同时确保了接口的一致性。

演出

所有后端都支持并发读/写操作。使用缓冲I/O以获得更好的性能。本地后端的开发和测试速度最快。云后端具有网络I/O开销。看 docs/testing.md 基准测试。

最佳实践

始终关闭Get（）返回的读取器，以防止资源泄漏
处理所有存储操作中的错误
使用上下文取消和超时长时间操作
启用生命周期策略以进行自动清理
根据成本、性能和耐用性需求明智地选择后端
当您需要标准文件系统操作时，请使用StorageFS
在部署到云端之前，使用模拟器进行测试

服务器接口

该项目提供了灵活的服务器部署选项，包括一体化多协议服务器和每种协议的单独服务器二进制文件。

CLI工具

# Run the CLI
./bin/objstore --help

# Store an object from file
./bin/objstore put myfile.txt mykey

# Store from stdin
echo "Hello World" | ./bin/objstore put - mykey
cat data.txt | ./bin/objstore put - mykey

# Retrieve an object to file
./bin/objstore get mykey output.txt

# Retrieve to stdout
./bin/objstore get mykey
./bin/objstore get mykey -

# Pipe between backends (copy/migrate data)
./bin/objstore get mykey --backend local | \
  ./bin/objstore put - mykey --backend s3

# List objects
./bin/objstore list

# Configure via config file, env vars, or flags
./bin/objstore --config .objstore.yaml put mykey data.txt

多协议一体服务器

使用单个二进制文件同时运行所有四个服务器协议：

# Start all services (gRPC, REST, QUIC, MCP)
./bin/objstore-server --quic-self-signed

# Customize ports and addresses
./bin/objstore-server \
  --grpc-addr :50051 \
  --rest-port 8080 \
  --quic-addr :4433 \
  --mcp-addr :8081 \
  --quic-self-signed

# Disable specific services
./bin/objstore-server --quic=false --mcp=false

# With production TLS for QUIC
./bin/objstore-server \
  --quic-tls-cert cert.pem \
  --quic-tls-key key.pem

单个服务器二进制文件

针对重点部署分别运行单独的协议：

gRPC服务器：

# Start gRPC server only
./bin/objstore-grpc-server --addr :50051

# With TLS
./bin/objstore-grpc-server --addr :50051 --tls-cert cert.pem --tls-key key.pem

REST API服务器：

# Start REST server only
./bin/objstore-rest-server --port 8080

# Access via HTTP
curl http://localhost:8080/objects/mykey

QUIC/HTTP3服务器：

# Start QUIC server only
./bin/objstore-quic-server -addr :4433 -tlscert cert.pem -tlskey key.pem

# With self-signed certificate (testing only)
./bin/objstore-quic-server -addr :4433 -selfsigned

MCP服务器：

# Start MCP server only (stdio mode for Claude Desktop)
./bin/objstore-mcp-server -mode stdio

# HTTP mode
./bin/objstore-mcp-server -mode http -addr :8081

部署模式

开发-所有服务：

# Quick development setup with all protocols
./bin/objstore-server --quic-self-signed

生产-负载平衡：

# Multiple instances of specific protocols behind load balancers
./bin/objstore-grpc-server --addr :50051 &
./bin/objstore-rest-server --port 8080 &

微服务-专用服务：

# Different protocols in different containers/hosts
docker run objstore-grpc-server
docker run objstore-rest-server
docker run objstore-quic-server

______________________________________________________________________

许可证

![AGPL-3.0](https://www.gnu.org/licenses/agpl-3.0.html)

go objstore在双许可模式下可用：

选项1:GNU Affero通用公共许可证v3.0（AGPL-3.0）

go objstore的开源版本根据 AGPL-3.0.

这是什么意思？

免费使用、修改和分发
非常适合开源项目
如果您修改并部署为网络服务（SaaS），则必须披露您的源代码
衍生作品也必须根据AGPL-3.0获得许可

AGPL-3.0要求，如果您修改此软件并将其作为网络上的服务提供（包括SaaS部署），则必须在相同的许可证下提供修改后的源代码。

选项2：商业许可

如果您希望在没有AGPL-3.0源代码披露要求的情况下在专有软件中使用go-objstore，可以从Automated the Things，LLC获得商业许可证。

商业许可证的好处：

在闭源应用程序中使用
无源代码披露要求
修改并保持更改私有
专业支持和SLA选项
可定制开发
法律保护和赔偿

商业许可联系人：

有关定价和商业许可咨询，请发送电子邮件至licensing@automatethethings.com或访问https://automatethethings.com

看许可证商业.md 了解更多详情。

选择正确的许可证

用例	推荐许可证
开源项目	AGPL-3.0
内部使用与源代码披露	AGPL-3.0
SaaS/云服务（开源）	AGPL-3.0
专有SaaS产品	商业
闭源应用程序	商业
嵌入商业产品	商业
需要专业支持	商业