StoreFS
StoreFS is a distributed S3-compatible object storage system
2 followers
StoreFS is a distributed S3-compatible object storage system
2 followers
StoreFS is a distributed, S3-compatible storage system implemented in Go, utilizing the Gossip protocol for cluster membership management and communication. The system supports dynamic node management, data distribution, and fault tolerance, providing users with high-performance, scalable object storage services. - bidzhao/storefs
StoreFS v0.5.2 — A Major Milestone in S3 Compatibility
A major milestone for AWS S3-compatible, self-managed object storage. Here’s what’s new:
🔑 Identity & Access — IAM + STS
AWS IAM-compatible identity management: users, groups, roles with trust policies, inline & managed policies, OIDC identity providers, and account-level summaries.
IAM principals have no implicit permissions — access is granted only through explicit policy Allow, bucket-policy Allow, or ACL grants.
AWS STS-compatible temporary credentials with AssumeRole, GetSessionToken, GetCallerIdentity, and AssumeRoleWithWebIdentity, including OIDC/JWKS signature verification.
Session policies, ABAC (aws:PrincipalTag, s3:ResourceTag), and iam:<Action> gating.
IAM management is available through the Web Console and MCP.
📦 More S3 Compatibility
Bucket Logging, Bucket Policy + PolicyStatus, Public Access Block, Bucket Encryption, CORS, Request Payment, and Ownership Controls.
Object ACLs, Object Legal Hold, GetObjectAttributes, UploadPartCopy, and flexible checksums including CRC32, CRC64NVME, and SHA256.
Background Object & Bucket Cleanup Services for automatic reclamation of stale data and soft-deleted buckets.
⚡ Under the Hood
TiDB replaces Apache Doris as the metadata backend, using the MySQL protocol — delivering 20×+ higher metadata concurrent read/write performance in our benchmarks.
Ceph s3-tests suite passing, providing broad coverage of AWS S3 compatibility.
Atomic concurrent overwrites — preventing stale or ghost keys during concurrent writes.
Strict SSE-KMS failure handling — returns 503 when KMS is unavailable instead of silently falling back to plaintext.
🖥️ Web Console & MCP
New IAM management pages.
Bucket Logging, Bucket Policy, and Cleanup Configuration pages.
User Access Key management.
New IAM management tools in the MCP server.
👉 See the full release notes for all changes and details.
StoreFS — your own AWS S3-compatible object storage, running on your hardware.
Distributed. Fault-tolerant. RDMA-accelerated. And now with enterprise-grade identity, access control, and security.
#StoreFS #S3 #ObjectStorage #AWS #IAM #STS #RDMA #OpenSource #CloudNative #DistributedStorage
StoreFS v0.5.1: Built-in Apache Iceberg Lakehouse Support
We are excited to announce the release of StoreFS v0.5.1, taking distributed object storage to the next level by delivering a fully built-in Apache Iceberg Lakehouse engine.
Traditional data lakehouses often require gluing together separate S3 storage, external Hive metastores, and complex access control proxies. With StoreFS v0.5.1, one StoreFS cluster = complete Lakehouse.
✨ Key Highlights
All-in-One & Self-Contained: No separate Hive Metastore or third-party S3 dependencies. Catalog management and data storage run seamlessly within StoreFS.
Full Iceberg v1 REST Spec: Native implementation of the Apache Iceberg REST Catalog protocol (/api/iceberg/{catalog}/v1/), ensuring instant compatibility with your favorite query engines.
Seamless Multi-Engine Integration: Connect Apache Spark, Flink, Trino, and DuckDB directly out of the box with zero external catalog setup.
Multi-Catalog Isolation: Dynamic warehouse bucket resolution (s3://bucket/prefix/) for granular access control and environment isolation.
Robust Doris Metadata Engine: Powered by Apache Doris under the hood to ensure atomic commits, fast catalog scans, and exact-once transaction guarantees.
Unified Management Panel: Manage catalogs, tables, and namespaces via our Web Admin Console or automate workflows using MCP / AI Agent tools.
🏗️ How It Works (Architecture Overview)
Engine Layer: Analytics engines (Spark, Flink, Trino, DuckDB) issue metadata requests to the Iceberg REST Catalog (Port :7946) and perform parallel data I/O directly against the StoreFS S3 Data Plane (Port :8901).
StoreFS Core: Processes metadata commits in-memory with atomic CAS head transitions while hosting the lakehouse warehouse bucket natively without S3 loopbacks.
Persistence Layer: Data files (.parquet, .avro) reside directly in the storage bucket, while catalog state and table heads are managed via low-latency Doris metadata tables.
💡 Ready to simplify your modern data stack? Upgrade to StoreFS v0.5.1 today and experience zero-overhead analytics on your object storage!
🚀 Introducing StoreFS v0.5.0 — an intelligent S3-compatible storage platform built for modern data workloads.
With this release, we focused on two major challenges in large-scale object storage: metadata scalability and data discoverability.
🔥 Metadata Engine Upgrade
We migrated our metadata backend from StarRocks to Apache Doris, bringing:
High-concurrency write optimization with Group Commit
Efficient Unique Key Merge-on-Write
Improved insert/update performance for massive object operations
🔍 AI-Powered Catalog Search
StoreFS now includes a powerful search engine powered by OpenSearch + k-NN:
Full-text search across object names, content, tags, and metadata
Semantic vector search using HNSW + cosine similarity
Hybrid search combining keyword and AI search with RRF
Content understanding for PDF, DOCX, HTML, EPUB, audio (Whisper), and images/videos (Vision LLM)
SQL-like queries and advanced filters
Horizontally scalable CatalogBuilder indexing service
Permission-aware Web UI search experience
Our goal is to evolve object storage from simply "store and retrieve data" into "store, understand, and discover data."
We’d love to hear your feedback and ideas as we continue building StoreFS into an intelligent storage platform for the AI era. 🚀
#OpenSource #ObjectStorage #S3 #AIStorage #VectorSearch #DistributedSystems