Loon, the new storage engine behind Milvus 3.0 and Zilliz Vector Lakebase, serves real-time search, large-scale discovery, and analytics from a single copy of vector data on low-cost object storage — even as teams continuously re-embed, re-label, and re-index that data.

Zilliz, a leading AI data infrastructure company and the creator of Milvus, recently announced Loon, the new storage engine that powers Zilliz Vector Lakebase and ships in Milvus 3.0. Loon is the lake-native foundation that lets a single copy of vector data serve real-time search, large-scale discovery, and batch analytics at once — the storage layer behind Zilliz Cloud’s evolution from a vector database into a unified data platform for AI.

Vector Lakebase is built on a demanding premise: one logical copy of vector data should serve every AI workload — production search, discovery, and batch analytics — without copying or moving data between systems. The hardest part is the storage layer, because the same dataset has to behave like two systems at once: fast, record-level lookups for serving and wide scans for analytics, all on inexpensive object storage. It also has to handle data that never stops changing, as teams re-embed, re-label, and re-index the same records while their models improve.

“Vector retrieval is no longer the whole problem; Vector Lakebase is our answer to what happens after vector databases succeed,” said James Luan, Cofounder and CTO of Zilliz. “The systems that win will make continuous serving and continuous discovery feel like part of the same machine — and that only works when the storage layer can serve a single copy of data to every workload. Loon is that storage layer.”

Storage Built for AI Data That Evolves

To make that possible, Loon treats a vector dataset as what it actually is — physically heterogeneous — and is built on three ideas:

• Hybrid file formats: Each kind of column is stored in the format that fits it. Scalar and filter fields use Parquet for efficient scans; dense and sparse vectors use the open Vortex format for fast, byte-precise row-level reads on object storage; and raw videos, PDFs, and images stay in object storage, referenced rather than copied into the database.
• Row ID alignment: Columns split across different formats still behave as one logical table, so a new embedding model can be added as its own column without rewriting the captions, metadata, or vectors already stored.
• A versioned Manifest: A single source of truth defines the dataset’s current version — its files, indexes, delete logs, and statistics — so serving clusters, on-demand compute, and external engines such as Spark and Ray can all read and safely update the same dataset instead of maintaining separate copies.

Together, these let one copy of data on object storage feed many engines at once. In Zilliz’s internal testing of object storage, Loon’s Vortex-based layout reduced the data pulled per record read by about 135x compared to Parquet — the difference between practical and impractical low-latency serving on inexpensive object storage. And because the same data evolves in place, adding a new embedding model becomes a lightweight version update rather than a multi-hundred-gigabyte rewrite.

That is the architecture behind Vector Lakebase: real-time serving clusters stay fast and stable; elastic, on-demand compute runs discovery and batch analytics without touching production; and External Collections index data that stays in a customer’s own S3 or GCS bucket — all on one semantic foundation, with no duplicate pipelines and no ETL. It is the same Milvus and Zilliz Cloud foundation that more than 10,000 enterprises and AI-native teams — including MiniMax, OpenEvidence, Filevine, Exa, and Salesforce — already build on.

Availability

Loon now powers Milvus 3.0 and serves as the storage layer for Zilliz Vector Lakebase on Zilliz Cloud, which is available across more than 30 regions on AWS, Google Cloud, and Microsoft Azure — with Serverless, Dedicated, and BYOC deployment options. Teams whose stack splits online serving, offline analysis, backfills, and external lake workflows across separate systems can create a free account — new work-email signups receive $100 in free credits — or connect with the Zilliz team to discuss a specific use case.

Source: BUSINESS WIRE