DuckDB Labs has shipped DuckLake 1.0, a lakehouse format that keeps table metadata in a SQL catalog rather than scattering it across many files in object storage. The release adds data inlining for small updates, sorted tables, bucket partitioning and deletion vectors compatible with Iceberg-style features. DuckLake is available as a DuckDB extension and has clients for Apache DataFusion, Apache Spark, Trino and Pandas; MotherDuck offers a hosted service to manage the catalog and storage.

What DuckLake changes

DuckLake flips a common lakehouse pattern. It stores table metadata in a SQL database. Data files remain Parquet. The DuckLake extension for DuckDB reads and writes both the catalog and the Parquet data.

The DuckDB team argues file-based metadata causes complex coordination. It says metadata scattered as files slows operations and creates lots of tiny files in object stores. DuckLake instead places schema, partitioning and snapshot records inside a single catalog database file or service.

That choice affects several routine operations. Listing tables and reading schema can hit the catalog database rather than walking many objects. Small updates and deletes can be recorded in the catalog instead of creating new files. The DuckDB team calls that capability data inlining, and it's a headline feature in v1.0.

Features in the v1.0 release

DuckLake 1.0 brings multiple features aimed at operational speed and fewer small files. Data inlining lets small inserts, updates and deletes be applied inside the catalog database. The default threshold is 10 rows.

That means very small mutations avoid producing new Parquet files.

The release also adds sorted tables to speed filtered queries. It offers bucket partitioning for high-cardinality columns. It improves support for geometry types. And it integrates deletion vectors compatible with Iceberg-style implementations.

On the technical side, DuckLake keeps data in Parquet files on object storage and holds a reference layer in the catalog. The DuckLake extension uses an ATTACH syntax so a DuckDB session can mount a DuckLake database. The catalog itself can be a DuckDB file, often named metadata.ducklake in examples, or an external database when a hosted service manages it.

Interoperability and clients

DuckLake is designed to be multi-tool. This project already provides clients for Apache DataFusion, Apache Spark, Trino and Pandas. That means engines which already read Parquet can be taught to consult the DuckLake catalog to find table files and snapshot metadata.

Mastheads

MotherDuck, the managed DuckDB service, offers a hosted DuckLake option. It runs and manages the catalog database and storage. For organisations that don't want to operate the catalog themselves, a hosted service removes that burden.

The GitHub repository for the DuckLake extension shows examples of common workflows. Developers can attach a catalog file, create tables, insert rows, perform updates and even run time-travel queries at a prior snapshot version. The extension also exposes change data feeds so clients can inspect row-level changes between snapshots.

How users and engineers have reacted

Reaction in developer communities has been lively. On Hacker News, Alexander Dahl, a data platform engineer, called the numbers "very exciting" and asked whether anyone had tried DuckLake on real workloads. On Reddit, a user with the handle SutMinSnabel4 requested stronger support for SMB environments, noting many enterprises still use on-premises Windows file sharing with DFS and Kerberos.

Those responses point to two kinds of questions users will ask. One is about performance and scale on production workloads. The other is about how DuckLake fits into enterprise environments that rely on traditional file protocols and access controls. DuckDB Labs has prioritized the catalog model in this release, and community feedback is already shaping the roadmap.

The DuckDB team has published a roadmap that lays out short and medium-term improvements. A v1.1 release will probably add variant inlining across catalogs and improved multi-deletion vector files, called Puffin files in the roadmap notes. The team also lists a planned v2.0 feature that adds Git-like branching for datasets.

The project is open source and accepts contributions. The GitHub page includes build instructions and a note that contributors should coordinate on new features. This project also maintains a main branch and active feature branches for ongoing work.

Iceberg, Delta Lake and Apache Hudi use file-based metadata stored in object storage. They sometimes layer a separate catalog service on top. DuckLake instead places the catalog inside a SQL database by design. That removes many of the small-file operations normally required for small mutations.

That design choice doesn't remove Parquet files from the architecture. DuckLake keeps Parquet for columnar storage. It changes where the metadata lives and how small changes are recorded. The release also ensures compatibility with several Iceberg-style features like deletion vectors so existing tools and patterns can interoperate.

For data engineers and platform teams the immediate impact is operational ergonomics. DuckLake shifts many metadata operations into a database transaction model. Transactions against the catalog are atomic and can avoid expensive object store listings. That means faster metadata operations on reads and writes in some workloads.

It also alters how small updates are stored. With inlining enabled, small changes are kept in the catalog until they cross the configured row threshold. That reduces file churn in object storage. It should make everyday maintenance tasks, like compaction or vacuuming, less frequent for workloads dominated by small updates.

For teams that run multiple DuckDB instances against the same dataset, DuckLake can unlock a multiplayer model. Multiple DuckDB clients can read and write the same dataset while coordinating via the shared catalog. That model offers a concurrency pattern not supported by a single-node DuckDB process when metadata is only file-based.

Some practical questions remain. Enterprises that rely on SMB or other OS-dependent share protocols raised concerns about integration with existing on-premises storage stacks. The DuckDB team has built the extension to work with standard object stores, and the hosted option moves the catalog into a managed service. But support for legacy file protocols will be an item for platform teams to vet.

Another area to evaluate is multi-engine concurrency. DuckLake provides clients for several engines. But teams will need to test real workloads to confirm the promised gains in metadata speed and lower object-store file counts. Community feedback and early benchmarks will shape operators' decisions in the months after v1.0's release.

Related Articles

DuckDB Labs plans v1.1 with expanded inlining and v2.0 dataset branching; DuckLake clients are available and MotherDuck offers a hosted service to manage the catalog and storage.

This article was created with AI assistance.