Polars 2.0
AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

Polars says version 2.0 makes its streaming engine the default for lazy queries and enables initial spill-to-disk support, allowing some workloads to exceed available memory. The release also expands SQL support and adds a Map data type; benchmark results cited by the project are vendor-run and depend on the tested hardware and setup.

Polars has released version 2.0, making its streaming engine the default for lazy queries and enabling initial spill-to-disk support for some operations. The changes can reduce memory pressure on supported workloads, but they also alter a behavior users may rely on: certain operations no longer preserve row order by default.

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

Under the new default, calling collect on a LazyFrame uses the streaming engine. Polars says streaming can bring memory and performance improvements on many queries. However, operations such as joins, group-bys and unpivots may not preserve observable row order. Users who require that behavior can set maintain_order=True where supported.

Version 2.0 also enables out-of-core processing by default. According to Polars, supported operations, including sorts, window functions and some expressions, can spill data to disk when memory use reaches about 80% of RAM; the default disk budget is 64 GB. The release post says support for joins and group-bys is planned, but does not describe those operations as supported by the initial spill implementation.

Other additions include expanded SQL support and a native Polars Map dtype corresponding to Arrow’s MapType. The new type represents key-value data in a dictionary-like form and offers operations such as key lookup, checking whether a key exists, and retrieving keys or values. Polars also describes stricter handling of data types and explicitness as part of the release.

At a glance
announcementWhen: Announced in the Polars 2.0 release pos…
The developmentPolars has released version 2.0, changing lazy query execution to use its streaming engine by default and enabling initial out-of-core support.

Streaming Changes Query Behavior

The default change may affect both resource use and query results’ ordering as experienced by applications. Streaming can help process workloads without requiring all intermediate data to stay in memory, while disk spilling gives supported operations a way to continue when memory use rises. That may make the library more practical for users working with datasets larger than available RAM, although the release’s support is not yet universal across operation types.

The row-order change means teams upgrading should check whether their pipelines depend on the prior ordering behavior. Where ordering matters, the project points users to maintain_order=True. This is a compatibility consideration, not merely a performance setting: downstream steps that assume a particular order may need explicit configuration and testing.

Polars also presents SQL as a first-class interface and reports strong benchmark results against DuckDB and DataFusion. Those results may interest teams choosing an analytical query engine, but they are not independent findings. They were produced by Polars using particular datasets, machines and benchmark procedures, so readers should treat them as a project-run comparison rather than a universal ranking.

Products Worth Considering

Sony WH-1000XM6 Wireless Noise-Cancelling Headphones – Midnight Blue

Sony WH-1000XM6 Wireless Noise-Cancelling Headphones – Midnight Blue

As an affiliate, we earn on qualifying purchases.

SQL Benchmarks and Engine Work

Polars says version 2.0 was not intended as a major feature release, but it includes work on the optimizer and execution engine. The project identifies join reordering, common-subplan elimination and dynamic predicates or bloom filters among the performance changes supporting SQL workloads.

For its comparison, Polars tested queries derived from TPC-H and TPC-DS against DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0. It used an AWS c7a.4xlarge machine with 16 vCPUs and 32 GB of RAM, and a c7a.metal machine with 192 vCPUs and 384 GB. Each query was run five times in a hot setting, with the best run used for comparisons; a 60-second timeout applied. The project says the data was stored on EBS and that its benchmark repository is available for replication.

Polars reports that it and both DuckDB versions completed all queries. DataFusion timed out on TPC-DS query 72, timed out once on query 67, and ran out of memory on TPC-H query 18 on the smaller machine; those queries were excluded for all engines. Polars also reports overhead when scaling to 192 threads, which hurt some small-data queries, and says it hopes to address that in a later release.

Limits of the New Defaults

The release post does not specify how much faster or less memory-intensive typical users’ workloads will be. Its performance statements are broad, while the published benchmark results reflect a defined test setup and were produced by the Polars team. Independent replications and results on different hardware or data layouts may differ.

Some details of the spill behavior also remain open. Polars says spilling begins at approximately 80% of RAM and that this threshold may need tuning, but the post does not provide a workload-by-workload account of limitations or say when disk use will be supported for joins and group-bys. The 64 GB default disk budget may also be relevant to users with constrained storage.

The release material provided here does not state the publication date, migration guidance beyond the row-order setting, or the exact release schedule for subsequent improvements. The project’s description of stricter dtype handling is not detailed enough to identify every affected workflow.

Watch for Compatibility Guidance

Users moving to Polars 2.0 should test lazy queries that depend on row order, and set maintain_order=True where the documented operation supports it and preserving order is required. They should also check whether their workload’s operations are covered by the current spill-to-disk implementation and account for the configured disk budget.

Polars says it plans to extend out-of-core support to joins and group-bys and hopes to fix the scaling overhead it observed on the 192-vCPU system in a later release. The post gives no dates for either item. The project has published a benchmark repository for readers who want to reproduce its SQL comparisons; further testing and release notes will clarify how the new defaults perform across other workloads.

Key Questions

What is the main change in Polars 2.0?

LazyFrame.collect now uses the streaming engine by default. The release also enables initial spill-to-disk support for selected operations and expands SQL and data-type features.

Does Polars 2.0 preserve row order?

Not by default for some streaming operations, including joins, group-bys and unpivots, according to Polars. Users who need observable ordering can set maintain_order=True where available.

Which operations can spill data to disk?

The release post lists sorts, window functions and many expressions as supported by the initial out-of-core implementation. Polars says joins and group-bys are planned for later, but gives no delivery date.

Did Polars prove it is faster than DuckDB and DataFusion?

No independent proof is established by the release post. Polars reports favorable results from its own TPC-H and TPC-DS tests under specified conditions; other hardware, data and methods could produce different results.

Source: hn

You May Also Like

Fiscal policy response to Iran war risks worsening inflation: BIS chief

BIS General Manager Pablo Hernandez de Cos warns that fiscal responses to Iran conflict may heighten inflation, urging central banks to stay prepared.

Canton Network developer Digital Asset eyes $300 million raise at $2 billion valuation, led by a16z crypto: Bloomberg

Digital Asset, developer of Canton Network, aims to raise $300 million at a $2 billion valuation, with a16z crypto leading the funding round, according to Bloomberg.

Minecraft: Java Edition Now Uses SDL3

Minecraft Java Edition has officially transitioned to using SDL3, marking a significant update for the game’s graphics and input handling.

Unikernels Were Hard. Key Word: Were

Geoffrey Huntley argues AI coding tools could ease old unikernel barriers, though security benefits and engineering effort remain open questions.