TL;DR
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Polars says version 2.0 makes its streaming engine the default for lazy queries and enables initial spill-to-disk support, allowing some workloads to exceed available memory. The release also expands SQL support and adds a Map data type; benchmark results cited by the project are vendor-run and depend on the tested hardware and setup.
Polars has released version 2.0, making its streaming engine the default for lazy queries and enabling initial spill-to-disk support for some operations. The changes can reduce memory pressure on supported workloads, but they also alter a behavior users may rely on: certain operations no longer preserve row order by default.
Under the new default, calling collect on a LazyFrame uses the streaming engine. Polars says streaming can bring memory and performance improvements on many queries. However, operations such as joins, group-bys and unpivots may not preserve observable row order. Users who require that behavior can set maintain_order=True where supported.
Version 2.0 also enables out-of-core processing by default. According to Polars, supported operations, including sorts, window functions and some expressions, can spill data to disk when memory use reaches about 80% of RAM; the default disk budget is 64 GB. The release post says support for joins and group-bys is planned, but does not describe those operations as supported by the initial spill implementation.
Other additions include expanded SQL support and a native Polars Map dtype corresponding to Arrow’s MapType. The new type represents key-value data in a dictionary-like form and offers operations such as key lookup, checking whether a key exists, and retrieving keys or values. Polars also describes stricter handling of data types and explicitness as part of the release.
Streaming Changes Query Behavior
The default change may affect both resource use and query results’ ordering as experienced by applications. Streaming can help process workloads without requiring all intermediate data to stay in memory, while disk spilling gives supported operations a way to continue when memory use rises. That may make the library more practical for users working with datasets larger than available RAM, although the release’s support is not yet universal across operation types.
The row-order change means teams upgrading should check whether their pipelines depend on the prior ordering behavior. Where ordering matters, the project points users to maintain_order=True. This is a compatibility consideration, not merely a performance setting: downstream steps that assume a particular order may need explicit configuration and testing.
Polars also presents SQL as a first-class interface and reports strong benchmark results against DuckDB and DataFusion. Those results may interest teams choosing an analytical query engine, but they are not independent findings. They were produced by Polars using particular datasets, machines and benchmark procedures, so readers should treat them as a project-run comparison rather than a universal ranking.
high-performance data analysis laptop
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
SQL Benchmarks and Engine Work
Polars says version 2.0 was not intended as a major feature release, but it includes work on the optimizer and execution engine. The project identifies join reordering, common-subplan elimination and dynamic predicates or bloom filters among the performance changes supporting SQL workloads.
For its comparison, Polars tested queries derived from TPC-H and TPC-DS against DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0. It used an AWS c7a.4xlarge machine with 16 vCPUs and 32 GB of RAM, and a c7a.metal machine with 192 vCPUs and 384 GB. Each query was run five times in a hot setting, with the best run used for comparisons; a 60-second timeout applied. The project says the data was stored on EBS and that its benchmark repository is available for replication.
Polars reports that it and both DuckDB versions completed all queries. DataFusion timed out on TPC-DS query 72, timed out once on query 67, and ran out of memory on TPC-H query 18 on the smaller machine; those queries were excluded for all engines. Polars also reports overhead when scaling to 192 threads, which hurt some small-data queries, and says it hopes to address that in a later release.
large memory capacity SSD external drive
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the New Defaults
The release post does not specify how much faster or less memory-intensive typical users’ workloads will be. Its performance statements are broad, while the published benchmark results reflect a defined test setup and were produced by the Polars team. Independent replications and results on different hardware or data layouts may differ.
Some details of the spill behavior also remain open. Polars says spilling begins at approximately 80% of RAM and that this threshold may need tuning, but the post does not provide a workload-by-workload account of limitations or say when disk use will be supported for joins and group-bys. The 64 GB default disk budget may also be relevant to users with constrained storage.
The release material provided here does not state the publication date, migration guidance beyond the row-order setting, or the exact release schedule for subsequent improvements. The project’s description of stricter dtype handling is not detailed enough to identify every affected workflow.
As an affiliate, we earn on qualifying purchases.
Watch for Compatibility Guidance
Users moving to Polars 2.0 should test lazy queries that depend on row order, and set maintain_order=True where the documented operation supports it and preserving order is required. They should also check whether their workload’s operations are covered by the current spill-to-disk implementation and account for the configured disk budget.
Polars says it plans to extend out-of-core support to joins and group-bys and hopes to fix the scaling overhead it observed on the 192-vCPU system in a later release. The post gives no dates for either item. The project has published a benchmark repository for readers who want to reproduce its SQL comparisons; further testing and release notes will clarify how the new defaults perform across other workloads.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main change in Polars 2.0?
LazyFrame.collect now uses the streaming engine by default. The release also enables initial spill-to-disk support for selected operations and expands SQL and data-type features.
Does Polars 2.0 preserve row order?
Not by default for some streaming operations, including joins, group-bys and unpivots, according to Polars. Users who need observable ordering can set maintain_order=True where available.
Which operations can spill data to disk?
The release post lists sorts, window functions and many expressions as supported by the initial out-of-core implementation. Polars says joins and group-bys are planned for later, but gives no delivery date.
Did Polars prove it is faster than DuckDB and DataFusion?
No independent proof is established by the release post. Polars reports favorable results from its own TPC-H and TPC-DS tests under specified conditions; other hardware, data and methods could produce different results.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
