Polars is the reason most people on an Apple silicon Mac have Arrow-shaped data in memory at all.
This document is how you point that data at the GPU.
There are four tiers, the first three in 0.1.0 and the engine (tier 4) in 0.2.0, and they differ in where the GPU sits
relative to the Polars plan:
All four move data over the Arrow C Data Interface. For a single-chunk numeric Polars column that
is no copy at all -- the GPU reads the buffer Polars already owns. A String column crosses the
same way in tiers 1, 3 and 4: in Polars' own Utf8View layout, which the string kernels read
directly (tier 2's plugin still asks Polars for large_string). Categoricals and multi-chunk Series
each cost one conversion pass -- see Limits. The evidence is below.
Install
From the wheel
scripts/build_wheel.sh # swift build, cargo build, then the wheel
pip install python/dist/arrowmetal-*.whl polars
A wheel built from this tree carries both native libraries in arrowmetal/_lib/:
libArrowMetalC.dylib (the GPU library) and libarrowmetal_polars.dylib (the tier-2 expression
plugin, built by cargo during the wheel build, against that same libArrowMetalC.dylib). All four
tiers work from that install, with no Xcode, no cargo and no DYLD_LIBRARY_PATH. The packaged plugin
has one rpath, @loader_path, so its @rpath/libArrowMetalC.dylib resolves to the copy beside it,
the same file Python loads. The 0.2.0 wheel on PyPI carries libArrowMetalC.dylib only; with it,
tier 2 needs the cargo build below.
scripts/check_wheel.sh checks a built wheel: it installs the wheel with its polars extra into a
fresh virtualenv outside the repository, with a scrubbed environment and no cargo on PATH, runs one
expression or plan per tier, checks that the process loaded exactly one libArrowMetalC.dylib, the
packaged one, and runs python -m arrowmetal.bench with and without --parquet. On an M4 Max
(macOS 26.6.2, Homebrew Python 3.13.9) it passed with polars 1.44.2 and pyarrow 25.0.1 as pip
resolved them, NumPy not installed: the packaged plugin, built against
polars 0.55 crates, loads in py-polars 1.44.2. The plugin adds 5.0 MB to the compressed wheel
(3.2 MB before, 8.2 MB after) and is 21 MB on disk after strip -x.
Which Polars
pip install 'arrowmetal[polars]' installs polars>=1.44,<2.1: polars 1.44.1, 1.44.2 and 2.0.0 are the releases the engine suites passed on.
Per tier:
A Polars outside 1.44 installed without the extra keeps tiers 1 and 3, and on 2.0.0 tier 4.
From source
# 1. The GPU library (every tier needs it)
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer \
swift build -c release --product ArrowMetalC
# 2. The Python package
pip install polars pyarrow
export PYTHONPATH=python # or: pip install ./python
# 3. Tier 2 only: the Polars expression plugin
cd polars-plugin && cargo build --release && cd ..
That is the whole build, and both halves are checked: on an M4 Max the Swift product and the cargo
release build both go through from an empty target/, with no other flags and no
DYLD_LIBRARY_PATH. cargo build --release is enough for the plugin -- it is a plain cdylib
that Polars dlopens, not a Python extension module, so maturin is optional, and
arrowmetal.polars_plugin.plugin_path() finds polars-plugin/target/release/ on its own.
plugin_path() looks, first hit wins, at $ARROWMETAL_POLARS_PLUGIN (the full path to a plugin
dylib); the packaged arrowmetal/_lib/libarrowmetal_polars.dylib when Python loaded the packaged
libArrowMetalC.dylib beside it, which is what a wheel install has; polars-plugin/target/release/
and target/debug/ in a source checkout; the packaged copy in any other case; and a maturin install
on sys.path. When $ARROWMETAL_LIB pins a development build, a cargo build linked against it
therefore wins over a packaged plugin left in _lib/ by a wheel build in the same checkout.
maturin develop --release is the untested path: polars-plugin/ has no pyproject.toml,
and plugin_path() looks for a maturin install at
<sys.path>/arrowmetal_polars/libarrowmetal_polars.dylib, which is not the layout maturin
produces for a cdylib crate. Use cargo build --release, or set ARROWMETAL_POLARS_PLUGIN to
whatever you did build.
The plugin's build.rs links libArrowMetalC.dylib by rpath. The dylib's install name is
@rpath/libArrowMetalC.dylib, so polars-plugin/arrowmetal-sys/build.rs finds the directory
holding it -- $ARROWMETAL_LIB_DIR, $ARROWMETAL_LIB, <repo>/.build/release,
<repo>/.build/debug, /usr/local/lib, /opt/homebrew/lib, first hit wins -- and bakes it in as
an LC_RPATH entry. No DYLD_LIBRARY_PATH is needed at run time. It republishes that directory
to the plugin crate through its links = "ArrowMetalC" key (DEP_ARROWMETALC_LIB_DIR), because
Cargo passes a build script's link arguments only to the crate that owns it.
Version pinning
polars-plugin/Cargo.toml pins polars 0.55.1 and pyo3-polars 0.28.0. Those are the Rust
crates py-polars 1.44.x is built from: the polars workspace at tag py-1.44.1 carries
version = "0.55.1", and pyo3-polars 0.28 is the release that depends on polars ^0.55.1. The
requirement is a caret, so Cargo.lock currently resolves polars 0.55.2 and polars-ffi 0.55.2;
the ABI check is on polars_ffi's major/minor, so that patch bump loads against py-polars 1.44.1
without complaint.
Polars checks the plugin ABI when it loads the library (_polars_plugin_get_version, which
returns polars_ffi's major/minor packed into a u32) and refuses a mismatched pair with
this Polars engine doesn't support plugin version: .... Moving to a different Polars means
re-pinning both crates to whatever that release's workspace version is -- read it from
https://raw.githubusercontent.com/pola-rs/polars/refs/tags/py-<version>/Cargo.toml, do not
guess. Tiers 1 and 3 have no such constraint: they are pure Python over the C Data Interface.
import polars as pl
import arrowmetal as am
Importing arrowmetal does not import Polars -- the package's only hard dependency is
pyarrow. The Polars half arrives on first use. Any of these arms it:
import arrowmetal.polars_bridge # explicit
am.from_polars(df) # first touch of a bridge function
import polars as pl; import arrowmetal # Polars already loaded -> registered at import
Two functions
am.from_polars(series) # -> MetalArray
am.from_polars(dataframe) # -> dict[str, MetalArray]
am.to_polars(metal_array) # -> pl.Series
am.to_polars({"a": ...}) # -> pl.DataFrame
Three namespaces
# Series
s.arrowmetal.sum() / .min() / .max() / .mean() / .count()
s.arrowmetal.top_k(100) / .bottom_k(100) / .sort(descending=True) / .arg_sort()
s.arrowmetal.filter(mask) / .unique() / .cum_sum() / .hash64()
s.arrowmetal.contains("x") / .starts_with("x") / .ends_with("x") / .upper() / .lower()
s.arrowmetal.to_metal() # keep the column on the GPU
# DataFrame
df.arrowmetal.group_by("k").sum("v")
df.arrowmetal.group_by("k", "region").agg(total=("v", "sum"), avg=("f", "mean"), n=(None, "len"))
df.arrowmetal.query(am.filter(am.col("k") == 2).sum(am.col("v")))
df.arrowmetal.sort(["k", "v"], descending=[False, True])
df.arrowmetal.top_k(10, by="v")
df.arrowmetal.filter(df["k"] > 8)
df.arrowmetal.join(other, on="k", how="inner")
df.arrowmetal.to_metal() / .device()
# LazyFrame
lf.arrowmetal.collect_gpu(query_or_callable)
Results come back as Polars objects. Scalar reductions come back as Python scalars, as
pl.Series.sum() does -- with one difference, in both tier 1 and tier 2: the sum of an empty or
all-null column is None, where pl.Series.sum() answers 0. am_reduce has nothing to add
up and says so; min, max and mean are None on both sides.
Grouped aggregates available through .agg: sum, mean, min, max, count, len,
n_unique, first, last, median, std, var, product, any, all, plus
gb.quantile(column, q). One am_group_by_keys pass builds the dense group ids and every
aggregate after that reuses it, so .agg(...) with six outputs costs one group-by.
What each tier-1 method runs
The join
The bridge does not call the C ABI's am_join hash join; df.arrowmetal.join is built out of
two other kernels. am_index_in finds, for every left key, the row of the right key column it matches;
am_take and am_filter then gather both sides. That is a complete inner or left join
whenever the right key is unique -- the usual dimension-table shape -- and the whole thing,
uniqueness check included (one am_group_by_keys: as many groups as rows means every key is
distinct), runs on the GPU with no row-by-row work on the host.
Everything else falls back to pl.DataFrame.join: a duplicated right key (which changes the row
count in a way index_in cannot express), a multi-column key, or an outer/semi/anti join. Pass
allow_cpu_fallback=False to get an error instead, so a benchmark can be sure what it measured.
Null keys never match, which is Polars' default join_nulls=False.
query imports only the columns the query names -- it reads them out of the serialised query's
(col "name") nodes. This matters more than it sounds: on the benchmark frame, importing all
four columns (one of them 50M strings) cost more than the entire query did.
Tier 2 -- the expression plugin
import polars as pl
import arrowmetal.polars_plugin # registers the namespace
lf.select(pl.col("amount").arrowmetal.sum())
lf.with_columns(pl.col("name").arrowmetal.upper())
lf.select(pl.col("amount").arrowmetal.filter_sum(pl.col("region") == 2))
lf.group_by("k").agg(pl.col("v").arrowmetal.sum())
These are real Polars expressions: they compose with select, with_columns, filter,
group_by, over, and they take part in projection and predicate pushdown. That is the whole
point of the tier -- tier 1 needs a materialised frame, tier 2 does not.
.filter_sum is the shape that pays for itself: one GPU compaction plus one reduction, with the
filtered column never crossing back into Polars.
What tier 2 accepts
Narrower than the bridge, and not the same list as the "Types" paragraph under Limits -- that one
is about what round-trips, which is a tier-1 and tier-3 question.
Everything else -- Boolean, the temporal types, Binary, Categorical, Enum, Decimal, List, Struct,
Null -- raises a Polars ComputeError whose message starts arrowmetal:. Nothing panics through
pyo3. There are two places tier 2 is behind tier 1. Categorical and Enum: the Python
bridge recodes Polars' dictionary<uint32> index buffer to int32 for the GPU, and
polars-plugin/src/bridge.rs does not, so a Categorical column reaches the kernel as-is and is
refused with "dictionary indices must be int32 or int64". And Decimal: am_decimal_op
backs s.arrowmetal.sum() in tier 1, and the plugin does not reach for it.
The scalar in .add / .sub / .mul / .truediv
"Integers wrap" is about the arithmetic: 127 + 1 is -128 on an Int8 column, and integer
division by zero is 0. It is not about the operand. A scalar the column's type cannot hold
exactly is an error, the same call the tier-1 bridge makes (it packs the scalar with
struct.pack at the column's own width, and struct.pack raises):
pl.col("i8").arrowmetal.add(1000) # raises: 1000 is out of range for an Int8 column
pl.col("u8").arrowmetal.add(-1) # raises: -1 is out of range for a UInt8 column
pl.col("i64").arrowmetal.add(1.5) # raises: an integer column takes an integer scalar
pl.col("i64").arrowmetal.add(2**60+1) # exact -- the scalar does not go through an f64
A float column takes either an integer or a float, and an operand too large for Float32 becomes an
infinity, which is what struct.pack("f", 1e300) gives tier 1.
group_by_sum, and what the plugin API cannot do
pl.col("k").arrowmetal.group_by_sum(pl.col("v")) returns a struct column of n_groups rows:
df.select(pl.col("k").arrowmetal.group_by_sum(pl.col("v")).alias("g")).unnest("g")
It has to be a struct because a plugin function answers with a single Series. And it runs as a
projection over the whole frame, not inside df.group_by(...).agg(...): Polars' expression
plugin API (polars.plugins.register_plugin_function) registers expressions, and has no hook
for contributing a hash aggregate to the group-by engine itself. The flags it accepts are
is_elementwise, changes_length, returns_scalar, cast_to_supertype,
input_wildcard_expansion and pass_name_to_apply -- none of them says "I am an aggregation the
group-by engine should call per group". df.arrowmetal.group_by(...) (tier 1) is the ergonomic
spelling; pl.col("v").arrowmetal.sum() inside .agg(...) also works and is called once per
group, which is the wrong granularity for a GPU on small groups.
How the plugin gets the data across
polars-plugin/src/bridge.rs, in both directions:
Series --rechunk--> polars_arrow::ffi::ArrowArray --am_import--> am_array (Metal-resident)
am_array --am_export--> ArrowArray --import_array_from_c--> Series
polars_arrow::ffi::ArrowArray and arrowmetal_sys::ArrowArray are both #[repr(C)]
transcriptions of the same C struct, so the hand-off is a pointer cast. Ownership follows the C
Data Interface's consumer-releases rule: am_import takes over the exported array (so the Rust
side mem::forgets its copy, exactly as the Python binding does after _export_to_c), and
import_array_from_c takes over the exported one coming back.
Strings go through CompatLevel::oldest() -- Arrow LargeUtf8, not the Utf8View layout Polars
uses natively. That conversion is the plugin's only copy, and it is why the plugin's string row
below is 2.3x where tier 1's, which takes the column as views, is 17.6x. Tiers 1, 3 and 4 hand
String columns over as Utf8View, which the kernels read directly (the Strings paragraphs under
Numbers and Limits).
arrowmetal-sys is a hand-written transcription of the header, not bindgen output: the surface
is small, the header is stable, and a checked-in file needs no libclang on the build machine.
cargo test in polars-plugin/arrowmetal-sys runs 10 tests against the real dylib (import,
export, reductions, compare + filter, scalar arithmetic, top-k + take, hash64, group-by, and the
error path).
Tier 3 -- the streaming hand-off, and Polars' engine hook
lf = pl.scan_parquet("trades/*.parquet").filter(pl.col("day") == "2026-09-01")
lf.arrowmetal.collect_gpu(am.filter(am.col("region") == 2).sum(am.col("amount")))
lf.arrowmetal.collect_gpu(lambda df: df.arrowmetal.group_by("k").sum("v"))
lf.arrowmetal.collect_gpu() # just lf.collect()
collect_gpu collects the Polars plan and then runs one ArrowMetal pass over the result. Its
engine= and any other keyword go straight to LazyFrame.collect, so the Polars half can still
use the streaming engine: projection pushdown, predicate pushdown and slice all happen before a
single byte reaches the GPU. It is an explicit hand-off. Polars' own engine= hook is the other
way in, and tier 4 uses it; this section is what the hook is, read from the installed package.
The engine= hook, as it stands in polars 1.44.1
Read from the installed package, not from memory:
polars/_typing.py:443 --
EngineTypeName: TypeAlias = Literal["auto", "in-memory", "streaming", "gpu"], and
EngineType: TypeAlias = Union[EngineTypeName, "Engine"]. So collect(engine=...) does accept
an Engine object, not only a name.polars/lazyframe/engine.py:77 -- class Engine(ABC), documented as "Subclass this to plug a
new backend into Polars", with an abstract name property, a plan_engine property, and
collect / execute / collect_async / collect_batches / the sink_* family.polars/lazyframe/engine.py:330 -- class _LocalEngine(Engine), "Base for in-process engines
backed by PyLazyFrame". Its collect ends in wrap_df(ldf.collect(self.name, callback)).- The callback comes from
_LocalEngine._post_opt_callback(*, background, eager), typed
PostOptCallback | None where PostOptCallback: TypeAlias = Callable[[Any, int | None], None]
(polars/_typing.py:450). The base returns None.
polars/lazyframe/engine.py:884 -- class GPUEngine(_LocalEngine) with _name = "gpu". Its
_post_opt_callback imports cudf_polars and returns
partial(cudf_polars.execute_with_cudf, config=self). It refuses background collection and
opts out in eager mode.polars/lazyframe/engine_config.py:28 --
SUPPORTED_ENGINE_NAMES = ("auto", "in-memory", "streaming", "gpu"), and _engine_from_name
maps the string "gpu" to GPUEngine().- The callback's first argument is a
NodeTraverser (polars/_plr.pyi:2595), whose surface is:
get_exprs(), get_inputs(), version(), get_schema(), get_dtype(expr_node),
set_node(node), get_node(), set_udf(function, is_pure=False), view_current_node(),
view_expression(node), add_expressions(expressions), set_expr_mapping(mapping),
unset_expr_mapping().
So a Metal backend is not blocked on Polars adding an API -- the API is there, and tier 4 below
is built on exactly this surface. One fact about names: the engine's name is passed to Rust as
ldf.collect(self.name, callback), and Rust only knows the four in SUPPORTED_ENGINE_NAMES (a fifth
string raises ValueError). When a callback is supplied, Rust invokes it for any known name,
"in-memory" and "streaming" included (checked on 1.44.1), and an Engine object passed to
collect(engine=...) bypasses the Python-side name check. So a third-party engine runs by passing
"in-memory" to Rust and reporting itself through plan_engine; what it cannot do is carry its own
name through Rust, so explain and the callback's error message ('cuda' conversion failed) name
the wrong engine.
collect_gpu stays the explicit form of the same idea: Polars
owns the plan, ArrowMetal owns one pass over the result.
import polars as pl
import arrowmetal as am
engine = am.MetalEngine()
df = lf.collect(engine=engine) # the same frame lf.collect() returns
print(engine.last_report) # which nodes ran on Metal, and why the rest did not
MetalEngine is python/arrowmetal/polars_engine.py. Polars optimises the plan as it always does
and hands the optimised IR to the engine's post-optimisation callback. The callback walks every
node and expression, translates the subtrees it can into an ArrowMetal plan (the grammar in
Query engine), and replaces each one with a function that runs that plan on the GPU and
returns a Polars DataFrame. Everything it does not take stays with Polars' in-memory engine
(on polars 2.0.0 too, where a plain collect() runs the streaming engine), which also runs whatever
sits above a replaced subtree. Every plan collects; the most that can
happen is that nothing moves, and then the answer is plain Polars'.
import arrowmetal still does not import Polars: am.MetalEngine loads the module on first touch,
like the other three tiers.
How it plugs into Polars 1.44.1 and 2.0.0
Read from the installed package (polars 1.44.1, and where it says so 2.0.0) and checked by
python/tests/test_polars_engine.py, which passes on both:
lf.collect(engine=<an Engine object>) passes the object through unchanged, so no Polars change
is needed. The name the engine gives Rust is "in-memory": Rust accepts only its four engine
names, and with a callback supplied it runs the callback for any of them; the in-memory engine is
also what runs every node the callback leaves. engine.name is "in-memory", engine.plan_engine
and repr(engine) say metal.- The callback receives the
NodeTraverser and, on polars 1.44.1, a second argument that is None
under collect and an integer under profile (the time since the query started, which the
engine uses to place its rows in the profile). polars 2.0.0 passes the NodeTraverser alone.
- On polars 2.0.0
collect() without engine= runs Polars' streaming engine (on 1.44.1 the
in-memory engine). MetalEngine gives Rust the name "in-memory" on both, so the nodes it leaves
run on the in-memory engine.
set_udf turns the current node into a PythonScan whose function Polars calls as
f(with_columns, predicate, n_rows, should_time). That function takes no input, so a replaced
subtree is a leaf: the only subtrees that can move are ones whose leaves are in-memory frames
(DataFrameScan) or Parquet files the engine reads itself (Scan, below). Any other scan
(PythonScan, a CSV or IPC Scan) stays with Polars, and so does everything above it.view_current_node raises NotImplementedError: ipc scan for a scan_ipc node. The engine
leaves such a node, and everything above it, to Polars and names it in the report.- Polars does not check the replacement's output. The engine does: a frame whose schema is not
the one
get_schema() promised raises ArrowMetalError inside the query.
- An exception from the callback reaches the user as
ComputeError: 'cuda' conversion failed: <Type>: <message>; the 'cuda' is hardcoded in Polars.
The engine's own messages start with ArrowMetal MetalEngine: so they read correctly inside it,
name the node (Sort#3) and the reason, and end with the way to run the plan on Polars instead
(FORCE_POLARS: collect without engine=, or set ARROWMETAL_METAL_ENGINE=off).
LazyFrame.profile(engine=...) passes the callback only for a GPUEngine, so
lf.profile(engine=MetalEngine()) profiles plain Polars. engine.profile(lf) passes the callback
through profile's own keyword, and each replaced subtree appears as a metal:<Node>#<id> row.
Polars 2.0 removed LazyFrame.profile; on polars 2.0.0 engine.profile(lf) raises
NotImplementedError.- Only
collect and the paths built on it run the callback; the next section lists every path and
what the engine does on it.
- The IR versions the engine was written against,
TESTED_IR_VERSION = ((14, 7), (15, 2))
(polars 1.44.1 and 1.44.2 report (14, 7), polars 2.0.0 reports (15, 2)), are pinned by a test, as
is every Polars surface it touches (_LocalEngine, _post_opt_callback, the NodeTraverser
methods, the node classes), so a Polars upgrade that moves one fails a named test instead of
changing an answer. An IR major that neither tested version has makes the engine leave the whole
plan to Polars; a newer minor of a tested major runs and warns once, naming the tested version of
that major. A node kind outside KNOWN_NODE_KINDS (the 20 node classes of polars 1.44.1; polars
2.0.0 has 19, the same less ExtContext, since 2.0 removed LazyFrame.with_context) or a node
Polars fails to show to the engine also keeps the whole plan with Polars: the report line is
The plan holds an unknown node <kind> in polars <version>, so the whole plan stays with
Polars., and the query is not an error.
ARROWMETAL_METAL_ENGINE=off (or 0, false, no, polars) makes every MetalEngine leave every
plan to Polars, raise_on_fail=True included; the report says so.
Collect paths
What each Polars 1.44.1 entry point does with a MetalEngine (eng below), read from the installed
package and checked by test_every_collect_path_is_explicit and
test_polars_side_entry_points_that_never_call_the_engine. last_report.path names the path; a
path on which the whole plan runs on Polars says why in last_report.fallbacks and issues a
MetalEngineFallbackWarning (a UserWarning) once per process. On polars 2.0.0 the same tests
pass apart from the two profile rows, which they skip: Polars 2.0 removed LazyFrame.profile, and
eng.profile(lf) raises NotImplementedError there.
collect(background=False) with Polars' eager optimisation flag set asks for no callback either;
the report says eager and there is no warning, as for GPUEngine. For code that cannot change
its collect calls, ARROWMETAL_METAL_ENGINE=off sends every path to Polars.
What it translates
Polars engine capabilities is the same boundary measured on polars 1.44.1, and ENGINE_CAPABILITIES_POLARS2.md the same generator run on polars 2.0.0 (the engine takes the same 1,326 of 2,523 plans; 63 plans that 1.44.1 accepted are rejected by 2.0.0 itself and show as n/a): 29 plan shapes
(filters, projections, with_columns, slice, sorts, top-k, group-by with x as the key and as the
value of each aggregate, whole-frame aggregates, the four joins, unique, and Parquet scans) over 29
input dtypes and three null patterns, each run through the engine and marked Metal or Polars
with the engine's reason. It is generated by python/tests/engine_capabilities.py, and
test_the_capability_table_is_current fails when the committed file differs from a fresh run.
Everything else -- %, //, eq_missing, str.ends_with, regex, windows (over), rank,
median, an expression over an aggregate, a String-valued output, a narrowing or fallible cast,
true division by a scalar that is not a plain literal,
a column name holding a NUL byte (Polars' own Arrow export panics on one) -- falls back, and the
report says which one. Categorical, Enum, Decimal, List, Struct, Null, Binary
and Object columns in a subtree's input keep the whole subtree on Polars.
Where the answers would differ, and what the engine emits instead
Each line is a differential case in test_polars_engine.py or test_engine_conformance.py, run
against Polars itself.
- Float comparisons. Polars compares floats in a total order: NaN equals NaN and is greater than
every number, and -0.0 equals 0.0. The engine adds the NaN terms (
(ne x x) is "x is NaN") so the
fused comparison gives Polars' answer, nulls included. is_in matches the same way: a NaN in the
value list becomes an (ne x x) term, since ArrowMetal's is_in compares with IEEE equality.
- Float32 arithmetic. The GPU's float adds, multiplies and divides flush subnormals to zero,
which Polars does not. The engine computes Float32
+ - * / in ArrowMetal's correctly rounded
software binary64 and rounds once back to Float32, which is the correctly rounded Float32 result
for these four operations, subnormals included. Float64 + - * / is correctly rounded in both.
- Division by a scalar. Polars divides a column by a scalar as
x * (1 / c), with the
reciprocal rounded in the result type; that differs from the correctly rounded x / c by at most
one ulp, in a share of rows that depends on the divisor (about a third of Float64 rows for / 3.0,
none for a power of two). The engine emits the same multiply, so the bits match Polars'
(test_true_division_by_a_literal_is_polars_reciprocal_multiply, which also checks zero, infinite,
NaN, subnormal and null divisors). A column divisor is a true division in both. Over a column of
one row Polars divides element-wise instead (the scalar and the column have the same length),
so there the answer is the correctly rounded x / c. The engine emits whichever of the two Polars
computes for the row count of the node's input: from the in-memory frame when nothing between it
and the division changes the row count by an unknown amount, and otherwise by counting that input
when the plan runs, both forms in the plan and the count choosing between them
(test_scalar_division_and_minus_one_follow_polars_at_every_length).
- Multiplying by -1. Polars multiplies a float column by a scalar -1 (on either side, and divides
by -1) as a negation, which flips a NaN's sign bit where a multiply keeps the input NaN. The engine
emits ArrowMetal's
negate there, so NaN rows carry Polars' bits too
(test_multiply_by_minus_one_is_a_negation_like_polars, which compares the raw bits). Over a
column of one row Polars multiplies, and the NaN keeps its sign; the engine chooses by the row
count as for a division.
- The streaming engine. The two forms above are those of Polars' in-memory engine, the one
MetalEngine leaves the rest of a plan to, in polars 1.44.1 and 2.0.0 alike. Polars' streaming
engine, the default of collect() on 2.0.0, evaluated frames of 2 and 9 rows element-wise on both
versions (the correctly rounded x / c, and a NaN that keeps its sign under * -1), and frames
of 100 rows and more as the in-memory engine does.
- Aggregates. A
sum over no values is 0 in Polars (ArrowMetal: null) and gets a fill_null; a
min/max over only NaN is NaN in Polars (ArrowMetal: null over a whole frame, an infinity per
group), so the engine counts the non-null and non-NaN values and decides from the two; a mean of
an Int64/UInt64 column is planned over the values cast to Float64, because ArrowMetal's integer mean
sums in 64-bit integers and wraps on extreme values where Polars does not (the executor runs it from
the integer column in one pass when a min/max pass shows every value below 2^53 and the sum below
2^63, and over the cast values otherwise; the two give the same bits); every result is cast
to Polars' dtype (UInt32 counts, the Int32 sum of an Int32 column, Float32 of a Float32, UInt32
for the sum of a Boolean). A per-group count of a Float64 or Boolean column is the sum of its
validity bits, because ArrowMetal's group-by will not read those values even to count them, and
min/max of a Float64 column per group stays with Polars for the same reason. Polars' min
and max order -0.0 below 0.0, so a min over both zeros is -0.0 and a max 0.0; ArrowMetal
treats the two as equal and returns whichever it met first over a whole frame, and 0.0 per group.
The engine counts the zeros of the sign Polars prefers and takes the sign from that count
(test_min_and_max_over_both_zeros_are_polars_signed_zeros).
- Float sums and means. The one place the answers differ, and the engine leaves it: a Float32
or Float64
sum, and a Float64 mean, add the same values in the GPU's order where Polars adds in
its own, so the two can differ in the last bits. Both are sums of the same values, so they differ
by at most twice the rounding bound of a sum, 2(n - 1) u sum(|x|) for a sum of n values and
2 u sum(|x|) for a mean (u the unit roundoff of the type the sum accumulates in: 2^-24 for a
Float32 sum, 2^-53 otherwise), and the engine conformance grid holds each such case to that bound.
In the run recorded in Benchmarks/results/engine_conformance_2026-10-02_report.txt (totals in
engine_conformance_2026-10-02.csv) the largest difference was 0.199 u sum(|x|) for a Float32 sum
(3.55e-5 of the answer), 0.372 u sum(|x|) for a Float64 sum (2.00e-14 of the answer) and
0.0033 u sum(|x|) for a Float64 mean (7.05e-14 of the answer). A mean of an integer or Float32
column is computed in Float64 by both (the engine plans the cast as Polars does, and runs an
integer mean from the integer column where that gives the same bits); in that run
it matched bit for bit in every case but one int64 group mean over the integer extremes (1,000
rows), where the engine returns the correctly rounded mean of the cast values and the two differ
by 8.76e-8 u sum(|x|), as much as the answer itself. Every other output of the grid is compared
bit for bit. test_polars_engine.py compares these
aggregates to a relative tolerance.
- Sort order. Polars (1.44.1 and 2.0.0 alike, pinned by
test_polars_sort_order.py) places the nulls per key, first unless nulls_last, in either
direction; orders a float key with every NaN (of either sign) one value above every number, +inf
included, in both directions, and -0.0 equal to 0.0; keeps tied rows in input order in both
directions under maintain_order=True, a tie falling through to the next key; orders String keys
byte-wise and Boolean keys False first. top_k is a descending sort with nulls_last=True and a
slice, bottom_k the ascending one. The engine writes each Polars key as one ArrowMetal key whose
options give that order: "nulls": "first" where the nulls go first, and "float_order":
"nan_largest" on a float key (docs/ENGINE.md, "Sort key options"). No key column is added, and a
single-key sort with a slice stays a GPU top-k.
- Integer overflow wraps in both (checked on Int8 and Int64 extremes), and an integer true
division by zero is IEEE in both.
Each was first worked around here and pinned by a strict xfail; each is now fixed in ArrowMetal,
its test in test_polars_engine.py passes, and the workaround is gone
(test_engine_takes_the_plans_it_once_worked_around runs the plans the engine once changed or
declined):
- A null String slot with bytes under it. Polars exports a null String value with the bytes the
slot held (valid Arrow), and ArrowMetal's String gather copied those bytes over the next kept
value (
test_core_string_filter_null_slot). The engine now hands such a column over as exported.
- A Boolean column through the plan's sort came back with the right validity bitmap and a null
count of 0 (
test_core_bool_sort_null_count). Result columns are now used as ArrowMetal returns
them.
- A filter over a scan that carries a
date32 column was rejected by the expression compiler
(test_core_filter_carrying_a_date). Temporal columns now go to ArrowMetal as their own types.
- Sorting by a String column that holds a null and a value of 8 bytes or more returned wrong
rows (
test_core_string_sort_with_nulls, run in a child process because one run ended in a bus
error). The engine now sorts by String columns.
- A finite float literal of magnitude 2^63 or more trapped the process inside the expression
compiler, which converted every float literal to Int64 as well. The engine now runs those plans
on Metal (
test_a_float_literal_of_magnitude_2_63_or_more_runs_on_metal).
- A UInt64 literal above 2^63 - 1 (found by the conformance grid below, not worked around first:
is_in, a comparison or fill_null against a large
unsigned value) was rejected by the expression parser, which read every integer literal as an
Int64, and the plan stayed with Polars. The parser now keeps such a literal as its bit pattern,
the way the code generator writes unsigned literals (test_a_u64_literal_above_int64_max_reaches_the_gpu,
and testWideUnsignedLiteralsParsePrintAndStayUnfolded in Swift).
Which translatable subtrees it runs: the defaults
A subtree the engine can translate still has to be one where the GPU is ahead, because getting a
Polars column onto the GPU is not free: a single-chunk numeric column is imported without a copy,
but mapping its pages into Metal and releasing them costs time on every query. String columns are
handed over in Polars' own view layout (see "Strings" under Limits), in the crossover sweep and in
the benchmark of the defaults below alike.
MetalEngine() (shapes="measured") decides per subtree from measured crossovers (python/arrowmetal/_engine_policy.py). Each translated subtree has shape classes:
rowwise -- filters and projections only;aggregate:<family> (a whole-frame aggregate), group_by:<family> (one key) and
group_by_multi:<family> (two or more keys), the families being sum, count (count and
len), mean and minmax;sort (a full sort) and top_k (a sort with a slice);join:inner, join:left, join:semi, join:anti, and distinct (unique).
It also has a dtype class, string when a String column is among the columns it reads and numeric
otherwise, and an input, in-memory frames or a Parquet file. A subtree runs on Metal when its input
rows (the rows of its in-memory frames, or of its Parquet file as the footer states them; what a
predicate will keep is not estimated) are at or above the crossover of every class in it for its
dtype class and input; a group-by is judged at the bucket of its estimated number of groups
("Group-bys: the number of groups", below). A class whose row says not taken stays with Polars at
every size. The decision reads nothing but the subtree's classes, dtypes, input rows and input, a
group-by's estimate and the tables, and the estimate comes from a fixed-seed sample, so the same
subtree over the same frame always gets the same answer (python/tests/test_engine_policy.py checks
both across 200 calls and across processes). Where the policy leaves a node, the placement moves down to its inputs, so a
smaller subtree below it can still be taken: under a whole-frame sum over an inner join, the join
runs on Metal and Polars adds up its output.
The sweep, the fits, the tables and the benchmarks from here to On polars 2.0.0 below are polars
1.44.1's, the measurements behind the Polars 1.x table; the Polars 2.x table and its measurements are
under On polars 2.0.0.
The crossovers. Benchmarks/polars_engine_bench.py --crossover ran 93 LazyFrames over in-memory
frames, the 45 cases below and a group-by grid of 48 (each aggregate family over one int32 key and
over two, the keys drawn from 200, 1,000, 10,000, 100,000 or 1,000,000 values or from half the
frame's rows, (g1sum200) to (g2minmaxR2)), at 250,000, 500,000, 1,000,000, 2,000,000, 5,000,000,
10,000,000, 20,000,000 and 50,000,000 rows (the probe side of the joins grows with the size, the
build side is 1,000,000 rows), and the four Parquet scan cases over files of 1,000,000, 2,000,000,
5,000,000, 10,000,000, 20,000,000 and 50,000,000 rows, snappy and uncompressed, each through Polars'
in-memory and streaming engines and through MetalEngine(shapes="all", min_rows=0) cold, best of 7,
every MetalEngine result equal to Polars': Benchmarks/results/polars_engine_crossover_2026-09-26-final.csv,
run conditions (the load average, sampled each minute) in
Benchmarks/results/bench_conditions_2026-09-26-final.txt. Each group-by row of the results file also
records the number of groups the case's data holds at that size (groups). The in-memory sort and
top-k cases ((b), (m), (n), (o), (p), (q), (x2), (x3), (y6), and (x4), a top 100 by a nullable Float64
key descending, added then) were swept again on 2026-09-29, after each Polars sort key became one plan
key with Polars' null placement and float order as its options (see "Sort order" above), at the same
eight sizes, best of 7: Benchmarks/results/polars_engine_crossover_2026-09-29-sort.csv holds those rows
and the 2026-09-26 rows of every other case, run conditions in
Benchmarks/results/polars_engine_crossover_2026-09-29-sort_conditions.txt. The group-by cases that run
the grouped Float64 sum and mean or the grouped min/max ((c), (l), (t3), (t4), (t6), (t7), (v3), (v4)
and the grid's mean and min + max cases; a mean of an int64 column then ran as a Float64 mean, the engine
casting it first) were swept again on 2026-09-30, after both were rebuilt, at the same eight sizes,
best of 7, and the grid gained a Float64 sum over each of its key sets and group counts
((g1fsum200) to (g2fsumR2), a sum of q / 1e9): Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv
holds those rows and the earlier rows of every other case, run conditions in
Benchmarks/results/polars_engine_crossover_2026-09-30-groupby_conditions.txt, and the table is fitted
from it.
Benchmarks/polars_engine_crossover.py fits it the way Benchmarks/router_table.py fits the router
table (python/arrowmetal/_router_fit.py), with the MetalEngine as the GPU side and the faster Polars
engine as the CPU side, over each case's input rows. A case's fit is the first size from which it is
ahead at every larger size, placed between that size and the one below it where the two straight
lines meet; a case ahead at the largest size alone has none, and a size where the MetalEngine's answer
differed from Polars' counts as behind.
Four rules on the fit. The table keeps each case's and each bucket's fit as fit, and the rows
the default uses as rows:
- The margin (
MARGIN). A case is ahead at a size when its MetalEngine time, raised by a margin,
is at most the faster Polars engine's time: 15% for a numeric shape, 35% for a shape with a String
column. A String shape's advantage grows slowly with size: the String sort (o) is 0.98x the faster
Polars engine at 1,000,000 rows in the 2026-09-26 sweep (Benchmarks/results/polars_engine_crossover_2026-09-26-final.csv,
vs_fastest_polars), 1.2x at 2,000,000, 1.37x at 5,000,000 and 1.5x at
10,000,000, so a fit at 15% would land where the two engines are within noise of each other.
- The String floor (
STRING_FLOOR). A shape with a String column is taken from 5,000,000 rows
at the earliest, whatever its fit says. The floor is set from the 2026-09-26 runs, where the String
sorts below it are close to Polars: (o) 1.2x and (p) 1.38x at 2,000,000 rows in the 2026-09-26 sweep,
and 1.21x and 1.31x under shapes="all" at that size in the 2026-09-26 default benchmark
(Benchmarks/results/polars_engine_bench_2026-09-26-final3.csv, cold), inside that benchmark's noise
band (0.62x to 1.31x); from 5,000,000 rows that sweep has them at 1.37x and up. In the 0.4.0
benchmark below (Benchmarks/results/polars_engine_bench_2026-10-02.csv, cold) they are 1.39x and
1.84x under shapes="all" at 2,000,000 rows, against a noise band of 0.86x to 1.18x.
- The headroom (
HEADROOM). The default takes a shape from 1.5 times its fit (a String shape
from the larger of that and the floor), and a shape whose 1.5 times passes the largest size
measured is not taken. The fit interpolates between sizes measured 2 to 2.5 times apart
(250,000, 500,000, 1,000,000, 2,000,000, 5,000,000 and so on), and shapes just past their
crossover were within run-to-run noise of Polars: two runs of the same Polars plan differ by 0.62x
to 1.31x in the 2026-09-26 default benchmark and by 0.86x to 1.18x in the 0.4.0 benchmark below
(MetalEngine default, cold where it took nothing, against polars in-memory, rows of 5 ms or more
in Benchmarks/results/polars_engine_bench_2026-09-26-final3.csv and
Benchmarks/results/polars_engine_bench_2026-10-02.csv).
- The default benchmark (
BENCH_RATIO, BENCH). The sweep times each case back to back; the
default is also held to Benchmarks/polars_engine_bench.py --idle (a 100 ms warm-up, a first run
after 500 ms of idle on its own, another warm-up, then best of 5 and their median), over the files
BENCH names (Benchmarks/results/polars_engine_default_groupby_raw_2026-09-30.csv, "Group-bys
after the grouped Float64 sum, mean and min/max were rebuilt" below). A class, or a group-count
bucket for a case judged by its estimate, of which a case the default took there is behind the
faster Polars engine on the best run or on the median is taken only from the smallest benchmarked
size above the largest size it was behind at, and not at all when it is behind at the largest. A
case of several classes (a mean and a max in one group-by) that is behind counts against those of
its classes whose one-class cases at that bucket and size are not all ahead, and against all of
them only when every one of its classes has one-class cases there that are ahead, so a class that
is ahead on its own keeps its rows and the case itself is still left. The table records each
benchmarked row's cases and their ratios as benchmark.
A class's crossover is the largest over the cases whose classes all belong to its node, so every case
that measures a class has to be ahead; if one of them has no fit, the class has no crossover. A case
whose classes span two nodes, (b) (a group-by under a top-k) and (e) (a whole-frame sum over a join),
measures no class. There is one fitted table per Polars major, chosen by the running Polars' major
the first time the policy decides, and python -m arrowmetal.polars_engine check names the one in
force (compatibility()["crossover_table"]):
Each module records its own SOURCE, BENCH and the sweep's HEADER, which names the Polars
version the sweep ran on; polars_engine_crossover.py writes a major's module only from a sweep on
that major, and polars_engine_crossover.py --check fails when any module and its results files
disagree. The policy then
takes the larger of that crossover and the kernels' own: the router table in force
(am.router_table(), The crossover) for the kernels it routes, and for the sort
classes the crossover of argsort int64, argsort float64 and lexsort (2 int32 keys) against the
fastest CPU library in Benchmarks/results/router_2026-09-24.json, 1,000,000 rows. The fits below are
the Polars 1.44 table's, from Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv.
A class with no row (a String column in any other class, and every Parquet class but these) has no
measurement and is not taken. arrowmetal.polars_engine.placement_rules() returns the table under the router
table in force, and group_placement_rules() the group-count buckets below.
Group-bys: the number of groups. Where the GPU group-by is ahead depends on the number of groups
more than on the rows. In the sweep (Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv,
vs_fastest_polars of MetalEngine all, cold) the one-key sum over 200 groups is behind at every size ((t1) at
best 0.58x, (g1sum200) 0.51x); over 10,000, 100,000 and 1,000,000 groups it is ahead at 50,000,000
rows by 2.65x, 2.97x and 2.74x ((g1sum10k), (g1sum100k), (g1sum1M)); over about 0.43 times as many
groups as rows it is at 1.02x there ((g1sumR2)), inside the margin. So a group-by class is judged at
the bucket of its subtree's number of groups:
A count between the 1,000,000 bucket and a quarter of the rows has no bucket and stays with Polars.
Each bucket is fitted like a class, from the points of the group-by cases whose number of groups at
that size falls in it (the grid, (c), (i), (j), (k), (l), (t1) to (t7), (v1) to (v4), (y1), (s1) and
(s2)); at each size the bucket's point is the worst of its cases there, so its step is the largest of
the cases' own, and a bucket ahead at its largest size alone has no crossover. The grid's
1,000,000-group cases hold a quarter of the rows or more below 5,000,000 rows, where they count in the
rows/2 bucket, so the 1,000,000 bucket was measured from 5,000,000 rows, and that is where its fits
start. The rows the default uses, with the fit each came from in brackets (the router table's
group_by_sum row is below every one of them):
An empty cell has no measurement and is not taken. * Raised by the default benchmark (BENCH_RATIO;
Benchmarks/results/polars_engine_default_groupby_raw_2026-09-30.csv, the faster Polars engine ÷ the default over
every run in which the default took the case): at 2,000,000 rows the default took (c) (a mean and a max over two keys, 10,000 groups) at 0.82x of
the faster Polars engine on the best run and 0.75x on the median, (g2mean10k) at 0.87x and 0.75x
and (g2mean100k) at 0.65x and 0.57x (both tables' runs where the default took them), all ahead at
50,000,000 rows, so the two-key mean's 10,000 and 100,000 buckets are taken from 50,000,000 rows.
(c)'s min + max class keeps its 10,000 bucket: (g2minmax10k), its one-class case there, was 1.18x and
1.09x. The rows/2 bucket is never taken
(UNTAKEN_BUCKETS), whatever its fit, because the sweep (Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv) is not
monotone there: the one-key count over
0.43 times as many groups as rows, (g1countR2), is 0.78x, 1.9x, 4.04x, 2.21x and 1.18x the faster
Polars engine at 2,000,000, 5,000,000, 10,000,000, 20,000,000 and 50,000,000 rows, and the two-key
rows/2 cases peak at 10,000,000 rows the same way ((g2countR2) 6.47x there, 2.38x at 50,000,000).
(j), (v3) and (v4), whose keys range over 100,000 x 1,000 values and which hold 0.79 to 1.0 times as
many groups as rows in the sweep, are 1.22x, 1.17x and 1.51x at 50,000,000 rows. The grid's Float64
sums set two sum buckets: the one-key 10,000 bucket ((g1fsum10k) 1.04x at 2,000,000 rows and 1.25x
at 5,000,000, then 2.63x at 10,000,000) and the two-key 1,000 bucket ((g2fsum1k) 0.74x at 5,000,000
and 1.88x at 10,000,000). The one-key min + max over 200 and 1,000 groups is taken from about
20,000,000 rows: (g1minmax200) is 1.13x, 1.41x and 1.67x at 10,000,000, 20,000,000 and 50,000,000
rows, (g1minmax1k) 1.06x, 1.27x and 1.64x.
The probe. For a group-by whose keys are columns of one in-memory input frame, read as they are
(through filters, projections, joins and unique below it, which can only drop key values, so the
frame's count is an upper bound), the engine estimates the number of distinct key tuples in that
frame at plan time. It reads the keys of a stratified sample of n of the frame's N rows, row
i·(N / n) + h(i) with h a fixed-seed hash, so the same frame always gets the same sample; counts the
distinct tuples d and those seen exactly once (f1) and twice (f2); and scales them with the
bias-corrected Chao1 estimator (Chao, Biometrics 2005), D = d + f1 (f1 - 1) / (2 (f2 + 1)), clipped
to [d, N]. Its range is D with f2 moved by two Poisson standard deviations (and one or two more)
either way, the high end being the whole frame while fewer than about 6 pairs have been seen. The
samples are 512, 2,048, 8,192 and so on, at most 65,536 rows and a quarter of the frame, and the
probe stops at the first whose range the decision does not depend on: every bucket the range reaches
is taken at these rows, or none is. So a few hundred groups or ten thousand settle on 512 rows, a
hundred thousand on 2,048, and a count near a quarter of the rows takes more. A frame of at most
4,096 rows is counted exactly. One integer key is read as it is; several keys, and a String, Boolean
or temporal key or one holding nulls, through Polars' hash. The samples' counts are cached per frame,
key columns and sample size, keyed by the columns' buffers (the cache keeps those columns alive, so no
other frame can take their addresses while it stands), so collecting the same frame again samples
nothing and gets the same estimate; arrowmetal.polars_engine.clear_group_estimates() drops the
cache. The probe runs only when some bucket of the class is taken at the input rows; below every
bucket's crossover the subtree stays with Polars without one. A Parquet file's footer answers when
it states a distinct count for every key column in every row group, and only when the largest row
group's count and the sum over row groups fall in one bucket; the benchmark files state none. With no estimate (a key the plan computes, keys
from an aggregate below, two group-bys in one subtree, a footer without distinct counts) a group-by
class is judged by its row in the class table: group_by_multi:minmax from 11,675,406 rows,
group_by:minmax from 19,740,807, group_by_multi:mean from 24,143,946, group_by_multi:count from
26,547,327, the (String, int32) sum from 9,744,372, and every other numeric group-by class at no size.
What the probe costs, and how close it gets. Benchmarks/group_probe_bench.py collects a
group-by sum over the grid's frames (one int32 key, two, and one String key, at each group count)
through MetalEngine() nine times with its caches cleared and reads the probe's time from the
report: Benchmarks/results/group_probe_2026-09-26.csv, run conditions in
Benchmarks/results/group_probe_2026-09-26_conditions.txt. Against the fastest of Polars' two engines
and the default for the same group-by:
That file was measured under the earlier group-count table. Under the table in force, a
group-by at 2,000,000 rows is probed only for the two-key count and min/max, the classes with a
bucket taken at that size; a one-key, String or two-key sum group-by there is not probed,
and neither is a String group-by at 50,000,000 (only the (String, int32) sum has buckets). In the
default benchmark below, where the probe runs in a process that holds every case's frames
(probe_us in Benchmarks/results/polars_engine_bench_2026-10-02.csv, the probes of one collect summed), its median
is 150.5 µs at 2,000,000 rows (2.90% of the default's time for the case; 1 of the 14 probes under 1%)
and 126 µs at 50,000,000 (0.75%; 51 of 77 under 1%). The seven probes over 5 ms, 6.9 to 12.7 ms, are
all at 50,000,000 rows on frames with 0.43 times as many groups as rows or more ((g1minmaxR2) 6.9 ms,
1.41% of the default's time; (g2sumR2) 12.7 ms, 1.99%), and its largest shares are 15.96% at
2,000,000 rows ((g2minmax100k), 390 µs of 2.4 ms) and 14.60% at 50,000,000 ((g2sum1M), 3.8 ms of
26.2 ms). A second collect of the same frame reads its
samples from the cache. Sampled until its denominator settles instead of until the decision does (free_estimate in Benchmarks/results/group_probe_2026-09-26.csv), the
estimate falls in the bucket of the true count for every frame and key set of the grid at both sizes,
within 29% of it up to 1,000,000 groups at 50,000,000 rows, and 24% to 31% above it at half the rows.
Each probe is listed in the report, and the rule of a group-by the policy judged by its estimate
names it:
metal: GroupBy#1 [GroupBy > DataFrameScan] over 50,000,000 rows, ran in <t> ms -> 100,000 rows
rule: 50,000,000 input rows is at or above the 2,961,043-row crossover for group_by:minmax at an estimated 102,981 groups over (k1), a Chao1 estimate from a 2,048-row sample, 67,758 to 198,392 (engine table, the 100,000-group bucket, ...)
groups: GroupBy#1 over (k1): 102,981 groups, a Chao1 estimate from a 2,048-row sample, 67,758 to 198,392 (probed in 132 us)
polars: GroupBy#1: rule: estimated 191 groups over (region), a Chao1 estimate from a 512-row sample, 189 to 196: below the measured band for group_by:sum at 50,000,000 input rows (taken at 3,163 to 3,162,277 groups; no count in the estimate's range is; ...)
groups: GroupBy#1 over (region): 191 groups, a Chao1 estimate from a 512-row sample, 189 to 196 (probed in 76 us)
The other classes in the sweep (Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv, vs_fastest_polars
of MetalEngine all, cold):
- Joins and
unique on numeric keys: the left join is ahead from the smallest input measured,
1,250,000 rows, and taken from 1,875,000; the inner join is fitted at 1,353,218 and taken from
2,029,827, the anti join at 2,485,306 and from 3,727,959. The semi join is not taken: against a
1,000-row table, (f), it is behind at every size (at best 0.47x), though against a 1,000,000-row
table, (w3), it is fitted at 2,464,828. unique is fitted at 1,252,378 rows over 10,000 groups (r)
and at 3,662,727 over about as many groups as rows (x1), and taken from 5,494,090.
- Sorts over in-memory frames are fitted at 250,000 rows (x2, the smallest input measured; 1.42x
there), 456,986 (m) and 631,891 (q). The sort kernels' crossover, 1,000,000 rows, is above (q)'s
947,836 and sets
sort.
- Top-k is fitted at 6,600,767 rows by a Float64 key descending (n; 1.86x at 10,000,000 rows,
1.81x at 50,000,000) and at 6,210,387 by an int64 key (x3; 1.1x to 1.35x from 5,000,000), but the
top 100 by a nullable Float64 key descending (x4), whose answer is 100 of the null rows, is ahead at
no size (0.75x to 1.05x), so
top_k is not taken.
- Whole-frame aggregates and row-wise filters and projections are behind at every size:
(a) to (a4) at 0.09x to 0.24x, (d) at 0.14x to 0.42x.
- String shapes: the sorts (o) and (p) and
unique (y5) are taken from the 5,000,000-row
floor, and the (String, int32) group-by (k), about 1,000,000 groups from
5,000,000 rows, from 9,744,372. The filter by a String equality (y2, at best 0.13x), the prefix
filter and sum (y3, 0.5x), the join on a String key (y4, 0.67x) and the top-k with a String column
(y6, 0.88x) are behind at every size; the group-by on a String key with 1,000 values (y1) is ahead
by the margin at no size (1.27x at 20,000,000 rows, 1.09x at 50,000,000).
- Over a Parquet file, cold (the open-file cache cleared before each run), the sort is ahead from
the smallest file measured, 1,000,000 rows, with both codecs (4.37x uncompressed and 4.9x snappy
there), and taken from 1,500,000. The filter and group-by (s1) has no fit with either codec
(uncompressed at best 1.09x, at 50,000,000 rows; snappy 1.42x and 1.26x at 1,000,000 and
2,000,000, then 0.79x to 0.92x), nor has the aggregate (s3) (at best 0.84x); the filter that skips
row groups, (s2), is ahead at every size snappy (1.24x to 2.48x) and at none uncompressed (at best
0.99x). So the group-by and aggregate classes over a file are not taken.
The default against Polars and against shapes="all". Benchmarks/polars_engine_bench.py --scan
over the 106 in-memory cases at 2,000,000 and 50,000,000 rows and the four Parquet scan cases over the
50,000,000-row files, snappy and uncompressed, 220 case-size pairs, best of 5, every result equal to
Polars' (the cold runs of MetalEngine() clear the group-count cache too, so each pays its probe):
Benchmarks/results/polars_engine_bench_2026-10-02.csv, run conditions in
Benchmarks/results/bench_conditions_2026-10-02.txt. On polars 1.44.1 the default takes a subtree in
75 pairs (11 at 2,000,000 rows and 64 at 50,000,000), 55 of them group-bys, and every one is ahead of
the faster Polars 1.44.1 engine, from 1.52x ((g1minmax200) at 50M) to 11.54x ((r) at 50M):
In the other 145 pairs the default takes nothing and runs Polars' in-memory plan; there its time is
0.40 to 1.42 times the polars in-memory row of the same case (wall_ms in
Benchmarks/results/polars_engine_bench_2026-10-02.csv), and 0.86 to 1.18 times in the 112 of
them where that row is 5 ms or more. That is the spread of two runs of the same Polars plan in this
run, and the noise band the ratios above are read against. shapes="all" takes a subtree and is
ahead of the faster Polars engine in 64 of those 145 (vs_fastest_polars of MetalEngine all, cold),
47 of them above 1.18x, each left by the default because its class or bucket has a crossover above
that size or none:
- at 2,000,000 rows: two-key min + max and count over about 1,000,000 groups, estimated above every
bucket taken for their class at that size, (g2minmaxR2) 3.07x, (g2minmax1M) 2.54x, (g2countR2)
1.68x and (g2count1M) 1.64x; group-bys below the lowest crossover of their class, (g2fsum100k)
2.47x, (g2sumR2) 1.80x, (g2fsumR2) 1.61x, (g2sum10k) 1.58x, (v1) 1.57x, (g2sum100k) 1.51x,
(g2fsum1M) 1.48x, (g2sum1M) 1.27x, (g2fsum1k) 1.26x and (g2fsum10k) 1.22x (
group_by_multi:sum
from 2,288,101), (g2mean100k) 2.41x, (g2mean10k) 2.05x, (l) 2.00x, (c) 1.98x, (g2meanR2) 1.67x and
(g2mean1M) 1.65x (group_by_multi:mean from 7,500,000), (g1minmaxR2) 1.80x and (g1minmax1M) 1.56x
(group_by:minmax from 2,184,147), (g1fsum10k) 1.75x and (g1fsum100k) 1.35x (group_by:sum from
5,252,956), (g1mean10k) 1.57x and (t6) 1.31x (group_by:mean from 2,436,520) and (g1count10k)
1.35x (group_by:count from 3,123,562); unique (r) 2.07x and (x1) 1.24x (below 5,494,090); the
String shapes (p) 1.84x, (o) 1.39x and (y5) 1.31x (below the 5,000,000-row floor);
- at 50,000,000 rows: group-bys estimated at 11,180,716 to 16,775,168 groups, above every bucket
taken for their class, (g2minmaxR2) 2.26x, (g2countR2) 2.11x, (g2sumR2) 1.59x, (g2fsumR2) 1.52x,
(g2meanR2) 1.51x, (v4) 1.45x and (g1minmaxR2) 1.34x; group-bys in buckets not taken for their
class, (g2sum200) 3.08x and (g2fsum200) 2.70x (two-key sum, 200 groups), (t2) 1.61x and
(g1count200) 1.58x (one-key count, 200 groups) and (g1count1k) 1.57x (1,000 groups); the semi join
(w3) 2.60x (
join:semi is not taken, (f)); the top-k (n) 1.85x and (x3) 1.23x (top_k is not
taken, (x4)).
The other 17 are 1.01x to 1.18x, inside the noise band.
Sorts with one key per Polars key. The 2026-09-26 default benchmark
(Benchmarks/results/polars_engine_bench_2026-09-26-final3.csv) ran when a nulls-first nullable key and a
descending float key still carried an extra key column each. With Polars' order as options of the one
key, the sort and top-k cases and four group-by controls were timed old against new, alternating, three
rounds of best of 5 after a 100 ms warm-up (Benchmarks/results/polars_engine_sort_options_2026-09-30.csv,
run conditions in …_conditions.txt); best and median of the rounds, in ms:
The faster Polars engine in the same runs (best of the rounds of the new build, new_best_ms of the
Polars rows): (x4) 9.75 ms at 50M
and 1.21 ms at 2M (streaming), (n) 18.43 ms and 2.23 ms (streaming), (x2) 542.42 ms and 17.54 ms,
(p) 978.94 ms and 22.02 ms (in-memory).
The top-k cases are still left to Polars by the default (top_k has no crossover, above), so its rows
for them are Polars' in either build. The sorts whose keys carry no option ((m), (o), (q), the Parquet
(s4)), the top-k by an int64 key (x3), (y6), (b) and the group-by controls (c), (i), (j), (t1) are
unchanged: every row that was more than 5% slower in both best and median over the three rounds, all
of them on paths this change does not touch, was timed again alone over eight alternating rounds of
100 runs (the rows flagged row re-timed alone in the same file), where before ÷ after is 0.95 to
1.25 in best and 0.96 to 1.22 in median. No row's median
process CPU time rose by more than 1.5x.
Group-bys after the grouped Float64 sum, mean and min/max were rebuilt. The 75 group-by cases
(the grid's 60, (c), (i), (j), (l), (t1) to (t7), (v1) to (v4)) at 2,000,000 and 50,000,000 rows,
MetalEngine() with the table fitted before the 2026-09-30 re-sweep and with the table fitted from
it (before the default-benchmark rule), the same build, three alternating rounds of best of 5 after a
100 ms warm-up, the first run after 500 ms of idle recorded on its own:
Benchmarks/results/polars_engine_default_groupby_2026-09-30.csv (per row best of the rounds and
median of the rounds' medians; every run in …_raw_2026-09-30.csv, run conditions in
…_2026-09-30_conditions.txt). Every answer was equal to Polars'. The re-swept table took 57 of the
150 case-size pairs (8 at 2,000,000 rows, 49 at 50,000,000), the earlier one 48, and it left none
that the earlier one took. What it took that the earlier one left, in ms (best of the rounds; the
earlier table's default ran Polars' in-memory plan there):
Every other pair it took was ahead of the faster Polars engine on the best run and on the median
(new_vs_fastest_polars_best and _median in the same file): at
50,000,000 rows from 1.24x ((g1minmax1k)) to 10.69x ((g2count1M)), at 2,000,000 rows from 1.12x to
1.63x ((g2count10k)). Taken and behind at 2,000,000 rows were the two-key means (c) at 0.82x,
(g2mean100k) at 0.65x and (g2mean10k), which the earlier table took as well, at 0.87x (5.83 against
5.08 ms; 0.73x under the earlier table). The default-benchmark rule (above) raises the two-key mean's
10,000 and 100,000 buckets to 50,000,000 rows, so the table in force takes 54 of the 150 pairs (5 at
2,000,000 rows, 49 at 50,000,000), every one at 1.0x or more of the faster Polars engine on the best
run and the median in this benchmark, and leaves (c), (g2mean10k) and (g2mean100k) at 2,000,000 rows
to Polars, (g2mean10k) being the one pair the earlier table took that this one leaves. Those three and
(g2minmax10k), which shares (c)'s 10,000-group bucket of the min + max class, re-timed, the table
before the rule against the table in force, three alternating rounds
(Benchmarks/results/polars_engine_default_groupby_retime_2026-10-01.csv, run conditions in
…_retime_2026-10-01_conditions.txt), in ms, best / median, against the faster Polars engine's best
of 5.49, 4.98, 5.61 and 5.44 ms:
Left to Polars, MetalEngine() runs Polars' in-memory plan, which uses the CPU cores (77 to 90 CPU-ms
against 4 to 6 on the GPU path for these three, new_cpu_ms and old_cpu_ms in the retime file). In this benchmark each engine's timed runs follow Polars' runs
of the same case and a 500 ms idle gap, and these three take 4.75 to 9.79 ms (best of 5) under
shapes="all", cold or warm, as well (the reported rounds of …_raw_2026-09-30.csv); run back to back, 15 runs of each way of collecting in turn, three
rounds and no idle gap (Benchmarks/results/polars_engine_groupby_backtoback_2026-10-01.csv), the
default runs them in 2.89 ms (c), 2.41 ms (g2mean10k) and 3.03 ms (g2mean100k) against Polars'
streaming engine's 5.48, 5.09 and 5.74 ms, which is where the crossover sweep, measured the same
back-to-back way, put them (2.09x, 2.19x and 2.34x at 2,000,000 rows in
Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv). In the 150-pair benchmark the
pairs whose decision did not change ran the same plan under both tables; the rows of those that were
more than 5% slower in best and median under one table than the other ((g1count10k) and (g1count1M)
at 50,000,000 rows, taken by both; (g2count100k) at 2,000,000, taken by both; (g1minmax100k) and (t4)
at 2,000,000, Polars' plan under both) differ between the two runs of one plan.
Float64 group sums and means at 2^24 groups. The (v3) case of the crossover sweep, a mean over two keys with about as many groups as rows, found ArrowMetal's group-by returning null for most groups at 16,777,216 groups and above (16,777,215 were right): the per-group kernels dispatched one threadgroup per group and the grid wrapped past 2^32 threads. Fixed in the core (Findings against pyarrow, round 13; python/tests/test_group_by_2_24.py), so no engine rule is needed and every group-by shape follows the policy above. The sweep and the benchmark above ran with the fix: under shapes="all" the cases the guard once kept with Polars, (c), (l), (t3), (t6), (v3) and the Parquet cases (s1) and (s2), run on Metal and equal Polars' answers. At 50M rows, under shapes="all", cold (Benchmarks/results/polars_engine_bench_2026-09-26-final3.csv), (c) is 1.78, (l) 2.03 and (t6) 1.21 times the faster Polars engine; (t3) is behind at 35.5 ms against 12.4 and (v3) at 612.0 ms against 421.7.
On polars 2.0.0. Polars 2.0.0 (released 2026-10-06) runs collect() on its streaming engine by
default (1.44.1: the in-memory engine), and runs group-bys, joins and sorts faster than 1.44.1. Over
the 62 cases the 1.44 default takes at 50,000,000 rows, the faster Polars engine's time on 2.0.0 is
0.72x of its time on 1.44.1 at the median, and 0.25x on (r), unique over two keys (153.1 ms to
38.5 ms); on 2.0.0 the streaming engine is the faster of the two in 56 of the 62. ArrowMetal's times
on the same cases are unchanged (median ratio 1.03). Files: Benchmarks/results/polars_engine_bench_2026-10-02.csv
(1.44.1) and Benchmarks/results/polars_engine_bench_2026-10-07-polars2.csv (2.0.0, conditions in
bench_conditions_2026-10-07-polars2.txt), 50,000,000 rows, best of 5, AC power. A mean over an
integer column now runs in one pass from the integer column (CHANGELOG, Unreleased); on 2.0.0 the
two-key means (c), (g2mean200) and (g2mean1k) went from 0.98x, 0.88x and 0.97x of the faster Polars
engine to 1.41x, 1.26x and 1.41x (polars_engine_bench_2026-10-07-polars2.csv before,
polars_engine_bench_2026-10-08-polars2-after.csv after).
The Polars 2.x table, python/arrowmetal/_engine_crossovers_pl2.py, is fitted by the same rules from
the sweep on 2.0.0 (Benchmarks/results/polars_engine_crossover_2026-10-08-polars2.csv: the 107
in-memory cases at the same eight sizes, 250,000 to 50,000,000 rows, and the Parquet scan cases at
1,000,000 to 50,000,000 rows, best of 7, one process per size, conditions in
polars_engine_crossover_2026-10-08-polars2_conditions.txt) and the default benchmark on 2.0.0
(Benchmarks/results/polars_engine_default_groupby_raw_2026-10-08-polars2.csv: the group-by, unique
and sort cases at 2,000,000 and 50,000,000 rows, and the sort, unique and 1,000- and 10,000-group
cases at 5,000,000, 10,000,000 and 20,000,000). Where its rows differ from the 1.x table's:
join:left (1,875,000), the Parquet sort (1,500,000) and String distinct (5,000,000) are the same
in both, and so is every class the 1.x table does not take. The group-count buckets of the 2.x table:
With the 2.x table in force (Benchmarks/results/polars_engine_bench_2026-10-08-polars2-refit.csv,
the 107 in-memory cases at 50,000,000 rows, best of 5, AC power, conditions in
bench_conditions_2026-10-08-polars2-refit.txt) the default takes a subtree in 52, every result
equal to Polars', and every one is ahead of the faster Polars 2.0.0 engine, from 1.35x ((e) at 50M)
to 7.94x ((q) at 50M), none behind. Some of them:
To improve on polars 2.0.0, left to Polars by the 2.x table (the sweep on 2.0.0, vs_fastest_polars
of MetalEngine all, cold, 250,000 to 50,000,000 rows): whole-frame aggregates (a) to (a4) 0.11x to
0.47x; row-wise filters and projections (d) 0.16x to 0.40x; the semi join against a 1,000-row table
(f) 0.42x to 0.82x, which keeps join:semi with Polars, the semi join against a 1,000,000-row table
(w3, 1.21x to 2.14x) with it; top-k (n), (x3), (x4) and (y6) 0.20x to 1.02x; the join on a String key
(y4) 0.24x to 0.48x; group-by sum and mean over one or two keys at 200 to 10,000 groups ((t1),
(t3) and the grid's sums, Float64 sums and means) 0.36x to 0.97x at 250,000 and 500,000 rows, in the
buckets the table above leaves; and the sort with a String column (o), 0.89x at 50,000,000 rows under
the 1.x table on 2.0.0 (244.77 ms against Polars streaming 217.88 ms,
Benchmarks/results/polars_engine_bench_2026-10-08-polars2-after.csv), which leaves the String sort
class, and with it (p), with Polars. A plan the default leaves runs on Polars' in-memory engine, where
a plain collect() on 2.0.0 runs the streaming engine: for the sort (m) at 5,000,000 rows, below the
2.x table's sort crossover, Polars' in-memory engine takes 32.88 ms and its streaming engine 13.55 ms
(Benchmarks/results/polars_engine_default_groupby_raw_2026-10-08-polars2.csv).
am.MetalEngine() # shapes="measured": the crossovers and buckets above
am.MetalEngine(shapes="all") # every translatable subtree of at least 1,000,000 rows
am.MetalEngine(shapes="all", min_rows=0) # everything it can translate (what the tests use)
am.MetalEngine(shapes={"group_by", "sort"}) # the classes named (or their prefix), any size
am.MetalEngine(min_rows=5_000_000) # the crossovers, and at least 5,000,000 rows
am.MetalEngine(raise_on_fail=True) # raise instead of leaving anything to Polars
Each subtree the report lists as taken carries the rule that took it, and each node the policy left
has a rule: line (here under the Polars 1.x table):
metal: Sort#2 [Sort > DataFrameScan] over 2,000,000 rows, ran in <t> ms -> 2,000,000 rows
rule: 2,000,000 input rows is at or above the 1,000,000-row crossover for sort (crossover sweep, ...)
polars: Select#3: rule: aggregate:sum was not measured ahead of Polars up to 50,000,000 input rows (...)
polars: Join#2: rule: 900,000 input rows is below the 2,029,827-row crossover for join:inner (...)
shapes="all" is there for plans the benchmark did not cover and for moving work off the CPU cores:
the CPU-ms column of the results file is the process CPU time of each run.
The import cache
Each Polars column the engine imports is kept, keyed by its Arrow type, length, offset and the
address and size of every buffer, and the next query that reads the same column reuses the import.
That is safe because the cached import holds the exported array and, through it, Polars' own buffer:
while an entry lives, Polars can neither free that memory (so no other column can appear at the same
address) nor write to it in place (Polars copies a buffer it shares before writing). Only a column
that was imported without a copy is cached; a String column (handed over as views, without a copy,
but outside the cache's key) and a multi-chunk column (concatenated on every export) are not. Entries are evicted least recently
used above a byte budget, a quarter of physical memory by default:
from arrowmetal import polars_engine
polars_engine.import_cache_info() # {"entries", "bytes", "limit", "hits", "misses"}
polars_engine.import_cache_limit(2 << 30)
polars_engine.clear_import_cache() # and with it the Polars buffers it kept alive
The "warm" column of the results file is a second collect over the same frame; the defaults above
were read from the cold one.
Parquet scans
lf = (pl.scan_parquet("trades.parquet")
.filter(pl.col("price") > 500.0)
.group_by("qty").agg(pl.col("weight").sum()))
lf.collect(engine=am.MetalEngine(shapes="all"))
A Scan of one local Parquet file becomes a leaf the engine reads itself, so the subtree above it
needs no import at all: the columns are decoded on the GPU straight into Metal memory
(Parquet on the GPU) and the plan runs over them.
- Projection. The columns Polars' projection pushdown left on the
Scan are the reader's column
list; no other column chunk is touched. Dictionary-encoded columns are read materialised, which
is what Polars reads them as.
- Predicate. Polars pushes a filter into the
Scan as its predicate. The engine translates it
like any Filter and runs it on the GPU over the rows the reader returns. The comparisons in it
that the file's statistics can judge the way Polars compares go to the reader as well, which skips
row groups (and, with a page index, pages) that cannot hold a matching row: a comparison of a
column with a literal, joined by &, on an integer column (all six operators), a String column
(==, !=) or a float column (<, <= and == only, against a literal exact in the column's
type). Polars orders NaN above every number, so NaN rows pass >, >= and !=, and writers leave
NaN out of min/max; those three never reach the reader for a float column, so != x keeps a NaN
in a row group whose statistics say x .. x (the case of apache/arrow#51491). |, functions and
comparisons of two columns stay on the GPU filter only. A Filter Polars left directly above the
Scan (with its predicate pushdown off) is handed to the reader the same way.
scan_parquet(use_statistics=False) hands it nothing.
- Nulls. A column is treated as nullable unless the footer's statistics say it holds no null
(
ParquetFile.column_null_count), so a sort key by a column without nulls carries no null placement. The
read checks the footer's word, and a file whose data holds nulls its statistics deny fails the
query with ArrowMetalError rather than answering differently.
- The open file is kept. The reader goes through the open-file cache
(Parquet on the GPU, "The open-file cache"), keyed by the file's path, inode, modification
time and size, so the second query over a file does not map it again, and a rewritten file is read
afresh.
am.clear_parquet_cache() and am.parquet_cache_limit() control it.
- Checked once per plan and file. As for in-memory inputs, the plan first runs over a prefix
of the file (64 rows of its first row group) and its output schema is compared with Polars'. A
file whose stored Arrow schema Polars applies and ArrowMetal's reader does not (one with a
different number of fields than the file, which Arrow's own reader ignores) fails that check or
names a column the file does not have, and stays with Polars.
- A bare scan stays with Polars. A
Scan with nothing above it that does GPU work is Polars'
to read.
What stays with Polars, with the reason in the report: several files (a list, or a glob or
directory that matches more than one), hive partition columns, a URL or cloud path (file://
included), row_index_name, n_rows (and a head, tail or slice Polars pushed into the scan),
include_file_paths, schema=, deletion files, column mapping, a column whose dtype the plan does
not carry (Decimal, Categorical, Enum, List, Struct, Binary, ...), a predicate that does not
translate, and CSV and NDJSON scans. polars 1.44.1 cannot show an IPC scan to an engine
(NodeTraverser.view_current_node raises NotImplementedError: ipc scan), so scan_ipc stays with
Polars as well.
Each taken subtree's report entry lists what the reader did (scans: the file, the filters it was
given, row groups read and skipped, pages skipped):
metal: GroupBy#2 [GroupBy > SimpleProjection > Scan] over 50,000,000 rows, ran in <t> ms -> 1,000 rows
parquet /data/bench-none-50000000.parquet: filters [('id', '<', 5000000)], 5 row groups read, 45 skipped, 0 pages skipped
MetalEngine() judges a scan subtree by the same rule as an in-memory one, with the Parquet rows of
the crossover table and the file's row count from its footer: a sort of a file of at least 1,500,000
rows is taken, and the filter, group-by and aggregate shapes over a file are not taken at any size
("Which translatable subtrees it runs: the defaults" above). The scan cases below are in line with
that: the sort is ahead of both Polars engines cold and warm, and the filter, group-by and aggregate
cases are behind cold.
The report
MetalEngine report for collect (polars 1.44.1, IR (14, 7))
metal: Sort#3 [Sort > HStack > Filter > DataFrameScan] over 10,000,000 rows, ran in <t> ms -> <n> rows
polars: Select#1: Function rank has no ArrowMetal translation.
engine.last_report is a MetalPlanReport: taken (one entry per subtree that ran on Metal: the
node at its top, the node kinds inside it, the rows it read, the ArrowMetal plan text, and after the
run its wall time and output rows), fallbacks (one Kind#id: reason line per node that stayed
with Polars for a reason of its own; every translation reason is a full sentence, which
test_every_unsupported_reason_is_a_full_sentence checks, and a node the placement policy left
has a rule: ... line, as above), walked (every node of the optimised
plan), nodes, and path (the collect path it is for, "Collect paths" above). With
POLARS_VERBOSE=1 the fallback lines are also issued as a PerformanceWarning. The report belongs to
the engine object, so two threads collecting through one MetalEngine overwrite each other's.
Tests
PYTHONPATH=python python -m pytest python/tests/test_polars_engine.py -q
DIFF_QUICK=1 PYTHONPATH=python python -m pytest python/tests/test_polars_engine.py -q # without the 100,003-row datasets
PYTHONPATH=python python -m pytest python/tests/test_engine_policy.py -q
test_engine_policy.py checks the default policy: the same decision for the same subtree across 200
calls and in a second process, every row of the crossover table one row below its crossover (left,
with the reason) and at it (taken), a subtree needing every class's crossover, the String and Parquet
rows, the router-table and sort-kernel floors, the overrides, a Parquet scan judged by its footer's
row count under a filter that keeps a handful of rows, the report's rule lines, that the committed
table is the fit of the results file it names, and the Float64 group-by guard at 2^24 groups. For
the group-count buckets it checks that the buckets tile the counts, every bucket one row below its
crossover and at it, a losing bucket named below or above the band, an estimate's range taken only
where every count in it is, the fallback to the class row without an estimate, where the probe finds
a group-by's keys, the probe's estimate in the true count's bucket over one int key, two and a String
key at 2,000,000 rows, the same estimate across 200 calls and in a second process, the cache, and the
engine taking a group-by in a winning bucket and leaving one in a losing bucket at the same rows.
The differential cases collect each LazyFrame on Polars and through
MetalEngine(raise_on_fail=True, min_rows=0, shapes="all") -- so nothing may fall back -- and
compare the frames: schema first, then values, exactly for integers, Booleans, Strings, nulls and
element-wise floats, with a relative tolerance for float aggregates. The inputs are
test_differential.py's generators at 0, 1, 33, 4,097 and 100,003 rows, null ratios 0, 0.3 and 1,
its "sliced" and "special" (extremes, NaN, infinities, subnormals) flavours, and two Polars-side
shapes, a two-chunk frame and a frame sliced with DataFrame.slice. Every numeric dtype runs
through the comparisons, arithmetic, null logic, is_in, casts, group-by and whole-frame aggregates,
sorts in both directions with nulls at both ends, and top-k; Boolean logic, String predicates and
carried temporal columns have their own cases, and so do the four join kinds on integer, String and
two-column keys with null keys and duplicates on both sides (a two-chunk right side among the
shapes), suffix collisions, different key names, a join inside a filter-join-aggregate plan, and
unique with and without a subset. Each fallback in the tables above has a case that
checks the result is still Polars' and the report names the reason. One case reruns
test_polars.py and test_lazy.py with every LazyFrame.collect() also collected through the engine
(python/tests/metal_engine_everywhere.py) and requires the two to agree.
test_every_collect_path_is_explicit runs every row of the collect-path table and checks where the
plan ran, the report's path and reason, and that the warning comes once.
test_the_capability_table_is_current regenerates Polars engine capabilities
and compares it with the committed file; no cell of it may be a Metal answer that differs from
Polars' or an engine exception. The committed table is from polars 1.44.1, IR (14, 7); on a Polars
of another IR major (2.0.0) the test is skipped. test_an_unknown_node_kind_keeps_the_whole_plan_on_polars and
test_a_node_polars_cannot_describe_keeps_the_whole_plan_on_polars check the fail-closed walk, and
test_the_check_command_prints_versions_and_the_capability_header the command below.
python -m arrowmetal.polars_engine check prints the installed Polars version and whether it is one
of TESTED_POLARS, the IR version against TESTED_IR_VERSION, whether the callback API is present,
the IR node kinds of this Polars the engine does not know, whether ARROWMETAL_METAL_ENGINE is set,
and the header of the capability table (in a source checkout), with a line naming both versions
when the table was generated on another Polars. It exits 1 when the callback API is missing or the
IR major is not a tested one, since then every plan stays with Polars.
The Parquet scan cases (212 tests) run fourteen scan shapes -- filters over every numeric dtype,
Polars' float total order, !=, String predicates, projections, group-by on one and two keys,
whole-file aggregates, sorts with nulls at both ends, top-k, a join with an in-memory frame and
unique -- over one 20,011-row dataset with nulls, NaN, -0.0 and infinities written by pyarrow
(snappy; uncompressed with a page index and no dictionary; ZSTD with v2 pages), by Polars and by
DuckDB; a filter, a sort and an aggregate over the flat columns of 30 nested fixtures from all three
writers; seventeen predicate-pushdown cases on every writer, each checking the answer, the filters
the reader was given and the row groups it skipped, including NaN under >, >= and != (the
apache/arrow#51491 shape, also on the pageindexnan__pa_constpage fixture); every fallback reason
above; the open-file cache's reuse, invalidation and bounds; and footer null counts, including a file
whose statistics deny its nulls.
The conformance grid. python/tests/engine_polars_grid.py generates the cases instead of
choosing them: every shape the engine translates (filters, select and with_columns expressions,
slice, sorts in both directions with nulls at both ends and over two keys, top-k, group-by with
each aggregate with the column as the key and as the value, whole-frame aggregates, the four join
kinds, unique) over every dtype it carries (the eight integer widths, Float32, Float64, Boolean,
String, Date, Datetime in ms, us and ns and with a time zone, Duration in ms, us and ns, Time), with
no, 5%, 70% or all nulls, at 0, 1, 7, 1,000 and 100,000 rows, plus the special values (integer
extremes, NaN, infinities, subnormals, -0.0). Each case collects through
MetalEngine(shapes="all", min_rows=0) and through Polars and compares the frames bit for bit.
The sorts cover each direction with the nulls first and last, a stable sort in each direction
(maintain_order=True: tied rows, the two zeros and every NaN keep their input order, compared bit
for bit), two and three keys with a per-key null placement, a tie on one key falling through to the
next, and top-k as sort().head(), top_k, bottom_k and over two keys. In the run recorded in
Benchmarks/results/engine_conformance_2026-10-02.csv (the per-shape table in
engine_conformance_2026-10-02_report.txt): 15,376 cases, 15,096 identical on Metal, 33 within the
float-summation bound above, none different otherwise, and 247
where Polars' optimised plan had nothing to run (all of them sorts over 0 or 1 row, which Polars
leaves out). python/tests/engine_report.py --engine polars reruns it and prints the
per-shape table.
To verify on your own machine
DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer swift build -c release --product ArrowMetalC,
then PYTHONPATH=python python -m pytest python/tests/test_polars.py python/tests/test_lazy.py -q:
tiers 1 to 3 still green.PYTHONPATH=python python -m pytest python/tests/test_polars_engine.py -q, after that fresh build.- In a REPL:
import sys, arrowmetal as am and check "polars" not in sys.modules; then collect one
plan through am.MetalEngine(shapes="all", min_rows=0) and read engine.last_report.
- With
POLARS_VERBOSE=1, collect a plan with an unsupported node (pl.col("v").rank()) and see the
PerformanceWarning listing it.
am.zero_copy_report(df["v"]) on a single-chunk numeric column of a million rows says the two
addresses are the same.- On polars 1.44,
out, timings = engine.profile(lf) shows a metal: row, and
PYTHONPATH=python python -m arrowmetal.polars_engine check prints the versions it was tested with.
- Run
PYTHONPATH=python python Benchmarks/polars_engine_bench.py --sizes 2000000,50000000 --out a.csv
twice on a quiet machine, compare the two files, and read the taken and rule columns of the
MetalEngine default, cold rows against the table above. --crossover with more sizes, then
python Benchmarks/polars_engine_crossover.py a.csv --print, fits the table from your own run.
- Read this section against
engine.last_report on your machine.
Numbers
These are tiers 1 and 2, and tier 4 over a Parquet file at the end; tier 4's measurements over
in-memory frames are in Benchmarks/results/polars_engine_bench_2026-10-02.csv and
Benchmarks/results/polars_engine_crossover_2026-09-30-groupby.csv (see "Tier 4" above).
Apple M4 Max, macOS 26.6.2, polars 1.44.1 (16 threads), pyarrow 25.0.1, ArrowMetal 0.1.0. Best of 5
runs after a warm-up, one process, one data set. Every figure below is from
Benchmarks/results/polars_bench_50000000_2026-09-07.txt, except the String rows: those are ArrowMetal
0.2.0 with String columns handed over in Polars' view layout, best of 5,
Benchmarks/results/polars_bench_strings_view_2026-09-26-quiet.txt (run conditions in
Benchmarks/results/bench_conditions_2026-09-26-quiet.txt). Reproduce with:
PYTHONPATH=python python Benchmarks/polars_bench.py 50000000 5
Columns: k Int32 with 1000 distinct values, v Int64, amount Float64, name String drawn
from 4096 distinct values. The "Polars" column is Polars' eager idiom; the lazy engine and
pyarrow's Acero are on the Compare tab and in the benchmark matrix.
50M rows
The String hand-off of that column, 50M rows, in the same run:
Reading the table
- The "Polars" column is Polars' eager idiom, which is what
Benchmarks/polars_bench.py measures
and what Benchmarks/results/polars_bench_50000000_2026-09-07.txt records. The published baseline is the
parallel run, Benchmarks/results/full_matrix_2026-09-07-parallel.csv, which adds Polars' lazy
engine (polars-lazy, a median of 11.5 of the 16 cores); ratios there are lower on the rows where
the lazy engine parallelises. The full matrix has both.
- "GPU-resident" is the same kernel with the column already in Metal memory -- the import is
outside the timed region, and, for the group-by rows, the
am.group_by([k]) key-mapping pass as
well: those rows time the aggregate only. It is what a pipeline that stays on the GPU sees, and it
is the column that shows what the kernels are worth.
- The group-by row is not a like-for-like ratio against Polars, whose 79.3 ms includes its whole
hash group-by. The comparable end-to-end figure is in the benchmark matrix:
sum by int32 key
(1000 groups) at 50M rows is 4.89 ms against Polars lazy's 81.93 ms, 16.8x
(Benchmarks/results/full_matrix_2026-09-07-parallel.csv), and 1.73 ms against 20.24 ms, 11.7x, at
10M. The fastest CPU idiom on that row is pyarrow's Acero (pyarrow-threaded), 18.45 ms at 50M,
which puts the ratio at 3.8x.
- The hand-off is the whole difference between the middle columns and the right one. At 50M
rows the import is 5.25 ms and the export 0.01 ms, and the run records the import as a no-copy
-- the Polars source buffer and the Metal buffer are the same address (
zero copy: ...
-> SAME). Every tier-1 and tier-2 row pays it once per call, so on a single sum Polars is
ahead, and a group-by is 4x.
- CPU-ms is the other half of the story. The 50M group-by costs Polars 1121 CPU-ms across 16
threads; ArrowMetal costs 14.0 CPU-ms end to end and 0.4 CPU-ms resident. On a laptop that is
battery, and on a shared box it is 16 cores left free for something else.
- Strings cross in Polars' own
Utf8View layout: 50M rows hand over in 30.0 ms with no bytes
copied, against 309.0 ms through large_string (offsets + bytes), which is a copy. Through the
namespace contains is 17.6x with views and 2.1x with offsets; resident it is 112.4x (106.9x over
offsets). The tier-2 plugin still asks Polars for large_string and is 2.3x. Keeping a string
column resident (s.arrowmetal.to_metal()) takes the 38.1 ms call to 6.0 ms.
Where each tier is worth using
Tier 4 over a Parquet file, 50M rows
pl.scan_parquet(file) under a filter, a group-by, an aggregate or a sort, over the 50,000,000-row,
8-column files of Benchmarks/parquet_bench.py (snappy, 1.65 GB, and uncompressed, 2.23 GB),
collected by Polars' in-memory and streaming engines and by MetalEngine(shapes="all", min_rows=0).
"cold" clears the open-file cache before every run, so the engine opens and maps the file each time;
"warm" keeps it open between runs. Polars reads the file on every run; the file stays in the OS page
cache throughout. Best of 5, every engine result equal to Polars',
Benchmarks/results/polars_engine_scan_2026-09-26-quiet.csv (run conditions in
Benchmarks/results/bench_conditions_2026-09-26-quiet.txt).
- The sort is ahead cold and warm: 4.4x (snappy) and 7.6x (uncompressed) the faster Polars
engine cold, 4.7x and 8.7x warm, and the default takes it (4.4x and 7.5x).
- Cold, the other three are behind and to improve, from 35.0 ms against 12.1 ((s2), snappy) to
13.3 ms against 13.1 ((s3), uncompressed) (
vs_fastest_polars). The cold runs include what the
open-file cache removes: opening the file and handing the pages the query reads to Metal; a
one-column read of these files is 6.9-12.2 ms through a fresh open and 3.2-7.6 ms through the cache
(Parquet on the GPU, "The open-file cache"). The default leaves them to Polars.
- Warm, over the uncompressed file (s3) is ahead of Polars' streaming engine (5.9 ms against
13.1, 2.2x) and so is (s1) (31.6 ms against 36.1, 1.1x); (s2) is behind at 6.3 ms against 5.1.
Over the Snappy file, which the GPU decompresses first (Parquet on the GPU, "What the
numbers say"), (s1) is behind at 91.3 ms against 45.6, (s2) at 33.4 against 12.1 and (s3) at 13.9
against 13.7.
- CPU time: the warm engine runs cost 2.9-7.8 ms of process CPU, against 37.5-6733.8 ms for
Polars (
cpu_ms).
PYTHONPATH=python python Benchmarks/polars_engine_bench.py --scan-only --scan-rows 50000000 \
--scan-codecs snappy,none --out scan.csv
Limits
Types. This paragraph is about what the bridge round-trips, which is a tier-1 and tier-3
question; tier 2's expressions accept a shorter list, set out in "What tier 2 accepts" above.
Every dtype Polars and Arrow share round-trips: all signed and unsigned integer widths,
Float32/64, Boolean, Date, Datetime (all units), Time, Duration, String, Binary. Categorical and
Enum also work, but Polars encodes them as dictionary<uint32> and dictionary<uint8> while
ArrowMetal wants int32 or int64 indices, so the bridge recodes the index buffer -- 4 bytes a row,
values untouched. An Enum comes back from to_polars as a Categorical: the dictionary crosses,
the fact that its value set was closed does not. List and Struct cross and round-trip, but only
the structural kernels operate on them -- you cannot sum or group by one. Object is not bridged.
Chunking. am.from_polars takes one Arrow array. A multi-chunk Series is rechunked once, which
does copy; am.from_polars(s, rechunk=False) raises instead, so the copy is never silent. The
MetalEngine hands a multi-chunk column of an in-memory frame over as its own chunks, which the
chunked import (am_import_chunks) copies straight into the final Metal buffers without Polars
concatenating them first; Categorical, Enum and nested columns are still concatenated by to_arrow.
Order. Group order is ArrowMetal's, not Polars': ascending by key for numeric, boolean,
temporal and decimal keys, first-seen for utf8 and binary, lexicographic in column order for
several keys. Polars' group_by promises no order at all, so sort both sides before comparing.
s.arrowmetal.unique() is first-seen, the order Polars' unique(maintain_order=True) gives,
and it keeps a null as one of the distinct values rather than dropping it. Sorts put nulls last in
both directions, where Polars' ascending default is nulls first -- pass nulls_last=True when
comparing.
Strings. A String or Binary column crosses in Polars' own Utf8View / BinaryView layout
(to_arrow(compat_level=newest)): its 16-byte views and data buffers are mapped into Metal without a
copy, and export hands them back to Polars the same way. s.arrowmetal.to_metal().string_layout says
"view". The string kernels read the views directly (the list is am.STRING_VIEW_KERNELS); the
rest (am.STRING_VIEW_CONVERTS: str_concat, the splits, match_like, strptime, the
byte-counting pads, utf8_zero_fill, the byte slices and reversals, casts from strings, the file
writers) convert the column to offsets + bytes once, on the GPU, and keep that form, which
string_layout then reports as "view (converted)". am.from_polars(s, string_layout="offsets")
asks Polars for large_string instead, the layout used before the kernels read views; the tier-2
plugin still uses that. upper/lower are Unicode's simple 1:1 case mapping over every script (the GPU
table covers U+0000–U+017F exactly; a row holding anything above is mapped on the host), so Greek
and Cyrillic come back mapped. Simple, not full: the multi-character expansions are not applied,
so U+00DF becomes ẞ where Polars' str.to_uppercase() gives SS, and fi stays put.
contains / starts_with / ends_with are literal, not regex.
Arithmetic. .add/.sub/.mul/.truediv in tier 2 keep the column's own type and follow Arrow's
unchecked rules for the operation: integers wrap, and integer division by zero yields 0 where
Polars raises. The operand is checked -- one the column's type cannot hold raises rather than
being clamped or truncated, and an integer operand is exact past 2^53. See "The scalar in .add"
above.
Joins. GPU path only for a single-column inner or left join against a unique right key;
everything else falls back to Polars (or raises with allow_cpu_fallback=False).
Aggregation inside group_by. Tier 2's group_by_sum is a projection, not a hash aggregate
Polars calls per group -- the plugin API has no hook for that. See the tier-2 section.
Threads. ArrowMetal serialises command-buffer commits behind its own lock and keeps its error
state thread-local, so Polars is free to call the plugin from several worker threads. The plugin
takes no lock of its own.
Version coupling. Tier 2: the plugin is pinned to polars 0.55.1 / pyo3-polars 0.28 for
py-polars 1.44.x. Tiers 1 and 3 speak the C Data Interface and are not coupled to a Polars
version. Tier 4 is pure Python but reads Polars' optimised IR through an API Polars calls unstable;
it was written against IR version (14, 7) of polars 1.44.1 and (15, 2) of polars 2.0.0, a test
pins them, and an IR major outside these makes it leave every plan to Polars.
Tests
PYTHONPATH=python python -m pytest python/tests/test_polars.py -q # 90 tests
cd polars-plugin/arrowmetal-sys && cargo test --release # 10 tests
PYTHONPATH=python python -m pytest python/tests/test_polars_engine.py -q # tier 4
test_polars_engine.py is described under "Tier 4", "Tests".
test_polars.py covers round trips for every shared dtype (plus Categorical, Enum, empty,
all-null and chunked), the namespace methods against native Polars at five sizes from 0 to
100,003 rows, the plugin expressions inside lazy plans, the join against pl.DataFrame.join,
zero-copy assertions on buffer addresses, and 50M-row timings as assertions with bounds generous
enough not to flake. The plugin tests skip when the Rust library has not been built, so a
checkout without a Rust toolchain still runs green -- build it before you trust a green run,
or 28 of the 90 are skips (62 passed, 28 skipped).
The tier-2 block at the end of the file is the hostile-input pass: the scalar operand against the
tier-1 bridge, nulls against native Polars, a sliced Series and a two-chunk one, an empty frame,
an all-set validity bitmap with no nulls, the twelve dtypes the numeric expressions refuse (each
one a Polars error carrying ArrowMetal's wording, never a pyo3 panic), and the expression inside
group_by().agg() and inside collect(engine="streaming").
