Run a query

Engine capabilities

What the engine supports, how correctness is established, and where the depth is. The SQL surface is narrow by design — the engineering is in the scan path.

Supported SQL

Shape
single-table SELECT
Clauses
WHERE · GROUP BY · ORDER BY · LIMIT
Aggregates
count · sum · avg · min · max
Predicates
= < <= > >= != · AND · OR · IN
Projection
explicit columns, *, aliases

SQL is parsed with sqlglot into a logical plan, so the accepted grammar is a subset chosen by the planner, not by a hand-rolled parser.

Deliberately out of scope

  • Joins. No join operator exists in the logical plan. Adding one means a join algorithm, build/probe side selection and spill handling — a second project, not a feature.
  • Subqueries and CTEs. The planner takes a single scan as its leaf.
  • Window functions. Would need a sort-based partitioned operator the executor has no shape for.
  • DML and DDL. Read-only by design; a dataset is a prefix of Parquet files registered in the catalog, and there is no load step to write back to.

What the engine actually does well

Partition pruning Hive-style partition values are parsed from the directory path. A predicate on a partition column eliminates whole files before any file is opened, so pruned files cost zero I/O — not a cheap read, no read at all.
Row-group skipping Every Parquet footer carries min/max statistics per row-group. The engine reads footers only, evaluates the predicate against those ranges, and skips any row-group it can prove holds no match before decompressing a byte.
Column projection Parquet is columnar, so unreferenced columns are never touched. A COUNT(*) resolves to the narrowest column on disk rather than the first declared one — on the sample dataset that is a 19x difference.

Execution

  • Scan tasks are fanned across a ProcessPoolExecutor above a 2M-row threshold, with a serial fallback below it — process startup costs more than it saves on small scans.
  • Aggregates are computed as partial aggregates per task and then merged, so AVG is sum(sums) / sum(counts) rather than an average of averages.
  • Row-group metadata is cached (Redis when REDIS_URL is set), so repeated queries against a dataset do not re-read every footer.

SQL NULL semantics

Partial-aggregate-then-merge quietly breaks SQL NULL rules unless each aggregate is guarded, because the merge step cannot otherwise tell "summed nothing" from "summed to zero".

  • AVG over zero non-null values returns NULL, not NaN.
  • SUM over zero non-null values returns NULL, not 0.
  • MIN, MAX and COUNT already matched SQL.
  • NULL group keys form their own group rather than being dropped.

Pinned by regression tests, and by the result grid rendering a real NULL rather than Python's None.

Byte accounting

Both sides of every ratio on this site are compressed on-disk bytes. bytes_scanned is the sum of total_compressed_size for the projected columns in the row-groups that were kept. bytes_total is the compressed size of every row-group including the pruned ones, plus the file size of files that were never opened.

That denominator is the point. An earlier version compared compressed bytes read against uncompressed total size and excluded pruned row-groups entirely, which made every query look about 1.5x better than it was. The sanity check is simple: a query with nothing to prune must report exactly 100%, and there is a regression test that fails if it ever does not.

How correctness is established

  • DuckDB is the oracle. The test suite runs each query through both engines over the same Parquet files and compares result sets. Agreeing with a mature engine is a much stronger claim than agreeing with hand-written expected values.
  • NaN is not NULL. The comparison helper normalises NaN to a distinct marker before sorting, so a NaN-where-NULL-was-expected divergence fails loudly instead of comparing equal.
  • The fixture contains NULLs on purpose — including a group whose aggregate column is entirely NULL, and a column that is NULL in every row. Without those rows the NULL-semantics fixes above would pass trivially.
  • The 100% full-scan assertion is its own test: a query with nothing to prune must report reading exactly 100% of the dataset. It is what keeps the byte accounting anchored to a real denominator.

On the roadmap

  • Metadata-only COUNT(*). An unfiltered count can be answered from Parquet footer row counts alone, at zero data I/O. Today it reads the narrowest column instead — cheap, but not free.
  • Streaming result delivery. Results are currently materialised before being sliced to a preview; chunked delivery would let the executor hand back batches as row-groups complete.
  • Bloom filter support. Parquet footers can carry bloom filters for high-cardinality columns, which would prune row-groups that min/max statistics cannot — equality predicates on IDs are the obvious win.
  • Predicate pushdown into page indexes. Row-group skipping is coarse; Parquet page-level statistics would push the same logic one level deeper.