trips · Query Engine

Column	Arrow type	Kind	Footer min	Footer max
trip_id	int64	data	0	2499
pickup_hour	int32	data	0	23
passengers	int32	data	1	4
distance	double	data	0.33	40.0
price	double	data	3.61	179.98
tip	double	data	—	—
payment	string	data	—	—
region	string	partition	—	—
date	string	partition	—	—

Min/max are read from the first row-group of one file and are shown for the first few columns only — they illustrate the statistics the engine prunes on, not the range across the whole dataset. A blank cell means that column carries no statistics in the footer.

Why partition columns are different

Partition columns are not stored inside the Parquet files at all — their values come from the directory names, which is why a predicate on one can eliminate whole files without opening any of them. Predicates on data columns have to fall back to footer statistics, which skip row-groups instead.

Per-column size on disk

one file, footer-only

Compressed bytes per column across one representative file. This is the table that explains column projection: reading passengers instead of price is the difference between 5.7 KB and 115.4 KB.

price 115.4 KB 28.6%

trip_id 108.4 KB 26.9%

distance 93.6 KB 23.2%

tip 59.4 KB 14.7%

pickup_hour 13.6 KB 3.4%

payment 7.1 KB 1.8%

passengers 5.7 KB 1.4%

A COUNT(*) needs row counts, not values, so the engine resolves it to the narrowest column on disk rather than the first declared one. On this file that is passengers at 5.7 KB — against 403.2 KB for every column.

Physical layout

Sampled file: /opt/render/project/src/data/trips/region=EU/date=2024-01-01/part.parquet
Row-groups in that file: 8
Rows in that file: 20000
Rows per row-group: ~2500
Column bytes in that file: 403.2 KB
Files in the dataset: 15
Dataset size: 6.0 MB

The row-group count is what sets the granularity of statistics-based skipping: more row-groups means finer pruning but a larger footer to read.

Partition values	Path	Size
region=EU date=2024-01-01	/opt/render/project/src/data/trips/region=EU/date=2024-01-01/pa…	409.9 KB
region=EU date=2024-01-02	/opt/render/project/src/data/trips/region=EU/date=2024-01-02/pa…	409.6 KB
region=EU date=2024-01-03	/opt/render/project/src/data/trips/region=EU/date=2024-01-03/pa…	409.3 KB
region=EU date=2024-01-04	/opt/render/project/src/data/trips/region=EU/date=2024-01-04/pa…	409.2 KB
region=EU date=2024-01-05	/opt/render/project/src/data/trips/region=EU/date=2024-01-05/pa…	409.6 KB
region=US date=2024-01-01	/opt/render/project/src/data/trips/region=US/date=2024-01-01/pa…	409.4 KB
region=US date=2024-01-02	/opt/render/project/src/data/trips/region=US/date=2024-01-02/pa…	409.7 KB
region=US date=2024-01-03	/opt/render/project/src/data/trips/region=US/date=2024-01-03/pa…	410.1 KB
region=US date=2024-01-04	/opt/render/project/src/data/trips/region=US/date=2024-01-04/pa…	410.1 KB
region=US date=2024-01-05	/opt/render/project/src/data/trips/region=US/date=2024-01-05/pa…	410.3 KB
region=APAC date=2024-01-01	/opt/render/project/src/data/trips/region=APAC/date=2024-01-01/…	409.6 KB
region=APAC date=2024-01-02	/opt/render/project/src/data/trips/region=APAC/date=2024-01-02/…	410.3 KB
region=APAC date=2024-01-03	/opt/render/project/src/data/trips/region=APAC/date=2024-01-03/…	410.1 KB
region=APAC date=2024-01-04	/opt/render/project/src/data/trips/region=APAC/date=2024-01-04/…	409.4 KB
region=APAC date=2024-01-05	/opt/render/project/src/data/trips/region=APAC/date=2024-01-05/…	409.6 KB

Showing 15 of 15 registered files. One row per file: these are the units a partition predicate eliminates whole.

One query per mechanism

Open in workspace

Each of these isolates a single thing the engine does, generated against this dataset's real schema and footer statistics. Submitting one runs it for real.

The read percentages here are not a descending sequence. Each query exercises a different mechanism; they do not stack. Two row-count queries can differ by 20x purely in which column had to be read.

1Full scan (baseline)

Every column, no WHERE clause — nothing can be pruned, so this reads 100% of the dataset. Run it first: it is the number every other query is measured against.

SELECT *
FROM trips
LIMIT 20

2Partition pruning

region is a partition column — it lives in the directory path, not inside the files. The engine drops every file that cannot match without opening a single one.

SELECT COUNT(*) AS total
FROM trips
WHERE region = 'EU'

3Row-group skipping

A highly selective filter on trip_id. Every Parquet footer carries min/max statistics per row-group, so most row-groups can be proven irrelevant and skipped before a single byte is decompressed.

SELECT COUNT(*) AS total
FROM trips
WHERE trip_id > 2374

4Column projection

Only distance is referenced, so only distance comes off disk — and region is free because it comes from the path, not the data. Parquet is columnar, so the unreferenced columns are never touched.

SELECT region, COUNT(*) AS n, AVG(distance) AS avg_distance
FROM trips
GROUP BY region
ORDER BY n DESC

5All three at once

A query someone would actually write, with all three optimizations firing. It reads more than the single-optimization demos above because it needs a real measure column rather than just a row count — but still a small fraction of the baseline.

SELECT region, COUNT(*) AS n, SUM(distance) AS total_distance
FROM trips
WHERE region = 'EU' AND trip_id > 2374
GROUP BY region

Run	SQL	Status	Duration	Read	When
#15	SELECT region, COUNT(*) AS n, SUM(distance) AS total_distan…	success	17.76 ms	16.5%	11 Oct 18:57
#14	SELECT region, COUNT(*) AS n, AVG(distance) AS avg_distance…	success	30.01 ms	23.2%	11 Oct 18:57
#13	SELECT COUNT(*) AS total FROM trips WHERE trip_id > 2374	success	34.74 ms	26.9%	11 Oct 18:57
#12	SELECT COUNT(*) AS total FROM trips WHERE region = 'EU'	success	7.99 ms	0.5%	11 Oct 18:57
#11	SELECT * FROM trips LIMIT 20	success	89.94 ms	100.0%	11 Oct 18:57
#10	SELECT region, COUNT(*) AS n, SUM(distance) AS total_distan…	success	13.7 ms	16.5%	11 Oct 18:57
#9	SELECT region, COUNT(*) AS n, AVG(distance) AS avg_distance…	success	28.05 ms	23.2%	11 Oct 18:57
#8	SELECT COUNT(*) AS total FROM trips WHERE trip_id > 2374	success	31.37 ms	26.9%	11 Oct 18:57
#7	SELECT COUNT(*) AS total FROM trips WHERE region = 'EU'	success	8.12 ms	0.5%	11 Oct 18:57
#6	SELECT * FROM trips LIMIT 20	success	101.34 ms	100.0%	11 Oct 18:57
#5	SELECT region, COUNT(*) AS n, SUM(distance) AS total_distan…	success	17.57 ms	16.5%	11 Oct 18:57
#4	SELECT region, COUNT(*) AS n, AVG(distance) AS avg_distance…	success	30.89 ms	23.2%	11 Oct 18:57
#3	SELECT COUNT(*) AS total FROM trips WHERE trip_id > 2374	success	31.97 ms	26.9%	11 Oct 18:57
#2	SELECT COUNT(*) AS total FROM trips WHERE region = 'EU'	success	9.06 ms	0.5%	11 Oct 18:57
#1	SELECT * FROM trips LIMIT 20	success	102.01 ms	100.0%	11 Oct 18:57