Run SQL GROUP BY on a CSV using DuckDB
Instantly query and aggregate large CSV files in-memory using DuckDB without importing them into a database.
Setup
- → Install DuckDB (e.g., brew install duckdb)
- → Have a valid CSV file named large_dataset.csv in your working directory
Cost per run
Free
The one-liner
$ duckdb -c "SELECT status, COUNT(*) as count FROM 'large_dataset.csv' GROUP BY status ORDER BY count DESC;"What each stage does
- [01] duckdb
duckdbInvokes the DuckDB command-line interface. - [02] duckdb
-cTells DuckDB to execute the following SQL query string and exit immediately. - [03] duckdb
FROM 'large_dataset.csv'DuckDB automatically infers the schema and reads the CSV file directly from disk without a prior import step. - [04] duckdb
GROUP BY status ORDER BY count DESCStandard SQL aggregation to group rows by the status column and sort the results by frequency.
Expected output (sample)
┌────────────┬───────┐ │ status │ count │ │ varchar │ int64 │ ├────────────┼───────┤ │ SUCCESS │ 45210 │ │ PENDING │ 8432 │ │ FAILED │ 1205 │ └────────────┴───────┘
Caveats & tips
- Memory usage scales with the size of the aggregation, though DuckDB is highly optimized for out-of-core processing if memory runs low.
- Ensure the CSV file path is correct and the file has read permissions for the executing user.