How to Compress Large CSV Files into Parquet Format Using Python

Learn to convert multi‑gigabyte CSV datasets like historical OHLCV into Parquet with Python, comparing Snappy and ZSTD compression for speed and size.

Why Compress Financial CSV Data into Parquet?

In quantitative research, historical price data is the bedrock of signal discovery. A typical OHLCV (open, high, low, close, volume) dataset for thousands of assets over a decade can easily weigh multiple gigabytes. Storing all that information in flat CSV files quickly becomes a bottleneck. CSVs are human‑readable text, which is great for a quick glance but disastrous for performance. Loading a 5 GB CSV into a pandas DataFrame or Arrow table forces your machine to parse millions of text records, often exhausting memory and slowing down every iteration.

The columnar Parquet format was designed precisely for such workloads. Instead of storing data as rows of plain text, Parquet organizes values by column, applies strong typing, and uses efficient compression codecs out of the box. This means that when you query only the “close” column for a single ticker, you don’t have to read the entire file. That sort of selective access turns multi‑minute CSV parsing into near‑instantaneous operations, making large‑scale backtesting and signal development feasible on modest hardware. In a domain where speed of iteration directly translates to better research, Parquet is not a luxury—it is a necessity.

CSV vs Parquet: A Technical Comparison

Row‑based vs Columnar Storage

A CSV file is a sequential list of rows. To find all closing prices for AAPL between two dates, your program must scan every line from the beginning, parsing all columns even if you only need one. Parquet flips this model. Data for each column is stored in contiguous chunks, so reading the “close” column for a filtered set of dates touches far fewer bytes. This columnar layout is a natural fit for financial time series, where queries often aggregate a handful of numeric fields and ignore the rest.

Schema, Types, and Compression Efficiency

CSVs have no notion of a schema beyond what your code infers from strings. A column meant to be datetime might be read as plain text, and every numeric value is stored as a variable‑length string. Parquet enforces a precise schema—dates stay dates, floats stay floats—eliminating repetitive parsing and allowing compression to exploit data patterns. A sequence of sorted timestamps with limited variation compresses dramatically better as native types than as character strings, because Parquet can use delta encoding and run‑length encoding directly on the integer representation of dates. This type awareness, combined with column grouping, is why Parquet files are often many times smaller than the equivalent CSV, and why analytical queries execute faster.

Setting Up the Python Environment

Required Libraries

You will need pyarrow (the official Apache Arrow Python binding) for reading and writing Parquet files. Although fastparquet is an alternative, pyarrow offers the most complete support for modern compression codecs and is the engine this guide will use. pandas remains useful for data manipulation, and the standard library’s os and time modules help with file management and benchmarking.

Installation Commands

If you’re starting fresh, create a virtual environment and install everything with pip:

python -m venv parquet-env
source parquet-env/bin/activate   # On Windows: parquet-env\Scripts\activate
pip install pyarrow pandas

If you prefer conda:

conda create -n parquet-env python=3.10 -y
conda activate parquet-env
conda install -c conda-forge pyarrow pandas -y

With the setup out of the way, you’re ready to start converting.

Choosing the Right Compression Codec: Snappy vs ZSTD

Understanding PyArrow Engine Support

PyArrow’s write_table function accepts a compression parameter that can be set to 'snappy', 'zstd', 'gzip', and others. The two most practical choices for quant workflows are Snappy and ZSTD (Zstandard). Both are lossless and integrate seamlessly with the Parquet format.

Snappy: Balanced Speed and Size

Snappy, originally developed by Google, prioritizes speed over maximum compression. It writes and reads extremely fast, often with minimal CPU overhead. For a typical OHLCV CSV, Parquet with Snappy often shrinks the data to 10–30 % of the original file size, making it 70–90 % smaller. ZSTD can go even further, frequently reducing the file to 5–20 % of its original CSV size (an 80–95 % reduction). These reductions are usually more than enough to keep datasets manageable, and the near‑native I/O speed makes Snappy the go‑to choice when you need to repeatedly read the entire dataset—for example, during exploratory analysis or when training models over many epochs.

ZSTD: Maximum Compression at a Modest Cost

ZSTD (Zstandard) aims for much higher compression ratios at the expense of slightly slower write times. On financial tabular data, it often achieves 80–90% size reduction relative to the original CSV. Read speeds remain excellent because ZSTD decompression is highly optimized, but the compression step itself uses more resources. This trade‑off is attractive when you write the data once and read it many times—classic behaviour in quantitative research, where a compressed dataset is stored and then accessed for thousands of backtesting loops. If your local storage is limited or you’re archiving historical snapshots, ZSTD is hard to beat.

Step‑by‑Step Conversion Guide

Reading a Large CSV with PyArrow

PyArrow’s CSV reader is highly optimised, but read_csv() still loads the whole file into memory. For files that are too large to fit in RAM, use the streaming interface: pyarrow.csv.open_csv() returns a reader that yields record batches. You can then write those batches incrementally to a Parquet file with ParquetWriter, keeping memory usage low. Always specify the parse_options to tell Arrow how to interpret dates, otherwise you may end up with string timestamps.

import pyarrow as pa
from pyarrow import csv, parquet as pq

# Read the file; infer schema automatically but override date columns
read_options = csv.ReadOptions(use_threads=True)
parse_options = csv.ParseOptions(delimiter=',')
convert_options = csv.ConvertOptions(
    timestamp_parsers=['%Y-%m-%d %H:%M:%S', '%Y-%m-%d']
)

table = csv.read_csv(
    'market_data.csv',
    read_options=read_options,
    parse_options=parse_options,
    convert_options=convert_options
)

Writing to Parquet with Snappy Compression

Once the table is in memory, converting to Parquet is a one‑liner:

pq.write_table(table, 'market_data.snappy.parquet', compression='snappy')

That’s it. PyArrow handles the columnar layout and applies Snappy automatically.

Writing to Parquet with ZSTD Compression

Switching to ZSTD requires only changing the compression parameter:

pq.write_table(table, 'market_data.zstd.parquet', compression='zstd')

You can also set a compression level (e.g., compression='zstd', compression_level=3) but the default is usually optimal.

Verifying the Output

You can quickly check that the conversion worked by reading the Parquet file back and inspecting its schema and row count:

reloaded = pq.read_table('market_data.zstd.parquet')
print(reloaded.schema)
print(f'Rows: {len(reloaded)}')

For a more in‑depth look at reading and writing Parquet files, see our companion guide: How to Read and Write Parquet Files in Python.

Benchmarking Speed and Storage Gains

Measuring Read/Write Performance

Quantitative workflows demand data, not guesswork. Use the time module to time each operation:

import time
import os

def time_conversion(csv_path, out_path, compression):
    t0 = time.perf_counter()
    table = csv.read_csv(csv_path)
    pq.write_table(table, out_path, compression=compression)
    elapsed = time.perf_counter() - t0
    size = os.path.getsize(out_path)
    return elapsed, size

snappy_time, snappy_size = time_conversion('market_data.csv', 'test.snappy.parquet', 'snappy')
zstd_time, zstd_size = time_conversion('market_data.csv', 'test.zstd.parquet', 'zstd')
print(f'Snappy: {snappy_time:.2f}s, {snappy_size/1e6:.1f} MB')
print(f'ZSTD  : {zstd_time:.2f}s, {zstd_size/1e6:.1f} MB')

Comparing File Sizes

Exact numbers will vary with the cardinality and distribution of your data. On a representative 2 GB OHLCV CSV covering 5 000 tickers over five years, Snappy typically compresses to roughly 300–500 MB, while ZSTD might shrink it to 150–250 MB. The write time for Snappy is usually shorter—sometimes by a factor of two or more—while read times are comparable. These benchmarks are instructive, but always measure your own dataset; highly random data will compress less, while neatly repeating patterns (like trading halts or constant bid/ask sizes) can compress even further.

Advanced Parquet Features for Quant Research Pipelines

Partitioning by Date or Asset

As your dataset grows to span decades or thousands of assets, a single massive Parquet file becomes unwieldy. Partitioning splits the data across a directory hierarchy based on column values. For example, you can partition OHLCV data by year and month:

import pyarrow.dataset as ds
ds.write_dataset(table, 'partitioned_data', format='parquet',
                 partitioning=['year', 'month'], compression='zstd')

This creates folders like year=2024/month=01/, each containing a small Parquet file. When your backtest only needs January 2024, the engine reads exactly that subset, dramatically reducing I/O and memory pressure.

Predicate Pushdown and Filter Optimisation

Parquet files store metadata about the min/max values of each column chunk. Modern readers like PyArrow use this metadata for predicate pushdown: they skip entire chunks that don’t match a filter before ever reading the data. Combine this with partitioning, and a query like “close > 100 AND date BETWEEN ‘2024-01-01’ AND ‘2024-01-31’” may only access a fraction of the total files. For walk‑forward, cross‑sectional signal forecasting—the exact evaluation framework used by AlphaNova—this ability to quickly retrieve relevant time slices and asset subsets can cut hours from a backtesting loop.

From Compressed Data to AlphaNova Signal Forecasting

Handling Obfuscated Tabular Data Efficiently

AlphaNova’s competitions revolve around a pure Python Predictor class that receives obfuscated tabular data for multiple assets at each evaluation period. Although the data formats are designed to be readable with standard tools, participants who store training and testing snapshots as Parquet gain an immediate edge. Compression reduces disk usage, but the real boost comes from fast columnar access during iterative feature engineering. When you’re experimenting with hundreds of signal variations, the time saved by reading only the necessary columns from a Parquet file compounds quickly.

Local Testing and Submission Workflow

Every AlphaNova participant gets a local runner for testing before submission. A typical workflow involves:

Downloading the obfuscated dataset once.
Converting it to Parquet (with ZSTD for archival, or Snappy for rapid iteration).
Using PyArrow to feed the Predictor during local tests.
Submitting the final Python class when satisfied.

Because the platform evaluates submissions out‑of‑sample using the Sharpe ratio, and a greedy quality‑selection process admits only uncorrelated, overfit‑filtered signals, the robustness of your model is paramount. A fast, repeatable data pipeline encourages the rigorous testing necessary to build signals that survive the selection filter. Participants retain full intellectual property, there are no staking requirements or token volatility to worry about, and prize pools scale with participation—top signals can even earn ongoing profit sharing. Cash prizes are paid in stablecoins or directly to a bank account.

If you’re ready to apply these Python skills to a real‑world forecasting challenge, Join the latest AlphaNova competition and start building the signals that could define your quant career.

Conclusion and Next Steps

Snappy or ZSTD? The answer depends on your priorities. If you rewrite and read data frequently during development, Snappy’s speed keeps your iteration loop tight. If you write once and read many times—common in systematic backtesting—ZSTD’s superior compression saves disk space and still loads quickly. In either case, moving from CSV to Parquet is one of the simplest, highest‑impact improvements you can make to your quant research infrastructure.

Experiment with your own datasets. Profile your workflow, measure the trade‑offs, and adopt the codec that fits. For a deeper dive into the Python ecosystem that supports production‑grade quant work, don’t miss our article on Python for Quants: Essential Libraries & Tools to Master Quantitative Finance. Armed with these tools, you’ll be ready to tackle the kind of walk‑forward forecasting competitions that separate robust signals from overfit noise.

Join the latest AlphaNova competition