How to Reduce Pandas DataFrame Memory Usage by 50%

How to Reduce Pandas DataFrame Memory Usage by 50% for Data Science Competitions

You’ve downloaded the competition dataset. It’s a tidy, zipped CSV of maybe 2 GB — data.csv.zip. You unzip, fire up a Jupyter notebook, and run pd.read_csv('data.csv'). Your laptop fan screams. Memory usage balloons to 10 GB, and before you can inspect the first five rows, the kernel dies.

This scenario is painfully common in data science competitions, especially in environments with constrained resources. The culprit isn’t just the file size; it’s how pandas stores that data in RAM. A 2 GB CSV routinely expands to 6–10 GB inside a DataFrame, leaving little room for actual modelling and iteration.

The good news: you can often cut that memory footprint in half — or more — by applying a few deliberate, well-understood techniques. This post shows you exactly how.

Why Pandas Eats More RAM Than You Think

Before optimizing, it helps to understand where the memory goes. Pandas builds on top of NumPy, but its default type choices and object-based string handling can introduce significant overhead.

Python object overhead for non-numeric data

Numeric columns in pandas are stored as NumPy arrays with fixed-width dtypes such as int64 or float64. Those are efficient. The real memory explosion happens when a column is stored as object dtype. Each value then becomes a pointer to a separate Python object such as a str, int, or float. A single Python int object may consume 28 bytes on a 64-bit system, while a NumPy int64 scalar needs only 8 bytes.

When pandas stores mixed-type columns or an object dtype column, every entry becomes a pointer to a separate Python object — and memory usage skyrockets. Even a numeric column read from CSV often lands as float64 or int64 by default, which brings us to the next issue.

Default heavy numeric types

Pandas assumes the broadest numeric containers: float64 for decimals and int64 for integers. In many competition datasets, the actual range of values can fit comfortably into a smaller subtype.

A column of integer asset IDs between 0 and 100,000 could be stored in int32 — 4 bytes instead of 8. If a column contains integer counts such as shares traded in thousands or days-to-expiry that remain within ±32,767, int16 may suffice. Float columns with only a few decimal places of precision can be squeezed into float32, halving memory instantly.

These default choices alone explain why your 2 GB file blows up to 6 GB or more.

Diagnosing Memory Hogs with `.memory_usage(deep=True)`

The first step to trimming memory is knowing where the fat lives. Pandas provides a straightforward method for that.

Identifying heavy columns

After loading your dataset, run df.memory_usage(deep=True) to see a breakdown of memory consumption per column, including the deep object memory of strings and any Python objects. The output is a Series that you can sort in descending order:

df = pd.read_csv('large_dataset.csv')
mem_usage = df.memory_usage(deep=True).sort_values(ascending=False)
print(mem_usage.head(10))

You’ll quickly spot columns of type object — likely strings — or wide numeric columns that are the top offenders. Often, a handful of columns account for 80% of the memory.

Measuring potential savings

Before making changes, record the total memory:

baseline_memory = df.memory_usage(deep=True).sum()
print(f"Baseline memory usage: {baseline_memory / 1e6:.2f} MB")

This baseline tells you the improvement you achieve later. By targeting the worst columns first, you get the largest gains with the least work.

Once you’ve identified the hogs, it’s time to slim them down.

Downcasting Numeric Types: From float64 to float32

The easiest and most impactful optimisation is to shrink numeric columns to the smallest dtype that can hold their values without overflow or loss of needed precision.

Downcasting ints: int64 → int32/16/8

For integer columns, check the minimum and maximum values. If the column is non-negative and max ≤ 255, uint8 works; if max ≤ 65,535, uint16; if max ≤ 2³¹‑1, int32.

You can downcast manually with .astype():

df['asset_id'] = df['asset_id'].astype('uint16')
df['day_of_week'] = df['day_of_week'].astype('int8')

A safer, automated approach uses pd.to_numeric(..., downcast='integer') or downcast='unsigned', which lets pandas choose the tightest integer subtype on the fly.

Downcasting floats: float64 → float32

Many financial features, like returns or ratios, don’t need 64-bit double precision. A float32 still offers about 7 significant digits, which is more than plenty for most competition features. Conversion is a one-liner:

df['feature_A'] = df['feature_A'].astype('float32')

If you want pandas to automatically downcast all numeric columns in a DataFrame, you can loop:

for col in df.select_dtypes(include=['float64']).columns:
    df[col] = pd.to_numeric(df[col], downcast='float')

Using `pd.to_numeric` with downcast='unsigned' or 'integer'

Pandas can handle integer downcasting similarly, applying the same column-wise logic:

for col in df.select_dtypes(include=['int64']).columns:
    # Use downcast='unsigned' if all values are non-negative
    df[col] = pd.to_numeric(df[col], downcast='integer')

After downcasting, re-run df.memory_usage(deep=True).sum() — you’ll frequently see a 50% reduction on numeric-heavy DataFrames, and sometimes much more.

Converting Strings to Category Dtype

Financial datasets are littered with repetitive strings: sector labels, ticker symbols, country codes, or, in AlphaNova competitions, obfuscated asset IDs that are stored as text. Storing each occurrence as an independent Python string object is breathtakingly wasteful.

When categories save memory

If a column has far fewer unique values than its total length, you can convert it to the category dtype. Internally, pandas replaces each repeated string with an integer code and keeps a small lookup dictionary. Memory drops dramatically.

df['asset_id'] = df['asset_id'].astype('category')

A good rule of thumb: if the number of unique values divided by the total rows is below 0.5 — and ideally below 0.1 — categorical will almost certainly be a net win. You can check cardinality before converting:

cardinality = df['sector'].nunique() / len(df)
print(f"Cardinality ratio: {cardinality:.2%}")

if cardinality < 0.5:
    df['sector'] = df['sector'].astype('category')

Avoiding high-cardinality pitfalls

Columns where nearly every row is unique — timestamps, IDs that are truly distinct per observation, or free-form notes — will not benefit from categorical. In fact, they can consume more memory because the internal integer codes are stored alongside the mapping. Always check nunique() before converting.

For high-cardinality string columns, consider whether they can be split, dropped, or processed outside of pandas.

When applied correctly, categorical conversion can turn a multi-gigabyte object column into a few tens of megabytes.

Selective Loading: Only Read What You Need

A smarter way to stay under memory limits is to never load the full dataset in the first place. Pandas offers three powerful parameters that let you control what enters memory.

Using the columns parameter

pd.read_csv can read only specified columns, discarding the rest on the fly:

cols_to_use = ['date', 'asset_id', 'return', 'feature_1', 'feature_2']
df = pd.read_csv('data.csv', usecols=cols_to_use)

This is especially useful when you know in advance which features your model needs. It reduces both reading time and memory consumption.

Specifying dtypes at load time

Instead of reading everything as the default heavy types and then downcasting, you can supply a dtype dictionary directly to read_csv(). This eliminates the overhead of an initial oversized allocation:

dtype_dict = {
    'asset_id': 'category',
    'return': 'float32',
    'feature_1': 'float32',
    'feature_2': 'int16'
}

df = pd.read_csv(
    'data.csv',
    usecols=cols_to_use,
    dtype=dtype_dict,
    parse_dates=['date']
)

The combination of usecols and dtype often brings the same dataset into memory at a fraction of the original size, avoiding the painful 6 GB spike entirely.

Chunking large files

For files that are still too large even after column and dtype selection, process them in smaller pieces with the chunksize parameter. Each chunk is a manageable DataFrame you can aggregate or filter before moving to the next:

chunks = []

for chunk in pd.read_csv(
    'data.csv',
    chunksize=100_000,
    dtype=dtype_dict,
    usecols=cols_to_use
):
    # Perform some filtering or feature engineering
    chunk = chunk[chunk['date'] >= '2020-01-01']
    chunks.append(chunk)

df = pd.concat(chunks, ignore_index=True)

Where appropriate, converting the raw CSV to Apache Parquet once and reading it with pd.read_parquet() can further slash load times and memory overhead. Learn more in our practical guide on how to read and write Parquet files in Python.

Applying These Techniques to AlphaNova Competitions

The memory-saving methods we’ve discussed are especially valuable in AlphaNova’s quantitative finance competitions, where participants work with obfuscated financial data and must submit a pure Python Predictor class.

Working with obfuscated financial data

AlphaNova and other competitions provide tabular datasets covering multiple assets per time period, with anonymised feature names and values.

Although the data is obfuscated, the underlying patterns — repetitive asset IDs, price-like float columns, and categorical indicators — remain. Converting asset IDs to category and downcasting numeric features to float32 or int16 can yield immediate, drastic memory reductions, letting you load the entire training set on a standard laptop.

Memory constraints in walk-forward forecasting

AlphaNova's competition format uses a walk-forward, cross-sectional signal forecasting setup. You train your model on historical windows, produce predictions for the next unseen period, and iterate. This process may run inside AlphaNova’s local runner — a lightweight simulation environment provided to test your code before submission.

Keeping your pipeline lean protects you from out-of-memory errors during local testing. Efficient data handling also speeds up the feedback loop, allowing you to try more ideas in less time.

Participants whose predictors handle memory efficiently can focus on signal quality rather than wrestling with infrastructure. And because AlphaNova evaluates submissions out-of-sample using the Sharpe ratio and admits only genuinely uncorrelated, overfit-filtered signals through a greedy selection process, every saved megabyte contributes to a faster, smoother experimentation loop — without any guarantee of returns, of course.

A Proven Workflow for Competition-Ready Data

Putting it all together, here is a repeatable recipe that consistently halves memory usage and sometimes goes well beyond.

Step-by-step recipe

Load a small sample — for example, the first 50,000 rows — to explore column types, cardinalities, and value ranges.
Diagnose memory with df.memory_usage(deep=True) on the sample.
Downcast numeric columns — automatically with pd.to_numeric(..., downcast='integer'/'float') or manually using astype().
Convert low-cardinality strings to category after checking nunique().
Create a dtype dictionary that captures these optimisations.
Use usecols and chunksize in pd.read_csv() to load — or reload — the full dataset efficiently.
Optional: Save the cleaned DataFrame as Parquet for instant restarts.

Validating memory savings

After each step, compare the output of df.memory_usage(deep=True).sum() to your original baseline. A 50% reduction is realistic, and many competition datasets shrink by 70% or more.

Once your data is lean, you can turn your attention to modelling without the fear of a sudden kernel death.

For further performance gains, learn to profile and speed up your pandas pipeline in our companion article: How to Profile and Speed Up a Slow Pandas Pipeline.

Frequently Asked Questions

How much memory can I save with these techniques?

Most numeric-heavy datasets shrink by at least 50% after downcasting. Datasets with many repetitive string columns often shrink by 70% or more when converted to category.

Is float32 safe for financial features?

For most competition features — returns, ratios, standardized values — float32 provides about 7 significant digits, which is generally sufficient. Avoid float32 for values that require exact decimal precision, such as monetary amounts stored as floats.

When should I avoid categorical dtype?

Avoid categorical when a column has very high cardinality — for example, a unique ID for every row. The mapping overhead can exceed the savings. Always check nunique() before converting.

Next Steps: Put It to the Test

The best way to internalize these techniques is to apply them to a real dataset under real constraints. AlphaNova competitions are free to enter, provide obfuscated financial data, and reward uncorrelated signals based on the Sharpe ratio. You retain full intellectual property, and prize pools are paid in stablecoins or directly to your bank account — no staking, no token volatility.

A local runner lets you test your Predictor class before submission, so you can immediately see how much memory you’ve saved and how much faster your iterations become.

Join the latest AlphaNova competition and put these memory-optimisation tips into practice. Your pipeline — and your model iteration speed — will thank you.

How to Reduce Pandas DataFrame Memory Usage by 50%