Polars vs. Pandas — An Independent Speed Comparison

Overview

1. Introduction — Purpose and Reasons
2. Datasets, Tasks, and Settings
3. Results
4. Conclusions
5. Wrapping Up

Introduction — Objective and Causes

Velocity is vital when coping with massive quantities of knowledge. If you’re dealing with information in a cloud information warehouse or comparable, then the pace of execution to your information ingestion and processing impacts the next:

• Cloud prices: That is in all probability the largest issue. Extra compute time equals extra prices in most billing fashions. In different billing primarily based on a specific amount of preallocated assets, you can have chosen a decrease service stage if the pace of your ingestion and processing was larger.
• Information timeliness: When you’ve got a real-time stream that takes 5 minutes to course of information, then your customers could have a lag of at the least 5 minutes when viewing the info by e.g. a Energy BI rapport. This distinction generally is a lot in sure conditions. Even for batch jobs, the info timeliness is vital. If you’re working a batch job each hour, it’s a lot higher if it takes 2 minutes relatively than 20 minutes.
• Suggestions loop: In case your batch job takes solely a minute to run, you then get a really fast suggestions loop. This in all probability makes your job extra pleasant. As well as, it lets you discover logical errors extra rapidly.

As you’ve in all probability understood from the title, I’m going to supply a pace comparability between the 2 Python libraries Polars and Pandas. If you already know something about Pandas and Polars from earlier than, then you already know that Polars is the (comparatively) new child on the block proclaiming to be a lot sooner than Pandas. You in all probability additionally know that Polars is carried out in Rust, which is a development for a lot of different trendy Python instruments like uv and Ruff.

There are two distinct causes that I wish to do a pace comparability check between Polars and Pandas:

Purpose 1 — Investigating Claims

Polars boasts on its web site with the next declare: In comparison with pandas, it (Polars) can obtain greater than 30x efficiency beneficial properties.

As you possibly can see, you possibly can comply with a link to the benchmarks that they’ve. It’s commendable that they’ve pace assessments open supply. However if you’re writing the comparability assessments for each your personal software and a competitor’s software, then there may be a slight battle of curiosity. I’m not right here saying that they’re purposefully overselling the pace of Polars, however relatively that they could have unconsciously chosen for favorable comparisons.

Therefore the primary cause to do a pace comparability check is solely to see whether or not this helps the claims introduced by Polars or not.

Purpose 2 — Larger granularity

One more reason for doing a pace comparability check between Polars and Pandas is to make it barely extra clear the place the efficiency beneficial properties may be.

This may be already clear if you happen to’re an knowledgeable on each libraries. Nonetheless, pace assessments between Polars and Pandas are largely of curiosity to these contemplating switching up their software. In that case, you may not but have performed round a lot with Polars since you are not sure whether it is value it.

Therefore the second cause to do a pace comparability is solely to see the place the pace beneficial properties are situated.

I wish to check each libraries on completely different duties each inside information ingestion and Data Processing. I additionally wish to contemplate datasets which might be each small and enormous. I’ll persist with frequent duties inside information engineering, relatively than esoteric duties that one seldom makes use of.

What I cannot do

• I cannot give a tutorial on both Pandas or Polars. If you wish to be taught Pandas or Polars, then place to start out is their documentation.
• I cannot cowl different frequent information processing libraries. This may be disappointing to a fan of PySpark, however having a distributed compute mannequin makes comparisons a bit tougher. You would possibly discover that PySpark is faster than Polars on duties which might be very straightforward to parallelize, however slower on different duties the place maintaining all the info in reminiscence reduces journey instances.
• I cannot present full reproducibility. Since that is, in humble phrases, solely a weblog put up, then I’ll solely clarify the datasets, duties, and system settings that I’ve used. I cannot host an entire working surroundings with the datasets and bundle all the things neatly. This isn’t a exact scientific experiment, however relatively a information that solely cares about tough estimations.

Lastly, earlier than we begin, I wish to say that I like each Polars and Pandas as instruments. I’m not financially or in any other case compensated by any of them clearly, and don’t have any incentive aside from being interested by their efficiency ☺️

Datasets, Duties, and Settings

Let’s first describe the datasets that I will probably be contemplating, the duties that the libraries will carry out, and the system settings that I will probably be working them on.

Datasets

A most firms, you have to to work with each small and (comparatively) massive datasets. In my view, information processing software can deal with each ends of the spectrum. Small datasets problem the start-up time of duties, whereas bigger datasets problem scalability. I’ll contemplate two datasets, each could be discovered on Kaggle:

• A small dataset on the format CSV: It’s no secret that CSV information are in every single place! Typically they’re fairly small, coming from Excel information or database dumps. What higher instance of this than the classical iris dataset (licensed with CC0 1.0 Universal License) with 5 columns and 150 rows. The iris model I linked to on Kaggle has 6 columns, however the classical one doesn’t have a working index column. So take away this column if you would like exactly the identical dataset as I’ve. The iris dataset is definitely small information by any stretch of the creativeness.
• A big dataset on the format Parquet: The parquet format is tremendous helpful for big information because it has built-in compression column-wise (together with many different advantages). I’ll use the Transaction dataset (licensed with Apache License 2.0) representing monetary transactions. The dataset has 24 columns and seven 483 766 rows. It’s shut to three GB in its CSV format discovered on Kaggle. I used Pandas & Pyarrow to transform this to a parquet file. The ultimate result’s solely 905 MB because of the compression of the parquet file format. That is on the low finish of what individuals name huge information, however it’ll suffice for us.

Duties

I’ll do a pace comparability on 5 completely different duties. The primary two are I/O duties, whereas the final three are frequent duties in information processing. Particularly, the duties are:

1. Studying information: I’ll learn each information utilizing the respective strategies read_csv() and read_parquet() from the 2 libraries. I cannot use any non-compulsory arguments as I wish to examine their default habits.
2. Writing information: I’ll write each information again to an identical copies as new information utilizing the respective strategies to_csv() and to_parquet() for Pandas and write_csv() and write_parquet() for Polars. I cannot use any non-compulsory arguments as I wish to examine their default habits.
3. Computing Numeric Expressions: For the iris dataset I’ll compute the expression SepalLengthCm ** 2 + SepalWidthCm as a brand new column in a duplicate of the DataFrame. For the transactions dataset, I’ll merely compute the expression (quantity + 10) ** 2 as a brand new column in a duplicate of the DataFrame. I’ll use the usual method to transform columns in Pandas, whereas in Polars I’ll use the usual features all(), col(), and alias() to make an equal transformation.
4. Filters: For the iris dataset, I’ll choose the rows similar to the factors SepalLengthCm >= 5.0 and SepalWidthCm . For the transactions dataset, I'll choose the rows similar to the explicit standards merchant_category == 'Restaurant'. I'll use the usual filtering technique primarily based on Boolean expressions in every library. In pandas, that is syntax equivalent to df_new = df[df['col'] , whereas in Polars that is given equally by the filter() operate together with the col() operate. I'll use the and-operator & for each libraries to mix the 2 numeric circumstances for the iris dataset.
5. Group By: For the iris dataset, I’ll group by the Species column and calculate the imply values for every species of the 4 columns SepalLengthCm, SepalWidthCm, PetalLengthCm, and PetalWidthCm. For the transactions dataset, I’ll group by the column merchant_category and depend the variety of cases in every of the courses inside merchant_category. Naturally, I’ll use the groupby() operate in Pandas and the group_by() operate in Polars in apparent methods.

Settings

• System Settings: I’m working all of the duties domestically with 16GB RAM and an Intel Core i5–10400F CPU with 6 Cores (12 logical cores by hyperthreading). So it’s not state-of-the-art by any means, however ok for easy benchmarking.
• Python: I’m working Python 3.12. This isn’t essentially the most present secure model (which is Python 3.13), however I feel it is a good factor. Generally the newest supported Python model in cloud information warehouses is one or two variations behind.
• Polars & Pandas: I’m utilizing Polars model 1.21 and Pandas 2.2.3. These are roughly the latest secure releases to each packages.
• Timeit: I’m utilizing the usual timeit module in Python and discovering the median of 10 runs.

Particularly fascinating will probably be how Polars can make the most of the 12 logical cores by multithreading. There are methods to make Pandas make the most of a number of processors, however I wish to examine Polars and Pandas out of the field with none exterior modification. In any case, that is in all probability how they’re working in most firms world wide.

Outcomes

Right here I’ll write down the outcomes for every of the 5 duties and make some minor feedback. Within the subsequent part I’ll attempt to summarize the details right into a conclusion and level out a drawback that Polars has on this comparability:

Process 1 — Studying information

The median run time over 10 runs for the studying activity was as follows:

# Iris Dataset
Pandas: 0.79 milliseconds
Polars: 0.31 milliseconds

# Transactions Dataset
Pandas: 14.14 seconds
Polars: 1.25 seconds

For studying the Iris dataset, Polars was roughly 2.5x sooner than Pandas. For the transactions dataset, the distinction is even starker the place Polars was 11x sooner than Pandas. We are able to see that Polars is far sooner than Pandas for studying each small and enormous information. The efficiency distinction grows with the dimensions of the file.

Process 2— Writing information

The median run time in seconds over 10 runs for the writing activity was as follows:

# Iris Dataset
Pandas: 1.06 milliseconds
Polars: 0.60 milliseconds

# Transactions Dataset
Pandas: 20.55 seconds
Polars: 10.39 seconds

For writing the iris dataset, Polars was round 75% sooner than Pandas. For the transactions dataset, Polars was roughly 2x as quick as Pandas. Once more we see that Polars is quicker than Pandas, however the distinction right here is smaller than for studying information. Nonetheless, a distinction of near 2x in efficiency is an enormous distinction.

Process 3 —Computing Numeric Expressions

The median run time over 10 runs for the computing numeric expressions activity was as follows:

# Iris Dataset
Pandas: 0.35 milliseconds
Polars: 0.15 milliseconds

# Transactions Dataset
Pandas: 54.58 milliseconds
Polars: 14.92 milliseconds

For computing the numeric expressions, Polars beats Pandas with a fee of roughly 2.5x for the iris dataset, and roughly 3.5x for the transactions dataset. It is a fairly large distinction. It must be famous that computing numeric expressions is quick in each libraries even for the massive dataset transactions.

Process 4 — Filters

The median run time over 10 runs for the filters activity was as follows:

# Iris Dataset
Pandas: 0.40 milliseconds
Polars: 0.15 milliseconds

# Transactions Dataset
Pandas: 0.70 seconds
Polars: 0.07 seconds

For filters, Polars is 2.6x sooner on the iris dataset and 10x as quick on the transactions dataset. That is in all probability essentially the most shocking enchancment for me since I suspected that the pace enhancements for filtering duties wouldn’t be this large.

Process 5 — Group By

The median run time over 10 runs for the group by activity was as follows:

# Iris Dataset
Pandas: 0.54 milliseconds
Polars: 0.18 milliseconds

# Transactions Dataset
Pandas: 334 milliseconds 
Polars: 126 milliseconds

For the group-by activity, there’s a 3x pace enchancment for Polars within the case of the iris dataset. For the transactions dataset, there’s a 2.6x enchancment of Polars over Pandas.

Conclusions

Earlier than highlighting every level under, I wish to level out that Polars is considerably in an unfair place all through my comparisons. It’s typically that a number of information transformations are carried out after each other in follow. For this, Polars has the lazy API that optimizes this earlier than calculating. Since I’ve thought of single ingestions and transformations, this benefit of Polars is hidden. How a lot this might enhance in sensible conditions will not be clear, however it could in all probability make the distinction in efficiency even greater.

Information Ingestion

Polars is considerably sooner than Pandas for each studying and writing information. The distinction is largest in studying information, the place we had an enormous 11x distinction in efficiency for the transactions dataset. On all measurements, Polars performs considerably higher than Pandas.

Information Processing

Polars is considerably sooner than Pandas for frequent information processing duties. The distinction was starkest for filters, however you possibly can at the least count on a 2–3x distinction in efficiency throughout the board.

Ultimate Verdict

Polars persistently performs sooner than Pandas on all duties with each small and enormous information. The enhancements are very important, starting from a 2x enchancment to a whopping 11x enchancment. With regards to studying massive parquet information or performing filter statements, Polars is leaps and certain in entrance of Pandas.

Nonetheless…Nowhere right here is Polars remotely near performing 30x higher than Pandas, as Polars’ benchmarking suggests. I’d argue that the duties that I’ve introduced are commonplace duties carried out on life like {hardware} infrastructure. So I feel that my conclusions give us some room to query whether or not the claims put ahead by Polars give a sensible image of the enhancements that you may count on.

Nonetheless, I’m in little question that Polars is considerably sooner than Pandas. Working with Polars will not be extra difficult than working with Pandas. So to your subsequent information engineering venture the place the info matches in reminiscence, I’d strongly counsel that you just go for Polars relatively than Pandas.

Wrapping Up

I hope this weblog put up gave you a special perspective on the pace distinction between Polars and Pandas. Please remark if in case you have a special expertise with the efficiency distinction between Polars and Pandas than what I’ve introduced.

If you’re all for AI, Data Science, or information engineering, please comply with me or join on LinkedIn.

Like my writing? Take a look at a few of my different posts: