koushikkhan commited on
Commit
bca2d8f
·
1 Parent(s): d727016

polars dir created

Browse files
polars/001_why_polars.py ADDED
@@ -0,0 +1,232 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import marimo
2
+
3
+ __generated_with = "0.11.0"
4
+ app = marimo.App(width="medium")
5
+
6
+
7
+ @app.cell
8
+ def _():
9
+ import marimo as mo
10
+ return (mo,)
11
+
12
+
13
+ @app.cell
14
+ def _(mo):
15
+ mo.md(
16
+ """
17
+ # What is Polars?
18
+
19
+ [Polars](https://pola.rs/) is a blazingly fast, efficient, and user-friendly DataFrame library designed for data manipulation and analysis in Python. Built with performance in mind, Polars leverages the power of Rust under the hood, enabling it to handle large datasets with ease while maintaining a simple and intuitive API. Whether you're working with structured data, performing complex transformations, or analyzing massive datasets, Polars is designed to deliver exceptional speed and memory efficiency, often outperforming other popular DataFrame libraries like Pandas.
20
+
21
+ One of the standout features of Polars is its ability to perform operations in a parallelized and vectorized manner, making it ideal for modern data processing tasks. It supports a wide range of data types, advanced query optimizations, and seamless integration with other Python libraries, making it a versatile tool for data scientists, engineers, and analysts. Additionally, Polars provides a lazy API for deferred execution, allowing users to optimize their workflows by chaining operations and executing them in a single pass.
22
+
23
+ With its focus on speed, scalability, and ease of use, Polars is quickly becoming a go-to choice for data professionals looking to streamline their data processing pipelines and tackle large-scale data challenges. Whether you're analyzing gigabytes of data or performing real-time computations, Polars empowers you to work faster and smarter.
24
+ """
25
+ )
26
+ return
27
+
28
+
29
+ @app.cell
30
+ def _(mo):
31
+ mo.md(
32
+ """
33
+ # Why Polars?
34
+
35
+ Pandas has long been the go-to library for data manipulation and analysis in Python. However, as datasets grow larger and more complex, Pandas often struggles with performance and memory limitations. This is where Polars shines. Polars is a modern, high-performance DataFrame library designed to address the shortcomings of Pandas while providing a user-friendly experience.
36
+
37
+ Below, we’ll explore key reasons why Polars is a better choice in many scenarios, along with examples.
38
+
39
+ ## (a) Easier Syntax Similar
40
+
41
+ Polars is designed with a syntax that is very similar to PySpark while being intuitive like SQL. This makes it easier for data professionals to transition to Polars without a steep learning curve. For example:
42
+
43
+ **Example: Filtering and Aggregating Data**
44
+
45
+ **In Pandas:**
46
+ ```
47
+ import pandas as pd
48
+
49
+ df = pd.DataFrame(
50
+ {
51
+ "Gender": ["Male", "Female", "Male", "Female", "Male", "Female",
52
+ "Male", "Female", "Male", "Female"],
53
+ "Age": [13, 15, 17, 19, 21, 23, 25, 27, 29, 31],
54
+ "Height_CM": [150, 170, 146.5, 142, 155, 165, 170.8, 130, 132.5, 162]
55
+ }
56
+ )
57
+
58
+ # query: average height of male and female after the age of 15 years
59
+
60
+ # step-1: filter
61
+ filtered_df = df[df["Age"] > 15]
62
+
63
+ # step-2: groupby and aggregation
64
+ result = filtered_df.groupby("Gender").mean()
65
+ ```
66
+
67
+ **In Polars:**
68
+ ```
69
+ import polars as pl
70
+
71
+ df = pd.DataFrame(
72
+ {
73
+ "Gender": ["Male", "Female", "Male", "Female", "Male", "Female",
74
+ "Male", "Female", "Male", "Female"],
75
+ "Age": [13, 15, 17, 19, 21, 23, 25, 27, 29, 31],
76
+ "Height_CM": [150, 170, 146.5, 142, 155, 165, 170.8, 130, 132.5, 162]
77
+ }
78
+ )
79
+
80
+ # query: average height of male and female after the age of 15 years
81
+
82
+ # filter, groupby and aggregation using method chaining
83
+ result = df_pl.filter(pl.col("Age") > 15).group_by("Gender").agg(pl.mean("Height_CM"))
84
+ ```
85
+
86
+ Notice how Polars uses a *method-chaining* approach, similar to PySpark, which makes the code more readable and expressive while using a *single line* to design the query.
87
+
88
+ Additionally, Polars supports SQL-like operations *natively*, that allows you to write SQL queries directly on polars dataframe:
89
+
90
+ ```
91
+ result = df.sql("SELECT Gender, AVG(Height_CM) FROM self WHERE Age > 15 GROUP BY Gender")
92
+ ```
93
+
94
+ ## (b) Scalability - Handling Large Datasets in Memory
95
+
96
+ Pandas is limited by its single-threaded design and reliance on Python, which makes it inefficient for processing large datasets. Polars, on the other hand, is built in Rust and optimized for parallel processing, enabling it to handle datasets that are orders of magnitude larger.
97
+
98
+ **Example: Processing a Large Dataset**
99
+ In Pandas, loading a large dataset (e.g., 10GB) often results in memory errors:
100
+
101
+ ```
102
+ # This may fail with large datasets
103
+ df = pd.read_csv("large_dataset.csv")
104
+ ```
105
+
106
+ In Polars, the same operation is seamless:
107
+
108
+ ```
109
+ df = pl.read_csv("large_dataset.csv")
110
+ ```
111
+
112
+ Polars also supports lazy evaluation, which allows you to optimize your workflows by deferring computations until necessary. This is particularly useful for large datasets:
113
+
114
+ ```
115
+ df = pl.scan_csv("large_dataset.csv") # Lazy DataFrame
116
+ result = df.filter(pl.col("A") > 1).groupby("A").agg(pl.sum("B")).collect() # Execute
117
+ ```
118
+
119
+ ## (c) Compatibility with Other Machine Learning Libraries
120
+
121
+ Polars integrates seamlessly with popular machine learning libraries like Scikit-learn, PyTorch, and TensorFlow. Its ability to handle large datasets efficiently makes it an excellent choice for preprocessing data before feeding it into ML models.
122
+
123
+ **Example: Preprocessing Data for Scikit-learn**
124
+
125
+ ```
126
+ import polars as pl
127
+ from sklearn.linear_model import LinearRegression
128
+
129
+ # Load and preprocess data
130
+ df = pl.read_csv("data.csv")
131
+ X = df.select(["feature1", "feature2"]).to_numpy()
132
+ y = df.select("target").to_numpy()
133
+
134
+ # Train a model
135
+ model = LinearRegression()
136
+ model.fit(X, y)
137
+ ```
138
+
139
+ Polars also supports conversion to other formats like NumPy arrays and Pandas DataFrames, ensuring compatibility with virtually any ML library:
140
+
141
+ ```
142
+ # Convert to Pandas DataFrame
143
+ pandas_df = df.to_pandas()
144
+
145
+ # Convert to NumPy array
146
+ numpy_array = df.to_numpy()
147
+ ```
148
+
149
+ **(d) Additional Advantages of Polars**
150
+
151
+ - Rich Functionality: Polars supports advanced operations like window functions, joins, and nested data types, making it a versatile tool for data manipulation.
152
+
153
+ - Query Optimization: Polars is significantly faster than Pandas due to its parallelized and vectorized operations. Benchmarks often show Polars outperforming Pandas by 10x or more.
154
+
155
+ - Memory Efficiency: Polars uses memory more efficiently, reducing the risk of out-of-memory errors.
156
+
157
+ - Lazy API: The lazy evaluation API allows for query optimization and deferred execution, which is particularly useful for complex workflows.
158
+ """
159
+ )
160
+ return
161
+
162
+
163
+ @app.cell
164
+ def _():
165
+ import pandas as pd
166
+
167
+ df = pd.DataFrame(
168
+ {
169
+ "Gender": ["Male", "Female", "Male", "Female", "Male", "Female",
170
+ "Male", "Female", "Male", "Female"],
171
+ "Age": [13, 15, 17, 19, 21, 23, 25, 27, 29, 31],
172
+ "Height_CM": [150, 170, 146.5, 142, 155, 165, 170.8, 130, 132.5, 162]
173
+ }
174
+ )
175
+
176
+ # query: average height of male and female after the age of 15 years
177
+ filtered_df = df[df["Age"] > 15]
178
+ result = filtered_df.groupby("Gender").mean()["Height_CM"]
179
+ result
180
+ return df, filtered_df, pd, result
181
+
182
+
183
+ @app.cell
184
+ def _():
185
+ import polars as pl
186
+ return (pl,)
187
+
188
+
189
+ @app.cell
190
+ def _(pl):
191
+ df_pl = pl.DataFrame(
192
+ {
193
+ "Gender": ["Male", "Female", "Male", "Female", "Male", "Female",
194
+ "Male", "Female", "Male", "Female"],
195
+ "Age": [13, 15, 17, 19, 21, 23, 25, 27, 29, 31],
196
+ "Height_CM": [150.0, 170.0, 146.5, 142.0, 155.0, 165.0, 170.8, 130.0, 132.5, 162.0]
197
+ }
198
+ )
199
+
200
+ # df_pl
201
+ # query: average height of male and female after the age of 15 years
202
+ result_pl = df_pl.filter(pl.col("Age") > 15).group_by("Gender").agg(pl.mean("Height_CM"))
203
+ result_pl
204
+ return df_pl, result_pl
205
+
206
+
207
+ @app.cell
208
+ def _(df_pl):
209
+ df_pl.sql("SELECT Gender, AVG(Height_CM) FROM self WHERE Age > 15 GROUP BY Gender")
210
+ return
211
+
212
+
213
+ @app.cell
214
+ def _(mo):
215
+ mo.md(
216
+ """
217
+ # 🔖 References
218
+
219
+ - [Polars official website](https://pola.rs/)
220
+ - [Polars Vs. Pandas](https://blog.jetbrains.com/pycharm/2024/07/polars-vs-pandas/)
221
+ """
222
+ )
223
+ return
224
+
225
+
226
+ @app.cell
227
+ def _():
228
+ return
229
+
230
+
231
+ if __name__ == "__main__":
232
+ app.run()
polars/layouts/001_why_polars.slides.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "type": "slides",
3
+ "data": {}
4
+ }