updated why_polars

polars/001_why_polars.py

@@ -36,14 +36,14 @@ def _(mo):
 
         Below, we’ll explore key reasons why Polars is a better choice in many scenarios, along with examples.
 
-        ## (a) Easier Syntax Similar 
+        ## (a) Easier & Intuitive Syntax
 
         Polars is designed with a syntax that is very similar to PySpark while being intuitive like SQL. This makes it easier for data professionals to transition to Polars without a steep learning curve. For example:
 
         **Example: Filtering and Aggregating Data**
 
         **In Pandas:**
-        ```
+        ```{python}
         import pandas as pd
 
         df = pd.DataFrame(
@@ -65,7 +65,7 @@ def _(mo):
         ```
 
         **In Polars:**
-        ```
+        ```{python}
         import polars as pl
 
         df = pd.DataFrame(
@@ -87,42 +87,62 @@ def _(mo):
 
         Additionally, Polars supports SQL-like operations *natively*, that allows you to write SQL queries directly on polars dataframe:
 
-        ```
+        ```{python}
         result = df.sql("SELECT Gender, AVG(Height_CM) FROM self WHERE Age > 15 GROUP BY Gender")
         ```
 
-        ## (b) Scalability - Handling Large Datasets in Memory
+        ## (b) Large Collection of Built-in APIs
+
+        Polars boasts an **extremely expressive API**, enabling you to perform virtually any operation using built-in methods. In contrast, Pandas often requires more complex operations to be handled using the `apply` method with a lambda function. The issue with `apply` is that it processes rows sequentially, looping through the DataFrame one row at a time, which can be inefficient. By leveraging Polars' built-in methods, you can operate on entire columns at once, unlocking the power of **SIMD (Single Instruction, Multiple Data)** parallelism. This approach not only simplifies your code but also significantly enhances performance.
+
+        ## (c) Query Optimization
+
+        A key factor behind Polars' performance lies in its **evaluation strategy**. While Pandas defaults to **eager execution**, executing operations in the exact order they are written, Polars offers both **eager and lazy execution**. With lazy execution, Polars employs a **query optimizer** that analyzes all required operations and determines the most efficient way to execute them. This optimization can involve reordering operations, eliminating redundant calculations, and more. 
+
+        For example, consider the following expression to calculate the mean of the `Number1` column for categories "A" and "B" in the `Category` column:
+
+        ```{python}
+        (
+            df
+            .groupby(by="Category").agg(pl.col("Number1").mean())
+            .filter(pl.col("Category").is_in(["A", "B"]))
+        )
+        ```
+
+        If executed eagerly, the `groupby` operation would first be applied to the entire DataFrame, followed by filtering the results by `Category`. However, with **lazy execution**, Polars can optimize this process by first filtering the DataFrame to include only the relevant categories ("A" and "B") and then performing the `groupby` operation on the reduced dataset. This approach minimizes unnecessary computations and significantly improves efficiency.
+
+        ## (d) Scalability - Handling Large Datasets in Memory
 
         Pandas is limited by its single-threaded design and reliance on Python, which makes it inefficient for processing large datasets. Polars, on the other hand, is built in Rust and optimized for parallel processing, enabling it to handle datasets that are orders of magnitude larger.
 
         **Example: Processing a Large Dataset**
         In Pandas, loading a large dataset (e.g., 10GB) often results in memory errors:
 
-        ```
+        ```{python}
         # This may fail with large datasets
         df = pd.read_csv("large_dataset.csv")
         ```
 
         In Polars, the same operation is seamless:
 
-        ```
+        ```{python}
         df = pl.read_csv("large_dataset.csv")
         ```
 
         Polars also supports lazy evaluation, which allows you to optimize your workflows by deferring computations until necessary. This is particularly useful for large datasets:
 
-        ```
+        ```{python}
         df = pl.scan_csv("large_dataset.csv")  # Lazy DataFrame
         result = df.filter(pl.col("A") > 1).groupby("A").agg(pl.sum("B")).collect()  # Execute
         ```
 
-        ## (c) Compatibility with Other Machine Learning Libraries
+        ## (e) Compatibility with Other ML Libraries
 
         Polars integrates seamlessly with popular machine learning libraries like Scikit-learn, PyTorch, and TensorFlow. Its ability to handle large datasets efficiently makes it an excellent choice for preprocessing data before feeding it into ML models.
 
         **Example: Preprocessing Data for Scikit-learn**
 
-        ```
+        ```{python}
         import polars as pl
         from sklearn.linear_model import LinearRegression
 
@@ -138,7 +158,7 @@ def _(mo):
 
         Polars also supports conversion to other formats like NumPy arrays and Pandas DataFrames, ensuring compatibility with virtually any ML library:
 
-        ```
+        ```{python}
         # Convert to Pandas DataFrame
         pandas_df = df.to_pandas()
 
@@ -146,67 +166,25 @@ def _(mo):
         numpy_array = df.to_numpy()
         ```
 
-        **(d) Additional Advantages of Polars**
-
-        - Rich Functionality: Polars supports advanced operations like window functions, joins, and nested data types, making it a versatile tool for data manipulation.
-
-        - Query Optimization: Polars is significantly faster than Pandas due to its parallelized and vectorized operations. Benchmarks often show Polars outperforming Pandas by 10x or more.
+        ## (f) Rich Functionality
 
-        - Memory Efficiency: Polars uses memory more efficiently, reducing the risk of out-of-memory errors.
-
-        - Lazy API: The lazy evaluation API allows for query optimization and deferred execution, which is particularly useful for complex workflows.
+        Polars supports advanced operations like **date handling**, **window functions**, **joins**, and **nested data types**, making it a versatile tool for data manipulation.
         """
     )
     return
 
 
 @app.cell
-def _():
-    import pandas as pd
-
-    df = pd.DataFrame(
-        { 
-            "Gender": ["Male", "Female", "Male", "Female", "Male", "Female", 
-                       "Male", "Female", "Male", "Female"],
-            "Age": [13, 15, 17, 19, 21, 23, 25, 27, 29, 31],
-            "Height_CM": [150, 170, 146.5, 142, 155, 165, 170.8, 130, 132.5, 162]
-        }
-    )
-
-    # query: average height of male and female after the age of 15 years
-    filtered_df = df[df["Age"] > 15]
-    result = filtered_df.groupby("Gender").mean()["Height_CM"]
-    result
-    return df, filtered_df, pd, result
-
-
-@app.cell
-def _():
-    import polars as pl
-    return (pl,)
+def _(mo):
+    mo.md(
+        """
+        # Why Not PySpark?
 
+        While **PySpark** is undoubtedly a versatile tool that has transformed the way big data is handled and processed in Python, its **complex setup process** can be intimidating, especially for beginners. In contrast, **Polars** requires minimal setup and is ready to use right out of the box, making it more accessible for users of all skill levels.
 
-@app.cell
-def _(pl):
-    df_pl = pl.DataFrame(
-        { 
-            "Gender": ["Male", "Female", "Male", "Female", "Male", "Female", 
-                       "Male", "Female", "Male", "Female"],
-            "Age": [13, 15, 17, 19, 21, 23, 25, 27, 29, 31],
-            "Height_CM": [150.0, 170.0, 146.5, 142.0, 155.0, 165.0, 170.8, 130.0, 132.5, 162.0]
-        }
+        When deciding between the two, **PySpark** is the preferred choice for processing large datasets distributed across a **multi-node cluster**. However, for computations on a **single-node machine**, **Polars** is an excellent alternative. Remarkably, Polars is capable of handling datasets that exceed the size of the available RAM, making it a powerful tool for efficient data processing even on limited hardware.
+        """
     )
-
-    # df_pl
-    # query: average height of male and female after the age of 15 years
-    result_pl = df_pl.filter(pl.col("Age") > 15).group_by("Gender").agg(pl.mean("Height_CM"))
-    result_pl
-    return df_pl, result_pl
-
-
-@app.cell
-def _(df_pl):
-    df_pl.sql("SELECT Gender, AVG(Height_CM) FROM self WHERE Age > 15 GROUP BY Gender")
     return
 
 
@@ -223,10 +201,5 @@ def _(mo):
     return
 
 
-@app.cell
-def _():
-    return
-
-
 if __name__ == "__main__":
     app.run()