Creating Data Transformation Pipelines

Often during analysis we come across situations where it is required to apply multiple transformations to the data before it is ready to be consumed by any algorithm. For example we often need to scale or need to impute missing values and even apply one-hot encoding . If we try to do this one by one it will be tedious and we may forget to apply any one of the required transformations.

One of the efficient way is to create a transformation pipeline — below we will see how to create a transformation pipeline using scikit learn library

Let me illustrate a process to create a pipeline for imputing missing values and applying one hot encoding

First we need to import the imputer , encoder and column transformers from the sklearn library

from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

Let say we have a data frame ‘car_sales’ that has following attributes: Price, Odometer_Reading, Make, Color, No_of_Doors

where Price and Odometer_Reading are numrical and Make, Color, No_of_Doors are categorical

so we will need to create lists for numerical attributes and categorical attributes

num_features = ['Price', 'Odometer_Reading']
cat_features = ['Make','Color','No_of_Doors']

Now lets create the instance of the transformer pipeline

transformer = ColumnTransformer([("num",SimpleImputer(strategy = 'median'),num_features),
("one_hot", OneHotEncoder(),cat_features)])

above pipeline does 2 things one it imputes the missing values with the median values for numerical attributes and then it performs one hot-encoding for the categorical

Now we just have to fit the transformation and apply the transformation

transformed_car_sales = transformer.fit_transform(car_sales)

thats it! now your ‘transformed_car_sales’ gets all imputations and one-hot encoding , and is ready for the algorithm

In fact in a pipeline you can define as many transformation as needed like we could have also included a scaling transformation , however care should be taken that these are in correct order.