Scaler and Transformer

Why Scaling?

• Avoid the features with large range have dominating importance
• Some models are sensitive to the relative scales of feature
• Faster converge
• Reduce collinearity in linear regression
• Enable contribution weight caculation in regression models
• Avoid saturation in neural network models

Scaling Sensitive Models

• KNN, K-Means, PCA, etc.

Scaling Insensitive Models

• Random Forest, LDA, Naive Bayes, etc.

Load Raw Data

import pandas as pd
import numpy as np

data = pd.read_csv('housing.csv')
data = data.drop('ocean_proximity', axis=1) # 20640*9

data.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

Normalization

$$𝑥_{𝑖}=\frac{𝑥_{𝑖}−𝑚𝑖𝑛(𝑥_{𝑖})}{𝑚𝑎𝑥(𝑥_{𝑖})−𝑚𝑖𝑛(𝑥_{𝑖})}$$

• Min-max normalization
• range from 0 to 1

Pros:

* All features will have the exact same scale
* Preserves the relationships among the original data values
* End up with smaller standard deviations, which suppresses the effect of outliers
* Responds well if the standard deviation is small and when a distribution is not Gaussian

Cons:

* Does not handle outliers very well
* Normalized data may not meet the need of some models, e.g., KDE

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler() # fit the scaler to the training data only
data_min_max = scaler.fit_transform(data) # scale can be use to scale test data

data_min_max = pd.DataFrame(data_min_max, columns = data.columns)
data_min_max.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

Standardization (Zero-mean normalization)

$$𝑥_{𝑖}=\frac{𝑥𝑖−𝑚𝑒𝑎𝑛(𝑥_{𝑖})}{𝑠𝑡𝑑(𝑥_{𝑖})}$$

• Pros

  • Does not bound values to a specific range
  • Less affected by outliers
  • Useful when the feature distribution is Normal or Gaussian

• Cons

  • Not the best Scaler if data is not normally distributed

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
data_standarization = scaler.fit_transform(data)

data_standarization = pd.DataFrame(data_standarization, columns = data.columns)
data_standarization.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

Max Abs Scaler

$$x_{i} = \frac{x_{i}}{|x_{max}|}$$

• Pros

  • Does not shift/center the data and thus does not destroy any sparsity

• Cons

  • Suffers from the presence of significant outliers

from sklearn.preprocessing import MaxAbsScaler
scaler = MaxAbsScaler()
data_abs_max = scaler.fit_transform(data)

data_abs_max = pd.DataFrame(data_abs_max, columns = data.columns)
data_abs_max.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

Robust Scaler

$$x_{i} = \frac{x_{i}-Q_{1}(x)}{Q_{1}(x)-Q_{3}(x)}$$

• $Q_{1}(x)$, value at the 1st quartile (25th quantile)
• $Q_{3}(x)$, value at the 3rd quartile (75th quantile)

• Pros
  • Not influenced by a few numbers of outliers
• Cons
  • Does not take the median into consideration and the only center of attention are on the parts where the bulk data lies

from sklearn.preprocessing import RobustScaler
scaler = RobustScaler()
data_robust = scaler.fit_transform(data)

data_robust = pd.DataFrame(data_robust, columns = data.columns)
data_robust.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

Quantile Transformer Scaler (Rank scaler)

• Transforms the features a uniform or a normal distribution

• Ranks the relationship between observations and map observations onto other distributions by the cumulative probability distribution (CDF), such as the uniform or normal distribution

• Pros

  • Spread out the most frequent values
  • Reduces the impact of (marginal) outliers
• Cons
  • May distort linear correlations

from sklearn.preprocessing import QuantileTransformer
scaler = QuantileTransformer(output_distribution='uniform') # default output distribution is uniform
data_quantile = scaler.fit_transform(data)

data_quantile = pd.DataFrame(data_quantile, columns = data.columns)
data_quantile.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

scaler = QuantileTransformer(output_distribution='normal')
data_quantile = scaler.fit_transform(data)

data_quantile = pd.DataFrame(data_quantile, columns = data.columns)
data_quantile.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

Power Transformer Scaler

• Transform data to follow the Normal Distribution via power function ($cx^{n}$)
• Box-Cox, needs the data to be positive
• Yeo-Johnson, allowed the data to be both negative and positive

from sklearn.preprocessing import PowerTransformer
scaler = PowerTransformer(method='yeo-johnson') # default method is yeo-johnson
data_power = scaler.fit_transform(data)

data_power = pd.DataFrame(data_power, columns = data.columns)
data_power.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

Unit Vector Scaler

• Divide each feature value by the norm of the values in the feature $$x_{ij} = \frac{x_{ij}}{\|x_{j}\|}$$

data_unit_vector = data.apply(lambda x: x/np.linalg.norm(x, 2))

data_unit_vector = pd.DataFrame(data_unit_vector, columns = data.columns)
data_unit_vector.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'total_bedrooms'}>,
        <AxesSubplot:title={'center':'population'}>],
       [<AxesSubplot:title={'center':'households'}>,
        <AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>]],
      dtype=object)

Unit Norm Scaler

• Divide each feature value of a record by the norm of the record $$x_{ij} = \frac{x_{ij}}{\|x_{i}\|}$$

data_temp = data.drop(['total_bedrooms'], axis=1) # not allow to have NaN values

from sklearn.preprocessing import Normalizer
scaler = Normalizer() # default norm is l2
data_unit_norm = scaler.fit_transform(data_temp)

data_unit_norm = pd.DataFrame(data_unit_norm, columns = data_temp.columns)
data_unit_norm.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'population'}>,
        <AxesSubplot:title={'center':'households'}>],
       [<AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>,
        <AxesSubplot:>]], dtype=object)

K-Bins Discretizations

• Transform the continuous feature into several bins
• n_bins, number of bins
• encode
  • onehot, one-hot encoding and return a sparse matrix
  • onehot-dense, one-hot encoding and return a dense array
  • ordinal, return the bin index
• strategy
  • uniform, all bins in each feature have identical widths
  • quantile, all bins in each feature have the same number of points
  • kmeans, values in each bin have the same nearest center of a 1D k-means cluster

data_temp = data.drop(['total_bedrooms'], axis=1) # not allow to have NaN values

from sklearn.preprocessing import KBinsDiscretizer
transformer = KBinsDiscretizer(n_bins = 5, encode = 'onehot-dense', strategy='uniform')
data_bin = transformer.fit_transform(data_temp)

data_bin = pd.DataFrame(data_bin, columns = data_temp.columns)
data_bin.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'population'}>,
        <AxesSubplot:title={'center':'households'}>],
       [<AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>,
        <AxesSubplot:>]], dtype=object)

Feature Binarization

• Values greater than the threshold map to 1, while values less than or equal to the threshold map to 0.
• With the default threshold of 0, only positive values map to 1

data_temp = data.drop(['total_bedrooms'], axis=1) # not allow to have NaN values

from sklearn.preprocessing import Binarizer
transformer = Binarizer( threshold = 20)
data_binary = transformer.fit_transform(data_temp)

data_binary = pd.DataFrame(data_binary, columns = data_temp.columns)
data_binary.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'population'}>,
        <AxesSubplot:title={'center':'households'}>],
       [<AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>,
        <AxesSubplot:>]], dtype=object)

Function Transformers

data_temp = data.drop(['total_bedrooms'], axis=1) # not allow to have NaN values

def transform_f(x):
    return x*10

from sklearn.preprocessing import FunctionTransformer
transformer = FunctionTransformer(transform_f, validate=True)
data_function = transformer.fit_transform(data_temp)

data_function = pd.DataFrame(data_function, columns = data_temp.columns)
data_function.hist(bins = 50, figsize = (20, 15))

array([[<AxesSubplot:title={'center':'longitude'}>,
        <AxesSubplot:title={'center':'latitude'}>,
        <AxesSubplot:title={'center':'housing_median_age'}>],
       [<AxesSubplot:title={'center':'total_rooms'}>,
        <AxesSubplot:title={'center':'population'}>,
        <AxesSubplot:title={'center':'households'}>],
       [<AxesSubplot:title={'center':'median_income'}>,
        <AxesSubplot:title={'center':'median_house_value'}>,
        <AxesSubplot:>]], dtype=object)

Reference

• Quora
• 5 Data Transformers to know from Scikit-Learn
• All about Feature Scaling