Preprocessing ¶
In [5]:
import pandas as pd
data = pd.read_csv('housing.csv')
data.info()
Data Cleaning¶
- Get rid of the corresponding districts
- Get rid of the whole attribute
- Set the values to some value (zero, the mean, the median, etc.)
In [ ]:
median = data['total_bedrooms'].median()
data['total_bedrooms'].fillna(median, inplace=True)
In [69]:
import numpy as np
from sklearn.impute import SimpleImputer
data_num = data.drop('ocean_proximity', axis=1)
imp = SimpleImputer(missing_values=np.nan, strategy='mean')
imp.fit(data_num)
X = imp.transform(data_num)
data_tr = pd.DataFrame(X, columns=data_num.columns)
data_tr['ocean_proximity'] = data['ocean_proximity']
data_tr.info()
Handling Text and Categorical Attributes¶
In [70]:
data_tr.head()
Out[70]:
In [66]:
data_tr['ocean_proximity'].value_counts()
Out[66]:
In [101]:
from sklearn.preprocessing import LabelBinarizer
encoder = LabelBinarizer();
data_cat_binary = encoder.fit_transform(data_tr['ocean_proximity']) # numpy array
data_cat_binary
Out[101]:
In [100]:
data_cat = pd.DataFrame(data_cat_binary, columns=encoder.classes_, index = data_tr.index)
data_cat = data_tr.join(data_cat)
data_cat = data_cat.drop('ocean_proximity', axis=1)
data_cat.head()
Out[100]:
Add More Features¶
In [105]:
def add_features(data):
data_copy = data.copy()
data_copy["rooms_per_household"] = data_copy["total_rooms"]/data_copy["households"]
data_copy["bedrooms_per_room"] = data_copy["total_bedrooms"]/data_copy["total_rooms"]
data_copy["population_per_household"]=data_copy["population"]/data_copy["households"]
return data_copy
data_add = add_features(data_cat)
data_add.head()
Out[105]:
Feature Scaling¶
- zero-mean normalization: $xi=(xi−mean(xi))/std(xi)$
- So called standardization
- Does not bound values to a specific range
- Less affected by outliers
- Min-max normalization: $xi=(xi−min(xi)/(max(xi)−min(xi))$
- So called normalization
- recommendation
- range from 0 to 1
In [116]:
data_scale = data_add.copy()
In [113]:
# min-max scaling
from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler() # fit the scaler to the training data only
scaler.fit(data_scale) # scale can be use to scale test data
data_scale = scaler.transform(data_scale)
data_scale.shape
Out[113]:
In [118]:
# standardization
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(data_scale)
data_scale = scaler.transform(data_scale)
data_scale.shape
Out[118]:
Transform Non-Gaussion Distribution to Bell-Shaped Distribution¶
In [124]:
# check feature distribution
data_dist = pd.DataFrame(data_scale, columns=data_add.columns, index=data_add.index)
data_dist.hist(bins = 50, figsize = (20, 15))
Out[124]:
In [138]:
# tranform tail heavy distribution to bell-shaped distribution
total_bedrooms = pd.DataFrame(data_dist['total_bedrooms'])
from sklearn.preprocessing import PowerTransformer
pt = PowerTransformer()
pt.fit(total_bedrooms)
bell = pt.transform(total_bedrooms)
In [135]:
import matplotlib.pyplot as plt
plt.hist(bell)
Out[135]:
Polymonial Features¶
In [144]:
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(2)
features = poly.fit_transform(X) # numpy array
features.shape
Out[144]:
Feature Selection¶
- Univariate Feature Selection
- L1 regularization