Encoder

Install Category Encoders

• pip install category_encoders
• pip install feature-engine

Load Data

import pandas as pd

training_data = pd.read_csv('train.txt', names = ["duration","protocol_type","service","flag","src_bytes","dst_bytes","land",
"wrong_fragment","urgent","hot","num_failed_logins","logged_in",
"num_compromised","root_shell","su_attempted","num_root","num_file_creations",
"num_shells","num_access_files","num_outbound_cmds","is_host_login",
"is_guest_login","count","srv_count","serror_rate", "srv_serror_rate",
"rerror_rate","srv_rerror_rate","same_srv_rate", "diff_srv_rate", "srv_diff_host_rate","dst_host_count","dst_host_srv_count","dst_host_same_srv_rate",
"dst_host_diff_srv_rate","dst_host_same_src_port_rate",
"dst_host_srv_diff_host_rate","dst_host_serror_rate","dst_host_srv_serror_rate",
"dst_host_rerror_rate","dst_host_srv_rerror_rate","attack", "last_flag"])

training_data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 125973 entries, 0 to 125972
Data columns (total 43 columns):
 #   Column                       Non-Null Count   Dtype  
---  ------                       --------------   -----  
 0   duration                     125973 non-null  int64  
 1   protocol_type                125973 non-null  object 
 2   service                      125973 non-null  object 
 3   flag                         125973 non-null  object 
 4   src_bytes                    125973 non-null  int64  
 5   dst_bytes                    125973 non-null  int64  
 6   land                         125973 non-null  int64  
 7   wrong_fragment               125973 non-null  int64  
 8   urgent                       125973 non-null  int64  
 9   hot                          125973 non-null  int64  
 10  num_failed_logins            125973 non-null  int64  
 11  logged_in                    125973 non-null  int64  
 12  num_compromised              125973 non-null  int64  
 13  root_shell                   125973 non-null  int64  
 14  su_attempted                 125973 non-null  int64  
 15  num_root                     125973 non-null  int64  
 16  num_file_creations           125973 non-null  int64  
 17  num_shells                   125973 non-null  int64  
 18  num_access_files             125973 non-null  int64  
 19  num_outbound_cmds            125973 non-null  int64  
 20  is_host_login                125973 non-null  int64  
 21  is_guest_login               125973 non-null  int64  
 22  count                        125973 non-null  int64  
 23  srv_count                    125973 non-null  int64  
 24  serror_rate                  125973 non-null  float64
 25  srv_serror_rate              125973 non-null  float64
 26  rerror_rate                  125973 non-null  float64
 27  srv_rerror_rate              125973 non-null  float64
 28  same_srv_rate                125973 non-null  float64
 29  diff_srv_rate                125973 non-null  float64
 30  srv_diff_host_rate           125973 non-null  float64
 31  dst_host_count               125973 non-null  int64  
 32  dst_host_srv_count           125973 non-null  int64  
 33  dst_host_same_srv_rate       125973 non-null  float64
 34  dst_host_diff_srv_rate       125973 non-null  float64
 35  dst_host_same_src_port_rate  125973 non-null  float64
 36  dst_host_srv_diff_host_rate  125973 non-null  float64
 37  dst_host_serror_rate         125973 non-null  float64
 38  dst_host_srv_serror_rate     125973 non-null  float64
 39  dst_host_rerror_rate         125973 non-null  float64
 40  dst_host_srv_rerror_rate     125973 non-null  float64
 41  attack                       125973 non-null  object 
 42  last_flag                    125973 non-null  int64  
dtypes: float64(15), int64(24), object(4)
memory usage: 41.3+ MB

training_data['protocol_type'].value_counts()

tcp     102689
udp      14993
icmp      8291
Name: protocol_type, dtype: int64

training_data['service'].value_counts()

http         40338
private      21853
domain_u      9043
smtp          7313
ftp_data      6860
             ...  
tftp_u           3
aol              2
http_8001        2
harvest          2
http_2784        1
Name: service, Length: 70, dtype: int64

Ordinal Encoder

• Converts each category into integer values and the encoded data represents the sequence of labels
• Input is a 2D array, output is a 2D array
• Pros
  • ease of collation, categorization and processing
• Cons
  • for categorical data where no common relationship between values exits, give the poor performance

from sklearn.preprocessing import OrdinalEncoder
encoder = OrdinalEncoder(handle_unknown = 'use_encoded_value', unknown_value = -1)
encoder.fit_transform(training_data[['service']])

import category_encoders as ce
encoder = ce.ordinal.OrdinalEncoder()
encoder.fit_transform(training_data[['service']])

Label Encoder

• Target labels with value between 0 and n_classes-1
• Functions same as ordinal encoder
• Input is a 1D array, output is a 1D array

from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder()
encoder.fit_transform(training_data[['service']])

One-hot Encoder

• Convert categories to one-hot numeric array
• Called dummy encoder if drop the first category in each feature
• Pros
  • the result is binary rather than ordinal and that everything sits in an orthogonal vector space
• Cons
  • high cardinality

from sklearn.preprocessing import OneHotEncoder
encoder = OneHotEncoder(handle_unknown='ignore')
encoder.fit_transform(training_data[['protocol_type']])

import category_encoders as ce
encoder = ce.one_hot.OneHotEncoder()
encoder.fit_transform(training_data[['protocol_type']])

Hashing Encoder

• maps each category to an integer within a pre-determined range
• Pros
  • deal with categorical features with high cardinality
• Cons
  • may hash different keys to the same integer value
  • slow

import category_encoders as ce
encoder = ce.HashingEncoder(n_components=8) # use 8 bins
encoder.fit_transform(training_data[['service']])

Count Encoder

• replace categories with their counts computed on the train set
• Pros
  • straightforward to implement
  • not expand the feature space
  • work well with tree-based algorithms
• Cons
  • can lose valuable information if there are two different categories with the same amount of observations count

import category_encoders as ce
encoder = ce.count.CountEncoder()
encoder.fit_transform(training_data[['service']])

from feature_engine.encoding import CountFrequencyEncoder
encoder = CountFrequencyEncoder(encoding_method='count')
encoder.fit_transform(training_data[['service']])

# Frequency Encoder
from feature_engine.encoding import CountFrequencyEncoder
encoder = CountFrequencyEncoder(encoding_method='frequency')
encoder.fit_transform(training_data[['service']])

Sum Encoder

• Or effect encoder, or deviation encoder
• Similar to dummy encoder
• The row containing only 0s in dummy encoding is encoded as -1 in effect encoding.
• Intercept represents the grand mean
• Pros
  • the result is binary rather than ordinal and that everything sits in an orthogonal vector space
• Cons
  • high cardinality

data=pd.DataFrame({'City':['Delhi','Mumbai','Hyderabad','Chennai','Bangalore','Delhi','Hyderabad']}) 
display(data)
encoder=ce.sum_coding.SumEncoder(cols='City',verbose=False,)
encoder.fit_transform(data)

Helmert Encoder

• Useful where levels of the categorical variable are ordered, from lowest to highest, or from smallest to largest |Level|Variable 1|Variable 2|Variable 3| |:---: |:----: |:----: |:----: | |1. Hispanic|0.75|0|0| |2. Asian|-0.25|0.666|0| |3. African American|-0.25|-0.333|0.5| |4. White|-0.25|-0.333|-0.5|
• L = 4
  • Variable 1, (L-1)/L, rest of the values are -1/L, 0.75, -0.25, -0.25, -0.25
  • Variable 2, 0 before level 2, the number of the rest levels 3, (L-1)/L, rest of the values are -1/L, 0, 0.666, -0.333, -0.333
  • Variable 3, 0 before level 3, the number of the rest levels is 2, (L-1)/L, rest of the values are -1/L, 0, 0, 0.5, -0.5
• Pros
  • useful for categorical variables that are ordered
• Cons
  • high cardinality, create L-1 variables

import category_encoders as ce
encoder = ce.helmert.HelmertEncoder()
encoder.fit_transform(training_data[['service']])

Binary Encoder

• a combination process of hash and one hot encoding
• steps
  • the categorical feature is first converted into numerical using an ordinal encoder
  • then the numbers are transformed into a binary number
  • split the binary value into different columns
• Pros
  • reduces the high cardinality of one hot encoder, create $log_{2}n$ variables
  • fewer computations and less computational errors
• Cons
  • binary number is difficult to read and write for humans

import category_encoders as ce
encoder = ce.binary.BinaryEncoder()
encoder.fit_transform(training_data[['service']])

Base N Encoder

• steps
  • the categorical feature is first converted into numerical using an ordinal encoder
  • then the numbers are transformed into a number with the base N
  • split the number into different columns
• N is the base, N = 1, one hot encoder, N = 2, binary encoder
• Pros
  • reduce the high cardinality of binary encoder
• Cons
  • base N number may be difficult to read and write for humans

import category_encoders as ce
encoder = ce.basen.BaseNEncoder(base = 5)
encoder.fit_transform(training_data[['service']])

Rare Label Encoder

• Groups rare or infrequent categories in a new category

from feature_engine.encoding import RareLabelEncoder
# tol, the minimum frequency a label should have to be considered frequent
# n_categories, the minimum number of categories a variable should have for the encoder to find frequent labels
encoder = RareLabelEncoder(n_categories = 10, tol=0.05)
encoder.fit_transform(training_data[['service']])

String Similarity Encoder

• Replaces categorical variables with a set of float variables that capture the similarity between the category names.

from feature_engine.encoding import StringSimilarityEncoder
encoder = StringSimilarityEncoder()
encoder.fit_transform(training_data[['protocol_type']])

Target Encoder

• Or impact encoder, likelihood encoder
• Replace a categorical value with the mean of the target variable in that category $$code(𝐱) = E[y|x] − E[y]$$
  • x, is a category of a feature, y, is the targets
  • targets need to be binomial or continuous
  • use the mean of the y values of a category substract the mean of the y values of entire dataset
• Pros
  • really easy and powerful
• Cons
  • by using the probability of the target to encode the features like “cheating”, may cause the overfitting

import category_encoders as ce
encoder = ce.target_encoder.TargetEncoder()
encoder.fit_transform(training_data[['service']], train_y)

Mean Encoder

• replaces categories by the target mean

from feature_engine.encoding import MeanEncoder
encoder = MeanEncoder()
encoder.fit_transform(training_data[['service']], train_y)

Quantile Encoder

• replace a categorical value with the statistical quantile instead of the mean in target MEstimate encoder

import category_encoders as ce
encoder = ce.QuantileEncoder()
encoder.fit_transform(training_data[['service']], train_y)

M-estimate Encoder

• a simplified version of target encoder, which goes under names like m-probability estimate or additive smoothing with known incidence rates

import category_encoders as ce
encoder = ce.m_estimate.MEstimateEncoder()
encoder.fit_transform(training_data[['service']], train_y)

Weight of Evidence Encoder

• A target-based encoder
• A measure of the “strength” of a grouping for separating good and bad risk

from feature_engine.encoding import WoEEncoder
encoder = WoEEncoder()
encoder.fit_transform(training_data[['service']], train_y)

Decision Tree Encoder

• Replaces categories by predictions of a decision tree

from feature_engine.encoding import DecisionTreeEncoder
encoder = DecisionTreeEncoder(cv=3)
encoder.fit_transform(training_data[['service']], train_y)

Leave One Out Encoder

• similar to target encoding, excludes the current row’s target when calculating the mean target for a level to reduce the effect of outliers

import category_encoders as ce
encoder = ce.leave_one_out.LeaveOneOutEncoder()
encoder.fit_transform(training_data[['service']], train_y)

CatBoost Encoder

• similar to target encoding, involves an ordering principle in order to overcome this problem of target leakage

import category_encoders as ce
encoder = ce.cat_boost.CatBoostEncoder()
encoder.fit_transform(training_data[['service']], train_y)

James-Stein Encoder

import category_encoders as ce
encoder = ce.james_stein.JamesSteinEncoder()
encoder.fit_transform(training_data[['service']], train_y)

GPS Coordinates Transforming

• Categories
  • Turn each record into a category or categories. Categories could be the country, city, point of interest (POI) or a different geo-spatial layer.
• Clusters
  • Cluster the points and then assign the cluster ID as a categorical feature
• Distances
  • Distance from one point to the next or distance to a fixed point (e.g., origin)

IP

• IP is not an indicator of an attack.
• The relationship between IP addresses might indicate some types of attacks

# 1. Convert IP to a binary number
# Convert each of the integer of an IP to an 8-bit integer, then merge them together to be a 32-bit integer
# Maintains this relationship between IP addresses, as the system would be able to learn that IP addresses that 
# are close to each other can be within the same network
ip = '192.168.0.1'
parts = ip.split('.')

ip_1 = int(parts[0])<<24
ip_2 = int(parts[1])<<16
ip_3 = int(parts[2])<<8
ip_4 = int(parts[3])

ip_converted = ip_1+ip_2+ip_3+ip_4
ip_converted

# 2. Convert IP to an integer by removing dots
# '192.168.0.1' to 192168000001
# Maintains the relationship between IP addresses and helps the recognition of adjacent nodes and networks
ip = '192.168.0.1'
parts = ip.split('.')
ip_converted = '{:03d}'.format(int(parts[0]))+'{:03d}'.format(int(parts[1]))+\
                '{:03d}'.format(int(parts[2]))+'{:03d}'.format(int(parts[3]))
ip_converted = int(ip_converted)
ip_converted

# 3. One-hot encoding
# The relationship is not maintained between IP addresses
# Be helpful in recognizing data flows, and doing statistics analysis of data originating or 
# landing in a specific node

# 4. One-hot-encoding the top frequent IPs

Port

• Well-known ports, the well known ports are those from 0 - 1,023
• Registered ports, the registered ports are those from 1,024 - 49,151
• Dynamic and/or private ports, the dynamic and/or private ports are those from 49,152 - 65,535

Reference

• Comparison of Different Encoding Methods using TIBCO Data Science
• Documentation of feature-engine
• Here’s All you Need to Know About Encoding Categorical Data (with Python code)
• A Complete Guide to Categorical Data Encoding
• 11 Categorical Encoders and Benchmark
• Target-encoding Categorical Variables
• Quantile Encoder
• Dealing with categorical features with high cardinality: Feature Hashing
• Documentation of Category Encoders