Exploring data with pandas, numpy and pyplot, preprocess data through convertion of non-numerical data to numerical data, make predictions with appropriate supervised learning algorithms such as Naive Bayes, Logistic Regression and Support Vector Machines, and calculate and compare F1 scores.
Building a Student Intervention System using Machine Learning

Project 2: Building a Student Intervention System

Supervised Learning

Machine Learning Engineer Nanodegree

This notebook contains extensive answers and tips that go beyond what was taught and what is required. But the extra parts are very useful for your future projects. Feel free to fork my repository on Github here.

Question 1 - Classification vs. Regression

Your goal for this project is to identify students who might need early intervention before they fail to graduate. Which type of supervised learning problem is this, classification or regression? Why?

Answer:

  • This should be a classification problem.
  • This is because there possibly two discrete outcomes, typical of a classification problem:
    1. Students who need early intervention.
    2. Students who do not need early intervention.
  • We can classify accordingly with a binary outcome such as:
    1. Yes, 1, for students who need early intervention.
    2. No, 0, for students who do not need early intervention.
  • Evidently, we are not trying to predict a continuous outcome, hence this is not a regression problem.

Exploring the Data

Run the code cell below to load necessary Python libraries and load the student data. Note that the last column from this dataset, 'passed', will be our target label (whether the student graduated or didn't graduate). All other columns are features about each student.

In [1]:
# Import libraries
import numpy as np
import pandas as pd
from time import time
from sklearn.metrics import f1_score

# Read student data
student_data = pd.read_csv("student-data.csv")
print "Student data read successfully!"
Student data read successfully!
In [2]:
# Further Exploration using .head()
student_data.head()
Out[2]:
school sex age address famsize Pstatus Medu Fedu Mjob Fjob ... internet romantic famrel freetime goout Dalc Walc health absences passed
0 GP F 18 U GT3 A 4 4 at_home teacher ... no no 4 3 4 1 1 3 6 no
1 GP F 17 U GT3 T 1 1 at_home other ... yes no 5 3 3 1 1 3 4 no
2 GP F 15 U LE3 T 1 1 at_home other ... yes no 4 3 2 2 3 3 10 yes
3 GP F 15 U GT3 T 4 2 health services ... yes yes 3 2 2 1 1 5 2 yes
4 GP F 16 U GT3 T 3 3 other other ... no no 4 3 2 1 2 5 4 yes

5 rows × 31 columns

In [3]:
# This is a 395 x 31 DataFrame
student_data.shape
Out[3]:
(395, 31)
In [4]:
# Type of data is a pandas DataFrame
# Hence I can use pandas DataFrame methods
type(student_data)
Out[4]:
pandas.core.frame.DataFrame

Implementation: Data Exploration

Let's begin by investigating the dataset to determine how many students we have information on, and learn about the graduation rate among these students. In the code cell below, you will need to compute the following:

  • The total number of students, n_students.
  • The total number of features for each student, n_features.
  • The number of those students who passed, n_passed.
  • The number of those students who failed, n_failed.
  • The graduation rate of the class, grad_rate, in percent (%).
In [39]:
# TODO: Calculate number of students
n_students = student_data.shape[0]

# TODO: Calculate number of features
n_features = student_data.shape[1] - 1

# TODO: Calculate passing students
# Data filtering using .loc[rows, columns]
passed = student_data.loc[student_data.passed == 'yes', 'passed']
n_passed = passed.shape[0]

# TODO: Calculate failing students
failed = student_data.loc[student_data.passed == 'no', 'passed']
n_failed = failed.shape[0]

# TODO: Calculate graduation rate
total = float(n_passed + n_failed)
grad_rate = float(n_passed * 100 / total)

# Print the results
print "Total number of students: {}".format(n_students)
print "Number of features: {}".format(n_features)
print "Number of students who passed: {}".format(n_passed)
print "Number of students who failed: {}".format(n_failed)
print "Graduation rate of the class: {:.2f}%".format(grad_rate)
Total number of students: 395
Number of features: 30
Number of students who passed: 265
Number of students who failed: 130
Graduation rate of the class: 67.09%

Preparing the Data

In this section, we will prepare the data for modeling, training and testing.

Identify feature and target columns

It is often the case that the data you obtain contains non-numeric features. This can be a problem, as most machine learning algorithms expect numeric data to perform computations with.

Run the code cell below to separate the student data into feature and target columns to see if any features are non-numeric.

In [6]:
# Columns
student_data.columns
Out[6]:
Index([u'school', u'sex', u'age', u'address', u'famsize', u'Pstatus', u'Medu', u'Fedu', u'Mjob', u'Fjob', u'reason', u'guardian', u'traveltime', u'studytime', u'failures', u'schoolsup', u'famsup', u'paid', u'activities', u'nursery', u'higher', u'internet', u'romantic', u'famrel', u'freetime', u'goout', u'Dalc', u'Walc', u'health', u'absences', u'passed'], dtype='object')
In [7]:
# We want to get the column name "passed" which is the last 
student_data.columns[-1]
Out[7]:
'passed'
In [8]:
# This would get everything except for the last element that is "passed"
student_data.columns[:-1]
Out[8]:
Index([u'school', u'sex', u'age', u'address', u'famsize', u'Pstatus', u'Medu', u'Fedu', u'Mjob', u'Fjob', u'reason', u'guardian', u'traveltime', u'studytime', u'failures', u'schoolsup', u'famsup', u'paid', u'activities', u'nursery', u'higher', u'internet', u'romantic', u'famrel', u'freetime', u'goout', u'Dalc', u'Walc', u'health', u'absences'], dtype='object')
In [9]:
# Extract feature columns
# As seen above, we're getting all the columns except "passed" here but we're converting it to a list
feature_cols = list(student_data.columns[:-1])

# Extract target column 'passed'
# As seen above, since "passed" is last in the list, we're extracting using [-1]
target_col = student_data.columns[-1]

# Show the list of columns
print "Feature columns:\n{}".format(feature_cols)
print "\nTarget column: {}".format(target_col)

# Separate the data into feature data and target data (X_all and y_all, respectively)
X_all = student_data[feature_cols]
y_all = student_data[target_col]

# Show the feature information by printing the first five rows
print "\nFeature values:"
print X_all.head()
Feature columns:
['school', 'sex', 'age', 'address', 'famsize', 'Pstatus', 'Medu', 'Fedu', 'Mjob', 'Fjob', 'reason', 'guardian', 'traveltime', 'studytime', 'failures', 'schoolsup', 'famsup', 'paid', 'activities', 'nursery', 'higher', 'internet', 'romantic', 'famrel', 'freetime', 'goout', 'Dalc', 'Walc', 'health', 'absences']

Target column: passed

Feature values:
  school sex  age address famsize Pstatus  Medu  Fedu     Mjob      Fjob  \
0     GP   F   18       U     GT3       A     4     4  at_home   teacher
1     GP   F   17       U     GT3       T     1     1  at_home     other
2     GP   F   15       U     LE3       T     1     1  at_home     other
3     GP   F   15       U     GT3       T     4     2   health  services
4     GP   F   16       U     GT3       T     3     3    other     other

    ...    higher internet  romantic  famrel  freetime goout Dalc Walc health  \
0   ...       yes       no        no       4         3     4    1    1      3
1   ...       yes      yes        no       5         3     3    1    1      3
2   ...       yes      yes        no       4         3     2    2    3      3
3   ...       yes      yes       yes       3         2     2    1    1      5
4   ...       yes       no        no       4         3     2    1    2      5

  absences
0        6
1        4
2       10
3        2
4        4

[5 rows x 30 columns]

Preprocess Feature Columns

As you can see, there are several non-numeric columns that need to be converted! Many of them are simply yes/no, e.g. internet. These can be reasonably converted into 1/0 (binary) values.

Other columns, like Mjob and Fjob, have more than two values, and are known as categorical variables. The recommended way to handle such a column is to create as many columns as possible values (e.g. Fjob_teacher, Fjob_other, Fjob_services, etc.), and assign a 1 to one of them and 0 to all others.

These generated columns are sometimes called dummy variables, and we will use the pandas.get_dummies() function to perform this transformation. Run the code cell below to perform the preprocessing routine discussed in this section.

In [10]:
def preprocess_features(X):
    ''' Preprocesses the student data and converts non-numeric binary variables into
        binary (0/1) variables. Converts categorical variables into dummy variables. '''

    # Initialize new output DataFrame
    output = pd.DataFrame(index = X.index)

    # Investigate each feature column for the data
    for col, col_data in X.iteritems():

        # If data type is non-numeric, replace all yes/no values with 1/0
        if col_data.dtype == object:
            col_data = col_data.replace(['yes', 'no'], [1, 0])

        # If data type is categorical, convert to dummy variables
        if col_data.dtype == object:
            # Example: 'school' => 'school_GP' and 'school_MS'
            col_data = pd.get_dummies(col_data, prefix = col)

        # Collect the revised columns
        output = output.join(col_data)

    return output

X_all = preprocess_features(X_all)
print "Processed feature columns ({} total features):\n{}".format(len(X_all.columns), list(X_all.columns))
Processed feature columns (48 total features):
['school_GP', 'school_MS', 'sex_F', 'sex_M', 'age', 'address_R', 'address_U', 'famsize_GT3', 'famsize_LE3', 'Pstatus_A', 'Pstatus_T', 'Medu', 'Fedu', 'Mjob_at_home', 'Mjob_health', 'Mjob_other', 'Mjob_services', 'Mjob_teacher', 'Fjob_at_home', 'Fjob_health', 'Fjob_other', 'Fjob_services', 'Fjob_teacher', 'reason_course', 'reason_home', 'reason_other', 'reason_reputation', 'guardian_father', 'guardian_mother', 'guardian_other', 'traveltime', 'studytime', 'failures', 'schoolsup', 'famsup', 'paid', 'activities', 'nursery', 'higher', 'internet', 'romantic', 'famrel', 'freetime', 'goout', 'Dalc', 'Walc', 'health', 'absences']

Implementation: Training and Testing Data Split

So far, we have converted all categorical features into numeric values. For the next step, we split the data (both features and corresponding labels) into training and test sets. In the following code cell below, you will need to implement the following:

  • Randomly shuffle and split the data (X_all, y_all) into training and testing subsets.
    • Use 300 training points (approximately 75%) and 95 testing points (approximately 25%).
    • Set a random_state for the function(s) you use, if provided.
    • Store the results in X_train, X_test, y_train, and y_test.

Pro Tip: Data assessment's impact on train/test split

  • When dealing with the new data set it is good practice to assess its specific characteristics and implement the cross validation technique tailored on those very characteristics, in our case there are two main elements:
    • Our dataset is small.
    • Our dataset is slightly unbalanced. (There are more passing students than on passing students)

What can we do?

In [11]:
# TODO: Import any additional functionality you may need here
from sklearn.cross_validation import train_test_split
In [29]:
# For initial train/test split, we can obtain stratification by simply using stratify = y_all:
X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, stratify = y_all, test_size=95, random_state=42)

# Show the results of the split
print "Training set has {} samples.".format(X_train.shape[0])
print "Testing set has {} samples.".format(X_test.shape[0])
Training set has 300 samples.
Testing set has 95 samples.
In [40]:
# To double check stratification
print np.mean(y_train == 'no')
print np.mean(y_test == 'no')
0.33
0.326315789474

Training and Evaluating Models

In this section, you will choose 3 supervised learning models that are appropriate for this problem and available in scikit-learn. You will first discuss the reasoning behind choosing these three models by considering what you know about the data and each model's strengths and weaknesses. You will then fit the model to varying sizes of training data (100 data points, 200 data points, and 300 data points) and measure the F1 score. You will need to produce three tables (one for each model) that shows the training set size, training time, prediction time, F1 score on the training set, and F1 score on the testing set.

The following supervised learning models are currently available in scikit-learn that you may choose from:

  • Gaussian Naive Bayes (GaussianNB)
  • Decision Trees
  • Ensemble Methods (Bagging, AdaBoost, Random Forest, Gradient Boosting)
  • K-Nearest Neighbors (KNeighbors)
  • Stochastic Gradient Descent (SGDC)
  • Support Vector Machines (SVM)
  • Logistic Regression

Question 2 - Model Application

List three supervised learning models that are appropriate for this problem. For each model chosen

  • Describe one real-world application in industry where the model can be applied. (You may need to do a small bit of research for this — give references!)
  • What are the strengths of the model; when does it perform well?
  • What are the weaknesses of the model; when does it perform poorly?
  • What makes this model a good candidate for the problem, given what you know about the data?

How do we choose algorithms?

Answer:

  • We will be covering 3 supervised learning models.
    1. Naive Bayes
    2. Logistic Regression
    3. Support Vector Machines
  • But before we move on to cover the 3 supervised learning models, we will be discussing about the data itself because it is an important discussion to determine if the model makes a good candidate for the problem at hand.

DATA OVERVIEW

1. Skewed classes:

  • As we can see, there is almost twice as many students who passed compared to students who failed.
    • Number of students who passed: 265 (majority class)
    • Number of students who failed: 130 (minority class)
  • This would pose problems when we are splitting the data.
    • The training test could be populated with mostly the majority class and the testing set could be populated with the minority class. This would affect the accuracy calculated.
    • Hence, there should be emphasis on how we split the data and which metric to choose.

2. Lack of data:

  • There is a lack of examples in the dataset.
    • 395 students
  • This would have implications on some algorithms that require more data.
    • Generally we want more data, except when we are facing a high bias problem.
    • In this case, we should keep to simpler algorithms.

3. Too many features:

  • For such a small dataset with 395 students, we have a staggeringly high number of features.
    • 48 features
  • Curse of Dimensionality
    • For each additional feature we add, we need to increase the number of examples we have exponentially due to the curse of dimensionality.

EXPLANATION OF MODELS

1. Naive Bayes

  • Industry usage (Barbosa et. al 2014)
    • Classifying eggs
    • There is an interesting use of naive bayes (NB) as a learning algorithm to classify eggs into 2 groups:
      1. Free-range eggs
        • These are eggs from hens who are able to roam freely
      2. Battery eggs
        • These are eggs from hens who are kept in a small cage. Some may call this "unethical eggs"
    • The study revealed that NB provided a high accuracy of 90% when classifying between the 2 groups of eggs.
  • Strengths
    1. Converges quicker than discriminative models like Logistic Regression hence less data is required
  • Weaknesses
    1. Requires observations to be independent of one another
      • But in practice, the classifier performs quite well even when the independence assumption is violated
    2. Simple representation without opportunities for hyperparameter tuning
  • Suitability for the problem
    • This algorithm performs well for this problem because the data has the following properties:
      • Low number of observations
        • Naive bayes performs well on small datasets

2. Logistic Regression

  • Industry usage (Penderson et. al 2014)
    • Classification of protein sequences
      • Identify and automatically categorize protein sequences into one of 11 pre-defined classes
      • Tremendous potential for further bioinformatics applications using Logistic Regression
  • Strengths
    1. Many ways to regularize the model to tolerate some errors and avoid over-fitting
    2. Unlike Naive Bayes, we do not have to worry about correlated features
    3. Unlike Support Vector Machines, we can easily take in new data using an online gradient descent method
  • Weaknesses
    1. Requires observations to be independent of one another
    2. It aims to predict based on independent variables, if there are not properly identified, Logistic Regression provides little predictive value
  • Suitability for the problem
    • Many features may be correlated
      • And Logistic Regression, unlike Naive Bayes, can deal with this problem
      • Regularization to prevent overfitting due to dataset having many features

3. Support Vector Machines (SVMs)

  • Industry usage (Di Pillo et. al 2016):
    • Sales forecasting when running promotions
      • Originally, statistical methods like ARIMA and smoothing methods are used like Exponential Smoothing
      • But they could fail if high irregularity of sales are present
      • Hence SVMs provide a good alternative
  • Strengths
    1. SVM have regularization parameters to tolerate some errors and avoid over-fitting
    2. Kernel trick: Users can build in expert knowledge about the problem via engineering the kernel
    3. Provides a good out-of-sample generalization, if the parameters C and gamma are appropriate chosen
      • In other words, SVM might be more robust even when the training sample has some bias
  • Weaknesses
    1. Bad interpretability: SVMs are black boxes
    2. High computational cost: SVMs scale exponentially in training time
    3. Users might need to have certain domain knowledge to use kernel function
  • Suitability for the problem
    • Many features may be correlated
      • Regularization to prevent overfitting due to dataset having many features

Works Cited

  1. Barbosa, R. M., Nacano, L. R., Freitas, R., Batista, B. L. and Barbosa, F. (2014), The Use of Decision Trees and Naïve Bayes Algorithms and Trace Element Patterns for Controlling the Authenticity of Free-Range-Pastured Hens’ Eggs. Journal of Food Science, 79: C1672–C1677. http://dx.doi.org/10.1111/1750-3841.12577.
  2. Pedersen, B. P., Ifrim, G., Liboriussen, P., Axelsen, K. B., Palmgren, M. G., Nissen, P., . . . Pedersen, C. N. S. (2014). Large scale identification and categorization of protein sequences using structured logistic regression. PloS One, 9(1), 1. http://dx.doi.org/10.1371/journal.pone.0085139
  3. Di Pillo, G., Latorre, V., Lucidi, S. et al. 4OR-Q J Oper Res (2016) 14: 309. http://dx.doi.org/10.1007/s10288-016-0316-0

Setup

Run the code cell below to initialize three helper functions which you can use for training and testing the three supervised learning models you've chosen above. The functions are as follows:

  • train_classifier - takes as input a classifier and training data and fits the classifier to the data.
  • predict_labels - takes as input a fit classifier, features, and a target labeling and makes predictions using the F1 score.
  • train_predict - takes as input a classifier, and the training and testing data, and performs train_clasifier and predict_labels.
    • This function will report the F1 score for both the training and testing data separately.
In [13]:
def train_classifier(clf, X_train, y_train):
    ''' Fits a classifier to the training data. '''

    # Start the clock, train the classifier, then stop the clock
    start = time()
    clf.fit(X_train, y_train)
    end = time()

    # Print the results
    print "Trained model in {:.4f} seconds".format(end - start)


def predict_labels(clf, features, target):
    ''' Makes predictions using a fit classifier based on F1 score. '''

    # Start the clock, make predictions, then stop the clock
    start = time()
    y_pred = clf.predict(features)
    end = time()

    # Print and return results
    print "Made predictions in {:.4f} seconds.".format(end - start)
    return f1_score(target.values, y_pred, pos_label='yes')


def train_predict(clf, X_train, y_train, X_test, y_test):
    ''' Train and predict using a classifer based on F1 score. '''

    # Indicate the classifier and the training set size
    print ""
    print "Training a {} using a training set size of {}. . .".format(clf.__class__.__name__, len(X_train))

    # Train the classifier
    train_classifier(clf, X_train, y_train)

    # Print the results of prediction for both training and testing
    print "F1 score for training set: {:.4f}.".format(predict_labels(clf, X_train, y_train))
    print "F1 score for test set: {:.4f}.".format(predict_labels(clf, X_test, y_test))

Implementation: Model Performance Metrics

With the predefined functions above, you will now import the three supervised learning models of your choice and run the train_predict function for each one. Remember that you will need to train and predict on each classifier for three different training set sizes: 100, 200, and 300. Hence, you should expect to have 9 different outputs below — 3 for each model using the varying training set sizes. In the following code cell, you will need to implement the following:

  • Import the three supervised learning models you've discussed in the previous section.
  • Initialize the three models and store them in clf_A, clf_B, and clf_C.
    • Use a random_state for each model you use, if provided.
    • Note: Use the default settings for each model — you will tune one specific model in a later section.
  • Create the different training set sizes to be used to train each model.
    • Do not reshuffle and resplit the data! The new training points should be drawn from X_train and y_train.
  • Fit each model with each training set size and make predictions on the test set (9 in total).
    Note: Three tables are provided after the following code cell which can be used to store your results.
In [35]:
# TODO: Import the three supervised learning models from sklearn
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC

# TODO: Initialize the three models
clf_A = GaussianNB()
clf_B = LogisticRegression(random_state=42)
clf_C = SVC(random_state=42)

# TODO: Set up the training set sizes
X_train_100 = X_train.iloc[:100, :]
y_train_100 = y_train.iloc[:100]

X_train_200 = X_train.iloc[:200, :]
y_train_200 = y_train.iloc[:200]

X_train_300 = X_train.iloc[:300, :]
y_train_300 = y_train.iloc[:300]

# TODO: Execute the 'train_predict' function for each classifier and each training set size
# train_predict(clf, X_train, y_train, X_test, y_test)

for clf in [clf_A, clf_B, clf_C]:
    print "\n{}: \n".format(clf.__class__.__name__)
    for n in [100, 200, 300]:
        train_predict(clf, X_train[:n], y_train[:n], X_test, y_test)
GaussianNB:


Training a GaussianNB using a training set size of 100. . .
Trained model in 0.0012 seconds
Made predictions in 0.0003 seconds.
F1 score for training set: 0.7752.
Made predictions in 0.0003 seconds.
F1 score for test set: 0.6457.

Training a GaussianNB using a training set size of 200. . .
Trained model in 0.0026 seconds
Made predictions in 0.0004 seconds.
F1 score for training set: 0.8060.
Made predictions in 0.0002 seconds.
F1 score for test set: 0.7218.

Training a GaussianNB using a training set size of 300. . .
Trained model in 0.0008 seconds
Made predictions in 0.0004 seconds.
F1 score for training set: 0.8134.
Made predictions in 0.0003 seconds.
F1 score for test set: 0.7761.

LogisticRegression:


Training a LogisticRegression using a training set size of 100. . .
Trained model in 0.0020 seconds
Made predictions in 0.0002 seconds.
F1 score for training set: 0.8671.
Made predictions in 0.0001 seconds.
F1 score for test set: 0.7068.

Training a LogisticRegression using a training set size of 200. . .
Trained model in 0.0018 seconds
Made predictions in 0.0002 seconds.
F1 score for training set: 0.8211.
Made predictions in 0.0001 seconds.
F1 score for test set: 0.7391.

Training a LogisticRegression using a training set size of 300. . .
Trained model in 0.0034 seconds
Made predictions in 0.0003 seconds.
F1 score for training set: 0.8512.
Made predictions in 0.0002 seconds.
F1 score for test set: 0.7500.

SVC:


Training a SVC using a training set size of 100. . .
Trained model in 0.0011 seconds
Made predictions in 0.0008 seconds.
F1 score for training set: 0.8354.
Made predictions in 0.0007 seconds.
F1 score for test set: 0.8025.

Training a SVC using a training set size of 200. . .
Trained model in 0.0043 seconds
Made predictions in 0.0021 seconds.
F1 score for training set: 0.8431.
Made predictions in 0.0011 seconds.
F1 score for test set: 0.8105.

Training a SVC using a training set size of 300. . .
Trained model in 0.0066 seconds
Made predictions in 0.0066 seconds.
F1 score for training set: 0.8664.
Made predictions in 0.0019 seconds.
F1 score for test set: 0.8052.

Tabular Results

Edit the cell below to see how a table can be designed in Markdown. You can record your results from above in the tables provided.

Classifer 1 - Naive Bayes

Training Set Size Training Time Prediction Time (test) F1 Score (train) F1 Score (test)
100 0.0012 0.0003 0.7752 0.6457
200 0.0026 0.0002 0.8060 0.7218
300 0.0008 0.0003 0.8134 0.7761

Classifer 2 - Logistic Regression

Training Set Size Training Time Prediction Time (test) F1 Score (train) F1 Score (test)
100 0.0020 0.0001 0.8671 0.7068
200 0.0018 0.0001 0.8211 0.7391
300 0.0034 0.0002 0.8512 0.7500

Classifer 3 - Support Vector Machines

Training Set Size Training Time Prediction Time (test) F1 Score (train) F1 Score (test)
100 0.0011 0.0007 0.8366 0.8025
200 0.0043 0.0011 0.8552 0.8105
300 0.0066 0.0019 0.8615 0.8052

Choosing the Best Model

In this final section, you will choose from the three supervised learning models the best model to use on the student data. You will then perform a grid search optimization for the model over the entire training set (X_train and y_train) by tuning at least one parameter to improve upon the untuned model's F1 score.

Question 3 - Choosing the Best Model

Based on the experiments you performed earlier, in one to two paragraphs, explain to the board of supervisors what single model you chose as the best model. Which model is generally the most appropriate based on the available data, limited resources, cost, and performance?

Answer:

The predictive performance of SVMs is slightly better than Naive Bayes. However, it is important to note how SVMs' computational time would grow much faster than Naive Bayes with more data, and our costs would increase exponentially when we have more students. On the other hand, Naive Bayes' computational time would grow linearly with more data, and our cost would not rise as fast. Hence, Naive Bayes offers a good alternative to SVMs taking into account its performance on a small dataset and on a potentially large and growing dataset.

Consequently, we compare Naive Bayes and Logistic Regression. Although the results show Logistic Regression is slightly worst than Naive Bayes in terms of it predictive performance, slight tuning of Logistic Regression's model would easily yield much better predictive performance compare to Naive Bayes. This is in contrast to Naive Bayes where we do not have the opportunity to tune model. Hence, we should go with Logistic Regression.

Big O Notation for the 3 Algorithms

  1. Naive Bayes:
    • O(n)
  2. Logistic Regression:
    • O(C^n)
  3. Support Vector Machines:
    • O(n^3) with sigmoid kernel
    • O(n^2) with space complexities

Question 4 - Model in Layman's Terms

In one to two paragraphs, explain to the board of directors in layman's terms how the final model chosen is supposed to work. Be sure that you are describing the major qualities of the model, such as how the model is trained and how the model makes a prediction. Avoid using advanced mathematical or technical jargon, such as describing equations or discussing the algorithm implementation.

Answer:
First, the model learns how a student's performance indicators lead to whether a student will graduate or otherwise. The model can do this because we already have existing data on students who have and have not graduated, so our model can learn the performance indicators of those students. Based on the student's performance indicators, the model would output a weight for each performance indicator.

With this in mind, the second step is predicting whether new students would graduate or otherwise. This time we do not have information on existing students whether they have graduated or not as they are still studying. However, we have a model that learned from previous batches of students who graduated. The new students' performance indicators with their respective weights will be fed into our model and the model will output a probability and students will be classified according to whether they are "likely to graduate" or "unlikely to graduate". We can then take preventive measures on students who are unlikely to graduate.

Moreover, if we would like to play it safe and ensure that we spot as many students as we can who are "unlikely to graduate", even if they may be "likely to graduate", we can increase our strictness in determining their likelihood of graduating, and spot more of them. This is because there is no harm in flagging a student who is "likely to graduate" as "unlikely to graduate". But there may be severe repercussions if we flag a student who is "unlikely to graduate" as "likely to graduate" and do not attend to the student.

Implementation: Model Tuning (Logistic Regression)

Fine tune the chosen model. Use grid search (GridSearchCV) with at least one important parameter tuned with at least 3 different values. You will need to use the entire training set for this. In the code cell below, you will need to implement the following:

  • Import sklearn.grid_search.gridSearchCV and sklearn.metrics.make_scorer.
  • Create a dictionary of parameters you wish to tune for the chosen model.
    • Example: parameters = {'parameter' : [list of values]}.
  • Initialize the classifier you've chosen and store it in clf.
  • Create the F1 scoring function using make_scorer and store it in f1_scorer.
    • Set the pos_label parameter to the correct value!
  • Perform grid search on the classifier clf using f1_scorer as the scoring method, and store it in grid_obj.
  • Fit the grid search object to the training data (X_train, y_train), and store it in grid_obj.

Pro Tip:

In [36]:
# TODO: Import 'GridSearchCV' and 'make_scorer'
from sklearn.grid_search import GridSearchCV
from sklearn.metrics import make_scorer
from sklearn.cross_validation import StratifiedShuffleSplit
In [50]:
# Create the parameters list you wish to tune
C = [0.001, 0.01, 0.1, 1, 10, 100, 1000]
solver = ['sag']
max_iter = [1000]
param_grid = dict(C=C, solver=solver, max_iter=max_iter)

# Initialize the classifier
clf = LogisticRegression(random_state=42)

# Make an f1 scoring function using 'make_scorer' 
f1_scorer = make_scorer(f1_score, pos_label='yes')

# Stratified Shuffle Split
ssscv = StratifiedShuffleSplit(y_train, n_iter=10, test_size=0.1)

# TODO: Perform grid search on the classifier using the f1_scorer as the scoring method
grid_obj = GridSearchCV(clf, param_grid, cv=ssscv, scoring=f1_scorer)

# TODO: Fit the grid search object to the training data and find the optimal parameters
grid_obj = grid_obj.fit(X_train, y_train)

# Get the estimator
clf = grid_obj.best_estimator_

# Report the final F1 score for training and testing after parameter tuning
print "Tuned model has a training F1 score of {:.4f}.".format(predict_labels(clf, X_train, y_train))
print "Tuned model has a testing F1 score of {:.4f}.".format(predict_labels(clf, X_test, y_test))
Made predictions in 0.0005 seconds.
Tuned model has a training F1 score of 0.8040.
Made predictions in 0.0004 seconds.
Tuned model has a testing F1 score of 0.8050.

Scikit-learn's Pipeline Module: GridSearch Your Pipeline

Question 5 - Final F1 Score

What is the final model's F1 score for training and testing? How does that score compare to the untuned model?

Answer:

  • The final model's F1 scores are:
    • Training F1 score: 0.8040
    • Testing F1 score: 0.8050
  • There is an increase in the testing F1 score of the tuned model compared to the untuned model
    • It is now higher than Naive Bayes' F1 score