Assignment 4 for Course 1MS041

Make sure you pass the # ... Test cells and submit your solution notebook in the corresponding assignment on the course website. You can submit multiple times before the deadline and your highest score will be used.

---

Assignment 4, PROBLEM 1

Maximum Points = 24

This time the assignment only consists of one problem, but we will do a more comprehensive analysis instead.

Consider the dataset Corona_NLP_train.csv that you can get from the course website git. The data is "Coronavirus tweets NLP - Text Classification" that can be found on kaggle. The data has several columns, but we will only be working with OriginalTweetand Sentiment.

1. [3p] Load the data and filter out those tweets that have Sentiment=Neutral. Let $X$ represent the OriginalTweet and let $$ Y = \begin{cases} 1 & \text{if sentiment is towards positive} \\ 0 & \text{if sentiment is towards negative}. \end{cases} $$ Put the resulting arrays into the variables $X$ and $Y$. Split the data into three parts, train/test/validation where train is 60% of the data, test is 15% and validation is 25% of the data. Do not do this randomly, this is to make sure that we all did the same splits (we are in this case assuming the data is IID as presented in the dataset). That is [train,test,validation] is the splitting layout.

2. [4p] There are many ways to solve this classification problem. The first main issue to resolve is to convert the $X$ variable to something that you can feed into a machine learning model. For instance, you can first use CountVectorizer as the first step. The step that comes after should be a LogisticRegression model, but for this to work you need to put together the CountVectorizer and the LogisticRegression model into a Pipeline. Fill in the variable model such that it accepts the raw text as input and outputs a number $0$ or $1$, make sure that model.predict_proba works for this. Hint: You might need to play with the parameters of LogisticRegression to get convergence, make sure that it doesn't take too long or the autograder might kill your code

3. [3p] Use your trained model and calculate the precision and recall on both classes. Fill in the corresponding variables with the answer.

4. [3p] Let us now define a cost function

   • A positive tweet that is classified as negative will have a cost of 1
   • A negative tweet that is classified as positive will have a cost of 5
   • Correct classifications cost 0

   complete filling the function cost to compute the cost of a prediction model under a certain prediction threshold (recall our precision recall lecture and the predict_proba function from trained models).

5. [4p] Now, we wish to select the threshold of our classifier that minimizes the cost, fill in the selected threshold value in value optimal_threshold.

6. [4p] With your newly computed threshold value, compute the cost of putting this model in production by computing the cost using the validation data. Also provide a confidence interval of the cost using Hoeffdings inequality with a 99% confidence.

7. [3p] Let $t$ be the threshold you found and $f$ the model you fitted (one of the outputs of predict_proba), if we define the random variable $$ C = (1-1_{f(X)\geq t})Y+5(1-Y)1_{f(X) \geq t} $$ then $C$ denotes the cost of a randomly chosen tweet. In the previous step we estimated $\mathbb{E}[C]$ using the empirical mean. However, since the threshold is chosen to minimize cost it is likely that $C=0$ or $C=1$ than $C=5$ as such it will have a low variance. Compute the empirical variance of $C$ on the validation set. What would be the confidence interval if we used Bennett's inequality instead of Hoeffding in point 6 but with the computed empirical variance as our guess for the variance?

# Part 1

# Load the data from the file specified in the problem definition and make sure that it is loaded using
# the search path `data/Corona_NLP_train.csv`. This is to make sure the autograder and your computer have the same
# file path and can load the data correctly.

# Contrary to how many other problems are structured, this problem actually requires you to
# have X on the shape (n_samples, ) that is a 1-dimensional array. Otherwise it will cause a bunch
# of errors in the autograder or also in for instance CountVectorizer.

# Make sure that all your data is numpy arrays and not pandas dataframes or series.
X = XXX
Y = YYY

X_train = XXX
Y_train = YYY
X_test = XXX
Y_test = YYY
X_valid = XXX
Y_valid = YYY

# Part 2

# Train a machine learning model or pipeline that can take the raw strings from X and predict Y=0,1 depending on the
# sentiment of the tweet. Store the trained model in the variable `model`.

model = XXX

# Part 3

# Evaluate the model on the test set and calculate precision, and recall on both classes. Store the results in the
# variables `precision_0`, `precision_1`, `recall_0`, `recall_1`.

precision_0 = XXX
precision_1 = XXX
recall_0 = XXX
recall_1 = XXX

# Part 4

def cost(model,threshold,X,Y):
    # Hint, make sure that the model has a predict_proba method
    # think about how the decision is made based on the probabilities
    # and how the threshold can be used to make the decision.
    # For reference take a look at the lecture notes "Bayes classifier"
    # which contains how the decision is made based on the probabilities when the threshold is 0.5.
    
    # Fill in what is missing to compute the cost and return it
    # Note that we are interested in average cost
    
    return XXX

# Part 5

# Find the optimal threshold for the model on the test set. Store the threshold in the variable `optimal_threshold`
# and the cost at the optimal threshold in the variable `cost_at_optimal_threshold` evaluated on the test set.
optimal_threshold = XXX
cost_at_optimal_threshold = XXX

# Part 6

cost_at_optimal_threshold_valid = XXX
cost_interval_valid = XXX

assert(type(cost_interval_valid) == tuple)
assert(len(cost_interval_valid) == 2)

# Part 7

variance_of_C = XXX
interval_of_C = XXX

assert(type(interval_of_C) == tuple)
assert(len(interval_of_C) == 2)