CPSC 330 Lecture 10: Regression Metrics

Andrew Roth (Slides adapted from Varada Kolhatkar and Firas Moosvi)

Announcements

  • Important information about midterm 1
    • https://piazza.com/class/m4ujp0s4xgm5o5/post/204
    • Good news for you: You’ll have access to our course notes in the midterm!
  • Practice midterm questions available on PL
    • You can see an example of all the info you will have access to in the last question.
  • HW4 due Monday
  • Remember to attend tutorials!
    • This weeks tutorial will cover data imbalance and fairness

Recap: Confusion matrix

  • TN \(\rightarrow\) True negatives
  • FP \(\rightarrow\) False positives
  • FN \(\rightarrow\) False negatives
  • TP \(\rightarrow\) True positives

Recap: Precision, Recall, F1-Score

\[ f1 = 2 \times \frac{ precision \times recall}{precision + recall}\]

Recap: iClicker 9.2-C

Lowering the classification threshold generally increases the model’s recall.

\[ recall = \frac{TP}{TP + FN} \]

Recap: iClicker 9.2-D

Raising the classification threshold can improve the precision of the model if it effectively reduces the number of false positives without significantly affecting true positive.

\[ precision = \frac{TP}{TP + FP} \]

Recap: PR curve

  • Calculate precision and recall (TPR) at every possible threshold and graph them.
  • Better choice for highly imbalanced datasets because it focuses on the performance of the positive class.

Recap: Point vs curve metrics

  • What’s the difference between the average precision (AP) score and F1-score?
  • Which model would you pick?

Recap: ROC curve

  • Calculate the true positive rate (TPR aka recall) (\(\frac{TP}{TP + FN}\)) and false positive rate (FPR) (\(\frac{FP}{FP + TN}\)) at every possible thresholding and graph TPR over FPR.
  • Good choice when the datasets are roughly balanced.

Recap: ROC Curve

  • Not a great choice when there is an extreme imbalance because FPR can remain relatively low even if the number of false positives is high, simply because the number of negatives is very large.
    \[ FPR = \frac{FP}{FP + TN}\]
  • The area under the ROC curve (AUC) represents the probability that the model, if given a randomly chosen positive and negative example, will rank the positive higher than the negative.

Recap: ROC Curve

  • Which model would you pick?

Questions for you

  • What’s the AUC of a baseline model?

Source

Dealing with class imbalance

  • Under sampling
  • Oversampling
  • class weight="balanced" (preferred method for this course)
  • SMOTE

Liquid biopsy aside

Regression metrics class demo

Ridge and RidgeCV

  • Ridge Regression: alpha hyperparameter controls model complexity.
  • RidgeCV: Ridge regression with built-in cross-validation to find the optimal alpha.

alpha hyperparameter

  • Role of alpha:
    • Controls model complexity
    • Higher alpha: Simpler model, smaller coefficients.
    • Lower alpha: Complex model, larger coefficients.

Regression metrics: MSE, RMSE, MAPE

  • Mean Squared Error (MSE): Average of the squares of the errors.
  • Root Mean Squared Error (RMSE): Square root of MSE, same units as the target variable.
  • Mean Absolute Percentage Error (MAPE): Average of the absolute percentage errors.

Applying log transformation to the targets

  • Suitable when the target has a wide range and spans several orders of magnitude
    • Example: counts data such as social media likes or price data
  • Helps manage skewed data, making patterns more apparent and regression models more effective.
  • TransformedTargetRegressor
    • Wraps a regression model and applies a transformation to the target values.

iClicker Exercise 10.1

iClicker cloud join link: https://join.iclicker.com/HTRZ

Select all of the following statements which are TRUE.

    1. Price per square foot would be a good feature to add in our X.
    1. The alpha hyperparameter of Ridge has similar interpretation of C hyperparameter of LogisticRegression; higher alpha means more complex model.
    1. In Ridge, smaller alpha means bigger coefficients whereas bigger alpha means smaller coefficients.

iClicker Exercise 10.2

iClicker cloud join link: https://join.iclicker.com/HTRZ

Select all of the following statements which are TRUE.

    1. We can use still use precision and recall for regression problems but now we have other metrics we can use as well.
    1. In sklearn for regression problems, using r2_score() and .score() (with default values) will produce the same results.
    1. RMSE is always going to be non-negative.
    1. MSE does not directly provide the information about whether the model is underpredicting or overpredicting.
    1. We can pass multiple scoring metrics to GridSearchCV or RandomizedSearchCV for regression as well as classification problems.