One of the commonly used model performance assessment tools is a confusion matrix. It compares actual labels vs predicted labels and allows us to measure accuracy, the ratio of correct predictions to the total number of predictions, and a few other measures such as precision, recall, etc.
A cost matrix is a way to specify the relative importance of accuracy for different predictions.
For example, miss-classify a good applicant to bad applicant and reject has different cost as compare to miss-classifying a bad applicant to a good applicant and approve a loan.
Some examples of classification.
Scenario 1: Credit Risk
Based on a credit risk scorecard, applications for a credit card are classified as “Good” and “Bad”. “Good” indicates applicants paying back dues on credit card and “Bad” indicates customers defaulting on the dues. Now, the customers are compared against the actual performance of the customer payment behavior after say 18 months. So, comparison of Predicted Class (“Good” or “Bad”) to actual customer behavior state (“Defaulted” or “Regular”).
Scenario 2: Healthcare Disease Diagnostic
In healthcare, a doctor suggests a series of tests to identify a disease for a patient. Once the doctor identifies the disease, relevant drugs, and medication could be given to the patients. Again, the doctor takes judgment (based on the test results and symptoms) call whether the patient has a disease say “A” or not.
So, confusion matrix could be used to compare predicted judgment on whether patients have disease “A” and actual confirmation on whether the patients had disease “A”.
Now, let’s take a step back and understand the “Predicted Class/outcome”. Predicted Class is identified from predictive model output and typically the output is probability values. For example, you have used the Logistic Regression based Predictive Model to find the probability of a customer defaulting and from probability values, an outcome class “Accept” or “Reject” decision is taken. If a bank accepts more applications, a higher number of False Positives (actual defaulters). If a bank accepts less number of applications, some of the good (or actually non-defaulters) will be rejected.
There is always a trade-off between Type I error/False Positives (accepting Bad Customers) and Type II Error/False Negatives (Rejecting Good Customers).
Similarly, in healthcare, Type I error/False Positives (treating or conducting further tests on patients for a disease which the patients do not have) and Type II Error/False Negatives (Concluding that patients do not have disease though they actually do have).
Risk Matrix or Cost Matrix
By default, all the miss-classification type has equal weights. Also, one or other error rates may have lower value but the cost of an error may be very severe. For example, in healthcare, missing a patient from treatment may lead to delay/death of the patient, but investigating patients with a few more tests will only lead to an increase in expenses.
Now, consider one more example of applying classification and checking accuracy using a confusion matrix.
One has developed a classification model that identifies whether a transaction is a case of money laundering. Now the cost of valid transaction marked as money laundering is the cost of investigation for the transaction(s) and impact on customer experience. But the cost of ignoring a money laundering transaction (by marking it as a normal transaction) is facing a huge penalty and reputational risk.
Similarly, in many practical scenarios, the cost of False Positives and False Negatives are different. So, one may require to optimize between False Positive Rate (Type I Error) and False Negative Rate (Type II Error).
So, the role of the cost matrix comes in picture to find the optimal cut off value for a classification rule. Now, going back to the Credit Risk Model. The cut-off value optimizes between the cost of an opportunity loss (miss to accept a good customer/Type II Error) and the cost of accepting a potential defaulter (involved in loss due to default). And based on historical data, we can estimate per customer/applicant cost of each of these errors and below cost-matrix has arrived.
In a credit risk default scenario, the cost of False Negatives is arrived based on percentage and count of good customers getting rejected. Similarly, the cost of False Positive is calculated based on the default rate, count of defaulters, and value per default customer.
Now, the cost matrix suggests that there is no cost of correct classification. The cost of approving a potential defaulter is 7times the cost of rejecting a good customer. But in some of the scenarios, assigning cost of Type I and Type II is not very easy especially in healthcare.