Performance Evaluation of Machine Learning Classifiers

The diagnostic ability of classifiers has usually been determined by the confusion matrix and the receiver operating characteristic (ROC) curve [37 (link)]. In the machine learning research domain, the confusion matrix is also known as error or contingency matrix. The basic framework of the confusion matrix has been provided in Fig. 11a. In this framework, true positives (TP) are the positive cases where the classifier correctly identified them. Similarly, true negatives (TN) are the negative cases where the classifier correctly identified them. False positives (FP) are the negative cases where the classifier incorrectly identified them as positive and the false negatives (FN) are the positive cases where the classifier incorrectly identified them as negative. The following measures, which are based on the confusion matrix, are commonly used to analyse the performance of classifiers, including those that are based on supervised machine learning algorithms.
Fig. 11

a The basic framework of the confusion matrix; and (b) A presentation of the ROC curve

$$\mathit{\text{Accuracy}}=\frac{\mathit{TP}+\mathit{TN}}{\mathit{TP}+\mathit{TN}+\mathit{FP}+\mathit{FN}}{F}_1\mathit{\text{score}}=\frac{2\times \mathit{TP}}{2\times \mathit{TP}+\mathit{FN}+\mathit{FP}}$$
$$\mathit{\text{Precisioin}}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FP}}\mathit{\text{Sensitivity}}=\mathit{\text{Recall}}=\mathit{\text{True positive rate}}=\frac{\mathit{TP}}{\mathit{TP}+\mathit{FN}}$$
$$\mathit{\text{Specificity}}=\frac{\mathit{TN}}{\mathit{TN}+\mathit{FP}}\mathit{\text{False positive rate}}=\frac{\mathit{FP}}{\mathit{FP}+\mathit{TN}}$$
An ROC is one of the fundamental tools for diagnostic test evaluation and is created by plotting the true positive rate against the false positive rate at various threshold settings [37 (link)]. The area under the ROC curve (AUC) is also commonly used to determine the predictability of a classifier. A higher AUC value represents the superiority of a classifier and vice versa. Figure 11b illustrates a presentation of three ROC curves based on an abstract dataset. The area under the blue ROC curve is half of the shaded rectangle. Thus, the AUC value for this blue ROC curve is 0.5. Due to the coverage of a larger area, the AUC value for the red ROC curve is higher than that of the black ROC curve. Hence, the classifier that produced the red ROC curve shows higher predictive accuracy compared with the other two classifiers that generated the blue and red ROC curves.
There are few other measures that are also used to assess the performance of different classifiers. One such measure is the running mean square error (RMSE). For different pairs of actual and predicted values, RMSE represents the mean value of all square errors. An error is the difference between an actual and its corresponding predicted value. Another such measure is the mean absolute error (MAE). For an actual and its predicted value, MAE indicates the absolute value of their difference.