Two models were constructed and compared to predict sentinel lymph node status: one using logistic ridge regression and another using random forest classification. Model building and analysis were completed in Python using scikit-learn. Logistic regression was chosen as the final model for ELMO as it results in a highly interpretable model; ridge regression controls for variance in the dataset.20 (link) Logistic ridge regression calculates the probability that a patient has SLN metastasis and then uses a probability threshold to assign a positive or negative prediction.
We also analyzed a random forest classification model. These models often have more predictive power than logistic regression models, because they do not assume data linearity.21 (link) Random forests are an ensemble of decision trees that assign a probability to an outcome based on the number of decision trees that predict that outcome. We chose a probability threshold of 5% for both models, which matches the current standard of care.7 (link),10 (link),22 (link)Clinical attributes that have been shown in previous work to have independent, biological relevance with regards to patient outcomes were included in our model, including Breslow depth, stage, ulceration, mitotic rate, age, sex, and primary site.
The development set was used to create and train the models. The tuning set was used to select the appropriate hyperparameters (such as the amount of regularization, a parameter that assists in tuning the model to avoid overfitting) and to select the best performing model.23 The validation cohort was kept separate and was only used to test the final model. Final model parameters were compared to the MSKCC and clinicopathological and gene expression profile (CP-GEP model) parameters through direct comparison to their published validation statistics.5 (link),7 (link),9 (link) Sensitivity, specificity, negative predictive value (NPV), positive predictive value (PPV), and AUC were calculated for the logistic regression and random forest models and compared to previously published model parameters. SLNB reduction rate was calculated by using ELMO to evaluate the percent of patients with ≥T1b melanoma who would not be recommended for SLNB, as previously described ([TN+FN]/[TP+TN+FN+FP]).10 (link) All code necessary to reproduce this model is available at the following link:https://github.com/karenlarson/MelanomaSentinel .
We also analyzed a random forest classification model. These models often have more predictive power than logistic regression models, because they do not assume data linearity.21 (link) Random forests are an ensemble of decision trees that assign a probability to an outcome based on the number of decision trees that predict that outcome. We chose a probability threshold of 5% for both models, which matches the current standard of care.7 (link),10 (link),22 (link)Clinical attributes that have been shown in previous work to have independent, biological relevance with regards to patient outcomes were included in our model, including Breslow depth, stage, ulceration, mitotic rate, age, sex, and primary site.
The development set was used to create and train the models. The tuning set was used to select the appropriate hyperparameters (such as the amount of regularization, a parameter that assists in tuning the model to avoid overfitting) and to select the best performing model.23 The validation cohort was kept separate and was only used to test the final model. Final model parameters were compared to the MSKCC and clinicopathological and gene expression profile (CP-GEP model) parameters through direct comparison to their published validation statistics.5 (link),7 (link),9 (link) Sensitivity, specificity, negative predictive value (NPV), positive predictive value (PPV), and AUC were calculated for the logistic regression and random forest models and compared to previously published model parameters. SLNB reduction rate was calculated by using ELMO to evaluate the percent of patients with ≥T1b melanoma who would not be recommended for SLNB, as previously described ([TN+FN]/[TP+TN+FN+FP]).10 (link) All code necessary to reproduce this model is available at the following link:
