TpTp

The simulation study evaluates methods that (differentially) associate gene expression with pseudotime for three different trajectory topologies, i.e., a cyclic, a bifurcating, and a multifurcating trajectory. As independent evaluation, we use the extensive trajectory simulation framework dynverse that previously served for benchmarking trajectory inference methods in Saelens et al.^{1 (link)}. Interested readers should refer to the original publication for details on the data simulation procedure. Data set characteristics are listed in Table 1.Table 1

Overview of simulated data sets.

	Cyclic data set	Bifurcating data set	Multifurcating data set
Simulation framework	dyngen	dyntoy	dyntoy
Number of cells	505–508	500	750
Number of genes	312–444	5000	5000
% of DE genes	42–47%	20%	20%
Number of lineages	1	2	3
Topology	Cyclic	Bifurcating	Multifurcating
Number of data sets	10	10	1

Each data set is simulated using one of the frameworks from the dynverse toolbox (dyngen or dyntoy), which are designed to simulate scRNA-seq data according to trajectory topologies. Each data set can be characterized by the topology of the trajectory, as well as the number of cells and genes. Low-dimensional representations of representative data sets can be found in Fig. 3. Note that the cyclic data sets have some variation in the numbers of genes and cells and in the amount of differential expression, which is inherent to the dyngen simulation framework.

For each of the cyclic and bifurcating topologies, we generate and analyze ten data sets. Since the multifurcating topology is very variable across simulations due to its flexible definition, its analysis requires substantial supervision. Therefore, we analyze only one representative multifurcating data set.
Prior to trajectory inference, the simulated counts are normalized using full-quantile normalization^{35 (link),36 (link)}. For TI with slingshot, we apply principal component analysis (PCA) dimensionality reduction to the normalized counts and k-means clustering in PCA space. For the bifurcating and multifurcating trajectories, the start and end clusters of the true trajectory are provided to slingshot to aid it in inferring the trajectory. For the edgeR analysis, we assess DE between the end clusters that are also provided to slingshot. The BEAM method can only test one bifurcation point at a time. For the multifurcating data set, we therefore assessed both branching points separately and aggregated the p-values using Fisher’s method³⁷ . For the tradeSeq and edgeR analyses of the multifurcating data set, we perform global tests across all three lineages.
We assess performance based on scatterplots of the true positive rate (TPR) vs. the false discovery proportion (FDP), according to the following definitions $\mathrm{FDP}=\frac{FP}{\max \left(1,FP+TP\right)}\mathrm{TPR}=\frac{TP}{TP+FN},$ where FN, FP, and TP denote, respectively, the numbers of false negatives, false positives, and true positives. FDP-TPR curves are calculated and plotted with the Bioconductor R package iCOBRA^{38 (link)}.

BIG-PI [10 (link),12 (link),13 (link)], GPI-SOM [15 (link)], FragAnchor [17 (link)] and MemType-2L [18 (link)] web server predictors were interrogated to test our datasets, while DGPI [14 ] was run locally with the last free available distribution. When testing BIG-PI, which implements different parameterizations for the different kingdoms, the suitable predictor was used for each protein.
Four parameters were used to evaluate the prediction performances. We indicated with TP and TN the number of True Positive and True Negative predictions, respectively, and with FP and FN the number of False Positive and False Negative predictions, respectively.
The Coverage, or true positive rate, was calculated as the number of proteins correctly predicted as GPI-anchored over the total number of positive examples.
Cov=TPTP+FN
The Accuracy value corresponds to the number of proteins correctly predicted as GPI-anchored over the total number of protein predicted as GPI-anchored.
Acc=TPTP+FP
The false positive rate corresponds to the number of protein predicted as GPI-anchored but annotated as negative examples over the total number of negative examples.
The Matthews Correlation Coefficient was calculated as:
MCC=TP⋅TN−FP⋅FN(TP+FP)(TP+FN)(TN+FP)(TN+FN)
A thorough explanation of the purposes of these indexes can be found in [24 (link)].

To simulate class-effect proportion (CEP), class effects are applied onto 0, 0.2, 0.5 and 0.8 of measured proteins (Fig. 2). The magnitude of applied effect sizes is randomly selected from 0.2, 0.5, 0.8, 1 and 2. The class effect is applied in one class, but not the other, and is a proportionate increment. For example, a 0.2 class-effect level means a 20% increment from the original value. When CEP is high, it leads to sample classes whose basal expression states are drastically different.Figure 2

Simulation strategies for data with simulated class and batch effects (A) and data with real batch effects, but simulated class effects (B).

Batch effects are simulated similarly, except the batch effects are inserted according to batch factors (the categorization of technical batches). In this simplistic scenario, we simply assign half of the samples of each class, to each batch.
Since the set of differential variables are known a priori, normalization performance across the five strategies may be evaluated by statistical feature selection (based on the two-sample t test; α = 0.05 significance level) and overall batch-effect correction based on the gPCA delta^{21 (link)} (see below).
For statistical feature selection, the precision, recall and their harmonic mean (the F-score) are used. These are expressed as: $$\begin{array}{cc}& \text{Precision}=\frac{\mathit{TP}}{TP+FP}\hfill \\ \hfill & \text{Recall}=\frac{\mathit{TP}}{TP+FN}\hfill \\ \hfill & \text{F-score}=2\times \frac{Precision\times Recall}{Precision+Recall}\hfill \\ \hfill \end{array}$$
where TP, FP and FN refer to true positives, false positives and false negatives, respectively. The efficacy of batch correction is evaluated using gPCA^{21 (link)}. The gPCA delta measures the proportion of variance due to batch effects in test data, and is bound between 0 and 1. Ideally, we want this to be as low as possible following normalization.

We formulated the prediction of dementia as a binary classification problem (dementia, control); therefore, evaluation metrics, such as accuracy, F1-score, balanced accuracy, precision, specificity and sensitivity, were used to measure the performance of the subsets of features. The following evaluation metrics were used:

True positives (TP): number of dementia cases that were correctly classified.

False positives (FP): number of healthy subjects incorrectly classified as dementia cases.

True negatives (TN): number of healthy subjects correctly classified.

False negatives (FN): number of dementia cases incorrectly classified as healthy subjects.

Accuracy (%): the proportion of correct classifications among total classifications:

$$\text{Accuracy}=\frac{\text{TP}+\text{TN}}{n}$$
where n is the number of total classifications per test.

Sensitivity (%): The proportion of correctly classified dementia cases.

$$\text{Sensitivity}=\frac{\text{TP}}{\text{TP}+\text{FN}}$$

Specificity (%): The proportion of correctly classified healthy subjects.

$$\text{Specificity}=\frac{\text{TN}}{\text{TN}+\text{FP}}$$

Precision: The proportion of subjects classified as dementia cases who have dementia.

$$\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}$$

F1-score (F-measure) (%): Harmonic mean of precision and sensitivity.

$$\text{F}1=2\times \frac{\text{Sensitivity}\times \text{Precision}}{\text{Sensitivity}+\text{Precision}}=\frac{\text{TP}}{\text{TP}+\left(\text{FP},+,\text{FN}\right)/2}$$

Some effective parameters were used to evaluate the performance of the proposed hybrid deep learning approach. These parameters are given by the following equations^{60 (link)}: $$\text{Accuracy}=\frac{Tp+Tn}{Tp+Fp+Tn+Fn}\times 100\%,$$ $$\text{Precision}=\frac{\mathit{Tp}}{Fp+Tp}\times 100\%,$$ $$\text{Specificity}=\frac{\mathit{Tn}}{Tn+Fp}\times 100\%,$$ $$\text{Sensitivity}=\frac{\mathit{Tp}}{Tp+Fn}\times 100\%,$$ $${\text{F}}_1\text{-score}=\frac{2Tp}{2Tp+Fp+Fn},$$ $$\text{Matthew's correlation coefficient}\left(\text{MCC}\right)=\frac{\left(Tp\times Tn\right)-\left(Fn\times Fp\right)}{\sqrt{\left(Tp+Fp\right)\left(Fp+Tn\right)\left(Tn+Fn\right)\left(Fn+Tp\right)}},$$ where Fn and Tn denote false-negative and true-negative values, and Fp and Tp denote the false-positive and true-positive values, respectively.

For feature extraction task, we utilized the proposed CNN architecture, SIFT and MobileNetV2 model. We utilized machine learning classifiers to classify the extracted features as COVID-19 case or not. The machine learning classifier consumes numerical values, as extracted from the CNN model, rather than a CT scan image. We proposed comparing the performance of a single classifier and an ensemble learning classifier by classifying the extracted features from the feature extraction phase. We utilized random forest, an example for Bagging ensemble methods, which depends on a decision trees model as a base classifier. Boosting methods are applied also by utilizing a support vector machine (SVM) classifier as a weak learner. On the other hand, a single classifier, logistic regression, is used to classify the extracted features in the previous stage.
The standard MobileNetV2 model extracts 62,720 numerical values with shape 7 × 7 × 1,280 of the CT scan image; these features are used as an input to the classification models. MobileNetV2 model with the fine-tuned model on top of it extracts 128 numerical values of the CT scan image for the classification process, as listed in Table 2. The proposed CNN model extracts 32 features from the CT scan image that are used as input for the classification task. As the proposed CNN model achieves better results than the pre-trained MobileNetV2 model, we were encouraged to merge the features extracted from proposed CNN and SIFT to together. As listed in Table 2, utilizing SIFT features with our proposed CNN features achieves the superior results.
Several metrics can evaluate the performance of a classifier. In the following, we explain the utilized metrics for evaluating the proposed classifier. Besides, we mean by True Positive (TP) the outcome where the model correctly predicts the positive class, e.g., positive COVID-19 patients diagnosed as COVID-19 (+). A True Negative (TN) is a results where the model predicts the negative class flawlessly, e.g., a CT scan image of COVID-19 patient is diagnosed as a negative COVID-19 (-) case. A False Positive (FP) case where the model incorrectly forecasts the positive class, e.g., patients suffering from other lung diseases and incorrectly classified as COVID-19 (+). False Negative is a case where the model incorrectly forecasts the negative class, e.g., patients infected by COVID-19 (+) and incorrectly classified as COVID-19 (-). Accuracy is defined as the fraction of correct predictions (both True Positive and True Negative).
The first evaluation metric is precision; where can it can be defined as the rate of TP outcomes to the total number of positive outcomes (TP + FP), as shown in Eq 2.
$$\begin{array}{c}\hfill Precision=\frac{TP}{TP+FP}\end{array}$$
The second evaluation metric is the recall; it is the ratio of TP outcomes to the actual number of positive samples (TP + FN), as shown in Eq 3.
$$\begin{array}{c}\hfill Recall=\frac{TP}{TP+FN}\end{array}$$
The third evaluation metric is the F1-score; it is the harmonic mean of precision and recall, as shown in Eq 4.
$$\begin{array}{c}\hfill F1-score=\frac{TP}{TP+0.5(FP+FN)}\end{array}$$
The fourth evaluation metric is the specificity; It is the ratio of TN outcomes to the total negative outcomes (TP + FN) of the model, as shown in Eq 5.
$$\begin{array}{c}\hfill Specificity=\frac{TN}{TN+FP}\end{array}$$
The fifth evaluation metric is the accuracy; It is the ratio of correct outcomes (TP + TN) to the total number of outcomes, as shown in Eq 6.
$$\begin{array}{c}\hfill Accuracy=\frac{TP+TN}{TP+TN+FP+FN}\end{array}$$