Negative Binomial Generalized Linear Models for Transcriptome Analysis

GLMs are an extension of classical linear models to non-normally distributed response data (42 ,43 ). GLMs specify probability distributions according to their mean–variance relationship, for example the quadratic mean–variance relationship specified above for read counts. Assuming that an estimate is available for ϕ_g, so the variance can be evaluated for any value of μ_gi, GLM theory can be used to fit a log-linear model

for each gene (32 (link),41 ). Here x_i is a vector of covariates that specifies the treatment conditions applied to RNA sample i, and β_g is a vector of regression coefficients by which the covariate effects are mediated for gene g. The quadratic variance function specifies the negative binomial GLM distributional family. The use of the negative binomial distribution is equivalent to treating the π_gi as gamma distributed.