Differential Correlations and Information

In this section we leave neural networks, and turn to theoretical analysis of differential correlations. We analyze information when there is a ‘pure’ f′f′^T component and, just as importantly, when there is a not so pure component. We show that in the former case information saturates with N; in the latter case it doesn’t. We also show, somewhat surprisingly, that the optimal decoder doesn’t need to know about the f′f′^T component of the correlations. In the Supplementary Modeling, we provide further insight into differential correlations by expressing them in terms of the eigenvectors and eigenvalues of the covariance matrix, and we use that analysis to understand why, and when, it’s hard to accurately estimate Fisher information.
Here we ask how the linear Fisher information scales with the number of neurons, N, when the covariance matrix contains a pure f′f′^T component (the second term in equation (31)). Our starting point is a covariance matrix, Σ₀(s), that doesn’t necessarily contain an f′f′^T component. As in equation (3), the (linear) Fisher information associated with Σ₀(s), denoted I₀, is given by
$$I_0=\mathbf{f}\prime {(s)}^{\mathrm{T}}{\mathbf{\Sigma }}_0^{-1}(s)\mathbf{f}\prime (s)$$
where, as usual, f(s) is a vector of tuning curves,
$$\mathbf{f}(s)\equiv {(f_1(s),f_2(s),\dots ,f_N(s))}^{\mathrm{T}}$$
and a prime denotes a derivative with respect to s. Note that the information also depends on stimulus, s; we suppress that dependence for clarity. To add a pure f′f′^T component, we define a new covariance matrix, Σ_ε(s), via
$${\mathbf{\Sigma }}_{\epsilon }(s)={\mathbf{\Sigma }}_0(s)+\epsilon \mathbf{f}\prime (s){\mathbf{f}\prime }^{\mathrm{T}}(s)$$
The new information, denoted I_ε, is given by
$$I_{\epsilon }=\mathbf{f}\prime {(s)}^{\mathrm{T}}{\mathbf{\Sigma }}_{\epsilon }^{-1}(s)\mathbf{f}\prime (s)$$
To compute I_ε, we need the inverse of Σ_ε. As is easy to verify, this inverse is given by
$${\mathbf{\Sigma }}_{\epsilon }^{-1}(s)={\mathbf{\Sigma }}_0^{-1}(s)-\frac{\epsilon }{1+\epsilon I_0}{\mathbf{\Sigma }}_0^{-1}(s)\mathbf{f}\prime (s){\mathbf{f}\prime }^{\mathrm{T}}(s){\mathbf{\Sigma }}_0^{-1}(s)$$
Inserting equation (33) into (32), we arrive at
$$I_{\epsilon }=I_0-\frac{\epsilon I_0^2}{1+\epsilon I_0}=\frac{I_0}{1+\epsilon I_0}$$
which is equation (5).
Perhaps surprisingly, although f′f′^T correlations have a critical role in determining information, they are irrelevant for decoding, in the sense that they have no effect on the locally optimal linear estimator. To see this explicitly, note first of all that the locally optimal linear estimator, denoted w^T, generates an estimate of the stimulus near some particular value, s₀, by linearly operating on neural activity,
$$ŝ=s_0+{\mathbf{w}}^{\mathrm{T}}(\mathbf{r}-\mathbf{f}(s_0))$$
In the presence of the covariance matrix given in equation (31), the optimal weight, ${\mathbf{w}}_{\mathit{\text{opt}}}^{\mathrm{T}}$ is given by
$${\mathbf{w}}_{\mathit{\text{opt}}}^{\mathrm{T}}=\frac{{\mathbf{f}\prime }^{\mathrm{T}}{({\mathbf{\Sigma }}_0+\epsilon \mathbf{f}\prime {\mathbf{f}\prime }^{\mathrm{T}})}^{-1}}{{\mathbf{f}\prime }^{\mathrm{T}}{({\mathbf{\Sigma }}_0+\epsilon \mathbf{f}\prime {\mathbf{f}\prime }^{\mathrm{T}})}^{-1}\mathbf{f}\prime }$$
where we have dropped, for clarity, the explicit dependence on s₀. Using equation (33), this reduces to
$${\mathbf{w}}_{\mathit{\text{opt}}}^{\mathrm{T}}=\frac{{\mathbf{f}\prime }^{\mathrm{T}}{\mathbf{\Sigma }}_0^{-1}}{{\mathbf{f}\prime }^{\mathrm{T}}{\mathbf{\Sigma }}_0^{-1}\mathbf{f}\prime }$$
Thus, the locally optimal linear decoder does not need to know the size of the f′f′^T correlations.
In hindsight this makes sense: f′f′^T correlations shift the hill of activity, and there is, quite literally, nothing any decoder can do about this. This suggests that these correlations are in some sense special. To determine just how special, we ask what happens when we add correlations in a different direction—say correlations of the form uu^T, where u is not parallel to f′. In that case, the covariance matrix becomes (with a normalization added for convenience only)
$${\mathbf{\Sigma }}_u(s)={\mathbf{\Sigma }}_0(s)+\epsilon \frac{\mathbf{f}\prime {(s)}^{\mathrm{T}}{\mathbf{\Sigma }}_0^{-1}(s)\mathbf{f}\prime (s)}{{\mathbf{u}}^{\mathrm{T}}{\mathbf{\Sigma }}_0^{-1}(s)\mathbf{u}}\mathbf{u}{\mathbf{u}}^{\mathrm{T}}$$
Repeating the steps leading to equation (34), we find that
$$I_u\equiv \mathbf{f}\prime {(s)}^{\mathrm{T}}{\mathbf{\Sigma }}_u^{-1}(s)\mathbf{f}\prime (s)=I_0{\text{sin}}^2\mathit{\theta }+\frac{I_0{\text{cos}}^2\mathit{\theta }}{1+\epsilon I_0}$$
where I₀ is defined in equation (29) and
$$\text{cos}\mathit{\theta }\equiv \frac{\mathbf{f}\prime {(s)}^{\mathrm{T}}{\mathbf{\Sigma }}_0^{-1}(s)\mathbf{u}}{{[\mathbf{f}\prime {(s)}^{\mathrm{T}}{\mathbf{\Sigma }}_0^{-1}(s)\mathbf{f}\prime (s){\mathbf{u}}^{\mathrm{T}}{\mathbf{\Sigma }}_0^{-1}(s)\mathbf{u}]}^{1/2}}$$
Whenever θ ≠ 0—meaning u is not parallel to f′(s)—information does not saturate as N goes to infinity. Thus, in the large N limit, f′(s)f′(s)^T correlations are the only ones that cause saturation.
A Supplementary Methods Checklist is available.