Optimal Policies for Perceptual Decision-Making

Here we provide a technical description of the task and how to find the optimal behavior. More details and many helpful intuitions are provided in the Results section. We assume that the state of the world is either H₁ or H₂, and it is the aim of the decision maker to identify this state (indicated by a choice) based on stochastic evidence. This evidence δx ~ N(μδt, δt) is Gaussian for some small time period δt, with mean μδt and variance δt, where |μ| is the evidence strength, and μ ≥ 0 and μ < 0 correspond to H₁ and H₂ respectively. Such stochastic evidence corresponds to a diffusion model
$\frac{\text{d}x}{\text{d}t}=\mu +\eta (t)$ , where η(t) is white noise with unit variance, and x(t) describes the trajectory of a drifting/diffusing particle. We assume the value of μ to be unknown to the decision maker, to be drawn from the prior p(μ) across trials, and to remain constant within a trial. After accumulating evidence δx_0…t by observing the stimulus for some time t, the decision maker holds belief g(t) ≡ p(H₁|δx_0…t) = p(μ ≥ 0|δx_0t) (or 1 − g(t)) that H₁ (or H₂) is correct (Fig 1A). The exact form of this belief depends on the prior p(μ) over μ and will be discussed later for different priors. As long as this prior is symmetric, that is p(μ ≥ 0) = p(μ < 0) = ½, the initial belief at stimulus onset, t = 0, is always g(0) = ½.
The decision maker receives reward R_ij for choosing H_i when H_j is correct. Here rewards can be positive or negative, allowing for instance for negative reinforcement when subjects pick the wrong hypothesis, that is, when i is different from j. Additionally, we assume the accumulation of evidence to come at a cost (internal to the decision maker), given by the cost function c(t). This cost is momentary, such that the total cost for accumulating evidence if a decision is made at decision time T_d after stimulus onset is
$C(T_d)={\int }_0^{T_d}c(t)\text{d}t$ (Fig. 1B). Each trial ends after T_t seconds and is followed by the inter-trial-interval t_i and an optional penalty time t_p for wrong decisions (Fig. 1C). We assume that decision makers aim at maximizing their reward rate, given by
$$\rho =\frac{\langle R\rangle -\langle C(T_d)\rangle }{\langle T_t\rangle +\langle t_i\rangle +\langle t_p\rangle },$$ where the averages are over choices, decision times, and randomizations of t_i and t_p. We differentiate between fixed duration tasks and reaction time tasks. In fixed duration tasks, we assume T_t to be fixed by the experimenter and to be large when compared to T_d, and t_p = 0. This makes the denominator of Eq. (1) constant with respect to the subject’s behavior, such that maximizing the reward rate ρ becomes equal to maximizing the expected net reward 〈R〉 − 〈C(T_d) 〉 for a single trial. In contrast, in reaction time tasks, we need to consider the whole sequence of trials when maximizing ρ because the denominator depends on the subject’s reaction time through T_t, which, in turn, influences the ratio (provided that T_t is not too short compared to the inter-trial interval and penalty time),