Author response: Orbitofrontal neurons acquire responses to ‘valueless’ Pavlovian cues during unblocking

Full text Figures and data Side by side Abstract eLife digest Introduction Results Discussion Materials and methods References Decision letter Author response Article and author information Metrics Abstract The orbitofrontal cortex (OFC) has been described as signaling outcome expectancies or value. Evidence for the latter comes from the studies showing that neural signals in the OFC correlate with value across features. Yet features can co-vary with value, and individual units may participate in multiple ensembles coding different features. Here we used unblocking to test whether OFC neurons would respond to a predictive cue signaling a ‘valueless’ change in outcome flavor. Neurons were recorded as the rats learned about cues that signaled either an increase in reward number or a valueless change in flavor. We found that OFC neurons acquired responses to both predictive cues. This activity exceeded that exhibited to a ‘blocked’ cue and was correlated with activity to the actual outcome. These results show that OFC neurons fire to cues with no value independent of what can be inferred through features of the predicted outcome. https://doi.org/10.7554/eLife.02653.001 eLife digest Imagine you are at a restaurant and the waiter offers you a choice of cheesecake or fruit salad for dessert. When making your choice it is likely that you will consider the features of these desserts, such as their taste, their sweetness or how healthy they are. However, when you decide which dessert to have, you will pick the one that you judge to have the highest value for you at that moment in time. In this sense, ‘value’ is a subjective concept that varies from person to person, while ‘features’ remain relatively static. It is generally agreed that the orbitofrontal cortex (OFC) is involved in making these sorts of decisions, but its role is still a topic of debate. According to one theory the neurons in the OFC signal the subjective value of an outcome, whereas a rival theory suggests that they signal the features of the expected outcome. However, it has proved challenging to test these theories in experiments because it is difficult to say for certain that a given decision was clearly due to the value or a feature. Now, McDannald et al. have devised an approach that can tell the difference between neurons signaling value and neurons signaling features. They trained thirsty rats to associate different odours with either an increase in the amount of milk they were given (a change in both value and a feature), or a change in the flavor of the milk (a change in a feature without a change in value). Extensive testing showed that the rats did not value one flavor over the other. McDannald et al. then examined how the neurons in the OFC responded. If these neurons signal only value, they should only fire when the value of the outcome changes. On the other hand, if they signal features, they should fire when a feature changes, even if the value does not. It turned out that the neurons in the OFC responded whenever the features changed, irrespective of whether or not the value changed. These findings present a challenge to popular conceptions of the role of the neurons in the OFC. https://doi.org/10.7554/eLife.02653.002 Introduction The orbitofrontal cortex (OFC) is often described as signaling either an outcome expectancy, implying a knowledge of the features of the impending outcome (Schoenbaum et al., 1998; Delamater, 2007; Ostlund and Balleine, 2007; Steiner and Redish, 2012; Luk and Wallis, 2013), or a value that exists independent of those features (Padoa-Schioppa, 2011; Levy and Glimcher, 2012). Support for such pure or abstract value encoding comes largely from reports that single unit activity and the blood-oxygen level dependent (BOLD) response in the OFC tracks value, independent of outcome features such as identity or location or even the response required to obtain the outcome (Padoa-Schioppa and Assad, 2006; Plassmann et al., 2007; Levy and Glimcher, 2011). Yet in some cases, outcome features could still be the underlying basis of apparent abstract value signals. This is conceivable even if the signal correlates with value across different outcomes, since some features or feature combinations might vary with value across the limited number of outcomes used in any particular session (but see Padoa-Schioppa and Assad, 2006, supplemental, and our correction notice). Further, any neural element (voxel or single unit) may participate in ensembles responding to more than one feature, so it is also possible that a particular element that appears not to distinguish specific features of different outcomes is in fact coding independent features that co-vary with each outcome's value. So how can we address whether the OFC signals features of impending outcomes vs value independent of those features? One way is to strip away or ‘block’ the value portion of the outcome during learning, while leaving unblocked—free to enter into associations—the outcome's unique sensory and other features. This can be done by pairing a ‘target’ cue with a rewarding outcome in the presence of a cue that has been previously trained to predict a differently-flavored, but similarly-valued outcome. When this is done, the previously conditioned cue predicts the general value that is common to the two outcomes, but does not predict the unique features that distinguish the new outcome (note features are not limited to sensory properties, but might include the outcome timing, location, temperature, size, number, etc). As a result, the target cue acquires associations with the unique features of the new outcome but not with its general or common currency value (Rescorla, 1999; Burke et al., 2008). If OFC neurons represent only a general or common currency value, divorced from features, then they should respond no more to such a target cue than to a completely blocked cue (Kamin, 1969). However, if OFC neurons represent outcome features, independent of value, then they should respond to the target cue just as they do to a cue that has been explicitly unblocked by increasing the amount of the outcome delivered. Indeed, both pure value and outcome expectancy accounts of OFC function would predict neural activity to such an unblocked cue, but only an outcome expectancy account predicts encoding of the target cue signaling a valueless change in outcome flavor. Results We recorded single–unit activity in the OFC in six rats during an odor-based unblocking task (Figure 1A). Prior to implantation with electrodes rats were trained to sample an odor in a central port following house light illumination and then respond to a reward well below for two drops of Nestlé's flavored milk (chocolate or vanilla, counterbalanced). This training was meant to establish the initial odor as a reliable predictor of a specific flavor and number of drops of milk. Each rat had extensive experience with both flavors, thus neither flavor was novel. Following initial training rats were implanted with microelectrodes in the OFC. When recovered from surgery, rats were retrained on the initial odor; after retraining, each rat underwent 7–9 rounds of unblocking. Figure 1 with 1 supplement see all Download asset Open asset Experimental outline, behavior summary and recording sites. (A) Thirsty rats were initially trained to enter an odor port following illumination of a house light and respond at reward well below for two drops of flavored milk. Unblocking sessions consisted of four trial types. The first was a reminder of initial training. On the remaining three trial types, the originally trained odor was briefly presented followed by 1 of 3 novel odors. The reward following the novel odors was either unchanged (black; blocked trials), increased in number (blue; number trials), or its flavor was altered (green; flavor trials). Learning was assessed in a probe test in which the novel odors were presented in isolation, without reward. (B) 10-min consumption testing between chocolate and water, and vanilla and water on non-training days found that both were significantly and equally preferred to water (ANOVA, F1,5 > 5, p's < 0.05). (C) 2-min preference testing between chocolate and vanilla immediately following unblocking sessions found no flavor preference (t test, p > 0.1). Scatter plot (right) shows preference for the trained flavor on each individual test (n = 13). (D) 20-min consumption testing from a separate group of rats (n = 8) that received selective devaluation of one of the flavors found a significant difference in consumption between the non-devalued (Con) and devalued (Dev) flavors (t test, p < 0.01). This was true for every rat tested (right). (E) Time in the reward well is plotted for the probe test trials. ANOVA for time spent in the reward well with odor (blocked, number and flavor) and trial (1-15) as factors found a significant odor x trial interaction (F1,47 = 3.45, p < 0.05). Planned comparisons confirmed that on the first three trials rats spent significantly more time in the reward well following number and flavor odors compared to blocked (p's < 0.05) but responding to number and flavor did not differ (p > 0.1). (F) Single unit activity was recorded from the lateral orbital and agranular insular cortices at roughly 3.2 mm anterior to bregma. *p < 0.05; ns = not significant. https://doi.org/10.7554/eLife.02653.003 Each round of unblocking began with 2 days of training and consisted of four trial types. One type was a reminder; the initially trained odor was followed by the expected outcome. On the other three trial types (blocked, number, flavor), rats were presented with the initially trained odor, followed immediately by one of three novel odors. On ‘blocked’ trials, the novel odor was followed by the expected two drops of the same flavor used in initial training. This outcome is fully predicted by the initial odor, thus the novel odor should be blocked from acquiring associative significance (Kamin, 1969). On ‘flavor’ trials, the novel odor was followed by two drops of the flavor not used in initial training (i.e., chocolate or vanilla). Here the value is unchanged, since the two flavors are equally preferred and the same amount is delivered, but the features of the outcome are different. Thus the novel target odor should enter into associations with the unique features of the outcome (Rescorla, 1999; Burke et al., 2008). On ‘number’ trials, the novel odor was followed by an additional drop of the flavor used in the initial training. Since the initial odor does not predict anything after the second drop, the novel odor should enter into associations with both the features and the additional value of the outcome presented in the third drop (Holland, 1984). To ensure that learning in the flavor condition did not result from an explicit shift in value, two types of preference tests were administered in conjunction with the unblocking training procedures. In one test, given on days separate from unblocking, preference for each flavor over water was assessed. Both flavors were highly preferred to water (Figure 1B), demonstrating both are highly palatable. The more important preference test came just following unblocking sessions, from which the critical neural data came. In these tests, the two milk flavors were pitted directly against one another. This test is critical to demonstrate that specific satiety to one flavor did not develop over the course of the unblocking session, a finding that would strongly suggest different valuation of the two flavors. Indeed, we found no evidence of a preference in these tests (Figure 1C). In both types of tests the locations of the bottles were swapped every 20–30 s, meaning that rats were required to switch locations if the solution did in fact differ in value. This pattern was present when either milk flavor was compared to water but was absent when the two flavors were directly compared. The consumption data demonstrate that both flavors were highly palatable yet of equivalent value. Finally, to ensure the two flavors were discriminable we subjected another set of rats to a selective conditioned flavor aversion procedure. After initial exposure to both milk flavors, consumption of one flavor (fully counterbalanced) was devalued by pairing with LiCl-induced nausea while consumption of the other was paired with saline injection that is of minimal consequence. At no point did rats show a preference for the chocolate or vanilla flavor but all rats selectively reduced consumption of the devalued flavor. This was apparent both in conditioning (Figure 1—figure supplement 1) and in the final choice test (Figure 1D). Thus extensive consumption testing indicated that the flavors used were of equivalent value but readily discriminable. In the unblocking sessions rats were sensitive to presentation of the novel odors, exhibiting longer latencies to respond at the reward well following odor sampling on these three trial types. Longer latencies to the novel odors were most apparent on the very first trial of each session, particularly on day 1. In support, ANOVA revealed a main effect of trial (F1,47 > 2, p's < 0.01) and a trial × day interaction (F1,47 = 34.73, p < 0.01). However the rats also learned that the two novel odors that predicted changes in the outcome were meaningful. This was evident in the extinction probe test in which they initially spent more time in the fluid well following sampling of the flavor and number odors than following the blocked odor (Figure 1E). We recorded 240 single units during the first day of unblocking and 220 units on the second unblocking day in 48 rounds of training across all six rats (Figure 1F). To address our hypothesis, units from both unblocking days were screened for phasic responses to one of the four odors using a t test, which compared firing rates during the ITI and novel odor period (significance level = p < 0.0125; Bonferroni correction). This screen found 135 units (Day 1 = 79, Day 2 = 56) that showed a significant increase in firing to at least one of the odor cues. The majority–98/135 or 73% of the neurons within this population–exhibited activity that fell into one of two categories (see Figure 2—figure supplement 1 for analysis of other neurons). We will consider each category in turn below. The first major category, not directly anticipated by our hypothesis, consisted of neurons (55/135, Figure 2A) that showed a significant phasic response to each of the four odor cues. Since these neurons fired to all of the odors, even the blocked odor, their firing cannot be easily explained as signaling information about the predicted outcomes. However they might be signaling information about the cues themselves, such as their shared sensory features or intrinsic salience. Unlike shared sensory features, salience should be higher for the novel cues than for the pre-trained, initial odor in our design, and this pattern should be most noticeable early in training when the novel cues were first presented. As a population, these neurons did show greater activity to the blocked, number and flavor odors than to the more familiar, initial odor (Figure 2B). This effect was present despite the fact that we did not select based on this criterion. Further analyses of the individual units showed that many exhibited significantly higher firing to the novel odors than to the initial odor (Figure 2C), and few exhibited differences in firing among the three novel odors (Figure 2D,E). Moreover, when we examined the firing of neurons in this population on the first 10 trials of unblocking (31/79 odor-responsive neurons; Figure 2—figure supplement 2), analyzing the difference in firing between the novel and initial odor cues in a sliding, 300-ms window across each trial, we found that activity in this population was maximal at the onset of the novel odors and on the first exposure and then declined rapidly on subsequent trials (Figure 2F). This same pattern held when each novel odor was analyzed separately (Figure 2—figure supplement 3). Further, this pattern was only seen on the first day of unblocking (Figure 2—figure supplement 4). This pattern of activity is consistent with signaling of the salience of these cues. Figure 2 with 4 supplements see all Download asset Open asset Single unit and population firing of putative salience neurons. (A) Raster plots for firing of a single unit are shown for all initial (red), blocked (black), number (blue) and flavor trials (green). Odor on (On) is indicated by the first vertical line, onset of novel odor (Nov) by the second vertical line and odor offset (Off) by the third. Each tick represents a spike. Average activity across all trials for each odor is plotted (bottom). (B) Mean neural activity (novel odor epoch–ITI) for the putative salience neurons (n = 55) is plotted. Line color as indicated in raster plots; shaded areas indicate standard error of mean. ANOVA with bin and odor as factors found significant effects of bin, odor and the bin × odor interaction (F1,54 > 2.0, p's < 0.01). ANOVA restricted to the novel odor period with odor and time (first 500 ms vs second 500 ms, shown in upper right inset) as factors found only a main effect of odor (F1,54 = 13.0, p < 0.01). Significant firing to the novel odors over the initial odor was observed throughout the novel odor period. (C) A scatter plot of novel odor firing vs initial odor firing is shown for putative salience neurons (n = 55). A signed square root transformation of firing was used to best visualize population spread; all statistics were performed on non-transformed firing rates. Individual neurons showing significant differences in firing between the odors are outlined in black (t test, p < 0.05). A non-parametric sign test found significant, preferential firing to the novel odors (Z = 3.24, p < 0.01). The population bias towards novel odor firing is apparent in the bar histogram aligned to the diagonal axis; on which the difference score for each neuron is plotted. Light gray bars represent units showing no differential firing; dark gray bars represent units showing significant differential firing. (D) A scatter plot of predictive vs blocked odor firing is shown. A sign test found no differential firing to the predictive and blocked odors by the putative salience population (Z = 0.54, p > 0.1). (E) A scatter plot of flavor and number odor firing is shown. A sign test found no differential firing to the number and flavor odors (Z = 0.27, p > 0.1). (F) Differential firing to the novel odors vs the initial odor on the first 10 trials of the first unblocking day was calculated and plotted for the putative salience population (n = 31). Differential firing was calculated in a 300-ms sliding window for each 50-ms bin moving away from novel odor onset: (mean [(blocked odor–ITI) + (number odor–ITI) + (flavor odor–ITI)]–initial odor–ITI). The difference score for each bin was then plotted, with dark red bins indicating maximal differential firing to the novel odors and dark blue indicating the opposite pattern (y-axis shown on right of heat plot). (G) The significance of the increased firing to the novel odors was determined by performing a one-tailed t test, comparing increases in differential firing to 0, using a significance of p < 0.05 and a sliding window as in (F). Red bins indicate significant elevations in firing to the novel odors over the initial odor. https://doi.org/10.7554/eLife.02653.005 The second major category, of greater relevance to our hypothesis, consisted of neurons (43/135, Figure 3A–C) that showed a significant phasic response the flavor and/or number odors (but did not fire to all four odors). Activity across this population was greater in response to the two predictive odor cues than to either the blocked or initial odors (Figure 3D), and an analysis of individual units showed that nearly all of these neurons (38/43) fired more to the predictive odors than to the blocked one (Figure 3E). This result marks these neurons as candidates for encoding of associative information or meaning, since this is the primary feature that distinguishes the two odor cues from the blocked odor. Figure 3 with 1 supplement see all Download asset Open asset Single unit and population firing of putative predictive neurons. Single units plotted exactly as in Figure 2A showing (A) selective firing to the number odor (B) selective firing to the flavor odor (C) firing to both number and flavor odors. (D) Mean neural activity (novel odor epoch–ITI) for the putative predictive neurons (n = 43) is plotted. Meaning of line colors and shading is maintained. ANOVA with bin and odor as factors found significant effects of bin, odor and the bin × odor interaction (F1,42 > 2.0, p's < 0.01). ANOVA restricted to the novel odor period with odor and time (first 500 ms vs second 500 ms, shown in upper right inset) as factors found a main effect of odor (F1,42 = 58.5, p < 0.01) and an odor × time interaction (F1,42 = 3.9, p < 0.05). At both times firing to the predictive odors was significantly greater than the blocked and initial odors; blocked firing was greater than initial firing only in the first half. (E) A scatter plot comparing firing to the predictive odors (signed square-root transform) vs the blocked odor is shown for the predictive population (n = 43). Within this population there were three kinds of neurons based on firing vs ITI: number only (blue), flavor only (green) or flavor and number (purple). A sign test found significant, preferential firing to the predictive odors (Z = 4.88, p(s) < 0.01). Across all neurons there was zero correlation between predictive odor firing and blocked odor firing (R2 = −0.01, p(r) = 0.39). (F) A scatter plot comparing firing to the flavor and number odors is shown for the predictive population (n = 43). While some neurons did show differential firing (outlined in black) to either the number (n = 14) or flavor (n = 4) odor a sign test found no bias in firing to the number or flavor odor across the entire population. (Z = 0.60, p > 0.1). Across all neurons there was a highly significant, positive relationship between number and flavor odor firing (R2 = 0.88, p < 0.01). (G) Differential firing to the number odor vs the blocked odor on the first 10 trials of the first unblocking day was calculated and plotted for the number-responsive units within the predictive population (n = 21). This was done in as in (Figure 2F) except that the difference score was calculated as: (number odor–ITI)—(blocked odor–ITI). (H) Significance for increased firing to the number odor over the blocked odor was calculated as in (G). Red bins indicate significant elevations in firing to the number odor over the blocked odor. Blue bins would indicate significant decreases in firing to the number odor below the blocked odor. (I) Differential firing to the flavor odor vs blocked odor on the first 10 trials of the first unblocking day was calculated and plotted for flavor-responsive units within the predictive population (n = 18) as was done in (G). (J) Significance of differential firing calculated and displayed as in (H). https://doi.org/10.7554/eLife.02653.010 Interestingly, these neurons did not appear to distinguish, at least as a population, between the flavor and number odors. For example, they showed similar levels of activity in response to both the flavor and the number odor (Figure 3D,F), and when we examined the firing of neurons in this population on the first 10 trials of unblocking (25/79 odor-responsive neurons, Figure 3—figure supplement 1), we found that differential firing to each cue developed at a similar rate during training (Figure 3G–J). The acquisition of differential firing to the flavor and number odors demonstrates that selective odor encoding was not driven by physical properties of the odors. If neurons were encoding the odor itself, independent of its outcome signaling, this would have been apparent on the very first trial. Thus, as a population, these neurons responded more strongly to the unblocked ‘flavor’ and ‘number’ odors than to the ‘blocked’ odor. Even more striking, the population responded similarly to a cue signaling a valueless change in the outcome flavor as they did to a cue that signaling that more of the outcome would be delivered. Similar numbers of neurons fired to the flavor and number cues (flavor: 31, number: 37; χ2 = 0.3, p = 0.47). In our design, firing to the flavor cue cannot be readily explained as signaling general or common value. Thus these data confirm that many OFC neurons signal associative meaning independent of at least a general value. In support of this, odor firing in the flavor population, as well as the number population, was positively correlated with firing to the actual outcome delivered on each of these trials (Figure 4). This relationship supports the idea that cue-evoked activity in the flavor population is signaling features of the new outcome. Figure 4 Download asset Open asset Outcome selectivity of predictive neurons. (A) For each predictive neuron that significantly increased firing to the flavor odor (total n = 31; flavor-only n = 6 [green]; number and flavor n = 24 [purple]) we plotted its selective firing to the flavor odor (x-axis; [normalized flavor odor firing – mean(normalized initial odor firing, normalized blocked odor firing, normalized number odor firing)]) against its selective firing to the flavor outcome (y-axis; [normalized flavor outcome firing – mean(normalized initial outcome firing, normalized blocked outcome firing, normalized number outcome firing)]). Comparison of odor firing of single neurons to the population found a single outlier (neuron firing was 3 stdev > population firing). The outlier was omitted from this analysis. There was a significant, positive relationship such that greater selective firing to the flavor odor was associated with greater selective firing the flavor outcome (R2 = 0.41, p < 0.01). (B) This relationship was restricted to the flavor outcome; plotting selective flavor odor firing against selective number outcome firing revealed zero correlation (R2 = 0.01, p = 0.70; calculation identical to A only mean(blocked, initial and flavor outcome firing) was subtracted from number outcome firing. (C) For each predictive neuron that significantly increased firing to the number odor (total n = 37; number only n = 12 [blue]; number and flavor n = 23 [purple]) we plotted it's selective firing to the number odor (x-axis; [normalized number odor firing – mean(normalized initial odor firing, normalized blocked odor firing, normalized flavor odor firing)]) against its selective firing to the number outcome (y-axis; [normalized number outcome firing – mean(normalized initial outcome firing, normalized blocked outcome firing, normalized flavor outcome firing)]). Two neurons showed selective odor firing 3 stdev above the population mean and were excluded from analysis. There was a significant, positive relationship such that greater selective firing to the number odor was associated with greater selective firing the number outcome (R2 = 0.27, p < 0.01). (D) This relationship was restricted to the number outcome; plotting selective number odor firing against selective flavor outcome firing revealed zero correlation (R2 = 0.04, p = 0.26) calculation identical to C only mean(blocked, initial and number outcome firing) was subtracted from flavor outcome firing. Finally, these statistical patterns were maintained if the flavor-only and number-only neurons were analyzed in isolation: (A) R2 = 0.71, p = 0.03, (B) R2 = 0.01, p = 0.82, (C) R2 = 0.48, p = 0.01 and (D) R2 = 0.18, p = 0.17. Flav = flavor, Num = number, N&F = number and flavor. https://doi.org/10.7554/eLife.02653.012 Discussion Neural signals in the OFC are often described as representing either outcome expectancies or abstract value. Although many studies have argued for one or the other, few have used behavioral designs that clearly dissociate predictions of these two hypotheses. Here we tried to address this question by using an unblocking procedure to strip away or ‘block’ the abstract value of the outcome during learning, while leaving unblocked—free to enter into associations—the outcome's sensory and other unique features. This approach revealed two distinct populations of OFC neurons. One population consisted of neurons that fired immediately on initial presentation of all three target cues, perhaps reflecting these cues' novelty or salience. While unexpected and not directly relevant to the question motivating this study, this finding is consistent with reports of neural correlates of salience in OFC (Kahnt and Tobler, 2013; Ogawa et al., 2013) and with studies implicating the OFC in phenomena such as latent inhibition, set formation, and even auto-shaping (Chudasama et al., 2003; Schiller and Weiner, 2004; Chase et al., 2012) which depend in part on the appropriate attribution of salience to cues. Together these results point to a largely unappreciated role for this area in the modulation of attention for the purposes of learning (Esber et al., 2012). Of course novelty is just one instance of ‘salience’. Modern learning theories describe salience as a function of both intrinsic and acquired properties. Within this framework, this population was correlated with intrinsic salience. The second population, of more direct relevance to our hypothesis, consisted of neurons that fired preferentially to the target