Abstract
Reward outcomes are available in many diverse situations and all involve choice. If there are multiple outcomes each rewarding, then decisions regarding relative value lead to choosing one over another. Important factors related to choice context should be encoded and utilized for this form of adaptive choosing. These factors can include the number of alternatives, the pacing of choice behavior and the possibility to reverse one’s choice. An essential step in understanding if the context of choice is encoded is to directly compare choice with a context in which choice is absent. Neural activity in orbitofrontal cortex and striatum encodes potential value parameters related to reward quality and quantity as well as relative preference. We examined how neural activations in these brain regions are sensitive to choice situations and potentially involved in a prediction for the upcoming outcome selection. Neural activity was recorded and compared between a two-choice spatial delayed response task and an imperative ‘one-option’ task. Neural activity was obtained that extended from the instruction cue to the movement similar to previous work utilizing the identical imperative task. Orbitofrontal and striatal neural responses depended upon the decision about the choice of which reward to collect. Moreover, signals to predictive instruction cues that precede choice were selective for the choice situation. These neural responses could reflect chosen value with greater information on relative value of individual options as well as encode choice context itself embedded in the task as a part of the post-decision variable.
References
Apicella P, Ljungberg T, Scarnati E, Schultz W (1991) Responses to reward in monkey dorsal and ventral striatum. Exp Brain Res 85(3):491–500
Belguermmi A, Bovet D, Pascal A, Prévot-Julliard A, Jalme M, Rat-Fischer L, Leboucher G (2011) Pigeons discriminate between human feeders. Anim Cogn 14(6):909–914. https://doi.org/10.1007/s10071-011-0420-7
Berridge KC, Cromwell HC (1990) Motivational-sensorimotor interaction controls aphagia and exaggerated treading after striatopallidal lesions. Behav Neurosci 104(5):778–795
Berridge KC, Kringelbach ML (2013) Neuroscience of affect: brain mechanisms of pleasure and displeasure. Curr Opin Neurobiol 23(3):294–303
Blanchard TC, Hayden BY (2015) Monkeys are more patient in a foraging task than in a standard intertemporal choice task. PLoS One (United States) 10:e0117057
Bouton ME, Todd TP, Miles OW, Leon SP, Epstein LH (2013) Within- and between-session variety effects in a food-seeking habituation paradigm. Appetite 66:10–19
Braun S, Hauber W (2011) The dorsomedial striatum mediates flexible choice behavior in spatial tasks. Behav Brain Res (Netherlands) 220:288–293
Brown VJ, Robbins TW (1991) Simple and choice reaction time performance following unilateral striatal dopamine depletion in the rat. Impaired motor readiness but preserved response preparation. Brain 114(Pt 1B):513–525
Cai X, Kim S, Lee D (2011) Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice. Neuron 69:170–182
Cardinal RN, Howes NJ (2005) Effects of lesions of the nucleus accumbens core on choice between small certain rewards and large uncertain rewards in rats. BMC Neurosci 6:37
Catania AC (1963) Concurrent performances: reinforcement interaction and response independence. J Exp Anal Beh 6:253–263
Catania AC, Savgolden T (1980) Preference for free choice over forced choice in pigeons. J Exp Anal Behav 34(1):77–86. https://doi.org/10.1901/jeab.1980.34-77
Cromwell HC, Schultz W (2003) Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. J Neurophysiol 89(5):2823–2838
Cromwell HC, Hassani OK, Schultz W (2005) Relative reward processing in primate striatum. Exp Brain Res 162(4):520–525
Czoty PW, McCabe C, Nader MA (2005) Effects of the 5-HT(1A) agonist (+/-)-8-hydroxy-2-(di-n-propylamino)tetralin (8-OH-DPAT) on cocaine choice in cynomolgus monkeys. Behav Pharmacol (England) 16:187–191
Farashahi S, Donahue CH, Khorsand P, Seo H, Lee D, Soltani A (2017) Metaplasticity as a neural substrate for adaptive learning and choice under uncertainty. Neuron 94(2):401–414
Fisher WW, Thompson RH, Piazza CC, Crosland K, Gotjen D (1997) On the relative reinforcing effects of choice and differential consequences. J Appl Behav Anal 30(3):423–438. https://doi.org/10.1901/jaba.1997.30-423
Freidin E, Kacelnik A (2011) Rational choice, context dependence, and the value of information in European starlings (Sturnus vulgaris). Science 334(6058):1000–1002
Gourley SL, Olevska A, Zimmermann KS, Ressler KJ, Dileone RJ, Taylor JR (2013) The orbitofrontal cortex regulates outcome-based decision-making via the lateral striatum. Eur J Neurosci (France) 38:2382–2388
Grabenhorst F, Hernadi I, Schultz W (2016) Primate amygdala neurons evaluate the progress of self-defined economic choice sequences. Elife 5:e18731. https://doi.org/10.7554/eLife.18731
Hassani OK, Cromwell HC, Schultz W (2001) Influence of the expectation for different rewards onbehavior-related neuronal activity in primate striatum. J Neurophysiol 85:2477–2489
Hernádi I, Grabenhorst F, Schultz W (2015) Planning activity for internally generated reward goals in monkey amygdala neurons. Nat Neurosci (United States) 18:461–469
Hwang J, Kim S, Lee D (2009) Temporal discounting and inter-temporal choice in rhesus monkeys. Front Behav Neurosci (Switzerland) 3:9
Iyengar SS, Lepper MR (2000) When choice is demotivating: can one desire too much of a good thing? J Pers Soc Psychol 79(6):995–1006
Kacelnik A, Bateson M (1996) Risky theories—the effects of variance on foraging decisions. Integr Comp Biol 36:402–434
Kalivas PW, Volkow ND (2005) The neural basis of addiction: a pathology of motivation and choice. Am J Psychiatry (United States) 162:1403–1413
Keiflin R, Reese RM, Woods CA, Janak PH (2013) The orbitofrontal cortex as part of a hierarchical neural system mediating choice between two good options. J Neurosci (United States) 33:15989–15998
Kheramin S, Body S, Mobini S, Ho MY, Velazquez-Martinez DN, Bradshaw CM et al (2002) Effects of quinolinic acid-induced lesions of the orbital prefrontal cortex on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl) 165(1):9–17
Kheramin S, Body S, Ho M, Velazquez-Martinez DN, Bradshaw CM, Szabadi E et al (2003) Role of the orbital prefrontal cortex in choice between delayed and uncertain reinforcers: a quantitative analysis. Behav Process 64(3):239–250
Kheramin S, Body S, Herrera FM, Bradshaw CM, Szabadi E, Deakin JF et al (2005) The effect of orbital prefrontal cortex lesions on performance on a progressive ratio schedule: implications for models of inter-temporal choice. Behav Brain Res 156(1):145–152
Kim H, Sul JH, Huh N, Lee D, Jung MW (2009) Role of striatum in updating values of chosen actions. J Neurosci (United States) 29:14701–14712
Klapp S, Abbott J, Coffman K, Greim D, Snider R, Young F (1979) Simple and choice reaction time methods in the study of motor programming. J Mot Behav 11(2):91–101
Lak A, Stauffer WR, Schultz W (2016) Dopamine neurons learn relative chosen value from probabilistic rewards. Elife (England). https://doi.org/10.7554/eLife.18044
Lau B, Glimcher PW (2008) Value representations in the primate striatum during matching behavior. Neuron 58(3):451–463
Leung BK, Balleine BW (2013) The ventral striato-pallidal pathway mediates the effect of predictive learning on choice between goal-directed actions. J Neurosci (United States) 33:13848–13860
Louie K, Glimcher PW (2012) Efficient coding and the neural representation of value. Ann N Y Acad Sci 1251:13–32. https://doi.org/10.1111/j.1749-6632.2012.06496.x
Marcos E, Genovesio A (2016) Determining monkey free choice long before the choice is made: the principal role of prefrontal neurons involved in both decision and motor processes. Front Neural Circuits (Switzerland) 10:75
McGraw JJ, Zona LC, Cromwell HC (2017) The effects of ethanol on diverse components of choice in the rat: reward discrimination, preference and relative valuation. Eur J Neurosci (France) 46(3):1837–1849. https://doi.org/10.1111/ejn.13627
Melville CL, Rue HC, Rybiski LR, Weatherly JN (1997) Altering reinforcer variety or intensity changes the within-session decrease in responding. Learn Motiv 28(4):609–621
Nader MA, Woolverton WL (1992a) Choice between cocaine and food by rhesus monkeys: effects of conditions of food availability. Behav Pharmacol 3(6):635–638
Nader MA, Woolverton WL (1992b) Effects of increasing response requirement on choice between cocaine and food in rhesus monkeys. Psychopharmacology (Berl) 108(3):295–300
O’Neill M, Schultz W (2010) Coding of reward risk by orbitofrontal neurons is mostly distinct from coding of reward value. Neuron 68(4):789–800
Padoa-Schioppa C, Assad JA (2008) The representation of economic value in the orbitofrontal cortex is invariant for changes of menu. Nat Neurosci 11(1):95–102
Palmer CR, Kristan WB Jr (2011) Contextual modulation of behavioral choice. Curr Opin Neurobiol (England) 21:520–526
Pastor-Bernier A, Plott CR, Schultz W (2017) Monkeys choose as if maximizing utility compatible with basic principles of revealed preference theory. Proc Natl Acad Sci USA (United States) 114:E1766-E1775
Rainwater A, Sanz E, Palmiter RD, Quintana A (2017) Striatal GPR88 modulates foraging efficiency. J Neurosci (United States) 37:7939–7947
Ramirez-Lugo L, Penas-Rincon A, Angeles-Duran S, Sotres-Bayon F (2016) Choice behavior guided by learned, but not innate, taste aversion recruits the orbitofrontal cortex. J Neurosci (United States) 36:10574–10583
Riceberg JS, Shapiro ML (2017) Orbitofrontal cortex signals expected outcomes with predictive codes when stable contingencies promote the integration of reward history. J Neurosci (United States) 37:2010–2021
Rich EL, Wallis JD (2016) Decoding subjective decisions from orbitofrontal cortex. Nat Neurosci (United States) 19:973–980
Ricker JM, Kopchock RJ 3rd, Drown RM, Cromwell HC (2016a) Effects of striatal lesions on components of choice: Reward discrimination, preference, and relative valuation. Behav Brain Res (Netherlands) 315:130–140
Ricker JM, Hatch JD, Powers DD, Cromwell HC (2016b) Fractionating choice: a study on reward discrimination, preference, and relative valuation in the rat (rattus norvegicus). J Comp Psychol (United States) 130:174–186
Robbins TW (2002) The 5-choice serial reaction time task: behavioural pharmacology and functional neurochemistry. Psychopharmacol (Berl) 163(3–4):362–380
Rolls BJ, Rowe EA, Rolls ET, Kingston B, Megson A, Gunary R (1981) Variety in a meal enhances food intake in man. Physiol Behav 26(2):215–221
Schultz W (2004) Neural coding of basic reward terms of animal learning theory, game theory, microeconomics and behavioural ecology. Curr Opin Neurobiol 14(2):139–147
Schultz W (2010) Subjective neuronal coding of reward: temporal value discounting and risk. Eur J Neurosci 31(12):2124–2135
Schultz W (2011) Potential vulnerabilities of neuronal reward, risk, and decision mechanisms to addictive drugs. Neuron 69(4):603–617
Schultz W (2015) Neuronal reward and decision signals: From theories to data. Physiol Rev (United States) 95:853–951
Schultz W (2017) Reward prediction error. Curr Biol 27(10):R369–R371. https://doi.org/10.1016/j.cub.2017.02.064
Seymour B, McClure SM (2008) Anchors, scales and the relative coding of value in the brain. Curr Opin Neurobiol (England) 18:173–178
Shizgal P (1997) Neural basis of utility estimation. Curr Opin Neurobiol 7(2):198–208
St Onge JR, Floresco SB (2010) Prefrontal cortical contribution to risk-based decision making. Cereb Cortex 20(8):1816–1828
Stauffer WR, Lak A, Schultz W (2014) Dopamine reward prediction error responses reflect marginal utility. Curr Biol (England) 24:2491–2500
Sternberg S, Monsell S, Knoll RL, Wright CE (1978) The latency and duration of rapid movement sequences: comparisons of speech and typing. In: Stelmach GE (ed) Information processing in motor control and learning. Academic Press, New York, pp 117–152
Strait CE, Sleezer BJ, Hayden BY (2015) Signatures of value comparison in ventral striatum neurons. PLoS Biol (United States) 13:e1002173
Strait CE, Sleezer BJ, Blanchard TC, Azab H, Castagno MD, Hayden BY (2016) Neuronal selectivity for spatial positions of offers and choices in five reward regions. J Neurophysiol 115(3):1098–1111. https://doi.org/10.1152/jn.00325.2015
Suzuki S (1999) Selection of forced- and free-choice by monkeys (macaca fascicularis). Percept Mot Skills 88:242–250. https://doi.org/10.2466/pms.1999.88.1.242
Treit D, Spetch ML, Deutsch JA (1983) Variety in the flavor of food enhances eating in the rat: a controlled demonstration. Physiol Behav 30(2):207–211
Tremblay L, Schultz W (1999) Relative reward preference in primate orbitofrontal cortex. Nature (England) 398:704–708
Tsujimoto S, Genovesio A, Wise SP (2009) Monkey orbitofrontal cortex encodes response choices near feedback time. J Neurosci 29(8):2569–2574. https://doi.org/10.1523/JNEUROSCI.5777-08.2009
Tsutsui K, Grabenhorst F, Kobayashi S, Schultz W (2016) A dynamic code for economic object valuation in prefrontal cortex neurons. Nat Commun (England) 7:12554
Tversky A, Shafir E (1992) Choice under conflict: the dynamics of deferred decision. Psychol Sci 3:358–361
van Wingerden M, van der Meij R, Kalenscher T, Maris E, Pennartz CM (2014) Phase-amplitude coupling in rat orbitofrontal cortex discriminates between correct and incorrect decisions during associative learning. J Neurosci (United States) 34:493–505
Watanabe M, Cromwell HC, Tremblay L, Hollerman JR, Hikosaka K, Schultz W (2001) Behavioral reactions reflecting differential reward expectations in monkeys. Exp Brain Res 140(4):511–518
Webber ES, Chambers NE, Kostek JA, Mankin DE, Cromwell HC (2015) Relative reward effects on operant behavior: Incentive contrast, induction and variety effects. Behav Process (Netherlands) 116:87–99
Webber ES, Mankin DE, Cromwell HC (2016) Striatal activity and reward relativity: neural signals encoding dynamic outcome valuation. Eneuro 3:ENEURO.0022-16.2016
Williams BA (2002) Behavioral contrast redux. Animal Learning Behavior 30:1–20
Yang X, Gao M, Shi J, Ye H, Chen S (2017) Modulating the activity of the DLPFC and OFC has distinct effects on risk and ambiguity decision-making: a tDCS study. Front Psychol 8:1417. https://doi.org/10.3389/fpsyg.2017.01417
Zona LC, Fry BR, Cromwell HC (2017) Effects of anandamide administration on components of reward processing during free choice. Pharmacol Biochem Behav 158:14–21
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cromwell, H.C., Tremblay, L. & Schultz, W. Neural encoding of choice during a delayed response task in primate striatum and orbitofrontal cortex. Exp Brain Res 236, 1679–1688 (2018). https://doi.org/10.1007/s00221-018-5253-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-018-5253-z