EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration

Yuhao Wang iD, Armin Lak, Sanjay G Manohar, Rafal Bogacz iD

PLOS Computational Biology · 2024

DOI: 10.1371/journal.pcbi.1011516 ↗

Kurz gefasst

Fachartikel in der Zeitschrift PLOS Computational Biology, 2024. Verfasst von Yuhao Wang und 3 weiteren Autor:innen. Beteiligte Einrichtung: MRC Brain Network Dynamics Unit, University of Oxford, Oxford, United Kingdom. Hauptthemen laut MEDLINE: Dopamine, Models, Neurological, Basal Ganglia, Reward. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Wellcome Trust, Biotechnology and Biological Sciences Research Council, Medical Research Council.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action–reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

DopamineModels, NeurologicalBasal GangliaRewardUncertaintyAnimalsExploratory BehaviorReinforcement, PsychologyDopaminergic NeuronsComputational BiologyComputer SimulationMaleAlgorithmsDecision MakingBehavior, AnimalRats

Substanzen

Dopamine

Autor:innen und Institutionen

  • MRC Brain Network Dynamics Unit, University of Oxford, Oxford, United Kingdom.
  • Department of Physiology, Anatomy and Genetics, University of Oxford, Oxford, United Kingdom.
  • Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, United Kingdom.

ORCID-Profile: Yuhao Wang, Rafal Bogacz

Förderung

  • Wellcome Trust
  • Biotechnology and Biological Sciences Research Council
  • Medical Research Council
  • Royal Society
  • National Institute for Healthcare Research
  • James S. McDonnell Foundation

Zitieren

APA

Wang, Y., Lak, A., Manohar, S. G., & Bogacz, R. (2024). Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1011516

Vancouver

Wang Y, Lak A, Manohar SG, Bogacz R. Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration. PLOS Computational Biology. 2024. doi:10.1371/journal.pcbi.1011516

BibTeX
@article{Wang2024,
  author = {Wang, Yuhao and Lak, Armin and Manohar, Sanjay G and Bogacz, Rafal},
  title = {Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration},
  journal = {PLOS Computational Biology},
  year = {2024},
  doi = {10.1371/journal.pcbi.1011516}
}
Kostenlos starten →