Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration
PLOS Computational Biology · 2024
DOI: 10.1371/journal.pcbi.1011516 ↗Kurz gefasst
Fachartikel in der Zeitschrift PLOS Computational Biology, 2024. Verfasst von Yuhao Wang und 3 weiteren Autor:innen. Beteiligte Einrichtung: MRC Brain Network Dynamics Unit, University of Oxford, Oxford, United Kingdom. Hauptthemen laut MEDLINE: Dopamine, Models, Neurological, Basal Ganglia, Reward. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 49 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Wellcome Trust, Biotechnology and Biological Sciences Research Council, Medical Research Council.
Abstract (Originalsprache)
When facing an unfamiliar environment, animals need to explore to gain new knowledge about which actions provide reward, but also put the newly acquired knowledge to use as quickly as possible. Optimal reinforcement learning strategies should therefore assess the uncertainties of these action–reward associations and utilise them to inform decision making. We propose a novel model whereby direct and indirect striatal pathways act together to estimate both the mean and variance of reward distributions, and mesolimbic dopaminergic neurons provide transient novelty signals, facilitating effective uncertainty-driven exploration. We utilised electrophysiological recording data to verify our model of the basal ganglia, and we fitted exploration strategies derived from the neural model to data from behavioural experiments. We also compared the performance of directed exploration strategies inspired by our basal ganglia model with other exploration algorithms including classic variants of upper confidence bound (UCB) strategy in simulation. The exploration strategies inspired by the basal ganglia model can achieve overall superior performance in simulation, and we found qualitatively similar results in fitting model to behavioural data compared with the fitting of more idealised normative models with less implementation level detail. Overall, our results suggest that transient dopamine levels in the basal ganglia that encode novelty could contribute to an uncertainty representation which efficiently drives exploration in reinforcement learning.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Substanzen
Autor:innen und Institutionen
- MRC Brain Network Dynamics Unit, University of Oxford, Oxford, United Kingdom.
- Department of Physiology, Anatomy and Genetics, University of Oxford, Oxford, United Kingdom.
- Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, United Kingdom.
ORCID-Profile: Yuhao Wang, Rafal Bogacz
Förderung
- Wellcome Trust
- Biotechnology and Biological Sciences Research Council
- Medical Research Council
- Royal Society
- National Institute for Healthcare Research
- James S. McDonnell Foundation
Zitieren
APA
Wang, Y., Lak, A., Manohar, S. G., & Bogacz, R. (2024). Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration. PLOS Computational Biology. https://doi.org/10.1371/journal.pcbi.1011516
Vancouver
Wang Y, Lak A, Manohar SG, Bogacz R. Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration. PLOS Computational Biology. 2024. doi:10.1371/journal.pcbi.1011516
BibTeX
@article{Wang2024,
author = {Wang, Yuhao and Lak, Armin and Manohar, Sanjay G and Bogacz, Rafal},
title = {Dopamine encoding of novelty facilitates efficient uncertainty-driven exploration},
journal = {PLOS Computational Biology},
year = {2024},
doi = {10.1371/journal.pcbi.1011516}
}
EUVIMED