Language models and information theoretical complexity metrics
- Date: May 2, 2019
- Time: 13:00 - 15:00
- Speaker: Dr John T. Hale
- Department of Linguistics, University of Georgia, Athens, USA
- Location: Max Planck Institute for Human Cognitive and Brain Sciences
- Room: Lecture Hall
- Host: IMPRS Coordination
- Contact: firstname.lastname@example.org
It is said that the mind/brain is "predictive." For the prime case of human cognition—language comprehension—, one way of using this idea leverages the concept of a 'language model' as developed in the field of natural language processing.
Via information-theoretical complexity metrics such as surprisal and entropy reduction, language models can link theoretical proposals about grammar and processing to observable neural signals.
This tutorial session teaches the basics of language models for those with no background in computational linguistics, emphasizing their utility in formalizing hypotheses regarding language processing in the brain. Of use beyond the language domain, the course serves to inspire and foster the use of quantitative, naturalistic quantifications of cognitive processing demands in all of cognitive neuroscience.
(1) People who have said that the mind is predictive
- Karl Friston. A theory of cortical responses. Philos Trans R Soc Lond B Biol Sci. 2005 Apr 29;360(1456):815-36.
- Andreja Bubic, Yves von Cramon, and Ricarda I. Schubotz. Prediction, cognition and the brain. Frontiers in human neuroscience, 4:25, 2010. doi:10.3389/fnhum.2010.00025
- Andy Clark. Surfing Uncertainty: Prediction, Action & the Embodied Mind. https://global.oup.com/academic/product/surfing-uncertainty-9780190217013
(2) People who have sought to apply this idea to Language
- John Hale. Information-theoretical Complexity Metrics. Language and Linguistics Compass, volume 10 issue 9 September 2016 pages 397—412. doi:10.1111/lnc3.12196
- Jonathan R. Brennan. Naturalistic sentence comprehension in the brain. Language and Linguistics Compass, volume 10 issue 7 July 2016 pages 299—313. doi:10.1111/lnc3.12198
- Kristijan Armeni, Roel M. Willems and Stefan L. Frank. Probabilistic language models in cognitive neuroscience: Promises and pitfalls. Neuroscience & Biobehavioral Reviews, Volume 83, 2017, Pages 579-588. doi:10.1016/j.neubiorev.2017.09.001
- Gina R. Kuperberg & T. Florian Jaeger (2016) What do we mean by prediction in language comprehension? Language, Cognition and Neuroscience, 31:1, 32-59. doi:10.1080/23273798.2015.1102299
- Ashley G. Lewis and Marcel Bastiaansen. A predictive coding framework for rapid neural dynamics during sentence-level language comprehension. Cortex, Volume 68, 2015, Pages 155-168. doi:10.1016/j.cortex.2015.02.014
(3) The idea of a language model as it figures in computational linguistics
- Easy intro geared towards linguistics: Coleman Introducing Speech and Language Processing chapter 7; http://www.phon.ox.ac.uk/jcoleman/SLP/outline.htm; https://www.cambridge.org/us/academic/subjects/languages-linguistics/computational-linguistics/introducing-speech-and-language-processing
- Standard book:Jurafsky and Martin 3rd edition chapter 3
- The new kid on the block:Jacob Eisenstein chaper 6