LingLunch: Tal Linzen (ENS)

Dr. Tal Linzen (ENS) will be presenting an ad hoc LingLunch talk.

Using English subject-verb number agreement to evaluate the syntactic capabilities of contemporary neural networks

Recent technological advances have made it possible to train artificial neural networks more effectively and on a much larger scale than ever before. These networks have quickly become the most effective techniques on the market in natural language processing tasks such as machine translation and language modeling for speech recognition. These engineering advances are surprising from a linguistic perspective, since many of these highly successful systems have little by way of built-in linguistic representations: they are based on recurrent neural networks, which are sequence models without pre-engineered hierarchical syntactic representations. The success of these systems in learning syntax from a corpus could have profound cognitive implications: Is the computational problem of syntax acquisition easier than we thought? Do neural networks have the potential to serve as models of human linguistic competence? Before answering these questions in the affirmative, however, we need to better understand the syntactic capabilities of the networks: statistics such as average word error rate on a random sample of sentences may be sufficient for engineering purposes, but careful analysis using tools from psycholinguistics and theoretical linguistics is necessary if we are to draw any theoretical conclusions.

In this talk, I will report on the results of an ongoing analysis of the ability of contemporary recurrent neural networks to master English subject-verb number agreement, one of the numerous phenomena that serve as evidence for hierarchical syntactic structure (joint work with Emmanuel Dupoux and Yoav Goldberg). We probed the syntactic capabilities of the architecture using training objectives with explicit grammatical targets (verb number prediction and acceptability judgments) as well as using a word prediction task with no grammatical supervision ("language modeling"). The explicitly trained verb number prediction model achieved an impressive error rate of less than 1%; an inspection of the system's dynamics suggests that some of its units can detect syntactic structure, at least when there are overt cues to it (function words). Closer examination revealed that error rates increased in the sentences where structural and linear factors conflicted; in difficult cases, the model's performance approached chance levels. Models that were trained without negative evidence performed much worse, approaching a 100% error rate on challenging sentences. These results suggest that contemporary recurrent neural networks can learn to approximate certain aspects of syntactic structure, but only given explicit grammatical supervision; stronger inductive biases may be necessary to acquire syntax from a natural input.