Natural Language Processing

about this course

Machines have had some ability to understand and generate text for several decades. More recently, neural approaches have revolutionized the field. The most striking and impressive models are large Transformer-based models trained on large amounts of text.

But if "Attention is All You Need," then maybe we don't need those large language models at all. If "Language Models are Few-Shot Learners," then maybe small language models can learn from a few examples too. Maybe we can develop a small language model that better understands our language if we "pay attention" to the language.

Training a model to examine the links between words in a sequence and directly model those relationships trains the model to understand human language. Super-sizing a model super-sizes the cost. It does not super-size the understanding.

Larger models tend to perform better than smaller models, but performance gains diminish as model size increases. At large sizes, language models can become superficially fluent without truly understanding human language. Such "Stochastic Parrots" simply repeat long sequences that they learned from the training data.

More recent reasoning models partially overcome the "stochastic parrots" critique by exploring several solution paths and evaluating them against formal rules, code execution environments or math verifiers before responding to a prompt. Unlike their predecessors, which are (effectively) trained to ask: "what word most likely comes next in this sequence?", reasoning models trained with reinforcement learning ask: "which path successfully solves this logical puzzle?"

However, reasoning models are only a partial response to the "stochastic parrots" critique. Because they're trained to generate a coherent "chain of thought," they can generate a fluent-looking logic that's based on a fundamentally flawed premise.

what you will learn

This course will compare the performance of Transformer-based models to previous approaches. And it will pay particular attention to how those performance gains were achieved. Did the researchers develop a better model? Or did they train a larger model?

For example, the fluency and translation quality of neural translation models far surpasses that of phrase-based statistical models. And in low-resource cases too.

But what's important is how those performance gains were achieved. Instead of translating words or phrases, the neural approach attempts to understand context. Neural models translate better than phrase-based models because they attempt to create a sentence in the target language with the same meaning as the source language sentence.

In that spirit, this course will explore neural approaches to natural language processing. Comparing them, it will ask how we can develop models that better understand our language.

links and files

textbook

course outline

  1. context and background
  2. tools for our "NLP kitchen"
  3. word embeddings
  4. subword segmentation
  5. recurrent neural networks
  6. the Transformer
  1. multilingual translation
  2. GPT models
  3. BERT models
  4. fine-tuning and distillation
  5. "Stochastic Parrots"
  6. reasoning models

course outline

  • topic 00 — context and background
    • themes:
      • Before neural networks were used in NLP, count-based methods like term-frequency, inverse document frequency (TF-IDF) provided classifiers. And phrase-based methods provided statistical machine translation.
      • More recently, deep learning has given NLP simple, but sophisticated models to understand and generate human language. And at large scale, the new models provide impressive, but superficial fluency.
      • Developing such models, we develop the potential to learn a new language or create new text in a language. And we also develop the potential to generate environmental pollution and texts full of incorrect information.
    • readings:
  • topic 01 — tools for our "NLP kitchen"
    • theme:
      • In linear regression, the confidence interval around the prediction is smaller when the predictor variables lie close to the sample means in the dataset used to estimate the regression model.
      • Similarly, language models make better predictions when the subword distribution at inference matches the subword distribution at training. Incorporating linguistic theory into the pre-processing aligns those distributions.
      • In linear regression, we use supervised learning to minimize a loss function.
      • Similarly, we use supervised learning to train a translation model. After comparing a batch of predicted translations to their reference translations, we use gradient descent to adjust the model parameters in the direction that most reduces the errors.
      • Dissimilarly, we use unsupervised learning to train a causal language model. This method leverages the abundance of raw text (relative to labelled text) and pre-trains a general purpose task-agnostic model that can later be fine-tuned for specific tasks.
    • tools and methods we can use:
      • dictionaries and grammar books
      • regular expressions, lemmatizers, part of speech taggers and dependency parsers
      • supervised learning, unsupervised learning, reinforcement learning
    • readings:
  • topic 02 — word embeddings
  • topic 03 — subword segmentation
    • theme:
      • Newer segmentation methods provide language-independent tokenization, detokenization and segmentation from raw sentences, whereas previous methods assumed prior tokenization.
      • This choice is important because once a model has been trained, the subword vocabulary is "locked in." It's also important because some languages do not divide sentences into words.
      • And it's important because a language model predicts a sequence of subword units. Incorporating theory into the subword-splitting trains the model to predict a theoretic sequence of subword units.
    • readings:
  • topic 04 — recurrent neural networks
    • themes:
      • RNNs were the first neural model used in machine translation. Reading words sequentially, they employ gating mechanisms to identify relationships between separated words in a sequence. The gating mechanisms use word embeddings to understand context.
      • The output that RNNs produced was far more fluent than that of phrase-based models. Nonetheless, phrase-based models continued to outperform RNNs in low-resource cases. Only when trained on very large datasets, RNN models outperformed phrase-based models.
      • RNNs fell out of favor because recurrent processing requires long training times. And the gating mechanisms only partially solved the vanishing gradient problem, which often made RNNs difficult to train.
    • readings:
  • topic 05 — the Transformer
    • themes:
      • The Transformer quickly became the neural model of choice for NLP tasks.
      • Unlike RNNs, Transformers do not require recurrent processing of a hidden state. Instead, they encode and decode using only self-attention, performing computations in parallel, which reduces training costs.
      • Self-attention directly models the relationships between words in a sequence as it examines the links between them. So when training a Transformer, computations are spent modeling the language (not encoding/decoding a hidden state vector).
      • It's a simpler approach that scales upward to larger datasets. And it also scales downward to smaller datasets, enabling us to develop useful, meaningful models in low-resource cases.
    • readings:
  • topic 07 — GPT models
    • themes:
      • Using a decoder-only Transformer, researchers at OpenAI observed that pre-training a language model in an unsupervised fashion and then fine-tuning it for a given task is an effective strategy.
      • In their 2018 paper introducing GPT, they hypothesized that "the more structured attentional memory of the transformer assists in transfer compared to LSTMs."
      • Then in 2019 and 2020, they trained models at a range of sizes and observed that a model's ability to transfer learning across tasks increases with model size. Their largest model performed best on all tests. And sometimes it performed much better than the smaller models. But in many cases, the improvement was small.
    • readings:
  • topic 08 — BERT models
    • themes:
      • Using an encoder-only Transformer, researchers at Google took a different approach to pre-training and unsupervised learning. They developed a "bidirectional" model, BERT, which allows each token to attend to all tokens in the self-attention layers, so that the representation learns from context on both sides.
      • For comparison, GPT (like the original Transformer) only allows a token to attend to previous tokens.
      • So to prevent trivial predictions, the team that developed BERT introduced masked language modeling. During pre-training, they randomly replaced a fraction of the tokens with either a "mask" token, a random token or the same (unchanged) token, then they pre-trained BERT to predict those tokens.
      • And to capture the relationship between two sentences, they created paired training examples in which the second sentence was either the next sentence or a random sentence.
      • The bidirectional cross-attention between the two sentences makes BERT a good choice for question-answering tasks, inference tasks and classification tasks.
    • readings:
  • topic 09 — fine-tuning and distillation
    • themes:
      • Pre-trained language models provide a general purpose foundation. With a "few shots" we can fine-tune them for specific tasks.
      • Originally, fine-tuning consisted of further training the original model, updating all of its parameters. But as model sizes grew to billions of parameters, updating all parameters became expensive and infeasible on consumer hardware.
      • Parameter efficient fine-tuning methods (like low-rank adaptation) allow us to only train a small set of adapter parameters. So instead of retraining the whole base model for each specifi task, we can (more easily) train different LoRA adapters for different tasks.
      • To keep the cost of inference low, we often need a small model that can run with minimal computational resources, but we also wish to retain the performance of a large model.
      • For such cases, knowledge distillation techniques allow us to use a large "teacher" model to train a small "student" model" to replicate its behavior. Instead than training the student model from scratch on raw data, the student model learns from the teacher's outputs.
    • readings:
  • topic 10 — "Stochastic Parrots"
    • themes:
      • Large language models impressively generate coherent, fluent text by predicting a sequence of subword units. But what model size is optimal? And what do language models truly understand?
      • Large models perform better, but they also cost more to train and deploy. So given the diminishing marginal returns to model capacity, the profit-maximizing model size may be quite small.
      • Training and fine-tuning large language models consumes energy. And often times that energy does not come from renewable sources.
      • Training models on mostly-English datasets leaves other languages poorly served. And even within English, the data may not reflect the way the language has changed and is changing in response to changing social views, changing opinions.
      • But with careful thought and planning, we can develop language models that understand the language that we speak.
    • readings:
  • topic 11 — reasoning models

textbooks

Copyright © 2002-2026 Eryk Wdowiak