Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference

Paper · Note reference · Code · EACL 2021

Abstract

This paper introduces Pattern-Exploiting Training (PET), a semi-supervised training procedure that reformulates input examples as cloze-style phrases to help language models understand a given task.

These phrases are then used to assign soft labels to a large set of unlabeled examples. Finally, standard supervised training is performed on the resulting training set.

PET outperforms supervised training and strong semi-supervised approaches in low-resource settings by a large margin.

Introduction

Before this paper, applying standard supervised learning to small training sets often performed poorly.

For instance, assume the model is given the following pieces of text:

• $T_1$: This was the best pizza I’ve ever had.
• $T_2$: You can get better sushi for half the price.

And the labels of $T_1$ and $T_2$ are $l$ and $l'$.

If the model is asked to infer the correct label for $T_3$:

• $T_3$: Pizza was average. Not worth the price.

The task becomes easier if the model knows that the underlying task is to identify whether the text says anything about prices.

---

This work shows that providing task descriptions can be successfully combined with standard supervised learning in few-shot settings. It is called Pattern-Exploiting Training (PET), a semi-supervised training procedure that uses natural language patterns to reformulate input examples into cloze-style phrases.

PET works in three steps:

1. For each pattern, a separate PLM is fine-tuned on a small training set $mathcal{T}$.
2. The ensemble of all models annotates a large unlabeled dataset $mathcal{D}$ with soft labels.
3. A standard classifier is trained on the soft-labeled dataset.

The authors also devise iPET, an iterative variant of PET in which this process is repeated with increasing training-set sizes.

Related Work

• Zero-shot learning of challenging tasks such as question answering and reading comprehension, which requires a semantic parser.
• Cloze-style phrases to probe the knowledge that PLMs acquire during pretraining.
• Few-shot learning in NLP, including exploiting examples from related tasks and data augmentation.

The idea behind iPET is to train multiple generations of models on data labeled by previous generations.

Pattern-Exploiting Training

• $M$: masked language model with vocabulary $V$.
• A mask token $\_\_\_\_ \in V$ (also written as $[\mathrm{MASK}]$).
• $mathcal{L}$: a set of labels for target classification task $A$.

An input for task $A$ is a sequence of phrases $\mathbf{x} = (s_1, \ldots, s_k)$, with $s_i \in V^*$.

For example, $k = 2$ if $A$ is textual inference with two input sentences.

They define a pattern as a function $P$ that takes $\mathbf{x}$ as input and outputs a phrase or sentence $P(\mathbf{x}) \in V^*$ that contains exactly one mask token. Its output can be viewed as a cloze question.

Furthermore, they define a verbalizer as an injective function $v : \mathcal{L} \rightarrow V$ that maps each label to a word from $M$‘s vocabulary.

The pair $(P, v)$ is a pattern-verbalizer pair (PVP).

• $P$ creates a $[\mathrm{MASK}]$ expression from $\mathbf{x}$.
• $v$ defines the relation $mathcal{L} \rightarrow V$; for example, positive $\rightarrow$ great.

How PET solves task $A$ using a PVP $(P, v)$

Given an input $\mathbf{x}$, apply $P$ to obtain $P(\mathbf{x})$. Then use $M$ to determine the label $y \in \mathcal{L}$ for which $v(y)$ is the most likely substitute for the mask.

PVP Training and Inference

Given an input $\mathbf{x}$, define the score for label $l \in \mathcal{L}$ as:

$$s_{\mathbf{p}}(l \mid \mathbf{x}) = M(v(l) \mid P(\mathbf{x}))$$

Obtain a probability distribution over labels using softmax:

$$q_{\mathbf{p}}(l \mid \mathbf{x}) = \frac{e^{s_{\mathbf{p}}(l \mid \mathbf{x})}} {\sum_{l^{\prime} \in \mathcal{L}} e^{s_{\mathbf{p}}(l^{\prime} \mid \mathbf{x})}}$$

Auxiliary Language Modeling

The paper faces the problem that few training examples are available and catastrophic forgetting can occur. Therefore, it uses language modeling as an auxiliary task.

The final loss combines cross-entropy $L_{CE}$ and language-modeling loss $L_{\mathrm{MLM}}$, where $\alpha = 10^{-4}$:

$$L = (1 - \alpha) \cdot L_{CE} + \alpha \cdot L_{\mathrm{MLM}}$$

Combining PVPs

To handle differences in PVP performance, the authors use a strategy similar to knowledge distillation:

1. Define a set $mathcal{P}$ of PVPs that intuitively make sense for a given task $A$.

2. Fine-tune a separate language model $M_{\mathbf{p}}$ for each $\mathbf{p} \in \mathcal{P}$.

3. Use the ensemble $\mathcal{M} = \{M_{\mathbf{p}} \mid \mathbf{p} \in \mathcal{P}\}$ to annotate examples from $mathcal{D}$. The unnormalized class scores for each example $\mathbf{x} \in \mathcal{D}$ are combined as:

   $$s_{\mathcal{M}}(l \mid \mathbf{x}) = \frac{1}{Z} \sum_{\mathbf{p} \in \mathcal{P}} w(\mathbf{p}) \cdot s_{\mathbf{p}}(l \mid \mathbf{x})$$

   where $Z = \sum_{\mathbf{p} \in \mathcal{P}} w(\mathbf{p})$. The weighting term is either:

   • $w(\mathbf{p}) = 1$ for all $\mathbf{p}$; or
   • $w(\mathbf{p})$ is the accuracy obtained using $\mathbf{p}$ on the training set before training.

4. Fine-tune a PLM $C$ with a standard sequence-classification head on $mathcal{T}_C$.

Iterative PET (iPET)

iPET trains several generations of models on datasets of increasing size. This avoids the issue that some patterns may perform much worse than others, which could otherwise cause the final model’s training set $mathcal{T}_C$ to contain many mislabeled examples.

1. Enlarge the original dataset $mathcal{T}$ by labeling selected examples from $mathcal{D}$ using a random subset of trained PET models.
2. Train a new generation of PET models on the enlarged dataset repeatedly.

• $mathcal{M}^0 = \{M_1^0, \ldots, M_n^0\}$ is the initial set of PET models fine-tuned on $mathcal{T}$.
• Each $M_i^j$ is trained for PVP $\mathbf{p}_i$ on its own training set $mathcal{T}_i^j$.
• In each iteration, the training-set size is multiplied by a fixed constant $d \in \mathbb{N}$.

To maintain the label ratio of the original dataset:

1. Obtain $mathcal{N} \subset \mathcal{M}^{j-1} \setminus \{M_i^{j-1}\}$ by randomly choosing $\lambda \cdot (n - 1)$ models from the previous generation, with $\lambda \in (0, 1]$.

2. Use the subset to create a labeled dataset:

   $$\mathcal{T}_{\mathcal{N}} = \left\{ \left(\mathbf{x}, \arg\max_{l \in \mathcal{L}} s_{\mathcal{N}}(l \mid \mathbf{x})\right) \mid \mathbf{x} \in \mathcal{D} \right\}$$

   To avoid training future generations on mislabeled data, choose examples for which the ensemble is confident in its prediction. When drawing from $\mathcal{T}_{\mathcal{N}}$, set the probability of each $(\mathbf{x}, y)$ proportional to $s_{\mathcal{N}}(l \mid \mathbf{x})$.

3. Combine the new dataset as:

   $$\mathcal{T}_i^j = \mathcal{T} \cup \bigcup_{l \in \mathcal{L}} \mathcal{T}_{\mathcal{N}}(l)$$

4. After training $k$ generations of PET models, use $mathcal{M}^k$ to create $mathcal{T}_C$ and train $C$ as in basic PET.

Experiments

The paper evaluates four English datasets: Yelp Reviews, AG’s News, Yahoo Questions, and MNLI. It uses x-stance to investigate how well PET works for other languages.

RoBERTa large is used as the language model; for x-stance, XLM-R is used.

Sentiment Analysis Type Tasks

• Yelp

Thematic Classification Tasks

• AGNews / Yahoo

Sentence Pair Type Task

• MNLI

X-Stance

Multilingual stance-detection dataset with German, French, and Italian examples.

Analysis

Combining PVPs

Auxiliary Language Modeling

Iterative PET

In-Domain Pretraining

Conclusion

PET consists of defining pairs of cloze-question patterns and verbalizers that help leverage the knowledge contained within pretrained language models for downstream tasks.

When the initial amount of training data is limited, PET gives large improvements over standard supervised training and strong semi-supervised approaches.