Re-Implementation: Universal Sentence Encoder (Cer et al., 2018)

Advanced NLP — Group Project (Task 4)

Paper: Daniel Cer et al., Universal Sentence Encoder, arXiv:1803.11175v2, Google Research, 2018.

---

1. What this project does

The paper introduces two sentence encoders that turn an English sentence into a single 512-dimensional embedding vector. The idea is to train the encoder once on a large amount of text, then transfer it to downstream NLP tasks — and the main claim is that this transfer helps most when you only have a little labelled data for the downstream task.

We re-implemented both encoders from scratch in Python + PyTorch and reproduced the paper's transfer-learning experiment at a scale that runs on a normal laptop CPU. Everything below is our own code; nothing is imported from a pre-built Universal Sentence Encoder library.

Paper element	Where it lives in our code
Section 3.1 — Transformer encoder (hand-built attention, sum ÷ √len pooling)	transformer_encoder.py
Section 3.2 — DAN encoder (word + bigram averaging, feed-forward network)	dan_encoder.py
Eq. 1 — angular similarity	common.py → angularSim()
Table 1 data (SST: dev 872 / test 1,821)	data/sst2/
Unsupervised pre-training idea	train_transfer.py, Phase 1
Table 3 — accuracy vs. amount of training data	train_transfer.py, Phase 2 → results.txt
Figure 1 — similarity heatmap	similarity_demo.py → heatmap.png
Listing 1 — encode a sentence in a few lines	demo block at the bottom of each encoder file

---

2. Repository architecture

NLP-Project/
│
├── README.md                  ← you are here
├── requirements.txt           ← Python libraries (torch, numpy, matplotlib)
│
├── common.py                  ← shared helpers: tokenizer, vocabulary builder,
│                                 SST data loader, and Eq. 1 angular similarity
│
├── transformer_encoder.py     ← USE_T : Transformer encoder (paper §3.1)
├── dan_encoder.py             ← USE_D : Deep Averaging Network encoder (paper §3.2)
│
├── train_transfer.py          ← MAIN EXPERIMENT
│                                 Phase 1: unsupervised pre-training of both encoders
│                                 Phase 2: transfer vs. from-scratch on SST-2 (Table 3)
│
├── similarity_demo.py         ← Eq. 1 similarity + Figure-1-style heatmap
│
├── data/
│   └── sst2/                  ← input dataset (Stanford Sentiment Treebank, 2-class)
│       ├── train.tsv          ←  6,920 sentences
│       ├── dev.tsv            ←    872 sentences  (matches paper Table 1)
│       └── test.tsv           ←  1,821 sentences  (matches paper Table 1)
│
└── ── generated when you run the code ──
    ├── dan_encoder.pt         ← trained DAN weights
    ├── transformer_encoder.pt ← trained Transformer weights
    ├── results.json           ← raw accuracy numbers
    ├── results.txt            ← the formatted results table
    └── heatmap.png            ← the similarity heatmap image

How the pieces fit together (data flow):

data/sst2/*.tsv
      │
      ▼
   common.py   ──►  tokenize, build vocab, angular similarity
      │
      ├────────────►  dan_encoder.py  (USE_D)  ┐
      │                                        ├──►  train_transfer.py  ──►  results.txt / *.pt
      └────────────►  transformer_encoder.py (USE_T) ┘                              │
                                                                                    ▼
                                                                          similarity_demo.py  ──►  heatmap.png

---

3. How to run (from scratch, step by step)

You only need Python 3. No GPU is required — it runs on a plain CPU (and will use Apple MPS or CUDA automatically if available).

Step 1 — open a terminal in the project folder

cd "NLP Project"

Step 2 — install the libraries (only needed once)

pip install -r requirements.txt

Step 3 — train the encoders and run the experiment (≈ 15–20 min on CPU)

python3 train_transfer.py

This trains both encoders and writes results.txt, results.json, dan_encoder.pt, and transformer_encoder.pt.

Step 4 — generate the similarity heatmap (a few seconds)

python3 similarity_demo.py

This reads the trained encoders from Step 3 and writes heatmap.png.

Step 5 — look at the results

open results.txt      # the accuracy table   (use  cat  on Linux)
open heatmap.png      # the heatmap image

Quick single-encoder demo (optional)

Each encoder file can also be run on its own. It encodes one example sentence and prints the embedding shape — a fast way to check things are working:

python3 dan_encoder.py
python3 transformer_encoder.py

Re-training completely from zero

train_transfer.py resumes from saved progress if the output files already exist. To force a full retrain from a clean slate, delete them first:

rm -f results.json dan_encoder.pt transformer_encoder.pt
python3 train_transfer.py

---

4. Input data

data/sst2/ holds the sentence-level SST-2 split (Stanford Sentiment Treebank, binary positive/negative). Each line is sentence<TAB>label, already lowercased and PTB-tokenized — exactly the input format the paper's encoders expect. We report accuracy on the dev split (872 sentences), whose size matches Table 1 of the paper. The test split (1,821 sentences) is included in the folder but we did not use it in this experiment; all results below are on dev.

Source: the public SST-2 mirror at github.com/clairett/pytorch-sentiment-classification.

---

5. Our results

5.1 Transfer experiment (our small-scale Table 3)

Dev accuracy on SST-2 as we vary how much labelled training data the downstream classifier gets (100 → 6,919 sentences). "Transfer (frozen)" = the pre-trained encoder is kept fixed and only a small classifier on top is trained. "From scratch" = the same encoder trained only on the labelled task, no pre-training.

Model	100	250	500	1000	2000	4000	6919
USE_D transfer (frozen)	0.5229	0.5745	0.6216	0.6158	0.6158	0.6227	0.6422
USE_D from scratch	0.4828	0.5837	0.6009	0.5860	0.6720	0.7007	0.7030
USE_T transfer (frozen)	0.5677	0.5791	0.6284	0.6284	0.6525	0.6571	0.6560
USE_T from scratch	0.5275	0.5803	0.6089	0.6261	0.6617	0.7087	0.7477

(Bold marks whichever of transfer vs. from-scratch wins at that data size, per encoder. Numbers come straight from results.txt.)

What this shows. At the smallest data setting (100 labelled sentences), the frozen pre-trained encoder wins for both models — USE_T 0.568 vs 0.528, USE_D 0.523 vs 0.483. As we add more labelled data the from-scratch models catch up and clearly overtake at the full 6,919 set (USE_T 0.748, USE_D 0.703). That is exactly the paper's central point: pre-training buys you the most when labelled data is scarce, and its advantage shrinks as task data grows.

The middle of the table (250–2000) is a bit noisy in our runs — with only a few hundred to a couple thousand examples and a single training seed, a percent or two swings easily. The two ends of the curve, which are the part the paper actually argues about, come out clean.

5.2 Similarity heatmap (our Figure 1)

similarity_demo.py embeds six sentences with each encoder and colours the pairwise angular similarity (Eq. 1). Darker = more similar.

The diagonal is clearly the darkest (every sentence is most similar to itself). Off the diagonal the structure is soft — a few related sentences (like the two positive-sentiment ones, or the two negative ones) sit a little higher, but the similarity is driven mostly by shared movie-domain words rather than by clean sentiment or topic pairing. This is softer than the paper's Figure 1: our unsupervised-only pre-training captures topical and lexical similarity, while Google's full multi-task training (including supervised SNLI) is what sharpens the finer semantic distinctions, which is part of why the paper chose it.

---

6. Honest differences from the paper (it's a scale thing)

We were open about the fact that we cannot match Google's compute, so we kept the experiment faithful in shape and small in size:

• Pre-training data. Google pre-trained on Wikipedia, web news, Q&A pages, forums and SNLI. We pre-train unsupervised on just the 6,920 raw SST training sentences (no labels), using a bag-of-words reconstruction task as a stand-in for the paper's Skip-Thought objective.
• Training set. The paper's SST train set (67.3k) contains phrase-level fragments; we use the sentence-level SST-2 split (6,920 train / 872 dev / 1,821 test — dev and test match Table 1 exactly).
• Model size. The paper doesn't publish its exact sizes or hyperparameters, so we picked small ones that train on a CPU: 128-d word embeddings, 2 Transformer layers, 4 attention heads. The final sentence embedding is 512-d, same as the paper.
• Attention. Built by hand with separate w_q, w_k, w_v projections (the way it was taught in the lecture), not nn.MultiheadAttention.

These choices are the accuracy-vs-resources trade-off the paper itself discusses in Section 8 — a bigger pre-training budget would sharpen our Transformer results, but the qualitative finding already reproduces on a laptop.