Image CaptioningDeep Learning
Teaching a model to describe photos taken by blind users, in words.
The photos are real ones taken by blind users, so they are often blurry, dark or off-centre, and describing them is genuinely hard. I built two generations of the model end to end, measured the jump between them, and then looked inside the newer one to see where it looks in the image when it chooses each word.
The dataset and the task
VizWiz-Captions is built from photos taken by blind users, so images are often blurred, off-centre or poorly lit, and 13.3 percent of the raw captions are the pre-canned quality rejection sentence. After filtering, 7,532 images and 32,619 usable captions remained, with a 3,738-word vocabulary (words seen at least five times) and an image-level 70/15/15 split so no photo leaks between train and test. In a group of six, I built and owned two models end to end and evaluated them on BLEU-1 through BLEU-4 over the 1,131-image test split.
Two models, one variable
The comparison is designed like an experiment: the encoder stays identical and frozen, and only the decoder paradigm changes. Freezing EfficientNet-B0 (chosen for ResNet-50-level accuracy at roughly a fifth of the compute) means the costly image pass is paid once: all 7,532 feature maps are cached to disk, and an epoch then takes 7 to 12 seconds on a free Kaggle GPU. The baseline decodes with a single GRU fed one compressed image vector, deliberately the simplest classic design. The refined model is a three-layer Transformer decoder that cross-attends to all 49 spatial regions of the image at every word, so it never has to remember the picture through a bottleneck.
Beam search with guardrails
The decoding constraints map one-to-one to failures observed in the baseline's outputs: unknown-token flooding on text-heavy photos like medicine labels (so unknown is masked), captions collapsing to nothing (so a minimum length), and "that says that says" loops (so a repetition penalty). Beam search runs five candidates with length normalisation, engineered so all beams share one cached image encoding and batch through the decoder together. The result: BLEU-1 rose from 0.5278 to 0.5644 and BLEU-4 from 0.1365 to 0.1416, and it happened while validation loss got worse, a documented reminder that loss and caption quality are different judges. A cross-team finding landed too: my simple baseline matched a teammate's attention model built on ResNet-50, meaning the encoder choice bought more than the attention did on that stack.
Looking inside the model
The figures below are straight from the delivered notebook: both models' captions against the human references on twelve test photos, and the Transformer's cross-attention heatmaps. The word "barcode" visibly attends to the barcode; on a chocolate package the words are wrong but the attention sits exactly on the lettering, which pins the remaining failures where the report puts them: vocabulary and object naming, colour swaps and guessed label text, not broken visual grounding. Seeing where the model looks is how you debug a captioner.
@torch.no_grad()
def evaluate_beam(model, loader, beam_width: int = 5) -> dict:
model.eval()
hyps, ids = [], []
for batch_ids, feats, tokens, lens in loader:
feats = feats.to(DEVICE)
out_ids = model.generate_beam(feats, beam_width=beam_width)
hyps.extend([decode_ids(o) for o in out_ids])
ids.extend(batch_ids)
return {'scores': compute_bleu(ids, hyps), 'ids': ids, 'hyps': hyps}
m2_eval = evaluate_beam(model2, test_loader, beam_width=5)