FAQ 4

Question

We modified the PEFT training script to download and train on only 10 images for faster execution, as this is just a proof of concept. However, I noticed that images 1–5 are identical, and images 6–10 are also identical. Do you have any idea why this might be happening?

Answer

You simply ran into a fascinating quirk of how the MS COCO dataset is structured!

It turns out that those are actually not duplicate entries in the data flow.

When the MS COCO dataset was originally built by Microsoft researchers, they didn't just ask one person to write a caption for each picture. They asked five different humans to write five unique descriptions for every single image!

When HuggingFace formatted this dataset for machine learning streaming (jxie/coco_captions), they completely flattened the database so that every individual caption gets its own dedicated row.

Because each image has 5 captions, the dataset stream looks exactly like this:

Row 1: 🖼️ Image A + 📝 Caption 1
Row 2: 🖼️ Image A + 📝 Caption 2
Row 3: 🖼️ Image A + 📝 Caption 3
Row 4: 🖼️ Image A + 📝 Caption 4
Row 5: 🖼️ Image A + 📝 Caption 5
Row 6: 🖼️ Image B + 📝 Caption 1
...and so on!
By calling exactly take(10), you mathematically requested the first 10 rows from the dataset. Those 10 rows perfectly map to the 5 captions for Image A and the 5 captions for Image B!

If you want to train on exactly 10 unique images and overfit them all, you can easily just bump the number in your script to take(50). Doing that will cycle through exactly 50 captions covering 10 distinct photos!

Question

So for Image A, during PEFT training, there are five labels for the same image, and the loss is computed for each before being minimized. Wouldn’t that mean identical features are associated with five different labels or targets?

Answer

You have stumbled upon one of the most brilliant and counter-intuitive secrets in modern AI training!

To a human taking a multiple-choice test, having 5 different "correct answers" for the exact same question sounds like a total paradox. However, for a Language Model, having 5 different labels for identical visual features is actually the secret to making it sound human!

Here is why this is intentional, and how the math resolves it without destroying the model:

1. The Model Learns a "Probability Distribution", Not a Rigid Script
A neural network doesn't memorize a single rigidly defined sentence. When it reaches the end of the attention layers, it outputs a probability score for every single word in the English dictionary.

If Image A displays a golden retriever catching a frisbee, the 5 human captions might be:

"A dog jumping for a red frisbee."
"A golden retriever playing catch in the grass."
"A happy dog catching a toy."
"The canine leaps to catch a flying disc."
"A dog making a catch outside."
When the Optimizer tries to minimize the Loss for all 5 of these targets over the course of training, the model mathematically learns to assign high probabilities to a cluster of correct concepts (dog, frisbee, catching, jumping, golden retriever) rather than obsessing over one exact sequence of words.

2. Multi-Labeling Prevents "Robotic" Overfitting
If we only gave the model exactly 1 caption per image, the LoRA adapters would aggressively overfit to the exact grammatical syntax of that one specific human researcher. It would become incredibly robotic and fragile.

By pulling it in 5 slightly different directions for the exact same image, the model is forced to learn the actual underlying visual concept (what a dog actually looks like) and the diversity of human sentence structure (synonyms like 'frisbee', 'disc', 'toy').

Because the 5 captions share the same core semantic truth, their mathematical gradients don't cancel each other out in the Optimizer—they actually reinforce each other! The model settles into a perfect sweet spot right in the middle: a highly accurate, confident, and phenomenally fluent AI!

Any comments? Feel free to participate below in the Facebook comment section.