How Far Can We Go With ImageNet for Text-to-Image Generation?

TL;DR

Is "bigger is better" the only way?

Text-to-image models are usually trained on billion-scale web datasets, which are either closed or decaying over time. We show that a model trained on ImageNet alone (1.2M images), enhanced with well-designed text and image augmentations, reaches the GenEval score of FLUX-dev and outperforms SD3 and SDXL, with 1/1000th of the training images and 3× to 10× fewer parameters. The standardized training setup only needs about 500 H100 hours, which makes text-to-image research reproducible and accessible.

1 · Text challenges

ImageNet has no captions

ImageNet is a classification dataset. The simplest caption, "An image of ⟨class name⟩" (AIO), gives very poor text-to-image models: it has only about a thousand words, ignores everything in the image besides the class, and has no word for concepts such as person, road or water, which appear in many images but are not classes.

Long informative captions (TA). We re-caption every image with LLaVA, describing scene composition, background, secondary objects, attributes and actions. The same captioner describes our CutMix images (see below). Pick an image:

AIO An image of ⟨class name⟩

TA

Long captions transform ImageNet into a text-to-image dataset. On ImageNet validation, FID drops from 20.14 to 6.29 for DiT-I; on COCO, which is zero-shot for our models, only TA models follow the prompt (Jina CLIP 22.42 → 38.39). GenEval jumps from 0.11 to 0.55.

2 · Image challenges

Overfitting and object-centric images

Even with long captions, ImageNet-only training faces two limitations. Early overfitting: with only 1.2M images, latent-space models start to overfit after about 200k steps. Limited compositionality: ImageNet is object-centric, so models struggle with spatial relations, attribute binding and multi-object scenes.

Image augmentation (IA). We study two augmentations:

• CutMix: a smaller image from another class is pasted onto the original image, and the composite is re-captioned. To avoid learning the sharp paste boundaries, CutMix images are only used at noise levels where the boundary is no longer visible.
• Crop: the model only sees a random local crop of the image tokens, with the crop coordinates added to the caption. This helps decouple objects from their background.

Image augmentation helps the latent-space models, and SiT-I benefits the most: with CutMix, its GenEval jumps by +10 points (to 0.61), driven by color attribution (+16) and two objects (+13), and ImageReward goes from −0.12 to 0.22. In contrast, the pixel-space JiT-I degrades slightly. We attribute this to the VAE of latent models, whose encoder absorbs the sharp, locally unnatural CutMix boundaries; JiT-I models raw pixels and has no such filter.

Image augmentation also improves compositions. With TA only, the teddy bear sits awkwardly on the motorbike; with TA + IA it drives it. The goat scene is better composed, and color attribution improves for the butterfly and the baseballs.

TATA + IA

TA

TA + IA

TA

TA + IA

TA

TA + IA

TA

TA + IA

TA

TA + IA

TA

TA + IA

3 · Conditioning with MIRO

Learning preferences during pretraining

MIRO conditions text-to-image pretraining on multiple reward signals, so the model learns user preferences directly instead of relying on post-hoc reward fine-tuning. In our ImageNet-only setting, MIRO improves SiT-I by +5 GenEval points (0.51 → 0.56) and ImageReward from −0.12 to 0.29. The gain is smaller than in the original paper with web-scale data, and smaller than what CutMix alone brings.

25k50k75k100k125k150k200k SiT-I CutMix + MIRO

“a fluffy corgi dog mid-flight with a red superhero cape”

Training steps, left to right. Top: SiT-I CutMix. Bottom: with MIRO conditioning. ↻ Replay

4 · Scaling

Bigger models and higher resolution

Model size. Scaling SiT-I CutMix from L to XL and H does not improve compositionality: GenEval stays flat, suggesting that the L model already extracts most of what ImageNet offers for this benchmark. Aesthetic metrics improve slightly with size. Resolution. Going to 512², either by fine-tuning a 256² checkpoint (DiT-I) or training from scratch (SiT-I), improves both GenEval and aesthetics.

Putting everything together

We combine the best ingredients into a single model: SiT-H trained with CutMix and MIRO conditioning. At 512², it reaches 0.67 on GenEval and an ImageReward of 0.73.

5 · Comparison with the state of the art

GenEval

Our best model reaches 0.67 overall, matching FLUX-dev and surpassing SANA-0.6 (0.64) and SD3 Medium (0.62), all trained on orders of magnitude more data. It obtains the best Two Objects score (0.82), but falls behind on Counting.

DPGBench

On DPGBench, a more complex benchmark, our model reaches 81.2 overall, surpassing SDXL and PixArt-Σ and matching Janus. It trails FLUX-dev and SANA-0.6 by about 3 points but is the best on Relation (93.0).

6 · Generalization

Beyond ImageNet's objects

Despite ImageNet's object-centric bias, our models compose complex scenes and approximate concepts never seen during training, by combining patterns from training concepts. For example, the model creates a hybrid creature mixing a giraffe and a tortoise, leveraging the text understanding of the language model that encodes the prompt.

BibTeX

@article{degeorge2025farimagenettexttoimagegeneration,
  title   = {How far can we go with ImageNet for Text-to-Image generation?},
  author  = {Degeorge, Lucas and Ghosh, Arijit and Dufour, Nicolas and
             Picard, David and Kalogeiton, Vicky},
  journal = {arXiv preprint arXiv:2502.21318},
  year    = {2025}
}

Acknowledgments

This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD011015436 and 2025-AD011015594 made by GENCI, and by the SHARP ANR project ANR-23-PEIA-0008 funded in the context of the France 2030 program. The authors would like to thank Alexei A. Efros, Thibaut Loiseau, Yannis Siglidis, Yohann Perron, Louis Geist, Robin Courant and Sinisa Stekovic for their insightful comments, suggestions, and discussions.