Xavier Thomas (Rohan)

TLDR Humans can tell whether two shapes are the same under rotation or scale, even without recognizing what the object is. Leading vision language models cannot. They need a familiar visual context to reason spatially.

TLDR Humans can immediately spot a generated video where the action is wrong or the movement looks off. AI metrics give it a perfect score anyway. We build a metric grounded in how real people move so that bad motion actually gets flagged.

TLDR If you ask what you hear in a video, people listen. Ask what you see, and they look. Multimodal models often don't make that distinction. We put audio, video, and text in direct conflict and train models to answer from the sense the question actually asks about.

TLDR We recognize a dog whether it is a photo, cartoon, sketch, or painting. Classifiers often fail when the visual style shifts. Diffusion model features naturally group images by style without any labels, and plugging them into a classifier improves generalization to new visual styles.

TLDR Generative models produce rich, detailed images, but what do they actually understand internally? We open them up, find that they encode meaningful visual concepts in specific layers, and show those internal representations are useful for other vision tasks.

TLDR A painter sketches the scene first, then fills in objects, then texture. Text to image models try to do all of that from a single prompt at once. We split the prompt into those same steps and feed them in progressively during generation, so the output actually matches the description.

TLDR Given one handwritten character, people draw new versions that are still recognizable but not identical. Existing metrics for generative models miss this balance entirely. We measure recognizability and diversity as separate axes and test how closely models match what humans actually produce.

TLDR Training a video captioning model requires a human written caption for every clip, which is expensive. We rank which unlabeled videos are most worth annotating, so the model learns faster with fewer labels.

TLDR Training images often come from many different sources with no labels telling the model which is which. We automatically group them by visual style and train classifiers that adapt to each group, matching methods that had those source labels to begin with.