Xavier Thomas (Rohan)

TLDR Humans can tell whether two shapes match under rotation or scaling, even without naming what they are. Leading MLLMs cannot. Accuracy holds on photos but collapses on sketches and rare scripts where semantic cues becomes sparser.

TLDR Humans can spot the wrong action or implausible motion in a generated video. MLLMs and existing metrics cannot. We learn a human centric representation of real movement and score how far a generated clip deviates from realistic action.

TLDR If you ask what you hear in a video, people listen. Ask what you see, and they look. MLLMs ignore that distinction when audio, video, and captions disagree. We build benchmarks with deliberate conflicts and fine tune models to answer from the modality being asked about.

TLDR We recognize a dog across photos, cartoons, sketches, and paintings, but classifiers often fail when visual style shifts. Diffusion latents already separate these style domains without labels, so we use them as pseudo domain features to help classifiers generalize to unseen domains, even beating methods trained with ground truth domain tags.

TLDR Generative models produce a rich world of images, but what do they encode internally, and how is that world represented? We uncover interpretable semantic features at specific layers and timesteps with sparse autoencoders, and show they transfer to classification and other vision tasks through lightweight probes.

TLDR A painter sketches the scene first, then adds objects, then texture. Text to image models try to do all of that from one prompt. We break the prompt into the same coarse to fine steps and schedule them across denoising so the final image actually reflects what was asked for.

TLDR Given one handwritten character, people draw new examples that look like the same letter but vary in stroke. FID and likelihood miss this tradeoff, so we measure recognizability and diversity separately and compare one shot models to human samples on Omniglot.

TLDR Video captioning requires a human written caption for every clip. We rank unlabeled videos by multimodal uncertainty and caption semantics so annotators label the clips that most improve the model.

TLDR Training data often arrives mixed across domains with no domain tags. We cluster samples into pseudo domains and train a classifier on both the image and its cluster, reaching performance competitive with domain supervised methods.