HART, a hybrid image generator from MIT, Tsinghua University and NVIDIA, produces images about nine times faster than top diffusion models and can run locally on commercial laptops and smartphones. It pairs a fast autoregressive sketch with a compact diffusion “polish” to capture coarse structure quickly and then refine pixels to match or beat leading diffusion systems.

Hybrid design: fast first, polish second HART blends two established approaches to generative imagery. Autoregressive models—like those behind many large language models—generate content quickly by predicting tokens step by step. Diffusion models, used by image tools such as Stable Diffusion and DALL·E, iteratively de-noise pixels to build an image and typically produce very detailed results but at heavier computational cost. The HART team reversed the usual design trade-off by using an autoregressive transformer to sketch the broad scene; a compact diffusion network then takes that rough picture and refines the pixels. That two-stage move shortens the slow, repeated de-noising steps diffusion models normally run through while preserving—or improving—visual fidelity. The researchers report it matches or exceeds the image quality of state-of-the-art diffusion models while generating images about nine times faster. "If you are painting a landscape, and you just paint the entire canvas once, it might not look very good. But if you paint the big picture and then refine the image with smaller brush strokes, your painting could look a lot better. That's the basic idea with HART," said Haotian Tang SM ’22, PhD ’25, co-lead author of the HART paper. Less compute, more local runs The hybrid approach reduces the total compute HART needs compared with traditional diffusion pipelines. The paper says the process consumes fewer resources—enough that the system can operate on commodity hardware from a single natural-language prompt, including commercial laptops and smartphones. Local execution can cut time and cloud GPU costs for teams that need many synthetic images. Authorship and affiliations Yecheng Wu, an undergraduate at Tsinghua University, is listed as co-lead author alongside senior author Song Han, an associate professor in MIT's Department of Electrical Engineering and Computer Science, a member of the MIT-IBM Watson AI Lab, and a distinguished scientist at NVIDIA. Additional authors are affiliated with MIT, Tsinghua University and NVIDIA. Applications that benefit from speed - Robotics simulation: Faster, realistic frames help robots train in varied virtual scenes and generalize to messy real-world environments. - Self-driving development: Quick synthetic renders let teams create and test rare or dangerous scenarios (sudden pedestrian crossings, debris, unusual vehicle behaviour) without long wait times. - Game and visual design: Artists can iterate on scenes and characters more rapidly when a model sketches composition quickly and then refines detail in a second pass.

Related Articles

The HART paper will be presented at the International Conference on Learning Representations.

This article was created with AI assistance.