Panasonic Holdings develops LaViDa, a diffusion based vision language model

The mechanism for generating discrete data such as text using diffusion models consists of a masking process, in which each token*² that makes up a sentence is randomly replaced with a mask token, and an unmasking process, in which the original tokens are restored from the masked tokens (Figure 1). However, simply replacing an autoregressive multimodal AI model with a diffusion model gives rise to two problems: (i) the attention*³ computations required for generating tokens each time become heavy, and (ii) there is a high risk that words important for image captions will not be learned (i.e., will not be masked).
The problem in (i)—the heavy attention computations—stems from the fact that, in an autoregressive model, the tokens to be predicted only need to attend to previously generated past tokens, whereas in a diffusion model the attention computations must always reference all tokens in the entire sentence.

To reduce computation, we introduced a technique that excludes the answer tokens from the attention calculations between the input image tokens and the question tokens. The attention mask shown in Figure 2 highlights (in green) the query–key pairs over which attention is computed. Rows represent queries and columns represent keys; I denotes the image-token sequence, P the question-token sequence, and X_t the answer-token sequence at step t. In the autoregressive case (left), the attention map only needs to consider past tokens, so the white areas in the figure require no computation. In the diffusion model (center), attention must be computed over all tokens. In our proposed Prefix-DLM (right), we improve efficiency by removing the answer tokens from the attention calculations involving image and question tokens (the white areas).

To address issue (ii), during training we prepare two complementary masking patterns for the same sentence so that the tokens masked in one pattern are not masked in the other, and we train both unmasking processes. This ensures that every token in the sentence is learned.

In evaluation experiments, we validated effectiveness using a variety of datasets, ranging from natural-image Q&A tasks to mathematical and scientific proof problems and document-understanding tasks containing many charts and graphs. Figure 4 shows (left) performance comparisons with existing methods on each evaluation dataset and (right) a plot with generation time on the x‑axis and generation performance on the y‑axis. In the right-hand plot, “NFE” is an indicator of the proportion of tokens generated in a single diffusion step. A lower NFE means more tokens are generated per step, which reduces the number of steps required and thus speeds up generation, but performance tends to decrease — a trade-off.
LaViDa achieved higher performance than existing autoregressive methods on all datasets, and we also confirmed that it can generate text faster than existing autoregressive approaches.

Figure 5 shows examples of generating image content in structured formats such as poetry and JSON. With conventional autoregressive methods, it was necessary to specify detailed rules in the prompt, which often caused misinterpretation; with LaViDa, however, the unmasking mechanism enables these formats to be generated naturally.