A Survey on Knowledge Distillation and Lightweight Deployment of Multimodal Large Language Models

Abstract

In recent years, multimodal large language models (Multimodal Large Language Models, MLLMs) represented by LLaVA and Qwen-VL have achieved remarkable breakthroughs in tasks such as image understanding and visual question answering. However, the prohibitive computational and storage overhead brought by their billions or even hundreds of billions of parameters severely constrains the feasibility of on-device deployment. To this end, this paper adopts a systematic literature review methodology, retrieving and screening 68 core publications from six major academic databases spanning 2018 to 2025, and provides a panoramic survey of the MLLM knowledge distillation technology landscape. The study reveals that knowledge distillation has formed three core methodological categories—response, feature, and relation distillation—and has given rise to emerging paradigms including visual encoder distillation, cross-modal alignment distillation, and end-to-end instruction-tuning distillation. Quantitative analysis shows that LLaVA-MoD (2B) achieves 95.5% of the teacher model’s performance on ScienceQA [20]; TinyLLaVA (3.1B) achieves 51.5% on TextVQA [46]; and MobileVLM-3B achieves 59.0% on GQA with only 43% of the parameters of LLaVA-7B [41]. Overall, knowledge distillation has become a key enabling technology for on-device deployment of MLLMs. Nevertheless, how to strike a balance among cross-modal knowledge preservation, heterogeneous hardware adaptation, and extreme compression efficiency remains a core challenge urgently to be addressed in this field.