VS-BIM: a cognitive map-driven framework enhancing MLLM for automatic safety inspection in construction
The rise of Multimodal Large Language Models (MLLMs) offers new potential for automated construction safety inspection. However, current discriminative vision-language alignment approaches struggle with spatial understanding and complex reasoning, limiting proactive risk detection. To address this,...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | en |
| Published: |
Elsevier
2026
|
| Subjects: | |
| Online Access: | http://psasir.upm.edu.my/id/eprint/122268/1/122268.pdf http://psasir.upm.edu.my/id/eprint/122268/ https://linkinghub.elsevier.com/retrieve/pii/S147403462500878X |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | The rise of Multimodal Large Language Models (MLLMs) offers new potential for automated construction safety inspection. However, current discriminative vision-language alignment approaches struggle with spatial understanding and complex reasoning, limiting proactive risk detection. To address this, we propose VS-BIM, a generative zero-shot framework driven by cognitive maps. We reconstruct 3D scenes from panoramic video and Building Information Models (BIM), and align visual and semantic information into a queryable 3D cognitive map that serves as the spatial working memory for MLLMs. VS-BIM includes three modules: (1) panoramic video and BIM-based 3D reconstruction; (2) pixel-level semantic embedding using segmentation and pretrained vision-language models; and (3) cognitive map generation via multi-view consistency constraints. We evaluate VS-BIM using public datasets and custom construction environments. In addition, we introduce the CER-QA benchmark to test its performance in configuration recognition, spatial reasoning, spatiotemporal inference, and risk assessment. Results demonstrate VS-BIM excels in 3D object detection, achieves near-human spatial reasoning, and surpasses human averages in spatial estimation. Overall, the cognitive-map-based zero-shot paradigm endows MLLMs with stronger spatial reasoning while greatly reducing the need for large-scale annotation and retraining. By adapting seamlessly to complex, dynamic sites, it shifts safety inspection from passive checks to proactive risk prediction, enabling more efficient construction-site management. |
|---|