A retrieval-augmented generation model for multimodal medical question-answering system - Ismail, Amelia Ritahani; Hishamuddin, Farhan Haikal; Ramjee, Aisar Nasrun; Amir Hussin, Amir 'Aatieff; Afiqa, Nurul (IEEE, 2026)
This paper addresses the limitations of existing medical question-answering systems, which are often unimodal and lack retrieval-augmented capabilities or expert-guided learning. To overcome these challenges, a Retrieval-Augmented Generation (RAG) framework was developed to handle multimodal medical data by integrating GPT-2 for text generation and BLIP for visual understanding. The system was fine-tuned using the MedQuAD and VQA-RAD datasets, and a FAISS-based retriever was used to supply relevant external context. Additionally, reinforcement learning from human feedback (RLHF) was applied to align responses with expert knowledge. Experimental results showed that the GPT-2 model achieved a BERTScore F1 of 0.8204, while the multimodal RAG-enhanced GPT-2 model improved to 0.8411, demonstrating the slight effectiveness of combining retrieval and multimodal learning in enhancing medical answer quality. For RAG-enhanced BLIP, the model shows 0.8627 BERTScore with sample question and image.