Integrating clinical knowledge and imaging for medical report generation

Medical report generation is an important cross-modal task in the field of medicine, aiming to automatically generate professional and accurate reports for given medical images. Integrating clinical knowledge into the task of medical report generation can enhance the semantic accuracy of medical image feature descriptions and improves the interpretability and robustness of the model. In this paper, we propose to integrate clinical knowledge and image content to generate medical report. The clinical knowledge is presented by a clinical knowledge graph, where anatomical structures and observations are represented as nodes, and their relationships are represented as edges. We design a graph generation module to dynamically generate relevant knowledge graphs specific to each image, which can diversify the knowledge graph structures and expand the coverage of clinical knowledge to provide a more extensive range of clinical knowledge during report generation. Furthermore, we design a graph attention module to facilitate optimized feature representation within the clinical knowledge graph by incorporating message passing between nodes and edges. This fosters a more comprehensive understanding of the relationships and significance of clinical information. Experimental evaluations conducted on the IU X-Ray and MIMIC CXR datasets demonstrate the superiority of the proposed method in generating medical reports. The results highlight the potential of leveraging clinical knowledge for enhancing the precision and clinical relevance of the generated reports.