NA-MIC Project Weeks
Back to Projects List
Generative AI for Display layout/ Hanging protocols
Key Investigators
- Martin Bellehumeur (Germany)
Project Description
Generative AI / LLMs can today produce a good draft of report impressions based on learning the report history of the radiologist. We believe that GenAI could today produce reasonably valid hanging protocols if it had enough training data.
Therefore, we propose to standardized hanging protocol Gen AI training data.
This data would be used by PACS viewers or AI agents to infer the appropriate display of the images available.
Because of standardization, radiologists would no longer have to train/configure each viewer they encounter.
They (or their PACS admin) would instead provide access to their personal training data repository to the viewer software.
Ideally, the viewers would not only use the data for display but also provide the ability to add new hanging protocol training cases to the user’s repository.
We believe that having the full study/series/image metadata, in addition to the thumbnails, could allow AI to compete with rule based systems. Especially when those systems cannot be constantly maintained by highly skilled individuals.
Gen AI would also enable personal hanging protocols which is almost impossible today because of maintenance costs.
Objective
Raise interest in the radiology reporting community in the standardization of hanging protocol training data.
Create proof of concept training data by adding the functionality to an open-source viewer. This exercise should allow to refine the training data format required and could support an IHE standard request/grant request.
Approach and Plan
This week we will add a prototype extension to the OHIF viewer that will save display layout training data and load it. Ideally, we would do the same in 3DSlicer and demonstrate interoperability.
Criterias
-
The training data should be human readable. Therefore we would use DICOMweb/JSON encoding instead of DICOM binary.
-
The training data should be portable. The size of the data should be small enough that radiologist can keep their own backup copy, modify it manually if they wish and create multiple repositories if they need to.
Example training data format
Each stored hanging protocol training data point would be composed of a folder containing:
- A screenshot of the layout with privacy mode on.
- A hanging protocol DICOM information object defining the layout and which series was assigned to which viewport (and other details). This object would not try to define rules for the hanging protocol but just store what the user selected. Also of interest is the work in section V. THe hard work of defining monitor geometry, layouts, viewport types, viewport linking, ect, all that is already existing in the DICOM standard.
- Separate folder for each study of the patient containing the metadata of each study/series/image(object) of the patient in static DICOMweb format. Therefore each display set will also include a thumbnail image. Patient level information and bulk data (DICOM image data) would not be stored.
- A way to identify which study was the current.
Viewer support
Viewers would at a minimum need to support reading the hanging protocol object and assign the series specified by the AI to the layout. Viewers that already support the existing hanging protocol DICOM standard object would be advantaged as they should be.
The request for the hanging protocol object would be done by the worklist or the viewer. They would provide the same study/series metadata that is in the training data and the AI agent would return the most appropriate hanging protocol from your training data and assign series to the defined viewports in the hanging protocol object.
Progress and Next Steps
TBD
Illustrations
A hanging protocol AI agent could, using the studies and series level metadata of the current and prior studies, find the most appropriate layout.
How Hanging Protocol AI Agents Work
+-----------------------------------+ +---------------------+ +-------------------+
| | | | | |
| PACS Viewer +-------->+ Hanging Protocol +-------->+ Display Layout |
| (current & prior study metadata | | AI Agent | | (images assigned |
| and thumbnails) | | (selects protocol | | to viewports) |
| | | and assigns series | | |
+-----------------------------------+ +---------------------+ +-------------------+
The PACS viewer sends current and prior study metadata and thumbnails to the AI agent, which selects the best hanging protocol and returns the layout for display. The PACS viewer sends study metadata and images to the AI agent, which selects the best hanging protocol and returns the layout for display.