Electronic medical records (EHR’s) contain a lot of structured data and unstructured text about patients. It is most often messy, incomplete, inconsistent, duplicative, and as a result requires a lot of time from doctors and data professionals to get answers from. Large language model (LLM) generative AI interfaces can potentially improve the efficiency and completeness of these workflows by enabling providers or analysts to just ask natural language questions and get short, straight answers – but how accurate and reliable are they?

This session describes benchmarks and lessons learned from building such a pilot system on data from the US Department of Veterans Affairs, a health system which serves over 9 million veterans and their families. This collaboration with VA National Artificial Intelligence Institute (NAII), VA Innovations Unit (VAIU) and Office of Information Technology (OI&T) show that while out-of-the-box accuracy of current LLM’s on clinical notes is unacceptable, it can be significantly improved with pre-processing, for example by using John Snow Labs’ clinical text summarization models prior to feeding that as content to the LLM generative AI output. We will also review responsible and trustworthy AI practices that are critical to delivering these technology in a safe and secure manner.