With more than one million new biomedical research papers published per year, providing everyone with fast & reliable answers based on current scientific knowledge is paramount. This session by Veysel Kocaman, the Head of Data Science at John Snow Labs, presents a comprehensive evaluation of how well the company’s state-of-the-art Medical Chatbot meets that goal. The evaluated tasks include question answering on:

• Prebuilt knowledge bases such as Pubmed, MedArxiv, and Clinical Trials.
• User-specific documents, such as internal & confidential files.
• Structured data in relational databases, without users resorting to SQL or BI tools.

The evaluation covers these dimensions:

• Truthfulness: Ensuring answer fidelity, prioritizing trustworthy sources, and negating hallucinations.
• Accuracy: Delivering higher accuracy compared to general-purpose LLMs and vector databases with bespoke document splitting, reranking, post-filtering, and similarity search.
• Explain-ability: Citing resources while generating answers.
• Security: Supporting air-gapped deployment, functioning securely on-premise without requiring internet connectivity or external API calls.
• Expert Preference: A team of medical doctors applied a specialized methodology to evaluate the generated answers on relevance, style, consistency, and appropriateness.
• Recall: Medical doctors evaluated not only the generated answers, but also whether there exists additional research that can provide a more recent or complete answer.
• Latency: The speed of building & updating the body of knowledge, calculating embeddings, and running inference to answer user questions.

Gain insights into the capabilities, benchmarks, and applications of the Medical Chatbot, shaping the future of healthcare data retrieval and decision-making support.