During this talk we will present Stanford CRFM’s efforts to train foundation models for biomedical NLP, culminating in a 2.7B parameter GPT-style model trained on PubMed abstracts and documents in partnership with MosaicML. We will walk through the training process, results of our experiments, next steps, and the broader topic of recent advances in biomedical question answering by language models.

This model has achieved exciting results on QA tasks at a relatively small scale, and we hope it can be useful for studying domain specific language models and building biomedical NLP applications. We are aiming for a vibrant discussion with the biomedical NLP community so we can understand what users want in an open source biomedical language model and gain insights for future models we train.