The current data stack is built on top of foundations laid down a decade ago for tabular data. But AI datasets are much more complex and workloads are much more diverse. Enterprises scaling AI in production often find data management prohibitively expensive and overly complicated.

Lance columnar format is an open-source project designed to provide the new data foundations for AI, delivering much better performance and scalability for AI datasets, and makes them natively searchable using vector or full text queries.

In this talk we’ll dive into the main challenges that AI data poses, how Lance format works, and the value it delivers to AI teams training models or putting applications into production.