DocuSign is the world’s largest e-signatures provider and serves more than 500,000 customers and hundreds of millions of users worldwide.

This talk summarizes lessons learned from building a deep-learning model pipeline that automatically understands where fillable fields are in each document – such as signature boxes, checkboxes, initial fields, and text fill-in fields.

A major challenge is learning to handle the immense variety of documents that DocuSign processes, covering just about every business transaction from loans, insurance, healthcare, education, government, real estate, manufacturing, telecom, retail, employment, to enterprise-driven legal affairs.

This task requires a modeling pipeline that unifies three usually separate problems:

1. Computer Vision​ – since many documents are scanned or uploaded as photographs, the system must handle image correction / pre-processing, page segmentation, and object detection.
2.  OCR​ – text from documents must be extracted in order to correlate fields and their corresponding tag types – this information goes on to inform the “type” of tag that it is
3. NLP​ – text from OCR, along with field coordinates from Computer Vision is used to infer the “type” of a given field, EG signature, or text

We’ll cover experiments and learnings from:

1. Applying DDS, ResNet, Inception, and Faster R-CNN.
2. Object Character Recognition – classifying fields requires “reading” the text around them, hence training OCR, line detection, and page segmentation models are required.
3. Natural Language Processing – we’ll discuss results and lessons learned from training, annotating, feature engineering, and hyper-parameter tuning of the NLP models that classify field types with documents.

Critically, the unified pipeline must satisfy these four requirements:

1. State-of-the-art accuracy of the entire pipeline – in contrast to local optimization of each model
2. Interpretability – visually showing what the model inferred on each document, to enable debugging and manual corrections
3. Scalability – the ability to scale training and inference to many millions of documents per week
4. Compliance – given the high sensitivity of many business documents that DocuSign handles, the entire training & inference infrastructure must be locked down and run on bare metal, within custom data centers. The deep learning libraries and models utilized will all be open source and based on the TensorFlow ecosystem.

This talk is intended for data scientists looking for best practices and lessons learned on building real-world, large-scale document understanding systems based on these technologies, with insights also relevant to applications of generative AI in healthcare.