In the last years there has been significant progress in a large number of downstream NLP tasks, mainly through the development of new deep neural language models. While these models increase the performance of NLP tasks, they are also more complex, more difficult to understand and, ultimately, more prone to brittleness. This is especially the case when the input these systems are applied on does not come from exactly the same distribution as the one of the data they have been trained on.

In this talk we will introduce the notion and practice of adversarial evaluation, an approach to evaluating NLP systems by exposing them to input that has been intentionally created to result in incorrect output. By applying this approach we can better understand the capabilities and limitations of an NLP system, uncover its blind spots, and avoid unnecessary risks.