PhD Thesis Defense: Paul Lintilhac

"Property Testing AI: An Efficient Frontier"

Abstract

In this paper, we take a step towards addressing the major problem of a lack of standardized and rigorous approaches to testing and evaluation of AI systems. Taking inspiration from both the fields of property testing and property-based testing (for programs), we develop a novel taxonomy of partially overlapping classes of properties of AI systems, including simple properties, compound properties, higher order properties, data relation properties, and architecture-utility properties. We argue that this taxonomy categorizes a diverse set of AI traits—including accuracy, fairness, robustness, monotonicity, point-wise and global privacy properties, sensitivity, and more—according to the methods required to test for them.

Through a series of four case studies, we investigate some of the peculiarities and challenges of testing for each of these types of properties. We show that, for these examples of complex AI properties, we must go beyond confidence intervals and IID sampling, and turn to additional theoretical and engineering tools such as property testing (case studies 1 and 2), membership inference attacks (case study 3), and adversarial risk analysis (case study 4). In addition, our case studies have intellectual merit on their own, including a new generalization bound for transformers based on two key complexity properties, a new theoretical perspective on membership inference attacks as property tests, and a novel approach to evaluating overall risk of multimodal foundation models in an adversarial context. Taken together, this work proposes property testing not only as a practical method but also as a unifying theoretical lens for building more principled and reliable AI evaluation strategies.

Thesis Committee

• George Cybenko (Chair)
• Eugene Santos
• Peter Chin
• David Watson (King's College London)

Contact

For more information, contact Thayer Registrar at thayer.registrar@dartmouth.edu.