PhD Thesis Proposal: Paul Lintilhac

Optional ZOOM LINK

"Property Testing AI: An Efficient Frontier"

Abstract

In this paper, we take a step towards addressing the major problem of a lack of standardized and rigorous approaches to testing and evaluation of AI systems. Taking inspiration from both the fields of property testing and property-based testing (for programs), we develop a novel taxonomy of partially overlapping classes of properties of AI systems, including simple properties, compound properties, higher order properties, and data function properties. We argue that this taxonomy categorizes a diverse set of AI traits—including accuracy, fairness, robustness, monotonicity, point-wise and global privacy properties, sensitivity, and more—according to the methods required to test for them.

Through a series of three case studies, we investigate some of the peculiarities and challenges of testing for each of these types of properties. We show that, for these examples of complex AI properties, we must go beyond confidence intervals and IID sampling, and turn to additional theoretical and engineering tools such as property testing (case study 1), linear programming (case study 2), and surrogate modeling (case study 3). While the results in this report are only preliminary, we hope they lay the groundwork for future investigations using our framework, including two larger case studies from the author to be published within the next year with more mainstream applications.

Thesis Committee

• George Cybenko (chair)
• Eugene Santos
• Peter Chin
• David Watson (King's College Longon)

Contact

For more information, contact Thayer Registrar at thayer.registrar@dartmouth.edu.