Statistics Colloquium: Karl Rohe
11:30 am–12:30 pm Jones 303
Karl Rohe
Department of Statistics
University of Wisconsin-Madison
Title: Even humans disagree: evaluating raters, human and AI, without gold labels
Abstract: Turning documents into data has taken a great deal of human labor. Doctors read trial reports, lawyers read court filings, and social scientists read survey responses. This labor goes by many names, including screening, data extraction, document review, content analysis, annotation, and document coding. When an AI reads the documents and fills in the spreadsheet, I refer to it as data minting. This talk gives examples from handwritten historical ledgers, bankruptcy court filings, open-ended survey responses, and clinical trial reports. Historically, two humans would do the work and compare answers, with their agreement measured by Cohen’s κ (or similar). Even trained humans often disagreed, so evaluating an AI at this work is hard. We can compute κ for an AI and a human, but when they disagree, we cannot tell which one erred.
We would like to measure the skill of each rater. To do so, we propose the solve-or-guess model. On each item (one question about one document), rater a (human or AI) either solves it, with probability p_a, or guesses. All solvers give the same label, which we call the latent consensus. A guess can match it by luck, so p_a is not accuracy. Under certain conditions, Cohen’s κ converges to κ(a,b) = p_a p_b. Thus, with three or more raters, the pairwise κ‘s identify each rater’s solving probability, with no gold labels. We give examples from abstract screening and risk-of-bias assessment in meta-analysis, and from the Cochrane Crowd. Time permitting, the talk will discuss various extensions of the model.
Joint work with Auden Krauska, Gary Collins, Julian Higgins, and James Pustejovsky.
Statistics Colloquium: Salar Fattahi
11:30 am–12:30 pm Jones 303
Salar Fattahi
Department of Industrial and Operations Engineering
University of Michigan
Title: Nonsmooth Composite Optimization at the Edge of Degeneracy
Abstract:
Many optimization problems arising in modern data-driven applications are inherently nonsmooth, including those involving robust losses, sparsity, low-rank structure, and activation functions. Unlike smooth optimization, where the vanishing of the gradient certifies stationarity or even optimality, the nonsmooth setting offers none of this, forcing algorithms to work with a set-valued subdifferential whose elements need not vanish even at a minimizer. Yet a powerful line of work, dating back to the 1980s, has shown that this apparent nonsmoothness often conceals a smooth structure underneath: near many critical points, algorithms eventually identify a lower-dimensional smooth manifold and thereafter behave much like smooth optimization methods.
But what happens when no such manifold exists? Such “degenerate” critical points have traditionally been viewed as pathological or nongeneric and are largely excluded from existing theory. In this talk, I will present a complementary perspective. I will argue that degeneracy can arise systematically in important classes of modern nonsmooth composite problems, including robust regression and low-rank matrix optimization, and yet need not obstruct efficient optimization. Simple first-order methods can still converge rapidly to degenerate solutions despite the failure of the usual manifold-based explanation. These results suggest that degeneracy does not destroy local structure, but instead replaces smooth manifold geometry with a richer, often stratified geometry.