ORIE Colloquium: Thodoris Lykouris (MIT)

Learning to Defer in Content Moderation: The Human-AI Interplay

Ensuring successful content moderation is vital for a healthy online social platform where it is necessary to responsively remove harmful posts without jeopardizing non-harmful content. Due to the high-volume nature of online posts, human-only moderation is operationally challenging, and platforms often employ a human-AI collaboration approach. A typical heuristic estimates the expected harmfulness of incoming posts and uses fixed thresholds to decide whether to remove the post and whether to send it for human review. This disregards the uncertainty in the machine learning estimation, the time-varying element of human review capacity and post arrivals, and the selective sampling in the dataset (humans only review posts filtered by the admission algorithm). We introduce a model to capture this human-AI interplay. Our algorithm observes contextual information for posts, makes classification and admission decisions, and schedules posts for human review. Only admitted posts receive human reviews on their harmfulness. These reviews help educate the machine-learning algorithms but are delayed due to congestion in the human review system. We propose a near-optimal learning algorithm that balances the classification loss from a selectively sampled dataset, the idiosyncratic loss of non-reviewed posts, and the delay loss of having congestion in the human review system. To the best of our knowledge, this is the first result for online learning in contextual queueing systems and hence our analytical framework may be of independent interest.

Paper information:
This talk is based on joint work with Wentao Weng (Ph.D. student at MIT). A preprint of the corresponding paper can be found here. This work has been selected as a finalist in the 2024 INFORMS Junior Faculty Interest Group (JFIG) Paper Competition.

Bio:
Thodoris Lykouris is an assistant professor of operations management at the MIT Sloan School of Management. Before joining MIT, Thodoris received his Ph.D. in computer science from Cornell University and was then a postdoctoral researcher in Microsoft Research New York. His research focuses on data-driven sequential decision-making and spans across the areas of machine learning, dynamic optimization, and economics. Thodoris publishes in journals such as Journal of the ACM, Mathematics of Operations Research, and Operations Research as well as conferences such as COLT, EC, ICML, NeurIPS, and STOC. His research has been recognized with a Google Ph.D. Fellowship and was a finalist for the Dantzig Dissertation Award, Junior Faculty Interest Group (JFIG) Paper Competition, Nicholson Student Paper Competition and Applied Probability Society (APS) Student Paper Competition. Thodoris recently co-organized an interdisciplinary semester-long program on “Data-Driven Decision Processes” at the Simons Institute for the Theory of Computing at Berkeley. He is currently serving as an Associate Editor in the Stochastic Models area of Operations Research and on the Executive Committee of the Learning Theory Alliance mentoring initiative.