location: Orchard View Room
SILO: Faster Diffusion Language Models
Abstract: Diffusion language models (DLMs) represent a nascent but promising alternative to GPT-style autoregressive (AR) language models: as opposed to generating one token at a time left to right, DLMs start from a set of noise tokens which they iteratively refine into text. The any-order generation can potentially result in …
SILO: Relying on the Metrics of Evaluated Agents
Abstract: Developers and regulators of online platforms and AI systems face a continuing problem of designing effective evaluation metrics. While tools for collecting and processing data continue to progress, this has not addressed the problem of “unknown unknowns”, or fundamental informational limitations on part of the evaluator. To guide the …
SILO: First-Order Algorithms for Large-Scale Optimization
Abstract: It is well known that for nonconvex unconstrained optimization with Lipschitz smoothness, gradient descent and stochastic gradient descent are the optimal first-order algorithms in the deterministic and stochastic settings, respectively. This naturally raises two questions: In the constrained setting, is it possible to design algorithms that achieve the same …
SILO: Searching for architectures and BERT moments in specialized AI applications
Abstract: In 2018, advances in architecture design and self-supervised learning led to the “BERT moment” in natural language processing, in which supervised learning workflows were permanently supplanted by the pretraining and fine-tuning of massive Transformer models. This spurred scientists in more specialized areas—e.g. genomics, satellite imaging, and time series forecasting—to develop …
SILO: Some Online Combinatorial Optimization and Dynamic Pricing Problems
Abstract: Optimizing subsets of items arises in many contexts, from designing antibiotic cocktails, to bundling cable channels or streaming services, to selecting the tap list at a pub. Such problems often exhibit diminishing returns: adding a third antibiotic may improve efficacy, but not as much as adding the second. Prior …
SILO: Stable Estimators for Fast Private Statistics
Abstract: We will discuss a new set of techniques for stable statistical estimation, leading to fast and near-optimal private algorithms for mean estimation, covariance estimation, and linear regression. The analysis proceeds by constructing a stabilizing wrapper around a greedy outlier-removal process. We will also discuss connections with a recent line …