Abstract From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence. Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated without considering a …
Abstract Markov chain-based methods are ubiquitous and have been highly successful at statistical inference, scientific simulation, and optimization. The common wisdom is that the reason for their success is that a Markov chain of interest “mixes” to its stationary distribution. But what if the Markov chain does not mix fast? …
Abstract Bayesian optimization (BO) is a principled framework for optimizing expensive, noisy objective functions, but traditional BO treats the system as a black box and learns only through input-output queries. In many scientific and engineering settings, this assumption is unnecessarily restrictive; valuable computational structure is often available, even if the …
Abstract The rapid proliferation of high-quality synthetic data — generated by advanced AI models or collected as auxiliary data from related tasks — presents both opportunities and challenges for statistical inference. Here, we introduce the GEneral Synthetic-Powered Inference (GESPI) framework that wraps around any statistical inference procedure to safely enhance …
Abstract Given the massive scale of modern ML models, we now only get a single shot to train them effectively. This restricts our ability to test multiple architectures and hyper-parameter configurations. Instead, we need to understand how these models scale, allowing us to experiment with smaller problems and then apply …
Abstract Modern LLMs process information by repeatedly applying a basic primitive of matrix multiplication. Estimates show that about 60-84% of the energy consumed by LLMs goes into memory load/store operations. How can we reduce this power consumption? LLM converts text into a sequence of tokens (which can be thought as …