Title: Bridging Linear MDP to Deep RL via Latent Variable Representation
Abstract: Despite the empirical success of RL with deep neural networks (Deep RL), the existing theory of MDP requires different variants of linear structures (e.g., Linear MDP and its various generalizations). On the other hand, the straightforward generalization of the existing theory to nonlinear transition kernels (e.g., deep neural networks) faces exponential lower bounds. To bridge the theory-practice gap, we take a different route. Instead of analyzing existing Deep RL algorithms equipped with vanilla neural architectures (e.g., feedforward neural networks), we develop a new family of algorithms equipped with novel neural architectures based on latent variable representations, which enforce structures while preserving expressiveness in modeling. The new approach allows us to incorporate an estimation subroutine, which takes advantage of existing contrastive learning or variational inference algorithms, and corresponding uncertainty quantification techniques, which play a critical role in the computational and sample efficiency. The resulting method demonstrates superior empirical performance while achieving provable theoretical guarantees.
Bio: Zhaoran Wang is an assistant professor at Northwestern University, working at the interface of machine learning, statistics, and optimization. He is the recipient of the AISTATS (Artificial Intelligence and Statistics Conference) notable paper award, ASA (American Statistical Association) best student paper in statistical learning and data mining, INFORMS (Institute for Operations Research and the Management Sciences) best student paper finalist in data mining, Microsoft Ph.D. Fellowship, Simons-Berkeley/J.P. Morgan AI Research Fellowship, Amazon Machine Learning Research Award, and NSF CAREER Award.