SILO: Relying on the Metrics of Evaluated Agents
Abstract: Developers and regulators of online platforms and AI systems face a continuing problem of designing effective evaluation metrics. While tools for collecting and processing data continue to progress, this has not addressed the problem of “unknown unknowns”, or fundamental informational limitations on part of the evaluator. To guide the …