DIFF: A Relational Interface for Large-Scale Data Explanation

Abstract: A range of explanation engines assist data analysts by performing feature selection over increasingly high-volume and high-dimensional data, grouping and highlighting commonalities among data points. While useful in diverse tasks such as user behavior analytics, opera- tional event processing, and root cause analysis, today’s explanation engines are designed as standalone data processing tools that do not interoperate with traditional, SQL-based analytics workflows; this limits the applicability and extensibility of these engines. In response, we propose the DIFF operator, a relational aggregation operator that unifies the core functionality of these engines with declarative relational query processing. We implement both single- node and distributed versions of the DIFF operator in MB SQL, an extension of MacroBase, and demonstrate how DIFF can provide the same semantics as existing explanation engines while capturing a broad set of production use cases in industry, including at Microsoft and Facebook. Additionally, we illustrate how this declarative ap- proach to data explanation enables new logical and physical query optimizations. We evaluate these optimizations on several real- world production applications, and find that DIFF in MB SQL can outperform state-of-the-art engines by up to an order of magnitude.