text: 262,66,1460774809114136577,1254948077825294336,Mahdi Haghifam,"🔥 New paper 🔥on @shortstein and @zakynthinou's CMI framework, demonstrating its unifying nature for obtaining optimal or near-optimal bounds for the expected excess risk in the realizable setting. <LINK> Will be a spotlight at NeurIPS’21! <LINK> Different frameworks for proving generalization are not compatible. For instance, the minmax learnability of thresholds cannot be established using input-output MI, PAC-Bayes bounds, and privacy frameworks. How expressive is CMI in terms of explaining generalization? Some results 1- The CMI framework yields optimal risk bound for sample compression schemes. Covers many algorithms, including SVMs. 2- A broad class of proper ERMs achieves CMI of order O(1) (without suboptimal log n factor) for learning VC classes with finite star number. 3. Steinke and Zakynthinou introduced a variant of CMI called “evaluated"" CMI (eCMI). We show that, for every interpolating algorithm and data distribution, the expected risk vanishes as the number of samples (n) diverges if and only if its eCMI has sublinear growth with n. Joint work with @kdziugaite, Shay Moran, and @roydanroy. @lzamparo @shortstein @zakynthinou Thanks!",https://arxiv.org/abs/2111.05275,"In this work, we investigate the expressiveness of the ""conditional mutual information"" (CMI) framework of Steinke and Zakynthinou (2020) and the prospect of using it to provide a unified framework for proving generalization bounds in the realizable setting. We first demonstrate that one can use this framework to express non-trivial (but sub-optimal) bounds for any learning algorithm that outputs hypotheses from a class of bounded VC dimension. We prove that the CMI framework yields the optimal bound on the expected risk of Support Vector Machines (SVMs) for learning halfspaces. This result is an application of our general result showing that stable compression schemes Bousquet al. (2020) of size $k$ have uniformly bounded CMI of order $O(k)$. We further show that an inherent limitation of proper learning of VC classes contradicts the existence of a proper learner with constant CMI, and it implies a negative resolution to an open problem of Steinke and Zakynthinou (2020). We further study the CMI of empirical risk minimizers (ERMs) of class $H$ and show that it is possible to output all consistent classifiers (version space) with bounded CMI if and only if $H$ has a bounded star number (Hanneke and Yang (2015)). Moreover, we prove a general reduction showing that ""leave-one-out"" analysis is expressible via the CMI framework. As a corollary we investigate the CMI of the one-inclusion-graph algorithm proposed by Haussler et al. (1994). More generally, we show that the CMI framework is universal in the sense that for every consistent algorithm and data distribution, the expected risk vanishes as the number of samples diverges if and only if its evaluated CMI has sublinear growth with the number of samples. ",Towards a Unified Information-Theoretic Framework for Generalization,6,"[""🔥 New paper 🔥on @shortstein and @zakynthinou's CMI framework, demonstrating its unifying nature for obtaining optimal or near-optimal bounds for the expected excess risk in the realizable setting.\n<LINK>\nWill be a spotlight at NeurIPS’21! <LINK>"", 'Different frameworks for proving generalization are not compatible. For instance, the minmax learnability of thresholds cannot be established using input-output MI, PAC-Bayes bounds, and privacy frameworks. How expressive is CMI in terms of explaining generalization?', 'Some results\n1- The CMI framework yields optimal risk bound for sample compression schemes. Covers many algorithms, including SVMs.\n2- A broad class of proper ERMs achieves CMI of order O(1) (without suboptimal log n factor) for learning VC classes with finite star number.', '3. Steinke and Zakynthinou introduced a variant of CMI called “evaluated"" CMI (eCMI). We show that, for every interpolating algorithm and data distribution, the expected risk vanishes as the number of samples (n) diverges if and only if its eCMI has sublinear growth with n.', 'Joint work with @kdziugaite, Shay Moran, and @roydanroy.', '@lzamparo @shortstein @zakynthinou Thanks!']",21,11,1162