We consider a finite mixture of Gaussian regression models for high-dimensional heterogeneous data where the number of covariates may be much larger than the sample size. We propose to estimate the unknown conditional mixture density by an ℓ1-penalized maximum likelihood estimator. We shall provide an ℓ1-oracle inequality satisfied by this Lasso estimator with the Kullback-Leibler loss. In particular, we give a condition on the regularization parameter of the Lasso to obtain such an oracle inequality. Our aim is twofold: to extend the ℓ1-oracle inequality established by Massart and Meynet [12] in the homogeneous Gaussian linear regression case, and to present a complementary result to Städler et al. [18], by studying the Lasso for its ℓ1-regularization properties rather than considering it as a variable selection procedure. Our oracle inequality shall be deduced from a finite mixture Gaussian regression model selection theorem for ℓ1-penalized maximum likelihood conditional density estimation, which is inspired from Vapnik's method of structural risk minimization [23] and from the theory on model selection for maximum likelihood estimators developed by Massart in [11].
Keywords: finite mixture of gaussian regressions model, Lasso, ℓ1-oracle inequalities, model selection by penalization, ℓ1-balls
@article{PS_2013__17__650_0,
author = {Meynet, Caroline},
title = {An $\ell _1$-oracle inequality for the {Lasso} in finite mixture gaussian regression models},
journal = {ESAIM: Probability and Statistics},
pages = {650--671},
year = {2013},
publisher = {EDP Sciences},
volume = {17},
doi = {10.1051/ps/2012016},
language = {en},
url = {https://www.numdam.org/articles/10.1051/ps/2012016/}
}
TY - JOUR AU - Meynet, Caroline TI - An $\ell _1$-oracle inequality for the Lasso in finite mixture gaussian regression models JO - ESAIM: Probability and Statistics PY - 2013 SP - 650 EP - 671 VL - 17 PB - EDP Sciences UR - https://www.numdam.org/articles/10.1051/ps/2012016/ DO - 10.1051/ps/2012016 LA - en ID - PS_2013__17__650_0 ER -
%0 Journal Article %A Meynet, Caroline %T An $\ell _1$-oracle inequality for the Lasso in finite mixture gaussian regression models %J ESAIM: Probability and Statistics %D 2013 %P 650-671 %V 17 %I EDP Sciences %U https://www.numdam.org/articles/10.1051/ps/2012016/ %R 10.1051/ps/2012016 %G en %F PS_2013__17__650_0
Meynet, Caroline. An $\ell _1$-oracle inequality for the Lasso in finite mixture gaussian regression models. ESAIM: Probability and Statistics, Tome 17 (2013), pp. 650-671. doi: 10.1051/ps/2012016
[1] , and , ℓ1-regularized linear regression: persistence and oracle inequalities, Probability and related fields. Springer (2011).
[2] , Sélection de Modèle pour la Classification Non Supervisée. Choix du Nombre de Classes. Ph.D. thesis, Université Paris-Sud 11, France (2009).
[3] , and , Simultaneous analysis of Lasso and Dantzig selector. Ann. Stat. 37 (2009) 1705-1732. | Zbl | MR
[4] , and , A non Asymptotic Theory of Independence. Oxford University press (2013). | Zbl | MR
[5] and , On the conditions used to prove oracle results for the Lasso. Electr. J. Stat. 3 (2009) 1360-1392. | MR
[6] and , The Dantzig selector: statistical estimation when p is much larger than n. Ann. Stat. 35 (2007) 2313-2351. | Zbl | MR
[7] and , Conditional Density Estimation by Penalized Likelihood Model Selection and Applications, RR-7596. INRIA (2011).
[8] , , and , Least Angle Regression. Ann. Stat. 32 (2004) 407-499. | Zbl | MR
[9] , Quelques questions de sélection de variables autour de l'estimateur Lasso. Ph.D. Thesis, Université Paris Diderot, Paris 7, France (2009).
[10] , and , Risk of penalized least squares, greedy selection and ℓ1-penalization for flexible function librairies. Submitted to the Annals of Statistics (2008). | MR
[11] , Concentration inequalities and model selection. Ecole d'été de Probabilités de Saint-Flour 2003. Lect. Notes Math. Springer, Berlin-Heidelberg (2007). | Zbl | MR
[12] and , The Lasso as an ℓ1-ball model selection procedure. Elect. J. Stat. 5 (2011) 669-687. | Zbl | MR
[13] and , A non asymptotic penalized criterion for Gaussian mixture model selection. ESAIM: PS 15 (2011) 41-68. | MR | Numdam
[14] and , Finite Mixture Models. Wiley, New York (2000). | Zbl | MR
[15] and , Lasso type recovery of sparse representations for high dimensional data. Ann. Stat. 37 (2009) 246-270. | Zbl | MR
[16] and , Mixture densities, maximum likelihood and the EM algorithm. SIAM Rev. 26 (1984) 195-239. | Zbl | MR
[17] and , Exponential screening and optimal rates of sparse estimation. Ann. Stat. 39 (2011) 731-771. | Zbl | MR
[18] , , and , ℓ1-penalization for mixture regression models. Test 19 (2010) 209-256. | Zbl
[19] , Regression shrinkage and selection via the Lasso. J. Roy. Stat. Soc. Ser. B 58 (1996) 267-288. | Zbl | MR
[20] , and , On the Lasso and its dual. J. Comput. Graph. Stat. 9 (2000) 319-337. | MR
[21] , and , A new approach to variable selection in least squares problems. IMA J. Numer. Anal. 20 (2000) 389-404. | Zbl | MR
[22] and , Weak Convergence and Empirical Processes. Springer, Berlin (1996). | Zbl | MR
[23] , Estimation of Dependencies Based on Empirical Data. Springer, New-York (1982). | Zbl | MR
[24] , Statistical Learning Theory. J. Wiley, New-York (1990). | Zbl | MR
[25] and On model selection consistency of Lasso. J. Mach. Learn. Res. 7 (2006) 2541-2563. | Zbl | MR
Cité par Sources :
