Model selection for regression on a random design
ESAIM: Probability and Statistics, Tome 6 (2002), pp. 127-146

We consider the problem of estimating an unknown regression function when the design is random with values in ℝ k . Our estimation procedure is based on model selection and does not rely on any prior information on the target function. We start with a collection of linear functional spaces and build, on a data selected space among this collection, the least-squares estimator. We study the performance of an estimator which is obtained by modifying this least-squares estimator on a set of small probability. For the so-defined estimator, we establish nonasymptotic risk bounds that can be related to oracle inequalities. As a consequence of these, we show that our estimator possesses adaptive properties in the minimax sense over large families of Besov balls ℬ α,l,∞ (R) with R>0, l≥1 and α>α l where α l is a positive number satisfying 1/l-1/2≤α l <1/l. We also study the particular case where the regression function is additive and then obtain an additive estimator which converges at the same rate as it does when k=1.

DOI : 10.1051/ps:2002007
Classification : 62G07, 62J02
Keywords: nonparametric regression, least-squares estimators, penalized criteria, minimax rates, Besov spaces, model selection, adaptive estimation 
@article{PS_2002__6__127_0,
     author = {Baraud, Yannick},
     title = {Model selection for regression on a random design},
     journal = {ESAIM: Probability and Statistics},
     pages = {127--146},
     year = {2002},
     publisher = {EDP Sciences},
     volume = {6},
     doi = {10.1051/ps:2002007},
     mrnumber = {1918295},
     zbl = {1059.62038},
     language = {en},
     url = {https://www.numdam.org/articles/10.1051/ps:2002007/}
}
TY  - JOUR
AU  - Baraud, Yannick
TI  - Model selection for regression on a random design
JO  - ESAIM: Probability and Statistics
PY  - 2002
SP  - 127
EP  - 146
VL  - 6
PB  - EDP Sciences
UR  - https://www.numdam.org/articles/10.1051/ps:2002007/
DO  - 10.1051/ps:2002007
LA  - en
ID  - PS_2002__6__127_0
ER  - 
%0 Journal Article
%A Baraud, Yannick
%T Model selection for regression on a random design
%J ESAIM: Probability and Statistics
%D 2002
%P 127-146
%V 6
%I EDP Sciences
%U https://www.numdam.org/articles/10.1051/ps:2002007/
%R 10.1051/ps:2002007
%G en
%F PS_2002__6__127_0
Baraud, Yannick. Model selection for regression on a random design. ESAIM: Probability and Statistics, Tome 6 (2002), pp. 127-146. doi: 10.1051/ps:2002007

[1] Y. Baraud, Model selection for regression on a fixed design. Probab. Theory Related Fields 117 (2000) 467-493. | Zbl | MR

[2] A. Barron, L. Birgé and P. Massart, Risk bounds for model selection via penalization. Probab. Theory Related Fields 113 (1999) 301-413. | Zbl | MR

[3] A.R. Barron and T.M. Cover, Minimum complexity density estimation. IEEE Trans. Inform. Theory 37 (1991) 1738. | Zbl | MR

[4] L. Birgé and P. Massart, An adaptive compression algorithm in Besov spaces. Constr. Approx. 16 (2000) 1-36. | Zbl | MR

[5] L. Birgé and P. Massart, Minimum contrast estimators on sieves: Exponential bounds and rates of convergence. Bernoulli 4 (1998) 329-375. | Zbl | MR

[6] L. Birgé and P. Massart, Gaussian model selection. JEMS 3 (2001) 203-268. | Zbl | MR | EuDML

[7] L. Birgé and Massart, A generalized C p criterion for Gaussian model selection, Technical Report. University Paris 6, PMA-647 (2001).

[8] L. Birgé and Y. Rozenholc, How many bins should be put in a regular histogram, Technical Report. University Paris 6, PMA-721 (2002).

[9] O. Catoni, Statistical learning theory and stochastic optimization, in École d'été de probabilités de Saint-Flour. Springer (2001). | Zbl

[10] A. Cohen, I. Daubechies and P. Vial, Wavelet and fast wavelet transform on an interval. Appl. Comp. Harmon. Anal. 1 (1993) 54-81. | Zbl | MR

[11] I. Daubechies, Ten lectures on wavelets. SIAM: Philadelphia (1992). | Zbl | MR

[12] R.A. Devore and G.G. Lorentz, Constructive approximation. Springer-Verlag, Berlin (1993). | Zbl | MR

[13] D.L. Donoho and I.M. Johnstone, Ideal spatial adaptation via wavelet shrinkage. Biometrika 81 (1994) 425-455. | Zbl | MR

[14] D.L. Donoho and I.M. Johnstone, Minimax estimation via wavelet shrinkage. Ann. Statist. 26 (1998) 879-921. | Zbl | MR

[15] M. Kohler, Inequalities for uniform deviations of averages from expectations with applications to nonparametric regression. J. Statist. Plann. Inference 89 (2000) 1-23. | Zbl | MR

[16] M. Kohler, Nonparametric regression function estimation using interaction least square splines and complexity regularization. Metrika 47 (1998) 147-163. | Zbl | MR

[17] A.P. Korostelev and A.B. Tsybakov, Minimax theory of image reconstruction. Springer-Verlag, New York NY, Lecture Notes in Statis. (1993). | Zbl | MR

[18] C.J. Stone, Additive regression and other nonparametric models. Ann. Statist. 13 (1985) 689-705. | Zbl | MR

[19] M. Wegkamp, Model selection in non-parametric regression, Preprint. Yale University (2000). | Zbl

[20] Y. Yang, Model selection for nonparametric regression. Statist. Sinica 9 (1999) 475-499. | Zbl | MR

[21] Y. Yang, Combining different procedures for adaptive regression. J. Multivariate Anal. 74 (2000) 135-161. | Zbl | MR

[22] Y. Yang and A. Barron, Information-Theoretic determination of minimax rates of convergence. Ann. Statist. 27 (1999) 1564-1599. | Zbl | MR

Cité par Sources :