Model selection for (auto-)regression with dependent data

Baraud, Yannick; Comte, F.; Viennet, G.

ESAIM: Probability and Statistics, Tome 5 (2001), pp. 33-49

In this paper, we study the problem of non parametric estimation of an unknown regression function from dependent data with sub-gaussian errors. As a particular case, we handle the autoregressive framework. For this purpose, we consider a collection of finite dimensional linear spaces (e.g. linear spaces spanned by wavelets or piecewise polynomials on a possibly irregular grid) and we estimate the regression function by a least-squares estimator built on a data driven selected linear space among the collection. This data driven choice is performed via the minimization of a penalized criterion akin to the Mallows’ $C_p$. We state non asymptotic risk bounds for our estimator in some ${Ł}_2$-norm and we show that it is adaptive in the minimax sense over a large class of Besov balls of the form ${ℬ}_{\alpha ,p,\infty }\left(R\right)$ with $p\ge 1$.

Classification : 62G08, 62J02
Keywords: nonparametric regression, least-squares estimator, adaptive estimation, autoregression, mixing processes

[1] H. Akaike, Information theory and an extension of the maximum likelihood principle, in Proc. 2nd International Symposium on Information Theory, edited by P.N. Petrov and F. Csaki. Akademia Kiado, Budapest (1973) 267-281. | Zbl | MR

[2] H. Akaike, A new look at the statistical model identification. IEEE Trans. Automat. Control 19 (1984) 716-723. | Zbl | MR

[3] P. Ango Nze, Geometric and subgeometric rates for markovian processes in the neighbourhood of linearity. C. R. Acad. Sci. Paris 326 (1998) 371-376. | Zbl | MR

[4] Y. Baraud, Model selection for regression on a fixed design. Probab. Theory Related Fields 117 (2000) 467-493. | Zbl | MR

[5] Y. Baraud, Model selection for regression on a random design, Preprint 01-10. DMA, École Normale Supérieure (2001). | MR | Numdam

[6] Y. Baraud, F. Comte and G. Viennet, Adaptive estimation in autoregression or $\beta$-mixing regression via model selection. Ann. Statist. (to appear). | Zbl | MR

[7] A. Barron, L. Birgé and P. Massart, Risks bounds for model selection via penalization. Probab. Theory Related Fields 113 (1999) 301-413. | Zbl | MR

[8] L. Birgé and P. Massart, An adaptive compression algorithm in Besov spaces. Constr. Approx. 16 (2000) 1-36. | Zbl | MR

[9] L. Birgé and Y. Rozenholc, How many bins must be put in a regular histogram. Working paper (2001).

[10] A. Cohen, I. Daubechies and P. Vial, Wavelet and fast wavelet transform on an interval. Appl. Comput. Harmon. Anal. 1 (1993) 54-81. | Zbl | MR

[11] I. Daubechies, Ten lectures on wavelets. SIAM: Philadelphia (1992). | Zbl | MR

[12] R.A. Devore and C.G. Lorentz, Constructive Approximation. Springer-Verlag (1993). | Zbl | MR

[13] D.L. Donoho and I.M. Johnstone, Minimax estimation via wavelet shrinkage. Ann. Statist. 26 (1998) 879-921. | Zbl | MR

[14] P. Doukhan, Mixing properties and examples. Springer-Verlag (1994). | Zbl | MR

[15] M. Duflo, Random Iterative Models. Springer, Berlin, New-York (1997). | Zbl | MR

[16] M. Hoffmann, On nonparametric estimation in nonlinear AR(1)-models. Statist. Probab. Lett. 44 (1999) 29-45. | Zbl | MR

[17] I.A. Ibragimov, On the spectrum of stationary Gaussian sequences satisfying the strong mixing condition I: Necessary conditions. Theory Probab. Appl. 10 (1965) 85-106. | Zbl | MR

[18] M. Kohler, On optimal rates of convergence for nonparametric regression with random design, Working Paper. Stuttgart University (1997).

[19] A.R. Kolmogorov and Y.A. Rozanov, On the strong mixing conditions for stationary Gaussian sequences. Theory Probab. Appl. 5 (1960) 204-207. | Zbl

[20] K.C. Li, Asymptotic optimality for $C_p$, $C_l$ cross-validation and generalized cross-validation: Discrete index set. Ann. Statist. 15 (1987) 958-975. | Zbl | MR

[21] G.G. Lorentz, M. Von Golitschek and Y. Makokov, Constructive Approximation, Advanced Problems. Springer, Berlin (1996). | Zbl | MR

[22] C.L. Mallows, Some comments on $C_p$. Technometrics 15 (1973) 661-675. | Zbl

[23] A. Meyer, Quelques inégalités sur les martingales d'après Dubins et Freedman, Séminaire de Probabilités de l'Université de Strasbourg. Vols. 68/69 (1969) 162-169. | Zbl | Numdam

[24] D.S. Modha and E. Masry, Minimum complexity regression estimation with weakly dependent observations. IEEE Trans. Inform. Theory 42 (1996) 2133-2145. | Zbl | MR

[25] D.S. Modha and E. Masry, Memory-universal prediction of stationary random processes. IEEE Trans. Inform. Theory 44 (1998) 117-133. | Zbl | MR

[26] M. Neumann and J.-P. Kreiss, Regression-type inference in nonparametric autoregression. Ann. Statist. 26 (1998) 1570-1613. | Zbl | MR

[27] B.T. Polyak and A. Tsybakov, A family of asymptotically optimal methods for choosing the order of a projective regression estimate. Theory Probab. Appl. 37 (1992) 471-481. | Zbl | MR

[28] R. Shibata, Selection of the order of an autoregressive model by Akaike's information criterion. Biometrika 63 (1976) 117-126. | Zbl

[29] R. Shibata, An optimal selection of regression variables. Biometrika 68 (1981) 45-54. | Zbl | MR

[30] S. Van De Geer, Exponential inequalities for martingales, with application to maximum likelihood estimation for counting processes. Ann. Statist. 23 (1995) 1779-1801. | Zbl | MR

[31] V.A. Volonskii and Y.A. Rozanov, Some limit theorems for random functions. I. Theory Probab. Appl. 4 (1959) 179-197. | Zbl | MR