Evaluation of autoregressive time series prediction using validity of cross-validation

Research paper

EVALUATION OF AUTOREGRESSIVE TIME SERIES PREDICTION USING VALIDITY OF CROSS-VALIDATION

Tags: Statswork | Cross-Validation | Time Series Prediction | Time Series Analysis | Statistical Model | Regression Analysis | Serial Correlation | Statistical Models | Time Series Data Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright © 2019 Statswrok. All rights reserved

Cross-Validation (CV)

CV is a validation technique used to explain how well the estimated values from a fitted statistical model will generalize to the explanatory variables under study.

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright Š 2019 Statswrok. All rights reserved

Why Cross Validation technique is different for time series data?

Standard procedure for model validation is the Kfold CV as in Regression Analysis and classification problem.

However, for predicting the time series data, this K-fold is not a straightforward because of the non-stationarity and serial correlation present in the data.

K-fold CV is applicable for the autoregressive models with uncorrelated errors and have wide scope in predicting the model using Machine Learning methods.

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright Š 2019 Statswrok. All rights reserved

Types of Cross Validation K-fold Cross Validation K-fold Cross Validation for time series for purely autoregressive models particularly for the dependent case instead of applying to the independent case.

01

Hold-out Cross Validation 02

Hold-out cross-validation technique is performed by splitting the data into training and test set.

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright Š 2019 Statswrok. All rights reserved

ILLUSTRATIVE EXAMPLE Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright © 2019 Statswrok. All rights reserved

01

Model selection adopted in that paper is 5-fold Cross Validation using leave-one-out, nondependent, and out-of-sample evaluation methods using AR(1) to AR(5) models and residuals have been taken care for the uncorrelated errors using Serial Correlation.

02

Three different experiments are carried out to forecast the AR model using Monte Carlo simulation.

03

The monthly seasonality from the ACF and PACF plots for the data considered for MonteCarlo simulation is depicted in the following figure and the results of the experiment is discussed.

04

Applicability of the procedure using the yearly sun spot data available in R. The data involves 289 observations recorded from 1700 to 1988 and are depicted in the following figure.

Contd...

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright Š 2019 Statswrok. All rights reserved

Results

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright Š 2019 Statswrok. All rights reserved

Relevant auto correlation is captured using Ljung-Box test with the residual series of 1560 model configurations.

Out of 1560, 763 satisfies the Box test and the configuration which have minimum root mean square error is chosen.

The 5-fold CV yields an error rate of 2.247 and out-of-sample (OOS) method yields 2.281 and the CV procedure is not an over fitting model and the results are tabulated.

Auto correlation

Discussions

Method

RMSE

5 fold CV

2,247

005

2,281

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright Š 2019 Statswrok. All rights reserved

Summary From the Monte-Carlo simulation, the K-fold classification yields better result than out-ofsample method for time series data and the real-time data explains the application of this using AR model with uncorrelated errors.

01

Discussed the applicability of the K-fold Cross Validation procedure for the purely AR models beyond the usual practice and it is found to be useful when the residuals in the model is uncorrelated.

02

Examines the applicability of the K-fold cross validation using neural network model with a monte-carlo simulations, & it can also be extended to the other regression models such as random forest and other machine learning techniques.

03

The over-fitting and under-fitting of the model is accessed using the Ljung-Box test and it results in there is no such issues occurred in the model.

04 05

The interesting point is that if we use whatever regression model for these kinds of Time Series Data, this method still proves the K-fold cross validation is the best method.

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright Š 2019 Statswrok. All rights reserved

Statswork Lab @ Statswork.com www.statswork.com

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright Š 2019 Statswrok. All rights reserved

PHONE NUMBER UK INDIA

: +44-1143520021 : +91-4448137070

EMAIL ADDRESS

Freelancer Consultant

info@statswork.com

Guest Blog Editor

GET IN TOUCH WITH US

CONTACT hr@workfoster.com

Research Planning | Data Collection | Semantic Annotation | Business Analytics | Bio Statistics | Econometrics

Copyright © 2019 Statswrok. All rights reserved