TARU PUBLICATIONS
Journal of Statistics and Management Systems cover

Journal of Statistics and Management Systems

(Since 1998)
Open Access ·Peer-reviewed·ISSN (Online): 2169-0014·ISSN (Print): 0972-0510
Powered by:Powered by

The Journal of Statistics and Management Systems (JSMS) is a world leading journal publishing high quality, rigorously peer-reviewed original research on theoretical and applied statistics and management systems since 1998. The scope is intentionally broad, but papers must make a novel contribution to the field to be considered for publication. Topics include, but are not limited to, the following: • Statistics • Applied Statistics • Industrial Statistics • Statistical Inference • Interdisciplinary role of Statistics • Actuarial Sciences • Decision Sciences • Managerial Aspects • Management Sciences • Management Information Systems

Issues up to 2022 co-published with and available at:Taylor & Francis Online
submissions@tarupublications.com
Taru Online First Current IssueSpecial IssuesBrowse all Issues
Journal of Statistics and Management Systems
Open Access Research Article

Statistical time-series forecast error and bias assessment using  hold-out samples and a single benchmark (data-reduction) scoring metric

*

* Corresponding author · click or hover a name for details

pp. 569–578Vol. 27Issue 3March 2024DOI: 10.47974/JSMS-974XML
Received:
09 Mar 2022
Published Online:
30 Mar 2024
Article type:
Research Article
Language:
EN
Article no.:
JSMS-974
Pages:
569–578

Abstract

In Time-Series (TS) forecast modeling, we utilize a Hold-Out (HO) sample to assess a candidate model forecast error and bias, with the Mean Absolute Percentage Error (MAPE) as a single benchmark (data-reduction) for a scoring metric. The purpose of such HO sampling is to assess error rates and accuracy (unbiasedness) level whether the forecasted values of a candidate TS model is within a pre-specified (targeted) margin-of-error (MOE) rate of α-percent. For instance, for an accuracy expectation of 80% level, the MOE rate of α is tolerated to be, 20%. If the MAPE of a candidate TS model is less than the MOE rate of α, then the model is considered reasonably accurate enough. Otherwise, we select the next available candidate TS model with the smallest-possible MAPE. For an illustration, we apply such method to a TS sales-data example.

Keywords

Time seriesHold-out sampleMean absolute percent error (MAPE)Margin of error (MOE)

Subject Classifications

(2010) 37M10 − Time series analysis62M10 – Time seriesauto-correlationRegressionetc91B84 − Economic time series analysis

References

[1] Sypsas, P.T. (1989). Identifying patterns in multiple time series data. Journal of Information and Optimization Sciences, 10(3): 471–494. 
[2] De Gooijer, J.G., Hyndman, R.J. (2006). 25 years of time series forecasting. International Journal of Forecasting, 22(3): 443–473. 
[3] Rahardja, D. (2020). Statistical methodological review for time-series data. Journal of Statistics and Management Systems, 23(8): 1445–1461. DOI: https://doi.org/10.1080/09720510.2020.1727618. 
[4] Rahardja, D. (2021a). Statistical Computing for Time-Series Forecasts via SAS Forecast Studio. International Journal of Applied Science and Mathematics, 8(4): 38–45. 
[5] Rahardja, D. (2021b). Statistical Time-Series Forecast via Microsoft Excel (FORECAST.ETS) Built-In Function. Quest Journals Publisher. Journal of Research in Applied Mathematics, 7(11): 69–73. 
[6] Holdout Sample, The Institute for Statistics Education, LLC (2021). © Copyright 2021 - Statistics.com, LLC. All Rights Reserved. Retrieve from https://www.statistics.com/glossary/hold-out-sample/. 
[7] Holdout Sample, Open Risk Manual (2016). Retrieve from https://www.openriskmanual.org/wiki/Holdout_Sample. 
[8] Holdout Sample, Oxford Reference (2021). Oxford University Press. Copyright © 2021. All rights reserved. Retrieve from https://www.oxfordreference.com/view/10.1093/oi/authority.20110803095941196. 
[9] Mean Absolute Percentage Error, MAPE, Wikipedia (2021). Last edited on 21 November 2021, at 23:28 (UTC). Retrieve from https://en.wikipedia.org/wiki/Mean_absolute_percentage_error. 
[10] Mean Absolute Percentage Error, MAPE (2000). In: Swamidass P.M. (eds) Encyclopedia of Production and Manufacturing Management. Springer, Boston, MA. https://doi.org/10.1007/1-4020-0612-8_580. 
[11] Exceljet (2021a). Excel FORECAST.ETS Function. © 2012-2021 Exceljet. Retrieve from https://exceljet.net/excel-functions/excel-forecast.ets-function. 
[12] Excel 2019 Support (2019). What’s new in Excel 2019 for Windows. © Microsoft 2021. https://support.microsoft.com/en-us/office/what-s-new-in-excel-2019-for-windows-5a201203-1155-4055-82a5-82bf0994631f. 
[13] Holdout Sample, KDD Analytics (2018). Practical Time Series Forecasting – Know When to Hold ‘em. Posted 8 January 2018. © 2021 KDD Analytics. Retrieve from https://www.kddanalytics.com/practical-time-series-forecasting-holdout-sample/. 
[14] Exceljet (2021b). Excel FORECAST.ETS.SEASONALITY Function. © 2012-2021 Exceljet. Retrieve from https://exceljet.net/excel-functions/excel-forecast.ets.seasonality-function. 
[15] Polhamus, Aaron (2015). Validation with holdout sampling — why you (or your data provider) better be doing it. Retrieve from https://aaronpolhamus.medium.com/validation-with-holdout-sampling-why-you-or-your-data-provider-better-be-doing-it-bdc3a838c5f5. 
[16] Akaike Information Criteria (AIC), Wikipedia (2020). Retrieve from https://en.wikipedia.org/wiki/Akaike_information_criterion. 
[17] Akaike, H. (21 December 1981), “This Week’s Citation Classic” (PDF), Current Contents Engineering, Technology, and Applied Sciences, 12(51): 42 [Hirotogu Akaike comments on how he arrived at AIC].    
[18] Schwarz Information Criteria (SIC), Wikipedia (2020). Retrieve from https://en.wikipedia.org/wiki/Bayesian_information_criterion. 
[19] Bhat, H. S., and Kumar, N (2010). On the derivation of the Bayesian Information Criterion. Archived from the original on 28 March 2012. https://web.archive.org/web/20120328065032/http://nscs00.ucmerced.edu/~nkumar4/BhatKumarBIC.pdf. 
[20] Findley, D. F. (1991). “Counterexamples to parsimony and BIC”. Annals of the Institute of Statistical Mathematics, 43(3): 505–514. DOI:10.1007/BF00053369.
[21] Kass, R. E.; Wasserman, L. (1995). “A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion”. Journal of the American Statistical Association, 90(431): 928–934. DOI:10.2307/2291327. JSTOR 2291327. 
[22] Hannan, E. J., and B. G. Quinn (1979). The Determination of the order of an autoregression. Journal of the Royal Statistical Society, Series B, 41: 190–195. 
[23] Hannan-Quinn Information Criteria (HQC), Wikipedia (2020). Retrieve from https://en.wikipedia.org/wiki/Hannan%E2%80%93Quinn_information_criterion. 
[24] Gilliland, Michael (2010). Forecasting FAQ. The Business Forecasting Deal: Exposing Myths, Eliminating Bad Practices, Providing Practical Solutions. Copyright © 2010 by SAS Institute, Inc. Retrieve from https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781119199885.app1. Also available: https://www.coursehero.com/file/115043927/9781119199885app1pdf/. 
[25] SAS Institute Inc. (1995). SAS manual SAS/ETS Software: Time series forecasting system, Version 6, First Edition: 225–235. Cary, NC: SAS Institute Inc.
[26] SAS Institute Inc. (2013). Base SAS® 9.4 Procedures Guide: Statistical procedures, Second edition. Cary, NC: SAS Institute Inc.
[27] SAS® Help Center (2017). The ESM Procedure. Copyright © SAS Institute Inc. All rights reserved. SAS® 9.4 and SAS® Viya® 3.3 Programming Documentation. SAS Documentation website: https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/etsug/etsug_esm_syntax02.htm. 

Views: 178Downloads: 9Citations: 0