Skip to main content

Advertisement

Log in

Coherent quality management for big data systems: a dynamic approach for stochastic time consistency

  • S.I.: Reliability and Quality Management in Stochastic Systems
  • Published:
Annals of Operations Research Aims and scope Submit manuscript

Abstract

Big data systems for reinforcement learning have often exhibited problems (e.g., failures or errors) when their components involve stochastic nature with the continuous control actions of reliability and quality. The complexity of big data systems and their stochastic features raise the challenge of uncertainty. This article proposes a dynamic coherent quality measure focusing on an axiomatic framework by characterizing the probability of critical errors that can be used to evaluate if the conveyed information of big data interacts efficiently with the integrated system (i.e., system of systems) to achieve desired performance. Herein, we consider two new measures that compute the higher-than-expected error,—that is, the tail error and its conditional expectation of the excessive error (conditional tail error)—as a quality measure of a big data system. We illustrate several properties (that suffice stochastic time-invariance) of the proposed dynamic coherent quality measure for a big data system. We apply the proposed measures in an empirical study with three wavelet-based big data systems in monitoring and forecasting electricity demand to conduct the reliability and quality management in terms of minimizing decision-making errors. Performance of using our approach in the assessment illustrates its superiority and confirms the efficiency and robustness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Notes

  1. These are layers of hardware, firmware, software applications, operating platforms, and networks that make up IT architecture.

  2. A six sigma process is one in which 99.99966% of all opportunities are statistically expected to be free of defects (i.e., 3.4 defective features per million opportunities).

  3. PTC white paper. PTC is a global provider of technology platforms and solutions that transform how companies create, operate, and service the “things” in the Internet of Things (IoT). See www.ptc.com.

  4. See www.rte-france.com.

  5. Not because of making a round number.

References

  • Agarwal, R., Green, R., Brown, P., Tan, H., & Randhawa, K. (2013). Determinants of quality management practices: An empirical study of New Zealand manufacturing firms. International Journal of Production Economics, 142, 130–145.

    Article  Google Scholar 

  • Artzner, P., Delbaen, F., Eber, J., Heath, D., & Ku, K. (2007). Coherent multiperiod risk adjusted values and Bellman’s principle. Annals of Operations Research, 152, 5–22.

    Article  Google Scholar 

  • Baucells, M., & Borgonovo, E. (2013). Invariant probabilistic sensitivity analysis. Management Science, 59(11), 2536–2549.

    Article  Google Scholar 

  • Bion-Nadal, J. (2008). Dynamic risk measures: Time consistency and risk measures from BMO martingales. Finance and Stochastics, 12(2), 219–244.

    Article  Google Scholar 

  • Bion-Nadal, J. (2009). Time consistent dynamic risk processes. Stochastic Processes and their Applications, 119(2), 633–654.

    Article  Google Scholar 

  • Chen, Y., & Sun, E. (2015). Jump detection and noise separation with singular wavelet method for high-frequency data. Working paper of KEDGE BS.

  • Chen, Y., & Sun, E. (2018). Chapter 8: Automated business analytics for artificial intelligence in big data \(@\)x 4.0 era. In M. Dehmer & F. Emmert-Streib (Eds.), Frontiers in Data Science (pp. 223–251). Boca Raton: CRC Press.

    Google Scholar 

  • Chen, Y., Sun, E., & Yu, M. (2015). Improving model performance with the integrated wavelet denoising method. Studies in Nonlinear Dynamics and Econometrics, 19(4), 445–467.

    Google Scholar 

  • Chen, Y., Sun, E., & Yu, M. (2017). Risk assessment with wavelet feature engineering for high-frequency portfolio trading. Computational Economics. https://doi.org/10.1007/s10614-017-9711-7.

  • Cheridito, P., & Stadje, M. (2009). Time-inconsistency of VaR and time-consistent alternatives. Finance Research Letters, 6, 40–46.

    Article  Google Scholar 

  • Chun, S., Shapiro, A., & Uryasev, S. (2012). Conditional value-at-risk and average value-at-risk: Estimation and asymptotics. Operations Research, 60(4), 739–756.

    Article  Google Scholar 

  • David, H., & Nagaraja, H. (2003). Order statistics (3rd ed.). Hoboken: Wiley.

    Book  Google Scholar 

  • Deichmann, J., Roggendorf, M., & Wee, D. (2015). McKinsey quarterly november: Preparing IT systems and organizations for the Internet of Things. McKinsey & Company.

  • Hazen, B., Boone, C., Ezell, J., & Jones-Farmer, J. (2014). Data quality for data science, predictive analytics, and big data in supply chain management: An introduction to the problem and suggestions for research and applications. International Journal of Production Economics, 154, 72–80.

    Article  Google Scholar 

  • Keating, C., & Katina, P. (2011). Systems of systems engineering: Prospects and challenges for the emerging field. International Journal of System of Systems Engineering, 2(2/3), 234–256.

    Article  Google Scholar 

  • Liu, Y., Muppala, J., Veeraraghavan, M., Lin, D., & Hamdi, M. (2013). Data center networks: Topologies architechtures and fault-tolerance characteristics. Berlin: Springer.

    Book  Google Scholar 

  • Maier, M. (1998). Architecting principles for systems-of-systems. Systems Engineering, 1(4), 267–284.

    Article  Google Scholar 

  • Mellat-Parst, M., & Digman, L. (2008). Learning: The interface of quality management and strategic alliances. International Journal of Production Economics, 114, 820–829.

    Article  Google Scholar 

  • O’Neill, P., Sohal, A., & Teng, W. (2015). Quality management approaches and their impact on firms’ financial performance—An Australian study. International Journal of Production Economics. https://doi.org/10.1016/j.ijpe.2015.07.015i.

  • Parast, M., & Adams, S. (2012). Corporate social responsibility, benchmarking, and organizational performance in the petroleum industry: A quality management perspective. International Journal of Production Economics, 139, 447–458.

    Article  Google Scholar 

  • Pham, H. (2006). System software reliability. Berlin: Springer.

    Book  Google Scholar 

  • Riedel, F. (2004). Dynamic coherent risk measures. Stochastic Processes and their Applications, 112(2), 185–200.

    Article  Google Scholar 

  • Shooman, M. (2002). Reliability of computer systems and networks: Fault tolerance analysis and design. Hoboken: Wiley.

    Book  Google Scholar 

  • Sun, E., Chen, Y., & Yu, M. (2015). Generalized optimal wavelet decomposing algorithm for big financial data. International Journal of Production Economics, 165, 161–177.

    Google Scholar 

  • Sun, E., & Meinl, T. (2012). A new wavelet-based denoising algorithm for high-frequency financial data mining. European Journal of Operational Research, 217, 589–599.

    Article  Google Scholar 

  • Sun, W., Rachev, S., & Fabozzi, F. (2007). Fractals or I.I.D.: Evidence of long-range dependence and heavy tailedness from modeling German equity market returns. Journal of Economics and Business, 59, 575–595.

    Article  Google Scholar 

  • Sun, W., Rachev, S., & Fabozzi, F. (2009). A new approach for using Lèvy processes for determining high-frequency value-at-risk predictions. European Financial Management, 15(2), 340–361.

    Article  Google Scholar 

  • Wu, S., & Zhang, D. (2013). Analyzing the effectiveness of quality management practices in China. International Journal of Production Economics, 144, 281–289.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the three anonymous reviewers and the guest editor for providing valuable comments. This work was supported in part by the Ministry of Science and Technology (MOST) under Grant 106-2221-E-009-006 and Grant 106-2221-E-009-049-MY2, in part by the “Aiming for the Top University Program” of National Chiao Tung University and the Ministry of Education, Taiwan, and in part by Academia Sinica AS-105-TP-A07 and Ministry of Economic Affairs (MOEA) 106-EC-17-A-24-0619.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Edward W. Sun.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, YT., Sun, E.W. & Lin, YB. Coherent quality management for big data systems: a dynamic approach for stochastic time consistency. Ann Oper Res 277, 3–32 (2019). https://doi.org/10.1007/s10479-018-2795-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10479-018-2795-1

Keywords

JEL Classification

Profiles

  1. Edward W. Sun