Skip to main content
SpringerLink
Log in
Book cover

Machine Learning for Health Informatics pp 357–376Cite as

Reasoning Under Uncertainty: Towards Collaborative Interactive Machine Learning

  • Chapter
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9605))

Abstract

In this paper, we present the current state-of-the-art of decision making (DM) and machine learning (ML) and bridge the two research domains to create an integrated approach of complex problem solving based on human and computational agents. We present a novel classification of ML, emphasizing the human-in-the-loop in interactive ML (iML) and more specific on collaborative interactive ML (ciML), which we understand as a deep integrated version of iML, where humans and algorithms work hand in hand to solve complex problems. Both humans and computers have specific strengths and weaknesses and integrating humans into machine learning processes might be a very efficient way for tackling problems. This approach bears immense research potential for various domains, e.g., in health informatics or in industrial applications. We outline open questions and name future challenges that have to be addressed by the research community to enable the use of collaborative interactive machine learning for problem solving in a large scale.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    http://www.wekinator.org/.

  2. 2.

    See also [44] for the chapter.

References

  1. Holzinger, A.: Trends in interactive knowledge discovery for personalized medicine: cognitive science meets machine learning. IEEE Intell. Inform. Bull. 15, 6–14 (2014)

    Google Scholar 

  2. Von Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior. Princeton University Press, Princeton (1944)

    MATH  Google Scholar 

  3. Fox, J., Glasspool, D., Bury, J.: Quantitative and qualitative approaches to reasoning under uncertainty in medical decision making. In: Quaglini, S., Barahona, P., Andreassen, S. (eds.) AIME 2001. LNCS, vol. 2101, pp. 272–282. Springer, Heidelberg (2001). doi:10.1007/3-540-48229-6_39

    Chapter  Google Scholar 

  4. Ma, W., Xiong, W., Luo, X.: A model for decision making with missing, imprecise, and uncertain evaluations of multiple criteria. Int. J. Intell. Syst. 28, 152–184 (2013)

    Article  Google Scholar 

  5. Simon, H.A.: A behavioral model of rational choice. Q. J. Econ. 69, 99–118 (1955)

    Article  Google Scholar 

  6. Dequech, D.: Bounded rationality, institutions, and uncertainty. J. Econ. Issues 35, 911–929 (2001)

    Article  Google Scholar 

  7. Holzinger, A.: Lecture 8 biomedical decision making: reasoning and decision support. In: Biomedical Informatics, pp. 345–377. Springer, Heidelberg (2014)

    Google Scholar 

  8. March, S.T., Hevner, A.R.: Integrated decision support systems: a data warehousing perspective. Decis. Support Syst. 43, 1031–1043 (2007)

    Article  Google Scholar 

  9. Hansson, S.O.: Decision theory: a brief introduction (2005)

    Google Scholar 

  10. Bell, D.E., Raiffa, H., Tversky, A.: Descriptive, normative, and prescriptive interactions in decision making. Decis. Making Descriptive Normative Prescriptive Interact. 1, 9–32 (1988)

    Article  MATH  Google Scholar 

  11. Pearl, J.: Heuristics: Intelligent Search Strategies for Computer Problem Solving. Addison-Wesley, Reading (1984)

    Google Scholar 

  12. Alan, D., Janet, F., Gregory, A., Russell, B.: Human-Computer Interaction. Pearson Education Limited, Harlow (2004)

    MATH  Google Scholar 

  13. Kohavi, R., Provost, F.: Glossary of terms. Mach. Learn. 30, 271–274 (1998)

    Article  Google Scholar 

  14. Ankerst, M., Elsen, C., Ester, M., Kriegel, H.P.: Visual classification: an interactive approach to decision tree construction. In: Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 392. ACM (1999)

    Google Scholar 

  15. Wakker, P., Deneffe, D.: Eliciting von neumann-morgenstern utilities when probabilities are distorted or unknown. Manage. Sci. 42, 1131–1150 (1996)

    Article  MATH  Google Scholar 

  16. Domingos, P.: A few useful things to know about machine learning. Commun. ACM 55, 78–87 (2012)

    Article  Google Scholar 

  17. Mitchell, T.M.: Machine Learning. McGraw-Hill, Boston (1997)

    MATH  Google Scholar 

  18. Martin, J.H., Jurafsky, D.: Speech and language processing. In: International 710th edn. (2000)

    Google Scholar 

  19. Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17, 734–749 (2005)

    Article  Google Scholar 

  20. Li, Q., Zheng, N., Cheng, H.: Springrobot: a prototype autonomous vehicle and its algorithms for lane detection. IEEE Trans. Intell. Transp. Syst. 5, 300–308 (2004)

    Article  Google Scholar 

  21. Marsland, S.: Machine Learning: An Algorithmic Perspective. CRC Press, Boca Raton (2015)

    Google Scholar 

  22. Ankerst, M., Ester, M., Kriegel, H.P.: Towards an effective cooperation of the user and the computer for classification. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 179–188. ACM (2000)

    Google Scholar 

  23. Ware, M., Frank, E., Holmes, G., Hall, M., Witten, I.H.: Interactive machine learning: letting users build classifiers. Int. J. Hum. Comput. Stud. 55, 281–292 (2001)

    Article  MATH  Google Scholar 

  24. Fails, J.A., Olsen Jr., D.R.: Interactive machine learning. In: Proceedings of the 8th International Conference on Intelligent User Interfaces, pp. 39–45. ACM (2003)

    Google Scholar 

  25. Fiebrink, R., Cook, P.R., Trueman, D.: Human model evaluation in interactive supervised learning. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2011, pp. 147–156. ACM, New York (2011)

    Google Scholar 

  26. Fogarty, J., Tan, D., Kapoor, A., Winder, S.: Cueflik: interactive concept learning in image search. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2008, pp. 29–38. ACM, New York (2008)

    Google Scholar 

  27. Simard, P., Chickering, D., Lakshmiratan, A., Charles, D., Bottou, L., Suarez, C.G.J., Grangier, D., Amershi, S., Verwey, J., Suh, J.: Ice: enabling non-experts to build models interactively for large-scale lopsided problems. arXiv preprint arXiv:1409.4814 (2014)

  28. Amershi, S., Chickering, M., Drucker, S.M., Lee, B., Simard, P., Suh, J.: Modeltracker: redesigning performance analysis tools for machine learning. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 2015, pp. 337–346. ACM, New York (2015)

    Google Scholar 

  29. Talbot, J., Lee, B., Kapoor, A., Tan, D.S.: Ensemblematrix: interactive visualization to support machine learning with multiple classifiers. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2009, pp. 1283–1292. ACM, New York (2009)

    Google Scholar 

  30. Ankerst, M., Ester, M., Kriegel, H.P.: Towards an effective cooperation of the user and the computer for classification. In: Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2000, pp. 179–188. ACM, New York (2000)

    Google Scholar 

  31. Holzinger, A.: Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inform. 3, 119–131 (2016)

    Article  Google Scholar 

  32. Holzinger, A., Plass, M., Holzinger, K., Crişan, G.C., Pintea, C.-M., Palade, V.: Towards interactive Machine Learning (iML): applying ant colony algorithms to solve the traveling salesman problem with the human-in-the-loop approach. In: Buccafurri, F., Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) CD-ARES 2016. LNCS, vol. 9817, pp. 81–95. Springer, Heidelberg (2016). doi:10.1007/978-3-319-45507-5_6

    Chapter  Google Scholar 

  33. Baron, J.: Normative Models of Judgment and Decision Making. Wiley, New York (2004)

    Book  Google Scholar 

  34. Raiffa, H.: Applied statistical decision theory (1974)

    Google Scholar 

  35. Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge (2012)

    MATH  Google Scholar 

  36. Friedman, J., Hastie, T., Tibshirani, R.: The Elements of Statistical Learning. Springer Series in Statistics, vol. 1. Springer, New York (2001)

    MATH  Google Scholar 

  37. Tulabandhula, T., Rudin, C.: Machine learning with operational costs. J. Mach. Learn. Res. 14, 1989–2028 (2013)

    MathSciNet  MATH  Google Scholar 

  38. Pitz, G.F., Sachs, N.J.: Judgment and decision: theory and application. Annu. Rev. Psychol. 35, 139–164 (1984)

    Article  Google Scholar 

  39. Fischhoff, B.: Judgment and decision making. Wiley Interdisc. Rev. Cogn. Sci. 1, 724–735 (2010)

    Article  Google Scholar 

  40. Russakovsky, O., Li, L.J., Fei-Fei, L.: Best of both worlds: human-machine collaboration for object annotation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2121–2131 (2015)

    Google Scholar 

  41. Rapoport, A.: Decision Theory and Decision Behaviour: Normative and Descriptive Approaches, vol. 15. Springer, Amsterdam (2013)

    MATH  Google Scholar 

  42. Bazerman, M.H., Moore, D.A.: Judgment in managerial decision making (2013)

    Google Scholar 

  43. Bonner, S.E.: Judgment and Decision Making in Accounting. Prentice Hall, Upper Saddle River (2008)

    Google Scholar 

  44. Robert, S.: Informationsverarbeitung in Prognosen: Experimentelle Evidenz. dissertation, University of Osnabrueck (2016)

    Google Scholar 

  45. Goldstein, W.M., Hogarth, R.M.: Research on Judgment and Decision Making: Currents, Connections, and Controversies. Cambridge University Press, Cambridge (1997)

    Google Scholar 

  46. Milkman, K.L., Chugh, D., Bazerman, M.H.: How can decision making be improved? Perspect. Psychol. Sci. 4, 379–383 (2009)

    Article  Google Scholar 

  47. Baron, J.: Thinking and Deciding. Cambridge University Press, Cambridge (2000)

    Google Scholar 

  48. Tversky, A., Kahneman, D.: Judgment under uncertainty: heuristics and biases. In: Wendt, D., Vlek, C. (eds.) Utility, Probability, and Human Decision Making, pp. 1124–1131. Springer, Amsterdam (1974)

    Google Scholar 

  49. Libby, R.: Accounting and Human Information Processing: Theory and Applications. Prentice Hall, Englewood Cliffs (1981)

    Google Scholar 

  50. Ashton, R.H.: Human Information Processing in Accounting. American Accounting Association, Sarasota (1982)

    Google Scholar 

  51. Over, D.: Rationality and the normative/descriptive distinction. In: Blackwell Handbook of Judgment and Decision Making, London, pp. 3–18 (2004)

    Google Scholar 

  52. Gilovich, T., Griffin, D., Kahneman, D.: Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University Press, New York (2002)

    Book  Google Scholar 

  53. Newell, B.R.: Judgment under uncertainty (2013)

    Google Scholar 

  54. Tversky, A., Kahneman, D.: Extensional versus intuitive reasoning: the conjunction fallacy in probability judgment. Psychol. Rev. 90, 293 (1983)

    Article  Google Scholar 

  55. Tversky, A., Kahneman, D.: Availability: a heuristic for judging frequency and probability. Cogn. Psychol. 5, 207–232 (1973)

    Article  Google Scholar 

  56. Mokdad, A.H., Marks, J.S., Stroup, D.F., Gerberding, J.L.: Actual causes of death in the United States, 2000. JAMA 291, 1238–1245 (2004)

    Article  Google Scholar 

  57. Strack, F., Mussweiler, T.: Explaining the enigmatic anchoring effect: mechanisms of selective accessibility. J. Pers. Soc. Psychol. 73, 437 (1997)

    Article  Google Scholar 

  58. Plous, S.: Thinking the unthinkable: the effects of anchoring on likelihood estimates of nuclear war1. J. Appl. Soc. Psychol. 19, 67–91 (1989)

    Article  Google Scholar 

  59. Ritov, I.: Anchoring in simulated competitive market negotiation. Organ. Behav. Hum. Decis. Process. 67, 16–25 (1996)

    Article  Google Scholar 

  60. Galinsky, A.D., Mussweiler, T.: First offers as anchors: the role of perspective-taking and negotiator focus. J. Pers. Soc. Psychol. 81, 657 (2001)

    Article  Google Scholar 

  61. Chapman, G.B., Johnson, E.J.: Incorporating the irrelevant: anchors in judgments of belief and value. In: The Psychology of Intuitive Judgment, Heuristics and Biases, pp. 120–138 (2002)

    Google Scholar 

  62. Wilson, T.D., Houston, C.E., Etling, K.M., Brekke, N.: A new look at anchoring effects: basic anchoring and its antecedents. J. Exp. Psychol. Gen. 125, 387 (1996)

    Article  Google Scholar 

  63. Mussweiler, T., Strack, F.: Comparing is believing: a selective accessibility model of judgmental anchoring. Eur. Rev. Soc. Psychol. 10, 135–167 (1999)

    Article  Google Scholar 

  64. Chapman, G.B., Johnson, E.J.: Anchoring, activation, and the construction of values. Organ. Behav. Hum. Decis. Process. 79, 115–153 (1999)

    Article  Google Scholar 

  65. Furnham, A., Boo, H.C.: A literature review of the anchoring effect. J. Socio-Econ. 40, 35–42 (2011)

    Article  Google Scholar 

  66. Gigerenzer, G.: Why the distinction between single-event probabilities and frequencies is important for psychology (and vice versa). In: Subjective Probability, pp. 129–161 (1994)

    Google Scholar 

  67. Gigerenzer, G., Czerlinski, J., Martignon, L.: How good are fast and frugal heuristics? In: Shanteau, J., Mellers, B.A., Schum, D.A. (eds.) Decision Science and Technology, pp. 81–103. Springer, New York (1999)

    Google Scholar 

  68. Gigerenzer, G., Hoffrage, U., Kleinbölting, H.: Probabilistic mental models: a brunswikian theory of confidence. Psychol. Rev. 98, 506 (1991)

    Article  Google Scholar 

  69. Kahneman, D., Tversky, A.: Prospect theory: an analysis of decision under risk. Econometrica: J. Econometric Soc. 47(2), 263–291 (1979)

    Article  MATH  Google Scholar 

  70. Xu, L., Jackowski, M., Goshtasby, A., Roseman, D., Bines, S., Yu, C., Dhawan, A., Huntley, A.: Segmentation of skin cancer images. Image Vis. Comput. 17, 65–74 (1999)

    Article  Google Scholar 

  71. Królczyk, G., Legutko, S., Raos, P.: Cutting wedge wear examination during turning of duplex stainless steel. Tehnički Vjesnik-Technical Gazette 20, 413–418 (2013)

    Google Scholar 

  72. Baron, J.: Rationality and Intelligence. Cambridge University Press, New York (2005)

    Google Scholar 

  73. Lee, S., Holzinger, A.: Knowledge discovery from complex high dimensional data. In: Michaelis, S., Piatkowski, N., Stolpe, M. (eds.) Solving Large Scale Learning Tasks. Challenges and Algorithms. LNCS (LNAI), vol. 9580, pp. 148–167. Springer, Heidelberg (2016). doi:10.1007/978-3-319-41706-6_7

    Chapter  Google Scholar 

  74. Holzinger, A., Malle, B., Giuliani, N.: On graph extraction from image data. In: Slezak, D., Peters, J.F., Tan, A.H., Schwabe, L. (eds.) Brain Informatics and Health, BIH 2014. LNAI, vol. 8609, pp. 552–563. Springer, Heidelberg (2014)

    Google Scholar 

  75. Valdez, A.C., Dehmer, M., Holzinger, A.: Application of graph entropy for knowledge discovery and data mining in bibliometric data. In: Dehmer, M., Emmert-Streib, F., Chen, Z., Li, X., Shi, Y. (eds.) Mathematical Foundations and Applications of Graph Entropy, pp. 259–272. Wiley, New York (2016)

    Chapter  Google Scholar 

  76. Cao, X., Balakrishnan, R.: Visionwand: interaction techniques for large displays using a passive wand tracked in 3d. In: Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, UIST 2003, pp. 173–182. ACM, New York (2003)

    Google Scholar 

  77. Jones, B.R., Benko, H., Ofek, E., Wilson, A.D.: Illumiroom: peripheral projected illusions for interactive experiences. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 2013, pp. 869–878. ACM, New York (2013)

    Google Scholar 

  78. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on the reality-virtuality continuum. In: Photonics for industrial applications, International Society for Optics and Photonics, pp. 282–292 (1995)

    Google Scholar 

  79. Azuma, R.T.: A survey of augmented reality. Presence: Teleoperators Virtual Environ. 6, 355–385 (1997)

    Article  Google Scholar 

  80. Fuchs, H., et al.: Augmented reality visualization for laparoscopic surgery. In: Wells, W.M., Colchester, A., Delp, S. (eds.) MICCAI 1998. LNCS, vol. 1496, pp. 934–943. Springer, Heidelberg (1998). doi:10.1007/BFb0056282

    Chapter  Google Scholar 

  81. Paelke, V., Röcker, C., Koch, N., Flatt, H., Büttner, S.: User interfaces for cyber-physical systems. at-Automatisierungstechnik 63, 833–843 (2015)

    Google Scholar 

  82. Büttner, S., Sand, O., Röcker, C.: Extending the design space in industrial manufacturing through mobile projection. In: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, MobileHCI 2015, pp. 1130–1133. ACM, New York (2015)

    Google Scholar 

  83. Büttner, S., Funk, M., Sand, O., Röcker, C.: Using head-mounted displays and in-situ projection for assistive systems - a comparison. In: Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments, vol. 8. ACM (2016)

    Google Scholar 

  84. Wilson, A.G., Dann, C., Lucas, C.G., Xing, E.P.: The human kernel. arXiv preprint arXiv:1510.07389 (2015)

Download references

Acknowledgements

We thank our colleague Henrik Mucha who provided insight and expertise that greatly assisted this research. We also thank the anonymous reviewers for their encouraging reviews.

Author information

Authors and Affiliations

  1. Fraunhofer-Institute of Optronics, System Technologies and Image Exploitation, Application Center Industrial Automation (IOSB-INA), Lemgo, Germany

    Sebastian Robert & Carsten Röcker

  2. Ostwestfalen-Lippe University of Applied Sciences, Lemgo, Germany

    Sebastian Büttner & Carsten Röcker

  3. Holzinger Group, HCI-KDD, Institute for Medical Informatics, Statistics and Documentation, Medical University Graz, Graz, Austria

    Carsten Röcker & Andreas Holzinger

Authors
  1. Sebastian Robert
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Büttner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carsten Röcker
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andreas Holzinger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sebastian Robert .

Editor information

Editors and Affiliations

  1. Institute for Medical Informatics, Statistics and Documentation, Medical University Graz, Graz, Austria

    Andreas Holzinger

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this chapter

Cite this chapter

Robert, S., Büttner, S., Röcker, C., Holzinger, A. (2016). Reasoning Under Uncertainty: Towards Collaborative Interactive Machine Learning. In: Holzinger, A. (eds) Machine Learning for Health Informatics. Lecture Notes in Computer Science(), vol 9605. Springer, Cham. https://doi.org/10.1007/978-3-319-50478-0_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-50478-0_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-50477-3

  • Online ISBN: 978-3-319-50478-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation