Skip to main content

Advertisement

SpringerLink
Log in

Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?

  • Published:
Journal of Behavioral Medicine Aims and scope Submit manuscript

Abstract

Behavioral weight loss (WL) trials show that, on average, participants regain lost weight unless provided long-term, intensive—and thus costly—intervention. Optimization solutions have shown mixed success. The artificial intelligence principle of “reinforcement learning” (RL) offers a new and more sophisticated form of optimization in which the intensity of each individual’s intervention is continuously adjusted depending on patterns of response. In this pilot, we evaluated the feasibility and acceptability of a RL-based WL intervention, and whether optimization would achieve equivalent benefit at a reduced cost compared to a non-optimized intensive intervention. Participants (n = 52) completed a 1-month, group-based in-person behavioral WL intervention and then (in Phase II) were randomly assigned to receive 3 months of twice-weekly remote interventions that were non-optimized (NO; 10-min phone calls) or optimized (a combination of phone calls, text exchanges, and automated messages selected by an algorithm). The Individually-Optimized (IO) and Group-Optimized (GO) algorithms selected interventions based on past performance of each intervention for each participant, and for each group member that fit into a fixed amount of time (e.g., 1 h), respectively. Results indicated that the system was feasible to deploy and acceptable to participants and coaches. As hypothesized, we were able to achieve equivalent Phase II weight losses (NO = 4.42%, IO = 4.56%, GO = 4.39%) at roughly one-third the cost (1.73 and 1.77 coaching hours/participant for IO and GO, versus 4.38 for NO), indicating strong promise for a RL system approach to weight loss and maintenance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Abbeel, P., & Ng, A. Y. (2004). Apprenticeship learning via inverse reinforcement learning. Paper presented at the Proceedings of the twenty-first international conference on Machine learning.

  • Anderson, J. W., Vichitbandra, S., Qian, W., & Kryscio, R. J. (1999). Long-term weight maintenance after an intensive weight-loss program. Journal of the American College of Nutrition, 18, 620–627.

    Article  CAS  PubMed  Google Scholar 

  • Association of Behavioral and Cognitive Therapies. (2018). Find a CBT Therapist. Retrieved May 23, 2018, from http://www.findcbt.org.

  • Auer, P., Cesa-Bianchi, N., & Fischer, P. (2002). Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47, 235–256.

    Article  Google Scholar 

  • Aydin, M. E., & Öztemel, E. (2000). Dynamic job-shop scheduling using reinforcement learning agents. Robotics and Autonomous Systems, 33, 169–178.

    Article  Google Scholar 

  • Brindal, E., Freyne, J., Saunders, I., Berkovsky, S., Smith, G., & Noakes, M. (2012). Features predicting weight loss in overweight or obese participants in a web-based intervention: Randomized trial. Journal of medical Internet research, 14, e173.

    Article  PubMed  PubMed Central  Google Scholar 

  • Butryn, M. L., Forman, E. M., Lowe, M. R., Gorin, A. A., Zhang, F., & Schaumberg, K. (2017). Efficacy of environmental and acceptance-based enhancements to behavioral weight loss treatment: The ENACT trial. Obesity, 25, 866–872.

    Article  PubMed  Google Scholar 

  • Butryn, M. L., Zhang, F., Remmert, J. E., Roberts, S. R., & Forman, E. M. (2018). Baseline Executive Functioning Predicts Weight Loss and Physical Activity Outcomes in A Lifestyle Modification Program. Paper presented at the 39th Annual Meeting and Scientific Sessions of the Society of Behavioral Medicine, New Orleans, LA.

  • Carels, R. A., Darby, L., Cacciapaglia, H. M., Douglass, O. M., Harper, J., Kaplar, M. E., et al. (2005). Applying a stepped-care approach to the treatment of obesity. Journal of Psychosomatic Research, 59, 375–383.

    Article  PubMed  Google Scholar 

  • Carels, R. A., Darby, L., Cacciapaglia, H. M., Konrad, K., Coit, C., Harper, J., et al. (2007). Using motivational interviewing as a supplement to obesity treatment: A stepped-care approach. Health Psychology, 26, 369.

    Article  PubMed  Google Scholar 

  • Carels, R. A., Hoffmann, D. A., Hinman, N., Burmeister, J. M., Koball, A., Ashrafioun, L., et al. (2013). Step-down approach to behavioural weight loss treatment: A pilot of a randomised clinical trial. Psychology & Health, 28, 1121–1134.

    Article  Google Scholar 

  • Carels, R. A., Selensky, J. C., Rossi, J., Solar, C., & Hlavka, R. (2017). A novel stepped-care approach to weight loss: The role of self-monitoring and health literacy in treatment outcomes. Eating Behaviors, 26, 76–82.

    Article  PubMed  Google Scholar 

  • Centers for Disease Control and Prevention (CDC). (2014). National Health and Nutrition Examination Survey Data. Hyattsville, MD. Retrieved May 13, 2018, from https://www.cdc.gov/nchs/fastats/obesity-overweight.htm.

  • Diabetes Prevention Program Research Group. (2003). Costs associated with the primary prevention of type 2 diabetes mellitus in the diabetes prevention program. Diabetes Care, 26, 36–47.

    Article  Google Scholar 

  • Forman, E. M., Butryn, M. L., Manasse, S. M., Crosby, R. D., Goldstein, S. P., Wyckoff, E. P., et al. (2016). Acceptance-based versus standard behavioral treatment for obesity: Results from the mind your health randomized controlled trial. Obesity, 24, 2050–2056.

    Article  PubMed  Google Scholar 

  • Jakicic, J. M., Tate, D. F., Lang, W., Davis, K. K., Polzien, K., Rickman, A. D., et al. (2012). Effect of a stepped-care intervention approach on weight loss in adults: A randomized clinical trial. JAMA, 307, 2617–2626.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Jeffery, R. W., Epstein, L. H., Wilson, G. T., Drewnowski, A., Stunkard, A. J., & Wing, R. R. (2000). Long-term maintenance of weight loss: Current status. Health Psychology, 19, 5.

    Article  CAS  PubMed  Google Scholar 

  • Joo, N.-S., & Kim, B.-T. (2007). Mobile phone short message service messaging for behaviour modification in a community-based weight control programme in Korea. Journal of Telemedicine and Telecare, 13, 416–420.

    Article  PubMed  Google Scholar 

  • Korinek, E. V., Phatak, S. S., Martin, C. A., Freigoun, M. T., Rivera, D. E., Adams, M. A., et al. (2018). Adaptive step goals and rewards: A longitudinal growth model of daily steps for a smartphone-based walking intervention. Journal of Behavioral Medicine, 41, 74–86.

    Article  PubMed  Google Scholar 

  • Kramer, F. M., Jeffery, R. W., Forster, J. L., & Snell, M. K. (1989). Long-term follow-up of behavioral treatment for obesity: Patterns of weight regain among men and women. International Journal of Obesity, 13, 123–136.

    CAS  PubMed  Google Scholar 

  • Krukowski, R. A., Tilford, J. M., Harvey-Berino, J., & West, D. S. (2011). Comparing behavioral weight loss modalities: Incremental cost-effectiveness of an internet-based versus an in-person condition. Obesity, 19, 1629–1635.

    Article  PubMed  Google Scholar 

  • Kuderer, M., Gulati, S., & Burgard, W. (2015). Learning driving styles for autonomous vehicles from demonstration. Paper presented at the 2015 IEEE International Conference on Robotics and Automation (ICRA).

  • McGraa, K. L. K. (2010). The effects of persuasive motivational text messaging on adherence to diet and exercise programs across different personality traits. Santa Barbara: Fielding Graduate University.

    Google Scholar 

  • Ng, A. Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., et al. (2006). Autonomous inverted helicopter flight via reinforcement learning. In M. H. Ang & O. Khatib (Eds.), Experimental Robotics IX (pp. 363–372). Berlin: Springer.

    Chapter  Google Scholar 

  • Ontanón, S. (2013). The combinatorial multi-armed bandit problem and its application to real-time strategy games. Paper presented at the Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.

  • Ontanón, S. (2017). Combinatorial multi-armed bandits for real-time strategy games. Journal of Artificial Intelligence Research, 58, 665–702.

    Article  Google Scholar 

  • Paredes, P., Gilad-Bachrach, R., Czerwinski, M., Roseway, A., Rowan, K., & Hernandez, J. (2014). PopTherapy: Coping with stress through pop-culture. Paper presented at the Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare.

  • Patrick, K., Raab, F., Adams, M. A., Dillon, L., Zabinski, M., Rock, C. L., et al. (2009). A text message–based intervention for weight loss: Randomized controlled trial. Journal of medical Internet research, 11, e1.

    Article  PubMed  PubMed Central  Google Scholar 

  • Phatak, S. S., Freigoun, M. T., Martín, C. A., Rivera, D. E., Korinek, E. V., Adams, M. A., et al. (2018). Modeling individual differences: A case study of the application of system identification for personalizing a physical activity intervention. Journal of Biomedical Informatics, 79, 82–97.

    Article  PubMed  Google Scholar 

  • Piette, J. D., Farris, K. B., Newman, S., An, L., Sussman, J., & Singh, S. (2014). The potential impact of intelligent systems for mobile health self-management support: Monte Carlo simulations of text message support for medication adherence. Annals of Behavioral Medicine, 49, 84–94.

    Article  PubMed Central  Google Scholar 

  • Piette, J. D., Krein, S. L., Striplin, D., Marinec, N., Kerns, R. D., Farris, K. B., et al. (2016). Patient-centered pain care using artificial intelligence and mobile health tools: Protocol for a randomized study funded by the US Department of Veterans Affairs Health Services Research and Development Program. JMIR Research Protocols, 5, e53.

    Article  PubMed  PubMed Central  Google Scholar 

  • Poirier, J., Bennett, W. L., Jerome, G. J., Shah, N. G., Lazo, M., Yeh, H.-C., et al. (2016). Effectiveness of an activity tracker-and internet-based adaptive walking program for adults: A randomized controlled trial. Journal of Medical Internet Research, 18, e34.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rabbi, M., Aung, M. H., Zhang, M., & Choudhury, T. (2015). MyBehavior: Automatic personalized health feedback from user behaviors and preferences using smartphones. Paper presented at the Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing.

  • Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., et al. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529, 484–489.

    Article  CAS  PubMed  Google Scholar 

  • Snook, K. R., Hansen, A. R., Duke, C. H., Finch, K. C., Hackney, A. A., & Zhang, J. (2017). Change in percentages of adults with overweight or obesity trying to lose weight, 1988–2014. JAMA, 317, 971–973.

    Article  PubMed  Google Scholar 

  • Society of Clinical Psychology. (2018). Therapists Near Me. Retrieved May 23, 2018, from https://www.div12.org/therapist-search/.

  • Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge: MIT press Cambridge.

    Google Scholar 

  • Wadden, T. A., Butryn, M. L., & Wilson, C. (2007). Lifestyle modification for the management of obesity. Gastroenterology, 132, 2226–2238.

    Article  PubMed  Google Scholar 

  • Wadden, T. A., Sternberg, J., Letizia, K., Stunkard, A., & Foster, G. (1989). Treatment of obesity by very low calorie diet, behavior therapy, and their combination: A five-year perspective. International Journal of Obesity, 13, 39–46.

    PubMed  Google Scholar 

  • Webber, K. H., Tate, D. F., & Bowling, J. M. (2008). A randomized comparison of two motivationally enhanced Internet behavioral weight loss programs. Behaviour Research and Therapy, 46, 1090–1095.

    Article  PubMed  Google Scholar 

  • Wilson, G. T. (1994). Behavioral treatment of obesity: Thirty years and counting. Advances in Behaviour Research and Therapy, 16, 31–75.

    Article  Google Scholar 

  • Wilson, G. T., & Brownell, K. D. (2002). Behavioral treatment for obesity. In C. G. Fairburn & K. D. Brownell (Eds.), Eating disorders and obesity. New York: Guilford Press.

    Google Scholar 

  • Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., Tennenholtz, M., & Hochberg, I. (2017). Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of Medical Internet Research, 19, e338.

    Article  PubMed  PubMed Central  Google Scholar 

  • Zhou, M., Fukuoka, Y., Mintz, Y., Goldberg, K., Kaminsky, P., Flowers, E., et al. (2018). Evaluating machine learning-based automated personalized daily step goals delivered through a Mobile Phone App: Randomized controlled trial. JMIR mHealth and uHealth, 6, e28.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This project was funded by a Drexel University DARE (Drexel Areas of Research Excellence) Award to Evan Forman. We would also like to acknowledge the assistance of Pedrag Krasnja whom we consulted during the planning of this study.

Author information

Authors and Affiliations

  1. Department of Psychology, WELL Center, Drexel University, Stratton Hall, 3141 Chestnut Street, Philadelphia, PA, 19104, USA

    Evan M. Forman, Stephanie G. Kerrigan, Meghan L. Butryn, Adrienne S. Juarascio, Diane H. Dallal, Rebecca J. Crochiere & Danielle Moskow

  2. Center for Weight, Eating, and Lifestyle Sciences (WELL Center), Drexel University, 3201 Chestnut Street, Philadelphia, PA, 19104, USA

    Evan M. Forman, Stephanie G. Kerrigan, Meghan L. Butryn, Adrienne S. Juarascio, Stephanie M. Manasse, Diane H. Dallal, Rebecca J. Crochiere & Danielle Moskow

  3. Department of Computer Science, Drexel University, 3401 Market Street, Philadelphia, PA, 19104, USA

    Santiago Ontañón

Authors
  1. Evan M. Forman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephanie G. Kerrigan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meghan L. Butryn
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Adrienne S. Juarascio
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stephanie M. Manasse
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Santiago Ontañón
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Diane H. Dallal
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Rebecca J. Crochiere
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Danielle Moskow
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Evan M. Forman.

Ethics declarations

Conflict of interest

Evan M. Forman, Stephanie G. Kerrigan, Meghan L. Butryn, Adrienne S. Juarascio, Stephanie M. Manasse, Santiago Ontañón, Diane H. Dallal, Rebecca J. Crochiere and Danielle Moskow declare that they have no conflict of interest.

Human and animal rights and Informed consent

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Forman, E.M., Kerrigan, S.G., Butryn, M.L. et al. Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?. J Behav Med 42, 276–290 (2019). https://doi.org/10.1007/s10865-018-9964-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10865-018-9964-1

Keywords

Search

Navigation