Youngja Park
ABSTRACT
Customer satisfaction is a very important indicator of how successful a contact center is at providing services to the customers. Contact centers typically conduct a manual survey with a randomly selected group of customers to measure customer satisfaction. Manual customer satisfaction surveys, however, provide limited values due to high cost and the time lapse between the service and the survey.
In this paper, we demonstrate that it is possible to automatically measure customer satisfaction by analyzing call transcripts enabling companies to measure customer satisfaction for every call in near real-time. We have identified various features from multiple knowledge sources indicating prosodic, linguistic and behavioral aspects of the speakers, and built machine learning models that predict the degree of customer satisfaction with high accuracy. The machine learning algorithms used in this work include Decision Tree, Naive Bayes, Logistic Regression and Support Vector Machines (SVMs).
Experiments were conducted for a 5-point satisfaction measurement and a 2-point satisfaction measurement using customer calls to an automotive company. The experimental results show that customer satisfaction can be measured quite accurately both at the end of calls and in the middle of calls. The best performing 5-point satisfaction classification yields an accuracy of 66.09% outperforming the DominantClass baseline by 15.16%. The best performing 2-point classification shows an accuracy of 89.42% and outperforms both the DominantClass baseline and the CSRJudgment baseline by 17.7% and 3.3% respectively. Furthermore, Decision Tree and SVMs achieve higher F-measure than the CSRJudgment baseline in identifying both satisfied customers and dissatisfied customers.
- Anderson, E. W., C. Fornell, and R. T. Rust: 1997, 'Customer Satisfaction, Productivity, and Profitability: Differences Between Goods and Services'. MARKETING SCIENCE 16(2), 129--145.Google Scholar
Digital Library
- Brennan, M., S. Benson, and Z. Kearns: 2005, 'The effect of introductions on telephone survey participation rates'. International Journal of Market Research 47(1), 65--74.Google ScholarCross Ref
- Burton, S. M.: 1997, 'Modelling the Determinants of Customer Satisfaction'. In: Ph.D. thesis, The University of New South Wales, Australia.Google Scholar
- Byrd, R. J. N. M. S., W. Teiken, Y. Park, K. F. Cheng, S. C. Gates, and K. Visweswariah: 2008, 'Semi-automated logging of contact center telephone calls'. In: CIKM '08: Proceeding of the 17th ACM conference on Information and knowledge management. pp. 133--142. Google ScholarDigital Library
- Chang, C. and C. Lin: 2001, 'LIBSVM: a library for support vector machines'. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.Google Scholar
- Devitt, A. and K. Ahmad: 2008, 'Sentiment Polarity Identification in Financial News: A Cohesion-based Approach'. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics (ACL). pp. 984--991.Google Scholar
- Feinberg, R., I.-S. Kim, L. Hokama, K. de Ruyter, and C. Keen: 2000, 'Operational determinants of caller satisfaction in the call center'. International Journal of Service Industry Management 11(2), 131--141.Google ScholarCross Ref
- Forbes-Riley, K. and D. Litman: 2004, 'Predicting Emotion in Spoken Dialogue from Multiple Knowledge Sources'. In: HLT-NAACL 2004: Main Proceedings. pp. 201--208.Google Scholar
- Gamon, M.: 2004, 'Sentiment classification on customer feedback data: noisy data, large feature vectors, and the role of linguistic analysis'. In: Proceedings of Coling 2004. pp. 841--847. Google ScholarDigital Library
- Godbole, S. and S. Roy: 2008, 'Text Classification, Business Intelligence, and Interactivity: Automating C-Sat Analysis for Services Industry'. In: Proceeding of the 14th ACM SIGKDD international conference on Knowledge Discovery and Data Mining (KDD). pp. 911--919. Google ScholarDigital Library
- Hallowell, R.: 1996, 'The relationships of customer satisfaction, customer loyalty, and profitability: an empirical study'. International Journal of Service Industry Management 7, 27--42.Google ScholarCross Ref
- Hu, M. and B. Liu: 2004, 'Mining and Summarizing Customer Reviews'. In: Proceedings of the Tenth International Conference on Knowledge Discovery and Data Mining. pp. 168--177. Google ScholarDigital Library
- Kanayama, H. and T. Nasukawa: 2006, 'Fully Automatic Lexicon Expansion for Domain-oriented Sentiment Analysis'. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 355--363. Google ScholarDigital Library
- Kim, S.-M. and E. Hovy: 2006a, 'Extracting Opinions, Opinion Holders, and Topics Expressed in Online News Media Text'. In: Proceedings of the ACL Workshop on Sentiment and Subjectivity in Text. pp. 1--8. Google ScholarDigital Library
- Kim, S.-M. and E. Hovy: 2006b, 'Identifying and Analyzing Judgment Opinions'. In: Proceedings of the Human Language Technology Conference of the NAACL. pp. 200--207. Google ScholarDigital Library
- Kobayashi, N., K. Inui, and Y. Matsumoto: 2007, 'Extracting Aspect-Evaluation and Aspect-Of Relations in Opinion Mining'. In: Proceedings of the Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. pp. 1065--1074.Google Scholar
- Likert, R.: 1932, 'A Technique for the Measurement of Attitudes'. Archives of Psychology 140. pp 1--55.Google Scholar
- Liscombe, J., J. Venditti, and J. Hirschberg: 2003, 'Classifying subject ratings of emotional speech using acoustic feature'. In: Proceedings of EUROSPEECH. pp. 725--728.Google Scholar
- Mierswa, I., M. Wurst, R. Klinkenberg, M. Scholz, and T. Euler: 2006, 'YALE: Rapid Prototyping for Complex Data Mining Tasks'. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 935--940. Google ScholarDigital Library
- Mitchell, T.: 1997, Machine Learning. McGraw Hill. Google ScholarDigital Library
- Monger, J., M. Rudick, and L. O'Flahavan: 2004, 'First call resolution: its impact and measurement'. Contact Professional. pp. 24--27.Google Scholar
- Nambiar, U., H. Gupta, and M. Mohania: 2007, 'CallAssit: Helping Call Center Agents in Preference Elicitation'. In: Proceedings of VLDB. pp 1338--1341. Google ScholarDigital Library
- Pang, B. and L. Lee: 2005, 'Seeing Stars: Exploiting Class Relationships for Sentiment Categorization with Respect to Rating Scales'. Ann Arbor, Michigan, pp. 115--124. Google ScholarDigital Library
- Pang, B., L. Lee, and S. Vaithyanathan: 2002, 'Thumbs up? Sentiment Classification using Machine Learning Techniques'. In: Conf. on Empirical Methods in Natural Language Processing (EMNLP). pp. 79--86. Google ScholarDigital Library
- Polanyi, L. and A. Zaenen: 2004, 'Contextual Valence Shifters'. In: Working Notes - Exploring Attitude and Affect in Text: Theories and Applications.Google Scholar
- Quinlan, J. R.: 1993, Programs for Machine Learning. Morgan Kaufmann Publishers. Google ScholarDigital Library
- Ranaweera, C. and J. Prabhu: 2003, 'The influence of satisfaction, trust and switching barriers on customer retention in a continuous purchasing setting'. International Journal of Service Industry Management 14. pp. 374--395.Google Scholar
- Reichheld, F. F.: 2001, Loyalty Rules: How Today's Leaders Build Lasting Relationships. Harvard Business School Press.Google Scholar
- Rifkin, R. M. and A. Klautau: 1998, 'In defense of one-vs-all classification'. Journal of Machine Learning 5. pp 101--141. Google ScholarDigital Library
- Shannon, C. E.: 1948, 'A Mathematical Theory of Communication'. Google ScholarDigital Library
- Soltau, H., B. Kingsbury, L. Mangu, D. Povey, G. Saon, and G. Zweig: 2004, 'The IBM 2004 conversational telephony system for rich transcription'. In: Proceedings of International Conference on Acoustics, Speech, and Signal Processing. pp. 205--208.Google Scholar
- Somasundaran, S., J. Ruppenhofer, and J. Wiebe: 2007, 'Detecting Arguing and Sentiment in Meetings'. In: SIGdial Workshop on Discourse and Dialogue.Google Scholar
- Takeuchi, H., L. Subramaniam, T. Nasukawa, and S. Roy: 2007, 'Combining Multiple Knowledge Sources for Dialogue Segmentation in Multimedia Archives'. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. pp. 1016--1023.Google Scholar
- Turney, P. D.: 2002, 'Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews'. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). pp. 417--424. Google ScholarDigital Library
- Vapnik, V.: 1995, The Nature of Statistical Learning Theory. New York: Springer-Verlag. Google ScholarDigital Library
- Wilson, T., J. Wiebe, and P. Hoffmann: 2005b, 'Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis'. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 347--354. Google ScholarDigital Library
- Yi, J., T. Nasukawa, R. Bunescu, and W. Niblack: 2003, 'Sentiment Analyzer: Extracting Sentiments about a Given Topic using Natural Language Processing Techniques'. In: Proceedings of the 3rd IEEE International Conference on Data Mining (ICDM-2003). pp. 427--434. Google ScholarDigital Library
- Zweig, G., O. Siohan, B. Ramabhadran, D. Povey, L. Mangu, and B. Kingsbury: 2006, 'Automated Quality Monitoring in the Call Center with ASR and Maximum Entropy'. In: Proceedings of International Conference on Acoustics, Speech, and Signal Processing.Google Scholar
Index Terms
- Towards real-time measurement of customer satisfaction using automatically generated call transcripts
Recommendations
Internet Pricing, Price Satisfaction, and Customer Satisfaction
Given the challenges of increasing margins and building a loyal customer base, the interaction between pricing and customer satisfaction is of great interest to e-tailers. The present research models the relationships between e-tailer pricing, price ...
Customer satisfaction with order fulfillment in retail supply chains: implications of product type in electronic B2C transactions
AbstractThis paper focuses on the proverbial “last mile” of the retail supply chain – i.e., delivering products to the end‐customer – and highlights the need for recognizing product type differences in configuring order fulfillment processes in electronic ...
Customer revisit intention to restaurants: Evidence from online reviews
One of the generally recognized marketing principles is that retaining customers is more profitable than winning prospective customers. Therefore, how to retain existing customers and improve their repeat purchases is an important consideration for ...
Comments