Weng-Keen Wong
Simone Stumpf
ABSTRACT
When intelligent interfaces, such as intelligent desktop assistants, email classifiers, and recommender systems, customize themselves to a particular end user, such customizations can decrease productivity and increase frustration due to inaccurate predictions - especially in early stages, when training data is limited. The end user can improve the learning algorithm by tediously labeling a substantial amount of additional training data, but this takes time and is too ad hoc to target a particular area of inaccuracy. To solve this problem, we propose a new learning algorithm based on locally weighted regression for feature labeling by end users, enabling them to point out which features are important for a class, rather than provide new training instances. In our user study, the first allowing ordinary end users to freely choose features to label directly from text documents, our algorithm was both more effective than others at leveraging end users' feature labels to improve the learning algorithm, and more robust to real users' noisy feature labels. These results strongly suggest that allowing users to freely choose features to label is a promising method for allowing end users to improve learning algorithms effectively.
- Attenberg, J., Melville, P., and Provost, F. A unified approach to active dual supervision for labeling features and examples, in Proc. European Conf. Machine Learn-ing (2010). Google Scholar
Digital Library
- Cleveland, W., and Devlin, S. Locally-weighted regres-sion: An approach to regression analysis by local fitting. J. American Statistical Assn. 83, 403 (1988), 596--610.Google ScholarCross Ref
- Cohn D. A., Ghahramani, Z., and Jordan, M. I. Active learning with statistical models. J. Artificial Intelligence Research, 4 (1996), 129--145. Google ScholarDigital Library
- Deng, K. Omega: On-line Memory-Based General Pur-pose System Classifier (PhD Dissertation). Carnegie Mellon University, Pittsburgh, PA, 1998. Google ScholarDigital Library
- Druck, G., Mann, G., and McCallum, A. Learning from labeled features using generalized expectation criteria, in Proc. SIGIR (2008), ACM, 595--602. Google ScholarDigital Library
- Hastie, T., Tibshirani, R., and Friedman, J. H. The Elements of Statistical Learning. Springer, 2003.Google Scholar
- Kulesza, T., Wong, W.-K., Stumpf, S., Perona, S., White, S., Burnett, M., Oberst, I. Ko, A. Fixing the program my computer learned: Barriers for end users, challenges for the machine, in Proc. IUI (2009), ACM, 187--196. Google ScholarDigital Library
- Kulesza, T., Stumpf, S., Burnett, M., Wong, W.-K., Riche, Y., Moore, T., Oberst, I., Shinsel, A., McIntosh, K., Explanatory debugging: supporting end-user debugging of machine-learned programs, in Proc. IEEE Symp. Visual Languages and Human-Centric Computing (2010), IEEE, 41--48. Google ScholarDigital Library
- Lang, K. Newsweeder: Learning to filter netnews, in Proc. ICML (1995), 331--339.Google Scholar
- Lewis, D. Reuters-21578. Available at http://www.daviddlewis.com/resoursce/testcollections/reuters21578.Google Scholar
- 1Lewis, D. D., Yang, Y., Rose, T., Li, F. RCV1: A new benchmark collection for text categorization research. JMLR, 5 (2004), 361--397. http://www.jmlr.org/papers/volume5/lewis04a/lewis04a.pdf. Google ScholarDigital Library
- Liu, H., and Singh, P. ConceptNet-a practical com-monsense reasoning tool-kit. BT Technology Journal 22, 4 (2004), 211--226. Google ScholarDigital Library
- McCallum, A., Mann, G., and Druck, G. Generalized Expectation Criteria (Technical Report UM-CS-2007-60). University of Massachusetts, Amherst, 2007.Google Scholar
- Nocedal, J. Updating quasi-Newton matrices with limited storage. Mathematics of Computation, 35 (1980), 773--782.Google ScholarCross Ref
- Raghavan, H., Madani, O., and Jones, R. Active Learning with Feedback on Both Features and Instances. JMLR 7 (2006), 1655--1686. Google ScholarDigital Library
- Raghavan, H. and Allan, J. An interactive algorithm for asking and incorporating feature feedback into support vector machines, in Proc. SIGIR (2007), ACM, 79--86. Google ScholarDigital Library
- Roth, D. and Small, K. Interactive feature space construction using semantic information, in Proc. CoNLL (2009), 66--74. Google ScholarDigital Library
- Sindhwani, V., Melville, P., and Lawrence, R. Uncertainty sampling and transductive experimental design for active dual supervision. Int. Conf. Machine Learning (2009), 953--960. Google ScholarDigital Library
- Speer, R., Havasi, C., and Lieberman, H. AnalogySpace: Reducing the dimensionality of common sense knowledge, in Proc. AAAI (2008). Google ScholarDigital Library
- Stumpf, S. Rajaram, V., Li, L., Wong, W.-K., Burnett, M., Dietterich, T., Sullivan, E., and Herlocker J. Interacting meaningfully with machine learning systems: Three ex-periments. Int. J. Human-Computer Studies 67, 8 (2009), 639--662.. Google ScholarDigital Library
Index Terms
- End-user feature labeling: a locally-weighted regression approach
Recommendations
End-user feature labeling: Supervised and semi-supervised approaches based on locally-weighted logistic regression
When intelligent interfaces, such as intelligent desktop assistants, email classifiers, and recommender systems, customize themselves to a particular end user, such customizations can decrease productivity and increase frustration due to inaccurate ...
End-user feature labeling via locally weighted logistic regression
AAAI'11: Proceedings of the Twenty-Fifth AAAI Conference on Artificial IntelligenceApplications that adapt to a particular end user often make inaccurate predictions during the early stages when training data is limited. Although an end user can improve the learning algorithm by labeling more training data, this process is time ...
Feature-induced labeling information enrichment for multi-label learning
AAAI'18/IAAI'18/EAAI'18: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial IntelligenceIn multi-label learning, each training example is represented by a single instance (feature vector) while associated with multiple class labels simultaneously. The task is to learn a predictive model from the training examples which can assign a set of ...
Comments