David W. McDonald
ABSTRACT
Finding a person who has the expertise to solve a specific problem is an important application of recommender systems to a difficult organizational problem. Prior systems have made attempts to implement solutions to this problem, but few systems have undergone systematic user evaluation. This work describes a systematic evaluation of the Expertise Recommender (ER), a system that recommends people who are likely to have expertise in a specific problem. ER and the organizational context for which it was designed are described to provide a basis for understanding this evaluation. Prior to conducting the evaluation, a baseline experiment showed that people are relatively good at judging coworkers' expertise when given an appropriate context. This finding provides a way to demonstrate the effectiveness of ER by comparing ER's performance to ratings by coworkers. The evaluation, the design, and results are described in detail. The results suggest that the participants agree with the recommendations made by ER, and that ER significantly outperforms other expertise recommender systems when compared using similar metrics.
- 1.Allen, T.J. Managing the Flow of Technology. MIT Press, Cambridge, 1977.Google Scholar
- 2.Balabanovic, M. and Shoham, Y. Fab: Content-Based, Collaborative Recommendation. Communications of the ACM, 40 (3). (1997). 66 - 72. Google ScholarDigital Library
- 3.Carroll, J.M. and Rosson, M.B. Getting Around the Task-Artifact Cycle: How to Make Claims and Design by Scenario. ACM Transactions on Information Systems, 10 (2). (1992). 181 - 212. Google ScholarDigital Library
- 4.Ehrlich, K. and Cash, D., Turning Information into Knowledge: Information Finding as a Collaborative Activity. in Digital Libraries '94, (College Station, TX, 1994), 119 - 125.Google Scholar
- 5.Foner, L.N., Yenta: A Multi-Agent, Referral-Based Matchmaking System. in First International Conference on Autonomous Agents (Agent'97), (Marina del Rey, CA, 1997), ACM Press. Google ScholarDigital Library
- 6.Furnas, G.W., Deerwester, S., Dumais, S.T., Landauer, T.K., Harshman, R.A., Streeter, L.A. and Lochbaum, K.E., Information Retrieval Using a Singular Value Decomposition Model of Latent Semantic Structure. in Proceedings of the 11th International Conference on Research and Development in Information Retrieval (SIGIR '88), (1988), 465 - 480. Google ScholarDigital Library
- 7.Glance, N.S., Aregui, D. and Dardenne, M., Making Recommender Systems Work for Organizations. in Practical Application of Intelligent Agents and Multi- Agents (PAAM'99), (London, UK, 1999).Google Scholar
- 8.Goldberg, D., Nichols, D., Oki, B.M. and Terry, D. Using Collaborative Filtering to Weave an Information Tapestry. Communications of the ACM, 35 (12). (1992). 61-70. Google ScholarDigital Library
- 9.Grinter, R.E. Supporting Articulation Work Using Software Configuration Management Systems. Computer Supported Cooperative Work: The Journal of Collaborative Computing, 5. (1996). 447-465. Google ScholarDigital Library
- 10.Hill, W., Stead, L., Rosenstein, M. and Furnas, G., Recommending and Evaluating Choices in a Virtual Community of Use. in CHI '95, (Denver, CO, 1995), ACM Press, 194-201. Google ScholarDigital Library
- 11.Kautz, H.A., Selman, B. and Shah, M. The Hidden Web. AI Magazine (Summer). (1997). 27 - 36.Google Scholar
- 12.Kautz, H.A., Selman, B. and Shah, M. Referral Web: Combining Social Networks and Collaborative Filtering. Communications of the ACM, 40 (3). (1997). 63 - 65. Google ScholarDigital Library
- 13.Kitchenham, B. Software Metrics: Measurement for Software Process Improvement. Blackwell Publishers, Inc., Cambridge, MA, 1996. Google ScholarDigital Library
- 14.Konstan, J.A., Miller, B.N., Maltz, D., Herlocker, J.L., Gordon, L.R. and Riedel, J. GroupLens: Applying Collaborative Filtering to Usenet News. Communications of the ACM, 40 (3). (1997). 77 - 87. Google ScholarDigital Library
- 15.Lutters, W.G., Ackerman, M.S., Boster, J.S. and McDonald, D.W., Mapping Knowledge Networks in Organizations: Creating a Knowledge Mapping Instrument. in Americas Conference on Information Systems, AMCIS'00, (Long Beach, CA, 2000), AIS, 2014-2018.Google Scholar
- 16.Mattox, D., Maybury, M. and Morey, D. Enterprise Expert and Knowledge Discovery. The MITRE Corporation (McLean, VA), September, 2000. http://www.mitre.org/support/papers/tech_papers99_00 /maybury_enterprise/maybury_enterprise.pdfGoogle Scholar
- 17.Maybury, M., D'Amore, R. and House, D. Awareness of Organizational Expertise. The MITRE Corporation (MacLean, VA), October, 2000. http://www.mitre.org/support/papers/tech_papers99_00 /maybury_awareness/maybury_awareness.pdfGoogle Scholar
- 18.McDonald, D.W. Supporting Nuance in Groupware Design: Moving from Naturalistic Expertise Location to Expertise Recommendation. University of California, Irvine. Ph.D. Thesis, 2000. Google ScholarDigital Library
- 19.McDonald, D.W. and Ackerman, M.S., Expertise Recommender: A Flexible Recommendation System and Architecture. in ACM 2000 Conference on Computer-Supported Cooperative Work (CSCW'00), (Philadelphia, PA, 2000), 231-240. Google ScholarDigital Library
- 20.McDonald, D.W. and Ackerman, M.S., Just Talk to Me: A Field Study of Expertise Location. in CSCW'98, (Seattle, WA, 1998), ACM Press, 315 - 324. Google ScholarDigital Library
- 21.Paepcke, A. Information Needs in Technical Work Settings and Their Implications for the Design of Computer Tools. Computer Supported Cooperative Work: The Journal of Collaborative Computing, 5. (1996). 63 - 92. Google ScholarDigital Library
- 22.Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P. and Riedl, J., GroupLens: An Open Architecture for Collaborative Filtering of Netnews. in CSCW '94, (Chapel Hill, NC, 1994), ACM Press, 175-186. Google ScholarDigital Library
- 23.Shepperd, M. Foundations of Software Measurement. Prentice Hall, London, 1995. Google ScholarDigital Library
- 24.Streeter, L.A. and Lochbaum, K.E., An Expert/Expert- Locating System Based on Automatic Representation of Semantic Structure. in Fourth Conference on Artificial Intelligence Applications, (San Diego, CA, 1988).Google Scholar
- 25.Streeter, L.A. and Lochbaum, K.E., Who Knows: A System Based on Automatic Representation of Semantic Structure. in RIAO '88, (Cambridge, MA, 1988), 380 - 388.Google Scholar
- 26.Terveen, L.G., Hill, W., Amento, B., McDonald, D. and Creter, J., Building Task-Specific Interfaces to High Volume Conversational Data. in CHI'97, (1997), ACM Press, 226 - 233. Google ScholarDigital Library
- 27.Vivacqua, A. and Lieberman, H., Agents to Assist in Finding Help. in ACM Conference on Human Factors in Computing Systems (CHI 2000), (2000), 65-72. Google ScholarDigital Library
- 28.Young, R.M. and Barnard, P., The Use of Scenarios in Human-Computer Interaction Research: Turbocharging the Tortoise of Cumulative Science. in Human Factors in Computing Systems and Graphics Interface, CHI+GI '87, (Toronto, Canada, 1987), ACM Press, 291-296. Google ScholarDigital Library
Index Terms
- Evaluating expertise recommendations
Recommendations
Mining usage expertise from version archives
MSR '08: Proceedings of the 2008 international working conference on Mining software repositoriesIn software development, there is an increasing need to find and connect developers with relevant expertise. Existing expertise recommendation systems are mostly based on variations of the Line 10 Rule: developers who changed a file most often have the ...
Expertise recommender: a flexible recommendation system and architecture
CSCW '00: Proceedings of the 2000 ACM conference on Computer supported cooperative workLocating the expertise necessary to solve difficult problems is a nuanced social and collaborative problem. In organizations, some people assist others in locating expertise by making referrals. People who make referrals fill key organizational roles ...
Just talk to me: a field study of expertise location
CSCW '98: Proceedings of the 1998 ACM conference on Computer supported cooperative work
Comments