Gregorio Convertino
John M. Carroll
ABSTRACT
Many failures in long-term collaboration occur because of a lack of activity awareness. Activity awareness is a broad concept that involves awareness of synchronous and asynchronous interactions over extended time periods. We describe a procedure to evaluate activity awareness and collaborative activities in a controlled setting. The activities used are modeled on real-world collaborations documented earlier in a field study. We developed an experimental method to study these activity awareness problems in the laboratory. Participants worked on a simulated long-term project in the laboratory over multiple experimental sessions with a confederate, who partially scripted activities and probes. We present evidence showing that this method represents a valid model of real collaboration, based on participants' active engagement, lively negotiation, and awareness difficulties. We found that having the ability to define, reproduce, and systematically manipulate collaborative situations allowed us to assess the effect of realistic conditions on activity awareness in remote collaboration.
- Arrow H., McGrath, J. E. Berdahl, J. L., Small Groups as Complex Systems: Formation, Co-ordination, Development, and Adaptation. Sage Publications, 2000.Google Scholar
- Baker, K. Heuristic Evaluation of Shared Workspace Groupware based on the Mechanics of Collaboration. MSc Thesis, Department of Computer Science, University of Calgary, Alberta, Canada, 2002.Google Scholar
- Campbell, D., & Fiske, D. W. Convergent and discriminate validation by the multitraitmultimethod matrix. Psychological Bulletin 4, (1959), 297--312.Google Scholar
- Carroll, J. M., Neale, D. C., Isenhour, P. L., Rosson, M. B., and McCrickard, D. S. Notification and awareness: synchronizing task-oriented collaborative activity. Int., Journ. of Human-Computer Studies 58, (2003) 605--32. Google ScholarDigital Library
- Clark, H. H. Using Language. New York: Cambridge University Press, 1996.Google Scholar
- Cockton, G., & Woolrych, A. Sale must end: Should discount methods be cleared off HCI's shelves?, interactions 9(5) (2002), 13--18. Google ScholarDigital Library
- Connolly, T., Jessup, L., & Valacich, J. S., "Effects of anonymity and evaluative tone on idea generation in computer-mediated groups," Management Science 36, (1990), 689--703. Google ScholarDigital Library
- Easterbrook, S. (ed.), CSCW: Co-operation or Conflict, Springer-Verlag, New York, 1993. Google ScholarDigital Library
- Endsley, M. R., Toward a theory of situation awareness in dynamic systems. Human Factors 37, (1995), 32--64.Google ScholarCross Ref
- Flanagan, J. C. 1954. The critical incident technique. Psychological bulletin 51(4), (1954), 327--358.Google Scholar
- Ganoe, C. H., Somervell, J. P., Neale, D. C., Isenhour, P. L., Carroll, J. M., Rosson, M. B., and McCrickard, D. S. Classroom BRIDGE: Using Collaborative Public and Desktop Timelines to Support Activity Awareness. Proc. of the ACM '03 Symposium on User Interface Software and Technology, ACM Press, (2003) 21--30. Google ScholarDigital Library
- Groove Networks Inc., http://www.Grove.com, 2003.Google Scholar
- Grudin, J., Why CSCW Applications Fail: Problems in the Design and Evaluation of Organizational Interfaces, Proc. CSCW'88, (1988), 85--93. Google ScholarDigital Library
- Gutwin, C. & Greenberg, S. (2000). The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces. IEEE 9th Int'l Workshop WET-ICE'00 (2000). Google ScholarDigital Library
- Gutwin, C. & Greenberg, S. Workspace Awareness Position paper for the ACM CHI'97 workshop on Awareness in Collaborative Systems, organized by McDaniel S. E., & Brinck T., Atlanta, Georgia (1997).Google Scholar
- Gutwin, C. & Greenberg, S., Workspace awareness for groupware. Proc. CHI'96, ACM Press, (1996), 208--209. Google ScholarDigital Library
- Hayes-Roth, B. & Hayes-Roth, F. A cognitive model of planning. Cognitive Science, 3, (1979), 275--310.Google ScholarCross Ref
- Kuutti, K. & Arvonen, T. Identifying potential CSCW applications by means of Activity Theory concepts: A case example. Proc. CSCW '92, ACM Press, (1992), 233--240. Google ScholarDigital Library
- McGrath, J. E., Methodology matters: Doing research in the behavioral and social sciences. In R. M. Baecker, J. Grudin, W. A. S. Buxton, & S. Greenberg (Eds.), Readings in human-computer interaction: Toward the year 2000, Morgan Kaufmann, (1995), 152--169. Google ScholarDigital Library
- Mohammed, S., & Dumville, B., Team Mental Models: Expanding Theory and Measurement Through Cross-Disciplinary Boundaries. Journal of Organizational Behavior, 22 (2001), 89--106.Google ScholarCross Ref
- Neale, D. C., Carroll, J. M., Rosson, M. B., Evaluating Computer-Supported Co-operative Work: Models and Frameworks, Proc. CSCW '04, ACM Press, (2004) (to appear). Google ScholarDigital Library
- Neale, D. C., Carroll, J. M., 1999. Multi-faceted evaluation for complex, distributed activities. Proc. of the CSCL'99 Computer Supported Co-operative Learning. Lawrence Erlbaum, 425--433. Google ScholarDigital Library
- Pinelle, D., & Gutwin, C., Groupware walkthrough: Adding context to groupware usability evaluation. Proc. 2002, ACM Press, (2002) 455--462. Google ScholarDigital Library
- Pinelle, D., Gutwin, C., & Greenberg, S., Task Analysis for Groupware Usability Evaluation: Modeling Shared-Workspace Tasks with the Mechanics of Collaboration, ACM Transactions on Computer-Human Interaction, 10, (4) (2003), 281--311. Google ScholarDigital Library
- Prinz W., Mark G., Pankoke-Babatz P. (1998), Designing Groupware for Congruency in Use, Proc. CSCW'98, ACM Press, 373--382. Google ScholarDigital Library
- Ramage M., The Learning Way: Evaluating Co-operatives Systems, PhD thesis, Lancaster Univ., 1999.Google Scholar
- Salas, E., Prince, C., Baker, D. P., & Shrestha, L., Situation awareness in team performance: Implications for measurements and training, Human Factors, 37 (1995), 123--136.Google ScholarCross Ref
- Scriven, M. The methodology of evaluation. In R. Tyler, R. Gagne & M. Scriven (Eds.), Perspectives of curriculum evaluation. Chicago: Rand McNally, p. 39--83, 1967.Google Scholar
- Steves, M., Morse, E., Gutwin, C., & Greengerg, S., A comparison of usage evaluation and inspection methods for assessing groupware usability. Proc. 2001 International ACM SIGGROUP Conference on Supporting Group Work. ACM Press (2001), 125--134. Google ScholarDigital Library
- Suchman, L. A. Plans and situated actions. New York: Cambridge University Press, 1986.Google Scholar
- Twidale, M. Randal, D., Bentley, R. (1994), Situated Evaluation for Co-operative Systems. Proc. CSCW'94, ACM Press (1994), 22--26. Google ScholarDigital Library
- Watts, L., Monk, A., & Daly-Jones, O. Inter-Personal Awareness and Synchronization: Assessing the Value of Communication Technologies. International Journal of Human-Computer Studies, 44 (6), (1996), 849--873. Google ScholarDigital Library
- Weisband, S., Overcoming social awareness in computer-supported groups: Does anonymity really help?, Proc. CSCW'94 (2) (1994), 285--297.Google Scholar
- Winograd, T., & Flores, F., Understanding computers and cognition: A new foundation for design. Norwood, NJ: Ablex, 1986. Google ScholarDigital Library
Recommendations
Evaluating computer-supported cooperative work: models and frameworks
CSCW '04: Proceedings of the 2004 ACM conference on Computer supported cooperative workEvaluating distributed CSCW applications is a difficult endeavor. Frameworks and methodologies for structuring this type of evaluation have become a central concern for CSCW researchers. In this paper we describe the problems involved in evaluating ...
Classroom BRIDGE: using collaborative public and desktop timelines to support activity awareness
UIST '03: Proceedings of the 16th annual ACM symposium on User interface software and technologyClassroom BRIDGE supports activity awareness by facilitating planning and goal revision in collaborative, project-based middle school science. It integrates large-screen and desktop views of project times to support incidental creation of awareness ...
Awareness and coordination in shared workspaces
CSCW '92: Proceedings of the 1992 ACM conference on Computer-supported cooperative work
Comments