Skip to main content
SpringerLink
Log in

Assessing the suitability of an honest rating mechanism for the collaborative creation of structured knowledge

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Creating and maintaining semantic structures such as ontologies continues to be an important issue. The approach investigated here is to let members of an online community create structured knowledge collaboratively and to use ratings to evaluate the data created. Obviously, such ratings have to be of high quality. Honest rating mechanisms (HRMs) known from literature are a promising means to gain such high-quality ratings. However, the design of such mechanisms for collaborative knowledge creation and their effectiveness have not been studied so far. To evaluate the effects of an HRM on rating quality in this context, we have conducted several experiments with online communities. We find that an HRM increases rating quality and “punishes” rating errors. We also find that rating-based rewards increase the quality of the structured knowledge created.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Albers, A., et al.: A new perspective on product engineering overcoming sequential process models. In: The Future of Design Methodology. Springer London (2011)

  2. Auer, S., Dietzold, S.: OntoWiki—a tool for social, semantic collaboration. In: Proceedings of the 5th International Semantic Web Conference ISWC (2006)

  3. Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., Kraut, R.E.: Using social psychology to motivate contributions to online communities. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, CSCW ’04, pp. 212–221. ACM, New York, NY, USA (2004)

    Chapter  Google Scholar 

  4. Benkler, Y.: The wealth of networks: how social production transforms markets and freedom. Yale University Press, New Haven, CT, USA (2006)

    Google Scholar 

  5. Bollacker, K., Evans, C., Paritosh, P., Sturge, T., Taylor, J.: Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, pp. 1247–1250. ACM, New York, NY, USA (2008)

    Chapter  Google Scholar 

  6. Braun, S., Schmidt, A., Walter, A., Nagypal, G., Zacharias, V.: Ontology maturing: a collaborative Web 2.0 approach to ontology engineering. In: Proceedings of the Workshop on Social and Collaborative Construction of Structured Knowledge at the 16th International World Wide Web Conference (WWW 07), Banff, Canada (2007)

  7. Buitelaar, P., Cimiano, P.: Ontology Learning and Population: Bridging the Gap Between Text and Knowledge. Frontiers in Artificial Intelligence and Applications. Ios Press (2008)

  8. Campanini, S.E., Castagna, P., Tazzoli, R.: Platypus wiki: a semantic wiki wiki Web. In: 1st Italian Semantic Web Workshop (2004)

  9. Cosley, D., Frankowski, D., Kiesler, S., Terveen, L., Riedl, J.: How oversight improves member-maintained communities. In: Proceedings of the SIGCHI conference on Human Factors in Computing Systems, CHI ’05, pp. 11–20. ACM, New York, NY, USA (2005)

    Chapter  Google Scholar 

  10. Cosley, D., et al.: Is seeing believing?: how recommender system interfaces affect users’ opinions. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2003)

  11. Douceur, J.R.: The sybil attack. In: IPTPS (2002)

  12. Eckert, K., et al.: Crowdsourcing the assembly of concept hierarchies. In: Proceedings of the 10th Annual Joint Conference on Digital Libraries - JCDL ’10 (2010)

  13. Efron, B.: Forcing a sequential experiment to be balanced. Biometrika 58(3), 403–417 (1971)

    Article  MATH  MathSciNet  Google Scholar 

  14. Fleiss, J.L.: Statistical Methods for Rates and Proportions, 2nd edn. Wiley, New York (1981)

    MATH  Google Scholar 

  15. Holsapple, C.W., Joshi, K.D.: A collaborative approach to ontology design. ACM Commun. 45(2), 42–47 (2002)

    Article  Google Scholar 

  16. Hütter, C., Kühne, C., Böhm, K.: Peer production of structured knowledge—an empirical study of ratings and incentive mechanisms. In: Proceeding of the 17th ACM Conference on Information and Knowledge Mining - CIKM ’08 (2008)

  17. Jäschke, R., et al.: Organizing publications and bookmarks in bibsonomy. In: CKC (2007)

  18. Jurca, R., Faltings, B.: Minimum payments that reward honest reputation feedback. In: Proceedings of the 7th ACM Conference on Electronic Commerce - EC ’06 (2006)

  19. Jurca, R., Faltings, B.: Collusion-resistant, incentive-compatible feedback payments. In: Proceedings of the 8th ACM Conference on Electronic Commerce, EC ’07, pp. 200–209. ACM, New York, NY, USA (2007)

    Chapter  Google Scholar 

  20. Kittur, A., Kraut, R.E.: Harnessing the wisdom of crowds in Wikipedia: quality through coordination. In: Proceedings of the 2008 ACM Conference on Computer Supported Cooperative work, CSCW ’08, pp. 37–46. ACM, New York, NY, USA (2008)

    Chapter  Google Scholar 

  21. Kochhar, S., Mazzocchi, S., Paritosh, P.: The anatomy of a large-scale human computation engine. In: Proceedings of the ACM SIGKDD Workshop on Human Computation, HCOMP ’10, pp. 10–17. ACM, New York, NY, USA (2010)

    Chapter  Google Scholar 

  22. Kotis, K., Vouros, A.: Human-centered ontology engineering: The hcome methodology. Knowl. Inf. Syst. 10(1), 109–131 (2006)

    Article  Google Scholar 

  23. Miller, N., Resnick, P., Zeckhauser, R.: Eliciting informative feedback: the peer-prediction method. Manag. Sci. 51(9), 1359–1373 (2005)

    Article  Google Scholar 

  24. Moor, A.D., Leenheer, P.D., Meersman, R., Starlab, V.: Dogma-mess: a meaning evolution support system for interorganizational ontology engineering. In: In Proc. of the 14th International Conference on Conceptual Structures, ICCS 2006, pp. 189–203. Springer-Verlag (2006)

  25. Noy, N., Chugh, A., Alani, H.: The ckc challenge: exploring tools for collaborative knowledge construction. IEEE Intell. Syst. 23(1), 64–68 (2008)

    Article  Google Scholar 

  26. Pepper, S., Architect, S.I., Infotek, S.: The TAO of Topic Maps—Finding the Way in the Age of Infoglut. In: Proceedings of XML Europe (2000)

  27. Prelec, D.: A Bayesian truth serum for subjective data. Science (New York, N.Y.) 306, 462–466 (2004)

    Article  Google Scholar 

  28. Schaffert, S.: IkeWiki: a semantic wiki for collaborative knowledge management. In: Tolksdorf, R., Simperl, E.P.B., Schild, K. (eds.) In: Proceedings of the First International Workshop on Semantic Technologies in Collaborative Applications STICA 06, pp. 388–396 (2006)

  29. Shaw, A.D., Horton, J.J., Chen, D.L.: Designing incentives for inexpert human raters. In: Proceedings of the ACM 2011 Conference on Computer Supported Cooperative work, CSCW ’11, pp. 275–284. ACM, New York, NY, USA (2011)

    Chapter  Google Scholar 

  30. Siorpaes, K., Hepp, M.: Games with a purpose for the semantic Web. IEEE Intell. Syst. 23(3), 50–60 (2008)

    Article  Google Scholar 

  31. Siorpaes, K., Simperl, E.: Human intelligence in the process of semantic content creation. World Wide Web 13(1–2), 33–59 (2010)

    Article  Google Scholar 

  32. Souzis, A.: Building a semantic wiki. IEEE Intell. Syst. 20, 87–91 (2005)

    Article  Google Scholar 

  33. Sunagawa, E., Kozaki, K., Kitamura, Y., Mizoguchi, R.: An environment for distributed ontology development based on dependency management. In: International Semantic Web Conference, pp. 453–468 (2003)

  34. Tijerino, Y.A., Embley, D.W., Lonsdale, D.W., Ding, Y., Nagy, G.: Towards ontology generation from tables. World Wide Web 8(3), 261–285 (2005)

    Article  Google Scholar 

  35. Tudorache, T., Noy, N.F.: Collaborative protege. In: CKC (2007)

  36. Viegas, F.B., Wattenberg, M., Kriss, J., van Ham, F.: Talk before you type: coordination in Wikipedia. In: Proceedings of the 40th Annual Hawaii International Conference on System Sciences, HICSS ’07, p. 78. IEEE Computer Society, Washington, DC, USA (2007)

    Google Scholar 

  37. Völkel, M., Krötzsch, M., Vrandecic, D., Haller, H., Studer, R.: Semantic Wikipedia. In: WWW, pp. 585–594 (2006)

  38. von Ahn, L.: Games with a purpose. IEEE Comput. 39(6), 92–94 (2006)

    Article  Google Scholar 

  39. Vrandecic, D., Pinto, H.S., Tempich, C., Sure, Y.: The diligent knowledge processes. J. Knowl. Manag. 9(5), 85–96 (2005)

    Article  Google Scholar 

  40. Wilkinson, D.M., Huberman, B.A.: Assessing the value of cooperation in Wikipedia. First Monday 12(4) (2007)

  41. de Winter, J.C., Dodou, D.: Five-point Likert items: t test versus Mann-Whitney-Wilcoxon. Pract. Assess., Res. Eval. 15(11) (2010)

  42. Xu, Z., Ni, Y., He, W., Lin, L., Yan, Q.: Automatic extraction of OWL ontologies from UML class diagrams: a semantics-preserving approach. World Wide Web 15(5–6), 517–545 (2011)

    Google Scholar 

  43. Zacharias, V., Braun, S.: Soboleo—social bookmarking and lighweight engineering of ontologies. In: CKC (2007)

Download references

Author information

Authors and Affiliations

  1. Institute for Program Structures and Data Organisation, Karlsruhe Institute of Technology, Am Fasanengarten 5, Building 50.34, 76131, Kalrsruhe, Germany

    Conny Kühne & Klemens Böhm

Authors
  1. Conny Kühne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Klemens Böhm
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Conny Kühne.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kühne, C., Böhm, K. Assessing the suitability of an honest rating mechanism for the collaborative creation of structured knowledge. World Wide Web 17, 85–104 (2014). https://doi.org/10.1007/s11280-012-0193-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-012-0193-1

Keywords

Search

Navigation