Abstract
In this paper we discuss evaluation of information retrieval, Web search and question answering systems, paving the way for the organization of an evaluation contest on IR for Portuguese. Inspired by current international setups, we motivate the need to study the specific problems posed by Portuguese, suggesting a collection suitable for multiple tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Voorhees, E., Harman, D.: Overview of TREC-2001. Proceedings of TREC’2001 (2001). Available at http://trec.nist.gov
Aires, R., Santos, D.: Measuring the Web in Portuguese. In: Brian Matthews, Bob Hopgood & Michael Wilson (eds.): Euroweb 2002 conference (2002) 198–199
Hirschman, L.: The evolution of Evaluation: Lessons from the Message Understanding Conferences. Computer Speech and Language 12(4) (1998) 281–305
Quaresma, P., Rodrigues, I.: PGR: Portuguese Attorney General’s Office Decisions on the Web. In: Bartenstein, Geske, Hannebauer, and Yoshie (eds): Web-Knowledge Management and Decision Support. LNCS/LNAI 2543 Springer-Verlag (2003) 51–61
Grishman, R., Sundheim, B.: Message Understanding Conference — 6: A Brief History. Proceedings of COLING-96 (1996) 466–471
Santos, D., Rocha, P.: AvalON: uma iniciativa de avaliação conjunta para o português. In: Actas do XVIII Encontro da Associação Portuguesa de Linguística 2002 (forthcoming).
Peters, C.: Results of the CLEF 2002 Cross-Language System Evaluation Campaign. In: Peters, C. (Ed.): Working Notes for the CLEF 2002 (2002). Available at: http://clef.iei.pi.cnr.it:2002/2002WN.html
Eguchi, K., Oyama, K., Ishida, E., Kando, N., Kuriyama, K.: Overview of the Web Retrieval Task at the Third NTCIR Workshop. NII Technical Report, No.NII-2003-002E (2003)
Mandl, T., Womser-Hacker, C.: Linguistic and Statistical Analysis of the CLEF Topics. In: Peters, C. (Ed.): Working Notes for the CLEF 2002 Workshop (2002)
Orengo, V., Huyck, C.: Portuguese-English Experiments using Latent Semantic Indexing. In: Peters, C. (Ed.): Working Notes for the CLEF 2002 Workshop (2002)
Saracevic, T.: Evaluation of evaluation in information retrieval. Proceedings of SIGIR 95 (1995) 138–46
Arasu, A. Cho, J. Molina, H. Paepcke, A., Raghavan, S.: Searching the Web. ACM Transactions on Internet Technology 1(1) (2001) 2–43
Brin, S., Page, L.: The Anatomy of a Large-Scale Hypertextual (Web) Search Engine. Computer Networks and ISDN Systems, 30(1–7) (1998) 107–117
Hawking, D., Craswell, N., Bailey, P., Griffiths, K.: Measuring Search Engine Quality. Information Retrieval l4(1) (2001) 33–59
Dean, J. Henzinger, M.: Finding related pages in the World Wide Web. Proceedings of the Eighth International World Wide Web Conference (1999) 389–401
Garfield, E.: Citation Analysis as a tool in journal evaluation. Science 178 (1972) 471–479
Small, H.: Co-citation in the scientific literature: A new measure of the relationship between two documents Journal of American Society for Information Science, 24,4, (1973) 265–269
Zamir, O., Etzioni, O.: Web document clustering: A feasibility demonstration. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (1998) 46–54
Silva, M. J.: The Case for a Portuguese Web Search Engine. DI/FCUL Technical Report 03-3 (2003). Available at http://www.di.fc.ul.pt/tech-reports/03-3.pdf
Gwizdka, J., Chignell M.: Towards Information Retrieval Measures for Evaluation of Web Search Engines. Unpublished manuscript (1999). Available at http://www.imedia.mie.utoronto.ca/~jacekg/pubs.html
Burton, M., Walther, J.: The value of Web log data in use-based design and testing. Journal of Computer-Mediated Communication, 6(3) (2001). Available at http://www.ascusc.org/jcmc/vol6/issue3/burton.html
Joachims, T.: Evaluating Retrieval Performance Using Clickthrough Data. Proceedings of the SIGIR Workshop on Mathematical/Formal Methods in Information Retrieval (2002). Available at http://www.cs.cornell.edu/People/tj/publications/joachims_02b.pdf
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Aires, R., Aluísio, S., Quaresma, P., Santos, D., Silva, M.J. (2003). An Initial Proposal for Cooperative Evaluation on Information Retrieval in Portuguese. In: Mamede, N.J., Trancoso, I., Baptista, J., das Graças Volpe Nunes, M. (eds) Computational Processing of the Portuguese Language. PROPOR 2003. Lecture Notes in Computer Science(), vol 2721. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45011-4_36
Download citation
DOI: https://doi.org/10.1007/3-540-45011-4_36
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-40436-1
Online ISBN: 978-3-540-45011-5
eBook Packages: Springer Book Archive