Staff View: Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment

Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment

Purpose: The effort in addition to relevance is a major factor for satisfaction and utility of the document to the actual user. The purpose of this paper is to propose a method in generating relevance judgments that incorporate effort without human judges’ involvement. Then the study determines the...

Full description

Saved in:

Bibliographic Details
Main Authors:	Rajagopal, Prabha, Ravana, Sri Devi, Koh, Yun Sing, Balakrishnan, Vimala
Format:	Article
Published:	Emerald Publishing 2019
Subjects:	QA75 Electronic computers. Computer science QA76 Computer software
Online Access:	http://eprints.um.edu.my/20008/ https://doi.org/10.1108/AJIM-04-2018-0086
Tags:	Add Tag No Tags, Be the first to tag this record!

id	my.um.eprints.20008
record_format	eprints
spelling	my.um.eprints.200082019-01-15T03:06:45Z http://eprints.um.edu.my/20008/ Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment Rajagopal, Prabha Ravana, Sri Devi Koh, Yun Sing Balakrishnan, Vimala QA75 Electronic computers. Computer science QA76 Computer software Purpose: The effort in addition to relevance is a major factor for satisfaction and utility of the document to the actual user. The purpose of this paper is to propose a method in generating relevance judgments that incorporate effort without human judges’ involvement. Then the study determines the variation in system rankings due to low effort relevance judgment in evaluating retrieval systems at different depth of evaluation. Design/methodology/approach: Effort-based relevance judgments are generated using a proposed boxplot approach for simple document features, HTML features and readability features. The boxplot approach is a simple yet repeatable approach in classifying documents’ effort while ensuring outlier scores do not skew the grading of the entire set of documents. Findings: The retrieval systems evaluation using low effort relevance judgments has a stronger influence on shallow depth of evaluation compared to deeper depth. It is proved that difference in the system rankings is due to low effort documents and not the number of relevant documents. Originality/value: Hence, it is crucial to evaluate retrieval systems at shallow depth using low effort relevance judgments. Emerald Publishing 2019 Article PeerReviewed Rajagopal, Prabha and Ravana, Sri Devi and Koh, Yun Sing and Balakrishnan, Vimala (2019) Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment. Aslib Journal of Information Management, 71 (1). pp. 2-17. ISSN 2050-3806 https://doi.org/10.1108/AJIM-04-2018-0086 doi:10.1108/AJIM-04-2018-0086
institution	Universiti Malaya
building	UM Library
collection	Institutional Repository
continent	Asia
country	Malaysia
content_provider	Universiti Malaya
content_source	UM Research Repository
url_provider	http://eprints.um.edu.my/
topic	QA75 Electronic computers. Computer science QA76 Computer software
spellingShingle	QA75 Electronic computers. Computer science QA76 Computer software Rajagopal, Prabha Ravana, Sri Devi Koh, Yun Sing Balakrishnan, Vimala Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment
description	Purpose: The effort in addition to relevance is a major factor for satisfaction and utility of the document to the actual user. The purpose of this paper is to propose a method in generating relevance judgments that incorporate effort without human judges’ involvement. Then the study determines the variation in system rankings due to low effort relevance judgment in evaluating retrieval systems at different depth of evaluation. Design/methodology/approach: Effort-based relevance judgments are generated using a proposed boxplot approach for simple document features, HTML features and readability features. The boxplot approach is a simple yet repeatable approach in classifying documents’ effort while ensuring outlier scores do not skew the grading of the entire set of documents. Findings: The retrieval systems evaluation using low effort relevance judgments has a stronger influence on shallow depth of evaluation compared to deeper depth. It is proved that difference in the system rankings is due to low effort documents and not the number of relevant documents. Originality/value: Hence, it is crucial to evaluate retrieval systems at shallow depth using low effort relevance judgments.
format	Article
author	Rajagopal, Prabha Ravana, Sri Devi Koh, Yun Sing Balakrishnan, Vimala
author_facet	Rajagopal, Prabha Ravana, Sri Devi Koh, Yun Sing Balakrishnan, Vimala
author_sort	Rajagopal, Prabha
title	Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment
title_short	Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment
title_full	Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment
title_fullStr	Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment
title_full_unstemmed	Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment
title_sort	evaluating the effectiveness of information retrieval systems using effort-based relevance judgment
publisher	Emerald Publishing
publishDate	2019
url	http://eprints.um.edu.my/20008/ https://doi.org/10.1108/AJIM-04-2018-0086
_version_	1643691152014049280
score	13.211869

Evaluating the effectiveness of information retrieval systems using effort-based relevance judgment

Similar Items