HIVK03041U System Evaluation and User Studies

Volume 2013/2014
Education
Kandidat i Informationsvidenskab og kulturformidling
Content

The module includes advanced studies of the theoretical and methodological aspects concerning evaluation of interactive information systems in context, the emphasis being on the correlation between information needs, situation and relevance as well as search behaviour seen in the light of dissemination processes and associated technologies.  Taking its starting point in the research literature, the module introduces models for information behaviour and information interaction in order to identify approaches to evaluation of system performance and usability, and explicitly to discuss methods for evaluation of interactive information retrieval systems.

The module consists of the following three sub-courses and a project seminar:

Information behaviour

  • Basic information behaviour models and interactive information retrieval research approaches
  • Types of information needs, including work task related information needs
  • The concept of relevance

Data collection and analysis

  • Research focus: behavioural studies vs. performance tests
  • Research design – test variables
  • Data collection methods: qualitative methods and statistical methods of analysis

Evaluation of interaction design

  • System centric evaluation; Cranfield-TREC-INEX; Performance measures
  • User centric evaluation; usability evaluation and analysis
  • Validation of test results

 

Learning Outcome

Competence objectives for the module

The objective of the module is to provide the student with:

Knowledge and understanding of:

  • Different types of evaluation and evaluation approaches.
  • Central concepts and models for information behaviour and search/retrieval and the related processes and functions, as well as methods for studying the same.

Skills in:

  • Reflecting on central models and methods for the study of information behaviour and search/retrieval as well as the associated processes and functions.
  • Evaluating applicable methods, and a combination of these, for the study/evaluation of system performance and information behaviour – including evaluation of interactive information retrieval systems.
  • Identifying and comparing central evaluation concepts and methods for system evaluation and user studies, and to assess and validate evaluation results in relation to given circumstances.

Competences in:

  • Preparing research design for assessing system performance and information behaviour – including evaluation of interactive information retrieval systems.
  • Conducting various types of system evaluation and user studies.

Academic objectives:
The examinee is able to

  • Give an account of different types of evaluation and evaluation approaches.
  • Assess suitable methods, and combinations of these, for studies/evaluation of system performance and information behaviour.
  • Conduct various types of system evaluation and user studies independently.
Literature

Text books, research articles and conference papers up to 1500 normal pages.

Sample literature

  • Lazar, Feng & Hochheiser (2010): Research Methods in Human-Computer Interaction. Chichester, Wiley, 426 s.
  • Ingwersen, P. & Järvelin, K. (2005). The Turn: Integration of Information Seeking and Retrieval in Context. Berlin/Heidelberg : Springer. 448 s.
  • Kelly, D. (2009). Methods for Evaluating Interactive Information Retrieval Systems with Users. Foundations and Trends in Information Retrieval, 3(1-2), 1-224.
  • Lazar, Feng & Hochheiser (2010): Research Methods in Human-Computer Interaction. Chichester, Wiley, 426 s.
E.g. class teaching, lectures, student presentations and workshops
  • Category
  • Hours
  • Class Instruction
  • 0
  • Total
  • 0
Credit
15 ECTS
Type of assessment
Written assignment
Examination form: Written paper with subject chosen by the student
Assessment form: 7-step scale
Marking form: Internal marking
Examination language(s): English
Scope: The examination may be taken individually or as a group examination with max. 5 participants and individual assessments. The scope of the assignment is max. 20 standard pages + 10 standard pages per student in addition to one. Each student's contribution must be a complete whole, which is identified and can be assessed on its own. The participants' joint contribution must not exceed 50 % of the entire length of the paper.
Marking scale
7-point grading scale
Censorship form
No external censorship
Exam period
June 2014
Re-exam
Re-examination form is the same as for the ordinary examination in August 2014
Criteria for exam assesment

The module is assessed according to the 7-point grading scale.

The mark 12 is given to:
For an excellent performance displaying a high level of command of all aspects of the relevant material, with no or only a few minor weaknesses

  • The student is very certain in his/her analyses and reflections with respect to discussing and applying central models and methods for studies of information behavior and retrieval, as well as related processes and functions.
  • The student can with great certainty identify and compare central evaluation concepts and methods in relation to interactive information retrieval systems as well as assess and validate evaluation results in relation to given contexts.
  • The student can design experiments for the studies or evaluation of information behavior.

The mark 7 is given to:
For a good performance displaying good command of the relevant material but also some weaknesses

  • The student is somewhat certain in his/her analyses and reflections with respect to discussing and applying central models and methods for studies of information behavior and retrieval, as well as related processes and functions.
  • The student can with some certainty identify and compare central evaluation concepts and methods in relation to interactive information retrieval systems as well as assess and validate evaluation results in relation to given contexts.
  • The student can with some difficulty design experiments for the studies or evaluation of information behavior.

The mark 02:
For a performance meeting only the minimum requirements for acceptance

  • The student is uncertain in his/her analyses and reflections with respect to discussing and applying central models and methods for studies of information behavior and retrieval, as well as related processes and functions.
  • The student can with modest certainty identify and compare central evaluation concepts and methods in relation to interactive information retrieval systems as well as assess and validate evaluation results in relation to given contexts.
  • The student can with modest certainty design experiments for the studies or evaluation of information behavior.