Timing and frequency of mental effort measurement: Evidence in favor of repeated measures.

Abstract

Subjective mental effort rating scales are widely used in research on learning, instruction, and training. However, the timing and frequency of application of those rating scales differs between studies. Some apply a rating scale repeatedly after every task in a learning or test phase, others only once at the end of a phase. Four experiments are presented that investigated how timing and frequency of mental effort measurements affect the results obtained. The findings from Experiment 1 (between-subjects) and 2 (within-subjects), using different arrangements of simple and complex tasks, showed that a single rating after a series of tasks resulted in a higher mental effort score than the average of ratings provided immediately after every task. A similar result was obtained in Experiment 3 with series of complex tasks, but not with simple tasks. Experiment 4 showed that knowing beforehand that mental effort rating will be required after completing all tasks results in lower scores, but average retrospective ratings per task still differed from a single retrospective rating. Taken together, these experiments suggest that timing and frequency of effort ratings do affect the results obtained and that repeatedly measuring mental effort after each task in the series seems to be preferable.