Ardalan et al. (2007)
A comparison of student feedback obtained through paper-based and web-based surveys of faculty teaching
Summary: This paper looks at the differences in student behaviour and responses between a paper-based and a web-based satisfaction survey. The research looked at the difference between the survey completed online in the Spring against a previous paper-based one from the previous Spring. Both questionnaires were from the same faculties and same courses. The demographics were also similar. The questionnaire combined quantitative and qualitative responses. However, though the quantitative responses could be easily compared, the authors needed to develop a way of comparing the qualitative responses. Comments were categorised as positive, negative or mixed and as constructive or non-constructive, depending on whether they provided something that could be used to make improvements. Constructive comments were argued to reflect thoughtfulness. Where students provided a rationale for their constructive comment, this was defined as a qualified constructive comment and represented a meaningful comment.
The conclusions were that there were fewer responses, quality of teaching was rated no differently and ratings were less extreme in web-based surveys. The number of comments overall were the same and were no more thoughtful, but were more meaningful. Like Bos et al, the data was tested statistically for difference. The conclusion was that overall there was little difference in the quality of responses, but a significant reduction in the number of responses would result in small samples from which managers could draw conclusions and therefore the need to encourage more responses.
Strength of this research: The study made valid conclusions based on the findings. The proxy measures of thoughtfulness and meaningfulness were sound. Even though ‘thoughtfulness’ and ‘meaningfulness’ are themselves quite vague terms, the categorisation did distinguish between increasing levels of feedback, which made a useful comparison over the two formats.
Weakness of study: Weak explanations of why response was low and comments more meaningful, but less thoughtful. Researchers could question students on their attitudes to the survey. The two surveys involved two different cohorts, and attitudes towards surveys may be dfferent from one year to the next. Ideally response rates and comment quality should have been averaged over several paper surveys and then over several online surveys to control for this.
Though the media being researched is new, the research techniques are not, they are standard, using standard statistical tests for difference, such as chi-squared and t-test.