E835, 1997, Educational Research in Action pp33-35
Validity
By ‘validity’, we mean truth: the extent to which an account accurately represents the phenomena to which it refers. There are few concepts that have led to more controversy than ‘truth’. Indeed, many educational researchers reject this concept as inapplicable to the assessment of their work. One reason for this is that they believe that use of the term ‘truth’ implies the possession of knowledge which is absolutely certain, proven beyond all possible doubt; and they think (quite rightly in our view) that knowledge can never be certain in this sense.
These critics point out that in deciding the validity or otherwise of some claim we always rely on presuppositions whose own validity we must take for granted. And if we seek to test any of those presuppositions, we shall be forced to rely on further ones. For instance, even in simple measurements of physical objects with a ruler we take for granted certain properties of rigid bodies, in this case of the ruler and of the object. We assume, for example, that small changes in temperature will not have any significant effect on our measurements. And testing these presuppositions would involve us in measuring temperature, which itself involves presuppositions about the operation of thermometers, and so on.
The second half of this argument is sound, but the first half – that use of the concept of truth implies knowledge that is beyond all possible doubt – is not. To claim that some statement is true is not incompatible with a recognition that it could turn out to be false. In fact, we can never be absolutely certain about the truth of anything, not even in the natural sciences or in our personal lives. There are, however, many things about whose truth we are very confident, and justifiably so. This means that there can be good reasons for believing that some statement is true or false without our ever having absolutely conclusive evidence for it. We rely on a whole host of presuppositions about the world in our everyday lives, and we can never be confident beyond all doubt about the truth of any of them. Yet this ever-present uncertainty does not undermine our use of the concept of truth in that context, and there is no reason why it should do so in the research situation either.
A second source of problems with ‘truth’ arises from beliefs about the nature of human social life. There are those who hold that in the case of social phenomena there is no single reality to which claim made in research reports correspond. It is argued that human beings create multiple social worlds or realities, that all perception and cognition involves the construction of phenomena rather than their mere discovery. It may be concluded from this that contradictory views of the ‘same’ phenomena by different cultural groups are equally ‘true’ in their own terms. Of course, if we apply this argument to educational research itself, as we should do if we are to be consistent, we must see that research as also creating a world (or, given the disagreements among social scientists, as creating multiple worlds!). This is the sort of relativism that we discussed in Section 1, from which point of view research reports do not provide an account of an independent reality, but create the social worlds they purport to describe, for instance through textual strategies of various kinds. Effectively, the argument here is not just that we can never be sure of the truth or falsity of our claims about reality, but that we have no grounds for believing that there are phenomena which are independent of our knowledge of them, since all the knowledge we can ever have is formed by our culture, and that culture is only one of many.
While this argument makes an important point about the limits and difficulties of understanding resulting from cultural diversity, the conclusion that there are no phenomena independent of researchers for them to document does not follow from it. All social phenomena are human products and are therefore, in some sense, not independent of humanity as a whole. However, much social life is independent of any particular researcher or group of researchers. Furthermore, we are able to learn other cultures to one degree or another and, thereby, to understand human behaviour that is framed in terms of them. This implies some commonalities among cultures on which knowledge may be built. It is also worth pointing out that to claim that there are diverse cultures itself implies the claim that there is a larger world within which those cultures are to be found, a world which is not itself relative. In this way, relativism is self-contradictory.
Of course, even if we accept that validity (interpreted as truth) is a feasible and legitimate criterion in terms of which to assess educational research findings, the question remains: on what basis can this assessment be carried out; given that there is no foundation of knowledge whose validity is certain, in terms of which researchers can validate their accounts? The only basis available, it seems to us, is a judgement about what is beyond reasonable doubt. From this point of view, there are three steps in assessing the validity of research claims:
1) The first question that we must ask about a claim is how plausible is it: that is, whether we judge it as very likely to be true given our existing knowledge. In the case of some claims, they will be so plausible that we can reasonably accept them at face value (until further notice) without needing to know anything about how researchers came to formulate them or what evidence is offered in support of them. (Examples might include that most schools in Britain are now mixed sex, or that the ‘setting’ of students in ability groups for different subjects is very common in secondary schools.) The first test, then, is plausibility.
2) A second question we may need to ask is whether it seems likely that the researcher’s judgement of matters relating to the claim is accurate, given the nature of the phenomena concerned, the circumstances of the research the characteristics of the researcher, etc. We shall call this credibility’. In assessing credibility, we make a judgement about the likely threats to validity involved in the production of a claim and the likely size and direction of their effect. As with plausibility, there are claims whose credibility is such that we can reasonably accept them without further ado (unless and until we discover new doubts about them).
3) Of course, where we conclude that a claim is neither sufficiently plausible nor sufficiently credible to be accepted at face value, to be convinced of its validity we shall require evidence. When we examine the evidence, however, we will have to employ much the same means to assess its validity as we applied to the claim itself: we will judge its plausibility and credibility. And, of course, we may require further evidence to support the evidence, which we shall again judge in terms of plausibility and credibility.
In many respects, this seems to us to be the sort of basis on which we judge claims in everyday life. It is important to point out, though, that we do not do it on a purely individual basis. As co-participants in various communal activities we compare our judgements with those of others; and where there are disagreements these may need to be resolved In a research situation, moreover, there is an obligation to make public our findings and to try to resolve disagreements through rational discussion. Such discussion may reveal to us that what we have been disposed to accept as adequately plausible or credible should not be accepted; or it may reveal errors in the presuppositions on which others’ judgements were made. It is the function of the research community to act as a corrective to the beliefs of individual researchers in this way.
Needless to say, plausibility and credibility are a relatively weak basis for judging the validity of claims compared to the idea that we can assess claims directly according to their correspondence with reality or by relying on some body of knowledge whose validity is certain. The approach we are suggesting provides no guarantee that our judgements will be correct, nor any way of knowing for certain whether they are correct. Neither will judgements always be consensual, since there are very likely to be different views about what is plausible and credible, and rational discussion carries no guarantee of agreement. In our view, however, this is the only viable basis for assessing the validity of claims to knowledge that is available to us.
Truth or validity is the first criterion, then, in terms of which we believe that research accounts should be judged. Equally important, though, in many ways, is relevance.
It is also of significant in this context that research reports are communications addressed to an audience. When we communicate with people, they assume that we are telling them something which is likely to be of significance to them. If it turns out that we merely communicate facts, any facts, we shall soon find that we have few listeners. In the case of most research, the aim is to communicate with a relatively large audience (otherwise, what would be the point of publication?).
It follows that what is communicated should be relevant in some way to such an audience. The obvious questions that follow from this are who are the appropriate audience and what sort of relevance should educational research have for them?
Intended audiences for research reports vary of course. Sometimes reports are directed primarily towards other researchers; sometimes they are addressed to policymakers or to a particular practitioner audience; sometimes they are directed at a general audience. Whatever the intended audience, though, there are two aspects of relevance that need to be distinguished:
The importance of the topic. The research topic must relate (directly or indirectly) to an issue of importance to the intended audience.
The contribution of the conclusions to our existing knowledge. The research findings must add something to our knowledge of the issue to which they relate. Research that merely confirms what is already beyond reasonable doubt makes no contribution. (Though research which corroborates what was previously suspected but not known with confidence is of value.)
In these terms, research findings may connect with an important topic, but still not be relevant since they do not tell us anything new about it. Conversely, research may add new knowledge, but this may relate to no topic of any significance and so still lack relevance. Importance and contribution are necessary and jointly sufficient conditions for relevance, and we must assess any set of findings in terms of both; though, of course, there may be disagreements about this assessment.
Relevance
To be of value, research findings must not only be valid, but must also be relevant to issues of legitimate public concern, however remotely. This second criterion is curiously neglected in many treatments of educational research methodology. It is rarely mentioned explicitly in discussions of standards of assessment, whether those concerned with quantitative research or those offering distinctive qualitative criteria.
Our interest in facts, in everyday life as much as in educational research, is selective. All descriptions are for some purpose, and the nature of the purpose will crucially shape the character of the description. If, say, we are describing a classroom-based lesson, it will make a considerable difference whether our interest is in the extent to which there is balanced participation between girls and boys, in whether the teacher seems to discriminate against black students, in how the knowledge which forms part of the syllabus is presented, or in something else. The descriptions produced on the basis of these various interests may overlap, but equally they may be so different as to be not recognizably referring to the same lesson (although if we believe that they are all true they should not contradict one another). And, just as there is always a large number of points of view from which we could describe a phenomenon, so too with explanations. These will not only be concerned with accounting for some aspects of a phenomenon rather than others, but will also involve the selection of explanatory factors in terms of the purpose that the explanation is to serve, as well as on the basis of their causal relationship with what is to be explained.