Abstract

It is common belief that evidence produced by qualitative research is bounded by the researcher’s personal interpretation, and as such is subjective and not generalizable. Qualitative researchers’ personal involvement could highlight hidden aspects of social life and provide insight into people’s perspectives. In educational contexts, research aims to critically inform educational judgments and decisions in order to improve educational action. This study assesses the argument that qualitative research can offer no more than subjective impressions with an emphasis on structured observation as opposed to audio recordings and field notes for the purposes of observation. This discussion includes related concepts such as “point sampling”, “(negative) demand characteristics” and “participants’ reactivity” and reflexivity. The study concludes by arguing that the claim that qualitative research can offer no more than subjective impressions is a rhetorical device rather than a methodological position. Quantitative researchers strive to produce scientific data based on objective evaluations by avoiding any personal involvement in their evaluations and by investigating causal relationships in the widest possible populations. It is this growing interest in generalizability in the qualitative tradition that indicates the researchers’ concern in making their research valid and as widely accepted as possible.

1. Introduction

Educational research and practice are closely related and this has consequences in choosing the appropriate research method for a given situation and setting ¹. The wide disagreement as to what constitutes good educational research is reflected in the existence of a variety of research methods. Positivist, interpretivist, critical and constructivist researchers provide their own arguments about the appropriateness of their research methods over others.

It is common belief that evidence produced by qualitative research is bounded by the researcher’s personal interpretation, and as such, it is subjective and not generalizable. One of the main reasons for this criticism is the practitioner’s involvement in the research process and participants’ reluctance and/or (inherent) inability to express themselves openly and freely ³. A serious implication has been the difficulty, and a lack of interest (at least until recently), in generalizing the evidence produced by qualitative researchers.

Qualitative researchers’ personal involvement could highlight hidden aspects of social life and provide insight into people’s perspectives. They provide their own account of social life, but they try to do it through the research subjects’ eyes and voice. In this study, we will use the term (research) participant rather than (research) subject, to underscore that those being studied are active participants, rather than passive subjects of an investigation. More and more of these researchers try to “fit” the findings of one situation into another, without ignoring the fact that the ultimate role of qualitative research is to examine aspects of social life within a given context ⁴.

This study critically assesses the argument that qualitative research can offer no more than subjective impressions. Particular emphasis will be given to the advantages and disadvantages of structured observation, in comparison with using audio recordings and field notes for the purposes of observation. The study concludes with the possibility of combining qualitative and quantitative research methods ⁵.

2. The Qualitative Nature of Educational Research

Bassey ¹ has brought together several different views on educational research and argues that “educational research aims critically to inform educational judgments and decisions in order to improve educational action”. He goes on to claim that good educational research involves both practice and policy. While there is almost unanimous agreement on this view ⁶, what is being greatly disputed is the means by which researchers should inform their educational judgments and decisions.

Qualitative research aims to explore an issue without a pre-specified hypothesis and to unravel the various aspects of that issue while collecting and analyzing data. The researchers usually work with unstructured data, which includes observational data, documents and unstructured interviews. They are eager to account for the perspectives and activities of the people and their discourses, usually within a given context, which can only be done if the researcher provides his own perspective of the events.

There is strong disagreement among researchers as to what constitutes good educational research practice and, in particular, about “the nature of knowledge and the proper relationship between research and other activities” ⁶. This is usually the case with researchers whose work is guided by a positivist paradigm, believing as they do that research should produce objective knowledge ¹. At the same time, this critique makes reference to research and its relation to practice. Educational research that seeks to improve practice must be based both on real educational events and on abstract theories. Lack of clarity over aims can lead to confusion on the part of the researcher as to the most appropriate method for the topic to be investigated.

Hellawell ⁷ claims that one of the hallmarks of research is the ability to be conscious of the selection and modification of the research methodology. If we accept such a reflexivity as remedial in research, then it would require an openness, the participation of others, and negotiation about the ownership of data and the uses that the researcher put it to ¹. The significance of being reflexive and ethical, in the sense of developing research relationships with the participants (at least in action research), is highlighted by Rallis and Rossman ⁸, who argue that “ethically reflexive practice is [important] to conducting a trustworthy study”.

In a classroom educational setting, teachers, students and other participants (e.g. invited subject/discipline experts) make up a challenging educational mixture. The teacher’s catalytic role in this bidirectional mode of learning is, according to Kiraly ⁹, an example of a social constructivist and collaborative learning approach to pedagogy. In such educational contexts, action research is a widely used methodology ¹⁰. A key concept in action research is reflection, because it is the last step of the cycle and the turning point that enables researchers to make the proper diagnosis and redesign the series of steps involved. The importance of reflection in teacher education is evident in the argument that:

reflective teaching has come to signify a movement in teacher education, in which student teachers or working teachers analyze their own practice and its underlying basis, and then consider alternative means for achieving their ends [...] The use of the term reflection in the context of instruction can be interpreted in the sense of (1) thoughtful consideration, as well as in the sense of (2) mirroring, symbolizing or representing ¹¹.

Yet, Bassey ¹ warns action researchers about the dangers of such an emotive and cognitive involvement in the research process. This type of qualitative approach requires a great deal of self-awareness and self-criticism by the researcher so as not to misguide the participants and bias the research findings.

Qualitative research is often characterized as subjective, suggesting that research practice should not be guided by the researcher’s personal interpretations. This is a positivist orientation to research which is based on causal or statistical relationships among variables ⁶. In general, it favors objective evaluations and rejects personal (subjective) or social evaluations, in an attempt to achieve so-called procedural objectivity. This kind of research should be linked directly to the past where we can trace linear causal, or even correlational, relationships where alternative explanations are removed or variables controlled.

According to Cage ¹², scientific methods can, and should, be employed to “explore social constructions of reality”. From such a positivist perspective, the researcher would employ empirical evidence to contribute to social and political progress by undermining well-established views. For example, the widespread view that using mobiles is harmful to students’ ability to learn is challenged for lack of a substantive number of students having been tested over a long period. The need for such quantitative studies is evident by the fact that the subjective impression that warns parents of the dangers that their children face by using cell phones has not prevented the former from providing the latter with cell phones.

3. Qualitative and Quantitative Aspects of Research in Educational Contexts

There are various ways of conducting educational research. Until recently, such research was dominated by the positivism of the quantitative method. This approach ensures the objective measurement of relationships between variables by means of statistical analysis, observational surveys, questionnaires, interviews and experiments, to mention but a few of them ⁶.

The basic principles of quantitative research, and scientific or positivistic methods, involve testing hypotheses, measuring effectiveness and seeking the causes of measurable effects while focusing on objectivity and statistics. They consider as feasible the mixing of these two research methods in a sort of sequential reflective chain or spiral. Cage ¹² might be right then to urge researchers to devote all their efforts in meeting their moral obligations in society as educators. No matter what research methods educational researchers use mostly, they should do their best to meet students’ needs even if this calls for adopting a rivaling research method.

Positivists have adopted the experimental natural sciences model. Accurate and valid data could be safely produced by the statistical analysis of data, and the careful observation of phenomena and the possible causal relationships between the variables. Randomized controlled trials, which exemplify the research aims of positivists, form the basis of an evidence-based practice movement which began in the field of health care ⁶. These trials are studies that randomly assign individuals to an intervention group or a control group, in order to measure the effects of the intervention.

In the field of teaching, possible ways of producing objective research data is to use structured classroom observation, statistics, experimental tests, attitude inventories and statistics to estimate the reliability and the interrelationships of such measures. For example, it is important to “search for relationships between classroom processes and students’ subsequent achievements” ¹². In relation to this, it is essential to explore teachers’ pedagogical content knowledge and the techniques they use ¹³. These content-specific ways refer to the teachers’ organization of their own knowledge, its presentation in the classroom and how it is presented to the students.

A method that shares grounds with objective research is Karl Popper’s “piecemeal social engineering” ¹². According to this method, causes and effects are inextricably bound together and the researcher is always looking back to link the past with the present in such a way as to avoid negative consequences for the future. This piecemeal social technology, as a scientific methodology, could be used to study classroom teaching, taking for granted that teaching, as a social arrangement that remains relatively unchanged over time, can be studied as long as it remains constant or tracked when a change occurs. A major limitation for such a “piecemeal theory” is that the application of causal scientific methods cannot by themselves account for the change of behavior of students as a direct result of a change of their teacher’s behavior. This change could have been brought about by the parents who, after consulting the teacher, have advised their children to change their behavior. Of course, someone could argue that the piecemeal technology (in this example) is applicable because the parents actually brought about the change in their children’s behavior only after the teacher had initiated the process in the first place.

At the same time, there is also the issue of the “extraneous factors”, that is, anything in an experiment other than the independent and dependent variables ¹⁴. Since the variables can present challenges and introduce errors, it is necessary in experiments to control these extraneous factors. The control of extraneous factors is a way to demonstrate the validity of theoretical inferences and this is usually achieved by carrying out an experiment. In several studies involving multiple causes, the validity of applying the experimental method to social situations is very questionable. This is so because inferences drawn from what people do in an experimental situation may not to be applied to what they would do in ordinary circumstances.

An example of objective research could have been research conducted on basic training courses for cadets recently graduated from military academies. The researcher would begin by forming a hypothesis; in this case, if the training courses fulfill the goals set by the army for a new officer. In Greece’s Army Infantry School, noncommissioned officers recently promoted to warrant officers take a preliminary test whose results will eventually become the indicator of the knowledge gained during their service in the army for the subsequent preparation of the school’s trainers. This test generates both qualitative and quantitative data. It is qualitative in the sense that the school has an indication of each trainee’s background knowledge. It is quantitative in the sense that data is gathered from the entire class, which will give an overall picture of the trainees’ professional level. This will, in turn, set the pace and modify, as appropriate, the goals set by the training school. In other words, these objectives will be modified accordingly at the start of school rather than letting them emerge over the training period. Though the school has a training program whose requirements must be met by all trainees, this positivistic approach helps trainers to be better prepared to meet the diverse needs of warrant officers that come from different branches. It could also be qualitative if an attempt was made to measure the value added, i.e. how much each student progressed in their learning due to the teacher or different teaching methods.

Unlike quantitative researchers who place great emphasis on generalizing their research findings for diverse populations and times, qualitative researchers have been reluctant to do so. This reluctance is partly exemplified by researchers (e.g. ¹⁵) who do not favor generalizations, on the assumption that every social interaction is unique and merits its own interpretative exploration. Though qualitative researchers do not reject generalizability in principle, the extent to which it could be generalizable is highly questionable ⁶. Though qualitative researchers might take into account all possible factors and adopt an open-ended approach at the beginning of their research, they will unavoidably have to rely on their personal judgment in due course. Even in a relatively unstructured interview, the researcher will provide a certain framework and try to channel the conversation. Moreover, there is the risk of evaluating the people being studied, which might lead the researcher to answer prematurely the whys and what instead of discovering things as they reveal themselves. The very fact that researchers with both a qualitative and quantitative orientation are influenced by their own attitudes, perspectives and cultural background unavoidably subjectively informs the research findings.

Yet, qualitative research could become a reference point for generalizations. For example, Denscombe ³ argues that “health-related critical incidents in the lives of young people could serve to change attitudes towards taking risks with health”. He goes on to explain that the evidence which this conclusion rests upon is exclusively experts’ opinion. Thus, such incidents are treated as objectively defined objects in the lives of young people. On the other hand, generalizations cannot come from such critical incidents which are actually produced by the way the researcher interprets them, setting aside the voices of the young people themselves ¹⁶.

There is doubt as to the whether such researchers can investigate “their own perspectives and the perspectives of the people they interact with daily” ⁶. Practitioner researchers should work hard to overcome their own assumptions, for the very fact that they claim insider knowledge and thus familiarity, or “bias”, with the setting which might blur their own reliability. At the same time, there is a danger of misunderstanding their own orientation. An in-depth understanding of the practitioner researchers’ practice could work the other way round and prove that their assumptions are false.

Another respect in which qualitative research may produce subjective impressions is the role of the interview in “penetrating fronts” ⁶. Briefly speaking, penetrating fronts are concerned with people’s sayings, actions, beliefs and claims in a particular situation. In this context, there are four main drawbacks in choosing interviewees as a data gathering source: they may not tell the truth, it might be difficult for them to express what is actually going on at a subconscious level, it would be hard to link their behavior with remote causes, and interviews can only serve particular discursive functions. Some researchers (e.g. ^{17, 18}) have overcome the problem of controlling the one-to-one interview by setting up group discussions, usually in small groups. However, this could cause other ethics-related problems; for example, one member of the group may not practice confidentiality or may show lack of respect to the other participants.

A qualitative approach would offer more scope to understand a participant’s perceptions of the topic under investigation. By combining qualitative and interventional methods, researchers would seek to form what Shortall ¹⁹ calls “a comprehensive tool which could provide insights into the specific and the general patterns of behaviour and translation skills”. In order to gain such insights, a teacher could design a practical qualitative tool to examine the processes that take place in an educational situation so as to gain (often subjective) insights into performance in the classroom.

Finally, research can be approached from the perspective of more than one paradigm. Each paradigm would provide researchers with the set of beliefs and a view of the nature of the world that define their place in the research ². For example, while post-positivist research accepts that absolutes are difficult to establish, it could still strive for a degree of objectivity. This type of research would combine quantitative and qualitative methods of data collection and analysis. The interpretive researcher would interpret this phenomenon in different ways, mainly because the participants and the situations differ. At the same time, from a postmodernist point of view, the researcher would seek to “break down conventional boundaries and draw attention to how permeable and movable [they] can be” (ibid.: 55). This approach would make possible the study of an unmapped research area which is expected to end with ideas for further research, because the conclusions could not have been regarded as definite.

4. Assessing the Argument that Qualitative Research Can Offer no more than Subjective Impressions

As stated in the introduction, the issue of qualitative research and its subjective impressions will be discussed with close reference to structured observation, audio recordings and field notes.

Briefly speaking, structured observation is the systematic recording of data about a set of events or interactions using a predetermined format. This source of data collection comes from the positivist tradition and involves counting, ranking and measurement of frequency and degree ¹⁹. While structured observation is expected to be full of frequency and counting data, audio recording and field notes provide detailed descriptions and qualitative judgments. The statistical control of structured observation allows for stronger generalizations, but the researcher’s access to the research area is not always secured.

The quantitative aspect of structured observation provides a complete list of expected behaviors and requires only that the observer check which ones occurred. This method sometimes involves several observers collecting data located in different and controlled environments. A common form of structured observation is “point sampling” ⁴. This involves counting how many times a particular behavior occurred at particular points over a period of time. In the field of education, structured observation is often used to observe lessons, though not all of the structure can be determined in advance. Therefore, some structure must be imposed on the data after they have been collected. For example, during an empirical study they conducted, Wilson and Streatfield ²¹ devised five pre-determined categories for structured observation (time, source/receiver, channel of communication, medium of communication and location of event) and derived three categories from the data after the observation period (activity engaged in while communicating, response to communication and purpose of communication). This modification suggests the need for a mixing of methods of data collection, where researchers may have to resort to qualitative analysis of the parameters used in quantitative methods of data collection ²⁰.

Structured observation can produce more reliable data because the results can be replicated either by the same researcher at a different time, or by other researchers. It allows data to be collected at the time they occur and does not have to rely on the recall of participants or their interpretation of events. Also, it collects data that participants themselves may not realize are important. It allows for greater control of sampling error associated with the selection of observation sites and times, and of measurement error associated with the methods of recording observations. This control, in turn, permits stronger generalizations as well as checks on reliability and validity. If an adequate number of observations are made, structured observation can also offer a statistical sample.

The issue of participants’ reactivity is another crucial point when considering structured observation as a research method. Social scientists often call this reactivity “demand characteristics” or “negative demand characteristics” ²². That is, the participants act in a way they think the researcher wants or they deliberately try to subvert a study, respectively. In this research mode, structured observation on one hand and audio recordings and field notes on the other could help researchers to enhance the validity and reliability of their research findings ¹⁰.

This mixed-methods use is often compromised by research ethics. For instance, it is not always possible to get permission for the audio recording of an event and thus observation may be limited to certain settings or sub-settings. Where permission is obtained, it is likely that reactivity will increase. The fact that subjects may become accustomed to the presence of recording equipment, and that modern equipment is more compact and unobtrusive than in the past, does not eliminate the likelihood of reactivity, which is a serious drawback. Normally, if permission cannot be obtained, it is either sought after the observation or efforts are made to disguise the identity of those observed.

The researcher may also be swamped by a large amount of data that exceeds the data necessary for the research at hand or the quantity of data that it is possible to analyze. When audio recording is possible, there is a great temptation to avoid selection decisions and to try to record almost everything ¹⁹. For the purposes of data analysis, it is usually necessary to have a written record of verbal interaction. Audio and tapes therefore have to be transcribed, a process which is extremely time-consuming and sometimes difficult if the recording quality is poor. Audio recording obviously misses non-verbal communication, and it cannot provide an adequate record of the wider social context in which behavior occurs. Pertinent observations, especially the quality of the recordings, are often compounded without any visual references to contextualize them. This limitation may dictate the analysis of certain data later in the research process. For such reasons, field notes may not be used because they have to be written in real time, which makes it too difficult to produce a detailed account of the event.

Structured observation requires observers’ continuous attention if they are to record data reliably, and this is much more difficult than using an audio recording device to record the entire event, for example a lesson. Though this drawback could be mitigated by training a number of observers, this solution is costly and time-consuming. In structured observation, the researcher must be present where the events are occurring and at the appropriate time, but only overt actions can be observed. Additionally, the presence of the observer can affect the subjects’ behavior and thereby bias the data. Although a structured format may limit the type of information collected, the researcher could not refrain from making subtle inferences.

Although actual observations of behavior may be accurate, participants’ reactivity, both personal and procedural, is possible because they do not always behave in the way they normally behave ²³. Personal reactivity occurs when subjects behave differently because of personal characteristics or the observer’s behavior. Structured observation, similarly to audio recording, could be carried out in a covert manner and this can only really be done if the observer does not participate in the group’s activities. The categories used may overlap, be open to dispute, may not cover all the possible behavioral forms, etc. Categorizing observed behavior in this way still depends upon researchers’ interpretations, because different researchers may categorize the same event differently. In any case, the covert use of a research method raises several methodological and ethical issues and should be given careful consideration.

Audio recording can provide a more complete and accurate record of behavior and can be used to supplement or check data records produced by the researcher through systematic observation. It may therefore be useful in assessing the validity of data recorded live. Using audio recording, it is also possible to conduct a more careful and sometimes more complex analysis of data, since we can stop and replay the recording in order to consider the coding or categorization of the data. Audio recording may actually be essential in some studies where information is needed on the details of interaction and/or on the specific language that is used, as in conversation analysis.

Although there are clear differences between structured and less structured observation, they are not opposed or mutually exclusive. The observational researcher often employs both styles, which at times raise similar issues and problems. Structured observation can be an excellent method of collecting data. More types of data categories can be observed, in less time, compared to unstructured observation, and the data analysis is simpler. Conclusive evidence based on a list of categories could be drawn merely by counting how many times a particular type of category occurred. The results of unstructured observation, on the other hand, consist of quantities of descriptive data, since the observer is trying to record everything that has happened. These data must be sorted out to see if there are any patterns to the observed category, which is a very time consuming process. Moreover, training and ensuring the availability of a large number of observers increase the cost of this method, which may threaten its validity and reliability. This type of mixing of qualitative and quantitative methods has been demonstrated by several researchers (e.g. ⁵).

The reliance on “expert opinions” and the “marginalization of the subject” in research has not passed unnoticed by qualitative researchers ⁶. Oriented by the investigative experience, they try to give voice to the subjects’ perspective. In this sense, the researchers provide their own interpretation, but through the subjects’ voice. At the same time, these researchers are aware that they have to deal with their own personal assumptions and prejudices which, at least partly, could help them remain open-minded and mitigate possible biases. Yet, it is difficult to be truly self-critical and this process can become paralyzing.

5. Conclusion

This study has argued that there isn’t any hard and fast rule for choosing or rejecting one research method over another. The claim that qualitative research can offer no more than subjective impressions is a rhetorical device rather than a methodological position. Most researchers, even those who are not qualitative research oriented, would acknowledge its contribution to educational research ⁵.

We have seen that quantitative researchers, influenced by the theory of positivism, strive to produce scientific data based on objective evaluations, while avoiding any personal involvement in their evaluation. Research should be based on objective data derived mainly from empirical evidence. The investigation of causal relationships is the safest way to lead to objective data. The main criticism against qualitative research is the researcher’s personal involvement in the research process. It is also claimed that researchers should remain as detached from the objects and settings they research as possible.

Another main criticism against qualitative research is directed at its inability to generalize its research findings. Though this is true, at lease to an extent, recently qualitative researchers have started to brush aside the merits of generalization. The researchers’ involvement in the research process is not in itself a disadvantage. There is of course the danger of biases, but many times it is necessary for the researcher to become involved in the research process if an in-depth understanding is required to produce research data. The participant is given voice even if this voice is filtered through the researcher. The increased interest in the generalizability of qualitative research could be seen as an attempt to build bridges with other research methods. For this reason, more and more researchers (e.g. ^{5, 24}) are in favor of methodologies where qualitative and quantitative research methods are mixed to enable them to increase the validity of their results.

This increasing interest in generalizability in the qualitative tradition indicates the researcher’s concern in making their research valid and as widely accepted as possible ²⁵. If the findings of qualitative research on one situation studied could fit in other situations as well, then the researcher’s impressions acquire an air of objectivity. Though this is not the primary concern of the qualitative researcher, it could be an overall contribution to other researchers who might be interested in investigating similar situations.

References