1. Introduction

The most general characterization of competence-orientated education is that it is an education based on competencies. Competencies as according to Weinert are defined as “the cognitive abilities and skills possessed by or able to be learned by individuals that enable them to solve particular problems as well as the motivational, volitional and social readiness and capacity to use the solutions successfully and responsibly in variable situations” [^[1], p. 27, [translation by K. B.]]. Many countries all over the world have competence-oriented curricula or are on the way to introduce them. Examples are Canada ^[2], Germany ^[3], Switzerland ^[4], or Vietnam ^[5].

Textbooks are defined as didactically prepared workbooks for students to work individually and to develop a deeper understanding. In addition, a textbook supports teachers for instruction [^[6], p. 486]. Wiater, a textbook researcher, defines textbooks as plain printed media [^[7], p. 43]. In contrast, there are also definitions of textbooks incorporating learning programs [^[8], p. 8]. In summary, there is a certain disagreement whether a textbook is only in print or whether electronic didactically prepared media are also considered as textbooks. In accordance with this ambiguity, Depaepe and Gorp [^[9], p. 10] question if there exist consistent textbook features and consequently if there exists one uniform definition for textbooks.

Moreover, the demarcation of the term textbook to other terms such as teaching materials, teaching aids, educational materials, and tools for teaching is vague ^[10].

Consequently, in this article only the term textbook is used and is defined as follows: Textbooks are teaching materials incorporating student’s materials, teacher’s guides, additional teaching materials including films and experimental material.

Textbook research is classified in three sections: the process related, the product related and the performance related research ^[11]. In the process related research, the production process, the approval and implementation of textbooks as well as the function and use for tuition are investigated. The product related research analyzes the quality of textbooks. Finally, the performance related research studies the impact of textbooks on teachers and students [^[11], p. 5].

Checklists to analyze and evaluate textbooks belong to the product related research. Checklists to conceptualize textbooks belong to the process related research.

The first checklists originated from the U.S. In the meantime, there are over hundred checklists such as the checklist by Astleitner, Sams and Thonhauser ^[12], Bamberger ^[13], Funk ^[14], Jander ^[15], Jazbek ^[16], Kast and Neuner ^[17], Kesidou and Roseman ^[18], Knütter ^[19], Miekley ^[20], Rauch and Tomaschewski ^[21], Sams and Thonhauser ^[22], Sitte and Wohlschlägl ^[23], Uhe ^[24], Weinbrenner ^[25] and Winkler ^[26]. In addition, the first checklists for digital media have been published such as the one from Graeber ^[27]. Most checklists are reduced to a few aspects, some contain up to 480 items ^[28].

Only a few checklists have been published for science textbooks. Examples include the checklist by Kahveci ^[29], Kesidou and Roseman ^[30], Lee ^[31], Metzger and Stuber ^[32], as well as Rauch and Tomaschewski ^[33].

Another desideratum is that most checklists are not research based. The aspects of the checklist are either a combination of aspects from existing checklists or normatively determined. The best known research based checklist in science is the Reutlinger checklist ^[33]. However, originating from the 1990’s, it is no longer up to date.

In addition, most checklists only focused on one aspect of a textbook. The American checklist of Lee ^[31] focuses on nature of science. Kesidou und Roseman ^[30] composed criteria for “good” instruction in the sense of the project AAAS 2061. Kahveci ^[29] set up a checklist for Turkish chemistry textbooks focusing on the students’ perspective.

An empirically generated checklist covering all aspects of a scientific textbook and considering current teaching methods, didactics and content is lacking at present. The absence of such basic research is serious as checklists are vital tools to analyze and write new textbooks ^[29]. As competence-oriented curricula are one of the most vastly distributed type of curricula at present, this research project aimed to empirically set up a checklist for competence-oriented textbooks. The research question is: Which standards should science textbooks meet in order to support competence-oriented teaching and learning in primary and secondary schools?

2. Material and Methods

To generate the empirical standards of the competence-oriented checklist, a sequential mixed-method approach was used [^[34], p. 254], comprised of a qualitative step followed by a quantitative one, to be described in the following sections.

Sixty nine persons were asked per email or telephone to partake in the qualitative survey. In the anonymous qualitative online survey, forty experts took part (response rate: 58%). Among the experts were experienced science teachers (N=17), employees of a publishing house (N=2), textbook authors (N=4), science educationalists (N=11) and educational scientists (N=6). The experts originate from Germany, Austria and Switzerland.

Participants in the anonymous quantitative online survey included 843 secondary school teachers from grade seven to nine, and 339 primary school teachers from grade four to six originating from 14 German speaking districts in Switzerland. In addition, all 56 German speaking science educationalists in Switzerland were asked to participate in the anonymous quantitative online survey.

All persons having participated in the qualitative survey originating from Switzerland were asked to partake in the qualitative study, too.

In total, 171 secondary school teachers (response rate: 20%), 178 primary school teachers (response rate: 53%) and 44 science educationalists (response rate: 79%) took part. Most participants (29%) were between 30 and 40 years old, 26% were younger than 30, 19% were between 41 and 50 years old, and 25% were older than 51 years old. There were 56% women and 44 % men.

A qualitative survey with open-ended questions was carried out. The nine questions asked about aspects to be considered in a competence-oriented textbook. By summarizing the answers of the survey by using qualitative content analysis ^[35], preliminary standards for competence-oriented textbooks were developed. To ensure reliability, the criteria suggested by Steinke [^[36], p. 324ff.] were considered in the qualitative survey including intersubjective reproducibility, indication of the research process, empirical anchoring, and limitation.

The standards were revised three times. First, the preliminary standards were compared with existing checklists for textbooks. Second, the content of the preliminary standards was matched with the results of current empirical studies. Third, the preliminary standards were complemented with the normative guidelines of the Swiss government concerning competence-oriented tuition in science.

The validated preliminary standards served to construct a quantitative survey. In this survey, participants weighted the relevance of the preliminary standards on a five-point Likert scale (1: I do not agree; 5: I fully agree).

The scores of the Likert-scale were analyzed with a one-sample t-test in IBM Corp. SPSS Version 21. The t-tests determined the relevance of the preliminary standards [^[37], p. 67]. If both groups of teachers and/or the group of science educationalists weighted a preliminary standard significantly higher than the test score 4, it could be assumed that the relevance was higher than simply by random chance. Consequently, such a preliminary standard became a determined standard. If only primary school teachers weighted a preliminary standard significantly higher than the test score 4, it became a determined standard for primary school only and an underdetermined standard for secondary school. The same procedure was applied for standards only scoring significantly higher than 4 by secondary school teachers. If a preliminary standard scored higher than 4 without any significance, this preliminary standard became an underdetermined standard, too.

The relevance of an underdetermined standard was analyzed by an additional validation step. In this qualitative validation, the content of an underdetermined standard was compared to results of approximately one hundred empirical studies in the field of didactics or pedagogy with the help of a standardized point system [^[37], p. 68ff.]. According to the score reached, an underdetermined standard became a determined or a rejected standard.

If both teacher groups as well as the science educationalists weighted a preliminary standard lower than the test score 4, it became directly a rejected standard.

For the checklist for Competence-Oriented TEXtbooks (COTEX), only determined standards were considered and are called weighted standards.

Besides the quantitative approach to gain weighted standards (see chapter 2.4.), there were some preliminary standards not undergoing the quantitative weighting process. The reason was that during the validation process (see chapter 2.3.) the relevance of these preliminary standards has been shown obvious. Such standards were for instance: “The content of the textbook is correct (text, images, graphs)”.

The significance of another group of preliminary standards showed to be dependent on the school or class. Therefore, these standards were also not weighted in the quantitative survey. An example of such a standard is: “The media used in the textbook match the infrastructure of the school”.

In the checklist, these standards are called standards individually to be weighted and can be weighted by the user of the checklist itself.

3. Results

The checklist for Competence-Oriented TEXtbooks (COTEX) contains empirically weighted standards and standards individually to be weighted (see chapter 2.).

The weighted standards are grouped in the following categories: subject matter, skills, tasks, experiments, class heterogeneity, student materials, and teacher materials.

The standards individually to be weighted are divided in two categories: fit to own institution, and everyday applicability.

Table 1. Part of the empirically weighted section of COTEX

Download as

PowerPoint Slide

Larger image(png format)

• Tables index

Veiw figure

View current table in a new window

View next table

The checklist COTEX for primary school contains 77 weighted standards and 25 standards individually to be weighted. COTEX for secondary school incorporates 74 weighted standards and 25 standards individually to be weighted.

COTEX can be used to conceptualize or assess competence-oriented textbooks. When assessing textbooks, it can be tested how far a science textbook represents the standards using a five-point Likert scale (Tab.1).

In the section with the standards individually to be weighted, the user determines the relevance of the standards on a five-point Likert scale from 1=not relevant; 5=highly relevant (Tab. 2). Apart from that the user rates the textbook similarly to the section with the empirically weighted standards.

Using the checklist to assess textbooks, a quotient is calculated by dividing the rating of the textbook by the weighting. The quotient indicates the fit of the textbook with the standard for competence-oriented textbooks.

Quotients above 1 indicate the standard to be more prominent than necessary. A quotient of 1 points out that the examined textbook has a perfect fit with the expectations of checklist COTEX. A quotient below 1 indicates that the textbook is below the expectations made by the checklist.

Table 2. Part of the individually to weight section of COTEX

Download as

PowerPoint Slide

Larger image(png format)

• Tables index

Veiw figure

View current table in a new window

View previous table

4. Discussion

Comparing the empirical approach to setting up the checklist for Competence-Oriented TEXtbooks (COTEX) with other checklists ^{[28, 38, 39]}, one noticeable feature of COTEX is the three validation steps. In the first validation step, the (empirical) studies ensured the theoretical relevance of the preliminary standards as the Bielefelder checklist did as well ^[28]. The second validation with about twenty existing checklists helped to formulate the preliminary standards more concisely and to summarize the standards. Finally, the third validation step ensured competence-orientation to be fully enclosed ^[4].

As done in other checklists ^[38], the inclusion of teachers could secure the often criticized lack of applicability of checklists ^[40]. Analogous to the checklist of Metzger and Stuber ^[32], there were both teachers and science educationalists involved. Inherent for COTEX is the representation of both teachers and science educationalists in the qualitative and quantitative survey.

Maybe the most characteristic feature of the checklist COTEX is its division in two sectors with either empirically weighted standards or standards individually to be weighted. The significance of the weighted standards was determined quantitatively by over 350 persons. If the relevance of a standard was uncertain, the so called underdetermined standard was additionally weighted qualitatively. The benefit of this procedure is twofold. First, the weighting allowed to reduce the number of standards empirically. Second, the checklist itself consists of weighted standards and consequently taking into account that the standards differ in their relevance. The standards individually to be weighted allow the users to carry out the weighting themselves in the categories: fit to own institution, and everyday applicability. This mix of both sections has several advantages. The weighted section ensures the objectivity of the rating process and enables the user to profit from the knowledge of over 350 persons. The section with standards individually to be weighted ensures the fit of a textbook to the individual situations in school by allowing the users to adjust the checklist to their particular situation in school and in class. No previously published checklist could be located that contain two such sections (e.g. ^{[29, 30, 31, 32]}). In contrast, most checklists do not include any weighting (e.g. ^{[28, 41, 42]}). Some checklists allow the users to weight the standards themselves (e.g. ^{[14, 39]}). An example of a checklist with weighted standards is the Reutlinger checklist ^[21]. In this checklist, the research team itself performed the weighting. Therefore, they could be blamed for a lack in objectivity. In addition, all standards were weighted and consequently not differentiating between the various qualities of standards. Last but not least, they left the user to decide whether to use the weighting or whether to weight for themselves ^[33]. An advantage of this option may be the possibility to adjust the checklist to the users own needs. A risk may be that the freedom of choice ends up in a complete acceptance or a complete ignorance of the weighting.

Finally, the evaluation of the textbook rating is another specialty of COTEX. By forming the quotient between the textbook rating divided by the weighting of the standards, the interpretation of the result is quite simple (see 3. Results). For instance, a quotient of 1 signifies a perfect fit between the weighting and the expression in the textbook. Other checklists multiplied the textbook rating by the weighting of the textbook (e.g. ^{[21, 39]}). Even though the calculation may be simpler, the drawback is the lack of a fast interpretation of the result as it is not visible in the product whether the textbook has expressed a standard high or whether the rating of the standard is high. More explicitly, given an exceptionally high expression of a standard with low relevance, the multiplication may be 5*1=5. Given an exceptionally low expression of a standard of a highly relevant standard, the multiplication may be 1*5=5. Even though the textbook in the first example is good and in the second bad, the result does not show this. With the quotient: 5:1=5 and 1:5=0.2 the difference in quality is directly visible.

5. Conclusions

In summary, the Competence-Oriented checklist for TEXtbooks in science (COTEX) is an empirically generated checklist being used to analyze and assess the quality of textbooks. Consequently, COTEX may enhance the process of authorizing science textbooks. In addition, COTEX serves as a guide for authors to set up competence-oriented textbooks in science. Consequently, COTEX may improve the quality of competence-oriented textbooks.

Even though the checklist COTEX is designed for science textbooks, the majority of standards are universal for all subjects. Future research is planned to adjust COTEX to other subjects. The adaptation is expected to be exceptionally easy for subjects such as earth sciences or history having similar structures in tuition. More extensive adjustments are anticipated to for languages.

In the future, COTEX could be used as a research tool to comparing the quality of textbooks within a country and for a cross-country comparison.

The checklist for Competence-Oriented TEXtbooks (COTEX) is available in English and German under: www.schulbuchforschung.ch.

Acknowledgement

I would like to thank the Ministry of Research and Development from the University of Teacher Education Lucerne as well as the University of Teacher Education Heidelberg for their funding. In addition, I would like to thank Prof. Dr. Markus Rehm and Prof. Dr. Markus Wilhelm for their scientific support and substantial help in improving the study.

References

[1]	Weinert, F. (2001). Leistungsmessung in Schulen. Beltz, Weinheim; Basel.
	In article		PubMed
[2]	Province-of-British-Columbia (2015). Science - Core Competencies. Ministry of Education, Province of British Columbia, Canada.
	In article
[3]	KMK (2004). Bildungsstandards in Biologie/Chemie/Physik für den Mittleren Schulabschluss. Kultusministerkonferenz, Bonn
	In article
[4]	EDK (2011). Grundkompetenzen für die Naturwissenschaften: Nationale Bildungsstandards | Frei gegeben von der EDK-Plenarversammlung am 16. Juni 2011.
	In article
[5]	World-Bank (2014). Vietnam - Renovation of General Education Project: safeguards diagnostic review: Social assessment (English).
	In article
[6]	Zwahr, A. (2006). Brockhaus: Enzyklopädie in 30 Bänden. Brockhaus, Leipzig.
	In article		PubMed
[7]	Wiater, W. (2005). Lehrplan und Schulbuch. In: Matthes, E., Heinze, C. (Hrsg.) Das Schulbuch zwischen Lehrplan und Unterrichtspraxis. Klinkhardt, Bad Heilbrunn, 41-63.
	In article
[8]	Grill, G., Digel, W. (1992). Meyers grosses Taschenlexikon: In 24 Bänden. B.I.-Taschenbuchverlag, Mannheim.
	In article		PubMed
[9]	Depaepe, M., Gorp, A. v. (2009). Einleitung: Auf der Suche nach der wahren Art von Textbüchern. In: van Gorp, A., Depaepe, M. (Hrsg.) Auf der Suche nach der wahren Art von Textbüchern. Klinkhardt, Bad Heilbrunn, 9-16.
	In article
[10]	Bölsterli Bardy, K. (2015). Kompetenzorientierung in Schulbüchern für die Naturwissenschaften: Eine Analyse am Beispiel der Schweiz. Springer Spektrum, Wiesbaden.
	In article		View Article
[11]	Mayer, D. P., Mullens, J. E., Moore, M. T. (2000). Monitoring school quality - An Indicators Report. National Center for Education Statistics, U.S. Department of Education., Washington DC.
	In article
[12]	Astleitner, H., Sams, J., Thonhauser, J. (1998). Womit werden wir in Zukunft lernen? Schulbuch und CD-ROM als Unterrichtsmedien. Ein kritischer Vergleich. öbvhpt, Wien.
	In article
[13]	Bamberger, R. (1995). Methoden und Ergebnisse der internationalen Schulbuchforschung im Überblick. In: Olechowski, R. (Hrsg.) Schulbuchforschung. Lang, Frankfurt am Main, 46–94
	In article
[14]	Funk, H. (2004). Qualitätsmerrkmale von Lehrwerken prüfen - ein Verfahrensvorschlag. Babylonia 3/, 4--47.
	In article
[15]	Jander, L. (1982). Schulbücher im Geographieunterricht. In: Jander, L., Schramke, W., Wenzel, H. J. (Hrsg.) Metzler Handbuch für den Geographieunterricht. Metzler, Stuttgart, 359–361.
	In article
[16]	Jazbek, S. (2007). Exemplarische Analyse zur Einsetzbarkeit französischer Biologiebücher zum Thema „Die Atmung des Menschen“ im bilingualen Biologieunterricht in Deutschland. Didaktik Biologie 16/, 13-31.
	In article
[17]	Kast, B., Neuner, G. (Hrsg.) (1994). Zur Analyse, Begutachtung und Entwicklung von Lehrwerken für den fremdsprachlichen Deutschunterricht. Langenscheidt, Berlin.
	In article		PubMed
[18]	Kesidou, S., Roseman, J. E. (2002). How well do middle school science programs measure up? Findings from Project 2061's curriculum review. Journal of Research in Science Teaching 39/6, 522-549.
	In article		View Article
[19]	Knütter, H. H. (1979). Schulbuchanalyse - Intention und Kriterien eines fachspezifischen Fragenkatalogs. In: Stein, G. (Hrsg.) Schulbuch-Schelte als Politikum und Herausforderung wissenschaftlicher Schulbucharbeit. Klett-Cotta, Stuttgart, 165-172.
	In article
[20]	Miekley, J. (2005). ESL Textbook Evaluation Checklist. The Reading Matrix 5/2, 9-17.
	In article
[21]	Rauch, M., Tomaschewski, L. (1986). Reutlinger Raster zur Analyse und Bewertung von Schulbüchern und Begleitmedien. Pädagogische Hochschule Reutlingen, Reutlingen.
	In article
[22]	Sams, J., Thonhauser, J. (1998). Schulbuchforschung - Ein Beitrag zur Schulentwicklung? Salzburger Beiträge zur Erziehungswissenschaft 1/2, 5-22.
	In article
[23]	Sitte, W., Wohlschlägl, H. (2001). Das GW-Buch. In: Vielhaber, C., Wohlschlägl, H. (Hrsg.) Beiträge zur Didaktik des Geographie und Wirtschaftskunde“-Unterrichts. Universität Wien, Wien, 447-472.
	In article
[24]	Uhe, E. (1979). Schulbuchanalyse mit Hilfe eines allgemeinen Beurteilungsrasters – Intention und Gesichtspunkte. In: Stein, G. (Hrsg.) Schulbuch-Schelte als Politikum und Herausforderung wissenschaftlicher Schulbucharbeit. Klett-Cotta, Stuttgart, 158–164.
	In article
[25]	Weinbrenner, P. (1986). Kategorien und Methoden für die Analyse wirtschafts- und sozialwissenschaftlicher Lehr- und Arbeitsmittel. Internationale Schulbuchforschung 8/, 321-337.
	In article
[26]	Winkler, W. (1998). A Suggested Tool for Earth Science Textbook Selection. Mitt Österr Miner Ges 143/, 11-16.
	In article
[27]	Graeber, W. (Hrsg.) (1990). Das Instrument MEDA. Ein Verfahren zur Beschreibung, Analyse und Bewertung von Lernprogrammen. Institut für die Pädagogik der Naturwissenschaften, Kiel.
	In article
[28]	Laubig, M., Peters, H., Weinbrenner, P. (1986). Methodenprobleme der Schulbuchanalyse. Abschlußbericht zum Forschungsprojekt 3017 an der Fakultät für Soziologie der Universität Bielefeld in Zusammenarbeit mit der Fakultät für Wirtschaftswissenschaften. Universität Bielefeld, Bielefeld
	In article
[29]	Kahveci, A. (2010). Quantitative Analysis of Science and Chemistry Textbooks for Indicators of Reform: A complementary perspective. International Journal of Science Education 32/11, 1495-1519.
	In article		View Article
[30]	Kesidou, S., Roseman, J. E. (2002). How well do middle school science programs measure up? Findings from Project 2061's curriculum review. JRST 39/6, 522-549.
	In article		View Article
[31]	Lee, Y. (2007). Procedures for conducting content analysis of science textbooks: Original by E.L. Chiappetta, D.A. Fillman, and G.H. Sethna, 2004, modified by M. Philips (2006), italicized modifications by Y. Lee (2007). University of Houston.
	In article
[32]	Metzger, S., Stuber, T. (2011). Folgerungen für Lehr - und Lernmittel: aus den Leitlinien für den Unterricht in Naturwissenschaften und Technik auf der Volksschulstufe. In:Bildungsdirektion Kanton Zürich, Zürich.
	In article
[33]	Rauch, M., Tomascheski, L. (1995). REUTLINGER RASTER: Short version: a checklist for evaluating teaching materials. College of Education, Freiburg
	In article
[34]	Foscht, T., Angerer, T., Swoboda, B. (2007). Mixed Methods: Systematisierung von Untersuchungsdesigns. In: Buber, R., Holzmüller, H. H. (Hrsg.) Qualitative Marktforschung. Gabler, Wiesbaden, 247-259.
	In article		View Article
[35]	Mayring, P. (2010). Qualitative Inhaltsanalyse, Grundlagen und Techniken. Beltz, Weinheim.
	In article
[36]	Steinke, I. (2000). Gütekriterien qualitativer Forschung. In: Flick, U. (Hrsg.) Qualitative Forschung: ein Handbuch. Rowohlt, Reinbek bei Hamburg, 319-331.
	In article
[37]	Bölsterli, K. (2014). Kompetenzorientierung in Schulbüchern für die Naturwissenschaften: aufgezeigt am Beispiel der Schweiz. Pädagogische Hochschule Heidelberg.
	In article
[38]	Metzger, S. (unveröffentlicht). Bericht mit Empfehlungen betreffend Lehrmittel und Unterrichtsmaterialien für die Volksschule (Kindergarten bis Ende Sekundarstufe I) im Bereich Naturwissenschaften und Technik. Pädagogische Hochschule Zürich.
	In article
[39]	Wirthensohn, M. (2012). LEVANTO - Ein Tool zur praxisorientierten Schulbuchevaluation. In: Doll, J., Frank, K., Fickermann, D., Schwippert, K. (Hrsg.) Schulbücher im Fokus. Waxmann, Münster, 199-213.
	In article
[40]	Fritzsche, P. K. (Hrsg.) (1992). Schulbücher auf dem Prüfstand. Perspektiven der Schulbuchforschung und Schulbuchbeurteilung in Europa: Unter Mitarbeit von Prof. Dr. Ursula A. J. Becher. Diesterweg, Frankfurt am Main.
	In article
[41]	Chiappetta, E. L., Fillman, D. A. (2007). Analysis of Five High School Biology Textbooks Used in the United States for Inclusion of the Nature of Science. International Journal of Science Education 29/15, 1847-1868.
	In article		View Article
[42]	Tairab, H. (2006). Representation of Scientific Literacy in United Arab Emirates Science Textbooks. UAEU Funded Research Publications 19/, 233-240.
	In article