Abstract

This article describes how a deliberate commitment to use a research-based framework of teaching effectiveness can be combined with existing hiring practices to yield better outcomes from the teacher screening and selection process. The author proposes that when school leaders apply a well-developed and agreed-upon framework of teaching effectiveness – one that reflects the expectations of what it means to be effective in a local school context - that the result will be a better hiring process than many of those currently employed. In the end, such a process means that those who demonstrate the greatest promise of being effective in the classroom will be more likely given the opportunity to teach.

1. Introduction

“What books have you read recently?”

“What did you think about the artwork hanging on the walls of the lobby area where we had you wait?”

“If you were an animal, what kind of animal would you be and why would you choose that one?”

After twenty years of working as both a school-based and central-office administrator, and now as a university professor of educational leadership who studies teacher recruitment, retention, and professional development, these are just a few examples of actual questions that teacher candidates have reported being asked during a job-interview for a teaching position.

I am continuously amazed to hear these recollections; am slightly surprised as the teacher-candidates divulge their responses to them; and utterly stunned when I consider how the answers to such questions might be cogently used by an interviewer or interview team to make an evidence-based decision about who might be the best candidate for a coveted teaching job. However, based on the feedback I have received over ten years of teaching university classes on personnel leadership in schools it is evident that these questions are asked to desperate job-seekers by real people who occupy leadership positions. Somehow a fair number of those who serve on teacher hiring committees believe that the responses to such obscure questions will yield insight that might be run through a sifting and sorting process that results in a decision of who might be the most likely to find success in a classroom full of students.

As a tonic for such persistent hiring maladies, in this article I propose that what are needed are better approaches to glean useful insight into a candidate’s knowledge, skills, and dispositions and insight that correlates to measures of capacity to be effective in a classroom setting. Such approaches need to be guided at the outset by a framework for teaching effectiveness and rely on an analysis of the available sources of data against the framework to improve the outcomes of the hiring decision.

2. The Teaching Effectiveness Prediction Problem

In his 2008 “The Most Likely to Succeed” article in The New Yorker, author and social commentator Malcolm Gladwell ¹ described the “quarterback problem.” He outlined how difficult it is to predict the on-the-field success of a potential professional football quarterback from college game analysis, combined with testing results and a general intelligence test.

Professional football scouts are notorious for telling stories of how much time they spend crisscrossing North America to sit in varsity stadiums and watch football games. To an outsider, it might seem to be a new form of the millennia old activity of prospecting for gold. It is reminiscent of the process of searching for that “nugget” or, perhaps, even the “motherload” that might lead to unimaginable wealth through a process called: “The Combine.” The Combine is a series of standardized tests of speed, strength and agility that many prospective professional athletes are measured against. In addition to The Combine, the National Football League (the NFL), as Gladwell explains, also administers the “Wonderlic Personnel Test,” which is a widely used intelligence test to assess the aptitude of prospective employees for learning and problem solving.

After all of the results of these assessments are collected and analyzed by the scouts, coaches and general managers to make predictions of who should succeed at the professional rank, Gladwell explains, there is actually little if any correlation between the testing-regimes’ results and the season-over-season track record of success or lack thereof when measured against the statistical history of quarterbacks’ successes playing in the NFL. After reviewing the failures of the measures to predict who “makes it” in the big league as a quarterback, Gladwell concludes, “There are certain jobs where almost nothing you can learn about candidates before they start predicts how well they’ll do once they’re hired” (p. 8). Such a statement seems apt in the context of hiring new teachers.

The reality of teaching - a job that requires combination of a specialized form of subject-knowledge, an adaptive pedagogical expertise, and well-developed dispositions that demonstrate genuine care towards students and their lives ² – is particular enough to conclude that it may be somewhat more challenging to predict success in it than it is to do so with potential quarterbacks. In fact, a century’s worth of research on the relationship between teacher characteristics and teacher effectiveness, has yield little practical progress correlating assessments of teacher quality with factors that appear to be observable at the time of hire ³.

I am not suggesting that their jobs are easy, but quarterbacks can be substituted between games and even mid-game when matters go awry. If someone is not performing well, in response to the “sacks” and the ensuing “boos” from the paying crowd, the coach can send in a replacement or substitute mid-game. Fans might be irate and even complain about the starter’s lack of production but typically they get over it and move on with their lives; especially if the team begins to win.

It is, however, more difficult to recover from the effects of poorly performing teachers. Erik Hanushek ⁴, an economist at Stanford University, in a series of studies has estimated that a teacher’s failure to be effective in the classroom can have lasting negative impact on student success that can take years to recover. When you consider the rigorous assessment regime quarterback prospects go through, even with the weak correlation that Gladwell ¹ suggests, the tests have to measure live-performance, it is amazing how an interviewer’s understanding of a teacher candidate’s imaginary animal-self might help her or him predict potential teaching effectiveness.

It is worth stating a somewhat unsettling truth at this point: there is no perfect predictor of effectiveness in a classroom. Each class taught within a single day is almost certainly different from the others a teacher teaches in that day. And, no two days are exactly the same. This is what makes teaching both a difficult and also a rewarding profession. These realities are why having an adaptive pedagogical expertise allows teachers to balance effectively between the press of pragmatic efficiency in the classroom and the pull of innovative approaches to teaching.

Far too often the tendency in teacher hiring is to focus on only the short-term needs of the school. When, for example, a grade six teacher who coached the school choir departs the school the typical knee-jerk response is to simply place an advertisement for another grade six teacher with the hope he or she can sing. This is hardly a deeply intentional decision.

In reality, teacher hiring ought to take a longer more strategic view that extends beyond the current need to fill a vacancy. A teacher hired today will not only teach the current cohort of students in the schools, but may well impact an entire generation of students. Hiring is not only potentially a thirty-year financial commitment to pay someone a salary; rather, it is also an expression of confidence that the person hired is not just excellent for the needs of today’s students but is also going to be great for students whose parents may not even be born yet. To state the obvious: the outcome of a teacher hiring decision – be it a good, bad or indifferent teacher – extends well beyond a season or two.

It seems logical to conclude that those responsible for hiring teachers would benefit from a framework of teaching effectiveness to understand and guide their hiring decisions. Such a framework would allow them to better understand what teaching effectiveness means based on research and subsequently contextualized to fit their local needs and aspirations. The framework can serve as a guide to the hiring process so that personnel committees can collect and analyze the most meaningful data to inform the decision making processes they employ to better identify those most likely to succeed in the classroom ^{5, 6}.

3. Committing to a Framework of Teaching Effectiveness

In simple terms, if you do not have a clear picture of what constitutes teaching effectiveness you are not likely to know teaching effectiveness when you see “it.” While many principals claim they know it when they see it, far fewer can explain what “it” means in terms of research evidence that supports their idiosyncratic mental models of effective teaching practice ⁷.

When pressed, some principals resort to the reified discourses of educational leadership, which can sometimes sound like “edu-babble,” and suggest they are looking for a “team player” without being able to explain in concrete terms what it means for teachers to cooperatively act as a “team” of teachers who work collaboratively to support every student’s in-school success. An equal number struggle to clearly articulate an agreed-upon collective goal for the team. And, they equally have trouble articulating how many teachers are required to be involved in order for the group to be named as a “team.”

As Spector ⁸ wrote,

The distinction between a group and a team is an important one. All teams are groups, but not all groups are teams. A group consists of people who work together but can do their jobs without one another. A team is a group of people who cannot do their jobs, at least not effectively, without the other members of their team. (p. 303)

It is hard to imagine that the principals who invoke the team reference are suggesting that they want to hire teachers who cannot actually do their jobs – which is teaching – without their colleagues. Teaching continues to be, as it has been for centuries, a solitary act on the part of the classroom teacher.

Sometimes educational leaders hide behind opaque platitudes by suggesting that it is impossible to describe teaching effectiveness because it comes down to a matter of individualistic “teaching style” even though teaching effectiveness is not an ethereal concept that cannot be described and identified ⁹. Danielson ¹⁰ presents the argument that robust and rigorous teaching effectiveness assessments allow school leaders to have confidence in saying, “Everyone who teaches here is good; and here is how I know why”. In a similar argument, hiring committees should be able to declare that they have chosen the best prospect based on a robust and rigorous assessment of potential teaching effectiveness and be able to explain how the framework was used to guide the decision-making process.

Arguably what appears to be lacking in the many teacher-hiring processes at the outset is a research-based conceptual framework of teaching effectiveness that can be used guide the process from the time when it becomes clear that there is a need to hire someone towards a prized outcome; hiring a talented new teacher. If educational leaders anchored their hiring processes in a framework of teaching effectiveness that focused on attracting candidates who hold the greatest promise to become effective and then continued to be consistent with using the same framework to screen the applications and continue to use it develop the interview questions and then rely on it post-interview to guide the selection of a new teacher then the school community - teachers, parents, and possibly even students – would be afforded the opportunity to get a sense of what really matters in hiring decisions; finding effective teachers.

4. Choosing a Framework of Teaching Effectiveness

Teacher hiring processes are examples of a decision-making hierarchy where the end-goal and outcome is the identification of the best prospect for success in a classroom. Research suggests that analytical approaches to hiring decisions outperform those that are instinctual or based on a gut-feeling ¹¹. In the context of teacher hiring, a conceptual framework of teaching effectiveness creates a frame of reference for understanding and articulating what constitutes effective professional practice and allows for common language to be used when collecting, interpreting, and discussing the data gathered during the hiring process ¹². In addition, a framework can serve as a discursive guide that captures and describes some of the attributes and characteristics that are associated with effective professional practice. When agreed-upon frameworks are used, they tend to help focus conversations about the means through which teachers support student learning and success, and provide a lens through which the data gathered in the selection process might be analyzed ^{13, 14}.

While the concept of teaching effectiveness has been defined differently and inconsistently by scholars, there is still a consensus that teaching effectiveness matters a lot in relation to supporting students’ academic success ^{12, 15, 16, 17}. As an orienting point to explore what teaching effectiveness means, Campbell, Kyriakides, Muijs and Robinson ¹⁸ describe the concept as: “the power to realise socially valued objectives agreed for teachers’ work, especially, but not exclusively, the work concerned with enabling students to learn” (p. 4). As the authors admit, while the definition is imperfect, it reduces, in part, some of the challenges that arise from not having made an explicit one ¹⁸.

Regardless of whether they are labelled as “domains” ^{13, 16, 17}, or “components” ¹², or “areas of influence” ¹⁵, many scholars who study teaching effectiveness believe there are observable qualities that separate the very good from their less talented colleagues. In the hiring context, a framework for teaching effectiveness allows educational leaders to articulate in a public manner a commitment to hire teachers who are on a professional growth trajectory to become ever-more effective in professional practice.

It is worth noting that the framework method for the management and analysis of qualitative data argued for in this article has been used since at least the 1980s ¹⁹. The framework method originated in large-scale social policy research, and authors such as Ritchie and Lewis have argued that it is particularly well-suited to the analysis of cross-sectional descriptive data enabling different aspects of the phenomena under investigation to be captured, and for the data analysis to being more transparent. Based on the strengths of a framework method to collect and analyze meaningful data, it is hard to ignore the value such an approach has with respect to the teacher hiring process.

One of the more widely used frameworks for assessing teaching effectiveness, but not the only one, is the Danielson ¹³ “framework for teaching” model. It is being presented in this article as an illustrative example but without an endorsement that it is somehow superior to any of the other well-developed frameworks.

In Enhancing Professional Practice: A Framework for Teaching, Danielson ¹³ outlines measures relating to teaching effectiveness organized into four domains that are then divided into a total of 22 sub-components. The domains (see Figure 1. Danielson model of teaching effectiveness) are: I) planning and preparation, II) classroom environment, III) instruction, and IV) professional responsibilities.

The components in Domain I, “Planning and Preparation,” outline how a teacher organizes the content of what students are expected to learn - in other words, how the teacher designs instruction. These include demonstrating knowledge of content and pedagogy, demonstrating knowledge of the students, selecting instructional goals, demonstrating knowledge of resources, designing coherent instruction, and assessing student learning. The components in Domain II, “The Classroom Environment,” illustrate the interactions that occur in a classroom that are non-instructional. These consist of creating an environment of respect and rapport among the students and with the teacher, establishing a culture for learning, managing classroom procedures, managing student behaviour, and organizing the physical space. The components in Domain III, “Instruction" are what constitute the core of teaching - the engagement of students in the learning context. These include communicating clearly and accurately, using questioning and discussion techniques, engaging students in learning, providing feedback to students, and demonstrating flexibility and responsiveness. Finally, “Professional Responsibilities,” Domain IV, represents the wide range of a teacher’s responsibilities outside the classroom. These include reflecting on teaching, maintaining accurate records, communicating with families, contributing to the school and district, growing and developing professionally, and showing professionalism. Teachers who demonstrate these competencies are highly valued by their colleagues and administrators, as well as being seen as true professionals ¹³.

5. Imperfect Evidence-Informed Hiring Decisions

When it comes to predicting teaching effectiveness, there is no substitute for a demonstrated history of being effective ²⁰. Not every hiring committee, however, has the luxury of observing a prospective teacher teach in real-time before making a job offer. Even when they are able to observe prospective hires, the truth is that past performance does not absolutely guarantee future classroom success. One thing that the research on teacher effectiveness ^{20, 21} has demonstrated is while no data set will be perfect with respect to predicting the success of a teacher in a dynamic classroom full of children, more reliable measures of evidence ought to lead to better personnel decisions and, therefore, better outcomes for students.

Andrejeviv ²² - in the book Infoglut: How too much information is changing the way we think and know - makes the compelling case that a hallmark of our time is the fact that we are choking on data smog. This smog is a metaphoric reference for the sludge and muck that gets pushed out as “evidence” and “fact” in the unmoderated information age in which we live.

Without a doubt those charged with the responsibility of collecting useful information to screen and select new teachers are not, in fact, immune to the pathogenic effect of this smog. Information overload in the hiring process, specifically to the point where there is so much information that it is no longer helpful in the decision-making process affects principals and superintendents as much as anyone else. The end result of being stuck in a thick fog of data is hiring committees who try to wade through reams of data collected about a prospect that is detached from a clear understanding of what constitutes teaching effectiveness. Decision-makers become distracted by marginal pieces of information that not only draw attention away from the most reliable information but also obscure it from sight. The smog contributes to criticisms levied by scholars such as Levin and Quinn ²³ and Liu and Johnson ²⁴ that those in positions of leadership make hiring decisions with little useful information on which to base those decisions.

6. Collecting and Assessing Evidence Using a Framework of Teaching Effectiveness

A comprehensive review of the literature related to hiring as a selection process completed by Ryan and Ployhart ²⁵ concluded that two fundamental questions that are oftentimes overlooked but should be considered by employers before the hiring process is enacted are:

1) What evidence needs to be collected and assessed? and,

2) How should that evidence be assessed?

Based on the selected framework of teaching effectiveness and as adapted to reflect the local schools’ context, a hiring committee should determine what kind of evidence is required in a candidate’s application package that might reflect a relative level of competency in each of the domains.

While school divisions vary widely with respect to what they collect and use as data or evidence in their hiring decision process, many prospective employers require such things as university or college transcripts, practice teaching reports, and letters of reference or recommendation ^{3, 4, 5}. Even though each one of these sources of data can be criticized for lacking predictive validity and reliability related to in-class success in teaching, many of them are regarded as important sources of evidence of potential effectiveness to teach ²⁶. This apparent contradiction lies at the heart of teacher hiring challenge – it is fundamentally a high stakes prediction problem where the stakes are the in-school success of children and youth.

However, when direct assessments of classroom teaching effectiveness are not available by hiring authorities, the truth is that school administrators must rely on the above artefacts and proxy measures to try to gain some insight into an applicant’s potential in-classroom performance ^{27, 28, 29, 30, 31}.

Recognizing that this process will continue, hiring committees can improve it by using a framework of teaching effectiveness to outline what needs to be in an application package addressed by way of knowledge, skills, and dispositions, and advertised to interested applicants, and the public, that application materials will be assessed against a framework of teaching effectiveness. In this way, applicants are invited to present artefacts that demonstrate how they align with the desired end - an effective teacher in every classroom ³². With the material submitted, the committee should carefully screen how each applicant demonstrates her or his level of competency against the framework.

Hayton and Wagner ³³ propose that competency-based assessments are based on a review of the evidence presented to determine an individual’s current ability level against a given set of professional competency standards. As Lurie ³⁴ stated, “Competencies represent a negotiation of language based on expert opinion, experience, and consensus” (p. 52). They are imperfect and only socially constructed ideas of what it means to be proficient in a defined area.

By assessing the evidence presented in the application materials against an agreed-upon framework for teaching effectiveness, such as in the one illustrated in the Danielson framework ¹³, the material in the application package can be screened as demonstrating a level of professional competency that is: “unsatisfactory,” “basic,” “proficient,” or “distinguished” ³⁵ This four-level rating model aligns with Danielson’s work ^{13, 35} and follows the Higher Education Quality Council of Ontario’s recommendations with respect to the development of competency assessments ³⁶. Ultimately, it allows a hiring committee to make an evidence-based recommendation about who should be invited to an interview.

Prior to holding interviews with candidates, the committee should work to develop interview questions that reflect the domains of the framework of teaching effectiveness. This important first step creates the condition of soliciting useful date. It also allows the committee members to engage in conversations with the interviewees related directly to teaching effectiveness. If the questions are based on the framework, then the candidates’ responses can be measured against the chosen framework of teaching effectiveness. The framework approach serves as a calibration mechanism for the hiring committee to align the selection process from the beginning right through to the end goal of the hiring process; which is, to determine who among the interviewees has demonstrated the ability to be best suited to help children learn exceptionally well.

As McDaniel, Whetzel, Schmidt & Maurer ³⁷ state: “The interview is a selection procedure designed to predict future job performance on the basis of applicants’ oral responses to oral inquiries” (p. 599). As is the case with questions about the efficacy of the materials in an application package to serve as predictors for in-class success, the pre-employment interview process has been similarly criticized as being highly subjective and subject to interviewer bias ^{7, 26, 38}. Perhaps, however, because of its intuitive appeal ³⁷, the interview is relied on heavily to make determinations of who should be offered teaching jobs ²⁴. Thus even though teacher employment interviews are criticized based on the grounds that they are poor predictors of future teaching performance, their use continues to be prevalent ^{5, 6, 39}.

Despite its dubious history in early personnel selection research (for example see Schmitt, 1976; Ulrich & Trumbo, 1965), more recent research has indicated that the employment interview, when structured, is an effective predictor of job performance ^{37, 40}. A variety of strategies for imposing structure have been suggested, including providing interviewers with scripts and standard sets of questions, developing scoring guides for interviewee responses, and using multiple interviews. All of these methods appear to help in improving the usefulness and fairness of employment interviews ^{39, 41}.

Two principal types of structured interviews have emerged in the selection literature and in practice: the situational interview ⁴² and the behavioural description interview ^{43, 44}. Both the situational interview and the behaviourally-descriptive interview feature job-related questions and, in general, use behaviourally-anchored rating scales. The situational interview is anchored on the premise that an individual’s stated work-related intentions are the best predictor of his or her future on-the-job behaviour ⁴⁵, whereas the behaviourally-descriptive interview is based on the notion that the best predictor of future work-related behaviour is past work-related behaviour ^{43, 44}.

Situational interview questions require interviewees to describe what they would do if faced with job-related dilemmas. Alternatively, behaviourally-descriptive questions require interviewees to describe what they actually did in various job-related situations. Traditionally, the situational interview has utilized a scoring guide; however, in recent years some researchers have also used a scoring guide for the behaviourally-descriptive interview. Although there is substantial evidence indicating that both the situational interview and behaviourally-descriptive interview predict job performance ⁴⁶, there is no agreement about which type of interview is a better predictor of job performance. By developing scripted questions - either situational or behaviourally-descriptive - that are based on a robust framework of teaching effectiveness, and then assessing the interviewees’ responses against a scoring rubric that has also been developed to align with the framework of teaching effectiveness, the usefulness and efficacy of the pre-employment interview can be improved ^{39, 41}.

Figure 2, “Interview questions related to Danielson’s Domain 1, Planning and Preparation” is an example of how the Danielson ¹³ framework might be used to guide the development of interview questions.

Candidates’ responses to the questions such as the examples listed above can be assessed using a competency-based assessment rubric ³³ to evaluate them against the framework and measure them against an agreed-upon set of standards to determine if the response demonstrates a level of professional competency that is: “unsatisfactory,” “basic,” “proficient,” or “distinguished” for new teachers ³⁵. In the case of any “new to school division but not new teacher” hires, the competency-based assessment model can be adjusted to reflect what should be expected of an individual with a specific level of professional experience. In this way, the questions can remain largely unchanged but the level of competency-development can be adjusted to reflect teaching experience.

In the end, the same framework of teaching effectiveness can be used to assess applicants regardless of any differences they may have in years of experience. Newer teachers would be assumed to be developmentally less competent than those with say five or ten years of classroom experience.

7. Improving Outcomes by Collecting Useful Data and Analyzing this Data with Rigour

With limited resources at their disposal, namely time and money, educational leaders have little choice but to rely on proxy measures such as applications and pre-employment interviews to serve as predictors of potential teaching effectiveness ^{24, 47, 48, 49}. But, more evidence-rich, informative and realistically practical hiring approaches should result in more accurate predictions of who might be effective in a real classroom setting.

Quality teachers - that is those who demonstrate teaching effectiveness - are a key component for school effectiveness, and effective schools are staffed with effective teachers. The research is clear on this subject; more effective teachers, and by extension teaching, are key to improving educational opportunities for all children ^{50, 51}. Conversely, Staiger and Rockoff ⁵² argue that the greatest hiring cost faced by schools is the risk of exposing a group of students to a new teacher who turns out to be highly ineffective. The simple recognition that teachers are the most important in-school factor in explaining student learning gains demands that hiring committees use protocols that will better support them to identify the most effective teacher candidates in the hiring process and provide a rationale for how this assessment was determined to the public they serve.

While it is true that it takes effort and resources to hire well and find the best, this article demonstrates that effective hiring practices does not require a complicated system. What high quality personnel leadership does require are robust frameworks of teaching effectiveness that allow school leaders to wade through the “almost good enough to be teachers” pile of data to use information that will help them to find the most talented and gifted candidates. Hiring the best possible candidates makes a long-term difference to school and division quality and, most importantly, to student learning. By increasing student learning, good teachers gradually improve any division and often help improve their fellow teachers as well by pushing the professional conversation that colleagues engage in to deepen understandings of what constitutes effective teaching and learning practices.

In this era of data-driven decision-making in education ⁵³, it seems time to carefully examine how to improve the predictive potential of hiring processes to improve the likelihood of success. As is the case with high quality educational research, teacher hiring decisions rely on decision-makers possessing a conceptual understanding of what the end-goal of the process should be ⁵⁴. It seems trivial to argue that the teacher hiring process should be guided by anything other than an agreed-upon, research-based framework of teaching effectiveness. However, this often-overlooked but fundamentally important orientation requires that those invested in hiring the very best prospects commit to a process to develop a shared understanding of what “it” – in the form of teaching effectiveness - looks and sounds like in their schools. It is what every child deserves – an effective teacher in every classroom. By using a descriptive and research supported image of what it takes to be an effective teacher to guide the hiring process, the likelihood of finding one will increase.

References