Abstract

The study developed a culturally based standardized scale in Visible Learning for instructional leaders. Descriptive developmental research and qualitative descriptive research were employed in the study. Purposive sampling was used to select expert and target population judges who knew about visible learning, culture, and scale development. The measurement scale development followed the following procedures: Defining the knowledge domain that defined the domains in the most precise terms and then reviewing the literature. This stage searches for knowledge on standardizing scale instruments, gathering conceptual and operational definitions of visible learning and culture. The second stage is item development. This stage covered domain identification, item generation, and content validity. This also covered the specification of response format, and the pooled items were organized per domain. The third stage was scale development, where the different items were organized into measuring constructs. Items on culture were integrated into the different domains of visual learning to describe visual learning in Bicol classrooms. Theoretical or face validation of the developed scale was the fourth stage. This phase utilized expert and population judges to evaluate the items' content relevance and technical quality. The fifth stage was semantic validation, a confirmatory step to gauge the effectiveness of the developed scale when the pilot tested it on respondents. The sixth stage is statistical or empirical validation. The developed scale instrument was administered to the target population to determine its reliability. With Cronbach's alpha of 0.652, the developed culturally based scale possessed internal data consistency. Expert validators, starting from content validity until semantic validation, also established the validity of the scale instrument. Results of the statistical validation revealed that the school where the newly developed scale instrument exemplified visible learning to a great extent.

1. Introduction

The Philippine Department of Education (DepEd) offers various assessments for educational leaders to gauge student achievements (DepEd, 2019) ¹. However, Hoffmann et al. (2022) ² note that few assessments or checklists exist that support educational leaders- school administrators, teachers, non-teaching school staff members, and parents/student caregivers- in gaining insights and understanding into how students learn.

In addition, there is a scarcity of assessments that accurately measure the influence and impact of teachers and other educational leaders on students (Kemethofer et al., 2022) ³; the study also presents the role of culture in instructional effectiveness.

Numerous studies have uncovered interesting correlations regarding student achievements. The abundance of studies on achieving students' achievement is beneficial in many ways. For instance, Reardon et al. (2018) ⁴ and Anderson (2017) ⁵ identified significant correlations between socioeconomic status, academic performance, and teaching quality on student success. Jackson et al. (2020) ⁶ have highlighted how teaching quality impacts student achievements. Gupta and Joshi (2014) ⁷ highlighted the need for solid school-family partnerships, noting that parents' vital role in shaping children's expectations and learning styles enormously impacts student achievement.

On the other hand, Urbanovic et al. (2023) ⁸ and Hoque et al. (2023) ⁹ suggested that understanding students' ethnic and cultural backgrounds and appreciating cultural diversity through teaching could significantly increase student achievement.

Educational leaders are critical in shaping effective teaching and learning experiences, acting as catalysts for curriculum innovation, creating conducive learning environments, and building teacher-student relationships. However, there is little research about how to help teachers know their impact on the students, which is crucial to providing the best learning experience. Hattie's (2019) ¹⁰ concept of Visible Learning (VL) draws upon this lack of research by emphasizing "Know Thy Impact." The VL framework emphasizes that teachers should see themselves as evaluators of their own teaching, and the students should see themselves as their own teachers.

Several studies have primarily concentrated only on bits and pieces of factors that exert influence, produce effects, and impact student achievement; among these, pedagogical/instructional leadership is essential for promoting better student academic outcomes (Doyle et al., 2016) ¹¹. Socio-emotional learning long-term impacts academic growth (Mahoney et al., 2018) ¹². Digital tools, games, and group presentations in the classroom enhance student engagement and behavior. Instagram and TikTok discussions advance teachers' understanding of the computer-related text (Idrus et al., 2023) ¹³; Differentiated Instruction enhances students' academic achievement (Özüdoğru, 2022) ¹⁴.

Additionally, certain studies have specifically explored the favorable outcomes associated with cultivating a growth mindset in the context of teaching and learning, as exemplified by the work of Brock and Hundley (2017) ¹⁵, Dweck (2007) ¹⁶, and Hildrew (2018) ¹⁷. Other factors encompassing teacher-related aspects, parental involvement, support from the home environment, curriculum, constructive alignments, technology, feedback mechanisms, and community partnership., have been a focus of various academic achievements.

Knowing that Visible Learning is the teaching's Holy Grail (Fisher & Frey, 2018) ¹⁸, it presents an extensive compilation of educational evidence, comprising 2100 meta-analyses derived from 130,000 studies that encompassed a staggering 400 million students worldwide (Hattie, 2023) ¹⁹. The integration of the cultural aspect of the students in Hattie's Visible Learning Checklist offers inclusivity and seeks to comprehend and honor individual student experiences and perspectives. More importantly, it provides an opportunity for relevant and responsive feedback on the impact of teachers on student learning that works best in their culture and student successes; this study bridged the gap between the other studies.

Unfortunately, Hattie's studies remain scant in retrospect to culturally based aspects. It is not even an instrument to gauge Visible Learning with culture as an additional catalyst for academic achievement. It is for this reason that the current study was undertaken. There is a need for more attention on the role of culture in the Visible Learning implementations in different schools worldwide. As Martorana (2022) ²⁰ emphasizes, the central role culture plays in teaching and learning is such that it enhances student participation and learning outcomes.

The research is based on various theories, educational approaches, and principles that aim to comprehensively understand how culture influences various aspects of education, such as schools, classrooms, educational leaders, teachers, and students. The viewpoints that have inspired this research include the Visible Learning Theory, Culture-Learning Process, Socio-cultural Theory, Twelve Sources of Cultural Knowledge, Culturally Relevant Teaching, Culturally Responsive Teaching Approach, Culturally Sustaining Pedagogy Approach, and the Goldilocks Principle. These educational perspectives and theories form the foundation for recognizing the significant role that students' culture plays in their academic growth, progress, and overall success.

Moreover, they emphasized the importance of developing a standardized culturally based scale to measure the influence and impact of teachers and other educational leaders on students.

The Visible Learning Theory (VLT) of John Hattie (2012) ²¹ is the pillar of this research because it is the meta-analysis of the smorgasbord of school and classroom practices of students and educational leaders that impact student achievement. It is a meta-study that intensely analyzes the effect sizes of measurable influences on learning outcomes in educational settings and aims to move from "what works" to "what works best" and when, for whom, and why (Hattie & Zierer, 2018, p. 494) ²². In 2008, The Times Education Supplement magazine hailed this meta-study as the 'Holy Grail' of education because, after 25 years of rigorous investigation, it revealed the key factors that impact students' academic success (Corwin, 2023) ²³. It emerged as the ultimate outcome, shedding light on the factors that shape students' academic performance and the influences on students' achievement during their time in school.

Visible Learning (VL) highlights the roles and specific purposes that families, schools, their leaders, teachers, and students have designated. According to Hattie (2023) ¹⁹, these purposes include families taking on the role of their child's first learner, schools and their leaders leading the school-wide improvement processes so that all educators maximize their impact on students, and students becoming their own teachers and wanting to invest in school learning.

Moreover, teachers are encouraged to see their impact through students' eyes while teaching students to become their own teachers. As a result, teachers become students of their own teaching. In retrospect, the VL calls for an improved role for educators as evaluators of their own teaching and its effects on students. However, assessing the influence and impact of teachers on students is a complex undertaking. Teachers have the potential to impact not only the students they directly teach but also those they do not have direct contact with. While most studies focus on the effects of teachers on their own students within the classroom, teachers often engage with students in their school through additional avenues such as coaching or involvement in extracurricular activities, among others. With this as a backdrop, it is essential to note that learning is a socio-cultural process.

The Socio-Cultural Theory is advocated by Lev Vygotsky, who claims that children learn mainly through social interactions and that children's culture affects what they learn (Zacarian & Soto, 2020) ²⁴. More tellingly, he believes that social interaction with educators, or more knowledgeable others, is necessary for children to acquire new knowledge and that teachers must guide students' learning by tailoring intentional teaching for individual achievement (Nolan & Raban, 2015) ²⁵. Educational leaders, especially teachers, play a critical role in ensuring each student succeeds in the school and classroom.

To this point, teachers must customize intentional instruction to meet the individual needs of their students. This can be done by using instructional and classroom practices like differentiating the teaching and learning to ensure that each student has access to the appropriate level of support and resources needed to succeed. Teaching and learning at school take effort, sometimes much effort. Hence, valuing relationships and participation is vital to learning.

Relationships are the heart of teaching. It is about something other than knowing how to write the lesson plan but about creating an environment that fosters humanity and human relationships, enabling a seamless transfer of information (Nieto, 2013) ²⁶. Educators who endeavor to integrate the students and their family cultures into the lesson, actions, and language can significantly impact the students. By building a connection to students' home culture, educators can also help bridge the gap between home and school so that students can find more meaningful ways to connect with the materials and engage in the learning process. Valuing a student's culture can also help them to explore and celebrate their identity and unique experiences. All these elements can contribute to increased academic performance and self-worth. The same viewpoint shared by Westernoff et al. (2021) ²⁷ that students' feelings, self-worth, academic engagement, and achievement are enhanced when the students feel and see that their culture is valued because it reduces the student's affective filter, resulting a learning and linguistic risks as well as school and classroom participation.

Similarly, Walker-Dalhouse (2021) ²⁸ observed that a meaningful and effective caring relationship gradually evolves, resulting in mutual advantages for educators and learners. Notably, the most remarkable aspect is how this relationship fosters a profound transformation in teachers' perspectives, shaping their attitudes and beliefs regarding students and the essence of student learning. The latter is achieved through connecting learning to lived experiences, background knowledge, fund of knowledge, diversity, and culture.

Culture serves as a blueprint that shapes our thoughts, emotions, and actions within society (Gollnick & Chinn, 2017) ²⁹. It is a product of human interaction, socially constructed by individuals (Cushner et al., 2019) ³⁰. Cushner et al. identified twelve sources of cultural knowledge that contribute to cultural learning, including race, ethnicity/nationality, social class, gender, health, age, geographic region, sexuality, religion, social status, language, and ability/disability. Educators interpret and understand children's behavior as cultural agents based on their own cultural knowledge and beliefs.

On the other hand, children are viewed as cultural apprentices, eagerly seeking the wisdom and guidance of those with greater cultural experience. Over the years, the culture of students and teachers has been acknowledged as vital in promoting success among diverse students. It profoundly influences how students learn, teachers instruct, and leaders guide (Trumbull & Pacheco, 2005) ³¹. Admittedly, educators, being human, contribute their diverse cultural viewpoints, personal values, aspirations, and dreams to the educational setting. Simultaneously, they may inadvertently introduce their biases, stereotypes, and misconceptions into the classroom (Banks, 2016) ³². Hence, implementing effective teacher education programs or professional development opportunities is essential for the growth and development of educational leaders, specifically teachers, both before and during their service in the field. Consequently, it becomes imperative to adopt influential and empowering approaches established by the trailblazers of cultural education, namely Geneva Gay, Gloria Ladson-Billings, Django Paris, and Samy Alim.

Culturally Relevant Pedagogy, Culturally Responsive Teaching Approach, and Culturally Sustaining Pedagogy Approaches are joining forces in this study to construct a paradigm that is effective, transformative, and empowering approaches that will enhance students' academic and social achievement. The nomenclature used to describe these teaching approaches differs, encompassing culturally responsive teaching, culturally sustaining pedagogy, and the more foundational culturally relevant pedagogy. Scholar Gloria Ladson-Billings introduced the Culturally Relevant Pedagogy or Teaching in the 1990s. The Culturally Responsive Teaching Approach was coined by researcher Geneva Gay in 2000, and the Culturally Sustaining Pedagogy Approach was founded by Django Paris, who coined the term in 2012, and H. Samy Alim (Will & Najarro, 2022) ³³.

Culturally Relevant Pedagogy (CRP) emphasizes that this approach not only addresses student achievement but also assists them in accepting and affirming their cultural identities. Critiques will develop critical perspectives that challenge institutional inequities while challenging those perpetuated through schools (or other institutions). (Will & Najarro, 2022) ³³.This pedagogical approach also values students' cultural backgrounds but aims to empower students socially, academically, emotionally, and politically by using cultural references in teaching. It challenges the status quo and promotes critical thinking about societal and cultural norms (Mensah, 2021) ³⁴. In this context, the primary objective is not limited to improving academic or student achievement; it also encompasses the development of critical consciousness, cultural competence, and student learning. These three elements are commonly acknowledged as the key components of this educational approach (Snyder & Fenner, 2021) ³⁵.

Culturally Responsive Teaching (CRT) is an approach to education that uses methods for understanding, representing, and knowing various ethnic and cultural groups to teach academic subjects, processes, and skills (Gay, 2018) ³⁶. More importantly, Gay cited that Culturally Responsive Teaching rests on eight dimensions: validating, multidimensional, empowering, normative and ethical, transformative, emancipatory, humanistic, comprehensive, and inclusive. Aside from this, Gay contends that culturally responsive teaching necessitates a paradigm shift in pedagogy to address the academic underachievement of students from various ethnic backgrounds (Mensah, 2021) ³⁴. This approach focuses on the assets students bring to the classroom and emphasizes recognizing and leveraging the strengths they bring to the classroom rather than focusing solely on their limitations. In addition, culture is used effectively to enhance academic and social achievement. Doing so sets higher expectations and ensures that learning is meaningful and applicable to all students.

Culturally Sustaining Pedagogy (CSP) advocates for schooling to be a site for sustaining—rather than eradicating—the cultural ways of being of communities of color (Will & Najarro, 2022) ³³. The objective is to guarantee academic achievement for every student, particularly emphasizing students belonging to nondominant cultural communities. Paris (2012) emphasized the importance of incorporating heritage and traditional practices into teaching while allowing space to include new and evolving cultural practices (Lazar et al., 2022) ³⁷. Alim et al. (2020) ³⁸ further highlighted that this approach acknowledges culture as a dynamic combination of values, beliefs, and practices that vary based on students' social identities, including gender, sexuality, disability, socioeconomic status, geographical location, and time period.

Although these three approaches may have distinct philosophical orientations, they all strive towards shared objectives centered on culture, race, and equity. In simpler terms, they embody asset-based pedagogies and aim to ensure the success of historically marginalized students within the educational system. That being the case, CRT, CRP, and CSP are powerful and effective transformative approaches promoting inclusivity, equity, and academic success. By valuing and incorporating students' cultural backgrounds and experiences into the instructional, classroom, and school practices, educators can create a more meaningful and impactful learning experience for all students. Culturally Responsive, Relevant, and Sustaining approaches are not just about teaching diverse students but also about empowering them to embrace their cultural identities and excel academically.

The school and classroom represent a captivating intersection where diverse cultures converge. It is a space that brings together individuals from various walks of life and backgrounds, resulting in a rich tapestry of interactions. Within this environment, people are encouraged to share their unique values, traditions, beliefs, perceptions, experiences, knowledge, attitudes, and emotions, thereby fostering an environment that is truly enriching. Culture serves as the bedrock of this study. By understanding the culture of the learners, teachers and researchers can adapt their research methodology to ensure its cultural appropriateness. This can contribute to obtaining more accurate and reliable results. Additionally, understanding the culture of the individuals can provide researchers and teachers with valuable insights when designing interventions or programs that aim to address specific needs or concerns of a particular locality or community. Therefore, the connection between Visible Learning and Culture is undeniable.

The Goldilocks principle comes into play within the scope of this study. According to Renandya (2022) ³⁹, individuals are naturally inclined towards finding the perfect balance in various aspects. They prefer having just the right amount of something rather than too much or too little. This principle aligns with the concepts of Desirable Difficulty and Productive Struggle (Renandya, 2022) ³⁹. People often choose a middle ground regarding factors like the sweetness of their beverage, room temperature, task difficulty, and more.

In the context of this study, the Goldilocks principle (DeWitt, 2014) ⁴⁰ Goldilocks principle can be seen as the cornerstone of this study, emphasizing the need to strike a balance between difficulty levels and familiarity of content to ensure it is challenging yet remains accessible for students from different cultural backgrounds. Furthermore, this principle encourages teachers to utilize students' cultural traditions and beliefs to leverage prior knowledge with new content or skills that students are learning.

Similarly, it recognizes the significance of adapting educational practices to suit different locations or places' unique needs and cultural backgrounds. Just as the Visible Learning practices implemented in one country may not be suitable for another, the Goldilocks Principle emphasizes the importance of considering the specific culture and needs of a particular location or place, such as Naga City in the Philippines. By doing so, organizations like the Department of Education and Naga City-based schools can ensure that the instructional and classroom practices they implement meet the specific requirements of that area. This approach contributes to promoting academic growth, fostering student success, enhancing instructional practices, validity and standardization of measurement scales, and providing educational leaders with an accurate method of knowing and understanding their impact on the students. This is the true purpose of this research.

Figure 2 presents the conceptual framework that guided the study's conduct, using the methodological framework in the figure's title. The framework used the input, process, and output (IPO) model since the study resulted in the development of culturally based standardized scale measurement in visible learning.

Input

To begin the standardizing scale instrument, the initial stage involves conducting a comprehensive literature review on this subject matter, visible learning, culture, and adapting and developing standardized scales. The goal is to pool relevant empirical items that accurately depict each dimension.

The books of Hattie and his co-authors were essential literature sources for Visible Learning.

John Hattie's book on Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement, Hattie (2009) ⁴¹ is an extensive investigation analyzing over 800 meta-analyses about academic achievement.Visible Learning for Teachers: Maximizing Impact on Learning, by Hattie (2012) ²¹. This is where the researcher adapted the "Checklist for Visible Learning Inside" (Hattie, 2012, pp. 207-211) ²¹ and Visible Learning in Action: International Case Studies of Impact, Hattie et al. (2015) ⁴².

Culturally based Teaching & Learning. Geneva Gay's work, "Culturally Responsive Teaching: Theory, Research, and Practice" (2018) ³⁶, was another resource contributing to the formulation of items on culture that were integrated into the scale instrument for visible learning

Developing and standardizing the measurement. Hair et al. (2019) ⁴³, Boateng et al. (2018) ⁴⁴, Kyriazos & Stalikas (2018) ⁴⁵, and Carpenter (2018) ⁴⁶ shared processes for standardizing scale instruments that were combined and utilized as models in developing and validating (standardizing) our culturally based scale instrument for visible learning.

This study employed sound methods of standardizing scales, leading to six main stages for developing and validating measurement scales. These steps are Defining the Knowledge Domain, Item development (covering identification of domain, item generation, and content validity), scale development, Theoretical or face Validation, Semantic validation, and Statistical validation.

Process

The studies of Hair et al. (2019) ⁴³, Boateng et al. (2018) ⁴⁴, Kyriazos & Stalikas (2018) ⁴⁵, and Carpenter (2018) ⁴⁶ were the basis for developing and standardizing the scale measurement. The phases of development included the following:

Defining the knowledge domain. In scale development, it entails accurately defining the knowledge domain. Hair et al. (2019) ⁴³ advised that understanding this domain involves creating clear definitions and operationalizing terms before reviewing the literature. Key concepts defined are "visible learning," "culture," "scale," and "scale of measurement for culturally based scale" or standardized scale.”

Hattie's (2012) ²¹ work is referenced throughout, specifically, his indicators of visible learning listed in an appendix of his book Visible Learning for Teachers: Maximizing Impact on Learning." These indicators cover various facets of teacher-student interactions, such as teacher inspiration, decision-making, lesson phases (planning, starting, midway through with learning/feedback, and ending), and teacher mind frames. Furthermore, this Step defines scales (culturally based standardized scales and instructional leaders such as principals/departmental heads).

Item Development. This stage entails specifying response format requirements and organizing items into an initial inventory pool, divided into three steps. a). Identification of Domain. To effectively generate items, it is vitally important to identify conceptual and operational domains and subdomains and their boundaries to facilitate item generation. b). Item Generation. This Step involves gathering items pertinent to a domain and organizing them into an initial survey. c). Content Validation. Content validity involves evaluating whether individual items per domain meet the required standards of appropriateness and adequacy.

Scale Development. The third Step involves organizing individual items into an effective scale to assess a desired domain. It aims to ensure that both questions and answers accurately represent this area.

Theoretical or face Validation. In this fourth stage, expert validators gather data. All items that comprise each domain are evaluated through surveys for content relevance, representativeness, technical quality, and face validity. Interviews may also be held with validators to gauge face validity.

Semantic Validation. The fifth Step is a confirmatory one that measures the effectiveness of the developed scale when administered to respondents. Expert validators provide data that helps evaluate each domain item for content relevance, representativeness, and technical quality. This process was done by providing the expert validators with the survey or the proposed scale instrument. Then, a focus group discussion (FGD) was conducted.

Statistical Validation. In this sixth and final stage, an empirical validation process involves administering the developed scale instrument to a sample representing its intended population range to assess its reliability and validity measures.

Output. The output of the study is the Culturally Based Standardized Scale in Visible Learning for Instructional Leaders. The final culturally based scale for visible learning for instructional leaders has been expanded to 64 items to incorporate necessary culturally based items. There were 5 items for Inspired and passionate teaching, 9 items for planning, 14 items for Starting the lesson, 14 items for During the lesson: Learning, 11 items for During the lesson: feedback, 10 items for the end of the lesson, and 1 item for Mind frames.

This study developed a culturally based standardized scale in visible learning for instructional leaders.

2. Methods and Procedure

Descriptive development and qualitative descriptive research were employed to conduct the study. Developmental research can be defined as the systematic examination of creating, improving, and assessing instructional programs, procedures, and resources per standards for internal consistency and efficacy. (Richey et al., 2004) ⁴⁷. Richey et al. defined developmental research as performing instructional design, development, or evaluation activities while studying their process simultaneously or studying all or specific components of that process as it unfolds. So, developmental research involves situations in which the product development process is analyzed and described, and the final product is evaluated. The entire process of descriptive developmental research was applied to develop the culturally based standardized scale in visible learning for instructional leaders.

Qualitative descriptive research was employed as another research design. This design is suitable for research inquiries to explore experiences and acquire valuable insights from participants (Bradshaw et al., 2017). ⁴⁸ When providing a straightforward explanation for a phenomenon or collecting data for developing questionnaires or interventions, labels like these are frequently employed (Kim et al., 2017) ⁴⁹ and validate and corroborate findings in convergent studies (Doyle et al., 2016) ¹¹. Qualitative descriptive research was employed in this study to elucidate how culture was incorporated into the measurement scale and the comprehensive progression of developing, validating, and standardizing the culturally based standardized scale in visible learning.

Cronbach's Alpha Test was calculated to assess the internal consistency and reliability of the data. Reliability and validity measures ensure the scale's reliability, meaning repeated applications of the same scale must lead to consistent scores.

Determination of the validity of the scale instrument started from the definitions of culture and visual learning and their dimensions up to semantic validation. The scale should be valid to confirm that the scale measures the construct (or constructs) that is (are) intended to be measured.

Mean was computed to describe visible learning in the school where the developed culturally based standardized scale was piloted. The scale instrument used the 6-point Likert Scale where 1 means strongly disagree while 5 means strongly agree. The range is determined to determine the interpretation of the result.

Range = Highest Point – Lowest Point/ No. of Categories

6-1/6 =0.83

Applying the range in the 6-point Likert Scale result, the interpretation is as follows:

The measurement scale underwent stages of development and validation, resulting in its reliability and validity. Consequently, the culturally based scale in visible learning for instructional leaders has been standardized.

The study aimed to develop a culturally based standardized scale in visible learning for instructional leaders. The process of standardizing scale instruments shared by Hair et al. (2019) ⁴³, Boateng et al. (2018) ⁴⁴, Kyriazos & Stalikas (2018) ⁴⁵, and Carpenter (2018) ⁴⁶ were amalgamated and made models in the development and validation (standardization) in the development and validation (standardizing) of the proposed culturally based scale instrument for visible learning.

The combined sound methods of standardizing scale resulted in the six main stages of developing and validating measurement scale applied in the present study. These are Defining the Knowledge Domain, Item development (covering Identification of domain, item generation, and content validity), Scale development, Theoretical or face Validation, Semantic validation, and Statistical validation, as shown in Table 3.

3. Result and Discussion

Various resources provided insights into the procedures and contents of the developed scale instrument. Hattie and his co-authors' books were sources of information on effective teaching methods, teaching strategies, visible learning techniques, and evidence-based practices necessary for optimizing student outcomes.

Research conducted on scale development and validation by Hair et al. (2019) ⁴³, Boateng et al. (2018) ⁴⁴, Kyriazos & Stalikas (2018) ⁴⁵, and Carpenter (2018) ⁴⁶ were valuable models in the development of the culture-based standardized scale in learning for instructional leaders.

Defining the Knowledge Domain. This step entails defining the knowledge domain in the most precise terms. Hair et al. (2019) ⁴³ suggested this step, saying it is logical to first know the domain under study by determining its exact definition and the items that can operationalize the terms before proceeding to the next steps of scale development. So, the essential terms defined conceptually and operationally are visible learning, culture, scale, culturally based scale, and standardized scale.

The different dimensions were taken from the indicators of visible learning from the books of Hattie (2012) ²¹, particularly in Appendix A: Checklist for Visible Learning Inside of his book Visible Learning for Teachers Maximizing Impact on Learning. The dimensions start with the teacher being inspired and passionate, followed by the sequence of decisions teachers are asked to make regularly and the various phases of the lesson interaction between teacher and students. These are Planning, Starting the lesson, During the lesson learning, During the lesson feedback, End of the lesson, and Mind frames.

Then, definitions of scale and culturally based standardized scales were looked into. The definition of instructional leaders was fixed on teachers and school heads such as (principals and department heads).

The precise definitions were the researcher's guide in conducting the literature review. Conducting a sound literature review or interviews with experts was considered the first step in developing the measurement scale by the model researchers. The literature review was performed instead of interviews with experts because the literature on visible learning, culture, scale development, and standardization is abundant. The review is essential for two reasons: using authoritative publications from respected journals as the cornerstone of good research should come naturally, and selecting such references ensures that state-of-the-art research in any scientific field is relied upon.

Books, journals, thesis, dissertations from other universities, and relevant electronic papers that ensure state-of-the-art research were chosen as references. This process was done to gain more knowledge on standardizing scale instruments and a concrete understanding of the constructs of visible learning, culture-based teaching, and learning. Moreover, further search was conducted for conceptual definitions of each dimension and relevant empirical items that describe each dimension. Item pooling was done to gather items describing visible learning, culture-based teaching, learning, and measurement scale development.

Item Development. The definitions (conceptual rationale) of each construct, visible learning, culturally based teaching, and learning were finalized. Likewise, the boundaries of the domain were ascertained. The pooled items from the literature review were organized for each of the constructs under study. The response format in the Visible Learning Survey by Hattie (2012) ²¹ was adopted with six Likert scales: 1 represents strongly disagree, 2 generally disagree, 3 partly disagree, 4 partly agree,5 generally agree, and 6 strongly agree. This stage covers identifying domains, item generation, and content validity.

a. Identification of domain. The initial step in scale development is determining the domains and subdomains and establishing a common understanding by defining the knowledge domain in the most precise terms. The domain definitions provided working knowledge, and the domain's boundaries eased the item generation and content validation process.

Then, an operational set of items accurately reflecting the measured concept was formulated. Operational definitions are necessary so that constructs or knowledge domains that cannot be defined directly can be assessed indirectly through specific elements. Constructs or concepts that cannot be determined, measured, or evaluated should be provided with multiple indicators(Hair et al., 2019) ⁴³.

b. Item generation. During the literature review, item generation or item pooling was done simultaneously on the different domains and subdomains of visible learning and culture concepts as applied in instruction. These were infused in Hattie's (2012) ²¹ checklist for visible learning, constituting the proposed measurement scale in this study.

c. Content Validation. Content validity or theoretical analysis refers to the adequacy and appropriateness of items to assess the different domains and subdomains. The other items were organized to constitute the initial pool. Determining the adequacy and suitability of the items per domain of visible learning was done with the help of seasoned researchers in the field of education.

Scale Development. The checklist of visible learning by Hattie (2012) ²¹ was adopted to determine visible learning in school. The different subdomains of visible learning were adopted and retained in the measurement scale. These include: 1. Inspired and passionate teaching, 2. Planning, 3. Starting the lesson, 4. During the lesson: learning, 5. During the lesson: feedback, 6. The End of the lesson, and 7. Mind frames.

Then, concepts of culture as applied in instruction were integrated into the checklist to fit the culturally based standardized scale for visible learning. Items on culture were added to the items formulated by Hattie to make the measurement unique to the Philippine Educational setting. Then, the survey was prepared. The researcher developed this first draft of a culturally based standardized measurement scale for visible learning with the help of five experts. The rule of thumb is "the more, the merrier," – but if there is not enough money and time to perform this activity, around four to six experts should solve the problem (Hair et al., 2019, p. 494) ⁴³.

Initially, Hattie's scale instrument was composed of 43 items. There were 5 items for Inspired and passionate teaching, 6 items for Planning, 12 items for Starting the lesson, 9 items for During the lesson: learning, 6 items for During the lesson: feedback, 4 items for the End of the lesson, and one item for Mind frames with 8 sub-items. Then, when items on culture were added, the first draft survey was finally made of 64 items. There were 5 items for Inspired and passionate teaching, 9 items for Planning, 14 items for Starting the lesson, 14 items for During the lesson: learning, 11 items for During the lesson: feedback, 10 items for the End of the lesson, and 1 item for Mind frames. Writing more good items than required permits the selection of the best items to estimate the target construct best. Kyriazos and Stalikas (2018) ⁴⁵ mentioned Devellis (2017) ⁵⁰ that the initial item pool should be larger than the final scale set. As a rule, it can be 3 or 4 times larger, or if the construct is relatively narrow, 2 times larger. Initially, more items were pooled. Some may not perfectly fit the domain or may duplicate some items. These undesirable items, non-fit items, or items least related to the domain were eliminated (item reduction) from the initial pool in the theoretical or face validation. Since the scale instrument was adapted from Hattie's visible learning scale and integrated with values, the instrument was validated in the new context (Hattie et al., 2015) ⁴². This process is theoretical or face validation.

Theoretical or Face Validation. This stage gathers items constituting the domain for content relevance, representativeness, and technical quality. These different items pooled for visible learning and integration of culture were presented to experts for evaluation to determine if they fit in each of the domains of visible learning. Experts also evaluated the items for culture integrated in the different domains to determine if they fit in each of the domains of visible learning. In this phase, the content validity of each domain, which consisted of different items, was determined. The experts selected were seasoned researchers with experience developing surveys, questionnaires, and checklists).

Before the instrument scale was made available to validators, the necessary permit to conduct the study was secured from the identified heads of agencies, such as the Schools Division Superintendents of the Divisions of Naga City, Iriga City, and the province of Camarines Sur. Likewise, Presidents of Camarines Sur Polytechnic Colleges, Nabua, Camarines Sur, Bicol State College of Applied Sciences and Technology, Naga City, and Central Bicol State University of Agriculture, Pili, Camarines Sur, Ateneo de Naga University, Naga City, and Modern Learning Center of Iriga City approved the conduct of the study (Appendix A). Upon approval, coordination was done with the expert validators whom the researcher believed had full knowledge of visible learning and culture as integrated into instruction and the school. These experts are composed of school administrators and faculty members.

The culturally based measurement scale for visible learning was pretested to 36 respondents. According to Hair et al. (2019) ⁴³, between 20 and 40 respondents should be obtained to complete this step, although the rule is that the more, the merrier. Parsimony suggests getting the maximum information at a minimum cost.

The scale instrument was administered both individually and in group settings. The evaluation and suggestions of experts were collated and compiled (See Appendix D).

After considering the suggestions given by the experts, another draft of the instrument scale was developed for the fifth stage, the semantic validation. This edited part becomes the second draft of the scale instrument.

Semantic Validation. Hair et al. (2019) ⁴³ suggested collecting 20-40 responses as part of this confirmatory step to assess its efficiency based on use in research subjects representing potential clients (Hair et al., 2019) ⁴³. If more responses were collected for testing purposes, more would be better (the rule is still the same – the more, the merrier). Parsimony is also welcomed: get the maximum information with minimum cost.

The ideal situation to perform a semantic validation session is obtained by putting together the respondents in the same venue. Alternative measures were opted for since this was impossible to accomplish. Focus group discussion (FGD) and individual interviews were conducted. In both cases, the researcher explained that this step is an initial application of the developed scale and asked for the help of the respondents to determine the effectiveness of the culturally based scale for visible learning.

During the first draft of the instrument scale, the expert evaluators were invited to the FGD. Unfortunately, some declined. Those available for focus group discussion were invited to one venue to share their thoughts, experiences, and insights. Those who could not attend were contacted personally for an interview by providing the researcher with their feedback via a Google document link to the checklist provided for each expert. Some were contacted via video call. Participants shared their views and experiences and explained their evaluation of the culturally based scale for visible learning.

Since the FGD was done via Zoom, the researcher assigned a moderator/facilitator while the researcher was online. Participants were given a printed copy of the survey or the developed measurement scale. The moderator/facilitator introduced the participants to the FGD. The researcher explained to the participants that they were not expected to respond or rank any items; instead, they should focus on evaluating the meaning and understanding of each statement. Because in scale development, one must consider the difficulties of the respondents in understanding the statement (the semantics of the items). Then, the changes made based on the experts' evaluation and suggestions in the first try-out of the scale instrument were presented as they were considered in the finalization of the instrument scale. The researcher also explained some suggestions that should have been considered. At the focus group discussion, the participants were invited to engage in an in-depth exploration of the construct domain and subdomains. They were encouraged to identify any new items they thought should be added in these areas before moving onto discussing how applicable item terms were in relation to these construct domains and subdomains; participants were then asked to consider their comprehensibility by noting any words which might cause difficulty when understanding.

Discussion then focused on whether each item was properly classified within their domains and subdomains, with participants also asked to identify any technical terms which could be simplified for easier comprehension. As discussion concluded, participants shared their thoughts about ways in which culturally tailored scales might improve visible learning.

The expert judgments and suggestions were recorded to avoid missing essential data that can contribute to biases in item assessment (Appendix D). The suggestions were again incorporated into the third and final scale instrument.

Table 3 shows the final standardized scale that integrates culturally based items. Notably, items italicized represent culturally based items derived from research and input from experts. Furthermore, additional culture-related items have been distributed among the various domains of Visible Learning as defined by Hattie.

No new items were added in the Inspired and Passionate Teaching domain; four existing ones underwent redefinition. Meanwhile, in the Planning domain, there are four culture-specific items.(Items 9 sub-items f, 10, 13, 14, and 13 sub-items respectively). The Starting the Lesson domain added two cultural items, 15 and 27. Items 34, 37, 38, 39, 40, 41 up to 42 were the included cultural items on the During the Lesson Learning domain, and items 47, 48, 49, and 50 (accompanied by two sub-items and 51-53 with three sub-items) were added as cultural additions during the lesson: feedback domain. Finally, the End of Lesson dimension was expanded to encompass items 68-63, and 1 sub-item (i) was added to the Mind Frames domain.

Statistical Validation. The finalized culturally based scale for visible learning was then administered to school administrators (principals and department heads) and teachers in one of the elementary schools in the Naga City Division. The scale instruments were then retrieved, followed by data management for statistical computations. Then, the researcher created a spreadsheet with the items in the columns and the respondents in the rows.

Reliability of the scale measurement. Cronbach's alpha test was computed to determine the internal consistency and reliability of the data (Hair et al., 2019) ⁴³. Then, the mean was calculated to describe the participants' responses in each item per domain in the measurement scale. The Cronbach's alpha test yielded a value of 0.652 (Appendix E). According to Hair et al. (2019) ⁴³, values above 0.60 and up to 0.70 in exploratory studies are acceptable. This means that the culturally based standardized scale instrument on visible learning possessed internal consistency and reliability of the data.

Validity of the scale measurement. Content validity (also referred to as theoretical analysis) refers to the extent to which a measure accurately evaluates its domain of interest (Boateng et al., 2018) ⁴⁴. For any measure to meet its purpose effectively and accurately measure what was intended, content validity must accurately capture relevancy and representation, ensuring items represent experiences relevant to those being studied by capturing content relevance/representations and content representation and reflecting relevant experiences from populations being researched. Validation through content validity should be an ongoing process that begins with identification/delineation/applicability between study domains/constructs being validated against other constructs/domains being studied (Boateng et al., 2018) ⁴⁴.

Thus, the validity of the developed measurement scale started from the evaluation of experts on the content validity of the pooled items (Boateng et al., 2018) ⁴⁴. Expert and target population judges mainly assessed content validity. Expert judges (scale development or target construct experts) or target population judges (potential users of the scale) can play an essential role in assuring content validity. This approach aligns with recommendation that scale creation considers expert opinions and those from their target population.

Boateng et al.(2018) ⁴⁴ mentioned Guion, who proposed five conditions that must be satisfied to claim any form of content validity. These conditions are broadly applicable to scale development in any discipline. These include that the behavioral content has a generally accepted meaning or definition; the domain is unambiguously defined;the content domain is relevant to the purposes of measurement; qualified judges agree that the domain has been adequately sampled based on consensus and the response content must be reliably observed and evaluated.

Further, the population judges evaluated the face validity, a component of content validity. Face validity relates to how respondents or end users perceive that items of an assessment tool are relevant to its intended construct. Given its significance for maintaining data quality and instrument generalizability, future research should provide multiple methods to evaluate validity; doing so would strengthen the psychometric integrity of analyses.

4. Conclusions

Data analysis led to these conclusions. 1. The developed culturally based standardized scale in visible learning for instructional leaders was reliable and valid and can be used to measure visible learning in classrooms and schools. 2. Results of the statistical validation showed that the respondents confirmed that Visible Learning is exemplified to a great extent in the school where the developed scale instrument was tested.

5. Recommendations

Due to the findings presented in this investigation, three key recommendations have been derived.1). Considering that the developed culturally based standardized scale in visible learning for instructional leaders was reliable and valid, it is suggested that other regions in the Philippines may adopt the standardized scale or may be changed by adding culture items to suit their own practices. 2). The reliability and validity of the scale measurement developed were determined mainly using expert and population judges at the elementary level. Further research may consider instructional leaders and students in the secondary and tertiary levels. However, items should be ensured to fit the instructional practices at these higher educational levels. 3). To foster a culturally inclusive and equitable educational environment, it is necessary to prioritize humanistic values, foster emancipatory components, and implement ethical behavior as the basis for equitable practices. Furthermore, include multicultural perspectives in instruction and curriculum development processes, integrate multidimensional approaches that reflect diverse experiences, validate cultural practices by acknowledging students from diverse backgrounds, and recognize students with diverse learning needs. Following these recommendations promotes inclusive learning experiences while furthering social justice, equity, and comprehensive education for all.

References