Abstract

This research aims to validate multiple representations instruments for measuring student’s multiple representations skill of electrolytes and nonelectrolytes. This research uses quantitative descriptive method with nonexperimental approach. This research applies stratified sampling method with total 94 students who respond to 40 items multiple choices that cover macroscopic, microscopic, symbolic, and mathematical aspects. Rasch model in this research analyzes the instruments such as determining the unidimensionality, reliability, item map, item measure, item fit, and test information function. Data analysis shows that the finding of good category of validity and reliability of the instrument can be used as a diagnostic tool in the assessment of student’s multiple representation abilities.

1. Introduction

Chemistry is the study of the structure, properties, and reactions of element and substance. Chemistry covers the understanding of macroscopic, submicroscopic, and symbolic levels ¹. The use of different representation can help learners connect one level to another in a better way. Conceptual understanding is the key aspect of chemistry learning. Learners can have strong chemical concepts understanding when they can connect their insights in different representations ^{2, 3, 4} and how to relate each new concept or fact in the three domains that is macroscopic: how chemical phenomena can be observed by using the five senses of human such as color, smell etc.). Submicroscopic is the interaction or the form of invisible molecules including atoms, molecules, ions, and so on. Symbolic is the representation that consists of formulas, equations, symbols, mathematics, and graphics ^{5, 6}. However, chemistry teaching rarely helps students connecting some representations. This teaching method often leads learners to confusion which has negative consequences on students' motivation and achievement in chemistry classes ⁷.

Generally, learners have difficulty in explaining chemical reactions and connecting between phenomena, then how to represent that ^{2, 8, 9}. Learners have many difficulties in studying microscopic and symbolic representations rather than macroscopic ones. It happens because the levels of microscopic and symbolic are invisible and abstract so it requires reasoning process ^{10, 11, 12}. Assessment is a vital component in education. The interaction between assessment, curriculum, and instruction is very important to improve the teaching and learning process ¹³. The research conducted by Setiadi ¹⁴ shows that at planning stage, many teachers ignore the function of pre-test and do not perform an instrument analysis before the assessment process. Teachers also find difficulties in getting the results of assessment to know student learning progress as well as student learning difficulties ¹⁵.

Credibility of the research refers to how accurately to answers research questions or the strength of research conclusions. Indicators of success in measurement instruments are reliability and validity of measurement. Validity refers to measurement accuracy. Validity refers to how well the assessment tool is able to measure the desired end result ^{16, 17}. Validity is used by researchers as evidence collected in the quality of assessment instruments. Research reliability refers to the how accurate a research can answer research questions and the validity of the research conclusion. The validity refers to the measurement accuration. It relies on how good measurement tools can measure the final result ¹⁷. It is used by the researcher as the collected evidence in the form of measurement instrument quality. Reliability shows that multiple choices measure something consistently. Reliability does not what kind of knowledge, ability, and/or skill that is measured. Therefore, the evidence of validity becomes an important aspect before concluding that the order of multiple choices is valid ¹⁸.The quality of multiple choices is determined by field trial process to evaluate the characteristics of each item. In fact, many teachers do not perform an instrument analysis before the assessment process. Many teachers never do pre-test and multiple choice analysis because they do not have the competency to analyze the test ¹⁹. Teacher develops assessment test instrument only to complete content they will give to the students. The test only focuses on the numbers arrangement. It does not stimulate the way how students has to solve the problem. Consequently, the ability of students to think high, critical, creative, and problem solving is not too prominent. It only plays on numbers ²⁰.

The study of the multiple representation instrument of test arrangement is important because it is useful to measure and analyze the students’ ability of multiple representations on electrolyte and nonelectrolyte materials. It helps teachers to use appropriate strategies, approaches, or learning models. This research gives many benefits to students, teachers, schools and stakeholders. It needs good and credible instrument and of course completed validation and reliability to measure students’ multiple representation ability. This research aims to validate the instrument test to measure multiple representations of senior high school students’ ability in Yogyakarta on electrolyte and nonelectrolyte materials. The research problem discussed relates to the material of electrolyte and nonelectrolyte solutions only.

2. Material and Methods

This research is a quantitative descriptive research by using sample of senior high students with the same tenth grade for electrolyte and nonelectrolyte material. It uses non-experiment design where researcher does not give special treatment to the students.

The population of the study is all senior high school tenth grade students in Yogyakarta. The sample is taken using stratified purposive sampling method from 3 senior high schools consisting of 94 students as research sample.

The tested multiple representation instruments consist of 40 multiple choice items covering the macroscopic, microscopic, symbolic, and mathematical aspects given after the learning process. The instrument are completed with answer sheets, manual, and key answer. The item of instrument are arranged systematically by paying attention to comprehensiveness and depth of the material.

The aims of this research is to validate the instrument test so the validation was done in two ways namely theoretical validation and empirical validation. For theoretical validation performed by expert judgement and education practitioners. The instruments that have been compiled are validated by validators using V-Aiken. For empirical validation done by testing the instrument to 94 students from 3 schools in Yogyakarta city. Empirical validation of the instruments is analyzed using Rasch Model to determine the validity, reliability, fit items, problem level, and function of information.

3. Results

The application of Rasch model in this research in determining instrument validity and reliability is important to define valid and reliable item construct and to give a clear definition about measurable contruction in constistent with theory. Interestingly, this model can be used effetively for consistently measurable items and valid respons pattern ⁶.

The instrument of the test consists of 40 items of multiple choice with pretest and key answer. The judgment of experts and education practitioners performs as theoretical validation for the questions that have been arranged in the aspect of content, construction, and language. Then the result of validation is analyzed using V-Aiken. Table 2 shows the results of Aiken index analysis.

The result of expert judgment and education practitioner judgement (4 rater in total) and 5 rating scale category in Table 2 shows that the average index of Aiken is 0.956 for the substance aspect, 0.967 for the construction aspect and 0.959 for the language aspect. Item with index more than 0.88 is acceptable. The result of the analysis indicates that total Aiken index from 40 item has an average more than 0.88. It can be concluded that the multiple representation instrument fullfills the theoretical validity and can be used for empirical validity testing ²¹.

Empirical validation is performed to construct validation and decision of sample measure by using SPSS 16 Program. The subject of empirical validation in this research is 94 tenth grade students of 3 senior high schools in Yogyakarta. The first phase is to do sample adequacy test. The measure (size) or number of samples is sufficient and eligible for analysis when the analysis of KMO-MSA test is more than 0.05 (> 0.50) and Bartlett test significance is less than 0.01 (<0.01). Table 3 shows the result of sample adequacy test.

Table 3 shows that the result of KMO-MSA test analysis is 0.694 and Bartlett Significance Test is 0.00. It concludes that the sample size of test is eligible for further analysis.

Unidimension is defined as the presence of a latent underlying data trait ²². Unidimension of the instrument is an important measure to evaluate whether the instrument can measure multiple representational abilities of students from dimension items of Winstep Program.

According to Figure 1, the result of raw variance measurement data is 31.7%. It indicates that the test instruments is only able to measure 31.7% of the desired capability with 68.3% of data that is unexplainable. The minimum unidimensionality requirement is 20% ²³ and it can be concluded that the instrument is only able to measure one representation of multiple capabilities ²⁴.

Item fit explains whether the items work normally to do measurement or not. The item fit analysis is done using Winstep program. Table 4 shows the criteria of conformity level of the items according to Sumintono & Widhiarso ²⁵.

The test instrument has to meet one criteria to be considered fit. When it does not meet one criteria, e.g. it does not meet the criteria of MNSQ outfit then it can be rematched with the criteria of ZSTD or Pt. Mean Corr outfit ²⁵. Table 5 shows the result of item fit analysis of this research.

Table 5 shows the results of item fit analysis to measure the students’ ability of multiple representations. According to information of Table 5, there are 3 unfit items from total 40 items after analysis process using Rasch model PCM 1-PL. The 3 unfit items are 9, 16 and 30. The unfit items are removed for the misconception of students on the items, leaving 37 items to measure the ability of multiple representations of students

Reliability refers to wether test instrument gives the same result every time it is used in the same setting with the same subject type. Basically, reliability means consistent or reliable results. Reliability is a part of the validity assessment ¹⁷. The analysis reliability test considers the value of Alpha Cronbach that is obtained from Winstep program analysis; summary statistics. According to Bhatnagar et al. ²⁶, it uses the criteria to see reliability based on Alpha Cronbach value that can be seen in Table 6.

The Alpha Cronbach value of this study is 0.86. It shows that the category of reliability in the test instrument to measure the multiple representation capabilities of students is categorized as Good ²⁶. The consistency of items can be used for other samples that have identical (or almost identical) characteristics ²⁷.

The Figure 2 shows that the value of person reliability is 0.83 and Figure 3 shows that the value of item reliability is 0.95. It can be concluded that both consistency of respondents answers and quality of items in special instruments are good ²⁴.

According to classical test theory, item difficulty is the percentage of students who answer an item correctly. The greater the percentage of students who work on the item correctly, the easier the item (question) is being worked on. If the test items are very difficult, then the majority of exam scores will be very low. If the test items are done very easily, then the test score will be very satisfying ²⁸. Item measure information is analyzed using Winstep program to find the difficulty level of items. Table 7 shows the criteria of problem level according to Adedoyin & Mokobi ⁷.

Table 8 provides information about the difficulty level of items from the instrument. The analysis results of the difficulty level of items in the instrument that measures students’ ability of multiple representations shows that there are 10 easy items, 20 medium items, and 10 difficult items.

Test information function is performed to obtain information of the measurements. The information function is analyzed using the Winstep program that can be performed from the menu of graph-information information function. Figure 4 shows the curve of test information function from this study.

According to the curve in Figure 4, it can be concluded that the result of 40 questions given to 94 students represents a very low ability levels, and the information from the measurement is also quite low. Similarly, at the very high ability level the information from the measurement is also quite low. At moderate ability level, the information from the measurement is very high. The conclusion shows that well arranged items determine the students' ability moderate level only ²⁴.

4. Discussion

The result of this study is theoretical validity analyzed by expert judgement and education practitioner judgement (4 rater in total) and 5 rating scale category shows that the average index of Aiken is 0.991 for the substance and language aspects, and 0.993 for the construction aspect. The highest Aiken index is 1,000 and the lowest is 0.963. Item with index more than 0.88 is acceptable. The result of the analysis indicates that total Aiken index from 40 item questions is more than 0.88. It can be concluded that the multiple representation instrument fullfills the theoretical validity and can be used for empirical validity testing ²¹.

The next step is empirical validation. The first phase is to do sample adequacy test. The result of KMO-MSA test analysis is 0.694 and Bartlett Significance Test is 0.00. It concludes that the sample size of test is eligible for further analysis. The second phase is unidimensional analysis. Unidimension is defined as the presence of a latent underlying data trait. The result shows that the instrument is only able to measure one representation of multiple capabilities ²⁴. The third phase is checking item fit. The results is 3 unfit items are number 9, 16 and 30 from total 40 items after analysis process using Rasch model PCM 1-PL. The unfit items are removed for the misconception of students on the items, leaving 37 items to measure the ability of multiple representations of students. The fourth phase is measure item and person reliability. The Alpha Cronbach value of this study is 0.86. It shows that the category of reliability in the test instrument to measure the multiple representation capabilities of students is categorized as Good. Item and person reliability showed that It can be concluded that both consistency of respondents answers and quality of items in special instruments are good. The fifth phase is analysis difficulty level of items. The analysis results of the difficulty level of items in the instrument that measures students’ ability of multiple representations shows that there are 10 easy items, 20 medium items, and 10 difficult items. The most difficult item is number 37 and the most easy item is number 9. At moderate ability level, the information from the measurement is very high. The conclusion shows that well arranged items determine the students' ability level only ²⁴.

5. Conclusions

The instrument of assessment must be reliable and valid to get credible measurement results. Thus, the reliability and validity of each assessment instrument that is used to measure the results of the study should be analyzed. Validity refers to how accurate a study answers research questions or the strength of research conclusions ¹⁷. The instrument of test has fulfilled the theoretical validity performed by the expert judgement and educational practitioners who have been analyzed using the V-Aiken index. The result of the analysis shows that the test instruments have fulfilled theoretical validity and can be used for further analysis. The arranged test instruments have fulfilled the empirical validity with empirical evidence of 37 fit items from a total of 40 items that have been analyzed using PCM 1-PL. The reliability of test instruments categorized as good ²⁶. Based on the value of person reliability and item reliability, it can be concluded that the consistency of answers from respondents and quality items in a special instrument are good. It indicates that the arranged instruments can be categorized as good questions. The test instrument can be used as a tool to diagnose or to measure students’ ability of multiple representations on electrolyte and nonelectrolyte solutions. The arranged items determine the students' ability moderate level only ²⁴.

Suggestion for next research is that chemistry teachers can apply the instruments that have been prepared to determine students’ ability of multiple representations to help teachers to decide the best strategy, approach, or appropriate learning models then students understand multiple representations in learning electrolyte and nonelectrolyte solutions. This instrument can be used as an example to prepare test instruments for other chemical contents.

Acknowledgements

The authors are thankful to all teachers and students at state senior high school in Yogyakarta, many thanks for their supported on this study.

References