Article Versions
Export Article
Cite this article
  • Normal Style
  • MLA Style
  • APA Style
  • Chicago Style
Research Article
Open Access Peer-reviewed

Identification of Criteria for Assessing the Quality of Research

Anjana , Rashmi Choudhuri
American Journal of Educational Research. 2018, 6(6), 592-595. DOI: 10.12691/education-6-6-2
Published online: May 24, 2018

Abstract

The purpose of this paper is to describe the process of identification of criteria which measure the quality of research. Assessment of the quality of research is an important issue in the academic community. Time to time some criteria of quality of research have been given by different authors but an adequate or appropriate existing scale to assess the quality of research is still lacking. This paper presents a systematic process employed to develop a scale for assessing the quality of research. Reliability and validity of the tool have also been reported.

1. Introduction

In today’s globalised world the quality of research is the highest priority for any academic community. The term quality refers to a characteristic or feature that someone or something has something that can be noticed as a part of a person or thing (Merriam-Webster, 2000). The quality research most commonly refers to the scientific process encompassing all aspects of study design; in particular, it pertains to the judgement regarding the match between the methods and questions, selection of subjects, measurement of outcomes, protection against systematic bias, non-systematic bias, and inferential error 1, 2, 3. Quality of research has always been at the centre of debates in both academics and professionals. These debates are prevalent in the multi-disciplinary fields of health, education, disability and social welfare 3. The debate is also due, in part, to the lack of consensus on the specific standards for assessing the quality of research. Thus, the concern for criteria of assessing the quality of research is an important issue and educational researchers are often worried because they feel that their responsibility is towards teachers and students and because they see many problems in educational process. Quality of research provides a rich source of information to aid policy makers to gain a better understanding of today’s most pressing and complex social and scientific issues such as making in new educational policies, educational rights, cultural diversity, human rights, disease prevention, and climate change. The benefits of the research cannot be realized unless the knowledge generated is both accurate and trustworthy.

1.1. Need for Identification of Indicators

A fruitful research is equally dependent on tools as it is dependent on other things like planning and procedure followed. For collecting data successfully for a research work suitability is of much importance. The present work is related to the perspectives of teachers and research scholars on criteria of quality of research. As no such unanimous criterion was available, the researcher realized the need of identifying criteria of quality of research and constructing rating scale to assess the perspectives of teachers and research scholars on criteria of quality of research work.

1.2. Review of Literature

The identification process was begun with the in-depth review of literature. The review of online articles, research papers and reports published from 2004 was done in the year 2015. Online articles (23) studied were research papers (9) and reports (3). Various available online articles were studied to collect the criteria of assessing the quality of research.

1.3. Enlisting of Parameters and Preparing the First Draft

After reviewing the article six criteria of quality of research were enlisted which are: (a) Journal Impact Factor, (b) Citations, (c) Ranking of Journal, (d) Peer Review, (e) Dialogue and Discourse and (f) Standardised Research Reporting Framework. Every criterion was clearly described after perusal of the literature. The first draft of the tool comprising of six criteria and their description was prepared carefully.

1.4. Expert Opinion and Editing the First Draft

The first draft was given to 20 experts (University teachers) of different disciplines such as Commerce, Education, Economics, Sociology, Psychology, Botany, Chemistry, Geography, Mathematics, Statistics of Banaras Hindu University and Mahatma Gandhi Kashi Vidyapith, in Varanasi, a prominent city in India. The experts were contacted in their departments after permission from the Head of department (HOD). The first draft of the list of criteria was given to them personally. Experts were contacted from 15 November 2015 to 26 November 2015. They were to judge the ‘clarity, appropriateness and relevance’ of each criterion of quality of research and suggest any other criterion for quality of research. After expert suggestions, two new criteria — ‘Journal in first quartile of its area’ and ‘Criteria of selection of expert referee in the field by the editorial board’—were included. Thus, the total criteria became eight in number.

1.5. Identification by University Teachers

Once the list of criteria was finalized, the next major step was that of identifying the useful criteria for assessment of quality of research. Teachers of university were approached for identification of useful criteria for assessing the research quality. Fifty-five teachers from Mahamata Gandhi Kashi Vidyapith and Banaras Hindu University in Varanasi city were approached. These teachers were from Assistant Professor grade (23), Associate Professor grade (18) and Professor Grade (14). All the criteria earlier enlisted were identified as important to less important. These are as follows in the sequential order of their importance/usefulness.

1. Citations

2. Journal Impact Factor

3. Peer Review

4. Criteria of selection of expert referee in the field by the editorial board

5. Standardised Research Reporting Framework

6. Ranking of Journal

7. Dialogue and Discourse

8. Journal in first quartile of its area.

A brief description of the final 8 indicators of quality of research is given below:

a) Citations- A quotation from or reference to a book, paper, or author, especially in a scholarly work (Oxford Dictionary of English).

b) Journal Impact Factor- This is to measure the frequency with which the average article in a journal has been cited in a particular year (Journal Citation Reports- JCR).

c) Peer Review- Evaluation of scientific, academic, or professional work by other researchers working in the same field (Oxford Dictionary of English).

d) Criteria of selection of expert referee in the field by the editorial board- Criteria of selection of expert referee refers to those selection criteria that are used by the editorial board for the selection of expert referee.

e) Standardised Research Reporting Framework- It is essential research information regarding sample and sampling technique, statistics, randomization, analysis and interpretation (National Centre for the Dissemination of Disability Research-NCDDR).

f) Ranking of Journal- Journal Rank is a measure of scientific influence of scholarly journals that account for both the number of citations received by a journal and the importance or prestige of the journals where such citations come from (Tripathy and Tripathy).

g) Dialogue and Discourse - An exchange of ideas or opinions on a piece of published research work between a community of scholars, each critiquing others’ work—corroborating and affirming or challenging and refuting and developing new propositions.

h) Journal in first quartile of its area- Journal in first quartile of its area refers to top 25 percent journals in a given subject category which has highest impact (Computer Vision Group).

The two indicators, Journal in first quartile of its area and Criteria of selection of expert referee in the field by the editorial board, emerged from the discussion held with teachers during acquisition of expert opinion.


1.5.1. Final Draft

In the final draft of the scale, demographical variables were also decided and added in the rating scale as background information about university teachers. The demographical variables included for teachers were: Name, Sex, Age, Designation, Educational qualification, Subject/Area of research, University and Department.


1.5.2. Scoring

Scores were assigned to every indicator in the following way: very important = 5, important = 4, undecided= 3, less important = 2, not important = 1 according to teachers and research scholars ratings on scale for Research Quality Criteria Scale.


1.5.3. Final Tryout of the Scale

After identification of criteria, final tryout was done on 30 teachers (Commerce, Education, Economics, History, Political Science, Psychology, Botany, Physics, Chemistry, Geography, Mathematics, Statistics) and 30 research scholars (Commerce, Education, Economics, History, Political Science, Psychology, Botany, Physics, Chemistry, Geography, Mathematics, Statistics) of the Banaras Hindu University. Final tryout was done in the month of January, 2016 and it took 24 days. Scores for various responses were added up to find a composite score.


1.5.4. Item Analysis

In order to study the degree of relatedness of one indicator of the test with the other seven criteria, the coefficient of the correlations among the scores on eight criteria were calculated. Inter-correlation among criteria has been calculated with the help of Pearson’s Product Moment coefficient of correlation(r). Table 1 shows the 8x8 correlation matrix, which shows the degree of relationship among eight criteria.

The coefficient of correlations(r) for maximum criteria obtained the values less than 0.50. It reflected orthogonality which means that there was no overlapping among the criteria. Thus, the obtained values were found to be satisfactory.


1.5.5. Reliability

Reliability refers to the consistency of measurement, i.e. as to how consistent test scores or other evaluation results are from one measurement to another 4. The reliability of the scale was calculated by the test-retest method. For determining the coefficient of test-retest reliability of the RQCS, the scores of 40 teachers and 40 research scholars obtained on the final 8 items were correlated with their own scores when they responded again after 20 days.

The reliability value of RQCS was found to be 0.87, which shows the scale as being homogeneous measures whatever it was meant to measure fairly in a reliable way.

To ensure internal consistency of the tool, reliability was calculated using Cronbach Alpha method which was found out to be 0.75.


1.5.6. Validity

Content validity of the tool was established by getting the tool evaluated by experienced Professors and professionally qualified faculty members of different universities including Mahatma Gandhi Kashi Vidyapith, Banaras Hindu University and Lucknow University.


1.5.7. Implications

• The present tool is uniformly useful for education, science and humanities researches.

• It will function as a guideline for researchers and teachers to conduct new researches in their field.

• It is helpful for researchers and teachers in conducting worthwhile researches.

• It is helpful in improving quality of research in terms of publication output.

• It is useful in assessing the quality of researches.

• It provides systematic steps for researchers and teachers in developing tool.


1.5.8. Limitations

• It is only applicable in disseminated result of research which is published in researchers’ publication.

• The result of this tool varies in different subjects.

2. Conclusion

Researcher has provided a systematic process for the identification of criteria which measure quality of research which is helpful for researchers of other disciplines. As the quantity of researches increases, we need to ensure the quality of research. Good research begins with good measurement. The scale developed after identification of criteria of quality of research is an example of systematic steps of scale development. Poor scale construction raises question on reliability and validity of the research results, no matter how careful the design of the study has been employed. In contrast, carefully constructed measures help to advance our understanding and ensure that the study will provide accurate and usable data. By using suggested steps, a researcher is more likely to create scales that will provide information and enhance the future of research.

References

[1]  Boaz, A., and Ashby, D. (2003). Fit for purpose? Assessing research quality for evidence based policy and practice. London: ESRC UK Centre for Evidence Based Policy and Practice. Retrieved from https://www.kcl.ac.uk/sspp/departments/politicaleconomy/research/cep/.../wp11.
In article      View Article
 
[2]  Lohr, K. N. (2004). Rating the strength of scientific evidence: Relevance for quality improvement programs. International Journal for Quality in Health Care, 16(1), 9-18.
In article      View Article  PubMed
 
[3]  Shavelson, R. J., and Towne, L. (Eds.). (2002). Scientific research in education. Washington, DC: National Research Council, National Academy Press.
In article      
 
[4]  Gronlund, N.E. (1976). Measurement and evaluation in teaching. Macmillan Publishing Company, New York.
In article      
 
[5]  Feuer, M. and Towne, L. (2002). The logic and the basic principles of scientific based research. Presentation given at a seminar on the use of scientifically based research in education, Washington, DC. Retrieved from http://www.ed.gov/nclb/methods/whatworks/research/page_pg11.html.
In article      View Article
 
[6]  Gersten, R. Baker, S., and Lloyd, J. W. (2000). Designing high-quality research in special education: group experimental design. The Journal of Special Education. 34(1), 2-18.
In article      View Article
 
[7]  Greenhalgh, T. (1997). How to read a paper: Assessing the methodological quality of published papers. British Medical Journal. 315, 305-308.
In article      View Article  PubMed
 
[8]  Journal Citation Reports (JCR). (1975). Philadelphia: Institute for Scientific Information retrieved from https://www.thomsonreuters.com.
In article      View Article
 
[9]  NCDDR. (2003). What Are the Standards for Quality Research?. The Research Exchange. 8(2), 16.
In article      
 
[10]  Ragin, C. C., Nagel, J. and White, P. (2003, July). Workshop on scientific foundations of qualitative research. Arlington, VA: National Science Foundation.
In article      
 
[11]  Stevenson, A. (2015). Oxford Dictionary of English. Oxford University Press.
In article      
 
[12]  Tripathy, P. and Tripathy, P.K. (2015). Fundamentals of Research. A Dissective View. Anchor Academic Publishing.
In article      
 

Published with license by Science and Education Publishing, Copyright © 2018 Anjana and Rashmi Choudhuri

Creative CommonsThis work is licensed under a Creative Commons Attribution 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Cite this article:

Normal Style
Anjana, Rashmi Choudhuri. Identification of Criteria for Assessing the Quality of Research. American Journal of Educational Research. Vol. 6, No. 6, 2018, pp 592-595. http://pubs.sciepub.com/education/6/6/2
MLA Style
Anjana, and Rashmi Choudhuri. "Identification of Criteria for Assessing the Quality of Research." American Journal of Educational Research 6.6 (2018): 592-595.
APA Style
Anjana, & Choudhuri, R. (2018). Identification of Criteria for Assessing the Quality of Research. American Journal of Educational Research, 6(6), 592-595.
Chicago Style
Anjana, and Rashmi Choudhuri. "Identification of Criteria for Assessing the Quality of Research." American Journal of Educational Research 6, no. 6 (2018): 592-595.
Share
[1]  Boaz, A., and Ashby, D. (2003). Fit for purpose? Assessing research quality for evidence based policy and practice. London: ESRC UK Centre for Evidence Based Policy and Practice. Retrieved from https://www.kcl.ac.uk/sspp/departments/politicaleconomy/research/cep/.../wp11.
In article      View Article
 
[2]  Lohr, K. N. (2004). Rating the strength of scientific evidence: Relevance for quality improvement programs. International Journal for Quality in Health Care, 16(1), 9-18.
In article      View Article  PubMed
 
[3]  Shavelson, R. J., and Towne, L. (Eds.). (2002). Scientific research in education. Washington, DC: National Research Council, National Academy Press.
In article      
 
[4]  Gronlund, N.E. (1976). Measurement and evaluation in teaching. Macmillan Publishing Company, New York.
In article      
 
[5]  Feuer, M. and Towne, L. (2002). The logic and the basic principles of scientific based research. Presentation given at a seminar on the use of scientifically based research in education, Washington, DC. Retrieved from http://www.ed.gov/nclb/methods/whatworks/research/page_pg11.html.
In article      View Article
 
[6]  Gersten, R. Baker, S., and Lloyd, J. W. (2000). Designing high-quality research in special education: group experimental design. The Journal of Special Education. 34(1), 2-18.
In article      View Article
 
[7]  Greenhalgh, T. (1997). How to read a paper: Assessing the methodological quality of published papers. British Medical Journal. 315, 305-308.
In article      View Article  PubMed
 
[8]  Journal Citation Reports (JCR). (1975). Philadelphia: Institute for Scientific Information retrieved from https://www.thomsonreuters.com.
In article      View Article
 
[9]  NCDDR. (2003). What Are the Standards for Quality Research?. The Research Exchange. 8(2), 16.
In article      
 
[10]  Ragin, C. C., Nagel, J. and White, P. (2003, July). Workshop on scientific foundations of qualitative research. Arlington, VA: National Science Foundation.
In article      
 
[11]  Stevenson, A. (2015). Oxford Dictionary of English. Oxford University Press.
In article      
 
[12]  Tripathy, P. and Tripathy, P.K. (2015). Fundamentals of Research. A Dissective View. Anchor Academic Publishing.
In article