Determining the Appropriate Blend of Blended Learning: A Formative Research in the Context of Spada-Indonesia

Deciding appropriate blend for certain learning outcomes is very crucial in designing an effective blended learning. Blended learning is not just simply combining face-to-face learning with online learning. The challenge is to determine the most appropriate blended learning strategies to achieve a series of desired learning outcomes. Blended learning designers utilize a model of criteria as a guideline to answer to this challenge. This research was aimed to develop a proposed model of criteria for determining the appropriate blended learning strategy. Hence, the research questions become: 1) Is the model created fitting a conceptual framework? and 2) Is the model created effective and implementable? Consequently, formative evaluation techniques were employed to answer these research questions, i.e. experts review, one-on-one evaluation, and field test. These methods were used to collect descriptive and formative data. This study was followed by 7 instructional designers and e-learning experts as well as 235 lecturers as respondents. Upon completing a series of formative evaluations, result of this study showed that the model created is conceptually fitting, effective and implementable as a lecturer’s guide in determining the appropriate blended learning strategy.


Introduction
E-learning has been popular since the 1980s [1]. It is an umbrella for the terms that refer to the same thing, such as stand-alone course, virtual classroom, embedded learning, blended learning, et cetera [2]. E-learning has tremendous potential and promises for education. It allows people to share information and learn widely and collaboratively without being blocked by space (territories) and time [3]. One of e-learning modes broadly used in higher education nowadays is blended learning.
Generally, blended learning can be viewed as the combination of face-to-face learning with online learning [4]. But, it is actually more than just combining face-to-face learning with online learning. Blended learning is essentially the combination of the best elements of face-to-face learning with the best elements of online learning [5]. It is the combination of positive aspects of both classroom learning and e-learning environment.
Consequently, finding and determining the best aspect among blended learning dimensions is the most essential block in building blended learning design. One of the most important questions to be considered is "how to blend? [4]. Further, Su et. al. also mentioned that instructors often encounter difficulties in this issue. It is not only caused by their lack of knowledge and skills but also caused by the mindset they have formed for many years in face-to-face learning [6]. In other words, blended learning designers may ask questions as such: "How to determine the right blend for certain learning outcomes? What criteria can be used to pick the right blend of it?"

Context of Study and Research Questions
This study was conducted at the Ministry of Research, Technology and Higher Education (MoRTHE), Republic of Indonesia. In response to the development of advanced technology and its application influence in education, especially application of e-learning in higher education, MoRTHE launched a pilot project called SPADA-Indonesia. SPADA stands for Sistem Pembelajaran Daring, which translates to Online Learning System. SPADA-Indonesia is a program aimed to improve the access of students to high quality of education through the application of blended/hybrid learning. Lecturers from certain universities in Indonesia are encouraged to offer blended learning courses. The courses can be taken by students from other universities as a credit earning method/transfer program. This method allows the credits students have earned through blended learning courses to be transferred into their own program at their own university. Until now, 51 universities have offered 395 blended learning courses, in which 6,927 students from 116 different universities participated.
Preliminary study of SPADA-Indonesia, followed by 50 lecturers as respondents, showed that most of them (84%) have no sufficient knowledge in blended learning. Most of them (74%) also have no sufficient knowledge on how to determine the appropriate blended learning activities. But, they have to be able to determine it without accompanied by a professional instructional system designer. Whereas, on the other hand, deep understanding of the characteristics of blended learning and the ability to determine the appropriate blend of it are very important in designing effective blended learning. Questions as such: "How to determine the appropriate blend for certain learning outcomes? What criteria can be used to pick the appropriate blend of it?" are relevant to the context of this study.
To solve the gap mentioned above, lecturers as blended learning designers in SPADA-Indonesia program context need a model of criteria as a guideline in deciding the most appropriate blend for particular desired learning outcomes. This study is intended to design a proposed model that can serve as a guideline for lecturers in SPADA-Indonesia program context. The research questions that this study hopes to be able to shed answers on include but are not limited to: 1. What model is both useful as a lecturer's guide in determining the appropriate blended learning strategy and suitable in SPADA-Indonesia program context? 2. Is the model created feasible enough to be used as a lecturer's guide in determining the appropriate blended learning strategy? 3. Is the model developed implementable enough to be used as a lecturer's guide in determining the appropriate blended learning strategy? Learning occurs in two kinds of learning setting, i.e. synchronous and asynchronous learning setting [7]; and online learning and blended learning are often assumed to be synonymous with asynchronous teaching and learning [8]. Therefore, the model created will be viewed from the perspectives of learning setting mentioned above. To the author's knowledge, no previous study related to this issue has been conducted. The results of this study are expected to constructively contribute to the field of instructional design, especially for blended learning design process at a micro level.

The Importance of Instructional Design Model
Design is actually a necessity. No activities or instructional activities without prior planning. Instructional design is actually planning activities. Instructional design is a necessary planning process without which any instructional activities can be completed. It is a careful planning prior to development [9]. An effective instructional system depends heavily on a good instructional design. Poor instructional design will lead to the poor result of blended learning process itself [10].
Instructional designers need a certain model as a guidance to make a good instructional design. Model is defined as a pattern which then can be used as a reference. It is a simple representation of a complex form, process, or idea [11]. A model should be able to show its element and structure and systematic working procedure [12]. It can also be represented in many forms, such as graphics, picture, diagram, and even descriptive representation, and interrelated elements that show working procedure.
Model, is also applied in the context of education, especially in the field of instructional design. Instructional design model used by instructional designers as a framework and a guideline in designing an effective instructional system.
In this study, the instructional design model needs to visualize the complex form and idea of criteria on how to determine the appropriate blended learning strategy. This context applies to the likes of the SPADA-Indonesia program. SPADA-Indonesia has yet to have a instructional design model that can be used as lecturer's guide in designing the appropriate blend of blended learning. Of course, hundreds instructional design model exist today. But, as mentioned by Bonk, the challenge is highly context dependent with a practically infinite number of possible solutions [4]. Study of Snyder recommended the needs of researchers to continue testing instructional-design theories and models in different online contexts, either build upon those theories and models or develop new ones that will provide appropriate and relevant guidance [13]. It means that the most suitable instructional design model is the one that is the most relevant to the context where the model will be applied. In this case of study, is the one that is most relevant to the context of SPADA-Indonesia.

Blended Learning
Blended learning stems from the term known as e-learning. It is a generic term used as an umbrella for any terms such as virtual learning, online learning, virtual class, mobile learning, blended learning et cetera. The authors define e-learning differently. We define e-learning as electronic technology-enabled learning. Some authors emphasize blended learning as access of learning resources anytime and anywhere [14], by using any electronic devices such as computer or mobile phones to deliver training, education or learning materials [15]. Naidu, relating e-learning with synchronous and asynchronous learning setting, defined e-learning as an educational process that utilizes information and communication technology to bridge learning and learning activities synchronously or asynchronously [16]. Horton emphasized e-learning as the application of information and computer technology to create learning experiences [2].
E-learning itself can be viewed as a continuum from adjunct to hybrid/blended and fully online learning [17]. Adjunct is a continuing traditional learning process, enhancing them or extending them beyond classroom hour with online resources particularly use computer mediated communication (CMC). Hybrid/blended has already becoming an integral part of curricula. Mixing delivery of content, CMC, or online collaboration with face-to-face session. Fully online learning encompasses all e-learning interactions which taken place online and include all learning materials being delivered online [17].
Blended learning is generally viewed as the combination of face-to-face learning with online learning. But, actually blended learning is an opportunity to integrate the advance of technology innovation that can be offered online and face-to-face [18]. As a response to the development of technology, blended learning is basically the combination the best of face-to-face learning with the best of online learning [5].
Blended learning also can be viewed from the perspective of synchronous and asynchronous learning. It is an effort to apply synchronous learning elements such as face-to-face interaction with instructors and collaborative work with peers as a complement of asynchronous learning activity of individual students [19]. It is also the combination of synchronous learning and asynchronous learning components to achieve optimum learning effectiveness [10].
Definitions mentioned above, explained how blended learning occurs in synchronous and asynchronous learning settings. Synchronous learning is a simultaneous learning activity between learner and instructor at the same time. Asynchronous learning is learning process between learner and learning resources anytime and anywhere [7].
Synchronous learning can be divided into two types. First, learning activities that occur between the learners and instructor/tutor at the same time and place are called synchronous physical format. Secondly, a learning process where learners and tutors/instructors are able to learn at the same time but at different places are called virtual synchronous [20]. Asynchronous learning can be divided into two types. Firstly, collaborative asynchronous learning, a learning process where learners and tutor/instructors are able to learn at anytime and anywhere with others, such as learning from discussion forum, assignment, mailing list, et cetera. Secondly, selfpaced directed asynchronous learning, a learning process where learners and tutor/instructors are able to learn at anytime and anywhere, such as reading online articles, watching video, slide presentation, et cetera [17].

The Importance of the Appropriate Blend in Designing Blended Learning
Bonk and Graham mentioned that one of the important questions that should be put into consideration in blended learning is "how to blend? [4]. Furthermore, So and Bonk also mentioned that there are three important questions that should be raised when blending blended learning. First, how much of interaction should take place in faceto-face or online settings? Second, when should we use online versus face-to-face interaction? Third, how can we integrate the two modes of interaction to increase learning experiences [21]?
Recent studies also showed the same recommendations. Designing course in a blended format is not easy. Precise plan or design in blended learning is very crucial [22]. So, for blended learning designers, learning how to effectively integrate online with face-to-face instruction is essential for effective blended learning [23]. They need to be aware of the limitation of blended learning [24] as well as its strength. Stacey and Gerbic also recommended the same thing. They mentioned that combination of blended learning should be based on an understanding of the strengths and weaknesses of each environment as well as the appropriateness of choice to the learners involved [25]. In this case, instructors/lecturers played a pivotal role in recognizing the potential and limitations of each learning medium, face-to-face and online learning environment [26]. Lecturers can reduce transactional distance by selecting the right instructional technology that will enhance interaction [27].
Instructional design, as well as other components, as such knowledge domain, conceptual learning theory, user interface design, and any related learning environment, is very important to make a successful e-learning [28]. That is why, in blended learning, interaction can be optimized by using relevant synchronous and asynchronous technology.
The International Association for K-12 Online Learning has constructed a blended learning teacher competency standard. It was stated in competency 3, management of blended learning experience, there are two standards related to it. Standard A is able to understand and manage the face-to-face and online components of lesson planning. Standard B is able to provide balanced opportunities for students to participate in synchronous and asynchronous modalities [29].
Recent studies above indicated the importance of the appropriate blend in blended learning design. Instructors or lecturers as blended learning designer need to know and able to pick the appropriate blend accordingly. Since, certain contexts and circumstances need different approach and model, this study is intended to create a model that relevant to the context of SPADA-Indonesia program.

Methodology
The purpose of this study is to create a design theory, which is a model of criteria for determining the appropriate blended learning strategy in the context of SPADA-Indonesia. The research method that is deemed as suitable for this purpose is formative research. Formative research as suggested by Reigeluth, consist of five steps as follows: 1) create a case that helps generate a design theory; 2) collect and analyze descriptive and formative data; 3) revise the instance; 4) repeat the data collection and revision cycle; and 5) fully develop tentative theory [30]. Its procedure can be described as in Figure 1 below [see Figure 1].
As can be seen in Figure 1, as the first step, researchers created a tentative model. The tentative model, then reviewed by experts to have descriptive and formative inputs. At the same time, the tentative model was tested to some respondents one by one (one-on-one evaluation) to have inputs from the perspective of users. Researchers, then, use descriptive and formative data to revise the model. The revised model, then be field-tested three times to different respondent until the model can be considered as the final model. In this case, until it is considered as effective and implementable. The experts review was completed by 7 instructional design and e-learning experts. One-on-one evaluation was done by 5 respondents who are experienced lecturers in e-learning implementation. Field tests include sample size of 45 respondents for 1 st field test, 120 respondents for 2 nd field test, and 65 respondents for the 3 rd field test.
The research instrument used in this study were questionnaires and rubric assessment. The instrument was based on the PCI (Perceived Characteristic of Innovation) that measure five dimensions of innovation characteristics: relative advantages, compatibility, complexity, observability and trial-ability of the model. [31].

The Tentative Model
The first step in formative research is creating a case to generate the design theory. In this study, the design theory created was a model of criteria for selecting and determining blended learning strategy. A design theory or a model must be based on a clear and logical theoretical foundation or framework. Therefore, for this purpose, researchers created a theoretical framework of blended learning as the underlying foundation of the developed model.
The framework that researchers created was a conceptual model of blended learning, called quadrant of blended learning setting. This conceptual model was generated from some concepts and principles of e-learning and or blended learning proposed by well-known authors such as Derek & Stockley [15], Thorne [18], Naidu [16], Horton [2], Bonk & Graham [4], Howard [19], Piskurich [10], Noirid [17], Noord [17], and Smaldino et. al. [7] as mentioned on literatures review above. The quadrant of blended learning setting can be described as in Figure 2 below [see Figure 2].  Figure 2 ilustrates blended learning setting that can be divided into four quadrants as follows: • Live Synchronous Learning (LSL); LSL is learning experience that occurs between the learner and the learning resources at the same time and place. LSL is the same as face-to-face learning, such as lecture, group discussion, lab practice, field study, etc. • Virtual Synchronous Learning (VSL); is learning experience that occurs between the learner and the learning resources at the same time, but diferent place. This learning setting can be mediated by synchronous tele-learning technologies such as audio-conference, web-based conference or videoconference. • Collaborative Asynchronous Learning (CAL); is learning experience that occurs between the learner and the learning resources at any time or place with other resource persons. CAL can be mediated by asynchronous learning tools such as discussion forum, miling list, online assignment, etc. • Self-directed Asyinchronous Learning (SAL); is learning experience that occurs between the learner and the learning resources at any time or place under their own pace and control. SAL can be facilitated by various high quality of learning objects in many forms of appropriate media, such as text, audio, visual, audio-visual, animation, and simulation. With references to the quadrant of blended learning settings described above, researchers then defined blended learning as a form of learning that combines in such a way the best potential of synchronous learning strategies with the best potential of asynchronous learning strategies to create optimum learning experiences which serve to achieve pre-determined learning outcomes.
This definition explains that designing blended learning is about combining appropriate synchronous and asynchronous learning strategies to achieve certain intended learning outcomes. Blended learning designers can combine the best of synchronous learning strategies with the best of asynchronous learning strategies. Different learning outcomes need different blended learning strategies. As consequences, blended learning designers need criteria on how to determine the appropriate blend of blended learning strategy.
Based on the conceptual model of blended learning described above and with reference to the work of Anderson [32] which combined the work of Edgar Dale's Cone Experience [32], Learning Modality [7] and revised Bloom's Objective Taxonomy (Anderson & Kratwhol, 2002), researchers created a model of criteria for determining the appropriate blended learning strategy, as described below [see Figure 3]. Figure 3 illustrates that there are several considerations in determining the appropriate blended learning setting, which include: • Does it require real and direct purpose experience to achieve a particular pre-determined learning outocome? If the answer is yes, then this can be achieved through live synchronous learning (LSL) activities.
• If the answer is no, the next question becomes: Does it require active participation and role-play situation? If the answer is yes, then it can be achieved through virtual synchronous learning (VSL) activities.
• If the answer for the first question is no, then it can be achieved by asynchronous learning activities, both self-directed asynchronous learning (SAL) and collaborative asynchronous learning (CAL).  This process can be illustrated as in table below [see Table 1].
Based on the conceptual framework mentioned above, blended learning designers may choose relevant synchronous and asynchronous learning activities, as illustrated below [see Table 2].

Validation and Revision of the Tentative Model
The second up to the fifth step of formative research are parts of an iterative process of formative evaluation designed to validate and revise the generated tentative theory or model. This study employed a series of formative evaluation, i.e. experts review, one-on-one evaluation and three times of field tests to obtain descriptive and formative data.
Related to the quadrant of blended learning and its definition, all experts stated that the model was clear, logical, relevant to the concept and principles of elearning, in alignment with sufficient underlying theories, and implementable as a conceptual model or framework. But, experts recommended researchers to distinguish more clearly the differences between live synchronous learning and face-to-face learning as well as between self-directed asynchronous learning and collaborative asynchronous learning. Consequently, researchers revised this framework by adding more detailed explanations and examples of learning activites for live synchronous learning (LSL), virtual synchronous learning (VSL), self-directed asynchronous learning (SAL), and collaborative asynchronous learning (CAL).
In relation to the model of criteria for determining blended learning strategy, all experts stated that the model was compatible with the needs of blended learning system design, clear, logical, and supported by sufficient underlying theories, and feasible as conceptual model or framework. Five out of seven experts stated that this guideline should be supported by more varied examples.
Rubric assessment results on one-on-one evaluation, also showed the same indication. Eighty percent of respondents were not able to select and determine the appropriate learnring setting. To respond to this outcome, researchers revised the model by providing adequate and varied examples with diferent case of predetermined learning outcomes.
After having revised the tentative model based on experts review and one-on-one evaluation formative data, the next formative steps was to conduct field tests. Field tests were conducted in three stages.
In the field test, respondents were given explanations and practices on how to determine the appropriate blended learning strategy based on the revised tentative model explained above. Then, respondents were asked to design a blended learning strategy by adhering to the model. Rubric assessment was used to measure their ability in determining the appropriate blended learning strategy. After that, they were asked to fill the questionnaires to collect their opinionson the model developed.
The results of rubric assessment on the 1 st , 2 nd and 3 rd field test can be described as in the Table 3 belows [see Table 3]:  Table 3 illustrates the improvement of respondent's ability in determining appropriate blended learning strategy. In the 2 nd and 3 rd field test, all respondents have moved up from Poor scale. In the 3 rd field test, most of the respondents gained excellent and good scale. This finding shows that after having three times revision, the tentative model can be considered as feasible, effective and implementable.  This study also measured the perception of respondents to the innovativeness of the model. It was measured by using Rogers's Perceived Characteristics of Innovation (PCI), i.e. relative advantages, compatibility, complexity, obesrvability and trialability of the model.
The results of the questionnaires distributed on the third field test showed that the model has many relative advantages. It can, be described as in the Figure 4.
Trend line on Figure 4 illustrates the relative advantages of the model. Most of respondents claimed they strongly agree and agree that the model is systematic and logic, interactive, appealing and easy to follow.
The results of the questionnaires distributed on the 3 rd field test can be seen below [see Figure 5]: Figure 5 illustrates that: 1) the model is compatible with the needs of blended learning design; 2) the model is simple and easy (user-friendly) to follow; 3) the results of design by using the model can be observed clearly; and 4) the model can be tested easily to other peer instructors.

Conclusion
After having a series of formative evaluation, i.e. experts review, one-on-one evaluations, and three stages of field tests, this study can be cluded as follows: • Experts review results showed that the model of criteria for determining appropriate blended learning strategy was clear, logical, systematic, supported by sufficient underlying theories, and implementable enough as a conceptual model or framework. • Rubric assessment results showed that after having three times revision on the three stages of field test, the model can be considered as feasible, effective and implementable. • Questionnaire results showed that the model has high characteristics of innovation, i.e.: 1) it has relative advantages (such as easy to use, logical and systematic, interactive and appealing); 2) it was compatible with the needs of blended learning design; 3) it was simple and easy to be followed (user-friendly); 4) the results of design by using the model can be observed clearly; and 5) the model can be tested easily to other peer instructors. • Finally, it can be concluded that that the model of criteria for determining appropriate blended learning strategy was feasible, effective and implementable in the context of SPADA Indonesia. The study is not without its limitations. The context is very specific in such that the model is unable to be easily implemented in other contexts. The model viewed only from the perspective of learning setting, i.e. synchronous and asynchronous learning [7]. The model also viewed only from the perspective of time and space [4]. The model, perhaps, can only be applied for designing blended learning at activity and or course level [4]. Further studies should be conducted to apply and evaluate the model formatively and widely in the same context with different participants.