Abstract

In this article, a digital teaching format tailored to the teaching in an introductory course in Statistics for Industrial Engineers is compared with the classic face-to-face lecture from previous years. The focus is on the comparison of the examination results of the classic lecture in the four academic years before the COVID-19 pandemic with the results of the online course of the academic year 2020/2021 − with the surprising result for the authors that the examinees performed significantly better under the new conditions than in the past. The discussion of the empirical results includes potential for improvement for future face-to-face, online and hybrid courses.

1. Introduction

In the spring of 2020, a strong earthquake went through the international university landscape. The spread of the COVID-19 pandemic has posed major challenges to the education system ever since. In the hurry, it was necessary to digitize the face-to-face courses with as little loss of quality as possible in order to be able to guarantee teaching as early as the summer term 2020. This article reports on the experiences of the two authors regarding an introductory course in Statistics, which is offered once a year for prospective industrial engineers in a bachelor's degree program at the Cologne university of technology, arts and sciences (CUT). Before the outbreak of the pandemic, lectures and exercises were held exclusively in presence. Until then, the predominant learning platform ILIAS was mainly used to provide materials such as the lecture script, presentation slides and exercise sheets.

The didactic requirements for university teaching have increased continuously in recent decades in the course of the gradual simplification of study access conditions. Due to a lack of an adequate university entrance qualification, many students have problems both in the individual organisation of their studies (despite rigid requirements in the study structure since the implementation of the Bologna reform in germany) and in the self-taught learning of specialist knowledge. The Bachelor's degree programme, whose curriculum includes the compulsory module Statistics in the third semester, is a good example of this development. Due to the thinning out of mathematics at schools, a considerable proportion of students does not possess the basic mathematical knowledge necessary for aspiring engineers. This is not only required to master the course Statistics, but also for other modules of the study such as technical mechanics, physics, cost and investment accounting. Without this knowledge, higher learning objectives become unattainable. In addition, in many cases, the students concerned are not aware of this circumstance due to a lack of realistic classification of mathematical abilities. The expectation is therefore to score a good grade in the module examination, as they showed good or very good mathematics performance in school before.

2. Sketch of the Learning Objectives and Contents

The course aims to provide the required basic knowledge on probability calculus and descriptive statistics: students are enabled to work independently on special questions from practice, e.g. from the areas of production technology or quality management, by means of statistical data analysis in further courses of their studies and later professional practice. The module descriptions of the degree programme are essentially based on the taxonomy of learning objectives in the cognitive area according to ¹. The focus is on learning objectives such as thinking, knowledge and problem solving. These are categorized in six stages in such an ascending way that each stage includes the previous stages – at least in theory. The Statistics module discussed here is assigned to the fourth level of taxonomy, what is called 'Analysis', and thus implicitly including the three underlying levels 'Knowledge', 'Understanding' and 'Applying'. The subject of achieving these levels are the usual basics of descriptive statistics and probability calculation being outlined subsequently.

The content is adapted to a four-hour course, with lectures and accompanying exercise lessons each taking up two hours per week. In addition to the processing of weekly exercise sheets, an introduction to the R programming environment is integrated into the exercise lessons. At the beginning, the different phases of a statistical survey are explained on the basis of the well-known life cycle of research data, together with typical graphical presentation of aggregated using small examples. This is followed by the concepts of discrete and continuous random variables including possible scale types, absolute and relative frequency distributions, location and scale parameters, various measures of correlation, and a brief introduction to regression and variance analysis.

The basics of combinatorics and probability are among the compulsory topics at secondary schools. Nevertheless, due to the heterogeneous distribution of previous mathematical knowledge among first-year students, it is necessary to repeat them before deepening them further. However, the repetition is compact due to the overall time restriction in the course. To compensate, the students receive various accompanying materials for independent catching up and consolidation. This is followed by different probability concepts with a special focus on the axioms of Kolmogoroff, the concept of stochastic independence combined with the usual theorems on conditional probability, parameters of random variables as well as prominent discrete and continuous probability distributions including their characteristic parameters. Finally, some theorems of reproduction and asymptotical behaviour are treated.

3. Classical and Digital Teaching Strategies

First, section 3.1 summarizes the similarities between the two teaching formats. Subsequently, it is explained how the two formats of classical classroom teaching (section 3.2) and digital teaching (section 3.3) were designed by the authors.

The course is offered annually in the winter term and is designed for students of the third semester. In addition to a rough sketch of the teaching content, the desired learning objectives and examination modalities are named right at the beginning of the course. Students are recommended to deal intensively with the teaching content during the semester and to become aware of their personal responsibility for achieving the learning objectives. The courses and accompanying materials are primarily to be understood as support in pursuing the desired objectives. On the university's internal ILIAS portal there exists a folder providing information on the organization of the course and accompanying materials, such as selected literature and current exercise sheets. These materials also include detailed mathematical derivations for some theorems as well as additional exercises together with the corresponding solutions. In addition, an open discussion forum for the event is offered, which can be structured by the students themselves according to freely selectable topics around the course. Experience has shown, however, that the forum is used to clarify organisational rather than substantive issues.

The final examination will be taken in the form of a written exam in attendance during the non-teaching period, whereby the framework conditions didn‘t change during the COVID-19 pandemic compared to previous academic years.

The weekly courses consist of lectures and exercises in equal parts. The lecture-accompanying materials are made available to students via the ILIAS platform described above. In addition, students can attend individual consultation hours with the teaching stuff.

The two-hour lectures take place in the lecture hall for around 150 students. As a rule, a PowerPoint presentation and the chalkboard are used simultaneously. On the one hand, the latter serves to demonstrate mathematical proofs and to develop in detail the content, which is usually presented very compactly via PowerPoint. On the other hand, the chalkboard makes it possible to respond spontaneously to student questions and to sketch suitable examples. The two-hour exercises are offered in four groups on different dates. This makes it possible for individuals to participate in the exercises several times. Every week a new exercise sheet, which was distributed in the previous week, is discussed.

In addition, a voluntary revision course is offered in the following summer term. In groups of a maximum of fifteen students, individual supervision is possible, which allows support for weaker students, for example, in the manual solution of mathematical equations.

The contents are dealt with in explanatory videos with a duration of 15 to a maximum of 40 minutes each. To better motivate the students, the content is broken down into smaller units. The solutions of the exercises are also recorded for each exercise sheet, but for didactic reasons they are provided a couple of days later. The contents of the explanatory videos are not exhaustive, especially concerning more complex contents, but rather intended as an introduction to the current chapters. In it, the statistical terms and methods are reduced to their core idea in order to enable the students to master more in-depth literature afterwards and to follow it better in the weekly online courses offered. If, for example, the various definitions of a mean (arithmetic, geometric and harmonic, median, etc.) are presented in the explanatory video, their differences and meaningful applications can be discussed in the online course by practical examples. A video on linear single regression can be deepened, for example, by the general linear model including the derivation of the normal equational system.

During the online courses, presentation slides are used and manually supplemented by a pen tablet as a chalkboard replacement. There is also some interaction with the students here, but with clear compromises as compared to the classic face-to-face courses. In the opinion of the authors, even a virtuoso use of modern media cannot replace equivalently personal eye contact between teaching staff and auditorium and a lecture hall equipped with good acoustics, chalkboard and so on. During the exercise lessons, students have the opportunity to address concrete comprehension problems when working on the weekly exercise sheets. Since there are complete solutions and explanatory videos for the exercise sheets, their 'slavish' processing is not necessary in contrast to the classical classroom course. Instead, the available time is better used to discuss additional tasks to the current content.

4. Analysis of the Learning Success

In the following, the question is investigated whether the change of teaching strategy forced by the COVID-19 pandemic had an influence on the achievement of the targeted learning objectives. For a serious comparison of the two teaching formats, it is necessary that the participants of both formats were at about the same level of prior mathematical and statistical education at the beginning of the course. Some considerations in this regard are given in Section 4.1. The subsequent section 4.2 then compares some characteristics on both teaching formats based on the examination results, i.e. on the points achieved by the participants in their exams. The differences of the characteristics in both cases are then checked for significance. It is assumed that the learning objectives are adequately checked at the end of the semester. According to the authors, this urgent need is dealt with under the concept of constructive alignment. For a serious comparison of the results, it is also essential that the examination requirements in both teaching formats are identical. This assumption is verified by the taxonomization of learning objectives in the cognitive field according to ¹ in section 4.3 using a χ2 adaptation test. Subsequently, the observable shifts in the achievement of learning objectives regarding both teaching formats are discussed.

The assumption of comparable prior mathematical knowledge of students from different academic years is very difficult to confirm. At least, there is nothing to indicate a large difference between the two groups in terms of prior knowledge. The admission criteria for the course have not changed during the past years. To take the exam, it has always been required that the mathematics exam from the first semester of study has been passed successfully. In the annual student teaching evaluations of the Statistics course, conducted in the middle of the current semester, the following question was asked: "Do you possess the technical skills to follow the course well and to participate actively?" The student response to this question (with three options ‚Yes‘, ‚Partially‘ and ‚No‘) is shown in Table 1 below. The table also includes the student-reported average weekly time spent (in brackets the standard deviation) on pre- and postprocessing the course.

Table 1 also gives no reason to assume different subject prior knowledge in different academic years (AY).

The individual data used for analysis have been conscientiously anonymized in accordance with the applicable general data protection regulation. Here, removing of personal characteristics such as name, semester of study, and so on etc. as well as replacing of the matriculation number by a sequential number (what is called pseudonymization) were sufficient for the purpose of protecting confidential individual information. Examination scores are subsequently considered in terms of the points achieved by students in the examination, rather than the grades awarded. A total of n = 798 observations from five consecutive academic years are available, with annual cohorts varying between 140 and 180 participants.

In the following, we measure the learning success on the basis of a target variable Y describing the number of points achieved in the exam as described above. The value range of this variable is determined by a (for practical reasons considered as continuous) scale from 0 to 100, describing the achieved percentage of points. The differences in the examination results in relation to the classic face-to-face semester on the one hand, and the digital semester on the other, can be seen on the basis of the kernel density estimate in Figure 1.

In determining the two core density functions, Gaussian kernel functions with adaptive bandwidth were used. The left-skewed distribution of examination results in the digital teaching format indicates an improvement in academic success compared to the classic teaching format. The empirical skewness of the distribution in the classical format, g_m^(cl) = 0.21, is observed as positive value (right-skewed). The corresponding kernel density is tilted slightly to the left in Figure 1 (solid line). This means that the majority of the exam results are in the lower and middle range of the scale and good to very good results are achieved less frequently. In the digital format, on the other hand, g_m^(dig) = - 0.39, is observed as negative value (left-skewed), which is also recognizable by the dashed line tilted to the right in Figure 1. The more robust quartile coefficients of skewness (regarding outliers) show a weaker tendency here and can also be estimated on the basis of the quartiles depicted in Figure 2, since they are calculated on the basis of the quartiles marked there. The difference between the upper and lower subsurface is set in relation to the total area of the box:

In the classical teaching format, the overall quartile coefficient of skewness is g_0,25^(kl) = 0 indicating a symmetrical rather than a right-skewed distribution. In the digital teaching format, the coefficient of g_0,25^(dig) = - 0.17 remains in the negative range and underlines the tendency towards better examination results in this teaching format.

Except for the lower quartile of the academic year 18-19 (piek of the bottom line), the results of the four classic face-to-face courses are at the same level. It is observed that the medians regarding the classic courses differ remarkably from each other with X_0.5 = 45 (academic years 2016/2017 up to 2019/2020) and X_0.5 = 57 (academic year 2020/2021). The same holds for the associated arithmetic means showing and In order to ensure the independence of the two groups, only examination participants who had not yet attempted an examination before the winter term 2020/2021 were taken into account in the calculations for the academic year 2020/2021.

The visible improvement of the learning success Y with the change of the teaching format X can now be ensured by means of a one-sided two-sample t-test. Since the standard deviations of the two groups with s₁ = 4.13 and s₂ = 4.58 are noticeably different (s₂ deviates by more than ten percent from s₁), a Welch correction is performed. (Note that the t-test is known to require identical variances in both groups. The p-value for the corresponding F-test is 0.16, thus the assumption of variance homogeneity may be doubted.) The null hypothesis is that with the introduction of the digital teaching format, there has been no improvement in study success and the measured increase in the position parameter is solely due to random fluctuations. As a result of the t-test, p < .0002 and |d| = 1.19 is reported. According to ² we can speak of a strong effect here. Accordingly, the examination results have systematically improved or, in other words, the null hypothesis can be rejected at a high level in favor of the alternative. Due to the inhomogeneity of the variances mentioned above occurring for the academic years, an analysis of variances does not appear to be meaningful. For the five years considered, the variances amount to 18.3, 17.5, 13.4, 21.9 and 21.0. For this reason, the associated p-value less than 4E-6 should be taken with caution.

Section 4.2 below examines in more detail the change in the examination results at the level of taxonomy and knowledge categories.

Various task formats were used and there were both multiple and free-text tasks. The difficulty levels of tasks varied between simple definition queries (e.g. determination of a characteristic type by marking with a cross), concrete applications and comprehension questions (e.g. the calculation of a regression function and interpretation of its coefficients) and analyses (e.g. the selection of a suitable measure of contingency and discussion of potential causal relationships).

The evaluation of the test results is based on the six taxonomy levels in the cognitive field proposed by ¹, taking into account the coarsening of the taxonomy to three levels proposed by ³. The original six stages are designed in such a way that a higher level includes the underlying stages with slight overlap. After the aforementioned coarsening or merging of levels, there are three underlying taxonomy levels as follows: T1 (recognition and repetition), T2 (comprehension and application) and T3 (analysis). In addition, the tasks are distributed among three knowledge categories or knowledge levels, namely K1 (factual knowledge), K2 (conceptual knowledge), and K3 (procedural knowledge) following the modification of Bloom's concept according to ⁴. As an additional level of knowledge, ⁴ introduces metacognitive knowledge, which can be classified as abstract and can rather be determined by self-assessment of the students. For this reasion it was not queried in the examinations considered here. When assigning tasks to taxonomy and knowledge levels, it should always be noted that - strictly speaking - these are not stages. Although the complexity of the questions tends to increase with increasing taxonomy, and the levels of knowledge also tend to move from concrete to abstract, there may occur overlapping of categories regarding both criteria. Nevertheless, this structure is suitable for systematically examining the degree of difficulty of single tasks.

In some cases, the conception allows different taxonomy and knowledge levels to be assigned to the subtasks of a particular task. This presupposes that the subtasks can be worked on independently of each other and that it is possible to clearly separate the levels of learning the examinees achieved by correct processing of a subtask. It also happens occasionally that on the way to solving some task, implicit considerations or calculations of lower taxonomy and other levels of knowledge lead at the same time to the achievement of a lower taxonomy level (measured by the taxonomy of the entire task), provided that the task has not been completed afterwards. In most cases, however, tasks are assigned uniquely to taxonomy and knowledge levels. In the present case, the majority of the tasks of a typical examination in statistical basics are assigned to taxonomy level T2 and knowledge level K2. The following is the distribution of tasks to taxonomy and knowledge levels in the classical (Table 2) and digital teaching format (Table 3), whereby the classic format was averaged over four years of study. In the calculation, all examinations in the respective academic years were considered. For each examination, the number of points achievable in a combination of taxonomy and knowledge level was determined and multiplied by the number of participants. In the end, the numbers were divided by the total number of points achievable in the whole of examinations.

The differences in the two tables can be explained by the fact that the distribution of requirements varies slightly from examination to examination. In order to substantiate the hypothesis that the two distributions actually differ only slightly, an χ2 adaptation test was carried out. The associated p-value of 0.7215 at least does not suggest a rejection of this null hypothesis and indicates similar examination designs. Nevertheless, a slight shift regarding the levels of knowledge can be observed. In the examinations for the digital teaching format, fewer tasks were asked at knowledge level K3, which is evidenced by the corresponding row sums of both tables. The rounded 0.0 entries in the tables show that the taxonomy and knowledge levels (K1,T1) and (K1,T3) were not or negligibly rarely tested, respectively.

In the following we take a look at the students‘ successes in processing the tasks with regard to taxonomy and knowledge level. The percentage of correctly edited tasks is shown for the two teaching formats in tables 4 (classic) and 5 (digital).

Obviously, these tables do not represent relative frequency distributions. For instance consider the field (K2,T2) in Table 4, which shows the value 67.2. This means that in the classic teaching format, two out of three tasks concerning the combination of T2 and K2 were processed correctly or that 67.2 percent of the achievable points in these tasks were achieved, respectively.

The differences between both tables are already visible with regard to their marginal distributions. Looking at the row sums, it is noticeable that the level of knowledge for the digital examination results has been increased compared to the results in the classic teaching format. The column sums of both tables indicate that there has been a shift from T1 to T2/T3 at the taxonomy level. For example, 74.8 percent of the tasks at level T1 were correctly completed in the examinations for the classic teaching format, while this holds for just 60.3 percent in the digital teaching format. On the other hand, tasks of higher taxonomy level (T2 and T3) were processed more successfully in the digital format. The authors attribute this to the fact that in face-to-face exercises it takes place a frequent repetition of simple tasks. Experience has shown that the questions of lower-performing students are dealt with in great detail in order to consolidate basic terms and definitions and to develop a routine through repetition. In the online exercises, on the other hand, the authors observed significantly fewer interactions between lecturer and students at all.

Due to the improvement of the examination results in the digital format, the overall positive feedback of the students regarding the asynchronous elements (especially the explanatory videos) and the associated lack of social interaction, it is planned to establish a hybrid teaching format, taking into account the current social framework conditions. Note that there is no uniform definition of the concept of hybrid teaching. A successful systematization can be found in ⁵. In ⁶, it is found that - based on a multiple regression model - an active engagement of students with course contents is significant and at the same time the strongest of all considered predictors for student satisfaction and self-perceived learning progress.

In the recent winter term 2021/22, the lecture-accompanying exercises were offered both in presence (divided into small groups) and digitally (via online platform 'Zoom'). In the future, it is essential to maintain the face-to-face offer, as some of the students have clearly spoken out in favour of the face-to-face offer of exercise lessons. In ⁷ the authors note that heterogeneous learning levels and a different prior knowledge of the students are among those diversity characteristics in the context of an university, which can best be countered by individual support. This is also confirmed by the empirical study ⁸: "With regard to media use, teaching and learning fail in the opinion of the students surveyed if digital teaching material is not prepared in a comprehensible way, if it cannot be used flexibly in terms of time, if there are no self-testing possibilities and forms of activation as well as interaction with other students and teachers." (Translated from German.) In particular, offering face-to-face exercise lessons can help to counteract the deterioration of the examination results observed in section 4.2 in the digital teaching format, namely in the lower taxonomic range. In addition, suitable multiple-choice tests at the lower taxonomy level T1 are planned at regular intervals on the ILIAS platform in order to consolidate basic concepts.

In the medium term - subject to technical availability - one part of the students can be taught in presence within the framework of a hybrid teaching format, while the other part is connected online. This teaching format (e.g. see ⁹), which is treated under the concept of 'Blended Synchronous Learning', undoubtedly requires an appropriate technical infrastructure in lecture halls and seminar rooms, respectively. In times of a pandemic, this includes an appropriate consideration of room capacities limited by distance rules as well as the use of air purifiers, room fans, CO₂ measuring devices and so on. The attribute 'synchronous' already indicates that teaching staff and students are simultaneously in the same place and/or in a common digital learning environment. With this hybrid teaching format, the application of a rotation principle is recommended, in which face-to-face and online teaching alternate for previously defined groups of students according to a rolling scheme. In addition, it makes sense to rely on the 'asynchronous' modules developed during the COVID-19 pandemic and to be further developed in the future, such as the positively evaluated explanatory videos and exercise materials, which guarantee students time and space flexibility. Maintaining and continuously improving these modules seems to make sense even after the (hoped-for) overcoming of the COVID-19 pandemic, as they create the necessary freedom to use the lecture time that is very limited for a broad introduction to statistics and elementary probability calculation more effectively.

5. Conclusion and Recommendation

In summary, it is stated that the classic teaching format performed significantly worse in terms of examination results than the digital one, whereby the reasons for this phenomenon should be critically questioned.

Whenever data sets are aggregated into a few quantities, information is lost. The interpretation of parameters, aggregates and especially significance tests is always afflicted with uncertainty. The statistical evaluations presented here only indicate that the examination results in the academic year 2020/2021 have improved significantly compared to previous years. This pleasing result is undisputed, but a causal relationship to the attribute 'teaching format' cannot be assumed. In other words, whether the improvement of examination results is solely due to the pandemic-enforced modification of the teaching format cannot be answered seriously. Other variables in the background that have not yet been taken into account may also be responsible for improving the results. In particular, the possibility of varying prior school knowledge over the years discussed in section 4.1 can have a major effect on the results. Hence, the prior knowledge should be appropriately queried and documented in advance for future studies. An additional factor could be the year-round online availability of teaching materials in the digital format, which improves the learning conditions particularly for exam repeaters, since the material is well suited for self-study despite the lack of exercise lessons and lectures.

Social interactions between teacher and students as well as students among themselves have always been part of everyday student life and are important for the development of professional and social skills as well as for the personal development of young people in general. However, there is no denying the student desire for flexibility in terms of time and space, which has been particularly expressed during the pandemic. According to the authors‘ opinion, both goals can be well balanced by hybrid teaching formats, i.e. some mix of online and face-to-face exercises and lectures. Especially in the first semesters of study, sufficient social contacts and familiarization with everyday university life are of great importance. Here, a well thought out hybrid teaching format can compensate for the weakness of a purely digital one.

The CUT dedicates a high degree of institutional attention to teaching and has received several awards over the years for the implementation of innovative teaching concepts. It has paid off that modern teaching and learning concepts have already been established and suitable structures have been set up in various areas in the years before the COVID-19 pandemic. Overall, the universities are well advised to keep up with the times and gradually expand and improve their digital offerings in order to remain attractive for students. In any case, regarding the choice of the alma mater, the criterion of spatial proximity will take in future on a lower weight than in the past.

References