A New Methodology of Usability Testing on the Base of the Analysis of User’s Electroencephalogram

Alfimtsev A.N., Basarab M.A., Devyatkov V.V., Levanov A.A.

Journal of Computer Sciences and Applications

A New Methodology of Usability Testing on the Base of the Analysis of User’s Electroencephalogram

Alfimtsev A.N.1, Basarab M.A.2,, Devyatkov V.V.1, Levanov A.A.1

1Department of Information Systems and Telecommunications, Bauman Moscow State Technical University, Moscow, Russia

2Department of Information Security, Bauman Moscow State Technical University, Moscow, Russia


This paper discusses the methodology of usability testing on the base of the real-time analysis of user’s electroencephalogram (EEG). In the experimental study of the control groups of users working with a real interface of commercially relevant applications, two distinct user states were revealed which correspond to the superposition of EEG beta and delta brain waves corresponding to the so-called «calm» state and «excited» state. These states are characterized by the user experience with a friendly user interface and an interface that is considered by experts as subjectively complicated and confusing. On the basis of the obtained data, some improvements in the users interface for mobile devices were proposed, which led to a threefold increase in its CRT (click-through rate).

Cite this article:

  • Alfimtsev A.N., Basarab M.A., Devyatkov V.V., Levanov A.A.. A New Methodology of Usability Testing on the Base of the Analysis of User’s Electroencephalogram. Journal of Computer Sciences and Applications. Vol. 3, No. 5, 2015, pp 105-111. http://pubs.sciepub.com/jcsa/3/5/1
  • A.N., Alfimtsev, et al. "A New Methodology of Usability Testing on the Base of the Analysis of User’s Electroencephalogram." Journal of Computer Sciences and Applications 3.5 (2015): 105-111.
  • A.N., A. , M.A., B. , V.V., D. , & A.A., L. (2015). A New Methodology of Usability Testing on the Base of the Analysis of User’s Electroencephalogram. Journal of Computer Sciences and Applications, 3(5), 105-111.
  • A.N., Alfimtsev, Basarab M.A., Devyatkov V.V., and Levanov A.A.. "A New Methodology of Usability Testing on the Base of the Analysis of User’s Electroencephalogram." Journal of Computer Sciences and Applications 3, no. 5 (2015): 105-111.

Import into BibTeX Import into EndNote Import into RefMan Import into RefWorks

At a glance: Figures

1. Introduction

Until the twentieth century, most machines created by human civilization, were mechanical, and interacted with a person mainly physically. But gradually the human inventions have become increasingly relevant to the field of abstract, intellectual, and virtual tasks. That is why such disciplines as usability and ergonomics are becoming increasingly important in the modern world. Usability means making products and systems easier to use, and matching them more closely to user needs and requirements. The international standard, ISO 9241-11, provides guidance on usability and defines it as follows: «The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use» [1]. To find this degree of efficiency and satisfaction, the usability testing of user interface is conducted. A direct relationship between usability of the user interface and its conversion was proven [2]. High rates of the conversion directly lead to higher sales of software with user friendly interface [3, 4]. That is why we can see an active development of new techniques for automation [5, 6], investigation [7], and evaluation of the results [8] of usability testing. However, even the combined techniques of usability testing, based on the aggregation of various external user data [9], cannot be considered objective. Such tests may contain an error in the interpretation of user data, an error in usability specialist methods, and above all, a mistake in explaining their actions by the users [10].

The main goal of this paper is to develop an objective methodology of usability testing, based on the analysis of user’s brain waves (electroencephalograms). The users cannot control his brain waves and so the use of brain waves for usability analysis suggests that the results thus obtained are objective in sufficient degree.

The structure of this paper is as follows. Section 2 includes the survey of brain waves and usability testing applications using an electroencephalogram, the novelty of this study is justified. Section 3 provides a brief description of the device used to retrieve information about the activity of brain waves of a user. Section 4 deals with the experimental methodology to identify quantitative values of brain waves characterizing usability and user-friendly interface. In section 5 conversion optimization is considered. Section 6 is the conclusion.

2. Brain Waves and User Interface Analysis

Modern methods of technical analysis of the electrical activity of the brain have found that the human brain constantly emits electrical impulses, called the brain activity waves [11]. When a person is awake, his brain generates waves of all ranges. However, when person is in a certain state of his body and mind only one of the types of waves will prevail. Human brain emits gamma, beta, alpha, theta and delta waves. Assumed that a brain produces gamma waves when a person needs to work simultaneously with different types of information and quickly relate them to each other [12]. A small amount of gamma-waves leads to the reduced ability to memorize anything. The left hemisphere of a human brain are generates beta waves [13]. The beta waves are responsible for the solution of problems, logical thinking, concentration, and decision-making. These waves allow being active in society. The amount of beta waves increases when working with the material world, talking, learning activities. Commonly believed that beta waves accelerate the work of the brain, improve the processing and assimilation of information, raise the overall level of energy of the body, and stimulate the nervous system. The right hemisphere of the brain generates alpha waves. Alpha waves increase the ability to take huge amounts of information, and develop abstract thinking and lead to inner balance and self-control [14]. In addition, alpha waves provide a link with the subconscious mind. It is in the alpha state the human brain produces more of so-called pleasure hormones that reduce pain, and are responsible for a positive outlook on life, happiness, joy. The left hemisphere of the human brain generate theta waves [15]. The theta waves lead human body into a state of deep relaxation, a state of slumber and dreams. These waves awaken and strengthen the feelings and emotions, and allow programing and reprograming the subconscious mind. It's believed that a human brain generate delta waves most actively during deep sleep to ensure its recovery [16]. A brain produces large amounts of growth hormone and the body's self-healing processes are intensive and self-healing in the delta state. The brain produce delta waves even when not all the other waves of brain activity generated. Delta waves are the slowest and mysterious of all types of waves. For a long time the delta waves were not available to scholars for study. Thus, beta waves in most characterize the conscious brain and the delta wave characterize the subconscious of the human brain.

Advances in the development of biosensor technologies in the new century allowed to use brain waves for user interface analysis [17], multimodal interaction [18] and augmented reality [19]. It has been demonstrated that the electroencephalogram (EEG) of a human brain enables an objective assessment of a user state [20], which can be used in automatic access control [21]. Extensive researches on usability evaluation in brain-computer interfaces (BCI) for operation in different applications were carried out [22, 23, 24] as well as the usability of different EEG headsets in the context of a BCI application was explored [25]. Another important trend - neuromarketing is an emerging field of marketing research which uses brain imaging techniques to study neural conditions and processes that underlie consumption [26].

The alpha and the beta brain waves at work with an interface were investigated in details [27], but the specific brain wave ranges for a friendly and not friendly interfaces have not been allocated. Moreover, unlike these papers, in our study not artificial interfaces for usability testing but user interfaces of the real relevant commercial applications are used. For the first time, the increased conversion of the application after UI improvements after usability testing with EEG is clearly shown. Usability testing methodology used in this work involves working outside laboratory conditions, without the use of large (32-256) pool electrodes and is based on an elegant mobile device, NeuroSky® MindWave [28], which is described in the next section.

3. Description of the NeuroSky MindWave Device

Appearance of the MindWave device produced by NeuroSky, Inc., is shown in Figure 1 (a), an example of a mobile device on the user's head is given in Figure 1(b).

Figure 1. NeuroSky MindWave device: (a) Appearance of the device; (b) Example of mounting the device on the head of the tester

Device’s neural network interface is based on specialized chip (ASIC), made by NeuroSky company, delivered as ready-made module TGAM1. The module has 3 inputs: EEG (electrode on the forehead), REM, and GND (electrode on the clip posted on the ear). All electrodes are connected by a shielded cable. The module power supply 3.3 V shaped DC-DC converter, posted on the main board. Unit's power comes from one element AAA 1.5 V. Data from the device is transmitted by radio to a special USB-receiver, which emulates a serial port of the computer.

The brain constantly produces electrical signals through interactions between cellular components of the brain, the neurons. At the macro level, they produce a wide range of frequencies. The apparatus detects MindWave brain electrical activity and divides the signal into different types of waves according to the frequency. To reduce electrical interference generated by the human body, the device has a "base" contact, which is attached to the earlobe and allows filtering electrical waves produced by the body (noise). The device includes a Bluetooth-module that allows the transmission of data for further processing and interpretation.

Figure 2. Simple scheme of interaction of the devices during testingand experiments

In our experiment, the data are transmitted to the mobile phone working under iOS operating system (iPhone). Figure 2 shows the scheme of interaction of the devices during testing, debugging, and experimentation. First, the program is downloaded to the mobile device (step 1), then the mobile device sends a request to connect to the headset (step 2). On step 3, MindWave device receives the request and starts sending data at specified intervals. Mobile device receives the data and sends them to the computer for storage and analysis (step 4).

With the library from the official website of the product it is possible to arrange the data of brain waves from the device to the following format:

Alpha rhythm (α-rhythm) - Rhythm of the electroencephalogram in the frequency band from 8 to 14 Hz, the average amplitude of 30-70 mV, however, can be observed high and low amplitude α-waves. Registered for 85-95% of the healthy adults. It is best manifested in occipital area of a brain. The greatest amplitude of α-rhythm is in a state of quiet wakefulness, especially with eyes closed in a darkened room. Blocked or attenuated with the increasing of the attention (especially visual attention) or mental activity.

Beta-rhythm (β-rhythm) - rhythm of the electroencephalogram in the range of 14 to 30 Hz with a voltage of 5-30 mV inherent state of active wakefulness. Most strongly this rhythm is expressed in the frontal areas, but various kinds of intense activity sharply increases and spreads to other areas of the brain. Thus, expression of β-rate increases at a presentation of a new unexpected stimulus, in a situation of attention, with mental stress, emotional excitement. They belong to the fast wave. Their amplitude is 4-5 times less than the amplitude of the α-waves.

Gamma wave (γ-rhythm) - potential of the electroencephalogram oscillations in the range from 30 Hz to 120-170 Hz, and according to some authors to - 500 Hz. The amplitude is very low - less than 10 mV, and is inversely proportional to frequency. If the amplitude of the gamma rhythm higher than 15 mV, the EEG is considered as abnormal.

Delta-rhythm (δ-rhythm) or delta waves - of the electroencephalogram rhythm. It consists of a high-amplitude (hundreds of microvolts) waves frequency of 1-4 Hz. Low-amplitude (20-30 uV) oscillations of this range may be recorded in the EEG rest in some forms of long-term stress and mental work.

Theta rhythm (θ-rhythm) - rhythm of the electroencephalogram characterized by a frequency of 4-8 Hz, the electric potential of 10 - 400 mV. Theta rhythm of low amplitude (25 - 35 mV) is included as a component of a normal EEG. This is the range of the upper connections of the brain hemisphere, and linking directly to the layers of the cerebral cortex with its frontal area.

4. Experimental Research Methodology

In an experimental research, 50 people were involved. The study considered such person parameters as gender and age (from 18 to 35 years old) according to [29]. Educational level was ranged from completed secondary school to PhDs. All subjects were healthy, studied and / or worked at the Bauman Moscow State Technical University (BMSTU). Nationality of the testers was typical for the Moscow region of Russia, the native language for all was the Russian language. Testers with sickness were not allowed for the examination. Strong situations of hunger, thirst, and fatigue were also excluded. Attitude of the majority of subjects was constructive, the cases of failure or aggression were not observed.

Before usability testing of the user interfaces, a brief survey was conducted to clarify psychotype of the testers (sanguine person, phlegmatic person, choleric person, and melancholic person) and obtain the data of brain activity in the normal state. Then the testers were presented the convenient and inconvenient user interfaces of Internet sites selected by experts. For the formation of a statistically representative sample, the stratification algorithm was used. According to this algorithm, the sample consisted of the results of brainwaves measurement results was selected in the following manner. First, a general list of all testers was made in alphabetical order. After checking for errors and lack of repeatability, every second item in the list was chosen. Then, the results were averaged [30].

Under the user-friendly (convenient) interface in this paper will be understood interface, which has a high level of usability (interface can be used in the particular context with due efficiency, productivity and contentment). Thus an awkward (inconvenient) interface is the interface that doesn’t not have high rates of usability. To select interfaces for objective usability testing, the experts filled out questionnaires that determine various parameters of web pages. Each of the parameters, such as color scheme, content overload and other defined as a number in the range [0..5]. If the geometric mean value exceeds a threshold value for determining the usability of user interface (adopted for 3.75), then such an interface was confessed to convenient. Otherwise, the interface was considered inconvenient by the experts. After the analysis of the data convenient and inconvenient interfaces were chosen. Among the convenient user interfaces were selected Dribbble.com, Facebook.com, Mail.ru. Among the inconvenient user interfaces were selected websites of small companies and individuals, and abounding advertising placed on free hosting: hosanna1.com, arngren.net (Figure 3), yyyyyyy.info.

Persons had worked with each interface during 65 seconds. An example of usability testing with fixation of brain activity is shown in Fig. 4. During the first 20 seconds of data is not taken into account. The time allotted for the user to start interacting with the interface, and to forget that he is testing. Analysis time interval was taken from 20 to 40 seconds - it was noticed that during this time interval all users interact with the interface the most active. On the implementation of the experiment, for each user were obtained 3 list of arrays corresponding to the normal state of the user while working with a friendly interface and state of the user during work with the awkward interface. These arrays contain dimensionless quantitative characteristics of brain waves for each channel.

The analysis was performed for each channel separately. On the basis of the total sample for each channel separately extreme values were chosen an expert way and then the data were normalized. Since the valuation was made on the basis of expert data for testing, it was additory used an algorithm restrictions / rounding to the borders.

Figure 4. Conducting a usability test using NeuroSky MindWave

Also it was obtained the confidence intervals for each channel in the quiescent state, when working with an expert user-friendly interface and working with expert awkward interface. It was found that confidence intervals at rest intersect with all confidence intervals when working with user-friendly interface for all users. When working with the awkward interface, the confidence intervals of the channels low beta and delta-waves do not overlap with the confidence intervals in the normal state.

In general, the analysis of the experimental data yielded the following results:

1. After analyzing the values of brain waves, two states were identified: a «calm» when working with a friendly interface and «excited» when dealing with inconvenient user interface. Excited state is characterized by 20 times falling the average value of the signal of the channel of low beta waves (Figure 5 and Figure 6) and the change in the average value of delta waves by an average of 56% (Figure 7 and Figure 8). Thus, the threshold values for beta waves in the excited state are 14-22 Hz, and those for delta waves are 1-3 Hz.

2. Averaged values of brainwaves in calm and excited states do not depend on gender, age, and psychotype (difference is less than 5 percent and was adopted for the measurement error).

Resulting data indicates that working with the user interface of Internet sites that the expert considered comfortable and attractive, the user's brain does not go to a state of excitement - the data are close to normal, relaxed state. While working with the sites marked as «inconvenient», a transition in «excited» state was observed, and it may indicate stress one experiences when dealing with uncomfortable interfaces and increased load on the user's brain realizes its functions of information perception and adaptation to the environment.

5. Conversion Optimization

In this experiment the results of an advertising campaign in an objective and biased usability tests of the advertising banners were compared. Five banners were created for advertising mobile application for the iOS platform Christmas Quiz Pro .

Originally application developers, not being experts in the field of advertising, have chosen banner for targeted audience: man or woman, catholic Europe, urban, mobile device, value convenience and benefit sought (Figure 9).

Figure 9. The potential audience, selected on several criteria

Figure 10 presents a banner originally selected by application developers. Conversion statistics was viewed in the next day, namely the CTR parameter, what means the ratio between shows and clicks. Conversion rate value was 1.6 %, which is good for a diverse audience, but is rather a poor indicator for targeted groups (Figure 11, the graph obtained by application Facebook ads).

Figure 10. Exterior view of an advertising banner on the first day of sales
Figure 11. Conversion for the first day of advertising via banner selected subjectively

To increase the conversion and therefore the software product sales, an experiment was undertaken as described above. The volunteer group of 15 students of BMSTU were asked to interact with the application and pay attention to the different advertising banners. Testers were using the NeuroSky MindWave device and computer application, written specifically for usability test, analyzed the condition of the user: calm or excited. After summing up the usability test, it became clear that users prefer another banner, on which there are no screenshots of the application (Figure 12).

Figure 12. Advertising banner selected using objective usability test

The next day the results of conversion were checked again. The obtained results were very different from the previous ones: the conversion increased more than 3 times, from 1.6 % to 4.9 % (Figure 13).

Figure 13. Conversion for the second day of sales, after improvement with the objective of usability testing

This has led to increased sales of application from 14 to 15 by more than two times, as seen in Figure 14.

Thus, it is possible to assert that the objective usability testing can improve the appearance of the interface, which increases the ratio shows to clicks on advertising and increase sales of software product or the advertised product.

6. Conclusion

The human-computer interaction can be divided into two components: human and machine. And, despite the considerable number of philosophical, cultural and socio-psychological theories, namely human component remains the most difficult to learn. This paper discusses the methodology to increase conversion by improving the user interface with the objective usability testing based on the analysis of the user’s electroencephalogram.

Key experimental results are as follows:

1. All people, regardless of gender, age or psycho type showed nearly identical performance of brain activity when working with user interfaces.

2. When working with resources that require high concentration and attention, the user enters the «excited» state (beta waves of 14-22 Hz, delta waves of 1-3 Hz).

3. When working with the resources that the user feels subjectively comfortable, the transition to the «excited» state is not observed.

4. Improved user interface for mobile devices, after a usability test using the techniques discussed, demonstrated a threefold increase in the conversion. High conversion in practical terms means an increase in attendance of an electronic resource, and therefore an increase in sales of software product or the advertised product.

The obtained data provide a new look at the problem of usability design and usability testing of user interfaces. Instead of subjective evaluation, now it is possible to obtain objective data. In future application of this technique is to test individual elements of the user interface. It is assumed that it will be possible to establish which buttons, switches and edit boxes are the most attractive for a particular software and to start new era of user interface design based on the real human-oriented interface elements.


[1]  ISO 9241-11: 1998. Ergonomic requirements for office work with visual display terminals (VDTs) – Part 11: Guidance on usability (IDT).
In article      
[2]  Jankowski, J., “Balanced approach to the design of conversion oriented websites with limited negative impact on the users”, Computational Collective Intelligence. Technologies and Applications. Lecture Notes in Computer Science, 8083, 527-536, 2013.
In article      View Article
[3]  Borza, I. C., Macías, J.A., “Incorporating Marketing Strategies to Improve Usability Assurance in User-Centered Design Processes”, Lecture Notes in Computer Science, 9169, 152-162, 2015.
In article      View Article
[4]  Issa, T., Turk, A., “Applying usability and HCI principles in developing marketing websites”, International Journal of Computer Information Systems and Industrial Management Applications, 4, 76-82, 2012.
In article      
[5]  Au, F.T.W., Baker, S., Warren, I., Dobbie, G., “Automated usability testing framework”, The 9th Australasian User Interface Conference (AUIC2008), Darlinghurst, Australia, 76, 55-64, 2008.
In article      
[6]  Muhi, K., Szőke, G., Fülöp, L.J., Ferenc, R., Berger, A., “A semi-automatic usability evaluation framework”, Int. Conf. on Computational Science and Its Applications (ICCSA 2013), 7972, 529-542, 2013.
In article      View Article
[7]  Darejeh, A., Singh, D., “An investigation on Ribbon interface design guidelines for people with less computer literacy”, Computer Standards and Interfaces, 36 (5), 808-820, 2014.
In article      View Article
[8]  Insfran, E., Fernandez, A., “A systematic review of usability evaluation in web development”, Web Information Systems Engineering – WISE 2008 Workshops. Lecture Notes in Computer Science, 5176, 81-91, 2008.
In article      View Article
[9]  Sim, G., Read, J.C., “The Damage Index: an aggregation tool for usability problem prioritization”, The 24th BCS Interaction Specialist Group Conference, Dundee, Scotland, 54-61, 2010.
In article      
[10]  Zicklera, C., Halderb, S., Kleihb, S.C., Herbertb, C., Küblera, A., “Brain Painting: Usability testing according to the user-centered design in end users with severe motor paralysis”, Artificial Intelligence in Medicine, 59 (2), 99-110, 2013.
In article      View Article  PubMed
[11]  Nijboer, F., Clausen, J., Allison, B.Z., Haselager, P., “The Asilomar Survey: Stakeholders’ Opinions on Ethical Issues Related to Brain-Computer Interfacing”, Neuroethics, 6(3), 541-578, 2013.
In article      View Article  PubMed
[12]  Baldauf, D., Desimone, R., “Neural mechanisms of object-based attention”, Science, 344 (6182), 424-427, 2014.
In article      View Article  PubMed
[13]  Zhang, Y., Chen, Y., Bressler, S.L., Ding, M., “Response preparation and inhibition: the role of the cortical sensorimotor beta rhythm”, Neuroscience, 156 (1), 238-46, 2008.
In article      View Article  PubMed
[14]  Rao, T.K., Lakshmi, M.R., Prasad, T. V., “An Exploration on Brain Computer Interface and Its Recent Trends”, International Journal of Advanced Research in Artificial Intelligence, 1(8), 17-22, 2012.
In article      
[15]  Huang, Y., Chen, L., Luo, H., “Behavioral Oscillation in Priming: Competing Perceptual Predictions Conveyed in Alternating Theta-Band Rhythms”, The Journal of Neuroscience, 35(6), 2830-2837, 2015
In article      View Article  PubMed
[16]  Carracedo, L.M., Kjeldsen, H., Cunnington, L., Jenkins, A., Schofield, I., Cunningham, M.,O., Davies, C.H., Traub, R.D., Whittington, M.A., “A Neocortical Delta Rhythm Facilitates Reciprocal Interlaminar Interactions via Nested Theta Rhythms”, The Journal of Neuroscience, 33(26), 10750-10761, 2013.
In article      View Article  PubMed
[17]  Hu, J., Nakanishi, M., Matsumoto, K., Tagaito, H., Inoue K., “A method of usability testing by measuring brain waves”. Int. Symposium on Future Software Technology (ISFST-2000), Guiyang, China, 159-164, 2000.
In article      
[18]  Gürköka, H., Nijholta, A., “Brain–Computer Interfaces for Multimodal Interaction: A Survey and Principles”, International Journal of Human-Computer Interaction, 28 (5), 292-307, 2012.
In article      View Article
[19]  Liao, L.-D., Lin, C.-T., McDowell, K., Wickenden, A.E., Gramann, K., Jung, T.-P., Ko, L.-W., Chang, J.-Y., “Biosensor Technologies for Augmented Brain–Computer Interfaces in the Next Decades”, Proceedings of the IEEE, 100, 1553-1566, 2012.
In article      View Article
[20]  Lee, H. and Seo, S., “A comparison and analysis of usability methods for web evaluation: The relationship between typical usability test and bio-signals characteristics (EEG, ECG)”, Conference of the Design Research Society, Montréal, Canada, 893-904, 2010.
In article      
[21]  Chuang, J., Nguyen, H., Wang, C., and Johnson, B. “I think, therefore I am: Usability and security of authentication using brainwaves”, Lecture Notes in Computer Science, 7862, 1-16, 2013.
In article      View Article
[22]  Bos, D.P.-O., Reuderink, B., Laar, B., Gurkok, H., Muhl, C., Poel, M., Nijholt, A., Heylen, D. “Brain-computer interfacing and games”, Brain-Computer Interfaces, 149-178, 2010.
In article      
[23]  do Amaral, V., Ferreira, L.A., Aquino, P.T., and de Castro, M.C.F., “EEG signal classification in usability experiments”, Biosignals and Biorobotics Conference (BRC), Rio de Janeiro, Brazil, 1-5, 2013.
In article      View Article
[24]  Yoshimura, N., Satsuma, A., DaSalla, C., and Hanakawa, T., “Usability of EEG cortical currents in classification of vowel speech imagery”, International Conference on Virtual Rehabilitation (ICVR), Zurich, Switzerland, 1-2, 2011.
In article      View Article
[25]  Nijboer, F., Laar, B., Gerritsen, S., Nijholt, A., Poel, M., “Usability of three electroencephalogram headsets for brain-computer interfaces: a within subject comparison. Interacting with computers”, 27 (5), 500-511, 2015.
In article      View Article
[26]  Hammou, K.A., Galib, M.H., Melloul, J., “The Contributions of Neuromarketing in Marketing Research”, Journal of Management Research, 5 (4), 20-33, 2013.
In article      View Article
[27]  Kimura, M., Uwano, H., Ohira, M., Matsumoto, K.-I., “Toward constructing an electroencephalogram measurement method for usability evaluation”, The 13th International Conference on Human-Computer Interaction, San Diego, USA, Part I: New Trends, 95-104, 2009.
In article      View Article
[28]  “MindWave Mobile Tutorial”. Available: http://store.neurosky.com/products/mindwave-mobile-tutorial/ Accessed November 25, 2015].
In article      
[29]  Ko, A.J. , LaToza, T.D., Burnett, M.M., “A practical guide to controlled experiments of software engineering tools with human participants”, Empirical Software Engineering, 20(1), 110-141, 2015.
In article      View Article
[30]  Tullis, T., Albert, B., “Measuring the User Experience, Second Edition: Collecting, Analyzing, and Presenting Usability Metrics”, Elsevier, 297, 2013.
In article      
  • CiteULikeCiteULike
  • MendeleyMendeley
  • StumbleUponStumbleUpon
  • Add to DeliciousDelicious
  • FacebookFacebook
  • TwitterTwitter
  • LinkedInLinkedIn