Global Assessment Method for System Quality

Kazuhiro ESAKI

  Open Access OPEN ACCESS  Peer Reviewed PEER-REVIEWED

Global Assessment Method for System Quality

Kazuhiro ESAKI

Faculty of science and engineering, HOUSEI University, Kajino-cho, Koganei-shi, Tokyo,Japan

Abstract

For a purpose of suitable system product acquisition or development successfully, it is extremely important to assess a total quality of system product. And, it is necessary to gather the individual results of measurement to one evaluation indicator through suitable algorithm and assess the total quality of a system. Traditionally, analysis of quality requirements of a product has been conducted through questionnaires or interviews with customers based on investigator's personal experience. On the other hand, the analysis of quality requirements for system is a very difficult work because it is non-formulaic and demands much of product customer's technical perceptiveness. For the purpose of improvement of these issues, we have been working in ISO/IEC JTC1/ SC 7 WG 6, and have developed the techniques for quality requirements and evaluation of system based on the ISO/IEC9126 quality model that includes six characteristics, which is widely recognized. However, independency among each six quality characteristic has not been confirmed and the suitability of quality model for quality requirement has not been certified statistically. In the precedent study, we have verified the suitability and effectiveness of quality model of system defined in ISO/IEC9126 statistically. However, even if each quality can be evaluated from the view point of six quality characteristics separately, but the general method of assessment for total quality of whole system has not been established. Therefore, quantitative and objective assessment or comparison of a total quality of final developed whole system has been extremely difficult work for developers or acquirers. For the purpose of solving this kind of issues, this paper proposes the method of implementing the global quality assessment indicators for evaluation of total quality of system product from the view point of six quality characteristics. Furthermore, this paper proposes the result of verification about effectiveness of method for implementing global quality assessment indicators and usefulness of adaptation of method to predicting total quality of system product.

At a glance: Figures

Cite this article:

  • ESAKI, Kazuhiro. "Global Assessment Method for System Quality." American Journal of Systems and Software 1.1 (2013): 11-19.
  • ESAKI, K. (2013). Global Assessment Method for System Quality. American Journal of Systems and Software, 1(1), 11-19.
  • ESAKI, Kazuhiro. "Global Assessment Method for System Quality." American Journal of Systems and Software 1, no. 1 (2013): 11-19.

Import into BibTeX Import into EndNote Import into RefMan Import into RefWorks

1. Introduction

For a purpose of suitable system product acquisition or development successfully, it is extremely important to perform the quantitative and total assessment of system product. And, it is necessary to gather the individual quantitative measurement results to one evaluation indicator through some kind of algorithm and assess the superiority and inferiority of the total quality of whole system generally. The customers should choose suitable product by defining the quality requirement from needs during possible early stage of acquisition. If we take the wrong choice of the product in accord with the real needs from the customers, it may cause the big risk as not be able to make use of the product which has purchased with much effort for a purpose of use. Generally, quality evaluation of a system product is performed by accordance with quality requirements of the system that is defined by customers.

On the other hand, the quality requirements analysis is a very difficult work because it is non-formulaic and demands much of product customer's technical perceptiveness, sense in balance and experiences. Traditionally, analysis of quality requirements of a product has been conducted through questionnaires or interviews of customers based on investigator's personal experience and preferences of product. But, lack of structure and incompleteness of the traditional survey introduced the lost or errors of importance of the quality requirement of a product during the analysis. For the purpose of improvement of these issues, we have been working on developing the ISO/IEC25000 (SQuaRE) series [1, 2, 3, 4, 5] of standards for quality requirements and evaluation for system and software product for a long time in ISO/IEC JTC1 (Joint Technical Committee 1 of the International Organization for Standardization and the International Electro technical Commission) SC7WG 6 (software and systems engineering under ISO technical committee, working group six).

As part of this project, we have also worked on the developments of ISO/IEC9126-1 [6] (This standard has revised to ISO/IEC25010:2011 [7]) which are the standards to provide supporting technology for above mentioned works and also we have developing the quality characteristics. Currently, the method of quantitative quality requirement definitions [3] and evaluation [4, 5] based on ISO/IEC9126-1 quality model is widely recognized and used . ISO/IEC9126-1 [6] defines the six quality characteristics of the system and these six quality characteristics are defined based on the model of Boehm [9] or McCall [10], or from the view point of a stakeholder's wide experience. Through analyzing customer requirements based on these six quality characteristics, it may be performed complete and objective evaluation of customer quality requirements for a system product. Although a certain level of improvement is expected in the completeness of describing product quality objectives by using the ISO/IEC 9126 quality model. But, the ambiguity and lack of verification of the ISO/IEC9126 quality model has made it incomplete to assure the quality objectives of a product which are completely described to satisfy the customer quality requirements. In the precedent study, we have introduced the requirements definition and evaluation method [11] for each quality of system product based on the system quality model defined in ISO/IEC9126 and proposed the effectiveness of using six quality characteristics statistically[12]. On the other hand, in order to assess a total quality of whole system, it is necessary to gather the individual quantitative evaluation results to one evaluation indicator in some kind of algorithm and assess the superiority and inferiority of the whole system generally. However, general method of integrated assessment of whole system quality has not been established. Even if quality of product can be evaluated from the view point of six quality characteristics, but the method of general quality assessment of system has not been sure. Therefore, quantitative and objective assessment or comparison of a quality of final developed whole system has been extremely difficult work for developers or acquirers. As the result, it is very difficult to select suitable product objectively from the alternative candidate products. In recent years, customers are able to purchase products based on an increasing number of customer reviews posted on the internet web site. This study focuses on negative opinion of Laptop Personal Computers (LPCs) posted by consumers and analyzes result of such reviews based on the six quality characteristics. Reasons for choosing LPCs are, at first, LPCs have at-tribute and characteristics that correspond to the six quality characteristics, and there is a large amount of data available on the non-functionality and non-quantitatively requirements on online review web-sites. Also, this study has used the statistical analysis approach based on the precedent study of system product and process improvement [12, 13].

For the purpose of solving this kind of issues, this paper proposed the three types of method of developing the global quality assessment indicator (GQAI: this paper call global quality assessment indicator as GQAI) for evaluation of total quality of system product from the view point of six quality characteristics. Also, this study has developed prediction model and verified these integrated assessment indicators by using inherent attributes of system product. Furthermore, this paper proposes the result of verification about effectiveness of method for developing GQAI and effectiveness of adaptation of method to predicting total quality of system product.

2. Concept

2.1. Quality Requirement and Evaluation

Figure 1 shows the concept of system product implementation supported by ISO/IEC25000 (SQuaRE) series of standard. Customers have requirements for an inherent attributes of system product. In order to perform development, at first, developers should specify a quality requirement from the view point of customer needs.

Developer should make specifications and confirm the real needs of the customers at the beginning of design, and should define the concrete inherent attribute quantitatively for implementing a product. After implementation, developer and customer should assess the target system product based on the quality requirement specification in order to assure the total quality of implemented product. From Figure 1, ISO/IEC25030 provides the requirements and recommendations for specifying the quality requirements from the view point of selected customer needs. The specified quality requirements should be used as the criteria of system product quality evaluation as shown in Figure 1.

Figure1. Total Frame work of Quality Requirement and Evaluation of SQuaRE

Quality requirements of system can specify by using the process defined in ISO/IEC25030 [3] based on the six quality characteristics included in the ISO/IEC9126-1 quality model. From Figure 1, system quality can evaluate by using ISO/IEC25040, 25041 [4, 5] after implementation based on the quality requirements defined during system design phase. A good system is the product which is high quality in a sufficiency degree of the product for the customer needs, and it is thought that the satisfaction of quality of the product for the customer is high. Therefore, negative opinion of inherent attribute of the product can be the target of decision for the purchase. It is thought that the customer satisfaction of the product depend on the inherent attribute of a product, which comes out from the customer needs for the target product. The developer should implement the inherent attribute of the product to the target system based on the customer needs.

After completion of products implementation, concrete inherent attribute of product and production cost is decided. If the customer satisfaction for an inherent attribute is high, as a result, the sales of the product may increase. Here, price is not inherent attribute but assigned attribute. If developers wish to evaluate the customer satisfaction from the view point of product quality, they should evaluate a product from the view point of inherent attributes of product. Therefore, this study performed only focus on the inherent attributes of system product.

This study has tried to develop the prediction model that can predict customer satisfaction judging from the viewpoint of six quality characteristics of the product by using the inherent attributes of the products.

2.2. Total Customer Satisfaction

Figure 2 shows the concept of total customer satisfaction, which is influenced by an inherent attributes of system product. Form Figure 2, inherent attributes of system product cause inherent quality of system products from the view point of six quality characteristics. Part of inherent attributes of product may influence to number of negative opinion from customers. And, inherent quality of product is evaluated by the negative opinion from customers.

Therefore, inherent quality of product may influence to customer satisfaction from the view point of each six characteristics.

Figure 2. Concept of influence between inherent attributes

Total customer satisfaction is obtained from each customer satisfaction taking into account the six quality characteristics. On the other hand, inherent attribute of system product may influence the assigned property of product such as cost and price. Furthermore, quality and price may influence the outcome of sales. In this paper, we tried to investigate about relationship among inherent attributes and each customer satisfaction of product, which have obtained from the viewpoint of six quality characteristics defined in ISO/IEC9126-1.

2.3. Global Quality Assessment Indicator

Acquirers have needs of assessment of full product for system acquisition. In order to perform acquisition, at first, specification of quality requirement for product should be performed from the view point of six quality characteristics by using ISO/IEC 25030 [3]. In defining the customer quality requirements of a system, identifying negative opinion of a product based on the quality model and setting quality objectives of a product should be performed in the product design phase.

Acquirer should evaluate the system product based on the requirement specification of product by using ISO/IEC 25040 [4] in order to accept the developed product.

Usually, customer needs include functional or non-functional requirements. Non-functional requirement can include quality requirement and other requirement such as hardware, data, and business requirement so on.

Quality requirement can be defined from the view point of six quality characteristics included in the quality model defined in ISO/IEC 9126-1 [6] and ISO/IEC 25020 [8].

Figure 3 shows the implementation process of GQAI of system quality based on the concept of total customer satisfaction as shown in Figure 2.

A quantitative degree of attributes of customer satisfaction can be measured by quality measure element by using actual negative opinion of inherent attributes of product. After that, quantitative degree of customer satisfaction from the view point of each six quality characteristics can be calculated by using the algorism defined in quality measures. GQAI can be implemented by using the quality measures from the view point of each six quality characteristics.

Therefore, a quantitative degree of total assessment of system product from the view point of customer satisfaction can be obtained by using the suitable algorism defined in GQAI.

Figure 3. Development Process of Total Quality Assessment Indicators
2.4. System and Software Quality Model

Figure 4 shows the structure of the System Quality Model defined in ISO/IEC9126-1. Recently, ISO/IEC9126-1 has replaced by ISO/IEC25010:2011,but it is widely recognized and actually used, then this study focus on the ISO/IEC9126-1.

Figure 4. System and software product quality model

From Figure 4, this model includes the six quality characteristics for system and software product as follows: Functionality, Reliability, Usability, Efficiency, Portability and Maintainability.

・Functionality can provide the ultimate function for supporting the improvement of work. And, Usability provides the ease of use of the system.

・Reliability and Efficiency represents the capability possessed in the system, and is the characteristic associated with sustaining the quality objectives determined by the Functionality and Usability.

・Maintainability and Portability provide the capability of the system to adapt to changes in the system environment and the usage environment.

3. Analysis Process

3.1. Target Entities of Analysis

This study first collects the customers complaints, i.e., expression of customer dissatisfaction posted on a review of website where customers who actually purchased personal computer related products post their complaints as shown in Table 1.

Table 1. Example of negative review from web-site

Table 2. Importance of customer needs by six quality

The study, then, classifies the posted customer negative opinions from the view point of six quality characteristics defined in the ISO/IEC9126-1.

After that, obtains the weight by the questionnaires about inherent attribute of each quality characteristic to represent importance of quality characteristic for the customers. Table 2 show the obtained result of quantitative degree of importance (weight) for each six quality characteristics.

Furthermore, this study has collected inherent attributes of target products, i.e., CPU, HDD capacity, Weight of body, Drive time, as shown in Table 3.

3.2. Verification Process

Step1: This study has calculated the observed customer satisfaction from the viewpoint of six quality characteristics by using the method for obtaining customer satisfaction, and effectiveness of this method has proposed in the precedent study [12]. The degree of customer satisfaction is able to obtain from the consideration of importance (weight) for customer satisfactions by each six quality characteristics for the target system as shown in Table 5.

Step2: This study has developed the GQAI of total customer satisfaction by using the quality measures of customer satisfaction based on the concept of implementation process of global quality assessment indicators as shown in Figure 3.

Step3: This study has performed correlation analysis among degree of total customer satisfaction obtained from GQAI and inherent attributes of LPCs, and has confirmed the correlation between each customer satisfaction and inherent attributes of the target system as shown in Table 5.

Step4: This study has developed the model that actually predicts the degree of total customer satisfaction of the target system products by using the degree of inherent attributes of product based on the concept of Figure 3.

Step5: This study has performed multiple-regression analyses and for performing the multiple regressions analysis has assigned the degree of each total customer satisfaction as an objective variable and assigned the degrees of inherent attributes as explanatory variables based on the consideration of result of correlation analysis.

Step6: This study has confirmed the possibility of whether or not the degree of total customer satisfaction of product could be derived from the degree of inherent attributes of target product by using developed GQAI.

Step7: Finally, this study verified the advantage of proposed method for assessment of total quality of system based on the result of multiple regression analysis.

Table 3. Example of the Inherent Attribute of LPCs

3.3. Degree of Customer Satisfaction

In recent years, due to the explosion of the Internet, purchasing behaviours of customers have significantly changed. For example, an increasing number of customers can order a product directly from an electric commerce site without visiting brick-and-motor shops while remaining at home.

The degree of customer satisfaction is an indicators used in marketing that represents how a product or service produced by a company meets customer expectations.

This study has focused on online reviews posted on the Internet, an effective alternative to face-to-face interviews of customers, and has used the online negative reviews of a system product as the data of investigation.

This study has collected online reviews of products posted at a web-site, kakaku.com [14] as customer's expression of his/her dissatisfaction of system products.

Table 1 show the part of collection data concerning negative opinions from web-site, which total number of type of LPCs is 35 and total number of review is 457. From Table 1, this study has counted the number of online negative opinions for each concrete category of interest of LPCs from the view point of the six quality characteristics.

This study has collected and classified online negative opinions from the view point of the six quality characteristics in this manner.

3.3Degree of Customer Satisfaction

Furthermore, for each product, this study has obtained the degree of importance (weight) for each six quality characteristics taking into account the interest of attribute of LPCs by six quality characteristics as shown in Table 2.

Table 2 shows the example of questionnaires and the obtained result of importance of quality needs by each six quality characteristics obtain from questionnaires.

For example, the questionnaires have asked to the customers, "in purchasing a LPC, what attributes are important?" and the customers have assigned the numeric order number between 1 and 15 based on the importance.

The meaning of order number 1 is the most important attribute for the quality characteristic, and the weights for the six quality characteristics have normalized in the range from 0 to 1.

This study has quantitatively calculated the importance of customer needs for each six quality characteristics.

For example, the importance of functionality as M is obtained from the equation (1).

(1)

M: Importance ratio of Functionality (weight)

Qa1n : Order number of importance (Sa1 =1~ON)

Qa2n : Order number of importance (Sa2 =1~ON)

n : Number of customers (n=1~TN)

ON: Maximum order number (ON=14)

TN: Total number of customers (TN=61)

From Table 2, TN is the total number of customers, which total number of customers is 61, and p is the weight for the efficiency determined from the questionnaires.

This study focused on laptop computers (LPCs).

Table 3 shows the part of collection data concerning inherent attributes of LPCs from web-site, which total number of type of LPCs is 35. Inherent attribute refers to the degree to which attributes of target product have the intrinsic potential to satisfy stated and implied needs when LPCs is used under specified conditions.

From the inherent point of view, Attribute of LPCs is product itself, in particular to shown in Table 3.

Table 4 shows the parts of result of degree of observed customer satisfactions from the view point of each six quality characteristics. By applying the weight for each of the six quality characteristics, this study has quantitatively calculated the degree of customer satisfaction for each six quality characteristics.

Table 4. Degree of observed customer satisfaction by six quality characteristics

For example, the degree of customer satisfaction for SAi (functionality) and Sa1i is obtained as following equation (2) and (3).

(2)
(3)

SAi : Customer satisfaction of functionality

sa1i: Ratio of un-satisfaction of efficiency by each category

a1i : Number of negative opinion of functionality

i : Number of sample product (i=1~35)

M: Importance ratio of functionality (weight)

RC: Total number of online reviews of a given product

3.4. Global Quality Assessment Indicators

This study has developed the three types of GQAI of degree of total customer satisfaction. For example, the degree of total customer satisfaction of Y1m, Y2m and Y3m are obtained by using each category of customer satisfaction of LPCs as following equations (4), (5) and (6).

(4)
(5)
(6)

m=1 or 2 (1: consider weight , 2: not consider weight )

Furthermore, this study has predicted the degree of total customer satisfaction in order to compare the advantage of method for implementing GQAI.

For example, the degree of predicted total customer satisfaction of yA is obtained by using inherent attributes of LPCs as following equations (7).

(7)

yA: Predicted customer satisfaction of functionality

rn : partial regression coefficient (n=0~14).

4. Verification of GQAI

This study has developed the model that actually predicts the degree of total customer satisfaction by each three types of GQAI from the degree of inherent attributes of target product.

And, the effectiveness of integration method to assess the total quality of system product has been verified based on the result that degree of total customer satisfaction by each three types of GQAI could be derived from the degree of inherent attributes corresponding to the target product.

4.1. Correlation Matrix between Observed Total Customer Satisfaction and Inherent Attributes

Table 5 shows the result of correlation analysis between inherent attribute and obtained total customer satisfaction of LPCs by each three types of GQAI.

The degree of total customer satisfaction by Y1m could obtained from equations (4) based on the Table 1 and Table 2.

The degree of total customer satisfaction by Y2m and Y3m of GQAI could obtained from equations (5) and (6) based on the degree of customer satisfaction by each six quality characteristics as shown in Table 4.

From Table 5, correlation of total customer satisfactions by each three GOAIs and some of inherent attributes of LPCs are recognized.

Table 5. Correlation Matrix between total customer satisfaction and inherent attributes

Table 6. Result of Multiple regression Analysis for verify the global assessment of indicator of system

4.2. Multiple Regressions Analysis

This study has performed multiple-regression analyses and has assigned the degree of total customer satisfaction by each three types of GQAI as an objective variable and assigned the degrees of intended inherent attributes as explanatory variables as shown in Table 6.

Table 6 shows the six types of developed prediction models based on the consideration of importance of customer needs and about the result of correlation analysis shown in Table 5. From the Table 6, the result of multiple-regression analysis between the observed total customer satisfaction Y11 and the concerning inherent attributes a, f, h, i, n, o” shows that the degree of the multiple-regression coefficients and the determination coefficients are 0.5830 and 0.3398, respectively. In addition, maximum value of F-test is 2.4023 and it is lower than 5% (F0 = 2.4205) not significance level.

The result of multiple-regression analysis between the observed total customer satisfaction Y21 and the concerning inherent attributes “a, f, h, j, n, o” shows that the degree of the multiple-regression coefficients and the determination coefficients are 0.6034 and 0.3641, respectively. In addition, maximum value of F-test is 2.6722 and it is higher than 5% (F0 = 2.4205) significance level.

Also, the result of multiple-regression analysis between an observed total customer satisfaction Y31 and the concerning inherent attributes “a, f, h, j, n, o” shows that the degree of the multiple-regression coefficients and the determination coefficients are 0.5941 and 0.3529, respectively.

The maximum value of F-test is 2.5451 and it is higher than 5% (F0 = 2.4205) significance level.

From Table 6, the cause and effect relationship between the degrees of observed total customer satisfaction obtained by GQAI such as Y21, Y31 and intended inherent attributes of those corresponding LPCs has observed.

On the other hand, the cause and effect relationship between the degree of observed total customer satisfaction obtained by another method such as Y12, Y22 and Y32, which has not considered the importance of customer needs for each six quality characteristics, could not be observed.

5. Conclusion

Based on the result of verification about the integration method of GQAI, this study confirmed the followings.

This study verified the accuracy of the method for implementing GQAI about Y21, Y31 which has integrated by using the degree of customer satisfaction from the view point of six quality characteristics and has considered importance of customer needs and usefulness of those indicators are recognized.

Also, effectiveness of method Y21 is recognized most high based on the consideration of Table 6.

This study also verified the accuracy of the method for implementing the method Y11, which has not considered the view point of six quality characteristics, and usefulness of those indicator are not recognized.

From Table 6, the necessity of consideration of importance (Weight) from the view point of six quality characteristics for integration to GQAI has recognized. In addition, relationship between degree of total customer satisfaction by using GQAI and the inherent attribute of product has recognized.

In the future study, the author plans to investigate the application of GQAI based the concept of this study to other systems and inspect the proposed method to assess the total quality of system products.

Acknowledgement

The authors is grateful to members of production system research office of Graduate School at Factory of Science and Engineering HOSEI University and their contributions and support for making the discussion.

References

[1]  ISO/IEC25000: Software engineering-Software product Quality Requirements and Evaluation (SQuaRE)-Guide to SQuaRE, Int’l Organization for Standardization, 2005.
In article      
 
[2]  ISO/IEC25001: Software engineering-Software product Quality Requirements and Evaluation (SQuaRE)-Planning and Management, Int’l Organization for Standardization, 2007.
In article      
 
[3]  ISO/IEC25030: Software engineering-Software product Quality Requirements and Evaluation (SQuaRE)-Quality requirement, Int’l Organization for Standardization, 2007.
In article      
 
[4]  ISO/IEC25040: Software engineering-System and software Quality Requirements and Evaluation (SQuaRE)-Evaluation process, 2011.
In article      
 
[5]  ISO/IEC25041: Software engineering-System and software Quality Requirements and Evaluation (SQuaRE)-Evaluation guide for developers, acquirers and independent evaluators, 2012.
In article      
 
[6]  ISO/IEC9126-1: Software engineering-Product Quality-Part1: Quality model,2001
In article      
 
[7]  ISO/IEC25010: Software engineering-System and software Quality Requirements and Evaluation (SQuaRE)-System and software Quality Model, Int’l Organization for Standardization, 2011.
In article      
 
[8]  ISO/IEC25020: Software engineering–Software product Quality Requirements and Evaluation (SQuaRE) -Measurement reference model and guide, Int’l Organization for Standardization, 2007.
In article      
 
[9]  Boehm,B.W.et al,Quantative Ev. Of Software Quality, 2nd ICSE pp.596-605, 1976.
In article      
 
[10]  McCall,J.A.et al,Factors in Software Quality,RADC TR-77369,1977.
In article      
 
[11]  K,Esaki, System Quality Requirement and Evaluation, importance of application of the ISO/IEC25000 series, Global Perspective on Engineering Management, Vol. 2, Iss. 2,2013.
In article      
 
[12]  K,Esaki, Verification of Quality Requirement Method, American Journal of Operations Research,Vol.2, No.1, pp.70-79,2013.
In article      
 
[13]  K,Esaki,Y,Ichinose,S,Yamada, Statistical Analysis of Process Monitoring Data for Software Process Improvement and Its Application, American Journal of Operations Research,Vol.3, No.1A, pp.43-50,2012.
In article      
 
[14]  Kakaku.com: http://www.kakaku.com/.
In article      
 
  • CiteULikeCiteULike
  • Digg ThisDigg
  • MendeleyMendeley
  • RedditReddit
  • Google+Google+
  • StumbleUponStumbleUpon
  • Add to DeliciousDelicious
  • FacebookFacebook
  • TwitterTwitter
  • LinkedInLinkedIn