Validation of the Assessment of Performance Competency

Abstract

Competency, which has complexity, means that competency assessment has to be done elaborately and indepth. The importance of assessment is that it has to show the evidence of validity. Is assessment of performance competency by the methods of rash analysis, inter-subtest correlation analysis, and confirmatory factor analysis valid or not. The purpose of research is to examine the validity of assessment of performance competency by the methods of rash analysis, inter-subtest correlation analysis, and confirmatory factor analysis. The sample group consisted of directors and employees. The instruments included three volumes - assessment by director (Volume I), self-assessment (volume II), and supervisor assessment (volume III). The data were analyzed by the rash analysis, inter-subtest correlation analysis, and confirmatory factor analysis. Results revealed that the examination by the rash analysis volume I fit almost all of the items, whereas volume II and volume III had all values in the range of statistical acceptance, the inter-subtest correlation analysis in all volumes had a high level of relationship, and the confirmatory factor analysis for all volumes was concordant with the empirical data. The evidence of examination on three methods based on the measurement standard for item response theory and classical test theory can indicate the quality of assessment of performance competence very well.

Keywords: ValidityPerformance competency;Rash analysisInter-subtest correlation analysisConfirmatory factor analysis

1. Introduction

Society is always changing; therefore, a successful organization has to be improved and developed

all the time. Organizational success depends on the drive of human resources. The organization which

has the ready, potentiality, and competent human resources for performance gets the advantage and has

the opportunity to achieve great success. Hence, the human factor is considered crucial factor for the

performance in the organization. The success of an organization, thus, depends on employee

performance as the crucial factor. The indicator leading to the success is essential for performance of

employees in the organization (Kesler, 1995; Beheshtifar, 2011). An individual will expose any

competency and usually depends on various components in terms of knowledge, skills/competency, and

other characteristics. These crucial components affect the competency of individuals (behavioral

characteristics of performance). Moreover, the competency also has a relationship with the

accomplishment (Office of the Civil Service Commission, 2005).

Competency is complicated; hence, the competency assessment should be done elaborately, deeply,

and with variation. This is in accordance with the concept of Baartman et al. (2006) who said that the

competency assessment is very complex, and that one single assessment method seems not to be

sufficient. Moreover, Koeppen et al. (2008) suggested that the competencies are conceptualized as

complex ability constructs. For the challenge point of this research about competency assessment, one

of the points is about psychological models (Frey & Hartig, 2009). In addition, Vazirani (2010)

represented the challenge of using competency models for measuring or appraising certain areas of

performance and providing developmental feedback based on these assessments. Thus, it can be seen

that competency is complex. It has to be assessed elaborately and with in-depth scrutiny. Only a single

method cannot be sufficient, and the interested point to be studied should be in terms of psychological

models.

Competency assessment has to be done indirectly; it cannot be measured directly but has to use

qualified instruments, and the qualified instruments have to be considered carefully in terms of validity.

According to the Educational Testing Service, ‘validity is the most important aspect of the quality of an

assessment’ (2002), and the American Educational Research Association et al. stated that ‘validity is

the most fundamental consideration in developing and evaluating tests’ (1999) According to the

definition of validity, from the Standards for Educational and Psychological Testing (as cited in AERA

et al., 1999), validity refers to “the degree to which evidence and theory support the interpretations of

test scores entailed by proposed uses of tests”. The validation process, therefore, involves the

accumulation of evidence to support the proposed test score interpretations and uses. The process of

accumulating evidence to support the validity of test score interpretations starts prior to the

development of an assessment (AERA et al., 1999). This suggests that the evidence of validity on the

instruments used to assess is the qualification which had to be considered at the first stage.

Regarding the backgrounds of this research, in terms of competency which had complexity,

competency assessment has to be done elaborately and with in-depth scrutiny. The importance of

assessment is that it has to show the evidence of validity by correct and accurate evaluation; thus,

various methods of assessment are necessary to be used in order to reach correctness and accuracy of

competency assessment. This will be the reflection of quality through competency assessment to lead to

measurement, assessment, and development of personnel for performance in the organization

effectively and successfully with sustainable performance.

2.Purposes of the study

The purpose of this research is to examine the validity of assessment of performance competency by

the methods of rash analysis, inter-subtest correlation analysis, and confirmatory factor analysis.

3.Research methods

3.1. Participants

1) The group of employees in the self-assessment is the TISCO company’s employees working

between 2013 and 2014. Tisco is a financial institute running the business with the highest standard of

good supervision. The Company is considered as a large company, consisting of Tisco Financial Group,

Tisco Bank, Tisco Securities, and Tisco Asset. The Company has been numerously rewarded for

success, which reflects the quality of the organization, such as the reward of Board of The Year 2013,

Top Corporate Governance Report Awards 2013, Outstanding Securities Company Awards 2013 –

Retail Investors, Outstanding Securities Company Awards 2013 – Institutional Investors, Outstanding

Investor Relations Awards 2013, “A” on Hewitt Best Employers in Thailand 2013, and Top Bank in the

Secondary Market for Corporate Bonds 2012 (Tisco, 2014). The subjects are from stratified random

sampling by having the strata pillar, as well as having the employees as the random unit from the

assessed employees for 3,929 persons.

2) The group of assessments by directors were obtained from purposive sampling. They are the

direct superiors of the employees.

3) The group of employees in the supervisor assessment consists of 516 TISCO Company’s

supervisors from stratified random sampling by having the strata pillar.

3.2. Instrumentation

content validity, by consideration of the experts, has the value of index of consistency between .50 and

1.00. The form is used by the employees who have to assess 52 supervisors. The value of item-total

correlation was between .22 and .53. The reliability was carried out by using the formula of Cronbach's

alpha coefficient equal to .93 and equal to Spearman-Brown coefficient .83.

3.3. Procedure

For the process of validity examination on instruments of three volumes, each volume is examined

by three methods: 1)Rash analysis — this method will examine the fundamental data of the uni-

dimensionality or multi-dimensionality by factors analysis considered by Eigen value, then it will be

analyzed by multi-dimensional random coefficients multinomial logit model (MRCMLM), which is

considered by INFIT mean square or weighted mean square (INFIT MNSQ), 2) Inter-subtest

correlation analysis — this can be analyzed by the relationship of five sub-competency scores and total

competency scores by using the Pearson correlation coefficient considered by the validity from

statistical significance, and 3) Confirmatory factor analysis — this is the confirmation of five

competency components considered by the concordance of the model with empirical data from the chi

square, goodness of fit index, adjusted goodness of fit index, root mean square residual, and root mean

square error of approximation

4.Findings

4.1. Rash analysis

4.1.1. Fundamental data examination

The analysis results of performance competency dimension by factors analysis was found that the

ratio between the Eigen value of factor 1 toward the Eigen value of factor 2 of the assessment by

directors was equal to 1.315, the self-assessment was equal to .001, and the supervisor assessment was

equal to 1.652. The values were less than the criteria, which would be concluded by the

unidimensionality. Morizot et al. (2007) proposed to consider the unidimensionality from the ratio

between Eigen value of factor1 toward the Eigen value of factor 2. If the value was more than or equal

to 3.00, it indicated the unidimensionality. With all evidence mentioned above, it can be concluded that

the performance competency assessed by the assessment by directors, self-assessment and supervisor

assessment were appropriate for multidimensionality assessment.

4.1.2. Research result of multidimensionality

According to this research, the researchers studied the base model called multidimensional random

coefficients multinomial logit model (MRCMLM) (Adams et al.,1997). Because there are five

competency assessment methods, with their fundamental characteristics about multi-dimensionality, it

fits with the analysis by the multidimensional form of the partial credit model. This is the model that

fits to the measurement having multi values response, personality measurement, metacognition and

attitudes measurement, and it fit to the measurement using the items of giving multi values scores by

ordered polytomous items (Ostini, & Nering, 2006). The data was analyzed by the ConQuest 2.0

computer program. The analysis results were concordant between the assessment model of performance

competency and question items considered by the INFIT mean square or weighted mean square (INFIT

MNSQ). These are the values based on the root mean square error (RMSE) concept between the

estimated and true item response (DeMars, 2004). The statistical values indicated that the data of each

item is concordant with the item response model determined, and does not accept the value between .60

and 1.40, which is the criteria for consideration of value measurement by rating scale (Wright et al.,

1994). The analysis result shown in Table 1 found that the assessment by the director had the INFIT

MNSQ value between .79 and 1.45. Only a single question item was beyond the acceptance range,

which was, COM2. Whereas the self-Assessment had the INFIT MNSQ value between .98 and 1.01,

and the supervisor assessment had the INFIT MNSQ value between .88 and 1.09, which its value was

in the acceptance range. This evidence shows the validity according to the theoretical construct of the

assessment of performance competency.

Table 1 -
See Full Size >

4.2. Inter-subtest correlation analysis

The construct validity is investigated by the inter subtest correlation method between the

dimensional or sub-test scores and total-test scores (Kline et al., 1994; Cohen, & Swerdlik, 2005; 2010;

Xi, 2008; Canivez et al., 2009). By using the Pearson correlation coefficient with the volume I,

assessment by director had the value between .675 and .940. The competency which had the highest

relationship with the total scores was LOG, secondly was BUS, LEA, COM, and PEO, respectively.

For the volume II, the self-Assessment had the value between .441 and .650. The competency which

had the highest relationship with the total score was LOG, secondly was COM, BUS, LEA, and PEO,

respectively. And the volume III, the supervisor assessment, the value was between .519 and .884. The

competency which had the highest relationship with the total score was LOG, secondly was BUS, LEA,

COM, and PEO, respectively. All values of all volumes had the statistical significance at .01 level, and

the relationship was at a high level. This evidence shows the validity according to the theoretical

construct of the assessment of performance competency as shown in Table 2 .

Table 2 -
See Full Size >

4.3. Confirmatory factor analysis

According to the analysis result of the models on assessment of performance competency, five major

indicators were found. These consisted of communication, people, logic, business and leadership within

the three volumes of the assessment form, When examining the concordance of performance

assessment models and empirical data, it was found that the assessment by director volume had the chi

square equal to 1.93 (df=2, p=.38), because the chi square was sensitive to the size of sample group.

Therefore, it had to mutually consider other statistical values. The goodness of fit index (GFI) was

equal to 1.00, the adjusted goodness of fit index (AGFI) was equal to 1.00, the root mean square

residual (RMR) was equal to .00, and the root mean square error of approximation (RMSEA) was equal

to .00. The self-assessment had the chi square = 5.65 (df = 2, p = .059), GFI = 1.00, AGFI = 1.00, RMR

= .00, RMSEA=.02, and the supervisor assessment had the chi square = 4.30 (df = 4, p = .37), GFI =

1.00, AGFI = .99, RMR = .01, RMSEA=.01. When considering the data according to the assessment

criteria of Bollen (1989), Kelloway (1998), and Hair et al. (2010) from the fit index value of the model,

the data showed that the assessment model of performance competency was concordant with the

empirical data. This evidence shows the validity according to the theoretical construct of assessment of

performance competency.

Table 3 -
See Full Size >

5. Conclusion

The validity examination result of assessment of performance competency was carried out by using

five competencies, which consisted of communication, personnel, logic, business, and leadership. The

examination was done by using the rash analysis method, which found that the volume I, assessment

by director, had the INFIT MNSQ value between .79 and 1.45 from all twenty-four items. Only a single

question item was beyond the acceptance range. For the volume II, self-assessment method, this had the

value of INFIT MNSQ between .98 and 1.01, and the volume III, supervisor assessment had the value

of INFIT MNSQ method between .88 and 1.09 which had the value in the range of statistical

acceptance. The examination by inter-subtest correlation analysis was found that the volume I had the

value between .675 and .940, the volume II had the value between .441 and .650, and the volume III

had the value between .519 and .884. The competency which had the highest relationship with the total

scores of all volumes was LOG. All values, of all volumes, had the statistical significance at .01 level,

and the relationship value was in a high level. For the examination that used the confirmatory factor

analysis method, it was found that the assessment model of performance competency of all volumes

was concordant with the empirical data. According to the evidence of examination on the three methods

based on the measurement standard for item response theory and classical test theory, these can indicate

the quality of assessment of performance competence very well.

6. Discussion

Regarding the instruments used for assessment created by the five aspects of competency, which are,

communication, personnel, logic, business, and leadership, for three volumes, consisting of the

assessment by director (Volume I), self-assessment (volume II), and the supervisor assessment (volume

III), the validity of these instruments were examined by three methods, which included the rash

analysis, inter-subtest correlation analysis, and confirmatory factor analysis.

The rash analysis results by considering the item fit value based on the concept of root mean square

error (RMSE) between estimated and true item response (DeMars, 2004) is the statistics value which

indicates whether the item data are concordant with the item response model. The item data, concordant

with the model, suggested that item has the response form concordant with the model in a form of

cumulative frequency curve (Chianchana, 2009), while only one question item was beyond the

statistical acceptance of Wright and others, 1994, which is, assessment by director (Volume I). Whereas

the question item of Com2 is “plan of representation by expecting that it can motivate others”, this

question item is still not concordant with the construct according to the item response theory; however,

this is considered as a small abnormality from the response structure, while other volumes showed that

every question item was concordant with the theoretical construct.

The result of inter-subtest correlation analysis for all volumes showed that all values had a highly

positive relationship. This was the evidence shown between the dimensional or sub-test scores and

total-test scores. These highly internal relationships reflect the validity of all series of assessment forms

developed under the same theory, considered as the aspect of homogeneity (Kline et al., 1994; Cohen,

& Swerdlik, 2005; 2010). All volumes have the logic competence related highly with the total

competence score. For the measured logic, it is the concept involved with the reason giving, and

reasonable reason criteria. The indicator consists of basic analysis, basic analysis and synthesis,

complex situations, and holistic and systematic. For the logic characteristic, Copi (2002) said that logic

is about reasoning on every subject — science and medicine, ethics and law, politics and commerce,

sports and games, and even the simple affairs of everyday life — which are considered as crucial

factors towards the performance.

The result of confirmatory factor analysis for all volumes showed that the assessment model was

concordant with the empirical data. The confirmatory factor analysis was then employed so that the

researcher could test the relationship between the model and the empirical data (McIntire and Miller,

2007). This means that all five structures of competency assessment forms were valid by the

assessment of structural models through the statistical index including chi-square test statistics,

goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI) , root mean squared residual

(RMR) , and root mean squared error of approximation (RMSEA). All indexes complied with a

consideration of Bollen (1989), Kelloway (1998) and Diamantopoulos and Siguaw (2000), who showed

the model congruence with empirical data (Bollen, 1989; Kelloway, 1998; Diamantopoulos, & Siguaw,

2000). Moreover, when the assessment of measurement model, through the consideration from the

value of indicator loading, which is the validity evidence (Diamantopoulos, & Siguaw, 2000), it

appeared that all loading volumes have statistical significance, which supports the validity of all

instruments for competency assessment. The crucial competency, when considering the loading value

in the assessment by director (Volume I) and supervisor assessment (volume III), is about the logic.

This is in accordance with the examination result of validity, by inter-subtest correlation analysis,

whereas the self-assessment (volume II) contradicts the result of inter-subtest correlation analysis.

Another crucial competency is communication, which is the personal characteristic which has fluency,

persuasion, presentation, and negotiation. These characteristics are crucial, but not more or less than the

logic. This is in accordance with the requirement of employers who need the employees who have

communication competency and presentation skills (Woodward et al., 2010). In addition, this is in

accordance with Kelley and Bridges’ study (2005), which showed that business communication skills

and presentation skills are the top two of twenty-three skills that are necessary for a successful career.

7. Recommendation

The instruments for competency assessment of performance have various volumes, and they are

conducted under the organizations which can control the examination effectively. For further research,

it should apply the technology to help originate the quickness, and saving. Moreover, it should be

created to be appropriate to the responders, until it builds the sustainability of the test. The conduction

should be done in terms of computer adaptive testing (CAT). Moreover, the nature within organizations

to have various groups of employees means that the instruments used for assessment should be

evaluated carefully to consider these differences of variance. For further research, the differential item

functioning (DIF), and differential test functioning (DTF) should be examined to show the evidence of

quality on the tests and questionnaires in terms of fairness through every group of responders.

Acknowledgement The research is financially supported by TISCO, Thailand. We would like to thank TISCO for this fund for the success of this research.

References

  1. Adams, R.Wilson, M.Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23
  2. American Educational Research Association., American Psychological Association., & National Council on Measurement in Education., (1999). Standards for educational and psychological ed.). , testing. (2nd
  3. (), Washington, DC: American Educational Research Association.
  4. Baartman, L. K. J.Bastiaens, T. J.Kirschner, P. A.Van Der Vleuten, C. P. M. (2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programs. Studies in Educational, 153-170, Evaluation. 32
  5. Beheshtifar, M. (2011). Role of career competencies in organizations. European Journal of Economics, Finance and Administrative Sciences, 42, 6-12
  6. Bollen, K. A. (1989). Structural equations with latent. , variables. New
  7. Chianchana, C. (2009). Multidimensional analysis. Journal of Education Khon Kaen University, 32(4), 13-22
  8. Cohen, R. J.Swerdlik, M. E. (2005). Psychological testing and assessment: An introduction to tests and measurement (6th ed.). Boston: McGraw-Hill.
  9. Cohen, R. J.Swerdlik, M. E. (2010). Psychological testing and assessment: An introduction to tests and measurements (7th ed.). Boston: McGraw-Hill.
  10. Copi, I. M. (2002). Introduction to logic (11th ed.). New Jersey: Pearson Education.
  11. DeMars, C. (2004). Measuring higher education outcomes with a multidimensional rasch model. Journal of Applied Measurement, 5(4), 350-361
  12. Diamantopoulos, A.Siguaw, J. A. (2000). Introducing LISREL: A guide for the SAGE Publication. , uninitiated. London
  13. Educational Testing Service (ETS)., (2002). ETS standards for quality and fairness.
  14. Frey, A.Hartig, J. (2009). Assessment of competencies. Studies in Educational Evaluation, 35, 55-56
  15. Hair, J. F.Anderson, R. E.Tatham, R. L.Black, W. C. (2010). Multivariate Data Analysis: A Global Perspective (7th ed.). New Jersey: Pearson education.
  16. Kellowey, E. (1998). Using LISREL for structural equation modeling: A researcher's guide.
  17. Kelly, C. A.Bridges, C. (2005). Introducing professional and career development skills in the marketing curriculum. Journal of Marketing Education, 27(3), 212-218
  18. Kesler, G. C. (1995). A model and process for redesigning the HRM role, competencies, and work in a major multi-national Resource Management. , 34, 229-252, corporation. Human
  19. Kline, R. B.Snyder, J.Gulmette, S.Castellanos, M. (1994). Evaluation of the construct validity of the kamphaus-reynolds supplementary scoring system for the K-ABC. Assessment, 1(3), 219-226
  20. Koeppen, K.Hartig, J.Klieme, E.and Leutner, E. (2008). Current issues in research on competency modeling and assessment. Zeitschrift fur Psychologie/Journal of Psychology, 216, 60-72
  21. McIntire, S. A.Miller, L. A. (2007). Foundations of psychological testing: A practical approach (2nd ed.).
  22. Morizot, J.Ainsworth, A. T.Reise, S. (2007). Toward modern psychometrics: Application of item response theory models’, In, R.C. Fraley, & R.F. Krueger. (Eds.), Handbook of Research Methods in Personality Psychology. , 407-423, R.W. Robins
  23. Office of the Civil Service Commission., (2005). Competency manual of civil service. , P.A. living
  24. html [29 September 2014] Vazirani, N. (2010). Competencies and competency model-A brief overview of its development and Journal of Management. , 7(1), 121-131, application. SIES
  25. Woodward, B.Sendall, P. (2010). Integrating soft skill competencies through project-based learning across the information systems curriculum. Information Systems Education Journal, 8(8), Retrieved from http://isedj.org
  26. Wright, B. D.Linacre, J. M.Gusafson, J. E.Martin-Lof, P. (1994). Reasonable mean-square fit values.
  27. htmXi, X. (2008). Methods of test E. Shohamy, and N.H. Hornberger. (Eds.), Encyclopaedia of Languate and Education: Vol.Language Testing and Assessment (pp.. 177-196), (2nd ed.). , 7, validation. In
  28. (), 212

Copyright information

This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

About this article

Cite this paper as:

Click here to view the available options for cite this article.

Publisher

Future Academy

First Online

18.12.2019

Doi

10.15405/epsbs.2016.05.21

Online ISSN

2357-1330