Validation of the Assessment of Performance Competency

Abstract

Competency, which has complexity, means that competency assessment has to be done elaborately and indepth. The importance of assessment is that it has to show the evidence of validity. Is assessment of performance competency by the methods of rash analysis, inter-subtest correlation analysis, and confirmatory factor analysis valid or not. The purpose of research is to examine the validity of assessment of performance competency by the methods of rash analysis, inter-subtest correlation analysis, and confirmatory factor analysis. The sample group consisted of directors and employees. The instruments included three volumes - assessment by director (Volume I), self-assessment (volume II), and supervisor assessment (volume III). The data were analyzed by the rash analysis, inter-subtest correlation analysis, and confirmatory factor analysis. Results revealed that the examination by the rash analysis volume I fit almost all of the items, whereas volume II and volume III had all values in the range of statistical acceptance, the inter-subtest correlation analysis in all volumes had a high level of relationship, and the confirmatory factor analysis for all volumes was concordant with the empirical data. The evidence of examination on three methods based on the measurement standard for item response theory and classical test theory can indicate the quality of assessment of performance competence very well.

Keywords: ValidityPerformance competency;Rash analysisInter-subtest correlation analysisConfirmatory factor analysis

1. Introduction

Society is always changing; therefore, a successful organization has to be improved and developed

all the time. Organizational success depends on the drive of human resources. The organization which

has the ready, potentiality, and competent human resources for performance gets the advantage and has

the opportunity to achieve great success. Hence, the human factor is considered crucial factor for the

performance in the organization. The success of an organization, thus, depends on employee

performance as the crucial factor. The indicator leading to the success is essential for performance of

employees in the organization (Kesler, 1995; Beheshtifar, 2011). An individual will expose any

competency and usually depends on various components in terms of knowledge, skills/competency, and

other characteristics. These crucial components affect the competency of individuals (behavioral

characteristics of performance). Moreover, the competency also has a relationship with the

accomplishment (Office of the Civil Service Commission, 2005).

Competency is complicated; hence, the competency assessment should be done elaborately, deeply,

and with variation. This is in accordance with the concept of Baartman et al. (2006) who said that the

competency assessment is very complex, and that one single assessment method seems not to be

sufficient. Moreover, Koeppen et al. (2008) suggested that the competencies are conceptualized as

complex ability constructs. For the challenge point of this research about competency assessment, one

of the points is about psychological models (Frey & Hartig, 2009). In addition, Vazirani (2010)

represented the challenge of using competency models for measuring or appraising certain areas of

performance and providing developmental feedback based on these assessments. Thus, it can be seen

that competency is complex. It has to be assessed elaborately and with in-depth scrutiny. Only a single

method cannot be sufficient, and the interested point to be studied should be in terms of psychological

models.

Competency assessment has to be done indirectly; it cannot be measured directly but has to use

qualified instruments, and the qualified instruments have to be considered carefully in terms of validity.

According to the Educational Testing Service, ‘validity is the most important aspect of the quality of an

assessment’ (2002), and the American Educational Research Association et al. stated that ‘validity is

the most fundamental consideration in developing and evaluating tests’ (1999) According to the

definition of validity, from the Standards for Educational and Psychological Testing (as cited in AERA

et al., 1999), validity refers to “the degree to which evidence and theory support the interpretations of

test scores entailed by proposed uses of tests”. The validation process, therefore, involves the

accumulation of evidence to support the proposed test score interpretations and uses. The process of

accumulating evidence to support the validity of test score interpretations starts prior to the

development of an assessment (AERA et al., 1999). This suggests that the evidence of validity on the

instruments used to assess is the qualification which had to be considered at the first stage.

Regarding the backgrounds of this research, in terms of competency which had complexity,

competency assessment has to be done elaborately and with in-depth scrutiny. The importance of

assessment is that it has to show the evidence of validity by correct and accurate evaluation; thus,

various methods of assessment are necessary to be used in order to reach correctness and accuracy of

competency assessment. This will be the reflection of quality through competency assessment to lead to

measurement, assessment, and development of personnel for performance in the organization

effectively and successfully with sustainable performance.

2.Purposes of the study

The purpose of this research is to examine the validity of assessment of performance competency by

the methods of rash analysis, inter-subtest correlation analysis, and confirmatory factor analysis.

3.Research methods

4.Findings

5. Conclusion

The validity examination result of assessment of performance competency was carried out by using

five competencies, which consisted of communication, personnel, logic, business, and leadership. The

examination was done by using the rash analysis method, which found that the volume I, assessment

by director, had the INFIT MNSQ value between .79 and 1.45 from all twenty-four items. Only a single

question item was beyond the acceptance range. For the volume II, self-assessment method, this had the

value of INFIT MNSQ between .98 and 1.01, and the volume III, supervisor assessment had the value

of INFIT MNSQ method between .88 and 1.09 which had the value in the range of statistical

acceptance. The examination by inter-subtest correlation analysis was found that the volume I had the

value between .675 and .940, the volume II had the value between .441 and .650, and the volume III

had the value between .519 and .884. The competency which had the highest relationship with the total

scores of all volumes was LOG. All values, of all volumes, had the statistical significance at .01 level,

and the relationship value was in a high level. For the examination that used the confirmatory factor

analysis method, it was found that the assessment model of performance competency of all volumes

was concordant with the empirical data. According to the evidence of examination on the three methods

based on the measurement standard for item response theory and classical test theory, these can indicate

the quality of assessment of performance competence very well.

6. Discussion

Regarding the instruments used for assessment created by the five aspects of competency, which are,

communication, personnel, logic, business, and leadership, for three volumes, consisting of the

assessment by director (Volume I), self-assessment (volume II), and the supervisor assessment (volume

III), the validity of these instruments were examined by three methods, which included the rash

analysis, inter-subtest correlation analysis, and confirmatory factor analysis.

The rash analysis results by considering the item fit value based on the concept of root mean square

error (RMSE) between estimated and true item response (DeMars, 2004) is the statistics value which

indicates whether the item data are concordant with the item response model. The item data, concordant

with the model, suggested that item has the response form concordant with the model in a form of

cumulative frequency curve (Chianchana, 2009), while only one question item was beyond the

statistical acceptance of Wright and others, 1994, which is, assessment by director (Volume I). Whereas

the question item of Com2 is “plan of representation by expecting that it can motivate others”, this

question item is still not concordant with the construct according to the item response theory; however,

this is considered as a small abnormality from the response structure, while other volumes showed that

every question item was concordant with the theoretical construct.

The result of inter-subtest correlation analysis for all volumes showed that all values had a highly

positive relationship. This was the evidence shown between the dimensional or sub-test scores and

total-test scores. These highly internal relationships reflect the validity of all series of assessment forms

developed under the same theory, considered as the aspect of homogeneity (Kline et al., 1994; Cohen,

& Swerdlik, 2005; 2010). All volumes have the logic competence related highly with the total

competence score. For the measured logic, it is the concept involved with the reason giving, and

reasonable reason criteria. The indicator consists of basic analysis, basic analysis and synthesis,

complex situations, and holistic and systematic. For the logic characteristic, Copi (2002) said that logic

is about reasoning on every subject — science and medicine, ethics and law, politics and commerce,

sports and games, and even the simple affairs of everyday life — which are considered as crucial

factors towards the performance.

The result of confirmatory factor analysis for all volumes showed that the assessment model was

concordant with the empirical data. The confirmatory factor analysis was then employed so that the

researcher could test the relationship between the model and the empirical data (McIntire and Miller,

2007). This means that all five structures of competency assessment forms were valid by the

assessment of structural models through the statistical index including chi-square test statistics,

goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI) , root mean squared residual

(RMR) , and root mean squared error of approximation (RMSEA). All indexes complied with a

consideration of Bollen (1989), Kelloway (1998) and Diamantopoulos and Siguaw (2000), who showed

the model congruence with empirical data (Bollen, 1989; Kelloway, 1998; Diamantopoulos, & Siguaw,

2000). Moreover, when the assessment of measurement model, through the consideration from the

value of indicator loading, which is the validity evidence (Diamantopoulos, & Siguaw, 2000), it

appeared that all loading volumes have statistical significance, which supports the validity of all

instruments for competency assessment. The crucial competency, when considering the loading value

in the assessment by director (Volume I) and supervisor assessment (volume III), is about the logic.

This is in accordance with the examination result of validity, by inter-subtest correlation analysis,

whereas the self-assessment (volume II) contradicts the result of inter-subtest correlation analysis.

Another crucial competency is communication, which is the personal characteristic which has fluency,

persuasion, presentation, and negotiation. These characteristics are crucial, but not more or less than the

logic. This is in accordance with the requirement of employers who need the employees who have

communication competency and presentation skills (Woodward et al., 2010). In addition, this is in

accordance with Kelley and Bridges’ study (2005), which showed that business communication skills

and presentation skills are the top two of twenty-three skills that are necessary for a successful career.

7. Recommendation

The instruments for competency assessment of performance have various volumes, and they are

conducted under the organizations which can control the examination effectively. For further research,

it should apply the technology to help originate the quickness, and saving. Moreover, it should be

created to be appropriate to the responders, until it builds the sustainability of the test. The conduction

should be done in terms of computer adaptive testing (CAT). Moreover, the nature within organizations

to have various groups of employees means that the instruments used for assessment should be

evaluated carefully to consider these differences of variance. For further research, the differential item

functioning (DIF), and differential test functioning (DTF) should be examined to show the evidence of

quality on the tests and questionnaires in terms of fairness through every group of responders.

Acknowledgement The research is financially supported by TISCO, Thailand. We would like to thank TISCO for this fund for the success of this research.

References

1. Adams, R.Wilson, M.Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23
2. American Educational Research Association., American Psychological Association., & National Council on Measurement in Education., (1999). Standards for educational and psychological ed.). , testing. (2nd
3. (), Washington, DC: American Educational Research Association.
4. Baartman, L. K. J.Bastiaens, T. J.Kirschner, P. A.Van Der Vleuten, C. P. M. (2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programs. Studies in Educational, 153-170, Evaluation. 32
5. Beheshtifar, M. (2011). Role of career competencies in organizations. European Journal of Economics, Finance and Administrative Sciences, 42, 6-12
6. Bollen, K. A. (1989). Structural equations with latent. , variables. New
7. Chianchana, C. (2009). Multidimensional analysis. Journal of Education Khon Kaen University, 32(4), 13-22
8. Cohen, R. J.Swerdlik, M. E. (2005). Psychological testing and assessment: An introduction to tests and measurement (6th ed.). Boston: McGraw-Hill.
9. Cohen, R. J.Swerdlik, M. E. (2010). Psychological testing and assessment: An introduction to tests and measurements (7th ed.). Boston: McGraw-Hill.
10. Copi, I. M. (2002). Introduction to logic (11th ed.). New Jersey: Pearson Education.
11. DeMars, C. (2004). Measuring higher education outcomes with a multidimensional rasch model. Journal of Applied Measurement, 5(4), 350-361
12. Diamantopoulos, A.Siguaw, J. A. (2000). Introducing LISREL: A guide for the SAGE Publication. , uninitiated. London
13. Educational Testing Service (ETS)., (2002). ETS standards for quality and fairness.
14. Frey, A.Hartig, J. (2009). Assessment of competencies. Studies in Educational Evaluation, 35, 55-56
15. Hair, J. F.Anderson, R. E.Tatham, R. L.Black, W. C. (2010). Multivariate Data Analysis: A Global Perspective (7th ed.). New Jersey: Pearson education.
16. Kellowey, E. (1998). Using LISREL for structural equation modeling: A researcher's guide.
17. Kelly, C. A.Bridges, C. (2005). Introducing professional and career development skills in the marketing curriculum. Journal of Marketing Education, 27(3), 212-218
18. Kesler, G. C. (1995). A model and process for redesigning the HRM role, competencies, and work in a major multi-national Resource Management. , 34, 229-252, corporation. Human
19. Kline, R. B.Snyder, J.Gulmette, S.Castellanos, M. (1994). Evaluation of the construct validity of the kamphaus-reynolds supplementary scoring system for the K-ABC. Assessment, 1(3), 219-226
20. Koeppen, K.Hartig, J.Klieme, E.and Leutner, E. (2008). Current issues in research on competency modeling and assessment. Zeitschrift fur Psychologie/Journal of Psychology, 216, 60-72
21. McIntire, S. A.Miller, L. A. (2007). Foundations of psychological testing: A practical approach (2nd ed.).
22. Morizot, J.Ainsworth, A. T.Reise, S. (2007). Toward modern psychometrics: Application of item response theory models’, In, R.C. Fraley, & R.F. Krueger. (Eds.), Handbook of Research Methods in Personality Psychology. , 407-423, R.W. Robins
23. Office of the Civil Service Commission., (2005). Competency manual of civil service. , P.A. living
24. html [29 September 2014] Vazirani, N. (2010). Competencies and competency model-A brief overview of its development and Journal of Management. , 7(1), 121-131, application. SIES
25. Woodward, B.Sendall, P. (2010). Integrating soft skill competencies through project-based learning across the information systems curriculum. Information Systems Education Journal, 8(8), Retrieved from http://isedj.org
26. Wright, B. D.Linacre, J. M.Gusafson, J. E.Martin-Lof, P. (1994). Reasonable mean-square fit values.
27. htmXi, X. (2008). Methods of test E. Shohamy, and N.H. Hornberger. (Eds.), Encyclopaedia of Languate and Education: Vol.Language Testing and Assessment (pp.. 177-196), (2nd ed.). , 7, validation. In
28. (), 212