Development of the Indicators and Instrument to Evaluate the Effectiveness of the Professional Nurse Training Project

Abstract

Professional nurse training is very important aspect for enhancing the effectiveness of continuing nursing education and to accumulate continuing nursing education unit for the renew of license; however, there are no specific the indicators and the measure instruments for the effective solution evidence in Thailand. The purposes of this research sought to develop the indicators and instrument to evaluate the effectiveness of the professional nurse training project, by using two approaches. An application of Kirkpatrick’s model focused on training and previous researches. Theory-based evaluation approach was built from literature reviews. The first phase of the study was the developing of the indicators and instrument, on the other hand of the second phase was the building instrument to evaluation effectiveness of training project and testing. This research proposes a comprehensive and systematic approach with a discussion of methodology related to developing of the indicators and instrument to evaluate on the professional nurse training project effectiveness in all steps of assessment, implementation and evaluation.

Keywords: Professional nursetraining projecttheory-based evaluationeffectiveness

Introduction

The training nurse assumes great responsibility in the effective and efficient implementation of the educational activities in health care (Atkins et al., 2014; Kaya Işik & Çalişkan, 2013; Mack, 2014; TNC, 2015) and professional nurse necessary to be on training for enhancement knowledge, professional or academic ability and accumulation continuing nursing education unit to renew license (TNC, 2015). Assessing training has historically been operationally oriented, emphasizing financial justification, content and quality of the training rather than the potential of improved project performance or organizational transformation (Divyaranjani & Rajasekar, 2014; Lee-Kelley, & Blackman, 2012). Unfortunately, post-training is not have tracked the success or effectiveness. Or even with the rate is usually assessed after training without monitoring behavior in workplace of trainees. Therefore, this training has not worth and it should be used to evaluate the training accordingly to reduce waste. Research aims needed to develop of the indicators and instrument to evaluate that it could be helpful and fruitful to stakeholders for their successful effectiveness of the implementation on their own training.

The research issue is the selection approaches to develop the indicators and instrument which evaluated the effectiveness of the professional nurse training project, because of their differentiated restriction. Kirkpatrick & Kirkpatrick (2006) proposed to evaluating training programs and focuses on four fundamental elements of effective evaluation: reaction, learning, behavior, and results. Actually behavior element is more difficult than reaction and learning elements because of the needs of following assessments in real workplace of the trainees. After training and matches the training the trainees usually occur on the outside of the training program and consequences ensue (Fraser, 2014; Praslova, 2010). It also provides an assessment their impact on the organization this was the most difficult to evaluate. Because of there are other variables much more than the training that is affecting the organization. The researchers did not assess the full training as Kirkpatrick’s model.

The highlight of the evaluation of the concept of training Kirkpatrick’s model will be employed to evaluate the effectiveness of training programs for nurses. It also the evaluation of the concept of training Kirkpatrick’s model is still limited information about behavioral changes after training and outcome of the organization. The literature reveals a lack of information that shows the causal factors affected the success of the training project. The idea is to provide supplemental information for appropriate approach by using theory-based evaluation which can be explained what factors were responsible for the program’s success or failure (Chen, 1990, Weiss, 1997).

This research is an integrating concept and applied Kirkpatrick’s model and theory-based evaluation approaches. An evaluation will the explanation the factors that stakeholders recognized the success factors of training in each of the assessed valuation of the reaction, learning, behavior and results. And the result of any barriers due to the effectiveness of the training are not occur for the program or nurses who were trained only. Furthermore, there are also factors about the self-efficacy and organizational support. Thus, efforts to find the factors that promote behavior of nurses who participated in the training will have a broader and deeper to gain information on how to improve the implementation of future training project (Donaldson, 2007; Kirkpatrick, & Kirkpatrick, 2006; Nakrošis, 2014) as well as into the strategies for management effectively serve their multiple stakeholders. Recognition of the training methods and measurement techniques are crucial for the organization’s training success (Alipour et al., 2009). But the approaches involving the combination of this integrating concept is currently just a proposal. Therefore, this research aims to develop the indicators and instrument to evaluate the effectiveness of the professional nurse training project by using an application of Kirkpatrick’s model and theory-based evaluation approaches.

Literature Reviews

Topics in the development of the indicators and instrument to evaluate the effectiveness of the professional nurse training project, the researcher reviews the essential topics as the followings;

2.1 Evaluating training programs

According to Kirkpatrick & Kirkpatrick (2006), the evaluating training programs consisted of four fundamental elements of effective evaluation: reaction, learning, behavior, and results. The previous research of the training was shown to lead to positive changes in the nurses' attitudes, knowledge and clinical practice. (Ballangrud, 2013; Pesiridis et al., 2015; Redhead et al., 2011). According to a meta-analysis by (Kalinoski et al., 2013) from 65 studies (N = 8465) revealed sizable effects on affective-based, cognitive-based, and skill-based outcomes. And the meta-analysis from 67 studies suggest that training were positively related to human resource outcomes and organizational performance (Tharenou et al., 2007)

2.2 Theory-based evaluation

According to Chen (1990), The theory-based evaluation approach can be explained what factors were responsible for the program’s success or failure (Birckmayer & Weiss, 2000; Chen, 1990; Chen, 2005; Coryn et al., 2011). The development of the indicators and instrument to evaluate the effectiveness of the professional nurse training project is an important process in the theory-based evaluation approach. Its needs was to be built from literature reviews. Self-efficacy contributed to cognitive development and functioning (Bandura, 1993., Sousa et al., 2012). Employee's self-efficacy has been related with the importance one of organizational outcomes such as service quality and job performance. Self-efficacy correlates positively very strong relationship with job performance,job satisfaction and behavioral outcomes (Khanna, 2012) And perceived organizational support-POS(Eisenberger et al., 1986), the previous evidence of a relationship between perceived organizational support and work outcomes such as job satisfaction, organizational commitment and supervisor support can be seen in a 2002 meta analysis by Rhoades and Eisenberger and from Riggle et al. (2009).

2.3 Design of evaluation effectiveness of the professional nurse training project aspects

Effective training enhances the knowledge, skills, attitudes and behavior of people and hence their performance (Alipour et al., 2009; Chen, 2013; Divyaranjani, & Rajasekar, 2014). Human resource development through training investment by the organization to develop human capital. Therefore, it must be evaluated training programs to improve the assessment. A competency development training and increasing the performance of employees and organizations (Borate et al., 2014; Falola et al., 2014; Siengthai et al., 2014;). The effectiveness of the training must result in behavioral changes, such as the transfer of knowledge and skills gained from the training into practice in order to achieve efficiency in the organization. (Borate et al., 2014; Chen, 2013). According to a meta-analysis by Taylor, Russ-Eft, & Chan, (2005) from 117 studies evaluated the effects of behavior modeling training (BMT) on 6 training outcomes, BMT effects were largest for learning outcomes, smaller for job behavior, and smaller still for results outcomes. The goal of this study is to development the indicators and instrument to evaluate the effectiveness of the professional nurse training project, by using two approaches. An application of Kirkpatrick’s model and theory-based evaluation approaches.

Research Methodology

A mixed method design was used in this research was divided into two phases:

The Phase I is the study of the developing of the indicators and instrument. The development was based on documents evaluation, for examples the success factors of training in each of the assessed valuation of the reaction, learning, behavior and results. The development of the instrument involved three steps (definition and content domain construction; item generation; and face validity examination), by the method of the focus groups discussion and in-depth interview style were used.A review of the instrument by seven experts were also obtained for the content validity evidences.

The Phase II was the building of instrument for evaluation effectiveness of the professional nurse training project and testing instrument. This phase consisted solely of testing the instrument developed during the Phase I. Research involved the pilot testing and the findings showed Cronbach’s alpha for the total scale was .98, while the range of Cronbach’s alpha for each component ranged from .95 to .98. Almost items (n = 30; 100%) had corrected item-total correlations between 0.25 and 0.81. As a result of the pilot testing, all instrument items were retained for future testing.

Research results

Taken together, the research results illustrated that:

4.1 Professional nurse training project success indicators

The indicators or indices used to assess the success indicators for professional nurse training project consisted of 6 components and 25 indicators as shown in Table 1 .

4.2 Conceptual framework

The conceptual framework for evaluating the successful effectiveness of the professional nurse training project in Thailand consisted of reaction, learning, behavior, results, self-efficacy and organizational support. Such factors are interrelated as shown in Fig. 1 .

See Full Size >

Conclusion and recommendations

Research results was the development of the indicators and instrument to evaluate the effectiveness of the professional nurse training project by application of Kirkpatrick’s four level model of training and theory-based evaluation approaches. Research found that (1) the intervention in terms of the professional nurse training project including educational activities (2) the determinant including reaction, learning, behavior, self-efficacy, and organization support (3) the outcomes including results. These components of the conceptual framework will be used as the indicators of success in term of the evaluation, and will consist of 25 indicators. The further research should be focused on the evaluation of the effectiveness of the professional nurse training project in all steps of assessment, implementation and evaluation.

Acknowledgements

The researchers would like to thank the Faculty of Medicine Ramathibidi Hospital, Mahidol University and Faculty of Education, Burapha University for funding this research. And special thanks for the dedicated involvement of Prof. Dr. Tongchai Sombon, Ramkhamhaeng University a primary writer and consultant throughout the development of this guide and editorial assistance.

References

1. Alipour, M., Salehi, M., & Shahnavaz, A. (2009). A Study of on the Job Training Effectiveness: Empirical Evidence
2. of Iran. International Journal of Business and Management, 4(11), p63.
3. Atkins, M. S., Strauman, T. J., Cyranowski, J. M., & Kolden, G. G. (2014). Reconceptualizing internship training
4. within the evolving clinical science training model. Clinical Psychological Science, 2(1), 46-57.
5. Ballangrud, R. (2013). Building patient safety in intensive care nursing: Patient safety culture, team performance
6. and simulation-based training.  Doctoral dissertation, Karlstad University Studies.
7. Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning. Educational psychologist,
8. 28(2), 117-148.
9. Birckmayer, J. D., & Weiss, C. H. (2000). Theory-Based Evaluation in Practice What Do We Learn?. Evaluation review,
10. 24(4), 407-431.
11. Borate, N. S., Gopalkrishna, B., Shiva Prasad, H. C., & Borate, S. L. (2014). A Case Study Approach for
12. Evaluation of Employee Training Effectiveness and Development Program. International Journal of Business
13. & Management, 2(6), 201-210.
14. Chen, H. T. (1990). Theory-driven evaluation. Thousand Oaks, CA: Sage.
15. Chen, H. T. (2005). Practical Program Evaluation: Assessing and Improving Planning, Implementation, and
16. Effectiveness. Thousand Oaks, CA: Sage.
17. Chen, Y. C. (2013). The effects of transfer of training and organizational culture on job satisfaction. Actual
18. Problems of Economics, (3), 484-492.
19. Coryn, C. L., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven
20. evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199-226.
21. Divyaranjani, R., & Rajasekar, D. (2014). Effectiveness of Training and Development Practices in an
22. Organization: An Evaluation. AMET International Journal of Management, 45-60.
23. Donaldson, S. I. (2007). Program Theory-Driven Evaluation Science: Strategies and Applications. New York,
24. NY: Lawrence Erlbaum.
25. Eisenberger, R., Huntington, R., Hutchison, S., & Sowa, D. (1986). Perceived organizational support. Journal of
26. Applied Psychology, 71, 500 –507.
27. Falola, H. O., Osibanjo, A. O., & Ojo, I. S. (2014). Effectiveness of Training and Development on Employees'
28. Performance and Organisation Competitiveness in the Nigerian Banking Industry. Bulletin of the Transilvania
29. University of Braşov, 7(1), 161.
30. Fraser, D. A. (2014). Trainer and Trainee Perceptions of a Knowledge Network Site Evaluating Training
31. Effectiveness: A Case Study (Doctoral dissertation, Northcentral University).
32. Kalinoski, Z. T., Steele‐Johnson, D., Peyton, E. J., Leas, K. A., Steinke, J., & Bowling, N. A. (2013).
33. A meta‐analytic evaluation of diversity training outcomes. Journal of Organizational Behavior, 34(8), 1076-1104.
34. Kaya, H., Işik, B., & Çalişkan, F. (2013). Metaphors used by in‐service training nurses for in‐service training
35. nurses in Turkey. Journal of advanced nursing, 69(2), 404-414.
36. Khanna, S. (2012). Self efficacy work commitment and perceived organizational support as predictors of work
37. outcomes. Department of Psychology, Kurukshetra University, Kurukshetra.
38. Kirkpatrick, D. & Kirkpatrick, J. (2006). Evaluating Training Programs: The Four Levels, 3rd edn. San Francisco.
39. Berrett-Koehler Publishers, Inc.,
40. Lee-Kelley, L., & Blackman, D. (2012). Project training evaluation: Reshaping boundary objects and assumptions.
41. International Journal of Project Management, 30(1), 73-82.
42. Mack, J. P. (2014). Where do allies come from? An evaluation of a university Safe Zone program. Doctoral
43. dissertation, Stata University of New York at Buffalo.
44. Nakrošis, V. (2014). Theory-based evaluation of capacity-building interventions. Evaluation, 20(1), 134-150.
45. Pesiridis, T., Sourtzi, P., Galanis, P., & Kalokairinou, A. (2015). Development, implementation and evaluation of a
46. disaster training programme for nurses: A Switching Replications randomized controlled trial. Nurse education in
47. practice, 15(1), 63-67.
48. Praslova, L. (2010). Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning
49. outcomes and program evaluation in higher education. Educational Assessment, Evaluation and Accountability,
50. 22(3), 215-225.
51. Redhead, K., Bradshaw, T., Braynion, P., & Doyle, M. (2011). An evaluation of the outcomes of psychosocial
52. intervention training for qualified and unqualified nursing staff working in a low‐secure mental health unit.
53. Journal of psychiatric and mental health nursing, 18(1), 59-66.
54. Rhoades, L., & Eisenberger, R. (2002). Perceived organizational support: a review of the literature. Journal of
55. applied psychology, 87(4), 698.
56. Riggle, R. J., Edmondson, D. R., & Hansen, J. D. (2009). A meta-analysis of the relationship between perceived
57. organizational support and job outcomes: 20 years of research. Journal of Business Research, 62(10), 1027-1030.
58. Siengthai, S., Boonkrob, D., & Siengthai, M. (2014). Evaluation of Training Effectiveness: A Case Study of the
59. Ministry of Interior, Thailand. Comprehensive Research Journal of Management and Business Studies
60. (CRJMBS), 2 (2), 021 -028.
61. Sousa, C. M., Coelho, F., & Guillamon‐Saorin, E. (2012). Personal Values, Autonomy, and Self‐efficacy:
62. Evidence from frontline service employees. International Journal of Selection and Assessment, 20(2), 159-170.
63. Taylor, P. J., Russ-Eft, D. F., & Chan, D. W. (2005). A meta-analytic review of behavior modeling training.
64. Journal of Applied Psychology, 90(4), 692.
65. Thailand Nursing and Midwifery Council(TNC). (2015, January 9). Nursing and Midwifery Education and
66. Accreditation. Retrieved from http://www.tnc.or.th/en/content/
67. Tharenou, P., Saks, A. M., & Moore, C. (2007). A review and critique of research on training and organizational-
68. level outcomes. Human Resource Management Review, 17(3), 251-273.
69. Weiss, C. H. (1997). How can theory-based evaluation make greater headway?. Evaluation review, 21(4), 501-524.