Content Validity of MELLE Instrument for Malaysian ESL Learners Using CVR Method
Abstract
With new paradigm of positive psychology in language teaching and learning, enjoyment in learning English as a second language (ESL) has been identified as a contributing factor to success in language learning. The pursuit to comprehend enjoyment of ESL learning leads to an effort to develop and validate a psychometric instrument known as Malaysian ESL Learners’ Language Learning Enjoyment (MELLE) Instrument. A vital step for ensuring the quality of a produced psychometric instrument is the computation of Content Validation Ratio (CVR). This study utilized Lawshe's Content Validity Ratio (CVR) methodology to investigate the content validity MELLE Instrument. Employing purposive sampling technique, 12 professionals and lay-experts were appointed and consulted to assess MELLE instrument. The MELLE instrument consists of 6 primary constructs represented by 140 items. The findings indicated that 87 items had at least met the minimum requirements of overall CVR value (CVR≥0.566), while 57 items were eliminated from the relevant instrument. The research results also open for new opportunities for educators and researchers, who can utilize the psychometric instrument to assess enjoyment among Malaysian ESL learners and improve instructional practices for successful ESL teaching and learning.
Keywords: Content validity, content validity ratio, enjoyment in learning ESL, instrument
Introduction
Positive psychology has been essentially defined as the scientific and empirical study of what makes life worthwhile, enhance individual strength, the processes by which individuals achieve success and flourish, develops individuals who are aware of their personal virtues, and the condition of recognising what goes right in life (Csikszentmihalyi & Nakamura, 2011; Peterson, 2006; Seligman & Csikszentmihalyi, 2000; Snyder et al., 2016). In term of language teaching and learning, it could be observed that researchers and scholars have highlighted psychology and emotion in language learning for a period of time especially on negative psychology. Positive psychology in the field of teaching and learning language has been emphasized recently and it serves as a springboard for these fields to make a progress towards the development of positive psychology (Mercer & MacIntyre, 2014).
The development of positive psychology has turned attention to the intricate role that positive emotion serves in second and foreign languages (Lake, 2013; Seligman & Csikszentmihalyi, 2000). Recently, the researchers' focus has been on enjoyment in particular (Dewaele & Dewaele, 2017) and it is acknowledged that one of the typical emotions experienced by language learners themselves is enjoyment (Gregersen et al., 2017). Enjoyment in language learning is viewed as a as a trait-like emotional experience and a state reaction originated from learners’ learning experience that serves their learning needs, and from the process of extending themselves to gain new experiences particularly when they encounters tasks that contributes to the sense of novelty, accomplishment and self-growth in the process of learning (Dewaele & MacIntyre, 2014; Dewaele & MacIntyre, 2016; Elahi Shirvan et al., 2020; Goetz et al., 2006; Seligman & Csikszentmihalyi, 2000).
Consequently, studies on enjoyment in language learning are thought to be significant and to have a variety of consequences. To date, learners' enjoyment of language learning, which has enhanced their speaking abilities and confidence, has favourably influenced their readiness to communicate in the language classroom (Dewaele, 2019). Additionally, improved language learning ability and accomplishment have been linked to enjoyment of language learning (Dewaele & Alfawzan, 2018). As enjoyment is claimed to outweigh anxiety in language learning, enjoyment may act as a protective role against negative emotions (MacIntyre, 2016). Enjoyment is also reported to play a major role to enhance learners’ motivation, particularly in the L2-Self and Ought-2-self domains (Lee & Lee, 2020) and contribute to positive effect on language learners’ emotional intelligence (Li, 2020). Besides, it is also reported that learners’ favoured choices and practices of learning strategy is associated with learning strategies that they enjoyed (Alfian, 2021).
In conjunction with the attention of enjoyment in language learning, researchers have initiated to focus on the psychometric instrument in measuring enjoyment in foreign language from different language perspectives and cultural contexts. The foundation of the first Foreign Language Enjoyment Scale, which has 21 items, was created by (Dewaele & MacIntyre, 2014) by adapting seven items from the Interest/Enjoyment subscale of (Ryan et al., 1990). The CFLES, also known as the Foreign Language Enjoyment Scale for Chinese Learners, is the next instrument developed by Li et al. (2018). The modified Chinese version of the 14-item, adapting two-factor Foreign Language Enjoyment Scale is called CFLES. Following the development of CFLES is English Classroom Enjoyment Scales (ECES) by Jin and Zhang (2018). ECES was specifically created to be used in English classes with Chinese students.
The first form of validity that will be evaluated in the development of a new instrument is content validity, which is a crucial component that must be addressed (DeVellis, 2017). The quantitative approach created by Lawshe (1975) is termed as the Content Validation Ratio (CVR), and it was chosen to quantify content validation for the MELLE instrument. CVR is employed to assess the mutual agreement among the appointed panel of experts on the significance of particular items for instrument development and to examine how well the items reflect the instrument's domain construct. Additionally, a three-point rating system was used to ask the experts to assess and validate the significance of the items namely (1) essential, (2) useful but not essential, and (3) not essential.
In order to guarantee that the measurement tool is measuring what it ought to measure, content validity is essential. Only pertinent and necessary items that adequately address the construct of the instrument should be included in a test with strong content validity (Nunnally & Bernstein, 1994). Besides that, other than being simple to administer and fast to implement, the CVR approach is one from the traditional measuring literatures that is more practical in terms of time and expenses (Noor et al., 2016; Tojib, 2006). It is a straightforward strategy to use because CVR includes a table for determining the essential cut-off value and emphasising the statistical significance of agreement at the item (Wilson et al., 2012). Due to the reasons, CVR is a validation method that had been selected by local researchers (Ab Aziz et al., 2019; Chong et al., 2021; Effendi et al., 2015; Ramli et al., 2018) and foreign researchers (Eskandari et al., 2018; Nazarnia et al., 2022; Shrotryia & Dhanda, 2019).
Problem Statement
The necessity for more research on positive emotion is justified by the fact that it aids in language learning by promoting wellbeing and offering advantages for the cognition and motivation (MacIntyre & Gregersen, 2012; Oxford, 2016). However, studies on language learning enjoyment in second and foreign languages have concentrated on its conception (Zeng, 2021) , assessment (Dewaele & MacIntyre, 2014; Jin & Zhang, 2019; Li et al., 2018) and factors as well as a variety of cultural, socioeconomic, and psychological aspects (Chin et al., 2022). This study, however, was motivated by a number of gaps in the literature, notably the absence of a psychometric measuring the enjoyment of language learning among Malaysian ESL students. The enjoyment of studying an ESL language in the Malaysian context based on the idea of second language learning has not been measured by any of the FLE psychometric established to measure foreign language enjoyment (Dewaele & MacIntyre, 2014; Jin & Zhang, 2019; Li et al., 2018). This could be as a result of the challenging and intricate process of constructing an instrument. Instrument development's primary concerns are validity and reliability; DeVellis (2017) and Bond and Fox (2015) point out that a defective validation method renders an instrument debatable. Since the Malaysian ESL Learners' Language Learning Enjoyment (MELLE) Instrument is a recent developed psychometric tool and had not yet undergone validation involving a panel of experts, the goal of this research is to fill the gap and advocate for the novelty of the development and validation of a psychometric instrument, specifically the Malaysian ESL Learners though CVR.
Research Question
Is each of the items in Malaysian ESL Learners’ Language Learning Enjoyment Instrument valid?
Research Methods
The MELLE instrument is developed to assess the latent trait of enjoyment among Malaysian ESL learners. The instrument was developed through literature review and focus group discussion conducted among excellent teachers (excellent teacher [Guru Cemerlang] is a post appointed to Malaysian teachers who has high knowledge, demonstrate skills and expertise and always dedicated and motivated in doing duties and responsibilities) and experienced ESL teachers, based on the validated six constructs namely engagement, autonomy, motivation, self-attainment, teacher-factor and peer-factor. The validation process of MELLE instrument was later conducted via Content Validation Ratio (CVR), which 12 panel of experts were appointed to rate the items. Table 1 shows the distribution on the number of items developed in MELLE.
Identification of panel of experts
Individuals with knowledge and experience in a specific field are known as experts. Before determining whether to retain or eliminate the items that have been proposed, the panel of experts is responsible to thoroughly evaluating each item. The two categories of experts enlisted for the purpose of conducting content validation ratio are professional and lay experts. Lay experts were chosen for their work in the relevant field, while professionals were elected based on their background in the field of study, specific expertise, relevant working experience, and knowledge (Powell, 2003; Rubio et al., 2003). In order to carry out the study, the researcher contacted the experts via phone, letter, and email to describe the study's goals and methods and request their approval to participate. Even though Lawshe's technique (1975) only requires for at least four experts on the panel, the researchers have chosen to involve as many experts as possible to increase the value of the model. Using purposive sampling, five professionals were chosen from five educational institutions based on the aforementioned criteria. These individuals are still engaged in research and publication and serve as consultants for Malaysia's educational system. On the other hand, the two SISC+ officers (School Improvement Specialist Coach), two excellent teachers (Guru Cemerlang) in ESL, and three experienced ESL teachers who participated in this study were chosen as lay experts. The lay experts assisted in addressing issues including language and imprecise terms and will suggest additional significant or important items. These lay experts were chosen based on their competence in teaching ESL, duration of teaching experience, knowledge in ESL teaching methods, and prior enrolment in ESL training. Additionally, experts were invited to share their opinions or viewpoints in the designated section. Table 2 summarizes the panel of experts’ information.
Rating of the items
The total number of panel experts determines the minimum CVR value which is the acceptance criteria for each indication. CVR computation employs a group of expert panels to determine whether an instrument's items accurately portray the domain construct on a three-point scale; which is (1) essential, (2) useful but not essential and (3) not necessary. The developed instrument’s item value, CVR, was calculated using the following formula, where ne represents the number of expert panels that find the item essential, N represents the total number of expert panels involved (N = 12). The formula is CVR = (Ne – N / 2) / (N / 2). The CVR value ranges from -1 to +1, with a value close to +1 denoting consensus among experts that the items are essential in content validity and vice versa. The CVR value ranges from -1 to +1 and a result close to +1 indicates agreement among experts that the items are necessary for content validity and vice versa. A critical value table was developed by Lawshe (1975) to determine the minimal standard for panel expert ratings. On the other side, (Wilson et al., 2012) revised the table and summarized it in Table 3. The minimum value that must be abided by for each item is 0.566 evaluated at α =.05. This was determined when there was a total of 12 experts. This signifies that items will be automatically retained, refined, or dropped if they fail to reach the minimum criteria value of 0.566.
Findings
Table 4 displays the CVR analysis based on the validation and evaluation of the items by a panel of 12 experts. For 12 panels, a minimum mean requirement of CVR 0.566 is required based on the CVR acceptance value in Table 3. A total of 53 items were found to not meet the CVR critical value's minimum value after an overall analysis of the items. items that were not fulfilled the requirements were Engagement (8 items), Autonomy (5 items), Motivation (12 items), Self-attainment (7 items), Teacher-factor (9 items) and Peer-factor (12 items). Following the experts’ advice, the items were deducted from the instruments due to several reasons such as the items do not suit the culture of Malaysian students and some items are not practiced in Malaysian ESL classrooms. The remaining 87 items were reviewed and refined in the focus group discussion.
Conclusion
In order to ensure the quality and efficacy of the final instrument, content validity computation is an essential step in the instrument development process. The CVR approach employed in this study has straightforward implementation requirements due to its clear steps and precise calculation formula. Additionally, it might contribute to improving the pedagogical component of the ESL teaching and learning process. Before being incorporated into the psychometric instrument for the pilot study with Malaysian ESL students, the items will be revised in accordance with the advice of the experts. The Rasch model allows for the selection of the items, following some careful consideration in appropriate statistics such as one-dimensionality, item fit, item polarity, local independence, differential item functionality to comply with Rasch models' assumptions.
References
Ab Aziz, A., Yusof, Z. M., & Mokhtar, U. A. (2019). Electronic document and records management system (EDRMS) adoption in public sector-instrument’s content validation using content validation ratio (CVR). Journal of Physics: Conference Series, 1196(1).
Alfian, A. (2021). The favored language learning strategies of Islamic university EFL learners. Studies in English Language and Education, 8(1), 47-64.
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model : Fundamental Measurement in the Human Sciences (3rd Ed.). Routledge.
Chin, N., Said, N., Pudin, C. S. J., & Din, W. A. (2022). The importance and factors of enjoyment in Malaysian ESL Classroom: A Multisite Case Study. International Journal of Education, Psychology and Counselling, 7(48), 221-240.
Chong, J., Mokshein, S. E., & Mustapha, R. (2021). A content validity study for Vocational Teachers’ Assessment Literacy Instrument (VoTAL). International Journal of Academic Research in Business and Social Sciences, 11(4), 868–883.
Csikszentmihalyi, M., & Nakamura, J. (2011). Positive psychology; where did it come from, where is it going. In S. Kennon, T. Kashdan, & M. Steger (Eds.), Designing Positive Psychology (pp. 1–3). Oxford University Press.
Dewaele, J.-M., & MacIntyre, P. D. (2014). The two faces of Janus? Anxiety and enjoyment in the foreign language classroom. Studies in Second Language Learning and Teaching, 4(2), 237-274.
Dewaele, J.-M., & Dewaele, L. (2017). The dynamic interactions in foreign language classroom anxiety and foreign language enjoyment of pupils aged 12 to 18. A pseudo-longitudinal investigation. Journal of the European Second Language Association, 1(1), 12–22.
Dewaele, J.-M. (2019). The Effect of Classroom Emotions, Attitudes Toward English, and Teacher Behavior on Willingness to Communicate Among English Foreign Language Learners. Journal of Language and Social Psychology, 38(4), 523-535.
Dewaele, J.-M., & Alfawzan, M. (2018). Does the effect of enjoyment outweigh that of anxiety in foreign language performance? Studies in Second Language Learning and Teaching, 8(1), 21-45.
Dewaele, J.-M., & MacIntyre, P. D. (2016). Foreign Language Enjoyment and Foreign Language Classroom Anxiety: The Right and Left Feet of the Language Learner. Positive Psychology in SLA, 215-236.
DeVellis, R. (2017). Scale development : Theory and Applications (4th ed.). SAGE Publications Inc.
Effendi, M., Mohd, E., & Khairani, A. Z. (2015). Assessing the content validity of IKBAR using content validity ratio. Australian Journal of Basic and Applied Sciences, (9), 255–257.
Elahi Shirvan, M., Taherian, T., & Yazdanmehr, E. (2020). The dynamics of foreign language enjoyment: An ecological momentary assessment. Frontiers in Psychology, 11(July), 1–14.
Eskandari, N., Simbar, M., Vadadhir, A. A., & Baghestani, A. R. (2018). Design and evaluation of the psychometric properties of a paternal adaptation questionnaire. American Journal of Men’s Health, 12(6), 2018–2028.
Goetz, T., Hall, N. C., Frenzel, A. C., & Pekrun, R. (2006). A hierarchical conceptualization of enjoyment in students. Learning and Instruction, 16(4), 323–338.
Gregersen, T., MacIntyre, P. D., & Olson, T. (2017). Do you see what i feel? An idiodynamic assessment of expert and peer’s reading of nonverbal language anxiety cues. In C. Gkonou, J.-M. D. Daubney, & J.-M. Dewaele (Eds.), New Insights into Language Anxiety: Theory, Research and Educational Implications (pp. 110–134). Multilingual Matters.
Jin, Y., & Zhang, L. J. (2018). The dimensions of foreign language classroom enjoyment and their effect on foreign language achievement. International Journal of Bilingual Education and Bilingualism, 0(0), 1–15.
Jin, Y., & Zhang, L. J. (2019). A Comparative Study of Two Scales for Foreign Language Classroom Enjoyment. Perceptual and Motor Skills, 126(5), 1024–1041.
Lake, J. (2013). Positive L2 self: Linking positive psychology with L2 motivation. Language Learning Motivation in Japan, 225–244.
Lawshe, C. H. (1975). A quantitative approach to content validity”.Personnel Psychology. Personnel Psychology, 28, 563–575.
Lee, J. S., & Lee, K. (2020). The role of informal digital learning of English and L2 motivational self system in foreign language enjoyment. British Journal of Educational Technology, 0(0), 1–16.
Li, C. (2020). A positive psychology perspective on chinese EFL students’ trait emotional intelligence, foreign language enjoyment and efl learning achievement. Journal of Multilingual and Multicultural Development, 41(3), 246–263.
Li, C., Jiang, G., & Dewaele, J. M. (2018). Understanding chinese high school students’ foreign language enjoyment: Validation of the chinese version of the foreign language enjoyment scale. System, 76, 183–196.
MacIntyre, P. D. (2016). So far so good: An overview of positive psychology and its contributions to sla. 3–20. https://doi.org/10.1007/978-3-319-32954-3_1
MacIntyre, P., & Gregersen, T. (2012). Emotions that facilitate language learning: The positive-broadening power of the imagination. Studies in Second Language Learning and Teaching, 2(2), 193–213.
Mercer, S., & MacIntyre, P. D. (2014). Introducing positive psychology to SLA. Studies in Second Language Learning and Teaching, 4(2), 153–172.
Nazarnia, M., Zarei, F., & Rozbahani, N. (2022). Development and psychometric properties of a tool to assess Media Health Literacy (MeHLit). BMC Public Health, 22(1839), 1–11.
Noor, N. M., Aziz, M., Mohamed, S., Moh, E., Abdullah, H., & Yon, H. (2016). Measuring the content validity of mepi using content validity ratio. Journal of ICT in Education, 3(1), 81–88.
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill.
Oxford, R. (2016). Toward a psychology of well- being for language learners: The ‘EMPATHICS’ vision. In P. MacIntyre, T. Gregersen, & S. Mercer (Ed.), Positive Psychology in SLA (pp. 10-88). Bristol, Blue Ridge Summit: Multilingual Matters.
Peterson, C. (2006). A Primer In Positive Psychology. Oxford University Press.
Powell, C. (2003). The Delphi Technique: myths and realities. Methodological Issues in Nursing Research, 41(4), 376–382. https://doi.org/10.1046/j.1365-2648.2003.02537.x
Ramli, N. F., Talib, O., Manaf, U. K. A., & Hassan, S. A. (2018). Content validity of STEMTIP using CVR method. International Journal of Academic Research in Business and Social Sciences, 8(7), 1118–1125.
Rubio, D. M., Berg-Weger, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003). Objectifying content validity: Conducting a content validity study in social work research. Social Work Research, 27(2), 94–104.
Ryan, R. M., Connell, J. P., & Plant, R. W. (1990). Emotions in nondirected text learning. Learning and Individual Differences, 2(1), 1–17.
Seligman, M. E., & Csikszentmihalyi, M. (2000). Positive psychology. An introduction. The American Psychologist, 55(1), 5–14.
Shrotryia, V. K., & Dhanda, U. (2019). Content validity of assessment instrument for employee engagement. SAGE Open, 9(1), 1–7.
Snyder, C. R., Lopez, S. J., Edwards, L. M., & Marques, S. C. (2016). The oxford handbook of positive psychology: Third edition. The Oxford Handbook of Positive Psychology, 1–1002.
Tojib, R. D. (2006). Content validity of instruments in IS research. Journal of Information Technology Theory and Application, 8(3), 31–56.
Wilson, F. R., Pan, W., & Schumsky, D. A. (2012). Recalculation of the critical values for Lawshe’s content validity ratio. Measurement and Evaluation in Counseling and Development, 45(3), 197–210.
Zeng, Y. (2021). A review of foreign language enjoyment and engagement. Frontiers in Psychology. 12(737613), 1-5.
Copyright information
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
About this article
Publication Date
25 September 2023
Article Doi
eBook ISBN
978-1-80296-964-1
Publisher
European Publisher
Volume
7
Print ISBN (optional)
-
Edition Number
1st Edition
Pages
1-929
Subjects
Language, education, literature, linguistics
Cite this article as:
Chin, N., Said, N., & Pang, V. (2023). Content Validity of MELLE Instrument for Malaysian ESL Learners Using CVR Method. In M. Rahim, A. A. Ab Aziz, I. Saja @ Mearaj, N. A. Kamarudin, O. L. Chong, N. Zaini, A. Bidin, N. Mohamad Ayob, Z. Mohd Sulaiman, Y. S. Chan, & N. H. M. Saad (Eds.), Embracing Change: Emancipating the Landscape of Research in Linguistic, Language and Literature, vol 7. European Proceedings of Educational Sciences (pp. 72-81). European Publisher. https://doi.org/10.15405/epes.23097.7