Achievement Monitoring Technology Of Mentally Retarded Schoolchildren On The Touch Screen

Abstract

The article actualizes the contradiction between practical relevance and insufficient development of technologies to monitor the educational achievements of students with mental retardation. The objective of the study is to determine the possibility of using a modified Russian-language version of GOM technology (measuring the overall result) for monitoring reading skills in students with moderate mental retardation on a touch screen (tablet computer). Empirical data were obtained both by the method of choosing (indicating) the correct option from the three proposed for selection (the requirement of GOM technology), and by the generally accepted method of assessing reading skills - reading aloud. The survey by pointing to the correct option was carried out using a computer program on the touch screen. To determine the reliability using the methods of statistical analysis, differences or coincidences, as well as the interdependence (correlation) between the stable results of showing two inclusions over a short period of time, were revealed. As a result of the study, it was concluded that the task sets developed in the Russian version taking into account the GOM requirements for monitoring reading skills of students with moderate mental retardation on a tablet computer are reliable and relevant. Despite the specifics of literacy in different languages ​​and in different cultural contexts, a number of requirements for the monitoring procedure defined in the framework of GOM technology are universal.

Keywords: Mental retardationeducational monitoringreading skills

Introduction

In connection with the inclusion of students with significant cognitive disorder (IQ – 49 and below) in education system in Russia in the 21st century, issues of educational achievements monitoring of schoolchildren in this category are relevant. Learning to read by a student with moderate mental retardation (IQ – 35-49) is considered as one of learning outcomes available and significant for socialization (Shipitsyna, 2002; Voronkova, 2016). Despite many methodological materials and developments, in recent years, interest in the problem of education in reading and writing for children with intellectual disabilities is unabated, there are new domestic and foreign studies in this field (Afacan et al., 2017; Ainsworth et al., 2016; Alnahdi, 2015; Katkova, 2017).

In order to determine the expected possible results in learning process in the nearest time, it is important to identify the zone of “proximal development” (the child’s real abilities that can be revealed and used for his development with minimal help or prompting from others) (Lubovsky, 1989; Vygotsky, 1935). In addition, teachers need constant feedback from students about the effectiveness of training, therefore, the problem of both “starting” diagnostics and monitoring of educational achievements is updated.

The use of generally accepted approaches used to monitor the learning outcomes of normatively developing pupils and schoolchildren with moderate mental retardation (independent and review work, testing, oral questioning) require adaptation in working with children with moderate mental retardation. When self-learning students of this category cannot always realize their potential, a situational decrease in results is possible due to poor health, increased fatigue, distractability, reduced motivation and the ability to program, regulate and control their actions, they need additional stimulating, organizing and guiding assistance; students with a lack of common speech need methods with possibility to use nonverbal answers (Shipitsyna, 2002; Zabramnaya & Isaeva, 2009).

Russian specialized literature widely presents data about using various diagnostic methods (psychological and pedagogical experiment, observation, collection and analysis of data about a child, studying products of activity, etc.) in order to identify the starting and potential abilities of children with intellectual disabilities (Lubovsky, 1989; Zabramnaya & Isaeva, 2009). However, their use for monitoring purposes in current educational process is not effective enough.

A number of foreign studies of recent years have been devoted to the problems of educational and administrative monitoring of educational achievements of students with cognitive disorders (Farley et al., 2016; Hill & Lemons, 2015; Jones et al., 2018; Kearns et al., 2015). “Formative assessment” technologies are used for applying in current educational process in order to identify the zone of “proximal development”, to monitor the progress of students for short periods and to promptly adjust programs for working with them. In particular, such a technology of formative assessment as general outcome measurements (GOMs) (Wallace et al., 2010), developed on the basis of curriculum-based measurement (CBM) (Deno, 2003). Both options, in essence, are testing, but the monitoring procedure and content are focused on the characteristics of students with cognitive disorder.

Of course, to monitor reading skills of children with mental retardation in Russia, modification of GOMs is required taking into account specific features of Russian writing and domestic approaches to literacy education. In addition, in our opinion, the use of GOMs technology on paper is not convenient enough: teacher needs to simultaneously record results and keep student's attention, navigate in a large number of cards with tasks, then it takes time to work with protocols and processing results.

The practical use of GOMs becomes much easier with help of computer technology, especially given the fact that the method of pointing at the correct option out of the three offered for selection, which was originally used in this technology to enable students with alalia (speechless children), is extremely convenient for a “programming language”. Using a computer program in this case has several advantages:

• objectivity of assessment, independence from expert opinion;

• ease of use;

• possibility to include children with alalia in the assessment system;

• optimization of time spent on processing results (processing is carried out by a computer program, automatically).

Problem Statement

Thus, the revealed contradiction between practical relevance and insufficiently developed technology for monitoring reading skills of students with moderate mental retardation allows us to formulate the research problem, which consists in determining whether it is possible to use a modified Russian-language version of GOMs technology (general outcome measurement) for monitoring reading skills on touch screen (tablet computer) of schoolchildren with moderate mental retardation.

Research Questions

In connection with the modification of the initial version of GOMs technology and change in the conditions for its implementation (transfer from paper to a computer program), the next question arises: how reliable and valid are the task sets developed in the Russian version taking into account the requirements of GOMs for monitoring reading skills of schoolchildren with moderate mental retardation on tablet computers?

Purpose of the Study

In accordance with the problem and research questions, we set forward the following goal: to determine the reliability and validity of task sets developed in the Russian version taking into account the main provisions of GOMs technology for monitoring reading skills of schoolchildren with moderate mental retardation on the touch screen (tablet computer).

Research Methods

The main methodological base in this study is a group of theoretical scientific methods related to the analysis of empirical data. The data for analysis was obtained both by using the method of pointing at the correct option out of the three offered on the touch screen (tablet computer), and the generally accepted method of “reading aloud”.

Examination using a computer program on a touch screen (tablet computer) was carried out on 6 indicators:

• Recognition of pictogram based on the sample.

• Recognition of grapheme based on the sample.

• Correlation of grapheme and phoneme.

• Reading syllables.

• Reading words.

• Determination of the first letter in a word.

For each indicator, sets of 60 assignments were developed. The procedure for presenting each task was as follows: 3 images appear on the touch screen (three pictograms, letters, syllables or words, depending on the indicator being examined) and the instruction sounds: “Point at ...”. The child touches on the touch screen one option out of three offered for selection. If there is no answer for 5 seconds or an incorrect answer, the question is repeated, if it is correctly chosen, they do the next task. The program automatically calculates the number of correct choices in 3 minutes for each indicator. Score: 2 points – correct answer after the first presentation, 1 point – correct answer after the second presentation, 0 points – incorrect answer or no answer even after repeated presentation.

The program is developed in accordance with the requirements of GOMs technology:

• The standard number of tasks distributed by indicators (at the stage of the “pilot study” 6 indicators were tested);

• Taking into account the student’s non-verbal reaction (pointing on the touch screen the correct option out of 3 offered for selection);

• Compliance with the “three errors” rule, which implies the completion of the survey by the indicator when three mistakes are made in a row;

• Short examination duration (no more than 3 minutes for each indicator);

• Using the first three tasks for each indicator as training (when presenting training tasks, “hint systems” are acceptable, their results are not taken into account when calculating points).

It is also necessary to clarify several points on the examination procedure:

1. Within the framework of this study, the task was not to clarify the wording of the instruction; therefore, within each indicator, various wording options were used: “Where ...”, “Point at ...”, “Where ... Point at”, “Find ...”. The ability to vary the wording was proved by us earlier, when examining children with serious mental retardation (Artemeva et al., 2018). In the future, it is advisable to clarify the formulation requirements for various degrees of mental retardation, which coincides with the opinion of foreign researchers (Hill & Lemons, 2015).

2. Based on the foreign studies of Wayman et al. (2009) on the possibility of applying the rules of both two and three errors, we settled on the rule of “three errors”.

The Russian version of the program takes into account the specifics of teaching reading to children with mental retardation in Russia:

• The process of studying letters for children with moderate mental retardation was prolonged for several years, so the examination was carried out on limited material of letters.

• Tasks are divided into difficulty levels within each indicator in accordance with the teaching methodology of reading in Russia (from two to four levels per indicator).

• A significant component of the learning content of children with moderate mental retardation is the mastery of alternative reading using non-verbal graphic characters (Baryayeva & Lopatina, 2018), therefore the new indicator has been introduced – “Pictogram recognition”. The specific difficulties in mastering alternative communication means by children with mental retardation require additional study and they are associated with immature stage of visual perception, memory, ability to purposeful activities, disordered speech understanding, gesture perception, limited social experience, etc. (Artemeva et al., 2018; Zashchirinskaia et al., 2017; Zabramnaya & Isaeva, 2009).

• Some of the children participating in the experiment have serious disordered motor functions, they are limited in the use of a computer mouse, so the computer program is designed to be implemented on a tablet computer with a touch screen.

To determine the reliability, all 6 indicators were offered to the same student several times over 2 weeks. Not always a reliable result was achieved immediately after the first inclusion, usually 2-4 training inclusions were required. It is also possible that episodes decrease in values ​​by indicators, which may be associated with the condition of the child during the examination (fatigue, poor mood, decreased performance, etc.). In cases of episodic decreases in results, we recommended that teachers repeat the tasks in 1-2 days (sometimes even after some time during the same day), when the child's condition stabilizes. Thus, taking into account the characteristics of students with moderate mental retardation, to determine the reliability, we used from 2 to 4 inclusions in a close time period and compared two stable results.

To determine the validity, the data obtained by the method of pointing on a tablet computer were compared with the data obtained by the generally accepted method – reading aloud. We used a three-point scale (0-2 points) when assessing reading for each task.

The statistical analysis was carried out using “Statistica 10” software package, the authors used non-parametric statistical criteria for related samples such as Wilcoxon test, and determined Spearman correlation coefficient.

It should be noted that the development of technology for monitoring reading skills of students with mental retardation, which allows you to track all stages of the skill formation, is a difficult task that requires a prolonged study. Therefore, an initial “pilot” study (usability – research) was organized to test sets of tasks on six indicators, the results of which are reflected in this article. The study was conducted in November – December 2016, in the framework of the joint project of KSPU named after V.P. Astafyev and the University of Minnesota “Student assessment with significant disabilities” (Mamaeva, 2017).

When conducting a “pilot” study, the goal was not to develop the full monitoring content, which covers all aspects and stages of reading skill, the authors also did not determine sensitivity to changes in the selected indicators and a number of other goals that are currently being implemented through a series of subsequent studies.

Findings

In total, during the “pilot” survey (usability research), 54 students with moderate mental retardation were examined. All in all, 35 boys and 19 girls. Age: 7 years old – 5 people, 8 years old – 9 people, 9 years old – 9 people, 10 years old – 10 people, 11 years old – 13 people, 12 years old – 4 people, 13 years old – 4 people. 4 people have mental retardation complicated by disorders of the autistic spectrum, 6 people with disorders of the musculoskeletal system (cerebral palsy), 4 people with visual disorders, 1 student with diminished hearing, 6 students with speechlessness (students with speechlessness were not examined by reading aloud or examined on a limited set of indicators). The study was conducted on the basis of four schools in Krasnoyarsk, 6 teachers participated in the experiment.

An initial analysis of the results is presented in Table 01 .

To determine the reliability of the proposed task sets, we conducted a comparison and also revealed the interdependence (correlation) between the stable results of two inclusions.

As can be seen from table 02 , for almost all indicators (with the exception of indicator VI - “Determination of the first letter in a word”), the obtained p-value (Wilcoxon) indicates the absence of statistical significance of indicator changes during repeated inclusions in a close time period. The results of two stable inclusions in a close time period are congruent at a significance level of 0.05, therefore, the proposed task sets (with the exception of indicator VI) are reliable.

According to the results of comparisons on indicator VI – “Determination of the first letter in a word”, we fall into the “zone of uncertainty”. The results of repeated inclusions in a close time period are congruent at a significance level of 0.01 but differ at a significance level of 0.05. At the same time, a high correlation coefficient (Spearman) – 0.94 indicates a close and statistically significant interdependence of the results of two stable inclusions in a close time period and allows us to conclude that the task set is reliable. To clarify the conclusions on the indicator VI, we additionally compared the results of two inclusions measured on a two-point scale (1 point – correct pointing after the first presentation, 0 points – incorrect pointing or no answer after the first presentation). The following values ​​were obtained: p-value (Wilcoxon) – 0.07; correlation coefficient (Spearman) – 0.94. The data obtained correspond to the conclusion about the reliability of the task set VI.

However, it should be noted once again that for the correct interpretation of the results of students with moderate mental retardation, it is important to consider a number of factors:

• To learn how to use the monitoring program on a tablet computer student may need several inclusions, therefore, when learning to use the program, the teacher should offer the student 2-4 inclusions for 2 weeks, until stable results are obtained inside the indicator;

• There may be occasional decrease in the values inside the indicator, in these cases it is recommended to repeat the inclusion when the child's condition is stabilized.

To determine the validity of the proposed task sets for each indicator, the authors revealed an interdependence (correlation) between the results obtained using a tablet computer and the data obtained by reading aloud.

As can be seen from table 03 , in almost all indicators (with the exception of the first and second, recognition of pictograms based on the sample and graphemes based on the sample), a close relationship was found between the results obtained using the pointing method on a tablet computer and the data obtained using the reading aloud method. The results achieve statistical reliability for all indicators. Moreover, for all indicators, the results of pointing are higher than the results of reading / naming (Table 01 ).

We explain this by the fact that in their psychological structure these actions are very close, but not identical. Being a complex psychophysiological process, reading aloud has a more complex psychological structure. Moreover, the display of a grapheme or pictograms based on a sample is focused on examining an undefined stage in mastering reading skills (such as tasks for naming letters, reading syllables and words), but examining one of the prerequisites for mastering reading skills, namely, visual perception. Therefore, according to indicators I and II, the correlation coefficient with the naming results is the lowest, but also reaches statistical significance.

Conclusion

As a result of the “pilot” study, the following conclusions were made:

• task sets developed in the Russian-language version taking into account the requirements of GOMs for monitoring reading skills of students with moderate mental retardation on a tablet computer are reliable and valid;

• assignments for correlating graphemes and phonemes, determining the first letter in a word, reading syllables and words are the most reliable for monitoring reading skills; pictograms and graphemes recognition tasks based on the sample also confirmed their potential for monitoring reading skills.

The scientific novelty of the study lies in the fact that the obtained experimental data do not contradict the possibility of using effective GOMs technology for monitoring reading skills of students with mental retardation in Russia on a tablet computer. Despite the specific features of literacy in various languages ​​and different cultural contexts, a number of requirements for monitoring procedure identified in the framework of GOMs technology are universal. Monitoring using computer technology greatly facilitates the practical use of GOMs and optimizes the results processing.

But the results of the “pilot” survey presented in this study do not solve all the issues related to the problems of monitoring educational achievements of students with mental retardation using computer technology. In a series of further studies conducted in 2017-19 the authors:

• tested a complete task set for monitoring reading skills;

• clarified the requirements for the procedure and content of monitoring when implemented in a single interactive environment.

The authors continue the research to determine the sensitivity to changes in the developed task sets in Russian-language version of monitoring.

Acknowledgments

The authors would like to thank:

(1) Eurasia Foundation that funded in 2016-17 the joint project of Krasnoyarsk State Pedagogical University (KSPU) named after V.P. Astafyev and the University of Minnesota “Assessments of Students with Significant Disabilities” (W15-1015);

(2) University of Minnesota faculty: Renate Teach (Ph. D.), Brian Abery (Ph. D.), and University of Minnesota programmers for developing a computer program for monitoring reading skills on a touch screen (tablet computer);

(3) Administration and employees of educational organizations of Krasnoyarsk who participated in the testing of innovative technology: Krasnoyarsk School No. 5, Krasnoyarsk School No. 3, Krasnoyarsk School No. 8, Krasnoyarsk School No. 2.

References

• Afacan, K., Wilkerson, K. L., & Ruppar, A. L. (2017). Multicomponent reading interventions for students with intellectual disabilities. Remedial and Special Education, 39(4), 229-242. https://doi.org/10.1177/0741932517702444
• Ainsworth, M. K., Evmenova, A. S., Behrmann, M., & Jerome, M. (2016). Teaching phonics to groups of middle school students with autism, intellectual disabilities and complex communication needs. Research in Developmental Disabilities, 56, 165-176. https://www.deepdyve.com/lp/elsevier/ teaching-phonics-to-groups-of-middle-school-students-with-autism-7ICzoO60o9
• Alnahdi, G. H. (2015). Teaching reading for students with intellectual disabilities: a systematic review. International Education Studies, 8(9), 79-86. https://files.eric.ed.gov/fulltext/EJ1074053.pdf
• Artemeva, N. V., Zadorozhnaia, T. V., & Mamaeva, A. V. (2018). Monitoring sformirovannosti bazovykh predposylok dlya ponimaniya piktograficheskikh izobrazheniy u obuchayushchikhsya 1-2 klassov s tyazheloy umstvennoy otstalostyu [Monitoring the formation of basic prerequisites for understanding pictographic images in first-second grade students with severe mental retardation] Humanities. Scientific and practical journal, 2, 168–172. http://www.gpa.cfuv.ru/ru/nauchnaya-deyatelnost/600-nauchnye-izdaniya/izdaniya-vak/zhurnal-gumanitarnye-nauki/arkhiv/gumanitarnye-nauki-v-2018-godu
• Baryayeva, L. B., & Lopatina L. V. (2018). Metodicheskiye aspekty raboty s negovoryashchimi detmi s ispolzovaniyem sistemy alternativnoy kommunikatsii [Methodological aspects of support for non-speaking children using alternative communication system]. Special Education, 4, 5–20. http://journals.uspu.ru/index.php?option=com_content&view=article&id=2221:izuchenie-i-obrazovanie-lic-s-ogranichennymi-vozmozhnostjami-zdorovja&catid=828&Itemid=370
• Deno, S. L. (2003). Developments in curriculum-based measurement. The Journal of Special Education, 37(3), 184 – 192. https://eric.ed.gov/?id=EJ785942
• Farley, D., Anderson, D., & Irvin, Sh. (2016). Modeling reading growth in grades 3 to 5 with an alternate assessment. Remedial and Special Education, 38(4), 195-206. https://doi.org/10.1177/0741932516678661
• Hill, D. R., & Lemons, C. J. (2015) Early grade curriculum-based reading measures for students with intellectual disability. Journal of Intellectual Disabilities, 19(4), 311-325. https://doi.org/10.1177/1744629515574812
• Jones, F. G., Gifford, D., Yovanoff, P., Al Otaiba, S., Levy, D., & Allor J. (2018). Alternate assessment formats for progress monitoring students with intellectual disabilities and below average IQ: an exploratory study. Focus on Autism and Other Developmental Disabilities, 34(1), 41-51. https://doi.org/10.1177/1088357618762749
• Katkova, I. A. (2017). Rezultaty izucheniya psikhologicheskogo bazisa i navykov chteniya umstvenno otstalykh mladshikh shkolnikov so slozhnymi narusheniyami razvitiya [The study results of the psychological basis and reading skills of mental retarded primary-school children with complex developmental disorders]. Defektologiya (Defectology), 6, 63–73. http://www.schoolpress.ru/ products/rubria/index.php?ID=79928&SECTION_ID=38
• Kearns, J. F., Kleinert, H. L., & Thurlow M. L. (2015). Alternate assessment as one measure of teacher effectiveness: implication for our field. Research and Practice for Persons with Severe Disabilities, 40(1), 20-35. https://doi.org/10.1177/1540796915585105
• Lubovsky, V. I. (1989). Psikhologicheskiye problemy diagnostiki anomalnogo razvitiya detey [Psychological problems in the diagnosis of abnormal children development]. Pedagogika. http://psychlib.ru/inc/absid.php?absid=18466
• Mamaeva, A. V. (2017). Aprobatsiya tekhnologii monitoringa navyka chteniya obuchayushchikhsya s umstvennoy otstalostyu s ispolzovaniem sensornogo ekrana [Testing the technology of monitoring reading skill of students with mental retardation using the touch screen]. In Itogoviy sbornik II Vserossiyskogo syezda defektologov [Final collection of the II All-Russian Congress of Defectologists]. (pp 271–277). Moscow. http://yarinternat-9.ru/wp-content/uploads/2019/03/ Сборник-II-Всероссийского-съезда-дефектологов.pdf
• Shipitsyna, L. M. (2002). «Neobuchayemiy» rebenok v semye i obshchestve. Sotsializatsiya detey s narusheniem intellekta [“Unteachable” child in a family and society. Socialization of children with intellectual disabilities]. Didaktika Plyus. https://infourok.ru/shipicinalm-neobuchaemiy-rebenok-v-seme-i-obschestve-1301906.html
• Voronkova, V. V. (2016). Differentsirovanniy podkhod v obuchenii umstvenno otstalykh detey mladshego shkolnogo vozrasta na primere usvoyeniya russkogo yazyka [Differentiated approach in training of mental retarded primary school children by the example of learning the Russian language]. ASOY. http://new.asou-mo.ru/images/files/Monografii/Изд.__1251__Воронкова.pdf
• Vygotsky, L. S. (1935). Dinamika umstvennogo razvitiya shkolnika v svyazi s obucheniyem [Mental development dynamics of a student in connection with training]. In Umstvennoye razvitiye detey v protsesse obucheniya: Sbornik statey [Mental development of children in the training process: Collection of Articles]. http://psychlib.ru/inc/absid.php?absid=79865
• Wallace, Т., Tichá, R., & Gustafson, K. (2010). Technical characteristics of general outcome measures (GOMs) in reading for students with significant cognitive disabilities. Reading& Writing Quarterly, 26(4), 333-360. https://doi.org/10.1080/10573569.2010.500264
• Wayman, M. M., Tichá, R., Wallace, T., Espin, C. A, Wiley, H. I, Du, X., & Long J. (2009). Comparison of different scoring procedures for the CBM maze selection measure. Technical Report. University of Minnesota. https://www.progressmonitoring.org/pdf/tr10asls.pdf
• Zabramnaya, S. D., & Isaeva, T. N. (2009). Psikhologo-pedagogicheskaya differentsiatsiya detey s umerennoy i tyazheloy umstvennoy otstalostyu [Psychological - pedagogical differentiation of children with moderate and severe mental retardation]. Vospitaniye i obucheniye detey s narusheniyami v razvitii [Education and training of children with developmental disabilities], 9, 49–53. https://elibrary.ru/contents.asp?id=34119738
• Zashchirinskaia, O. V., Nikolaeva, E. I., Rybnikov, V. Y., & Bysova V. M. (2017). Vliyaniye vnutrisemeynoy kommunikatsii na spetsifiku vospriyatiya zhestov detmi mladshego shkolnogo vozrasta s legkoy umstvennoy otstalostyu [The influence of family communication on the specificity of the gestures perception by children of primary school age with mild mental retardation]. Vestnik Novosibirskogo gosudarstvennogo pedagogicheskogo universiteta [Novosibirsk State Pedagogical University Bulletin], 7(5), 81-93. http://doi.org/10.15293/2226-3365.1705.06