The Requirements To Assessing Of Mentally Retarded Students Achievements In Interactive Environment

Requirements for assessment procedure

To answer the first question of the research, while determining the requirements to the procedure of mentally retarded schoolchildren achievements testing in an interactive environment, we relied on the main provisions of the GOMs method. However, the original version of the GOMs method, proposed by the University of Minnesota, was modified taking into account:

• specifics of the spoken and written Russian language;
• specifics of teaching children with mental retardation to read in Russian;
• features of interactive environment.

The following requirements are defined as basic:

Selecting the system of progressive indicators for a certain ability or skill assessment. This requirement is one of the key ones in the method and correlates with the principle of “gradual learning” in Russian correctional pedagogy with the focus on “zone of proximal development”. To assess the formation of initial reading skills, in cases where training is based on a sound analytical-synthetic method, we have identified the following indicators:

• knowing letters;
• language analysis skills (recognizing the first letter in a word, the last letter in a word, the presence of a letter in a word, syllabic analysis);
• reading syllables;
• reading words;
• reading sentences (matching a picture to a sentence; finishing a sentence with a word that matches the meaning).

For children who are not able to master reading on the basis of generally accepted approaches, alternative reading methods are proposed: visual perception, pictograms, “reading” of pictographic sentences.

Some of the above indicators have been tested on paper, local media using computer technologies (tablet computer with a touch screen) and in an interactive environment; others are at the stage of development and testing. This report will present the results of testing the requirements for the procedure of the initial reading skills formation assessment in mentally retarded schoolchildren in an interactive environment using the example of the following indicators: recognizing the first and last letters in a word, reading syllables and reading words.

Structuring of diagnostic tasks not only according to indicators, but also letters. Since the process of studying letters in mentally retarded schoolchildren is time-consuming (sometimes for several years) and students initially start reading syllables, words and sentences using a limited set of letters, it is appropriate to start tracking the formation of these indicators also with a limited letter sets. The tasks are divided into three groups.

Thus, assessment according to our proposed algorithm presupposes an initial testing the students’ alphabet skills, and then the rest of the indicators based on the letter set located in the "zone of proximal development" of a particular student. At the same time, within each set both recommended and additional indicators are shown. For example, in the set 3 (all letters) the recommended indicators are: reading syllables, words, sentences; the rest of the indicators are considered as extra.

It should be noted that one more difference from the foreign version, where all tasks within the indicator were considered as equivalent, is the allocation of difficulty levels within the indicator and an even distribution of tasks into difficulty levels. For example, according to the indicator "recognizing the first letter in a word" the first level of difficulty is identifying the first vowel in a word; the second level of difficulty is the first consonant from the consonant cluster; the third level of difficulty is the first consonant from the fusion syllable. Tasks for testing each of the indicators are presented sequentially by groups; each includes one task of all difficulty levels of the indicator in a random sequence; the choice of a task from a specific difficulty level is carried out randomly. Thus, the “phasing” we offer in the modified version of the GOMs method is more specific, which allows us to obtain more accurate data on the skills formation within each specific indicator, since the program automatically calculates the number of correct recognitions both for each indicator and for each difficulty level within indicator.

Due to the fact that the screening time for each indicator is limited, we have defined a requirement - no more than three difficulty levels within the indicator.

Short assessment time (three minutes per indicator). This requirement is indicated in GOMs method and is due to such psychophysical characteristics of students with cognitive disorders as: low performance, increased fatigue, distraction, difficulty concentrating, reduced motivation (Zashchirinskaia et al., 2017). Subsequent to allocation of difficulty levels within the indicator this provision is modified: for each difficulty level within the indicator the time is distributed evenly: 60 seconds per level in the case of three difficulty levels within the indicator, 90 seconds per level in the case of two difficulty levels within the indicator.

The student's answer recognizing the correct option from the three proposed. When completing the tasks a child must choose the correct answer from the three offered. Correct answers are automatically calculated per unit of time (3 minutes per indicator), both for the indicator as a whole and for each difficulty level within the indicator. During the research we used both showing the answer with a computer mouse, and with the help of a pointing gesture (in this case, the teacher controls the mouse), as well as with the help of a student's touching of the correct option on the touch screen. The choice of electronic medium and a method of display were made by the teacher, based on the motor characteristics of a particular student and material and technical support in a particular organization. At the same time, while working with each particular student during the experiment, only a uniform technical solution was used, since hypothetically it can be assumed that a change in conditions of presenting material will affect the quantitative diagnostic indicators. Comparison of the results when using different electronic media and different recognition methods was not carried out within the framework of this study, however, we are considering it as a further perspective.

The rule of "three errors" in the original foreign version presupposes the end of the test when a student makes three consecutive errors. Compliance with this requirement, on the one hand, allows to reduce the test time, while maintaining the accuracy of the results, and also prevents fatigue and students’ negative reaction to repetitive mistakes when presenting tasks that are beyond their proficiency level or not understood, which is especially difficult for students with mental retardation and the brain control of the affect is difficult. Based on foreign studies on the possibility of applying the rules of both two and three errors (Wayman et al., 2009), we settled on the “three errors” rule. However, its implementation in our program has its own specifics associated with the introduction of difficulty levels within the indicators: in the case of three consecutive incorrect choices in any difficulty level, the testing of only this level is stopped; testing by the indicator as a whole for the rest of the difficulty levels is continued according to the time assigned.

Using the first three tasks for each indicator as training. This provision of the GOMs method is used in our proposed version without changes and involves the use of a system of prompts with a gradual increase in the assistance provided with each subsequent repetition of the task (four presentations of training tasks are acceptable), which allows a student with mental retardation to navigate when applying knowledge in new conditions. The simplest tasks are used as trainers, which are not used as estimates afterwards. The results of training tasks are not taken into account when scoring.

Providing organizing assistance by repeating instructions at the first incorrect choice when performing assessment tasks (assessment tasks are tasks starting from the fourth within each indicator). If the choice is made with an error or does not fit within a certain time limit (from 5 to 30 seconds, depending on the indicator), then the question is repeated, but already expanded. The correct choice from the first presentation of the task in each assessment task is awarded with 1 point, while for the correct choice after repeating the question no points are awarded. This requirement is due to the need for additional assistance associated with the insufficient formation of motivational, energy and regulatory components of cognitive activity in children with mental retardation (Dmitriev & Verkhoturova, 2019; Verkhoturova et al., 2020). Repeating instruction in this case reduces the risk of simplifying instructions by the learners ("degrading" to random choice).

The possibility of up to four inclusions in a close time period (within two weeks) to obtain reliable results, which is an important condition for ensuring data reliability. Carrying out several inclusions is especially important when learning how to work with the program, as well as in case of a situational decrease in indicators results; for this, the program provides a function for selecting the basic value of the indicator for a period. The need to comply with this requirement was confirmed earlier through a series of studies in 2016 - 2020 (Mamaeva et al., 2020).

Interpretation of assessment results based on their quantitative and qualitative analysis. Scoring by indicators and difficulty levels is done automatically, but it is on the basis of quantitative and qualitative analysis that a teacher chooses a letter set, assessment indicators, the base indicator values for the period, and also supplies and interprets the numerical values, taking into account the qualitative features of the tasks, observation of behavior and children condition during the test period. For example, a student who chooses a "random guess" strategy while reading sentences may score more points than a student who completes assignments consciously, but slowly, and whose reading by syllables skill is just forming.

It should also be noted that within the framework of this message there is no need to present requirements for the content and design of tasks for each specific indicator (for example, material selection, voice instructions, images, font design, etc.), which is of a private nature and belong to exactly one specific indicator.

These requirements to the content and procedure for assessing the initial reading skills formation in mentally retarded schoolchildren in an interactive environment were tested in terms of the following indicators: recognizing the first and last letters in a word, reading syllables and reading words.

Determining the reliability and validity of data

To answer the second question of the research (to determine the reliability and validity of data obtained in an interactive environment in compliance with the requirements indicated), a series of studies was organized in the period from 2018 to 2020 on the above indicators. Testing of different indicators was carried out on the basis of different organizations, with different students and at different times. The main criteria for inclusion in the experiment were: the presence of mental retardation and the formation of the indicator (the examined indicator of the initial reading skills formation at the time of assessment should be in the "zone of proximal development"). Data on the organization of the experimental study by indicators are shown in Table 1.

The experimental study was organized as follows:

• to determine the reliability between two and four inclusions were carried out within two weeks, and then the results of two stable inclusions were compared;
• to determine the validity the interdependence between the data on recognitions in an interactive environment and the results of reading syllables and words aloud or a verbal answer to a question about the first and last letters in a word was analyzed.

The results were reviewed using the methods of statistical analysis, the software package "Statistica 10" and nonparametric statistical tests for related samples were used: Wilcoxon's test, sign test, and Spearman's correlation coefficient was calculated.

Initial analysis

Initial analysis data is presented in Table 2.

As can be seen from Table 2, for all indicators the medial, maximum and minimum values of recognitions during two stable inclusions carried out over two weeks are very close, which is consistent with the assumption of the tested sets of tasks reliability. The medial, maximum and minimum values ​​of recognitions and the corresponding values ​​of the verbal responses are also close, but the results of the recognitions are higher than the results of the verbal responses. This is explained by the fact that recognizing and reading aloud (or naming the first and last sounds in a word) are close, but not identical; reading aloud has a more complex psychological structure. At the same time, the results of the recognitions in an interactive environment and the results of verbal answers can be closely interrelated, which corresponds to the results of our earlier study on a local medium, and does not contradict with the assumption of the task sets validity tested in an interactive environment (Mamaeva et al., 2020). To confirm the outcomes on reliability and validity of tasks set placed in an interactive environment in compliance with the above requirements, the results of approbation were analyzed using methods of statistical analysis.

Determining the reliability of task sets

To determine the reliability of the tasks sets, the results of two stable inclusions were compared for each indicator in a close time period (up to two weeks).

As can be seen from Table 3 for all indicators, the obtained p-value values (Wilcoxon, Sing Test) allow us to conclude that there is no statistical significance of changes in the indicator when repeating inclusions for two weeks. The results coincide at a significance level of 0,05. Therefore, task sets placed in an interactive environment in compliance with the requirements indicated are reliable.

Determining the validity of the task sets

To determine the validity of the task sets for each indicator the interdependence (correlation) was determined between the results obtained in an interactive environment by the recognition method and the results obtained applying reading aloud.

As can be seen from Table 4 for all indicators, a close relationship was revealed between the data obtained in an interactive environment by the recognition method and the data obtained by traditional methods of reading aloud (or verbal response). The correlation coefficient reaches statistical reliability for all indicators, which allows us to make a conclusion about the validity of task sets placed in an interactive environment in compliance with the requirements we have indicated. Since according to indicators I and II (recognizing the first and last letter in a word) not the reading skill itself is examined, but one of the prerequisites for its formation, namely, elementary forms of sound-letter analysis, the correlation coefficient with the results from naming is somewhat lower than in indicators III and IV, within which testing of the syllables and words reading is carried out.