Text Skills Assessment In Russian Language’s Monitoring In Primary School

Abstract

The idea that the reading literacy is an integrative component of functional literacy and that all subjects must contribute to its development is firmly established in the Russian primary education. The Russian language subject has a strong potential in this matter as during the Russian language lessons working with texts is one of the priority activities. However, it is impossible to effectively trace the individual educational trajectory of each student and give pedagogical support without possessing data on the reading abilities level gathered during regularly performed monitoring. The absence of credible data seriously restrains the enhancement of teaching methods. This article contains the results of a study which goal was to develop models of tasks that assess text skills and are to be included in the Russian language tests for students from the 1st to 4th grade with the obligatory respect for the principle of results comparability. The article provides the principles of formation of tasks, the tasks themselves. The results of the study show that about 30% of primary school graduates have trouble with deep comprehension of texts leading to the fact that the students have difficulties inventing text title, identifying text microtopics, identifying the topic and the main idea of the text and describing them in their own coherent statement. The gained data can serve as a basis for taking decisions on necessary alterations of methods of working with texts during Russian language lessons and the presented tasks - as models for the realization of similar studies.

Keywords: Reading literacyprimary schooltext skills

Introduction

In the Russian primary school, work on reading literacy has always been one of the targeted priorities both in normative documents and in the practical work of teachers. Primary school learning time is the first stage in the development of a reader who is able to retrieve information from the text, integrate and interpret it, evaluate the form and content of the text. Based on the interpretation of reading literacy in the international comparative study PIRLS (Mullis & Martin, 2015; Zuckerman, Kovaleva, & Baranova, 2018) and taking into account the peculiarities of the Russian context, the following definition was developed:

reading literacy is a set of skills reflecting: the need for reading activity for the purpose of successful socialization, further education, self-development; readiness for semantic reading of written texts, analysis, evaluation, interpretation and synthesis of the information presented in them; the ability to extract the necessary information for its transformation in accordance with the educational task; to be guided with the help of various textual information in life situations. (Vinogradova et al., 2018, p.21)

A special feature of the last decade is the realization that reading literacy is an integrative component of functional literacy, which cannot be formed by means of only the subject “Literary reading”. All the main subjects of the curriculum should not only use the skill acquired in reading lessons for the development of the program material, but also contribute to the development of reading literacy of the younger student. At all lessons, students read, but there is a specificity of reading linguistic, mathematical, natural science texts, and the task is to help the younger student to master the strategies of reading different texts and the features of using information obtained from the text. At the lessons of the Russian language, both in elementary and basic schools, working with text is one of the priority activities of, informational, informative, instructional and reference texts are actively used (Aleksandrova, Aristova, Dobrotina, Gosteva, & Vasilevykh, 2017; Aleksandrova, Vasilevykh, Gosteva, Dobrotina, & Uskova, 2018; Ippolitova, 1998; Ladyzhenskaya & Ladyzhenskaya, 2005).

Problem Statement

In the presence of theoretical grounds (Universal'nye uchebnye dejstvija kak rezul'tat obuchenija v nachal'noj shkole: soderzhanie i metodika formirovanija universal'nyh uchebnyh dejstvij mladshego shkol'nika, 2016; Valgina, 2003; Wirza, 2017) and the development of methodological approaches to working with texts in the lessons of the Russian language is relevant to the question of how to assess the reader's literacy in the subject of the Russian language throughout primary school. The method of written text retelling used for many decades does not always make it possible to assess the depth of understanding of the text. In most cases, the actual written text retelling of the text based on its understanding is substituted by recording from memory: after listening to a small text, most students memorize it and then reproduce it verbatim. In addition, the criteria for evaluating the written retelling remain quite subjective, which makes it difficult to apply this method when conducting studies on large samples with evaluation according to unified criteria. And without regularly receiving information about the success of each student and his difficulties, it is impossible to build an individual educational trajectory, to provide pedagogical support. The lack of reliable data obtained from large samples seriously slows down the improvement of teaching methods. That is why the challenge was set to include tasks that involve working with text in modern monitoring and measuring materials on the Russian language. These tasks lie at the intersection of the evaluation of subject and metasubject results. As subject skills are estimated to choose a title of the text, to prove the choice, to make a plan of the text or to add missing items of the plan, to write a small coherent story about read. But the main condition for performing these tasks is a careful reading of the text, and this is already one of the metasubjective results: “mastering the skills of semantic reading texts of various styles and genres in accordance with the goals and objectives; skills to consciously build a speech statement in accordance with the objectives of communication and to compose texts in oral and written forms” (Federal'nyj gosudarstvennyj obrazovatel'nyj standart nachal'nogo obshhego obrazovanija, 2017, p.8).

Research Questions

As part of the study, a question was raised about the development of tasks in the framework of the monitoring work on the Russian language (grades 1–4), evaluating the planned results related to the work with the text. National and international experience in the development of such tasks was used (Campbell, 2017; Kuznetsova, 2013; Neugebauer, 2017). When selecting the content and form of these assignments, it was necessary to take into account compliance with the general criteria of monitoring work: the content of the assignment should be important for this class and be relevant to the next grade; the content of the works should allow to draw conclusions about the individual progress of each student, and this is possible when implementing the principle of continuity: there should be tasks of a certain type in the works thatallow one to judge the progress in mastering text skills.

Purpose of the Study

Development of models of tasks based on the text in the monitoring work on the Russian language from 1 to 4 class with the obligatory observance of the principle of comparability of results.

Research Methods

The goal set in the research was achieved within the framework of the monitoring work carried out by the Center for education quality assessment (head G. S. Kovaleva) in a number of regions of the Russian Federation. In the research theoretical and empirical methods were applied. Theoretical methods were used to search for answers to the following research tasks: selection of text tasks as objects of control in each parallel, taking into account the age possibilities of students and adherence to the principle of comparability; determination of the volume of tasks related to the text in the monitoring works from 1 to 4 class; volume of texts for compiling assignments, taking into account that text assignments are only part of the monitoring work; development of objective criteria for evaluating text tasks.

The objective complexity of the creations the tasks that evaluate text skills is related to the peculiarities of monitoring work, which presupposes individual work of the student one on one with the blank, some of these assignments imply the creation of a small creative work, and the monitoring features require the use of clear criteria for evaluating assignments that exclude subjectivism - this is for obtaining objective results. The inclusion of work with text in monitoring work required a solution to the question of a reasonable amount of text: the use of text of too small a volume did not allow an objective assessment of the planned result, and the use of text of too large a volume is impossible due to time limit for the work. Empirical methods included conducting monitoring work on large samples, analyzing the statistical characteristics of tasks, comparing the results. Statistical data processing was carried out using the ConQuest software. We monitored both the parameters of the monitoring work as a whole (the reliability of the test for the Cronbach coefficient alpha, standard measurement error) and the statistical characteristics of tasks related to the text (percent complete, differentiating ability, point-bisserial coefficients).

Findings

Analysis of the general list of textual skills in the list of planned results and the Russian language program by year of study allowed identifying appropriate control objects in monitoring work carried out at the end of grades 1-4 (table 1 ).

In the work for the 1-st grade one task of ten required was associated with the text. Let us give an example one of the task variants.

7. Divide the text into sentences. Write the text correctly.

in the forest the children saw a hedgehog he ran into the burrow on the back of a hedgehog leaf

Let us present the statistical characteristics of the task.

Item 7

Cases for this item 2092 Discrimination 0.63

--------------------------------------------------

Label Score Count % of tot Pt Bis

---------------------------------------------- ---

0 0.00 401 19.17 -0.48

1 1.00 666 31.84 -0.10

2 2.00 972 46.46 0.54

9 0.00 53 2.53 -0.22

===================================================

Out of 2092 first-graders, 1638 (78.3%) correctly defined the boundaries of the sentences, when making the proposal, no more than 2 mistakes were made, of which 972 (46.5%) completed the task absolutely correctly.

In grade 2, one of thirteen required tasks was associated with the text. Let's give an example of the text and the task.

13. Read the text.

Argued mind and kindness, which of them more people need. They came to the house where the two brothers lived together. The mind chose the elder brother, and the kindness chose the younger. Then the elder brother thought and said:

– It's time for us, brother, to live with our minds with every mind.

The younger brother sighed but agreed. The time has passed. Older brother was rich, but he lived alone. His mind noticed too many flaws in people, and he had neither friends nor a wife. A good brother, on the contrary, the house was poor, but friends and mates visited him every day. He helped everyone, and sometimes people did not hesitate to take the last from him.

Understand the mind and kindness that life has not become better for the brothers. They returned to the brothers and persuaded them to live together again and help each other.

Write down the main idea of the text. In your record should be 2-3 sentences.

Let us present the statistical characteristics of the task.

Item 13

Cases for this item 1798 Discrimination 0.53

--------------------------------------------------------

Label Score Count % of tot Pt Bis

--------------------------------------------------------

0 0.00 624 34.71 -0.35

1 1.00 554 30.81 0.09

2 2.00 430 23.92 0.45

9 0.00 190 10.57 -0.23

=========================================================

For correct performance of the task, it was possible to receive from 1 to 2 points. The main condition is the correct definition of the main idea, while 1 point was set if there were errors in the verbal design of the statement, 2 points were put if there were no comments on the coherence of the written statement. As can be seen from the results, 10.57% of second-graders refused to perform the task, 34.71% could not correctly identify the main idea of the text, and only 54.73% of second-graders coped with the task of determining the main idea of the read. It is important to note that 23.92% not only correctly identified the main idea, but also were able to make their coherent statement, to the form of which there were no comments. Having the data, the teacher got the opportunity to build work in the third grade in different ways: when working with 45.28% of students, it was important to work on defining the main idea of the text, with other students improving the existing skills when working with more complex texts, working out the ability to express the basic idea in a written statement.

At the end of grade 3 in the monitoring work of the 15 mandatory tasks for the evaluation of text skills were targeted 2 tasks. Let us give an example of these tasks of one of the options.

To complete tasks 14-15 you need to read the text.

Somehow the frogs decided to arrange a competition: who first gets to the top of the tower.

Gathered a lot of spectators. Everyone wanted to see how the frogs will jump and laugh at the participants. Of course, none of the spectators believed that at least one frog could climb up. Competitions began, and screams were heard from all sides:

— They will not succeed! It's too hard.

— No chance! The tower is too high!

One after another, the frogs fell down, but some still scrambled. The crowd screamed louder:

— Too difficult!!! None can do it!

Soon all the frogs were tired and fell. Except for one that went higher and higher… He was the only one who managed to climb to the top of the tower.

All began to ask the winner how he managed to find in himself so much strength. Turns out the winner was deaf. Isn't this helped him?

14.Make a plan of the read text.

15.Write down the main idea of the text. In your statement should be 2-3 sentences.

As in the second class, the text is devoid of direct didacticism, the main idea is not expressed explicitly, and for its understanding it is necessary to consciously read the text. Drawing up a plan for this text also requires conscious work with the text, since the presence of dialogues in the text does not allow a formal approach to determining the number of topics, based only on the number of paragraphs. We present the statistical characteristics of tasks.

Item 14

Cases for this item 2613 Discrimination 0.42

--------------------------------------------

Label Score Count % of tot Pt Bis

--------------------------------------------------

0 0.00 419 16.04 -0.31

1 1.00 2032 77.77 0.42

9 0.00 162 6.20 -0.26

Item 15

Cases for this item 2613 Discrimination 0.56

------------------------------------------------

Label Score Count % of tot Pt Bis

-----------------------------------------------

0 0.00 675 25.83 -0.33

1 1.00 930 35.59 0.05

2 2.00 740 28.32 0.47

9 0.00 268 10.26 -0.31

===========================================

77.77% of third-graders coped with the task to make a plan of the text, 16% of students began to perform the task, but the plan they proposed did not meet the criterion of reasonableness and balance, 6.2% of students did not perform this task. The task of formulating, in its own coherent statement of the main idea of the text, still caused difficulties. As it was already at the end of the 2nd grade, 10% of students did not start the task. 25.83% started the task, but did not cope with it (the positive thing is that it is 9% less than at the end of grade 2). 63.91% correctly identified the main idea of the text, while being able to express this idea in an immaculately coherent statement of 28.32% of the total sample (this is 4.4% more than at the end of grade 2). A comparison of the results of grades 2 and 3 leads to the conclusion that only 9% of students had a positive trend in identifying the main idea of the text during the year of study.

Of the 20 assignments in the 4th grade, three assignments were aimed at evaluating text skills. We give an example of one of the options.

Read the text and complete the tasks 18-20.

Katya went to the table and gasped: new colors are scattered, brushes are smeared.

– Alyoshka! – Katya screamed and, covering her face with her hands, cried loudly.

Alyosha looked into the room. His face was smeared with paint.

– I did nothing to you! – he said quickly.

Katya ran after him, but the brother jumped through the window into the garden.

– I'll take revenge on you! – Katya screamed with tears.

Alesha, like a monkey, climbed a tree.

– She cried! .. Because of some paints cried!

– You'll cry for me too! – shouted Katya. – You'll cry!

– It's something I'm going to cry? – Alyosha laughed and began to climb up.

Suddenly he stumbled and hung, clutching at a thin branch. A branch of a tree crunched and broke off. Alyosha fell.

Katya ran into the garden. She immediately forgot her spoiled paint and quarrel with her brother.

Little brother was sitting on the ground and looked at her in fear.

– Get up!

But Alyosha pulled his head into shoulders and closed his eyes.

– Cannot? – Kate screamed, feeling Alyoshin's knees. – Hold on to me.

She hugged the little brother by the shoulders and gently set him on his feet.

– Does it hurt you?

Alyosha shook his head and suddenly burst into tears.

– What, you cannot stand? – asked Katya.

Alyosha started crying even louder and pressed tightly to his sister.

– I will never touch your colors again ... never ... never ... never will!

(V. Osseyeva)

18.Valentina Osseyeva called her story “Revenge”. How else can you call this story? Suggest your own title.

19.Fill in the missing points of the plan.

1. Katya saw the spoiled paint.

2. ________________________________________.

3. ________________________________________.

4. Sister comes to the rescue.

5. I won't do it again!

20. What do you think this text is about? What is his main idea? Write about it. Your record should have 2-3 sentences, and it must be a coherent text.

When choosing tasks in the work addressed to primary school graduates, we proceeded from the fact that when assessing the text actions that were evaluated a year ago, it is necessary to complicate the tasks a little. For example, if at the end of grade 3 students were asked to make a plan of the text, and they themselves chose the degree of detail of the text in the event plan, in grade 4 students were asked to add items to the plan. A preliminary study confirmed our hypothesis that this is a more difficult action. The big difficulty is explained by the fact that the student needs to precisely match the already existing points of the plan with the text, understand what part of the text is not reflected in the plan, formulate micro-topics of this part and write down the missing points of the plan. The complication of the task of formulating the main idea of the text is due to the fact that students should relate the topic and the main idea of the test. We present the statistical characteristics of tasks.

Item 18

Cases for this item 6954 Discrimination 0.36

-----------------------------------------------

Label Score Count % of tot Pt Bis

---------------------------------------------

0 0.00 2073 29.81 -0.27

1 1.00 4570 65.72 0.36

9 0.00 311 4.47 -0.23

==============================================

Item 19

Cases for this item 6954 Discrimination 0.41

----------------------------------------------

Label Score Count % of tot Pt Bis

----------------------------------------------

0 0.00 2048 29.45 -0.32

1 1.00 4599 66.13 0.41

9 0.00 307 4.41 -0.23

=============================================

Item 20

Cases for this item 6954 Discrimination 0.53

-----------------------------------------------

Label Score Count % of tot Pt Bis

-----------------------------------------------

0 0.00 1505 21.64 -0.33

1 1.00 2694 38.74 0.05

2 2.00 2013 28.95 0.43

9 0.00 742 10.67 -0.27

==============================================

A generalized analysis of the results shows that approximately 30% of elementary school graduates (received 0 points or missed tasks) have difficulty understanding the text in depth, which makes it difficult for these students to choose a heading for the text, identify micro-topics of the text, highlight the topic and the main idea text and reflect them in their own coherent statement. Comparing the success of the task of formulating the main idea of the text at the end of 3-rd and at the end of 4-th grades, we can state that the dynamics are not too high: only 4% increased the number of students who coped with the task.

When conducting monitoring work, it is important to maintain parallel options, while due to the peculiarities of the author's text as a creative work; there are no absolutely identical texts. When selecting texts, it is important to carry out the most thorough analysis of the texts used in parallel versions of the work, to use expert assessments of the same difficulty for the perception of the text. However, even after the most thorough procedures, it is not possible to guarantee the complete identity of the texts and it is necessary to resort to statistical methods for analyzing the performance of tasks in variants with subsequent alignment.

Conclusion

Statistical characteristics of tasks evaluating text skills meet the requirements, which proves the possibility of using these task models in monitoring work. The above information about the percentage of students who received 1 and 2 points, point-bisserial coefficients show the correctness of the proposed criteria, which allows a more thorough description of the level of knowledge being assessed and provide more detailed information about each student. The ability to self-title a text, draw up or supplement a text plan, reconstructing the missing items, identify the main idea of the text, require both a high level of reading literacy and the ability to formulate your thought in the form of a text title, a plan item, a coherent statement. The results of the monitoring work show that one third of students during their primary school years do not master these skills. This means that it is necessary both to increase attention to the work with the text when learning the Russian language, and to include tasks that evaluate work with the text, because the data obtained will allow for a more subtle differentiation and provide timely assistance to students.

Acknowledgments

The work has been done within the framework of the state assignment to FSBSI “Institute for Education Development Strategy of the Russian Academy of Education” No. 073-00086-19-01 for 2019 and the planning period of 2020 and 2021. Project “Didactic support for the development of functional literacy of scholars in modern conditions”.

References

1. Aleksandrova, O. M., Aristova, M.A., Dobrotina, I.N., Gosteva, YU.N., & Vasilevykh, I. P. (2017). Language and literary literacy as components of learner’s functional literacy. In 2017 International Conference “Education Environment for the Information Age” (EEIA-2017). Moscow, Russia. DOI: 10.15405/epsbs.2017.08.6
2. Aleksandrova, O. M., Vasilevykh, I. P., Gosteva, Yu. N., Dobrotina, I. N., & Uskova, I. V. (2018). Realization of the principle of continuity in functional reading literacy forming. In 2018 International Conference “Education Environment for the Information Age” (EEIA-2018). Moscow, Russia.
3. Campbell, T.A. (2017). Verbal protocols in literacy research: Nature of global reading development. International Review of Education, 63(SI), 141-143.
4. Federal'nyj gosudarstvennyj obrazovatel'nyj standart nachal'nogo obshhego obrazovanija (2017). [Federal State Educational Standard of Primary General Education]. M-vo obrazovanija i nauki RF. Moscow: Prosveshhenie. [in Rus.].
5. Ippolitova, N. A. (1998). Tekst v sisteme obucheniya russkomu yazyku v shkole: Ucheb posobie dlya studentov ped. Vuzov [Text in the Russian language school system]. Moscow: Flinta, Nauka. [in Rus.].
6. Kuznetsova, M.I. (2013). Sistema kontrolya i ocenki obrazovatel'nyh dostizhenij mladshih shkol'nikov v sovremennoj nachal'noj shkole [The system of monitoring and evaluation of educational achievements of primary school students in the modern elementary school]. Moscow: Ventana-Graf [in Rus.].
7. Ladyzhenskaya, T.A., & Ladyzhenskaya, N.V. (2005). Tekstovye umeniya. Kak im uchit'? [Text skills. How do they teach?]. Nachal'naya shkola: do i posle, 5, 13-15.
8. Mullis, I.V.S., & Martin, M.O. (Eds.) (2015). PIRLS 2016. Assessment framework (2nd edition). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
9. Neugebauer, S.R. (2017). Assessing Situated Reading Motivations Across Content Areas: A Dynamic Literacy Motivation Instrument. Assessment For Effective Intervention, 42, 131-149. http://dx/doi.org/
10. Universal'nye uchebnye dejstvija kak rezul'tat obuchenija v nachal'noj shkole: soderzhanie i metodika formirovanija universal'nyh uchebnyh dejstvij mladshego shkol'nika [Universal learning activities as a result of training in primary school: the content and methodology of the formation of universal educational actions of the primary school student] (2016). Moscow: FGBNU «Institut strategii razvitija obrazovanija RAO». [in Rus.]
11. Valgina, N.S. (2003). Teoriya teksta [Text theory]. Moskva: Logos [in Rus.].
12. Vinogradova, N.F., Kochurova, E.EH., Kuznetsova, M.I., Petrashko, O. O., Romanova, V. Yu., Rydze, O. A., & Homyakova, I. S. (2018). Funkcional'naya gramotnost' mladshego shkol'nika [Functional literacy of the primary school student]. Moskva: Rossijskij uchebnik: Ventana-Graf [in Rus.].
13. Wirza, Y. (2017). Reading upside down: Identifying and addressing opportunity gaps in literacy instruction. Journal Of Language And Literacy Education, 13, 173-178.
14. Zuckerman, G. A., Kovaleva, G.S., & Baranova, V.Y (2018). Chitatel'skie umeniya rossijskih chetveroklassnikov: uroki PIRLS 2016 [Russian fourth-graders reading skills: lessons of PIRLS 2016]. Voprosy obrazovaniya, 1, 58-78 [in Rus.].