Assessment Of Schoolchildren’s Reading Literacy: Testing Methods And Results

Abstract

According to the international study of the reading literacy PIRLS, graduates of the Russian primary school are ready to study by means of texts. The PISA test shows a low level of readiness of 15-16 year old students to use texts for a wide range of tasks. In this regard, it is important to understand what happens with reading literacy at the beginning of the general school. In order to study the dynamics of reading literacy, a diagnostic technique was created. The article presents data on the reading literacy of pupils of grades 4 and 5, estimated with the help of this technique. It is shown that the year of study in the general school does not change anything in the ability of schoolchildren to understand information texts. These results show that reading texts does not contribute to the increase in the reading skills of the 5th grade students; therefore, they pose the problem of working on reading literacy in the teaching process in grades 5-6.

Keywords: Reading literacyreader's abilitymonitoring assessmenttesting methodsPIRLSPISA

Introduction

It is strongly believed in the pedagogical community that the educational achievements should be evaluated not by absolute learning outcomes, but by relative ones based on comparison between student’s previous and actual achievements. Taking into account the significance of the reading literacy achievement level both for studying and for successful day by day life, it is necessary to realize its achievement monitoring on a constant basis during the whole period of school life, reveal individual dynamics in order to help some certain groups of students in proper time and adjust the educational process. Based on two reading goals, reading for pleasure and reading to get information, reading literacy is evaluated during fiction and informational texts reading. Although accepting large significance of fiction texts reading, our study nevertheless is focused on informational texts reading as a component of reading literacy because the ability to read this type of texts is extremely important for education. This article reveals a tool that gives the possibility to evaluate the reading literacy dynamics of 4-5 grade students as well as the results of such evaluation.

Research Questions

The Russian educational system is characterized by a gap in reading literacy levels of primary and general school students. According to the PIRLS research data, students that finish primary school have a high reading literacy level and make part of the leaders group (Mullis et al., 2007), (Cole & Cole, 1993), (Luyten, Peschar & Coe, 2008), (Martin, Mullis & Foy, 2011; Mullis, Martin., Foy & Drucker, 2012), (Kintsch, 1998), (Mullis et al., 2012), ( Osnovniye resultati … PIRLS 2011 2013) . However, the PISA data shows that Russian 15 year old readers show a level below the international average ( Learning for tomorrow's world: First results from PISA 2003 2004) , ( PISA 2009 results, 2010), ( PISA 2009 results, 2010), ( PISA 2012 results, 2014). It is necessary to carry on searching for an answer to the following questions “Why quite a good reading literacy level that students have when they enter general school does not keep on rising? During which grade problems start to arise?” The answers for these questions will help to adjust the teaching methods. To solve the task that was set it is necessary using the comparable tools to evaluate reading literacy level at the end of each grade and therefore we chose two points: the end of fourth and fifth grade.

Purpose of the Study

To define the problems of the changes of reading literacy of graduates of the Russian primary school and to describe a diagnostic technique applied at the research.

Research Methods

During the organization of reading literacy monitoring as an ability to retrieve information from the text and use it in order to solve educational and practical tasks, we took as a basis the conception of two most famous and reliable reading literacy measurement instruments – PIRLS and PISA (Mullis & Martin, 2015), ( PISA 2009 framework, 2010), (Zuckerman, Kovaleva & Kuznetsova Marina, 2013), (Zuckerman, Kovaleva, & Kuznetsova, 2011), (Zuckerman, Kovaleva, & Kuznetsova, 2015). The research goal was to describe the individual level of 4 and 5 grade students’ reading literacy abilities. As the concrete objects of control three groups of readers’ actions were distinguished:

  • to read and understand texts that contain information exposed by different means (text, tables, brief notes), to reproduce or use information that is explicitly exposed (group 1);

  • to summarize and interpret information, check and formulate statements and conclusions that are based on it, work with data exposed in different forms (group 2);

  • to use information from the text in order to solve different practical and educational-cognitive tasks (group 3).

The 4th grade students performed work consisting of one part. The 5th grade students were asked to complete the work, which consisted of two parts: the first part of the work completely coincided with the work offered to the 4th grade students, and at the same time was close in content and form to the work that the 5th grade students performed a year ago, at the stage of completing primary education. This composition of the work for the 5th grade students made it possible to compare the level of performance of work by the same students in the 2014-2015 academic years and in the 2015-2016 academic years - the results of this comparison made it possible to talk about the dynamics of the reading literacy ability. Simultaneous implementation of the same work in the spring of 2016 by two age categories: 4th grade students and 5th grade students allowed to compare the level of performance of these two age categories. The second part of the work for the 5th grade students was more complicated, this part of the work is necessary both for clarifying the characteristics of the reader's literacy in grade 5, and for future research: in a year similar work will be offered to the same students who will become 6-th grade students. In the framework of this article, we will only discuss the results of the work common to 4th grade students and 5th grade students. The students performed the test work by reading the text and completing 16 tasks; two variants were used. The subject of the work presents a life situation that corresponds to the age features and level of training of students of grades 4-5: texts in both versions were guidebooks for exhibitions: "History of the bicycle" and "Genius of Leonardo da Vinci". The information was presented both verbally and in the form of diagrams and drawings. Four tasks (25% of the total number of tasks) were assigned to the assessment of the first group of skills, and 6 assignments (37.5% of the total number of assignments) for the assessment of the second and third groups. The proposed work was performed by 67501 4th grade students and 67068 5th grade students.

Findings

In accordance with the received scores for the performance of all tasks by the students, the qualitative level of the reading literacy ability was assessed. Four levels were identified: insufficient, basic, advanced and high.

Criteria for the allocation of levels and the percentage of students who completed work at each level are presented in Table 01 and in Figure 01 . They are based upon the research described in Kovaleva (2011), Kovaleva, Kuznetsova (2013), Kovaleva (2015), Kuznetsova (2009), Kuznetsova (2016).

Table 1 -
See Full Size >
Figure 1: Distribution according to the levels of reader literacy of students in grades 4 and 5 (Spring 2016).
Distribution according to the levels of reader literacy of students in grades 4 and 5 (Spring 2016).
See Full Size >

As follows from Table 01 , the outcomes of the 4th grade students and 5th grade students practically coincide (the difference of one percent is not statistically significant).

As was mentioned above, at the end of the 2014-2015 school years, the current 5th grade students did the work, compiled on the same basis. The similarity of approaches to designing the work and interpreting the results allows us to compare two datasets (Table 02 , Figure 02 ). The peculiarity of the analysis of results in 2015 was the allocation of a level "above the baseline" without dividing into two levels (advanced and high), while the criteria for the allocation of insufficient and basic levels coincided. That is why the data in Table 02 and Figure 02 are presented in three levels, while the results of the 5th grade students were restructured in 2016 (the percentages of fulfillment at advanced and high levels are combined into a level "above the base level").

Table 2 -
See Full Size >
Figure 2: The distribution according to the levels of reading literacy of the same students at the end of the 4th and the end of the 5th grade (Spring 2015, Spring 2016)
The distribution according to the levels of reading literacy of the same students at the end of the 4th and the end of the 5th grade (Spring 2015, Spring 2016)
See Full Size >

The data indicate that after the year of study there is no significant change in the distribution of students’ level of training, i.e. training in the fifth grade did not lead to a significant change in the level of reading literacy ability. Thus, the overall results of the two assessments almost coincided, i.e. the 5th grade students did not find any advantages over the 4th grade students in understanding the proposed text.

The total, most common result of the assessments requires a more detailed examination of the performance of tasks on the type of reading skills that they evaluate. The results of such analysis are shown in Figure 03 .

Figure 3: Results of performing tasks that are evaluating different reader skills
Results of performing tasks that are evaluating different reader skills
See Full Size >

There is no difference in the structure of reading literacy of the 4th grade students and the 5th grade students, all three groups of skills are approximately equally manifested in the work performance by two age groups. The students most successfully performed the tasks to reproduce information proposed explicitly in the text, when answering a specific question. The performance of the tasks of the second group of skills related to the analysis, interpretation of information (presented in the text verbally or in another form), presentation of the results of analysis in the form of a conclusion or statement, is 6% worse than the tasks performance of the first group. The success of the third group of tasks evaluating the use of information from the text for various purposes, the ability to link the information with the one obtained with prior knowledge, is 72-71%.

The third area of analysis was related to the consideration of the quality of the performance of each task included in the work. The parallelism of the variants and the direction of the tasks in two variants for evaluating the same skill allowed the two variant s assessment data to be combined. The results of tasks in two age groups of students are shown in Figure 04 .

Figure 4: The results of each task in two age groups
The results of each task in two age groups
See Full Size >

As can be seen from the figure, there are no significant differences in the performance of tasks by the 4th grade students and the 5th grade students, the quality of the performance is almost identical, while there are tasks both better performed by the 5th grade students and tasks that the 4th grade students performed better: the advantage of the 5th grade students do not exceed 5% , and for seven tasks their results are slightly worse. The biggest difference in the performance that is not in favor of the 5th grade students have been seen in task 9, it is all the more strange that the task is related to the definition of the century in which the event occurred, based on the year indicated in the text. For example, in the variant 2 it was necessary to answer the question, in what century Leonardo da Vinci was born, while in the text there is a sentence "Leonardo da Vinci (1452-1519) lived in Italy." For correct answer, it was necessary to understand that dates of life are indicated in parentheses, and to correlate year 1452 with the XV century (the 15th century record was also taken as correct). In the 5th grade there is more of this kind of information, but the percentage of the task performance was lower.

Taking into account the fact that if the task was fulfilled by 65% of students or more, the task is considered to be successfully completed, we can talk about three problematic tasks. The maximum difficulty for both the 4th grade students and the 5th grade students was caused by the fulfilment of tasks 1 and 12.

Task 1 assessed the ability of students to establish a correspondence between the text read and the purpose of this text, the ability to choose the most accurate and correct answer from the five proposed. Analysis of incorrect answers led to the conclusion that many students did not show the ability to determine the main purpose of the guidebooks, but chose too generalized answers. For example, after reading the guide to the exhibition "History of the bicycle", 18% of the 4th grade students and 23% of the 5th grade students chose the answer "get acquainted with the common types of transport" instead of the right answer "to inform about what can be seen at the exhibition".

Task 12 assessed the ability of students to find in the text evidence of the truth of the proposed statement. For example, in variant 1, this question was: "The text says: "A bicycle is one of the best inventions in the last 200 years." Find in the text evidence of the truth that the bicycle serves people for almost two hundred years." It was necessary to find in the text and write out the sentence "The first bicycle was invented in 1817". In order to correctly find this sentence, it was still necessary to keep in mind that at the time of the work, 2016 was going on. The task 14, aimed at assessing the ability to prove an assertion with the help of the information presented in the text, and with the help of the available subject knowledge and representations, was also rather complicated. For example, in variant 1 this task was formulated like this: "In the text, a bicycle is called an" iron horse ". Why is a bicycle compared to a horse?" In the correct answer students had to point out that comparing a bicycle with a horse is due to the fact that one can ride a bicycle like a horse / because the bicycle came to replace the horse / because the bicycle replaced the horse as a means of transportation.

Quite badly the students of both age groups performed the task 2, in which it was necessary to choose the answer about how long the exhibition will last. In variant 2 the text says that the exhibition will be held from March 1 to July 1, but at the same time, for example, only 55% of the 5th grade students chose the correct answer "4 months", and 30% of the 5th grade students chose the answer "5 months".

Discussion

Comparison of the results of the 4th grade students and the 5th grade students made it possible to reveal that there are practically no differences in the performance of tasks, and there is no difference in the distribution according to the levels of training. The question arises about the effectiveness of work on semantic reading in the course of training in the 5th grade. It is important to find pedagogical ways of such organization of training that would help the 5th grade students to improve the level of the semantic reading ability, to increase the motivation of students to raise their readership level. It is important for teachers to continue working to identify the potential of all subjects in the development of semantic reading ability and the ability to work with information as meaningful meta-subject results.

The main difficulties of students are related to the inability to link their knowledge with the text they read, to determine the coincidence and inconsistency of this knowledge, to use the text read and the available knowledge for answers to the question. Another difficulty is due to inaccurate perception of the information proposed in the text and inaccuracy (uncertainty) of the conclusions drawn, the answer.

Conclusion

Using the diagnostic method, the reading literacy of the students of the 4th and the 5th grades was measured, and it was found that the 5th grade students do not differ from the 4th grade ones in the level of understanding of information texts. At the same time due to the appearance of a number of new subjects, the volume of information texts in the educational process in grade 5 is much higher. The difficulty level of these texts is also higher: the content of the texts goes beyond the knowledge and experience of readers and is aimed at extending this experience. These studies show that reading texts does not contribute to the increase in the reading skills of the 5th grade students, they did not better extract information from texts, link information units into a single picture, reflect on the messages of texts, assess the completeness and accuracy of their understanding. It is important to improve the teaching methods in the 5th grade and, when studying all subjects, pay more attention to the development of reading skills.

The study was conducted in compliance with the State Assignment 27.7948.2017/BCh

References

  1. Cole M., & Cole, S. (1993). The development of children. N.Y.: Scientific American Books.
  2. Kintsch W. (1998). Comprehension: A paradigm for cognition. Cambridge. UK: Cambridge University Press.
  3. Knowledge and skills for life: First results from the OECD Programme for International Student Assessment (PISA) 2000. Paris, France: OECD Publications, 2001. 23 p.
  4. Kovaleva G.S. (2011) Resultati mezhdunarodnogo issledovaniya PISA: kachestvo obrasovaniya. Narodnoye obrasovanie. 2011. № 4. p. 193–200. [in Rus]
  5. Kovaleva G.S., Kuznetsova M.I. (2013) Mezhdunarodnoe sravnitelnoye issledovaniye PIRLS 2011 “Izuchenie kachestva chteniya i ponimaniya tekstov” i ego osnovniye resultati. Narodnoye obrasovanie. 2013. № 6. p. 199–209. [in Rus]
  6. Kovaleva G.S. (2015) Osvaivaem metapredmetniye rezultati – uchimsa dla zhizni. Vestnik obrasovaniya. 2015. № 10. p. 38–45. [in Rus]
  7. Kuznetsova M.I. (2016) Features of reading literacy of Russian primary students obtained from PIRLS results. SHS Web of Conferences, Volume 29 (2016). 2016 International Conference “Education Environment for the Information Age” (EEIA-2016), Moscow, Russia, June 6-7, 2016 / S.V. Ivanova and E.V. Nikulchev (Eds.). DOI:
  8. Kuznetsova M.I. (2009) Sil'nye I slabye storony chitatel'skoi deyatel'nosti vypusknikov rossiyskoi nachal'noi shkoly po resul'tatam PIRLS 2006. Voprosy Obrazovaniya. 2009. № 1. p. 107–136. [in Rus]
  9. Learning for tomorrow's world: First results from PISA 2003. Paris, France: OECD Publications, 2004.
  10. Luyten H., Peschar J., & Coe R. (2008). Effects of schooling on reading performance, reading engagement, and reading activities of 15-year-olds in England. American Educational Research Journal, 45(2), 319–342.
  11. Mullis I.V.S., Martin M.O., Kennedy A.M., & Foy P. (2007). PIRLS 2006 international report: IEA's progress in international reading literacy study in primary schools in 40 countries. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. 404 p.
  12. Mullis I.V.S., Martin M.O. (editors) (2015) PIRLS 2016. Assessment framework. 2-nd edition Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. 190 p.
  13. Martin M.O., Mullis I.V.S., & Foy P. (2011) Age distribution and reading achievement configurations among fourth-grade students in PIRLS 2006. IERI Monograph Series Issues and Methodologies in Large-Scale Assessments, 4, pp. 9–34.
  14. Mullis I.V.S., Martin M.O., Foy P. and Drucker K.T. (2012). PIRLS 2011 International Results in Reading. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College,
  15. Osnovniye resultati mezhdunarodnogo issledovaniya chitatelskoy gramotnosti PIRLS 2011: Analiticheskiy otchet. G.S. Kovaleva I dr. pod nauch. red. G.S. Kovalevoy. Мoscow, MAKS-Press, 2013. 132 p. [in Rus]
  16. PISA 2009 framework: Key competencies in reading, mathematics and science. (2010) Paris, France: OECD Publications.
  17. PISA 2009 results: What students know and can do: Student performance in reading, mathematics and science (Volume I). (2010) Paris, France: OECD Publications. 272 p.
  18. PISA 2009 results. Learning trends: Changes in student performance (Volume V). (2010) Paris, France: OECD Publications. 210 p.
  19. PISA 2012 results: What students know and can do: Student performance in mathematics, reading and science (Volume I). (2014) Paris, France: OECD Publications. 560 p.
  20. Zuckerman G.A., Kovaleva G.S., Kuznetsova M.I. Between PIRLS and PISA: The advancement of reading literacy in a 10–15-year-old cohort. Learning and Individual Differences, 26 (2013) p.64–73.
  21. Zuckerman G.A, Kovaleva G.S., Kuznetsova M.I. (2011) Pobeda v PIRLS I porazheniye v PISA: sudba chitatelskoy gramotnosti 10-15 -letnix shkolnikov. Voprosy Obrazovaniya. 2011. № 2. p. 123–150. [in Rus]
  22. Zuckerman G.A, Kovaleva G.S., Kuznetsova M.I. (2015) Stanovleniye chitatelskoy gramotnosti, ili noviye poxozhdeniya Tyani-Tolkaya. Voprosy Obrazovaniya. 2015. № 1. p. 284–300. [in Rus].

Copyright information

This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

About this article

Cite this paper as:

Click here to view the available options for cite this article.

Publisher

Future Academy

First Online

18.12.2019

Doi

10.15405/epsbs.2017.08.48

Online ISSN

2357-1330