Approaches To Assessing Students Learning Achievements: National And International Experience

Abstract

The article analyzes approaches to assessing students’ learning achievements, which international studies of quality of knowledge (PISA, International Civic and Citizenship Study (ICCS)) and main state examination (OGE) in “Social Sciences” use. The authors give an assessment of the place and role of international studies in the system of national education; show that the control and measurement materials affect the content and methods of teaching school subjects; and reveal the contradictory nature of this impact. Based on the comparison of tasks' models, common features were identified (the use of tests with one correct answer; the multidimensional, complex nature of tasks; the attraction of sources of social information, the ability to test knowledge and cognitive skills) and the differences associated with the problematic nature of the tasks of international monitoring. The paper states that the reflection of actual life problems and real situations in the international studies’ tasks creates the basis for evaluation of some personal qualities and civil competencies of students. The authors come to the conclusion, that the models of assignments, using in international studies of quality of education, would be effective in the teaching social science, and propose some ways of developing the control and measurement materials of the national exam taking into account international experience.

Keywords: PISAICCSSocial Sciencesfinal state examination (OGE)

Introduction

In the modern world assessing the achievements of students in various areas of education is not limited by the national framework. Globalization is a factor, which intensifies the international studies of the quality of education. International studies reveal similar tendencies in education in various states. National education faces “the problems of quality, the success of education, the development of global education as a condition for the stable development of the world community”. The solving of these problems helps “to determine the satisfaction with the state of the national school in comparison with other countries” (Batyukova, 2010; Brugelmann, 2006). The problem of education quality is considered widely. Students achievements are assessed as a basis for the economic and cultural development of the society (Clarke, 2012). The international monitorings reflect the formation of the personal qualities of the younger generation (Knowles & Di Stefano, 2015). Thus, based on the results of the ICSS, an assessment of civil competencies was given: the features, level of formation of civic norms (Hooghe, Oser & Marien, 2014), the specifics and relevance of civic values (Chow & Kennedy 2015).

An adequate assessment of the level of knowledge, skills, personal achievements of students is considered as the basis for making political decisions on the development of society as a whole, influences the formation of a citizen, attests to the achievement of the goals of education: “Achieving satisfactory learning outcomes for most of the population, which leads to a thoughtful and productive citizenry, is the fundamental goal of education” (Making use of assessments for creating stronger education systems and improving teaching and learning, 2017; Tyumeneva & Xavenson, 2012).

In Russia state educational authorities provide participation in international comparative studies of the quality of education (PISA, TIMSS, PIRLS, TALIS, ICCS, PIAAC) (Doklad ob itogah, 2016; Otchet o vypolnenii, 2017). Their results are included in a unified national system for assessing the quality of education (Kravczov, 2017), are used to improve the quality of national education and are considered as one of the approaches to educational programs improving (Bolotov, Valdman, Kovalyova & Pinskaya, 2013; Naydenova, 2014; Polovnikova, 2016).

Problem Statement

In domestic pedagogical studies, special attention is paid to comparing the results of monitoring and analyzing approaches to assessing the knowledge, skills and competences of participants (Polivanova, 2010; Valdman, 2013; Polovnikova, 2016; Lomovceva & Migunova, 2016; Polozhevecz, 2017; Polozhevecz, 2018; Polovnikova, Rutkovskaya & Sorokin, 2018). However, there is a lack of analyzing task models (control and measurement materials) in the aspect of improving the content and methods of teaching such school subject as “Social sciences”. The immediate purpose of using control and measuring materials in “Social sciences” is to provide reliable and objective indicators that testify to the nature and level of preparation of students. Nevertheless not only the content of the task, but also its form (model) affects the credibility of the results obtained. This dependence was revealed as far back as the 1920s, when so called “testological approach” was developing (Kostyuk, 1928; Bespal`ko, 2012). At the same time, as noted in the contemporary studies, control and measuring materials have a significant impact on the content of the subject and the methods of teaching (Lazebnikova, 2013; Lazebnikova, 2014). The nature of this impact is contradictory. As positive, it effects on involving tasks and models of the exam and on the expansion of interactive component in the study of a subject. But the content of the school subject is often narrowed to the boundaries defined by the framework of standardized verification. Important aspects (f. e., associated with the achievement of personal outcomes: the value-orientation of the personality, its regulatory, communicative and other qualities) stay behind the doors of the classroom. The tasks which are aimed at interdisciplinary outcomes and therefore skills, linked with the ability to transfer learned actions to other situations and meaningful contexts, also receive weak representation (Koval, 2017). In classroom teachers weaken attention to the achievement of these options. Thus, determining the directions for improving control-and-measuring materials, one should keep in mind the dual task. On the one hand, it is necessary to minimize the observed shortcomings and risks caused by the impact of standardized forms of testing on the content and methods of teaching. On the other hand, components from measuring instruments that have a positive impact on educational practice should be strengthened. The productive direction of resolving contradictions posed is the comparison of control and measuring materials used in international and national studies of the quality of education.

Research Questions

The search for ways to improve the control and measuring materials of the state final certification (OGE) in “Social sciences” in the 9th grade is connected with the use of international experience. To do this, it is necessary to determine: what is common and what is the difference in the approaches to the creation of models of examination tasks in international studies of the quality of education and in the national examination; which international approaches can be used to minimize risks while improving the models of the national exam.

Purpose of the Study

The purpose of the study was to identify common features and to determine the specificity of the measuring materials used in the main state examination (9th grade) in Russia and in international monitoring systems. The results would help to substantiate proposals for improvement of control and measurement materials of the main state examination in “Social sciences”.

Research Methods

In the framework of the study, a comparative analysis of the tasks of the national examination in “Social sciences” (2018) (Demonstracionnyj variant, 2018) and in international quality of education studies (PISA 2009, ICCS 2016) (Losito, Agrusti, Damiani, Schulz, 2017; PISA 2009 Results: Executive Summary, 2010; PISA 2009 Results: What Students Know, 2010; Osnovnye rezultaty, 2012; Schulz, Ainley, Fraillon, Losito, Agrusti & Friedman, 2017). The comparison considered correct, because the selection of data was made due to the following factors: a) partial coincidence of the objectives of the national exam and international monitoring; b) the coincidence of a number of components of the content of national and international monitoring (the problem of the ICCS international study covers the political and legal sphere that is widely represented in the OGE on social studies); c) the coincidence of skills verified (PISA 2009 research revealed the level of readers' literacy of students based on the work with the text, the OGE also checks the ability to work with the text and contains texts and a set of tasks to them); d) the coincidence of the form of the national examination and monitorings (standardized control); e) coincidence of the age of students, who participated in the national examination and international monitoring (about 15 years).

The task models’ comparison was carried out according to the following: 1) the type of cognitive activity (cognitive operations) required to carry out the task; 2) the type of task (with choice of answer or with freely constructed answer); 3) the representation of the task.

Findings

The fulfilment of the tasks of the national examination and international monitoring was connected with the wide range of similar cognitive operations (skills). The differences were bound with the goals of assessments. The purpose of international PISA monitoring was to assess the ability of students to understand a text or problem that is somehow related to the situation in the text, answer questions to the text or solve the problem using knowledge from a particular subject area. In ICCS, there were many tasks that test the formation of the ability to establish cause-effect relationships (for example: “What are two reasons why it is good for society that students, in addition to their own history and culture, understand the history and culture of other countries?”) So, the tasks contributed to the personal development and can assess the formation of social values and competences. The prevailing within the framework of the OGE were logical operations: compare, establish correlation, define a notion by an essential characteristic, and so on. Such tasks contributed to the formation of interdisciplinary skills (Koval & Lazebnikova, 2017).

The area of values was compensated by the content of “Social sciences”. Lack of attention to the identification of cause and effect relationships of social phenomena and processes appeared not only in the task models of the national exam, but also in the practice of teaching social science. This made it difficult for students to understand social reality in the full and could follow by the lack of a comprehensive approach to assessing social life.

The national examination and international monitoring actively used the model of tasks with the choice of one correct answer from the four proposed ones. The practice of using such tasks was controversial for the domestic pedagogy. Critics payed attention to the fact that the right answer for test tasks with a single choice could be based on guessing. In other words, even if the student made the right choice, one could not say with certainty that he mastered this concept or understood the meaning of the given judgment. Comparative analysis of the results of tasks from different parts of examination paper partly confirmed this conclusion. For example, the student correctly chose the characteristic of the federal state structure, but in interpreting the data of the diagram in the second part of the work he made a mistake, which showed the misunderstanding of the basic features of this state form. That was why since 2015 a substantial reduction of such tasks in the materials of final attestations was made. At the same time, some researchers noted the value of such tasks, their adaptability and efficiency: this model contributed to the efficiency of verification, the simplicity of evaluation, allowed to test a large body of knowledge (Avanesov, 2012) and even the “ability to think” (Bolotov, 2014).

In international studies, tasks with a single-choice answer were used very widely, even when working with text. For example, an assignment to the texts about telecommuting: “How do the texts “Such will be the future world” and “The world on the verge of misfortune” correlate with each other?

• They used different arguments in order to arrive at a general conclusion.

• They are written in one style, but are devoted to completely different topics.

• They express the same general point of view, but come to different conclusions.

• They express opposing views on the same topic.” (Osnovnye rezultaty, 2012).

To do this task properly, it was not enough to find the relevant information in the text, it was necessary to analyze and compare the judgments expressed by different authors.

It could be argued about ability of single-answer tests to determine the level of students’ skills and knowledge. However, this task model should be used along with other models, if it met certain requirements, one of which should be the “credibility” of incorrect answers.

The majority of international studies’ tasks were constructed on unfolded conditions, which described a specific situation that corresponded the reality of modern social life and a certain social problem, f. e. “Individuals or groups of individuals sometimes give money to political parties as donations. In some countries, there are laws in which political parties should provide the public with access to information about donations. Why do these countries have these laws?”

Situations of real life were used also in single-answer tests: “People who work at noisy workstations have received a disease associated with hearing damage. What could governments do to cope with the problem of noisy jobs?

• Immediately close all noisy workplaces.

• Give workers money so that they can find work in more regular jobs.

• Introduce laws that require employers to protect their employees from noise.

• Arrest all owners of noisy jobs.”

Similar tasks were used in OGE on social sciences’ tests also modelled situations. However, quite often these situations were abstract and did not reflect the realities and problems of today's life: “In country Z there is a commodity production and money circulation. What additional information will lead to the conclusion that the economy of country Z is of a market nature?”

The increase of the number of assignments in the national exam related to modern social issues in typical situations, familiar to students, would help to strengthen the connection between the schooling and life.

In study of students’ performance in reading (PISA), two texts were used in an examination paper. They related to common problems, but contained different assessments of the problem or approaches to solving it. In the OGE, students worked with one piece of text. Several years ago, as part of the testing of new assignments, a model was proposed using two fragments. However, it was rejected, partly because it was very difficult to select text pairs when creating a large number of variants of examination papers in social sciences.

Nevertheless, using of two fragments of text would allow to test knowledge and skills in several aspects. Consider the second task to the above-mentioned fragments about telecommunications: “What kind of work will be difficult to per-form under telecommuting conditions? Give one example. Justify your answer.” There was no information about work in telecommuting environment in both texts. What was more, the explanation of term “telecommuting” was given in a footnote to the texts. To perform this task, students needed to correlate the information with the types of professional activity known to them. In this case, they had to: a) analyze the features of work in various professional fields on the basis of their knowledge or social experience; b) correlate these features with telecommuting; c) justify answers.

Such tasks were actively used during the first years of the national exam in the beginning of 21st century. For example, to the text revealing the achievements of modern science, the following task was proposed: “Discoveries of scientists in the field of chemistry, biology, and pharmaceutics helped create new effective medicines against incurable diseases. What feature of modern science reflects this fact? Which piece of text supports your conclusion?” (Lazebnikova & Korolkova, 2011). Nowadays these task models included in OGE on social science seldom. Refusal from the assignments of that kind could hardly be approved.

The next task to the text was a variation of a single-answer test: to identify the statement among four with which the authors of both texts would agree. According to psychologists, the task of finding a similarity was more difficult for students than finding a difference (Vygotskij, 1999). The solution demanded deeper analysis and involved several types of training skills. So the simple test form was effective for assessing the complicated skills.

The first task to the text in the OGE required drawing up a detailed plan, i.e. was directly related to reading skills. Others tasks required the search for necessary information in the text; transformation of information based on establishing links with previously acquired knowledge; specifying the ideas put forward in the text. The students' ability to attract social science knowledge and personal social experience were tested in modelling social situations related to the problem considered in the text, for example: “What forms of recreation and entertainment do the authors recommend to young people? Give two examples of such forms; in each case, indicate what personal qualities this form develops.” Students also had to support the thesis given in the text with their own arguments. Thus, PISA assignments required to use subject knowledge and skills for solving a problem contained in a text, OGE text was also a material for checking knowledge.

Conclusion

Comparative analysis of the tasks of the national examination in “Social sciences” (2018) and international quality of education studies (PISA 2009, Schulz, Ainley, Fraillon, Losito, Agrusti & Friedman, 2017) showed the similar features of the models used (usage of single-answer tests and sources of social information, complex nature of tasks) and their effectiveness in checking the wide range of students' cognitive skills. The main difference was in the tasks content and level of reflection actual life problems and real situations in them. The latter created the opportunity for assessing in international studies not only knowledge and skills, but also civil competencies and some personal features. The choice of this approach to improve the tasks of the national exam required further research. However, tasks similar to the tasks of international monitoring, would be effective in the practice of teaching social science.

Acknowledgments

This paper is prepared in the framework of the topic: “The upgrade of the general education content and training methods in the conditions of a modern information environment”. Project No 27.6122.2017/BCh.

References

1. Avanesov, V.S. (2012). Osnovnye napravleniya razvitiya pedagogicheskih izmerenij. Shkolnye tehnologii, № 1. [in Rus.]
2. Batyukova, Z.I. (2010). Sravnitelnoe obrazovanie v kontekste mezhdunarodnyh sravnitelnyh issledovanij. Problemy sovremennogo obrazovaniya, № 3. [in Rus.]
3. Bespal`ko, V.P. (2012). Testy v obrazovanii: byt` ili ne byt`? Shkolnye tehnologii, № 1. [in Rus.]
4. Bolotov, V.A. (2014) Minusy vizhu, plyusov ne nablyudayu. Uchitelskaya gazeta, № 34. [in Rus.]
5. Bolotov, V.A., Valdman, I.A., Kovalyova, G.S., Pinskaya, M.A. (2013). Rossijskaya sistema ocenki kachestva obrazovaniya: glavnye uroki. Kachestvo obrazovaniya v Evrazii, № 1. [in Rus.]
6. Brugelmann, H. (2006). International tests and comparisons in educational performance: a pedagogical perspective on standards, core curricula, and the quality of schooling. International Perspectives on Educational Policy. Ed. By Ralph Rotte. Nova Science Pb.
7. Chow, J.K.F., & Kennedy, K.J. (2015). Asian students’ conceptions of future civic engagement: Comparing clusters using person-centered analysis. Research in Comparative and International Education, 10 (1).
8. Clarke, M. (2012). What Matters Most for Student Assessment Systems: A Framework Paper. Systems Approach for Better Education Results (SABER) student assessment working paper, no. 1. World Bank, Washington, DC. © World Bank. Retrieved from: https://openknowledge.worldbank.org/handle/10986/17471
9. Demonstracionnyj variant kontrolnyh izmeritelnyh materialov dlya provedeniya v 2018 godu osnovnogo gosudarstvennogo ekzamena po obshhestvoznaniyu, (2018). Retrieved from: http://vpr-ege.ru/zagruzki/oge2018-ob-demo.pdf [in Rus.]
10. Doklad ob itogah deyatelnosti Federalnoj sluzhby po nadzoru v sfere obrazovaniya i nauki za 2016 god i analiz ih rezultatov (2016). Retrieved from: http://www.obrnadzor.gov.ru/ru/activity/main_directions/plans_and_reports/ [in Rus.]
11. Hooghe, M., Oser, J., & Marien, S. (2014). A comparative analysis of ‘good citizenship’: A latent class analysis of adolescents’ citizenship norms in 38 countries. International Political Science Review, 37 (1).
12. Knowles, R.T., Di Stefano, M. (2015). International Citizenship Education Research: An Annotated Bibliography of Research using the IEA ICCS and IEA CIVED Datasets. Journal of International Social Studies, Vol. 5, No. 2.
13. Kostyuk, G.S. (1928). O zavisimosti rezultatov testirovaniya ot formy testa. Testy: teoriya i praktika. Moscow, Mosk. testologicheskoe ob`edinenie. [in Rus.]
14. Koval, T.V. (2017). O sformirovannosti metapredmetnyh umeniy uchashchikhsya v osnovnoy shkole: chto pokazala diagnostika. Prepodavanie istorii i obshhestvoznaniya v shkole, № 4. [in Rus.]
15. Koval, T.V., Lazebnikova, A.Yu. (2017). Interdisciplinary outcomes as a new component of educational standards. The European Proceedings of Social & Behavioural Sciences (EpSBS).
16. Kravczov, S.S. (2017). Uchastie Rossijskoj Federacii v mezhdunarodnyh sravnitelnyh issledovaniyah kachestva obrazovaniya. Pedagogicheskie izmereniya, № 2. [in Rus.]
17. Lazebnikova, A.Yu. (2013). Vliyanie trebovanij i standartizirovannyh form edinogo ekzamena na izuchenie obshhestvoznaniya v starshej shkole. Prepodavanie istorii i obshhestvoznaniya v shkole, № 10. [in Rus.]
18. Lazebnikova, A.Yu. (2014). Vliyanie trebovanij i standartizirovannyh form edinogo ekzamena na izuchenie obshhestvoznaniya v starshej shkole. Prepodavanie istorii i obshhestvoznaniya v shkole, № 1. [in Rus.]
19. Lazebnikova, A.Yu., Korolkova, E.S. (2011). GIA 2011. Obshhestvoznanie. 9 klass. Moscow. Drofa. [in Rus.]
20. Lomovceva, N.N., Migunova, L.P. (2016). Aktualnye napravleniya ocenki kachestva obrazovaniya v RF. Trebovaniya i perspektivy. Ekonomika truda, T. 3. № 2. DOI: 10.18334/et.3.2.36096. [in Rus.]
21. Losito, B., Agrusti, G., Damiani, V., Schulz, W. (2017). Young People's Perceptions of Europe in a Time of Change. IEA International Civic and Citizenship Education Study 2016 European Report. Retrieved from: http://iccs.iea.nl/cycles/2016/findings/single-finding/news/iccs-2016-european-report-young-peoples-perceptions-of-europe-in-a-time-of-change/
22. Making use of assessments for creating stronger education systems and improving teaching and learning. Paper commissioned for the 2017-8 Global Education Monitoring Report, Accountability in education: Meeting our commitments. (2017). Retrieved from: http://unesdoc.unesco.org/images/0025/002595/259564e.pdf
23. Naydenova, N.N. (2014). Rol mezhdunarodnyh sravnitelnyh issledovanij v razvitii standartizacii shkolnogo obrazovaniya. Novoe v psihologo-pedagogicheskih issledovaniyah, № 3. [in Rus.]
24. Osnovnye rezultaty mezhdunarodnogo issledovaniya obrazovatelnyh dostizhenij uchashhihsya PISA-2009. Analiticheskij otchyot (2012). Pod nauch. red. G. S. Kovalevoj. Moscow. MAKS Press. [in Rus.]
25. Otchet o vypolnenii Plana deyatelnosti Federalnoj sluzhby po nadzoru v sfere obrazovaniya i nauki za 2017 god. (2017). Retrieved from: http://www.obrnadzor.gov.ru/ru/activity/main_directions/plans_and_reports/ [in Rus.]
26. PISA 2009 Results. Executive Summary. OECD 2010. (2010). Retrieved from: https://www.oecd.org/pisa/pisaproducts/46619703.pdf
27. PISA 2009 Results. What Students Know and Can Do – Student Performance in Reading, Mathematics and Science. (2010). Volume I. Retrieved from: http://dx.doi.org/10.1787/9789264091450-en 28.03.2018
28. Polivanova, K.N. (2010). Nacionalnye i mezhdunarodnye programmy ocenki kachestva obrazovaniya: ucheb. posobie. Moskow. Logos. [in Rus.]
29. Polovnikova, A.V. (2016). Sozdanie sistemy nacionalnoj ocenki kachestva obrazovaniya v Rossijskoj Federacii i mezhdunarodnye issledovaniya. Aktualnye voprosy gumanitarnyh nauk. Teoriya, metodika, praktika. Materialy III Vserossijskoj nauchno-prakticheskoj konferencii. Pod redakciej A.A. Sorokina. [in Rus.]
30. Polovnikova, A.V., Rutkovskaya, E.L., Sorokin, A.A. (2018). Uchastie rossijskih shkolnikov v mezhdunarodnom issledovanii po finansovoj gramotnosti: rezultaty i perspektivy (ili chem mozhet pomoch shkolniku uchitel). Prepodavanie istorii i obshhestvoznaniya v shkole, № 3. [in Rus.]
31. Polozhevecz, P. (2017). Ryvok, kotoryj potryas mir. I nas samix. Uchitelskaya gazeta, №№ 48-52. [in Rus.]
32. Polozhevecz, P. (2018). Trudnosti mnimye i yavnye. Uchitelskaya gazeta, №№ 7-10. [in Rus.]
33. Schulz, W., Ainley J., Fraillon, J., Losito B., Agrusti, G., Friedman, T. (2017) Becoming Citizens in a Changing World. (2017). IEA International Civic and Citizenship Education Study 2016 International Report. Retrieved from: http://iccs.iea.nl/cycles/2016/findings/single-finding/news/iccs-2016-international-report-becoming-citizens-in-a-changing-world/
34. Tyumeneva, Yu.A., Xavenson, T.E. (2012). Mezhdunarodnye sravnitelnye issledovaniya kachestva obrazovaniya dlya ocenki effektivnosti reform. Problemy, ogranicheniya i vozmozhnye resheniya. Tendencii razvitiya obrazovaniya: problemy upravleniya i ocenki kachestva obrazovaniya. Materialy VIII Mezhdunarodnoj nauchno-prakticheskoj konferencii. [in Rus.]
35. Valdman, I.A. (2013). Podxody k upravleniyu kachestvom obrazovaniya na osnove ispolzovaniya procedur ocenki obrazovatelnyh dostizhenij uchashhihsya. Upravlenie obrazovaniem: teoriya i praktika, № 4. [in Rus.]
36. Vygotskij, L.S. (1999). Myshlenie i rech`. Moscow. Labirint. [in Rus.]