Does PISA Help Fighting for Social Equity?
Abstract
The Program for International Student Assessment (PISA) has been implemented for more than 20 years, amounting to 7 full cycles of data collection every three years. With this scale and quantity of data, PISA has had a profound impact on educational research and policies worldwide. Since the very beginning, PISA has claimed the promotion of equity as one of its major goals, and with every cycle of PISA we acquire more information and recommendations on equity promotion. Though some of these recommendations have been accepted and implemented at the national level, not every country has succeeded in reducing the inequity levels, and difficulties remain in understanding and assessing the role of PISA in this process. In this paper, we discuss the controversies around the usage of PISA results for shaping national policies. We discuss how the concept of equity appears in PISA framework, how the results of PISA are being used on the national level and the observed changes in equity over time. This is an exploratory study, and it intends to review the current state of knowledge about the connections between PISA and equity and find the gaps in this knowledge.
Introduction
Social equity is one of the most elusive goals set by educational organizations around the world (Ainscow, 2020; UNESCO, 2017). Throughout the last decades, scholars, teachers and politicians around the globe have been trying to find solutions to advance social equity, but the results are controversial (Sharma, 2020). The general idea of equity is rooted in the concept of fairness and equality of opportunity, which refers to the notion that personal results should depend more on individual effort than on personal circumstances (OECD, 2019b). Nevertheless, a concrete definition of the concept is not easy to achieve. For example, in the official UNESCO guide, the definition of equity is quite vague: “Ensuring that there is a concern with fairness, such that the education of all learners is seen as being of equal importance” (UNESCO, 2017).
Therefore, although the general idea is quite straightforward, it is difficult to provide a more specific definition of it, because the concept of fairness developed over time and transformed depending on contextual factors that need to be taken into a consideration; indeed, we can track how philosophical concepts of justice, fairness and equality developed over time (Roemer & Trannoy, 2016). Although inequality of opportunities due to arbitrary social status was once considered just and fair, nowadays it seems inappropriate, as society has moved towards more egalitarian views. The doctrine that all people deserve equal rights and opportunities is rather new in human history, having developed gradually during the 20th century, and still evolving currently. In western democratic countries, equity and equality are seen as basic human values that define the principles of fair society (UNESCO, 2017). Yet today, despite all the progress, economic and social barriers still exist, and opportunities are not equal for all the children, even in the developed countries. Therefore, the challenge remains.
Problem Statement
Quantitative research and data analysis of equity may help moving towards a more fair society. The OECD's Program for International Student Assessment (PISA) plays a role in this process, as it was designed to provide cross-nationally comparable evidence of students’ performance and equity levels in different countries, and of the factors that contribute to it (OECD, 2019c). PISA, currently one of the largest international education studies, provides countries and policymakers with the quality information they need to support their decision-making processes. It measures 15-year-old’s ability to use their mathematics, reading and science knowledge and skills in real-life situations (OECD, 2019a). In 2018, 79 countries from all over the world participated in PISA (OECD, 2019a). Since its beginning in 2000, PISA results and international rankings have received a lot of media attention and have had a profound effect on education policies globally (Hopfenbeck et al., 2018; Seitzer et al., 2021).
Research Questions
In this study we explored the connection between PISA and modern equity policies in education. We reviewed three major aspects of this connection. First, we explored how the concept of equity is understood and defined within the PISA framework and how PISA operationalizes it. In other words, we analised what indicators are used to measure and report the levels of equity. Next, we explored how the results of PISA were being used to shape national policies and educational standards, and what were the challenges, controversies, and critiques in that respect. Finally, we referred to literature that describes the observed changes in social equity over the last two decades and discussed how they can be related to PISA.
Purpose of the Study
Reducing educational inequity is part of the global PISA agenda, so the reports systematically feature analyses on each country’s equity indicators, as well as recommendations for its’ improvement. Despite this, 20 years after the first wave of PISA, there are still debates if the participating countries have improved their levels of equity since they began participating in PISA. Although it is impossible to track the causalities between PISA and equity, we can nonetheless explore the role of PISA in the search for increased equity. For this, we will review what PISA tells us about equity, how this information is being used in practice, and analise results that have been reported over the last two decades.
Research Methods
This study presents a literature analysis to tackle the research questions mentioned above. For that, we dug into the PISA technical documentation and reports. To search for the relevant literature for these three tasks, we used EBSCO, Web of Science and Scopus databases. We focused on the literature published since the beginning of 2000, because it is the year of the first PISA study. This is an exploratory study, and it intends to review the current state of knowledge about the connections between PISA and equity and find the gaps in this knowledge.
Findings
In this study, we reviewed three aspects of the connection between PISA and equity: how PISA frames the concept, how results are used and what changes are observed. The findings are structured according to the research questions, mentioned above, and present the main findings in each aspect. These results are used to find gaps in the current state of knowledge and to propose a methodology to fill these gaps.
According to its own reports and technical documentation, PISA operates with the concept of equity, and it concentrates on the principles of inclusion and fairness (OECD, 2019b). In this framework, equity does not mean that all students achieve the same results, but it means that every student is given the same opportunity to realize their potential and acquire the skills necessary to participate in society regardless of their family and social background. Thus, in the PISA framework (OECD, 2019b, 2019c), equity is measured as a connection between the personal and family background features (such as socio-economic status, gender, or nationality) and the educational outcomes, such as access to school, educational performance, student’s attitudes and beliefs, and student’s expectations for the future. In PISA reports, this connection is called a “socio-economic gradient” which shows how well student’s performance can be predicted by their background characteristics. The role of the school system is to weaken this relationship and to compensate for an unfavorable family background.
In the official reports, PISA uses several indicators of equity to highlight its different facets and to help building a multi-dimensional picture of the subject (OECD, 2018, 2019b). The PISA coverage index reflects the participation rate of the national 15-year-old population in PISA, which indirectly indicates the general access to education for this age group. Another indicator reveals the percentage of variation in PISA performance that is explained by student’s ESCS; this shows the intensity of the relationship between the family background and academic performance. In the reports, we can also find information about gender differences and the effects of an immigrant background of students. Finally, the PISA report includes an analysis of resilient students – those who have good performance results despite their unfavorable family backgrounds. Among these indicators, most studies focus on the strength of the relationship between performance and socio-economic status because it directly reflects the level of inequity in society. These are the indicators reported in PISA official documents; however, in other studies, we can find more, different indicators (Rolfe, 2021; Srinivasan, 2021; Sulis et al., 2020). Such plurality of indicators, on the one hand, gives us more opportunities to explore equity and to build a bigger picture of the phenomenon. Yet, on the other hand, it complicates the comparisons and the interpretations since different indicators might present different results.
PISA is commonly recognized as a robust and legitimate measure of success of educational systems by the public, politicians, and the media (though controversial between academics); thus, it can have a strong impact on the national politics in education. In the literature, we find examples of the so-called “PISA shock”, when unexpectedly low results in PISA lead to fundamental revisions of national educational systems. The case of Germany is one of the clearest and most studied examples of such reforms. The German educational system was believed to be “excellent” until, in 2001, the first results of PISA were published, showing that the country’s results were below OECD averages. Not only the performance indicators were below average, but the country demonstrated high levels of inequity between children as well. This sparked intense social and political debates about the state of education in country, which led to massive reforms in the entire education system. New education standards were introduced, as well as quality assurance and an increase of school autonomy. Special attention was paid to underachieving students and to early advancement of socio-economically disadvantaged children. As a result, the average scores in PISA started to increase, and the growth has been mostly motivated by the improvement of low-achieving students (Niemann et al., 2017).
It is argued that since the establishment of PISA, the OECD has become a major driver of national education reforms worldwide (Mzhavanadze, 2014; Seitzer et al., 2021). There are many examples of implementing OECD recommendations at the national level, for example in Germany (Niemann, 2016), Spain (Engel, 2015; Tiana Ferrer, 2017), Japan (Takayama, 2008), Portugal (Afonso & Costa, 2009; Carvalho et al., 2017), Poland (Białecki et al., 2017), Canada (Morgan, 2016) and others (Michel, 2017). Most of these authors credit PISA as being a very significant and powerful tool for the national educational systems in their respective countries; however, some of them are concerned about certain limitations of and even a “disenchantment” with PISA (Mangez & Hilgers, 2012; Morgan, 2016; Tiana Ferrer, 2017).
Breakspear (2014) describes the challenges that can arise from the use and potential misuse of PISA. The first challenge is rooted in the misuse of PISA ratings for political reforms, which can lead to rank-seeking policy actions. The media attention to the relative ranks and league-tables creates a competitive atmosphere between the countries and might lead to rank-seeking political decisions that disregard the actual quality of teaching and learning. Some countries explicitly define their educational goals in terms of PISA rankings and league-tables (Breakspear, 2012), although a relative ranking position is not an appropriate goal for an educational system, as this position depends not only on the success of a particular country, but also on the failure of its’ neighbours. Increasing national equity and well-being is not a competitive sport and the goals must be set based on internal standards and indicators.
The second controversy refers to PISA indicators, which are very specific and well-defined. In a massive project like PISA, it is necessary to create a common ground, which offers an opportunity to compare countries and to track the changes over the years. It creates a “common language” in PISA studies, where most concepts are operationalized in a familiar way. On the other hand, such focus can limit the end-goals of education. If we focus on the same indicators for the last 20 years, we narrow our vision and our understanding. According to Campbell’s (1979) law,
The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures, and the more apt it will be to distort and corrupt the social processes it is intended to monitor. (p. 85)
Ainscow (2020) writes about this problem:
On the one hand, data are required in order to monitor the progress of children, evaluate the impact of interventions, review the effectiveness of policies and processes, plan new initiatives, and so on. On the other hand, if effectiveness is evaluated on the basis of narrow, even inappropriate, performance indicators, then the impact can be deeply damaging. Whilst appearing to promote the causes of accountability and transparency, the use of data can, in practice: conceal more than they reveal; invite misinterpretation; and, worse of all, have a perverse effect on the behaviour of professionals. (p. 10)
Moreover, the notion that all countries can be measured by the same indicators arises strong concerns and questions about the role of cultural contexts. Since the beginning of the project, PISA indicators have been scrutinized and critisized for being “not valid”, “globalist”, “flawed” or “misleading” (Zhao, 2020). PISA is not perfect, but this discussion helps to promote awareness that these indicators should not be treated as an ultimate truth, but only as a piece of a bigger picture.
One final last controversy refers to PISA’s classification of educational systems as high, average, or low performing. Countries like Finland, Korea, and Canada, which were branded as “success” cases, now act as a “reference” system for other countries (Schleicher, 2009). This creates a tendency for “cherry-picking” the policies from the high-performing countries with disregard to their fit to national context (Breakspear, 2014). This problem is aggravated with the tendency towards “Westernization” and “Europeanization”. Though Schleicher tried to showcase four countries from the different parts of the world, recent research shows that western European countries are being referenced as models more often. Seitzer reviewed 296 national reports published by OECD between 1961 and 2019 on the topic of education, looking for countries mentioned in the reports. A total of 129 countries were mentioned, and the most referenced ones were Finland (1729 mentions), Germany (1685), Sweden (1407) and Australia (1300). On one hand, this can be explained by the community of European OECD-member countries that like to refer to the neighbours. On the other hand, it shows that OECD’s recommendations lean toward a western European model of education (Seitzer, 2020). Such an approach limits the possible sources of learning, because only the “top” countries are considered an appropriate source of inspiration. However, what is good for one country may not work, or even cause damage, in another (Zhao, 2020).
Thus, like any other powerful instrument, PISA should be used with caution and consideration of contextual factors. Most problems emerge from the misuse of PISA results and the disregard of cultural contexts, as discussed above. However, problems also occur at the national level in political decisions. Fischman shows that politicians often refer to PISA in their electoral agendas, but little impact is perceived by the stakeholders (that in this research include academics, NGO researchers, educators and policymakers). Often, PISA is used as a tool to legitimize educational reforms; yet, there is little evidence that there is any causal relationship between PISA results and the implementation of such reforms (Fischman et al., 2019). Fishman mentions examples where “national participation in international large scale assessments and their resulting rankings have been used to legitimize and accelerate pre-existing or pre-planned educational reform efforts” (Fischman et al., 2019, p. 7). Thus, he argues that PISA is both a facilitator and an impediment to national reforms. Another example comes from Japan, where Takayama described the context and the consequences of PISA shock in this country, when media representation and the political agenda shaped the discourse around PISA results to a point in which “PISA findings were appropriated to de-legitimise the yutori reform, the very curricular reform closely aligned with the global trend in curricular reform promoted by the OECD” (Takayama, 2008, p. 401). Thus, the relation between the political agenda and the interpretations (and misinterpretations) of PISA results are of a complex nature.
We discussed the critiques and controversies related to PISA, but also the fact that OECD and PISA play a significant role in educational policies. Next, we will try to see what results have been reported during the 20-year history of the project. Despite the problems, PISA reports indicate a positive trend in increasing equity over the past ten years, and in many countries this progress is based on the improvements in performance of the most disadvantaged students (OECD, 2017; OECD 2018). According to PISA studies, educational attainment and access to education have improved during the last decades, and children are attaining, on average, a higher level of education regardless of their level of income.
In 2015, ESCS was responsible for approximately 13% of the variation in students’ performance across OECD countries. The strength of this connection (also known as “socio-economic gradient”) varies across countries, from around 5% in countries like Iceland, Hong Kong, and the United Arab Emirates, to more than 20% in France, Argentina, and Peru. Between 2000 and 2015, equity in reading performance improved in 11 out of 35 countries with comparable data in these cycles of PISA (OECD, 2018, p. 63). In 2018, the average percentage of performance variation explained by ESCS in OECD, countries was 12% (OECD, 2019b). Although the percentage number went down, the interpretations should be done very carefully, because these numbers are not fully comparable due to differences in the samples and the methodologies. In many countries, the impact of SES on students’ performance is still very strong and has not changed significantly during the last years (OECD, 2017). The 2018 PISA report shows that the gap between disadvantaged and advantaged students results in an average difference of 88 points in the PISA score, which is equivalent to three years of schooling; this is a clear indication that the problem of equity is remains unsolved.
PISA surveys provide a variety of data and indicators, which, on the one hand, help to build a multi-dimensional picture, but, on the other hand, complicate the interpretation of the results and comparisons between different studies. There is more consistency within the OECD reports, where the same indicators and forms of representation are used, but independent studies sometimes operate with other indicators (Aloisi & Tymms, 2017; Sulis et al., 2020). Different indicators of inequality (for example, inequality of outcome and inequality of opportunity) do not necessarily correlate and might actually have a negative correlation across countries and over time (Gromada et al., 2021). The plurality of indicators, contextual factors and interpretations make it difficult to distinguish and evaluate the impact of PISA on equity in quantitative terms, especially since PISA scores are used both as goals for educational reforms and as a measure of their success. Therefore, we can argue that PISA plays an important role in the fight for social equity, but it is impossible to calculate the weight of this impact in quantitative measures, because it is impossible to isolate it from other factors (for example, economic growth).
Though there is a tendency to interpret PISA scores as indicators of effectiveness (or ineffectiveness) of educational policies in different countries, it is important to understand that the connection between them is not linear nor causal. Describing the effect of a complex educational reform in terms of PISA scores is a very limited view. It is also necessary to recall that PISA itself is not a neutral tool of observation because of its political connotations. Quantitative research in this situation should be combined with qualitative data, which helps to capture contextual characteristics and interpret the indicators considering the specificities of each case.
Conclusion
In this paper we explored how PISA works with the problems of social equity and how participation in PISA can help countries become more equitable. We reviewed how equity is understood with the PISA framework and discussed the potential dangers of misuse of PISA data. As every powerful instrument, PISA offers a lot of insight into the problem of equity, and just by putting it on the agenda, PISA may already have a relevant impact on promoting equity around the globe. However, it is difficult to calculate the precise weight of this impact in quantitative terms because of the plurality of definitions, interpretations, and contextual factors that need to be considered. There are different aspects of equity, and while some of them may improve at any given time, others may not.
This study shows that there is no consensus around the role of PISA in the fight for equity. While some studies recognize the importance of PISA, the others criticize it calling it a “juggernaut” (Niemann et al., 2017; Zhao, 2020). In this paper we have argued that the effect of PISA is hardly (if not impossible) to isolate from the internal processes in each country and from the global trends in education. Thus, changes can never be solely attributed to PISA effects. If we try to test this impact, we will end up in a “chicken or egg” situation: the change appeared because of PISA, or the results of PISA were accepted and implemented because there was a need for change? To even start comprehending the situation we will need a complex study combining quantitative and qualitative methods, which will capture not only the quantitative data trends, but also the discourses around them.
Such a study will need to combine the quantitative data trends with the contextual factors and help to understand the processes shaping equity levels in any given country for the last 20 years. Mapping the perspectives and discourses around PISA will provide an additional layer of understanding regarding how PISA is used within the educational equity debates beyond the official OECD reports and recommendations. Hopefully, such a project will aggregate the existing knowledge on the subject and help to move equity forward.
Acknowledgments
This work was funded by the FCT - ‘Fundação para a Ciência e a Tecnologia’ (Portuguese Foundation for Science and Technology) through the grant PTDC/CED-EDG/2124/2020
References
Afonso, N., & Costa, E. (2009). The influence of the Programme for International Student Assessment (PISA) on policy decision in Portugal: the education policies of the 17th Portuguese Constitutional Government. Sísifo. Educational Sciences Journal, 10(set/dez), 53-64. https://www.researchgate.net/publication/242571464_The_infl_uence_of_the_Programme_for_International_Student_Assessment_PISA_on_policy_decision_in_Portugal_the_education_policies_of_the_17th_Portuguese_Constitutional_Government
Ainscow, M. (2020). Promoting inclusion and equity in education: lessons from international experiences. Nordic Journal of Studies in Educational Policy, 6(1), 7-16.
Aloisi, C., & Tymms, P. (2017). PISA trends, social changes, and education reforms. Educational Research and Evaluation, 23(5-6), 180-220.
Białecki, I., Jakubowski, M., & Wiśniewski, J. (2017). Education policy in Poland: The impact of PISA (and other international studies). European Journal of Education, 52(2), 167-174.
Breakspear, S. (2012). The policy impact of PISA: An Exploration of the Normative Effects of International Benchmarking in School System Performance. OECD Journals, 71, 1-32.
Breakspear, S. (2014). How does PISA shape education policy making? Why how we measure learning determines what counts in education. Centre for Strategic Education: Seminar Series, 240. https://allchildrenlearning.org/wp-content/uploads/2019/11/Breakspear-PISA-Paper.pdf
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 6-90.
Carvalho, L. M., Costa, E., & Gonçalves, C. (2017). Fifteen years looking at the mirror: On the presence of PISA in education policy processes (Portugal, 2000-2016). European Journal of Education, 52(2), 154-166.
Engel, L. C. (2015). Steering the national: Exploring the education policy uses of PISA in Spain. European Education, 47(2).
Fischman, G. E., Topper, A. M., Silova, I., Goebel, J., & Holloway, J. L. (2019). Examining the influence of international large-scale assessments on national education policies. Journal of Education Policy, 34(4), 470-499.
Gromada, A., Rees, G., & Chzhen, Y. (2021). Comparing inequality in adolescents’ reading achievement across 37 countries and over time: outcomes versus opportunities. Compare, 51(4), 495-511.
Hopfenbeck, T. N., Lenkeit, J., el Masri, Y., Cantrell, K., Ryan, J., & Baird, J. A. (2018). Lessons Learned from PISA: A Systematic Review of Peer-Reviewed Articles on the Programme for International Student Assessment. Scandinavian Journal of Educational Research, 62(3), 333-353.
Mangez, E., & Hilgers, M. (2012). The field of knowledge and the policy field in education: PISA and the production of knowledge for policy. European Educational Research Journal, 11(2), 189-205.
Michel, A. (2017). The contribution of PISA to the convergence of education policies in Europe: MICHEL. European Journal of Education, 52(2), 206-216.
Morgan, C. (2016). Tracing the sub-national effect of the OECD PISA: Integration into Canada’s decentralized education system. Global Social Policy, 16(1), 47-67.
Mzhavanadze, N. (2014). PISA, Power, and Policy: The Emergence of Global Educational Governance edited by Heinz-Dieter Meyer and Aaron Benavot. Comparative Education Review, 58(1).
Niemann, D. (2016). Germany: The intersection of international achievement testing and educational policy development. In L. Volante (Ed.), The Intersection of International Achievement Testing and Educational Policy: Global Perspectives on Large-Scale Reform (p. 18). Routeledge.
Niemann, D., Martens, K., & Teltemann, J. (2017). PISA and its consequences: Shaping education policies through international comparisons. European Journal of Education, 52(2), 175-183.
OECD (2017). Where did equity in education improve over the past decade? In PISA in Focus: Vol. 2017/68, 1-6.
OECD (2018). Equity in Education (PISA). OECD Publishing.
OECD (2019a). PISA 2018 Results: What students know and can do. Volume I. OECD Publishing.
OECD (2019b). PISA 2018 Results: Where all students can succeed. Volume II. OECD Publishing.
OECD (2019c). PISA 2018 Assessment and Analytical Framework (PISA). OECD Publishing.
Roemer, J. E., & Trannoy, A. (2016). Equality of opportunity: Theory and measurement. Journal of Economic Literature, 54(4), 1288-1332.
Rolfe, V. (2021). Tailoring a measurement model of socioeconomic status: Applying the alignment optimization method to 15 years of PISA. International Journal of Educational Research, 106.
Schleicher, A. (2009). Securing quality and equity in education: Lessons from PISA. Prospects, 39(3), 251-263.
Seitzer, H. (2020). Finding the Poster Child: Reference Patterns in OECD Country Reports. In K. Martens, & M. Windzio (Eds.), Global Pathways to Education (pp. 163-189). Palgrave Macmillan, Cham.
Seitzer, H., Niemann, D., & Martens, K. (2021). Placing PISA in perspective: the OECD’s multi-centric view on education. Globalisation, Societies and Education, 19(2), 198-212.
Sharma, R. (2020). Interrogating Equity in Education for Sustainable Development. Cambridge Educational Research E-Journal, 7, 35-52.
Srinivasan, J. (2021). Inequalities in Access to Educational Opportunities: An Investigation of the PISA 2009 Dataset Using a Multilevel-IRT Framework [Doctoral dissertation]. University of California.
Sulis, I., Giambona, F., & Porcu, M. (2020). Adjusted indicators of quality and equity for monitoring the education systems over time. Insights on EU15 countries from PISA surveys. Socio-Economic Planning Sciences, 69.
Takayama, K. (2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education, 44(4), 387-407.
Tiana Ferrer, A. (2017). PISA in Spain: Expectations, impact and debate. European Journal of Education, 52(2), 184-191.
UNESCO (2017). A guide for ensuring inclusion and equity in education. https://unesdoc.unesco.org/ark:/48223/pf0000248254
Zhao, Y. (2020). Two decades of havoc: A synthesis of criticism against PISA. Journal of Educational Change, 21(2), 245-266. https://doi.org/10.1007/s10833-019-09367-x
Copyright information
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
About this article
Publication Date
22 April 2022
Article Doi
eBook ISBN
978-1-80296-956-6
Publisher
European Publisher
Volume
3
Print ISBN (optional)
-
Edition Number
1st Edition
Pages
1-310
Subjects
Education quality, education assessment, education and city, urban education, lifelong learning, Big Data in education
Cite this article as:
Enchikova, E., Toledo, C., Neves, T., & Nata, G. (2022). Does PISA Help Fighting for Social Equity?. In S. Vachkova, & S. S. Chiang (Eds.), Education and City: Quality Education for Modern Cities, vol 3. European Proceedings of Educational Sciences (pp. 80-89). European Publisher. https://doi.org/10.15405/epes.22043.8