Increases to the quantity of education – as measured for example by mean years of schooling – has, for a long time, been the central focus of policy makers and academic debate. While increasing the access to education is important, the actual goal of providing schooling is to teach skills and transfer knowledge to students in the classroom. This entry focusses on the outcomes of schooling – the quality of education.
While we have good empirical data on the access to education we know much less about the quality of education. Unfortunately, the data on the skills and knowledge of students is sparse and has limited spatial and temporal coverage. This is in part due to the difficulty and cost of creating and implementing standardized assessments that can be compared across borders and time.
Efforts to measure these outcomes are geographically more restricted (often only OECD countries are included) and even less is known about how the performance of students with respect to these outcomes has changed over time. A third limitation is that measures are sometimes not comparable between countries.
Most often these assessment are measuring learning outcomes of one or several of the following three dimensions:
- Reading and language proficiency
- Mathematics and numeracy proficiency
- Scientific knowledge and understanding
The assessments differ in respect to how children are selected. While some assessments select children by their age, there are other assessments which select children by the school grade the child attends. The aim of these studies is to test a representative random sample of the intended population.
The most widely available metric on the outcomes of education is literacy. Data and research on literacy is discussed in detail in our entry on the topic.
The Program for International Student Assessment (PISA) assessment, which is coordinated by the OECD, is the most well known international assessment of learning outcomes. The first PISA study was carried out in 1997 and since then it was repeated every three years.
The study is not select students by grade, but by age and only tests 15-year-olds, whatever their school level. In a two-hour assessment the competencies in reading, mathematics and science of these students are assessed. It is a very substantial undertaking and until 2017 "half a million students representing 28 million 15-year-olds in 72 countries and economies have participated in PISA" according to the OECD. The coverage of PISA can be seen in this map on the results in the reading dimension, it obviously includes the richer OECD countries and unfortunately only few poorer countries. Poorer countries are also not tested regularly and were only participating in one round only and additionally it can be the case that for poorer countries not students from the entire country were selected, but instead only from particular regions.
PISA, as well as TIMSS, are standardizing their results so that the average score of students from OECD countries is 500 and the student standard deviation is 100. Assuming a normal distribution of measurement outcomes this means 68% of OECD students are reaching scores in the range between 400 and 600.
While TIMSS focuses on content that is covered in the school curriculum PISA aims to "assesses the application of skills to real-life problems" and "emphasizes the importance of the context in which students should be able to use their skills (schools, home and society)".
PISA tests the students in three different dimensions, which they define as follows:1
- "Science literacy is defined as the ability to engage with science-related issues, and with the ideas of science, as a reflective citizen. A scientifically literate person is willing to engage in reasoned discourse about science and technology, which requires the competencies to explain phenomena scientifically, evaluate and design scientific enquiry, and interpret data and evidence scientifically.
- Reading literacy is defined as students’ ability to understand, use, reflect on and engage with written texts in order to achieve one’s goals, develop one’s knowledge and potential, and participate in society.
- Mathematical literacy is defined as students’ capacity to formulate, employ and interpret mathematics in a variety of contexts. It includes reasoning mathematically and using mathematical concepts, procedures, facts and tools to describe, explain and predict phenomena. It assists individuals in recognising the role that mathematics plays in the world and to make the well-founded judgements and decisions needed by constructive, engaged and reflective citizens."
Cross country comparability of the PISA results
When the PISA study includes non-OECD countries it is often the case that students are not selected from the entire country, but instead only from selected regions within that country. Results reported for China – and labelled as such by the OECD – in fact are only based on four Chinese provinces: Beijing, Shanghai, Jiangsu and Guangdong. These regions are not representative of China as a whole and there is every reason to expect that students from these provinces have access to better education than the average Chinese 15-year-olds. The four regions are among the very richest regions of China and incomes are up to twice that of the national Chines average.
Similarly when PISA reported on "India" in 2009 the assessments were made in two states of India only (Himachal Pradesh and Tamil Nadu).
The three maps below show the results for the three subjects in which students are assessed.
I.2 PIRLS and TIMSS – The assessments by the International Association for the Evaluation of Educational Achievement
As the name says, TIMSS is an assessment of the mathematics and science knowledge of students. As the closely linked PIRLS it is carried out by the International Association for the Evaluation of Educational Achievement (IEA).
While PISA selects the assessed students by age and focusses on 15-year olds, TIMMS selects students by the grade they attend and tests students in Grades 4, 8, and in their final year.
TIMSS started slightly earlier than PISA. The first TIMSS was conducted in 1995 and covered 45 national educational systems. Later rounds included more countries – the 2011 round included 77 countries/area. The coverage of TIMSS is similar to PISA and includes mostly richer countries which are part of the OECD and only few poorer countries. The coverage of TIMSS can be seen in this map on the results in the science dimension.
Precursors to TIMSS were already conducted from 1964 onwards.
The PIRLS is the other major study carried out by the International Association for the Evaluation of Educational Achievement (IEA). Here the objective is to assess primary school students in reading proficiency. PIRLS only started in 2001.
PASEC, which in the original French refers to Programme d’ Analyse des Systèmes Educatifs, has assessed educational outcomes in 13 countries in Francophone West Africa. Here is a map of outcomes in the dimension of the French language.
EGRA and EGMA are simple, low-cost assessments of literacy and numeracy.
APRESt is a large-scale randomized evaluation in the Indian state of Andra Pradesh that has been carried out since 2004.
ASER is a report that is based on a survey of more than 500,000 children in rural areas across India. It is not a school-based, but instead an in-home assessment of the reading and mathematical skills of children between 3 and 16. The first ASER was carried out in 2005.
UWEZO is an adaptation of ASER carried out in three east African countries (Tanzania, Uganda, and Kenya). It has been done since 2009 and is carried out annually.
An assessment carried out by education ministries from fifteen southern and eastern African countries. The study assesses students in grade 6 in the domains of reading and mathematics. The first study was carried out in 1995. The studies' website is: www.sacmeq.org
With a multitude on different assessments of learning outcomes which focus on different geographical areas and different skills it is an obvious idea to try to combine these measures into a common system to get a more comprehensive perspective on educational outcomes across the world. Several researchers have attempted to do this.
The education economists Eric Hanushek and Ludger Wößmann are combining the results from educational achievement tests to investigate the question whether the quality of education has a causal influence on the growth of the economy.2
They extend earlier efforts by Hanushek and Kimko (2000) published in the American Economic Review.3
The researchers Altinok, Diebolt & Demeulemeester published their dataset and their accompanying paper under the title A new international database on education quality: 1965–2010.4
The authors write that they "merge all existing regional and international student achievement tests by using a specific methodology". The resulting dataset includes indicators of the quality of student achievement for 103 countries/areas in primary education and 111 countries/areas in secondary education between 1965 and 2010.
For African countries Sandefur (2016) constructed internationally comparable mathematics scores.5
To describe the distribution of proficiency within countries PISA segments the achieved outcomes into levels of proficiency. To achieve higher levels, students need to be able to solve tasks of increasing complexity. The results are then reported as the share of the student population that reached each level.
The cut offs for the levels are the following:
- Below level 1B (below or equal to 262)
- level 1B (higher than 262)
- level 1A (higher than 335)
- level 2 (higher than 407)
- level 3 (higher than 480)
- level 4 (higher than 553)
- level 5 (higher than 626)
- level 6 (higher than 698)
The visualization shows the distribution of students proficiency levels in 2009, 2012, and 2015. It is possible to see the data for other countries by choosing the 'change country' option below the visualization.
Where the top students are better than in other countries, the worst students are less bad than in other countries too.
The test scores of the worst and best performing students in each country are correlated as this visualization shows. Countries in which the top students perform better than top students in other countries tend to be the same countries in which the worst students also perform better than the worst students in other countries.
In every country and in every year girls achieved higher PISA test scores in reading. The difference of sometimes more than 50 points is substantial as the test scores are standardized to have a standard deviation of 100 points.
In mathematics the difference between girls and boys is much more mixed than in the reading dimension. While in most countries boys tend to achieve better test scores there are also many countries and years in which girls performed better than boys.
Student Achievement in the United States over Time6
The OECD PISA rankings demonstrate the strength of education in East Asian countries. These results are broadly supported by similar research conducted by the International Association for the Evaluation of Educational Achievement (IEA) and Boston College. The IEA produces assessments of mathematics and science performance (TIMSS) as well as of reading and literacy (PIRLS). Countries that repeatedly rank among the top 5 in mathematics and the sciences are Singapore, South Korea, Hong Kong, Taiwan and Japan.
The OECD also surveys the skills of adults. This is done in the Programme for the International Assessment of Adult Competencies (PIAAC). Here is the world map showing the level of numeracy of adults. More information on PIAAC can be found here.
The Programme for the Analysis of Education Systems (PASEC) has assessed educational outcomes in 13 countries in Francophone West Africa. Here is a map of outcomes in French language.
SACMEQ reading and math assessments have been carried out in countries in Anglophone East Africa in 1995, 2000, and 2007. Here is a map of the outcome of the assessments of 6th graders on the mathematics scale.
EGRA and EGMA are the Early Grade Reading Assessment and Math Assessment which are simple, low-cost assessments of literacy and numeracy.
Another interesting aspect of school performance is the effect of family environments and culture on students exam performance. A study by John Jerrim finds that children of East Asian immigrants to Australia outperform their native counterparts in the PISA tests.7 In mathematics, he finds them to be ahead by 100 points representing two and a half years of education. This evidence suggests that the differences highlighted by PISA and the IEA may be driven by cultural or family factors rather than the schooling systems.
Indicators of skills and knowledge, such as the OECD PISA scores, are highly correlated with indicators of economic development. The following scatter plot shows the correlation between the PISA reading scores and the United Nations' Human Development Index (HDI) for a select group of countries.
There exists a substantial literature on the effects of competition on school quality and performance. Whether choice improves school quality remains an open question in economics. In general we might expect that more schools might be better for outcomes through competitive forces, however this relies on both schools and parents responding to the increased competition/choice. On the demand side, parents need some way of observing school quality accurately as well as the ability to change schools. Meanwhile, schools need some incentive to respond to any increase in competition. This is especially important since most public school systems lack any profit motive.
Much of the research into the effects of competition rely on indirect measures of demand for high quality schools such as local rents and house prices. Disaggregating the willingness to pay for better schools from neighbourhood effects and sociodemographic factors is highly technical and relies on models of sorting. For more information on these models see Rothstein (American Economic Review, 2006), Bayer and McMillan (NBER, 2005), and Bayer et al. (NBER, 2007).8
An alternative approach has been to use variables correlated with school competition but independent of the other the demand and supply factors to disaggregate the different effects of choice (instrumental variables approach). For more information on this approach please see Hoxby (American Economic Review, 2000) and Rothstein (American Economic Review, 2007).9
Recent research however suggests that the link between resources and school quality is not simple. The OECD looked into whether money can buy stronger PISA test performance. They concluded that the most important factor in PISA test performance is how resources are used: countries that prioritized the quality of teachers over class sizes performed much better. This view represents a growing consensus in the education literature that inputs such as class size and expenditure per pupil have little to no effect on the returns to schooling in the developed world.10
An argument made by Eric Hanushek and Ludger Woessmann is that the lack of any straightforward relationship between resources and school outcomes indicates a minimum resource requirement.11 Once the resource threshold has been reached, additional expenditure has little or no returns to school quality -- instead, teacher quality and other constraints matter far more.
The following scatter shows average spending per student from the age of 6 to 15 against reading test scores in 2009.
Average reading performance in PISA and average spending per student from the age of 6 to 15 - OECD12
The education economists Eric Hanushek and Ludger Wößmann combined the results from educational achievement tests to investigate the question whether the quality of education has a causal influence on the growth of the economy.13
The visualization below shows the correlation between the quality of education, as measured by Hanushek and Woessmann, and the level of prosperity of the country in 2016.
A detailed discussion of the assessment of learning outcomes is provided by Daniel Koretz in his book Measuring Up: What Educational Testing Really Tells Us.14
UNESCO Institute of Statistics
- Data: Comprehensive data on enrollments, out-of-school children, repetition, completion, gender, teachers, education expenditures, learning outcomes, educational attainment, education equality, literacy, population, labor, and EMIS.
- Geographical coverage: Global by country
- Time span: 1999-2015
- Available at: http://data.uis.unesco.org/
World Bank EdStats
- Data: indicators on educational attainment, enrolment, attendance, teachers, financing and more
- Geographical coverage: Global, over 200 countries
- Time span: 1970 to most recent data year; Projections to 2050
- Available at: It is online here
OECD Programme for International Student Assessment (PISA)
- Data: Standardised assessment scores for mathematics, reading and science
- Geographical coverage: OECD countries and other partners
- Time span: 2000-2012 (every three years)
- Available at: http://www.oecd.org/pisa/