Missing data on energy: our list of datasets that are needed but are not available

Below we list the key metrics on this topic that we think are missing from our coverage on Our World in Data. These are metrics we think are critical to understand the problem and build solutions.

The reason we list this missing data is because we are often asked by funders or researchers for examples of work that would provide a high-value return. This list should provide a useful reference for impactful projects to support. We believe these data gaps are important to fill; doing so would provide a lot of benefit.

There are several possible reasons why a particular metric or dataset is not available:

• Many are fundamental knowledge gaps where the relevant data or research simply does not exist at all.
• Some are gaps where the data does exist but it is not in the public domain. It is privately licensed or locked behind paywalls.
• Some are gaps where the existing data is very low-quality, outdated, or has poor coverage.
• Some are gaps where the data is published by some countries but it has not been aggregated into regional or global datasets.

If you are aware of existing data or research which fills these gaps, please do get in touch at hannah@ourworldindata.org.

It’s important that these datasets are regularly updated; and are open-access for everyone to use.

Electricity access at higher thresholds of electricity use

What is available: Data on access to electricity – presented as the share of the population with access to electricity – is available globally. The World Bank publishes this data, which we make available here.

What is missing: The threshold in the available statistics for what it means to have ‘access to electricity’ is very low. This issue is explained here. This means that a large number of people is counted as ‘having access to electricity’ which in fact only have very little access – we explain this problem in the middle-section of this essay on energy poverty.

It would be valuable to understand access and electricity use across higher thresholds as well. in one dataset that aggregates this information for countries around the world. For example, the share of people who consume x units of electricity per day; or electricity for at least x hours per day. Where x units of electricity is representative of a sufficient amount to complete standard household tasks. This would give us information on the distribution of electricity usage rather than the average. It would also allow us to understand inequalities in electricity consumption across a population.

As far as we are aware, this data does not currently exist.

Affordability of energy

For many households, energy can come at a high cost. It can account for a large share of their income – making it compete with other essentials such as food and shelter. This means many are living in energy poverty.

What is available: Data on global fuel prices (e.g. coal or oil prices) exists, from sources such as the Energy Institute Statistical Review of World Energy or energy market data. Some countries (for example, the UK) publish national statistics on fuel poverty.

What is missing: There is, as far as I am aware, no harmonized global data on the cost of energy in households (or for the end user) across the world. Nor are there comparisons to how this relates to the income of people in these countries. This means we lack the data we need to understand the depths of energy poverty globally.

A dataset which brings today’s energy costs across the world would provide a lot of value in helping to answer this question.

Reliability of electricity supply

Having access to electricity does not guarantee that is is reliable and available when it’s needed the most.

What is available: Data on access to electricity – presented as the share of the population with access to electricity – is available globally. The World Bank publishes this data, which we make available here. This does not tell us how reliable this supply is.

What is missing: No global, continuously maintained data exists on the reliability of electricity – when and how frequently blackouts occur – across the world. This would be an important measure to track progress on energy access.

Final energy consumption

What is available: We have good data on primary energy – the total amount of energy generated from all sources.

Primary energy includes energy that is finally consumed, plus energy that is wasted as heat in the combustion of fossil fuels. This measure is useful, but it does not represent the amount of energy we would need to produce in a low-carbon economy, and our rate of progress on decarbonization. Since a large amount of primary energy from fossil fuels is wasted as heat, we would need to produce much less energy from alternative sources to replace fossil fuel energy supplies.

What is missing: Final energy consumption, which removes these losses in the combustion process. It is primary energy minus these losses from heat.

Some individual countries publish this data (see the UK as an example). Some regions – for example, the European Environmental Agency – also publishes final energy estimates. But, unfortunately, there are few harmonized global datasets. The UN Energy Division makes some data publicly available.

The most comprehensive dataset – which includes this data – is published by the IEA. However, despite being almost entirely publicly funded, its licensing policies mean that this data is locked behind a paywall. Since this data is so crucial to understand the energy transition, we have made an open call for this data to be made public.

Final energy consumption is one of the most important energy metrics (perhaps the most important one) and it is unacceptable that no global, publicly usable dataset exists.

Final energy consumption by source

See above – ‘Final energy consumption’. This also applies to the breakdown of final energy consumption by source: how much comes from fossil fuels, renewables and nuclear energy.

Primary energy consumption (low-income countries)

What is available: Up-to-date data on primary energy consumption is available for most countries in publicly available datasets, such as the Energy Institute Statistical Review of World Energy. We present this data here. However, some low-income countries are absent from these datasets.

What is missing: Unfortunately many low income countries are absent from the Energy Institute Statistical Review of World Energy dataset. We try to supplement this with other public sources, such as the SHIFT Data Portal. However, this data is not up-to-date and only extends to 2016.

Complete global data on primary energy is available from organizations such as the IEA. However, despite being almost entirely publicly funded, its licensing policies mean that this data is locked behind a paywall. Since this data is so crucial to understand the energy transition, we have made an open call for this data to be made public.

Energy use by sector and sub-sector

It would be useful to know how our energy is used: how much is used for electricity in households; how much in cars; how much in global aviation.

What is available: Some countries publish data on the allocation of energy to end use sectors (for example, the UK and the US). Some regions also publish aggregated datasets – such as the European Environment Agency.  Often these estimates are only available at high-level sectors and do not provide breakdowns by sub-sector. For example, they stop at allocation to ‘transportation’ which further breakdown of ‘road, rail, aviation’.

What is missing: An open-access global dataset with sector-by-sector data for all countries. Ideally, this would be available at the sub-sector level.

This data is included in data published by the IEA. However, despite being almost entirely publicly funded, its licensing policies mean that this data is locked behind a paywall. Since this data is so crucial to understand the energy transition, we have made an open call for this data to be made public.

Electrical storage capacity and demand

As the world moves towards more low-carbon energy sources, and electrifies an increasing  number of sectors, energy storage will become increasingly important to balance supply and demand.

What is available: The DOE Global Energy Storage Database is a harmonized global database that attempts to quantify energy storage capacity across the world.

What is missing: This database is incomplete, with data for many countries missing. It would also benefit from the integration of estimates of electric storage demand today and in the future with decarbonization. This would allow us to track the storage gap that might exist.

Hannah Ritchie, Pablo Rosado and Max Roser (2021) - “Missing data on energy: our list of datasets that are needed but are not available” Published online at OurWorldinData.org. Retrieved from: 'https://ourworldindata.org/energy-missing-data' [Online Resource]

@article{owid-energy-missing-data,
    author = {Hannah Ritchie and Pablo Rosado and Max Roser},
    title = {Missing data on energy: our list of datasets that are needed but are not available},
    journal = {Our World in Data},
    year = {2021},
    note = {https://ourworldindata.org/energy-missing-data}
}

Reuse this work freely

All visualizations, data, and code produced by Our World in Data are completely open access under the Creative Commons BY license. You have the permission to use, distribute, and reproduce these in any medium, provided the source and authors are credited.

The data produced by third parties and made available by Our World in Data is subject to the license terms from the original third-party authors. We will always indicate the original source of the data in our documentation, so you should always check the license of any such third-party data before use and redistribution.

All of our charts can be embedded in any site.