Haris Gazdar and Fuzail Khan
The Benazir Income Support Programme (BISP) has been the flagship social protection programme of Pakistan’s federal government. The programme includes the National Socio-Economic Registry (NSER) which is a large database used for identifying beneficiaries. This article uses publicly available data from official sources to find that while BISP does well in terms of ensuring that programme beneficiaries are mostly from among the poor, it does less well in making sure that the programme covers most of the poorest.
Two types of indicators are often used to measure the quality of poverty targeting of social protection programmes – inclusion and exclusion errors (Hodgson 2007, cited in Devereux et al 2015). Inclusion error refers to the proportion of the beneficiaries who are non-poor. Exclusion error is defined as the proportion of the poor who are not covered by the programme. We estimate inclusion and exclusion errors in BISP poverty targeting using data from the Household Income and Expenditure Survey (HIES) of the Pakistan Bureau of Statistics (PBS).
BISP’s method for identifying the poor uses a poverty score which is a proxy for income. HIES data is used to identify household demographics, assets and housing conditions – variables on which reliable information can be collected cost effectively – that are good predictors of household income (or consumption expenditure). Weights assigned to these variables in constructing the poverty score are also estimated statistically from survey data. The NSER then collects data on households on these demographics, assets and housing conditions, and calculates a poverty score for each household by applying the appropriate weights. The poverty score, therefore, is a proxy for income or consumption expenditure.
HIES is a statistically representative sample survey that collects data on a range of household characteristics including household composition, consumption expenditure, assets and living conditions, as well as whether or not the household was a BISP beneficiary. The most recent HIES round was conducted in 2018-2019 and this remains the only publicly available statistically representative sample survey that includes information on household economic well-being (consumption and assets) as well as access to government social protection programmes. These are the data we used to estimate inclusion and exclusion errors in BISP targeting.
We identified the poor in the HIES sample using two approaches. First, households were ranked using their reported consumption expenditure per adult equivalent. In the absence of reliable data on individual or household incomes, this variable is often used as a proxy for income and the BISP poverty score is supposed to be a proxy for consumption expenditure. Second, a household well-being score was estimated using principal component analysis (PCA) which consolidated binary asset ownership and housing conditions (e.g. ownership of heater, vehicle, radio, television, agriculture land, fan, and type of cooking fuel used, house structure durability) into a single dimension. Intuitively, this method transforms the original variables, in this case household assets, into uncorrelated components called the principal components. While our household well-being score uses a different methodology from the BISP poverty score, many of the household characteristics (such as asset ownership and housing conditions) used in the construction of these two scores are common.
We then used our two methods of ranking, consumption expenditure and the household well-being score respectively, to divide the survey population into five equally-sized groups each representing exactly a fifth of the population. We labelled the lowest quintile as the poorest and the highest quintile as the richest, with the intermediate groups being labelled as “lower middle”, “middle”, and “upper middle” respectively. In 2018-2019 the headcount ratio of poverty according to the official poverty line was 21 per cent – so the “poorest” group in our analysis corresponds approximately with those who would have been below the national poverty line in that year.
Table 1 gives the distribution of BISP beneficiaries by economic class by the two methods respectively. If households are ranked according to consumption expenditure, 36 per cent of BISP beneficiaries were in the lowest quintile and 66 per cent, or two-thirds of them, were in the bottom two quintiles. In other words, inclusion error was 64 per cent if the targeting was aimed at identifying only those who are below the official poverty line, and 36 per cent if the “poorest” as well as the “lower middle”. Targeting appears more accurate if we use the household well-being score which is closer to the BISP’s own scoring method. The inclusion error is 55 and 24 per cent respectively – 45 per cent of the beneficiaries are in the “poorest” quintile or correspond with those below the poverty line, while another 30 per cent are in the ”lower middle” quintile.
Table 2 provides an analysis of the proportion of the population in each economic class that is covered by BISP. It shows that overall, around a tenth of the population was covered by the programme in 2018-2019. Coverage is higher in the poorest class – which corresponds roughly with the population below the official poverty line in that year than in other economic classes. Among the richest (top quintile) the probability of being a BISP beneficiary was between 0.5 to 1.7 per cent. This underlines the finding of Table 1 that BISP targeting is good at excluding the rich. But coverage is low even in the poorest economic class. Only around 18 to 22.5 per cent of those below the official poverty line were BISP beneficiaries in 2018-2019. If the goal of targeting policy was to reach the entire population below the poverty line, the exclusion error was between 77 to 82 per cent depending on the indicator used for classifying the population into economic classes. If the intention was to reach the population in the bottom two quintiles – i.e. the “poorest” as well as the “lower middle” – the exclusion error was much higher, ranging from 82 to 84 per cent.
Summing up, BISP targeting does well in keeping inclusion errors low, particularly if we assume the goal of the targeting strategy was to reach the bottom two quintiles – the “poorest” and the “lower middle”. In this case inclusion error was between 24 and 36 per cent, depending on whether we compare BISP targeting with consumption expenditure or household well-being score based on asset ownership and housing conditions. If the goal was to cover only the “poorest”, inclusion errors were between 55 and 64 per cent. Exclusion errors were high – ranging from 77 to 82 per cent – if targeting aimed to reach all of the ”poorest”, and were even higher – ranging from 82 to 84 per cent – if the programme had aimed to reach the “poorest” as well as the “lower middle” economic classes. There is clearly a trade-off between minimizing inclusion and exclusion errors.
The situation would have changed since 2018-2019 with respect to coverage. The NSER was renewed from 2016 onwards with updated information collected through a variety of methods, and beneficiary selection was updated in 2021 using the new data. In the early part of NSER updating data were collected using the same method which was used for the initial selection of beneficiaries in 2011 – namely, door-to-door visits by enumerators. This method was replaced later with the setting up of registration desks where applicants came to provide their data. A recent survey by the World Bank found that data quality was somewhat better in the door-to-door method than the desk-based method. Transition to the new NSER data, therefore, may have improved targeting quality because it gave a truer picture of the current poverty status of the household than the data used in 2018-2019. It may have deteriorated somewhat because a large proportion of the population were enumerated using the slightly less reliable desk-based method.
Another important change is that the number of beneficiaries increased by 55 per cent from around 5.8 million in 2018 to over 9 million in 2023. If targeting quality had remained unchanged, this increase in the total number of beneficiaries would take the proportion of households covered from 9.9 per cent (Table 2) to over 15 per cent. It would also have increased the coverage of the lowest quintile from 22.5 per cent to around 35 per cent.
Using publicly available data from the HIES 2018-2019 we have shown that Pakistan’s flagship social protection programme performs well with respect to identifying the poor and excluding the rich. Improvements in the programme since 2018-2019 in terms of data and scale will have further enhanced its quality and coverage. These are important achievements of social policy in Pakistan which have earned well-deserved accolades globally. The BISP does not do so well, however, in ensuring that all, most or even a majority of the poorest are covered. In 2018-2019 its coverage of the poorest quintile, who would also happen to be below the official poverty line, was under a quarter. Changes since then would have increased the coverage of the poorest to just over a third. This means that around two-thirds of the poorest in Pakistan remain without social protection coverage. Significantly increasing the coverage of the poorest remains the challenge ahead.
References:
1) Devereux, S., Masset, E., Sabates-Wheeler, R., Samson, M., Rivas, A. M., & te Lintelo, D. (2017). The targeting effectiveness of social transfers. Journal of Development Effectiveness, 9(2), 162-211.
https://www.tandfonline.com/doi/abs/10.1080/19439342.2017.1305981
2) Filmer, D., Pritchett, L.H. Estimating Wealth Effects Without Expenditure Data—Or Tears: An Application To Educational Enrollments In States Of India*. Demography 38, 115–132 (2001). https://link.springer.com/article/10.1353/dem.2001.0003
3) ASPIRE database, World Bank. . https://www.worldbank.org/en/data/datatopics/aspire/indicator-glance, https://thedocs.worldbank.org/en/doc/7bfac76609ef741b6f6fdb166c9f7bfb-0380082021/original/PER-Table2-Coverage.xlsx
4) Zaineb, Fnu; Wieser, Christina; Qazi, Maria; Guzman Fonseca, David Camilo; Pave Sohnesen, Thomas; Ibrahim Khan. Mind the Gap : Assessing Pakistan’s National Socio-economic Registry (NSER) (English). Washington, D.C. : World Bank Group. https://documents.worldbank.org/en/publication/documents-reports/documentdetail/099110524081521753/p1783481eab88003c1b61917de147c09798
