Machine Learning in Public Policy: Examples from Research
Companies use machine learning to improve their business decisions. Algorithms select ads, predict consumers’ interest or optimize the use of storage. However, few stories of machine learning applications for public policy are out there, even though public employees often make comparable decisions. Similar to the business examples, decisions by public employees often try to optimize the use of limited resources. Algorithms may assist tax authorities in improving the allocation of available working hours, or help bankers make lending decisions. Similarly, algorithms can be employed to guide decisions taken by social workers or judges.
This blogpost lists three research papers that analyze and discuss the use of machine learning for very specific problems in public policy. While the potential seems huge, we do not want to neglect some of the many potential pitfalls for machine learning in public policy. Business applications often maximize profits. For policy decisions, however, the maximizable outcome may be harder to define or multidimensional. In many cases, not all relevant outcome dimensions are directly observable and measurable, which makes it more difficult to evaluate the impact of an algorithm. Tech companies would usually obtain training datasets through experimenting, while datasets for public policy often contain only one outcome for a specific group of people. If tax authorities never scrutinize restaurants, how can we form a predictive model for this industry? Predictions for public policy problems often face this so-called selected labels problem and it needs innovative approaches and the willingness to perform randomized experiments to get around it. This is just a brief list. Susan Athey’s paper provides more food for thought on the potential - and potential pitfalls - of using prediction in public policy.
Research on Machine Learning Applications in Public Policy
Improving refugee integration through data-driven algorithmic assignment
Developed democracies are settling an increased number of refugees, many of whom face challenges integrating into host societies. We developed a flexible data-driven algorithm that assigns refugees across resettlement locations to improve integration outcomes. The algorithm uses a combination of supervised machine learning and optimal matching to discover and leverage synergies between refugee characteristics and resettlement sites. The algorithm was tested on historical registry data from two countries with different assignment regimes and refugee populations, the United States and Switzerland. Our approach led to gains of roughly 40 to 70%, on average, in refugees’ employment outcomes relative to current assignment practices. This approach can provide governments with a practical and cost-efficient policy tool that can be immediately implemented within existing institutional structures.
Bansak, K., Ferwerda, J., Hainmueller, J., Dillon, A., Hangartner, D., Lawrence, D., & Weinstein, J.; Science, 2018
Switzerland is currently implementing an algorithm based allocation of refugees. We are excited to see first results!
Human Decisions and Machine Predictions
Can machine learning improve human decision making? Bail decisions provide a good test case. Millions of times each year, judges make jail-or-release decisions that hinge on a prediction of what a defendant would do if released. The concreteness of the prediction task combined with the volume of data available makes this a promising machine-learning application. Yet comparing the algorithm to judges proves complicated. First, the available data are generated by prior judge decisions. We only observe crime outcomes for released defendants, not for those judges detained. This makes it hard to evaluate counterfactual decision rules based on algorithmic predictions. Second, judges may have a broader set of preferences than the variable the algorithm predicts; for instance, judges may care specifically about violent crimes or about racial inequities. We deal with these problems using different econometric strategies, such as quasi-random assignment of cases to judges. Even accounting for these concerns, our results suggest potentially large welfare gains: one policy simulation shows crime reductions up to 24.7% with no change in jailing rates, or jailing rate reductions up to 41.9% with no increase in crime rates. Moreover, all categories of crime, including violent crimes, show reductions; these gains can be achieved while simultaneously reducing racial disparities. These results suggest that while machine learning can be valuable, realizing this value requires integrating these tools into an economic framework: being clear about the link between predictions and decisions; specifying the scope of payoff functions; and constructing unbiased decision counterfactuals.
Jon Kleinberg Himabindu Lakkaraju Jure Leskovec Jens Ludwig Sendhil Mullainathan; Quarterly Journal of Economics, 2018
Using Text Analysis to Target Government Inspections: Evidence from Restaurant Hygiene Inspections and Online Reviews
Restaurant hygiene inspections are often cited as a success story of public disclosure. Hygiene grades influence customer decisions and serve as an accountability system for restaurants. However, cities (which are responsible for inspections) have limited resources to dispatch inspectors, which in turn limits the number of inspections that can be performed. We argue that NLP can be used to improve the effectiveness of inspections by allowing cities to target restaurants that are most likely to have a hygiene violation. In this work, we report the first empirical study demonstrating the utility of review analysis for predicting restaurant inspection results.
Kang, J. S., Kuznetsova, P., Choi, Y., Luca, M., 2013, Technical Report
Here is related paper on the same topic suggesting ways for governments on how to obtain the required expertise:
Crowdsourcing City Government: Using Tournaments to Improve Inspection Accuracy
Further readings:
Two papers with an excellent overview on the topic
Machine Learning: An Applied Econometric Approach
The Economist on the same topic:
Of prediction and policy, The Economist 2016