About Michael Wiseman

Doing Process Analysis Better

Doing Process Analysis Better

[Download PDF]

Summary:

Evaluation specifications for social experiments typically include a requirement for a “process analysis” of the intervention in addition to impact measurement and cost-benefit assessment. There is little agreement in the literature concerning what belongs in a process analysis. This report, prepared for the U.S. Social Security Administration’s Office of Program Development and Research in 2012 and revised in 2013, discusses the concept of process analysis, the role of process analysis in three SSA-supported demonstrations, and the use of the term in evaluation guides by various agencies. The end product is a set of guidelines for process analysis. Cast as 10 “steps,” the guidelines are intended for use in two ways. The first is as a checklist for demonstration planners, a response to the question “What are the essential elements of process analysis?” The second is as checklist for demonstration evaluators: What should be there, and is it?

Activation and Reform in the United States: What Time Has Told

Download PDF

Abstract:

The American “Personal Responsibility and Work Opportunity Reform Act” of 1996 (PRWORA) promoted “work over welfare” and included significant work and other activation requirements for adult recipients of Temporary Assistance for Needy Families (TANF), the safety net for poor families with children. Wiseman’s 1999 review of work-related requirements in TANF emphasized that states were still adjusting policy to various requirements of the law and that only “time would tell” what the ultimate consequences might be and what role activation would play. Time has now told. States have developed creative ways to meet the strict federal activation requirements without paying for workfare. Less than 40 percent of the TANF adult work-eligible caseload is “activated”—meaning that benefit receipt is conditional on some self-support incentive or obligation—and only about 12 percent of those persons engage in workfare. At the same time, mostly as a result of states’ efforts, access to the TANF safety net has contracted, inequities in federal support have increased, real benefits have declined, and fiscal responsibility for general social assistance has shifted upward from states to the federal government without significant rationale. States’ strategy of shifting funding from income support to more broadly-targeted services reduced the capacity of the TANF system to respond to the Great Recession of 2007-2009. The Supplemental Nutritional Assistance Program (formerly the Food Stamp Program) now provides what is de facto the national minimum income. Despite TANF’s shortcomings, the Obama administration reasonably chose to focus domestic policy on health care, rather than welfare reform. A national health insurance scheme was introduced in 2010 and gradual implementation is under way despite political opposition.  Little change has occurred in TANF; keeping the program small eases states’ difficulties with achieving required work activity participation rates.  And the highly partisan political struggle over taxes and the federal deficit precludes significant change in TANF outreach or activation requirements.

Suggested citation:
Wiseman, Michael, Anderson, Theresa, and Kairys, Katharine. “Activation in the United States: What Time Has Told.” Manuscript, August 28, 2013.

Misreporting of Food Stamp Participation in a Household Survey: Results from a Single-State Pilot Study of the National Health and Nutrition Examination Survey NHANES

Download PDF

Abstract:
Data from household surveys play an essential role in monitoring and assessing the consequences of food assistance programs for the well-being of Americans. Compared to other sources—most notably administrative data—surveys like the National Health and Nutrition Examination Survey (NHANES) provide more information on individual and household characteristics such as food security, nutrition, and body weight, and these are characteristics likely to be affected by participation in food assistance programs. Thus, by using survey information on both program participation and household or individual characteristics (and statistical methods for handling non-response), surveys can provide powerful evidence on the impacts of food assistance programs.

The utility of survey data for evaluating food program outcomes is diminished by respondent misreporting of participation in these programs. At least in principle, misreporting issues can be addressed by linking administrative records to household survey data. The potential benefits of such linkage are widely recognized, especially in social policy research. In practice, however, efforts at linkage encounter many problems, but experience is growing, and with gains in experience, more of the potential benefits are being realized.

The Texas NHANES Pilot (TNP) is part of a cooperative effort by the National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention of the U.S. Department of Health and Human Services, the Economic Research Service (ERS) of the U.S. Department of Agriculture, and the Ray Marshall Center for the Study of Human Resources (RMC) at the University of Texas, Austin to link administrative records from the Food Stamp Program (FSP)1 in Texas to survey records from NHANES. The Ray Marshall Center determined if NHANES participants had records in the Texas food stamp program database under contracts. We conducted this study in one state as a pilot to learn more of what would be required to develop and implement match protocols appropriate to the NHANES should more general effort ever be contemplated. The immediate goals of this venture were to: (a) assess the feasibility of matching records from these disparate sources; (b) evaluate the accuracy of the match process; (c) use the matched data to estimate the prevalence and correlates of misreporting of FSP participation for this one-state subset of NHANES; and (d) assess the likely impacts of matching error on our estimates of misreporting. Ultimately, we sought to obtain information to improve analyses of food assistance programs and policies through more informed use of valuable information gathered during household surveys.

This paper presents results to date from the TNP. Part 1 briefly describes the match process used to link data from food stamp administrative files in Texas to NHANES participants for interviews conducted in Texas for survey cycles 2005 through 2008. Part 2 summarizes the results of that data linkage in terms of how often food stamp participation status as reported in the survey did or did not correspond to food stamp participation status as indicated in the administrative records. Part 3 provides a general discussion of our results. In our larger working paper (available from the authors) we provide more specifics about the linkage algorithm, misreporting using weighted estimates and logistic regressions models, creating a face validity scoring algorithm for sensitivity analyses of the matched results, and lessons learned through this linkage project. This paper serves as a short summary of our work and findings.

It is important to remember that, while the NHANES is designed to produce nationally representative estimates of numerous measures of the health and well-being of the non-institutionalized U.S. population, the results presented herein are based on the sample from one state and cannot be generalized to the U.S. population. The results of this pilot should be interpreted with caution and in light of our objectives.

Suggested citation:
Misreporting of Food Stamp Participation in a Household Survey: Results from a Single-State Pilot Study of the National Health and Nutrition Examination Survey NHANES” (with John A. Kirlin, Lisa Mirel, and Daniel Schroeder). Unpublished paper presented at the 2012 Federal Committee on Statistical Methodology (FCSM) Research Conference, Washington, D.C., January 10-12, 2012.

Pulling Together: Linking Unemployment Insurance and Supplemental Nutrition Assistance Program Administrative Data to Study Effects of the Great Recession

Download PDF (presentation slides)

Abstract:
Coming soon

Suggested Citation:
“Pulling Together:  Linking Unemployment Insurance and Supplemental Nutrition Assistance Program Administrative Data to Study Effects of the Great Recession” (with Theresa Anderson and John A. Kirlin).  Unpublished paper presented at the 2012 Federal Committee on Statistical Methodology (FCSM) Research Conference , Washington, D.C., January 10-12, 2012.

The SSDI “Benefit Offset” Experiment: Landing the Pilots

Download PDF

Abstract:
The Social Security Disability Insurance (SSDI) program includes transitory incentives that promote the return to work by beneficiaries, but after these incentives end, beneficiaries lose their entire disability benefit for the first dollar earned above a monthly level termed “Substantial Gainful Activity” (SGA). It is widely presumed that this ―cliff‖ restrains employment among beneficiaries. Congress included requirements within the Ticket to Work and Work Incentives Improvement Act of 1999 for a demonstration project that examines the costs and benefits of replacing (―offsetting‖) the SGA cliff with a more gradual reduction in benefits: the SSDI benefit is to be reduced by $1 for every $2 earnings beyond that threshold. Skeptics questioned the ability of the Social Security Administration to administer such an experiment, and even SSA stakeholders acknowledged that conducting the experiment within the context of regular SSA operations posed substantial design and management challenges. SSA launched a pilot project intended to provide experience with new benefit offset administration procedures as well as with the conduct of a randomized trial of the new program. Volunteers for the experiment were recruited from beneficiaries likely to be in the process of returning to work in order to ensure that the pilot would produce experience administering the new program. However, this target group also poses the most serious cost issues because many of them would return to work even in the absence of the incentive and thus would receive a partial benefit under the new program instead of no benefit under the current program. This paper reviews challenges faced in implementing the pilot and the results. The pilot ―delivered‖: Random assignment was completed successfully and the revised benefits computation system was successfully performed for the treatment group, albeit with some informative missteps. The offset produced an increase in the prevalence of earnings above SGA. However, it also increased benefit payments because the savings generated from increases in employment above SGA were smaller than the costs of partial benefit payments to those who would have worked above SGA under the existing program rules. The outcomes underscore the need for a larger experiment that includes a representative sample of all SSDI beneficiaries. Such an experiment, called the Benefit Offset National Demonstration (BOND), will begin in 2011.

Suggested citation:
Wiseman, Michael, Weathers, Robert, and Hemmeter, Jeffrey. “The SSDI Benefit Offset Experiment: Landing the Pilots.” Paper prepared for the Fall 2010 Conference of the Association for Public Policy Analysis and Management, Boston, MA, November 4-6, 2010.

Poverty in the US and the UK: Relative Measurement and Relative Achievement

Download PDF

Abstract
By the government’s official measure, 18 percent of children in the United States were living in poor families in 2007. In the United Kingdom, where the Labour party has set a 2010 goal to reduce child poverty by 2010 to half the level observed in 1998/99 (and a 2020 goal to eliminate it), the official measure for 2006/2007 was 22 percent. While it may appear at first that US children are in a better position, this is misleading because of differences in procedures for measuring poverty in the two countries. Poverty in the UK is assessed by comparing a broadly defined measure of household income to a threshold amount equal to the 30th percentile of the overall income distribution. When a similar approach is used for US data, the estimated child poverty rate rises to 29 percent. It is likely that the new US administration will alter current procedures for poverty assessment in the US, and UK methods would be usefully studied. At the same time, the UK would benefit from study of American survey procedures and reform proposals.

Suggested citation
Wiseman, Michael, and Shwalb, Rebecca. “Poverty in the US and the UK: Relative Measurement and Relative Achievement.” Paper prepared for the 30th Annual Research Conference of the Association for Public Policy and Management, Los Angeles, CA, November 6-8, 2008.

Opening Up American Federalism: Improving Welfare the European Way

Download PDF

Abstract
The European Union is experimenting with using an Open Method of Coordination (OMC) for pursuing common social policy objectives in the context of the demographic, economic, and institutional diversity of Member States. OMC involves jointly developed and monitored procedures for benchmarking Member State accomplishments, developing goals for improvement, planning action, and assessing achievement. While American federalism differs from EU governing structures, OMC procedures appear applicable to problems of governance in various US social programs. This is illustrated by outlining changes that might occur were OMC procedures applied to joint federal-state management of the High Performance Bonus in the Temporary Assistance for Needy Families program. OMC techniques would help clarify program objectives, encourage performance orientation, and identify successful practices.

Suggested citation
Wiseman, Michael, and Walker, Robert. “Opening Up American Federalism: Improving Welfare the European Way.” Manuscript, January 4, 2006.