Transcription

An Introduction to CostEffectiveness and Benefit-CostAnalyses in Education ResearchNatalie A. Koziol, Ph.D.Presented as part of the Nebraska Methodology Applications SeriesSeptember 8, 2017

Outline Overview and MotivationCost AnalysisCost-Effectiveness AnalysisBenefit-Cost AnalysisAdditional Considerations

Acknowledgement The Center for Benefit-Cost Studies ofEducation (CBCSE; cbcse.org) is at the forefrontof economic evaluation in education research– Tons of resources and examples on their website– IES-funded methods training program (now closed) Content and organization of presentation isbased on their textbook (Levin, et al., 2018)

Overview andMotivation

Economic Evaluation/Analysis “Broad set of techniques for evaluation anddecision-making” (Levin et al., p. 3)– Cost Analysis or Cost Feasibility Analysis (CA/CF)– Cost-Effectiveness or Cost-Utility Analysis (CEA/CU)– Benefit-Cost Analysis (BCA)

Type ProcedureAdvantagesCA/CF Estimate averageannual cost ofprogramimplementation perparticipant (or total ormarginal cost)CEA Estimate incrementalcost to achieve 1-unitincrease ineffectiveness, relativeto an alternativeprogram with commongoalsCUEstimate incrementalcost to achieve 1-unitincrease in utility,relative to analternative programwith common goalsBCA Estimate a program’smonetized effects(benefits) relative toits monetized costsFully explicate programNo consideration of programresources and who finances outcomesresourcesRule out unaffordablealternativesConsider costs & outcomes“Straightforward” extensionto traditional effectivenessstudiesConsider costs & outcomesAllow for multipleeffectiveness measuresAccount for stakeholderpreferencesDisadvantagesBowden et al.(2017)Effectiveness must beHollands et al.represented by a single measure (2013)Limited to relative conclusionsAlternatives must target (&identically measure) the sameoutcomeMust determine how to define,weight, & combine utilitiesLimited to relative conclusionsAlternatives must target (&identically measure) the sameoutcomesConsider costs & outcomes Difficult to monetize someProvide absolute information impactsCapture all benefits and for Long-term benefits generallyas long as they lastrequire projectionCompare diverse alternativesBased in part on Table 1.1 (p. 22; Levin et al., 2018)ExamplesLewis et al.(1994)Belfield et al.(2015)Bowden et al.(2015)Levin & Garcia(2013)

Motivation Optimize scarce resources and maximizeeducational outcomes Improve education policy via evidence-baseddecision-making Meet the requirements of funding agencies Justify use of a particular program

Example Research Questions Is the program affordable? (CA/CF)Who primarily bears the costs? (CA/CF)How scalable is the program? (CA/CF)Given a fixed budget, which program (among those with acommon goal) is most effective (CEA)? Given a minimum level of effectiveness, which program(among those with a common goal) is least costly? (CEA)? Do the returns on the program justify its costs? (BCA) Among alternatives targeting the same OR differentoutcomes, which program is the most “socially desirableinvestment”? (BCA)

Key Considerations PurposeAudiencePerspectiveRelevant alternativesTimespan of evaluationAnalytic method

Cost Analysis

Purpose Fully explicate program resources andfinancing of resources Rule out unaffordable alternatives First step of larger evaluation (CEA or BCA)

Defining Costs “All the resources that are involved in ‘making theintervention work’” (p. 51; Levin et al., 2018)– Costs directly tied to program implementation– Induced costs (Bowden et al., 2017) Opportunity cost (value is next best use)Incremental vs. total costsCosts vs. who finances costsBudgets and expenditure reports cost analysis

Estimating and Reporting Costs:The Ingredients Method1)2)3)4)Determine program “ingredients”Assign a value (price) to each ingredientCalculate costsIdentify optimal presentation of costs

Determining Ingredients (Step 1) Identify all resources needed to replicate effects, e.g.,–––––Personnel (e.g., teachers, volunteers, coaches)Facilities (e.g., classroom space)Equipment/materials (e.g., software, textbooks)Other program inputs (e.g., scholarship funds)Client inputs (e.g., parents’ time but usually not children’s time) Fully detail resources, e.g.,– Qualifications (e.g., certifications, experience)– Dimensions/Characteristics (e.g., sq ft, special features)– Dosage/Quantity (e.g., % of usable time allocated to program)

Sampling Procedures Existing program descriptions are generally insufficient– Lack precise information about ingredients’ characteristics– Fail to capture site-by-site variability– Describe intended resources which may not correspond tothe actual resources tied to observed impacts Need to sample information from implementation sites– Use traditional data collection methods– Perform concurrently with program implementation

Assigning Prices (Step 2) Local (site-specific) prices vs. national (expected) prices– Local prices are subject to greater sampling error and are lessgeneralizable, but may be more meaningful to primary stakeholders– National prices are more generalizable and thus may be moremeaningful to secondary audience(s) Market prices vs. shadow prices– Market prices can be drawn from national databases (e.g., seeHollands et al., 2015) or site-specific documents (e.g., for local prices)– Shadow prices are needed in the absence of market prices and areestimated as the value of the ingredient’s next best use Account for characteristics of ingredients when identifying prices

Adjusting PricesHollands, Hanisch-Cerda, Menon et al. (2015) Inflation– Convert prices to the same time period– 𝐴𝑃 𝑃 𝐼𝐸 Τ𝐼𝑃 Geographic location– (Potentially) convert national prices to local prices– 𝐺𝑃 𝑃 𝑅𝑃𝑃𝑒 Τ𝑅𝑃𝑃𝑝 Amortization/Depreciation– Calculate adjusted annual costs for assets with lifetime 1 year (e.g., training, owned facilities)– 𝐴𝐶 𝑃 𝑅 1 𝑅 𝐿 Τ 1 𝑅 𝐿 1 Discounting– Compute present value (prefer future cost [immediate benefit] to immediate cost [future benefit])– 𝑃𝑉 𝑃 𝑒 𝐷 (1 𝑌) Personnel Benefits– Adjust salary/wage estimates to account for fringe benefits– 𝑇𝐶 𝑊 1 𝐵AP inflation-adjusted price; P unadjusted price; IE inflation index for year in whichprices will be expressed; IP inflation index for year of unadjusted price. GP geographically-adjusted price. RPPe geographical index for location in which prices willbe expressed. RPPp geographical index for location of unadjusted price. AC annualcost; R interest rate; L lifetime of asset; PV present value; D discount rate; Y yearof cost or impact; TC total compensation; W wages; B benefits rate (% of wages)

Calculating Costs (Step 3) Cost of ingredient 𝐶𝑖 𝑃𝑖 𝑄𝑖Total cost of program implementation σ𝑖 𝐶𝑖Average cost per program participant σ𝑖 𝐶𝑖 Τ𝑁Marginal cost (less straightforward)𝐶𝑖 cost of ith ingredient;𝑃𝑖 price of ith ingredient;𝑄𝑖 quantity of ith ingredient;N number of program participants

Reporting Costs (Step 4) Report annual (per intervention year) cost Provide aggregate-level information (e.g., average cost) Disaggregate information where meaningful, e.g., by–––––constituency (e.g., highlight who finances the costs)site (e.g., identify site-by-site variability)ingredient (e.g., inform scalability questions)year (e.g., identify savings due to increased efficiency)demographic group (e.g., highlight impact of dosage)

Example Cost Table(p. 9; Bowden et al., 2017)

CostOut – the CBCSE Cost Tool KitHollands, Hanisch-Cerda, Levin et al., 2015 Free online tool developed by CBCSE to facilitate cost analysesIncludes a multi-source database with national market pricesAutomatically makes price adjustmentsAutomatically calculates costs given ingredient quantities and pricesDisaggregates costs across ingredients and constituenciesIncorporates effectiveness information for CEAOffers simple comparative reportsFor more detail see Hollands, Hanisch-Cerda, Menon et al. (2015)

Hypothetical Example (HYPE!) HYPE! is a 9-wk afterschool intervention aimed atdecreasing aggression in K-3rd graders with behaviorconcerns Ingredients of intervention (100 kids across 20 schools)– Personnel 20 teachers (no special qualifications), each 3 hrs/wk 20 coaches, each 1 hr/wk– Facilities 20 classrooms, each 900 sq ft with lifetime of 30 years, 3 hrs/wk– Client Inputs 100 parents, each 1 hr/wk for meetings, 12 mi/wk and .5 hr/wk fortransportation

HYPE! CostsExpressed in constant 2016 dollars using national prices and inflation rate of 3.5%, and assuming 1440 hrs/academic year%Unadj. PriceBenefitIngredient Unit Quant UsePriceYear Price SourceRateAdjust. PriceTotal CostCostOut:3*9Elementary*20school teacher40*(711.104/709.997)*(1 .50)60.24*540TeacherHour 540 10040.002015grades K-60.50 60.24 .91*(711.104/673.818)*(1 .5) 50.64*180CoachHour 180 10031.912011 grades K-120.50 50.64 ementary*((0.035*(1 0.035) 30)/17111.86*1440classroom (900((1 0.035) 30-1))20*0.0188Classroom Unit 20 1.88 314232.21 2015sq ft) 17111.86 6434.06IndependentSector (value of100*volunteer time,1.5*9 includes fringe24.14*1350ParentHour 1350 10024.142016benefits)24.14 32589.00100*12*9 IRS (standard.54*10800TransportMile 10800 1000.542016 mileage rate)0.54 5832.00Total CostCost/Student86502.4686502.46/100 865.02

Cost-EffectivenessAnalysis

Purpose Identify, among alternatives with commongoals, the program that optimizes educationalresources with respect to maximizingeducational outcomes

Performing a CEA1) For all relevant alternatives, concurrentlyestimate incremental program costs and impacts(effects) relative to those of a baseline group2) Calculate cost-effectiveness ratios (CERs)3) Compare and rank alternatives programs

Measuring Effectiveness (Step 1b) Program impact as indicated by the theory of changeStrength of evidence depends on the study designStandardized effects facilitate interpretationSame effectiveness measure needed for all alternativesDiscounting needed if programs vary in timing of effectPrograms often target multiple domains at multiple timepoints but CEA requires a single measure of effectiveness

Calculating CERs (Step 2) CERs indicate the incremental cost perparticipant needed to achieve a 1-unit increasein effectiveness per participant, relative to analternative program with common goals CER 𝐶𝑇 𝐶𝐶 Τ 𝐸𝑇 𝐸𝐶 𝐶 Τ 𝐸𝐶𝑇 Average cost per participant for target program𝐶𝐶 Average cost per participant for alternative program𝐸𝑇 Average effectiveness per participant for target program𝐸𝐶 Average effectiveness per participant for alternative program

Comparing Alternatives (Step 3) Generally prefer alternative with smallest CERbut consider– sign of 𝐶 and 𝐸– relative magnitude of 𝐶 and 𝐸 in relation tobudget restrictions or minimum required level ofeffectiveness– heterogeneity in CERs

Example Cost-Effectiveness MapMore CostlyCERBCERCLess EffectiveCERAMore EffectiveCERECERDLess Costly Origin represents baseline comparison groupSoutheast quadrant is optimal (more effective and less costly)Northeast quadrant is more likely (more effective and more costly)

HYPE! Hypothetical CEA An RCT was performed to evaluate the impact ofHYPE! on aggression as measured by the BASC-3TRS Aggression scale The mean aggression score for HYPE! students was.13 SDs lower than the score for BAU students 𝐶𝐸𝑅 865.02Τ( .13) 6654, i.e., it takes anadditional 6654 per student to decreaseaggression by 1 SD unit (relative to BAU)

Multiple Measures of Effectiveness The CER necessitates one effectiveness measure Options for handling multiple measures include– calculating multiple CERs to either (a) reveal a clear“winner,” or (b) highlight program tradeoffs– conducting a CU– conducting a BCA

Performing a CU Similar to CEA but denominator represents utility (𝑈) Use Multiattribute Utility Theory as a guiding framework– Choose method for determining individual utilities, 𝑈𝑖 𝑥𝑖 (e.g.,proportional scoring, direct method, variable probability method)– Choose method for determining individual importance weights, 𝑤𝑖(e.g., direct method, variable probability method)– Choose function for combining 𝑈𝑖 𝑥𝑖 , e.g., 𝑈 σ𝑀𝑖 1 𝑤𝑖 𝑈𝑖 𝑥𝑖 Rarely performed in education research– Requires sampling preference data from relevant populations– Relies heavily on judgment and makes strong assumptions

Benefit-CostAnalysis

Purpose Determine, in an absolute sense, whether aprogram is a “socially desirable investment” Determine whether a program is a more“socially desirable investment” thanalternative programs, including– education programs with common goals– education programs with different goals– non-education programs with different goals

Performing a BCA1) Estimate immediate program costs and impacts, andpredict long-term impacts2) Monetize costs and impacts using appropriate priceadjustments3) Calculate economic metrics to combine monetizedcosts and impacts4) Evaluate program

Measuring Impacts (Step 1b) Consider ALL (ideally causally-linked) program impacts assoon as they occur and for as long as they last– Immediate impacts (e.g., decrease in externalizing behaviors)– Intermediate impact (e.g., decrease in HS dropout rate)– Long-term impacts (e.g., increase in wages, improved health) Immediate impacts are directly observed Longer-term impacts are typically predicted, e.g., via– secondary analyses of large-scale federally-funded datasets– published meta-analytic findings (e.g., see WSIPP, 2017)

Snapshot of Exhibit A1 (p. 184; WSIPP, 2017)

Monetizing Impacts (Step 2) A benefit (𝐵𝑖 ) is a monetized impact (𝑃𝑖 𝐸𝑖 )– 𝑃𝑖 is typically a shadow price– Benefits may be positive or negative– Some benefits