- sors. Some staff members lack the qualifications and training necessary to work with inmates during the assessment process, and agencies might not adequately allocate time for staff to complete assessments or access relevant, collateral information


service delivery requirements, resources, or other constraints, prisons might use nonthera- peutic staff to deliver programming. One participant referred to this as cognitive-behavioral intervention versus treatment or therapy. Both cognitive-behavio


cipants stressed the importance of process evaluations; it is not enough to know
whether a program is effective, but it is critical to understand why a program is effective so that it can be replicated.
Finally, the participants discussed the concept


elity.
Research has demonstrated that validated tools based on RNR principles can correctly predict recidivism risk (Des- marais and Singh, 2013). Moreover, research has shown that employing demonstrated, effective correctional practices and programm


eeds. Through this process, we developed the research agenda that structured the topics presented in the main report. The descriptions in this appendix are adapted from those in previous publications of the Priority Criminal Justice Needs Initiative


to make the group as represen- tative as possible of different disciplines, perspectives, and geo- graphic regions, the final output of the workshop likely will be significantly influenced by the specific group of experts invited to participate. It


, the ResponseCard RF LCD from Turning Tech- nologies). Each participant was asked to individually score each problem or opportunity and its associated needs using a 1–9 scale for two dimensions: importance and probability of suc- cess.
For the impor


to three tiers using a hierarchical clustering algorithm. The algorithm we used was the “ward.D” spherical algorithm from the “stats” library in the R statistical package, version 3.5. We chose this algorithm to minimize within-cluster variance when


d values—we also set a threshold that at least 25 percent of the workshop participants must have voted on that need (and then rounded to the nearest full par- ticipant). For this workshop, there were 15 participants, so for any votes to have an effec


tices on how to sequence and prioritize the order in which each item should
be addressed.
36%
14%
29%
7%
14%
17
        0% 0% 0% 0% 123456789
NOTE: Percentages on each question did not always sum to
100 percent because of rounding and variation in th


 the 50 identified needs,
• 11 were related to organizational issues (two of which were high priority)
• 16 were related to the selection and implementation of RNA tools (four of which were high priority)
• nine were related to the administration of


fidelity.
Need Tier
• Develop training and guidebooks for leadership on the state of knowledge about RNA principles and how EBPs can enhance institutional security.
• Develop a mechanism for the regular dissemination of useful information in digestib


 might not be sufficiently complete.
The aggregate results from RNA tools can be used to assess whether an agency’s programming is sufficiently meeting the needs of its population.
It is difficult to understand whether needs assessments are appropria


 the “big picture” with respect to the needs of an individual.
Allowing staff or committee overrides of the evidence-based risk assessment can result in less desirable outcomes.
Many programs are advertised to address the same need, but it is unclear


orrec- tional and rehabilitative objectives.
• Identify the costs and benefits of employing special groups of staff (e.g., change champions) to manage change initiatives.
    Administration of assessments
Programming
Organizational issues
2
2
3


   Selection and implementation of RNA tools
Administration of assessments
Programming
3
3
3
                 a Each unique pairing of a problem or opportunity with a potential solution is counted as a separate need. This need is associated with thre


truments and Methods of Assessment,” Criminal Justice and Behavior, Vol. 36, No. 6, 2009, pp. 567–590.
Carson, E. Ann, Prisoners in 2018, Washington, D.C.: Bureau of Justice Statistics, NCJ 253516, April 2020.
Casey, Pamela M., Jennifer K. Elek, Roge


n A., Duren Banks, John S. Hollywood, Dulani
Woods, Amanda Royal, Patrick W. Woodson, and Nicole J. Johnson, Fostering Innovation in the U.S. Court System: Identifying High-Priority Technology and Other Needs for Improving Court Operations and Outcom


nstructing the Risk-Need-Responsivity Model: A Theoretical Elaboration and Evaluation,” Aggression and Violent Behavior, Vol. 12, No. 2, March 2007, pp. 208–228.
Yang, Min, Stephen C. P. Wong, Jeremy Coid, “The Efficacy of Violence Prediction: A Meta


re, Maryland. He holds an M.S. in agricultural economics (applied economics).
Brian A. Jackson is a senior physical scientist at the RAND Corporation. His research focuses on criminal justice, homeland security, and terrorism preparedness. His areas


ion are those of the author(s) and do not necessarily reflect those of the Depart- ment of Justice, the RAND Corporation, or the organizations represented by any of the workshop participants.
Limited Print and Electronic Distribution Rights
This docu


  CORPORATION
LYNN A. KAROLY, STEPHANIE J. WALSH
Estimating the
Cost of Quality
Early Childhood Care and Education
in Oklahoma
 Sponsored by the Oklahoma Partnership for School Readiness Foundation
For more information on this publication, visit www.


d expenditures for the most recently completed fiscal year. The information supported estimation of the per-child cost of care by child age, setting, and quality rating. Although not intended to provide a representative sample, the data collected fro


 Centers by Reaching
for the Stars Rating and Selected Child Age Group..............................................................40 Table 3.4. Estimated Percentage of Per-Child Cost Recovered in Sampled Centers
Using Child Care Subsidy Program Rei


7 Table D.5. Assumed Unit Prices for FCCH Cost Model by
Reaching for the Stars Rating (2019 Dollars) ....................................................................138
viii
Summary
The affordability of licensed early childhood education and care


to understand their program structure and the associated expenditures for the most recently completed fiscal year (typically 2018); second, drawing on the findings from these illustrative providers, we developed a cost model that was used to examine


tart programs is expected to differ from other ECCE programs given the more comprehensive set of services required; and the latter because the study timeline could not accommodate the more-lengthy human subjects research review process required for i


 the three types of providers in our sample (centers, large FCCHs, and FCCHs), as well as generating estimates of per-child expenditures and the degree of variation by provider type and other characteristics.
Program Features of the Illustrative Prov


.
• Licensed ECCE providers do not routinely have well-organized financial records, according to our illustrative providers. Although some of the large centers and a few of the FCCHs kept precise records—some of which were maintained by bookkeepers o


n the attendance rate falls below 80 percent (so that the full subsidy, which is based on attendance rather than enrollment, would not be received). The reimbursement rates more than covered estimate costs for Two Star providers but undercompensated


riking. When a small FCCH with a total capacity of seven children maintains a group size of only six children, its costs increase 14 to 18 percent, and the amount by which the state subsidy covers its costs is reduced by 10 to 15 percent. For centers


are a few exceptions, mostly for Two Star programs and the older age groups, where the reimbursement rates are estimated to exceed the provider cost estimate by 10 percent or more (green shaded cells). The remaining few cases have reimbursement rates


ming 80% attendance with full enrollment
Two Star
Estimated per-child cost ($)a Per child year
Per child day
Estimated percentage of cost recovered using the subsidy reimbursement rate (%)
Assuming 100% attendance with full enrollment Assuming 90% at


r, unique considerations for the FCCH provider include how to properly treat their housing costs as a business expense and their own compensation through a salary or retained earnings. Access to expert financial advice periodically or on an ongoing b


, is a key source of cost variation to recognize, although any age groups with the same teacher-child ratio requirement could have similar reimbursement rates. Quality rating tiers are another potential key cost driver, especially if they incorporate


from our analyses and findings.
Recommendation 1. Use cost information collected directly from providers and/or through cost modeling to reconsider the current Child Care Subsidy program reimbursement schedule. The validity of any reimbursement syste


federal rather than state monies. This requires a better understanding of why some ECCE providers do not apply for the program. In some cases, this may be a provider choice. But understanding and addressing any barriers to take-up would ensure a mor


 family child care home
GED General Educational Development
FTE full-time equivalent
IDEA Individuals with Disabilities Act
IEP Individualized Education Plan
LEA local education agency
NAC National Accreditation Commission for Early Care and Educatio


oviding high-quality ECCE programs. Information on the resources that providers require to deliver quality ECCE can then inform issues about affordability on the part of families, as well as the public sector’s role in promoting quality and subsidizi


stand the cost of ECCE in Oklahoma in various types of licensed settings. Following prior studies (e.g., Karoly and Gomez, 2019), we adopted a two-pronged approach: (1) collecting program and expenditure data from a sample of licensed providers, purp


cost model was developed to assess how changes in any one cost driver affect the estimated cost per child, while holding all other program features constant. Key cost drivers that were examined include the teacher-child ratio, teacher qualifications


they reported their annual expenditures, compared with center-based programs, which tend to have a dedicated bookkeeper or accountant on staff or under contract. Fourth, providers vary in the start and end month of their annual fiscal year, which me


earlier, our analysis focused on private licensed center- and home-based ECCE providers purposively sampled across the range of the quality ratings produced by the state’s QRIS, Reaching for the Stars. In this section, we provide relevant context fo


lots. Many providers choose to serve fewer children than their license allows. We differentiate between licensed capacity versus desired enrolled, where the latter captures how many children a provider seeks to serve, which may be less than their lic


achers may be enrolled in high school.) Directors are also required to have some additional postsecondary coursework, credentials, or certificates. In other states, it is most common for center directors and master teachers to at least have a Child D


Excellence in Child Care program also covers up to 90 percent of the cost of college tuition and books for qualifying staff (e.g., those in One Star Plus programs with at least 10 percent of enrolled child receiving state child care subsidies).
Reach


d master teacher
qualifications
Same as licensing
Same as licensing
Same as licensing
• Added administration • PD follows ELG
• Improved learning
environment
• Added family engagement • Increased master teacher
qualifications
Birth to 11 months 1:4 1


each higher tiers in Reaching for the Stars, the reimbursement rates associated with the Oklahoma Child Care Subsidy program increase with each rating level. The Child Care Subsidy program assists low-income working families (those with income up to


 monthsa
Three Star
Daily rate
Birth to 12 months 13 to 24 months 25 to 48 months 49 to 72 months
Blended rate
49 to 72 monthsa
a Traditional rate and extended
SOURCE: OKDHS, 2020.
NOTE: Blended rates and rates for 73 months to 13 years are also incr


group size, teacher-child ratio), and other features associated with program quality (e.g., staff education and compensation, quality rating). Much of the literature has centered on cost analyses of preschool programs for three- and four-year-olds i


e staff, the salary scale and generosity of the fringe benefit package, the teacher-child ratio in the classrooms, and program intensity (e.g., part- versus full-day programs, academic-year versus calendar-year programs). ECCE teachers in public-scho


estimated at $35,354 per child on an annual basis, toddler care at $28,203 per child, and preschool-age care at $13,655 per child. These estimates are based on program features consistent with an earlier NASEM report that recommended bachelor’s-leve


ECCE cost survey was to collect information on program structure and expenditures for 30 licensed center- and home-based ECCE programs located across Oklahoma serving infants, toddlers, and preschool-age children. As we discuss next, this process req


roviders were selected purposively to ensure variation along key dimensions that would be expected to affect provider per-child costs, namely:
• provider type (three categories) defined as for-profit centers, not-for-profit centers, and FCCHs of any


al program information, staffing levels and benefits, sources of revenue, and annual expenditures. Separate versions of the instrument were used for center- and home-based providers. Most importantly, program features and expenditures were collected


licensed ECCE center- and home- based providers in Oklahoma. Replicating the same approach for selecting another sample may also produce different values for these statistics. For this reason, we do not report such statistics in the section that foll


ban 52310 Large rural 5218 Rural 3227
        Licensed Provider Type
         Characteristic Centers Large FCCHs FCCHs
Reaching for the Stars rating (number) One Star and One Star Plus
Two Star
Three Star
1 1 9 4 3 1
9 5
Yes 4116
Accredited (number)


oms or groups (number) 1
2
3to4 5to6
7 or more
Ages of children served (number) Children aged birth to 11 months Children aged 12 to 36 months Children aged 3 to 5 years Children aged 6 or more years
Has waiting list (number)
Has waiting list for spe


ther expensive fringe benefits, such as retirement, health, or dental benefits. Overall, FCCHs, especially the smaller ones, offer fewer benefits compared with centers. Although we did not collect salary or wage data for all center and FCCH staff in


 0 10 6 6 25 0 0 0 4 2 15 2 1 6
0 5 5 6 6 25
Speech screening
Speech services
Developmental assessment
Meals 13 Counseling 0 Referrals 9
Transportation to and/or from program
No services other than meals Total providers (number)
Sources of revenue. F


Hs
Total
Licensed Provider Type
   11 – – 11 18 – – 18 15 – – 15 18 – – 18 11 – – 11
5 – – 5
– 10 7 17 78 10 7 95 13 6 6 25
25
Total
               Characteristics that vary across classrooms, stratified by age group (see Table 2.9), include desired


age group is consistent with the licensing requirements in both centers and FCCHs. Yet many center-based providers operate with teacher-child ratios lower than the licensing ceiling (i.e., fewer children per teacher than required), which may result


less relevant. At FCCHs, children are served in mixed age groups in one or more groups or rooms and expenditures per child are not likely to vary in the same way by child age or room compared with center-based programs. Thus, for FCCHs, all expenditu


s.
 28
undated). 22 In the remainder of this section, we highlight key assumptions and the alternative scenarios considered. Additional details about the cost modeling approach is provided in Appendix D.
Assumptions for Cost Model
The cost analysis m


tors, that would allow more flexibility in the quantities and prices included in the model and more readily facilitate sensitivity analyses across many scenarios.
23 Again, for the cost model, we did not consider classrooms serving school-age childre


 classroom Staff Classroom expenditures per child
Same as One Star/ One Star Plus
Same as One Star/ One Star Plus
50th percentile of OK preschool teachers’ salaries 50th percentile of OK child care salaries Same as One Star/ One Star Plus
5%


assumption is different than prior tiers.
 31
maintaining accreditation (Three Star).25 A similar pattern of expenditures is assumed for small and large FCCHs, although the accreditation costs are assumed to be lower based on NAFCC fees.
The other ke


er enrollment, across NAEYC, NAC, and NECPA. When totaled across fees at enrollment, verification, and annual reporting, the total fees range from $2,000 to $2,200 for centers with up to 50 children, with NAEYC in the middle at $2,120. For centers wi


 top tier) and that Three Star large centers employ a full-time associate director (compared with a half- time associate director at the baseline top tier, see Appendix D, Table D.1).
• Geographic cost differences. Our baseline model assumes state-le


inistrative staff and shared resources). We repeat the same analyses for home-based care costs, although we do not attempt to generate cost estimates by child age group, given the mixed-age structure in the groups of children served in typical home-b


ce (e.g., facility rent or mortgage, utilities, maintenance), and transportation (when provided). Expenditures on staff PD is a low cost per child, sometimes with no such expenditures, while exceptional centers had annual expenditures of several thou


respectively, the sum of the cost components is not expected to add to the total because the minimum or maximum values for a given cost component are not necessarily for the same provider with the overall minimum or maximum total cost.
• Cost per chi


aid for by the program. Variation is also evident for other cost components in Table 3.2, but the magnitudes of the differences are smaller.
The small number of centers in our sample limits our ability to make robust comparisons across providers base


mates.
Overall, for the centers in our sample, the estimates indicate that less than full cost would be recovered on average in rooms serving children from birth to 11 months (infants). For infants, this finding holds across the range of per-child co


ed on the range for the estimated per-child cost. The minimum and maximum are reversed for the estimated percentage of cover recovered. In other words, the minimum per-child cost estimate will have the maximum percentage of cost recovered. For the ma


n sampled providers that served that age group ($6,900) and is well under the
30 The estimates in Table 3.5 are not necessarily comparable with other estimates of per-child ECCE costs, largely because of different assumptions about the ingredients pr


5,012 4,483 96 86 19.28 17.24
89.2 98.6 80.3 88.7 71.4 78.9
                                    Table 3.5. Model-Based Baseline Per-Child Cost Estimates for Centers (Continued)
              Birth to 11 12 to 23
Indicator Months Months 2 Years 3 Year


m materials 1.3 Food 7.6 Transportation 0.0 Space 23.1 Administration 1.6 Total 100.0
Three Star
Estimated per-child cost ($)a
Per child year 16,698 Per child week 321 Per child day 64.22
Estimated percentage of cost recovered using Child
Care Subsid


4,663
5,267
One Star One Star Plus
Two Star Three Star
-4,000
Title
-147 -661
0
4,000 8,000 12,000 Annual per-child cost (2019 $)
16,000 20,000
4,420
4,810
5,928
9,226
Reimbursement Unrecovered cost
  6,552
2,674
    NOTES: The amount of costs recove


e Star and One Star Plus, with an increment of about $200 to $400 dollars per child, depending on the age group, relative to a One Star rating. The next increment from One Star Plus to Two Star is about $1,600 for infants, declining to about $600 for


or four- year-olds. Assuming 100 percent attendance, the Two Star reimbursement rate would seem to consistently over-reimburse providers for two-year-olds and older (especially three-year-olds), based on our cost estimates. However, knowing that, in


mple, under-enrollment—enrolling fewer children than actual or desired capacity— would be expected to raise per child cost (see scenario C1 in Table 3.6). The C1 scenario shows that under-enrolling by about 10 percent of desired capacity (1 to 3 chil


ance with full enrollment Assuming 80% attendance with full enrollment
C4. Higher cost areas
Estimated per-child cost ($)a Per child year
Per child day
Change in per-child cost from baseline (%)
Estimated percentage of cost recovered using Child Care


indicators regarding the owner/operator’s compensation and treatment of housing cost.
We begin by noting that there is considerable variation across the 12 home-based sites in the per-child cost indicators, both within and between the two size catego


r the averages, the sum of the cost components may not add to the total
because of rounding. For the minimums and maximums, the sum of the cost components is not expected to add to the total because the minimum or maximum values for a given cost comp


a portion of their housing costs (e.g., rent or mortgage, utilities) as a business expense, based on the portion of annual hours that the home was used for business (both operating hours and other hours spent cleaning, preparing meals, and so on) an


nual per-child space costs and for classrooms serving two-year-olds and three-year-olds, they can operate with larger ratios (1:8 and 1:12, respectively), which lowers their per-child classroom personnel costs. Nevertheless, these potential cost savi


mated percentage of cost recovered using Child Care Subsidy program reimbursement rate (%)
Assuming 100% attendance at observed enrollment Assuming 90% attendance at observed enrollment Assuming 80% attendance at observed enrollment
Total number of l


n and operate with a 1:7 ratio (see Table 2.10). However, because they are assumed to have space to serve up to 12 children, their per-child space costs are higher than the small FCCHs operating with a lower square footage for the same enrollment of


 resource use in other cost categories, and cost-of-living differences across communities. We explore several of these other sources of variation and the implication for per-child costs in the next section.
Cost Variation Associated with Key Cost Dri


rly 100 percent of the per-child cost under the baseline Child Care Subsidy program reimbursements, the same large FCCH with under-enrollment would recover about 85 percent of its cost.
• If there are cost components that do not vary by region, cost


ntage of cost recovered (%)
85.5 91.7 73.5 76.6
Assuming 100% attendance with full enrollment
Assuming 80% attendance with full enrollment
58.8 61.3
68.4 73.4 NOTES: Costs are estimated in 2019 dollars. For more details on the alternative scenarios,


services provided to families, and sources of revenue, among other features. In part, this diversity may not be surprising, as we intentionally sampled providers to obtain variation across provider type, Reaching for the Stars rating tier, for-profit


ng to collect information on ECCE provider annual expenditures, many smaller centers and FCCH providers did not have clear financial statements that carefully differentiated revenues from expenses in well- structured cost categories. For FCCHs, docum


nt for the FCCHs, where lower-cost providers typically had their costs full covered, even with less than 100 percent attendance, but higher-cost providers would see less than full reimbursement. The reimbursement rates more than covered costs for Two


p between quality and per-child cost reflect the quality features accounted for in Reaching for the Stars, specifically those that have implications for program cost. The relationship between quality tiers and per-child cost could change if a differe


on other factors, such as local area wages and whether tiered child care subsidies will cover the higher compensation costs or whether families can afford to pay higher fees or tuition. The fact that we did not see higher- rated providers with highe


able business model and the required financial literacy.
ECCE Providers and Their Revenue
Although our primary focus has been on the cost side of ECCE program operations, the information we collected from the sample of illustrative providers also poi


age and rating tier. The cost estimates and the reimbursement rates are followed by the percentage change in dollar amounts moving across age groups within a given rating tier, first starting with four-year-olds and moving to younger ages (where cos


ercentage change in cost as move to younger ages (%)
One Star
One Star Plus Two Star Three Star
Percentage change in cost as move to older ages (%)
One Star
One Star Plus Two Star Three Star
1:4 1:6 1:4 1:6 1:4 1:6 1:4 1:6
39.63 29.46 41.21 30.63 47.


e groups, unless there are substantial changes in other cost factors across those age groups.
Which Dimensions of Cost Variation to Include in a Rate Schedule
The illustrative provider data reveal how much per-child costs can vary across ECCE provide


r-child requirement would have the same reimbursement rates. Quality rating tiers are another potential key cost driver, especially if they incorporate increases in classrooms staff education requirements and the associated compensation in moving up


of how providers structure their programs, the types and quantities of resources used, and the costs that they face. Even small nonrepresentative samples can illustrate the range of circumstances providers experience. Cost modeling provides a more s


itation, there are specific requirements for higher staff qualifications compared with Reaching for the Stars, but these standards do not require a provider to pay higher compensation for those staff, commensurate with their higher education and trai


s as well. For example, most small homes operate up to 12 hours per day with just the owner on site, a situation
72
that may lead to stress and burnout. FCCHs reported that it was difficult to find substitutes who were adequately trained and certifie


 are under 2 years old
• Or six children are in care and more than
three children are under 2 years old
FCCH All Groups
     Mixed-Aged Groups
                    Child Care Centers Large FCCH Single-Aged Groups All Groups
FCCH All Groups
          M


operation or 30 hours per week
• Has satisfactory work experience and a good
understanding of program requirements and
policies
• Is not a director or master teacher at another
program that operates concurrently at any
given time
• Can be one of thre


 providers that we sampled from represent 78 percent of licensed center- and home-based providers in Oklahoma.
Table B.1. Results of the Application of Criteria for Inclusion in the Provider Cost Survey
            Number of Providers Meeting Each
Cr


• an overview and the potential benefits of the study
• the study funder (Oklahoma Partnership for School Readiness [OPSR] through a PDG B–
5 award from the U.S. Department of Health and Human Services and the U.S.
Department of Education)
• RAND as


m information, staffing levels and benefits, sources of revenue, and annual expenditures. As noted earlier, when providers agreed to participate, they were asked a series of screening questions to verify the information we had from OKDHS and to obtai


 each classroom. When centers had more than one classroom serving the same age group, the direct and indirect costs were pooled to create a center- level total expenditure for each age group.
4. For each age group, calculate the cost per child on an


have as you go along. Please do not hesitate to reach out to Stephanie Walsh, RAND Survey Director, at swalsh@RAND.org or 412-683-2300 x 4195.
1
Provider ID:
INSTRUCTIONS
This questionnaire consists of four sections:
A. General Center / Program Infor


ary
     Most recent completed fiscal year
  For-profit / not-for-profit status
  Reach for the Stars rating
  Number of classrooms
  Licensed capacity
  Accepts state child care subsidies
  Program changes during the most recent completed fiscal yea


ission for Early Care and Education Programs (NAC)
       National Association for the Education of Young Children (NAEYC)
    Accredited Professional Preschool Learning Environment
                              pm
pm
A10. Are the regular hours just


ecent completed fiscal year, what was the number of unpaid interns in this classroom? [Enter number]
   A19. In this classroom, how many children were enrolled who had an IFSP or IEP for special needs, such as a physical disability, including hearing


physical disability, including hearing or sight problems, mental disabilities, or emotional disabilities. [Enter number]
                                       A24. For the most recent completed fiscal year, was a floater assigned to this classroom t


,whatwasthemaximumnumberofchildrenineachage group that were on your waiting list? [Enter number in each row as relevant.]
Maximum number of infants (0 to 12 months) on waiting list ................................. ________ Maximum number of toddlers


time
               Fully paid health insurance
               Paid health insurance for dependents
               Paid sick leave or personal leave
               Paid to attend staff meetings
                  B3. What is the minimum number of hour


e reported at the center level. Use the table on the next page to record the following types of classroom-specific expenditures:
D1. Wages and Salaries for Classroom Staff. In the most recent completed fiscal year, what was the total expenditure on w


$$ $$$$ $$$$
$$$$ $$$$ $$$$ $$$$ $$$$
$$$$ $$$$ $$$$ $$$$
                               D3. Classroom materials and supplies
                         17
Provider ID: _______________
                 D1. Wages and salaries for classroom staff


.
D9. Sub-Contractors: What was your total expenditure on contract workers for the year (i.e., people who work for you on a more irregular basis for whom you do not pay benefits)?
If possible/applicable, please record expenditures for the following t


__________
   No GO TO D13
D12. Which of the following is true?
a. All of our space is donated. If so, fill out this section (i and ii, or iii):
i. Number square feet donated: AND
ii. Estimated rent per square foot: OR
square feet
rent/square foot i


ty, which might include liability, fire, theft, flood, earthquake; vehicle; accident for children, staff or others; child abuse, etc. Do not include health insurance or any insurance programs that are part of employee benefits. Enter “0” if there wer


t question number, if applicable.
Provider ID: _______________
    Thank you for all your help! The information you have provided will be invaluable to our study.
24
RAND Corporation Oklahoma Child Care Cost Study
Family Child Care Homes and Large Ch


 completed fiscal year.
Topics and Respondents: This questionnaire will cover the topics listed in the table below. We also indicate potential documentation that may be useful to have on hand.
As indicated in the table, the owner/program director may


nization, such as a church or community agency? [Select one response.]
Independent  GO TO A6 Sponsored  GO TO A5
A5. Now I’d like to ask about sponsorship of your program. By sponsorship, we mean an organization that provides governance and/or fina


ses:
am NA pm
A14. How many weeks of the year is your program closed? (Enter 0 if the program is open 52 weeks a year. Please provide a copy of the program calendar with days off.)
Number of weeks closed:
6
Provider ID:
We would like to collect some


ours these staff work in a typical week (e.g., two part-time staff combined may work 45 hours per week in a typical week).
Yes / No
[select one response per position]
Yes No
Yes No Yes No Yes No Yes No Yes No Yes No
If yes, please note the:
       Nu


enue from each of the following public or private sources for your early childhood program. [Select only those that apply.]
    Public Sources
Yes No
Don’t know
    Program service fees paid by state Child Care Subsidies (i.e., OK Department of Human


t again here.
[You may record amounts by staff category using the rows under D1. If staff are shared across rooms/groups in your program (e.g., art, music, or physical education teachers), please prorate their wages/salary based on the share of their


 other administrative staff (e.g., director, associate director, and any other administrative staff)
b. Food preparation/service staff (e.g., cooks and other kitchen/meal service staff)
c. Transportation services staff (e.g., bus/van drivers)
d. Othe


ies
a. Curricula and related materials
b. Other learning materials and room/group supplies c. Other (specify:
D7. Staff education and training costs
a. Classroom staff
b. Leadership and other administrative staff
D8. Lost tuition or fees from staff d


al
Amount Used by Child Care
Square feet Percent Square feet Percent
    Interior square footage Exterior square footage
        b. Please describe the payment arrangements for building costs, including the total amount paid by the business and the t


ongoing training and support for a curriculum covering multiple domains of children’s development, assessing children’s progress, engaging with families, and overseeing fiscal accountability policies and procedures found in accreditation standards, s


her occupation equal to $36,510. We use 95 percent of the 25th percentile for a One Star lead teacher.
Table D.3. Assumed Occupational Salaries for Oklahoma Center and FCCH Cost Model Baseline by Reaching for the Stars Rating (2019 Dollars)
      One


0 percent of salary, respectively, for other fringe benefits, which might include a partial contribution toward health insurance premiums for full-time staff.
Other Unit Prices
Tables D.4 and D.5 display the assumed cost per unit of other cost compon


2019/
Davis, Elizabeth, Lynn A. Karoly, Bobbie Weber, Pia Caronongan, Kathryn Tout, Patti Banghart, Sara Shaw, and Anne Partika, Market Rate Surveys and Alternative Methods of Data Collection and Analysis to Inform Subsidy Payment Rates, Washington,


ouncil, and LaRue Allen and Bridget B. Kelly, eds., Transforming the Workforce for Children Birth Through Age 8: A Unifying Foundation, Washington, D.C.: National Academies Press, 2015. As of August 1, 2020: https://www.nap.edu/catalog/19401/transfor


gov/resource/trends-family-child-care-home-licensing- requirements-and-policies-2017-research-brief-2
141
National Center on Early Childhood Quality Assurance, Trends in Group Child Care Home Licensing Requirements and Policies for 2017, research bri


n Schmidt, and Tara Lynn Orlowski, 2017 Child Care Licensing Study, Minneapolis, Minn.: National Association for Regulatory Administration, 2017. As of August 1, 2020: https://nara.memberclicks.net/assets/docs/ChildCareLicensingStudies/2017CCStudy/NA


  CORPORATION
JORDAN R. FISCHBACH, MICHAEL T. WILSON, CRAIG A. BOND, AJAY K. KOCHHAR, DAVID CATT, DEVIN TIERNEY
Managing Heavy
Rainfall with Green
Infrastructure
An Evaluation in Pittsburgh’s Negley Run Watershed
  For more information on this public


le the main body of the report is more suited to technical readers.
This research was funded with grant support from the Henry L. Hillman Foundation, the Heinz Endowments, and 3 Rivers Wet Weather.
Community Health and Environmental Policy Program
RA


. . . . . . . . . . . . . . Recent Historical Negley Run Overflow Estimates, by Year. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Selected Depth-Duration-Frequency Estimates Used in Discrete Storm Analysis. . . . . . . . Runoff by


. . . . . . . . . 22
3.2. Summary of Recent Historical Negley Run Overflow Metrics for Different Periods. . . . 28
3.3. Summary of Recent Historical and Projected Future Negley Run Overflows . . . . . . . . . . . ..29
3.4. Flooding Return Period from


 as stormwater flooding and other negative out- comes can disproportionately affect low-income or majority-minority neighborhoods.
These challenges are especially difficult to overcome in complex and interconnected urban environments. In many cities,


County Sanitary Authority (ALCOSAN) is responsible for conveying and treat- ing wastewater from 83 municipalities in the county, including the city of Pittsburgh. But ALCOSAN functions as a regional wastewater utility governed by federal, state, and


e effectiveness of infiltration-based approaches. However, an alternate approach is to use GSI techniques as part of combined green-and-gray strategies that seek to capture and route rainfall to a centralized surface system—or daylighted
system, whic


oding given future rain- fall uncertainty. We then evaluate proposals for a phased series of large-scale GSI investments centered on a new daylighted system that would capture, store, and convey a large volume of rainfall to the Allegheny River. Spec


tional increment also includes the previously numbered increments so that the strategy builds “upward” from the bottom of the watershed into the surrounding neighborhoods. We also identified three options (A, B, and C) for different levels of investm


nge strat- egy, while Strategy 9C, which represents the highest level of investment considered, is the Max Build-Out strategy.
Study Results
This study builds closely on a prior pilot-scale effort focused on climate-resilient stormwater management ac


 a range of Recent Historical rainfall patterns compared with a single Typical Year (TY). We first evaluated sewer overflows in an FWOA (no additional investment) using the same TY approach adopted for regional planning
xviii Managing Heavy Rainfall


ategy under present and plausible future rainfall conditions.
Key Findings
A new centralized daylighted system and subsequent upstream investments could sub- stantially reduce, but not eliminate, sewer overflows in Negley Run. When we evaluated each


at build on a daylighted surface collection system could yield highly cost- effective sewer-overflow reduction across a range of assumptions. As modeled, the Cen- tral Daylighting strategy is highly cost-effective for overflow reduction. We estimate


ts in and of themselves under many plausible futures. The results suggest that the additional services provided by GSI strategies are economically desirable over a wide range of strategy and environmental assumptions.
The net economic value of the st


 in ALCOSAN’s Clean Water Plan.
Gather additional data to inform flood-risk assessment for Negley Run, includ- ing combined system descriptions and flow monitoring. This analysis was limited by the data available to inform model development and calib


peland (Pittsburgh Parks Con- servancy) but note that this effort would not have been possible without active and thoughtful engagement with the entire task force.
Mikaela Meyer (Carnegie Mellon University) helped to assemble this report, and RAND’s


ross the United States are struggling to effectively manage stormwater runoff from heavy rainfall (National Academies of Sciences and Medicine, 2019). This challenge has grown over time in response to several drivers, including impervious cover from


 related to regulatory compliance, flood risk and damage, water quality,
 1
2 Managing Heavy Rainfall with Green Infrastructure
ecosystem services, recreational opportunities, and economic development. It is also an envi- ronmental justice and equity


ntly expanded its efforts to more holistically invest in stormwater man- agement through new infrastructure investments and a proposed stormwater fee (Pittsburgh Water and Sewer Authority, 2020). PWSA’s investments are limited to its service area, ho


n Infrastructure
A key focus of public conversation surrounding the Pittsburgh region’s stormwater challenges since 2012 has been on green infrastructure. Green stormwater infrastructure (GSI) specifi- cally uses natural elements (vegetation, soils,


strategy. Additionally, the ALCOSAN “Starting at the Source” report explored areas in which gray infrastructure plans could be reduced in favor of an expansion of green infrastructure options. ALCOSAN subse- quently established the Green Revitalizat


ual visions for how it could implement large- scale systems of GSI and restore some amount of natural surface flows in these watersheds. It also conducted preliminary analysis of potential plan benefits but was not able to explicitly simulate how suc


f City Planning and Urban Redevelopment Authority of Pittsburgh, 2019; Pittsburgh Water and Sewer Authority, 2016).
Geography
Negley Run (Figure 1.2) is a large (3,300 acres) and urbanized watershed at the eastern end of the city of Pittsburgh, with


ata Center, 2017). Homewood, Larimer, and Lincoln-Lemington-Belmar also struggle with high rates of vacant or blighted properties, with over 20 percent of homes vacant in each neighborhood (Allegheny County/ City of Pittsburgh/Western PA Regional Dat


een First Plan concepts, for instance, in 2017 a local design collaborative developed the Negley Run Implementation Plan (envirosocialcapital, evolveEA, and eDesign Dynamics, 2017; hereafter NRIP). This plan envisions a new sepa- rated surface connec


escribes modeled strategy results for stormwater outcomes, such as reductions in sewer overflows or street flooding, while Chap- ter Six builds on this simulation modeling analysis to compare economic benefits, cobenefits, and costs from each strateg


cor- responding approach to cost estimation and summarize the methods applied to evaluate GSI cobenefits. The chapter concludes with a brief description of the Robust Decision Making (RDM) tools applied to support a comparison of GSI strategies under


ts provided by PWSA or obtained from other county geographic information system sources to add new spatial and modeling detail, such as dynamic estimates of groundwater and inflow and infiltration flows. Finally, they recalibrated the model using mon


en conducted additional validation steps to compare UA-NR model output to monitored flow data and prior Negley Run CSO estimates developed by ALCOSAN and PWSA. Additional detail on UA-NR model integration and validation is provided in Appendix B.
His


 per the methods applied in ALCOSAN’s WWP (ALCOSAN, 2019e). Annual overflow event durations and frequencies (as number of events) were then calculated using these thresholds for each strategy and uncer- tainty condition selected for simulation.
Note


house exceeded the home’s grade elevation. This conservative assumption maintains that water would only seep into the basement along the house foundation or from first floor flooding when flood waters exceed the house’s grade, although it is also po


en Infrastructure
discussed here but instead introduced in Chapter Four alongside a description of the resulting strategies. Additional methodological detail is also provided in Appendix E to this report.
Estimating Strategy Benefits and Cobenefits
A


emming from the mere existence of a good or service or the option to use it in the future) can be estimated using stated preference techniques.
• Benefit (or value) transfer methods use values from previous research in a new context to value nonmarke


e also seek to estimate the potential benefits from CSO reduction (water- quality improvement) directly in dollar terms, representing the social benefit that results from cleaner waterways. This allows us to sum water-quality benefits and other coben


an environments. Once this estimate is obtained and extrapolated to a calendar year, average values are multiplied by the physical quantities to convert the estimates to a monetary benefit. The National Cooperative Highway Research Program provides e


enity values.
Robust Decision Making
Finally, this study builds on the RDM approach previously described in the pilot study. RDM is a general analytical method intended to help identify policies or strategies that are robust against future uncertaint


ces a range of stormwater management challenges under present conditions, and as noted in Chapter One future climate change could exacerbate these chal- lenges. In this chapter, we explore recent rainfall trends and use simulation modeling to esti- m


• Storm variability by time and location
Model Relationships (R)
Updated Negley Run (UA-NR) SWMM model Downscaled climate rainfall projections Overflow analysis tools
Flood analysis tools
Policy Levers and Strategies (L)
FWOA (current system conditi


eshold, which is shown in the column header. Returning to the example, we expected approximately 8 24-hour events where at least 2.3 inches of rain fell, but over this period we observe 18 events meeting or exceeding this threshold. For the 5-year ev


—can increase surface stormwater runoff in the early stages of a heavy rainfall event (Rossman and Huber, 2016). To account for this, we evaluated discrete storms under two scenario conditions: (1) a default dry condition, where the assumption is tha


onnector
  2x 18” connectors and vortex gates
North Negley Run Blvd 66”
Overflow Weir elevation 725.02’
     Flow 48” split
Highland Drive 54”
East Washington Boulevard 96”
East Washington Boulevard 72”
   South Negley Run Blvd 85”
West Washington Bo


key modeling changes that might lead to these differences in Appendix B, but the most significant factors are likely to be the following:
• This model has been rebuilt and recalibrated for the Negley Run watershed and uses SWMM functionality not incl


oves some rainfall events to better match long-term historical statistics (see ALCOSAN, 2009, for more information).
2007
2011
Year
2003TY 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
Number of overflow events
74 80 90 1


As the wettest year on record as of the pub- lication of this report, 2018 remains an outlier but is nevertheless important to consider in a what-if analysis to understand how the current system and proposed strategies might perform under very extrem


rainfall represents an average across all 3 Rivers Wet Weather radar-adjusted pixels used in the UA-NR model simulation.
In general, these scenario comparisons reinforce similar conclusions as the pilot study. Looking at a 10- or 15-year average repr


Climate Assessment for precipitation across the Northeast (USGCRP, 2017), but here we see how this translates to a specific location in Pittsburgh. Note, however, that the range of uncertainty grows increasingly larger for the Average Future results


 conditions prior to the storm: with saturated soils, alternately, the peak values from a 1-hour, 10-year rainfall increase to 25 cfs (Historical rainfall scenario) or 39 cfs (Aver- age Future scenario), respectively.
Figure 3.8 compares the 1-hour p


e from Synthetic Rainfall Events
Figure 3.9 shows a sampling of street flooding maps from the discrete storm analysis using this approach for four events of varying duration and intensity (Historical rainfall scenario, dry starting conditions). Gray


feet. Results show the median Historical rainfall scenario estimate assuming default (dry) prior conditions. Rainfall volumes for each storm shown are as follows: 1-hr, 2-yr = 1.0 in; 1-hr; 10-yr = 1.5 in; 24-hr, 2-yr = 2.1 in; 24-hr, 10-yr = 3.2 in.


also evaluated a range of historical analogue storms that derive from the 3 Rivers Wet Weather historical radar-adjusted rainfall data set. The methods used to derive and adjust these storms to represent rainfall volumes from the DDF curves are desc


people who were trapped when floodwaters rose suddenly to a peak of over 8 feet around their idled cars (Balingit, 2013). Subsequent to this event, flood gates were installed at both ends of lower Washington Boulevard (from Negley Run Boulevard to A


quently homes might be affected by wet basements or sewer backups in Negley Run. This analysis does not include all homes in the watershed but is instead limited to the homes along the streets noted in the previous section.12 In total, we were able t


84 6 177 51 433 18
Total 2,367 100 Modeled
Homes
NOTE: Median rainfall estimates from the Historical rainfall scenario assuming default (dry) soil moisture, applying the synthetic rainfall approach. The sum of modeled homes in the first three columns


nsemble average (Average Future) to better illustrate the spread across climate models. Damage estimates com- bine all modes of basement flooding described above and range from approximate $450,000
15 Results mapped by structure are omitted from this


 year
44 Managing Heavy Rainfall with Green Infrastructure
Figure 3.13
Expected Annual Basement Damage for Modeled Houses from a 1- or 24-Hour Event, by Climate Scenario
Expected annual damage (thousands of 2019 constant $)
0
700
600
500
400
300
200


how Negley Run results would change with broader improvements to the regional system, including treatment plant expansion of the construction of deep tunnel interceptors as called for in the CWP. Future work could integrate the UA-NR model back into


ood-modeling capabilities were also limited by data available to inform model development, which constrained our estimates to only a subset of the watershed. A complete inventory and mapping of storm inlets, inlet invert elevations, and associated pi


 additional storage or detention performance associated with the conveyance infra- structure that may subsequently connect local interventions.
Our methods of preliminary analysis included plan and literature reviews, design-focused participatory wor


nt Venture) (Stantec / Tetra Tech JV, 2018)
• Negley Run Section 219 Environmental Infrastructure Pittsburgh, Pennsylvania Design Documentation Report: Draft 35 percent Submittal (hereafter DDR) from May 2019, in partnership with PWSA (prepared by Te


nd longitudinal sections found in Appendix G of CDAR to assess the area, length, and construction methodologies of key features. In addition, CDAR included detailed costing information with quantity takeoffs and unit prices that we used for our subse


y processes (for example, the Homewood Comprehensive Plan [Homewood Community Devel- opment Collaborative, Pittsburgh Department of City Planning and Urban Redevelopment Authority of Pittsburgh, 2019]). Due to the logistical complexity of land acquis


ed our strategy formulation. For example, the Meadow Street Microshed Concept Plan is a focused, community-based planning effort focused on a portion of the Larimer and East Liberty neighborhoods (Pittsburgh Parks Conservancy and Larimer Consensus Gr


son and remotely connected participatory workshops at our Oak- land office with NRWTF to assess important consensus concepts, facilitate strategy develop- ment, review preliminary results, and set the agenda for subsequent research efforts. These occ


 area along Paulson Avenue east of Washington Boulevard to the USACE design, which was designated Strategy 4. These areas within the lower Negley Run watershed are dia- grammed in Figure 4.6.
Silver Lake Retention Basin
One of the novel contributions


s not a dissimilar condition from that at Panther Hollow Lake (see Figure 4.7).
SLRB or SLRBX would have either a surface conveyance or subsurface piped infrastruc- ture connecting it down Washington Boulevard to the USACE Wetland Basin associated wi


 within the right-of-way improvements.
Strategies to Manage Stormwater in Negley Run 57
  2-acre water area
 10-yr 1-hr Overflow
72-hour Underdrain @2.8 cfs peak
2.5-acre Panther Hollow Lake for comparison SOURCE: (R) Image with Creative Commons Attr


Outfall
River
         Streets
Underdrain Underdrain
USACE Wetland Basin
Other impervious
Overflow
Overflow
Underdrain
Infiltration + evaporation
   Silver Lake “Rain Barrel”
   Stormwater overflow Combined drainage GSI underdrain Natural system los


7A, which includes a version of SLRB and investment in GSI to control 25 percent of impervious cover through the Kelly Street increment, is the Midrange strategy, while Strategy 9C, which represents the highest level of investment considered, is the


). Though we used the construction costs of individual GSI typologies of bioretention, tree pits, and so on for our own nominal estimates of neighborhood greening and conveyance units, we also bracketed the potential cost range using
Strategies to Ma


tage of construction cost by type of GSI, (2) percentage of construction cost by type of anticipated maintenance activity, and (3) cost per impervious acre treated.
The first method assumes annual maintenance costs are typically a percentage of con-


t the build- out for this system would take approximately 15 years. We used the USACE CDAR process to guide our anticipated timeline. After a five-year planning effort, CDAR projected that it would take five to six years to complete all concepts unde


ts were constrained to stay within available computing resources. We modified the UA-NR model to represent each of the 17 strategies described in Chapter Five and ran this model using both annual time series and discrete storms to consider performanc


rategies, 2003 Typical Year Rainfall
  Strategy name
FWOA Channel 1/2 Basins Swale/RSC Paulson Frankstown-SLRB-25 Frankstown-SLRB-50 Kedron-SLRB-25 Kedron-SLRB-50 Kelly-SLRB-25 Kelly-SLRB-50 Kelly-SLRBX-50 Lincoln-SLRB-25 Lincoln-SLRB-50 Lincoln-SLRB


rtion of CSO reduction in other rainfall scenarios considered. Figure 5.2
Figure 5.2
Combined Sewer-Overflow Volume Reduction with Strategies, Recent Historical Rainfall
  Strategy name
FWOA Channel 1/2 Basins Swale/RSC Paulson Frankstown-SLRB-25 Fra


ly average three projected future years (2043, 2045, and 2047). This subset of years was chosen to span the range of CSO volumes observed in the FWOA analysis, with a similar overall average. For more information, see Appendix B.
                  Gr


rategies with SLRB and more substantial invest- ments in local GSI, starting with the Midrange strategy (Strategy 7A), show a jump in perfor- mance relative to strategies focused primarily on the USACE CDAR region. Overall, the mod- eled results sugg


e discrete storm modeling results show that the system represented in the USACE CDAR report could reduce flood depths along the Washington Boulevard corridor during major rainstorms. Figure 5.6 shows an example of flood depth reduction from a 24-hour


Median 24-hour, 10-year rainfall volume is 3.2 inches from the Cook, McGinnis, and Samaras, 2020, historical DDF estimate. Figure shows change in 1D flood depth in the modeled street channels assuming default (dry) soil moisture conditions and using


focus on Strategies 1 through 4, representing the extent of the direct capture region identified by PWSA for the USACE CDAR effort. The UA-NR discrete storm modeling showed essentially no additional benefit in terms of street flood mitigation in the


hapter showed a mixed picture of benefits related to sewer overflows and flooding. CSO reduction benefits are notable and con- sistent across the range of strategies and rainfall scenarios considered and compare favorably to past estimates from PWSA’


 optimized for specific local site conditions.
As a result, additional research will be needed to follow up on these results and determine whether specific project designs could provide localized risk reduction. Because the initial high-level finding


nmotorized transportation infrastructure, benefits from the planting of trees (including urban heat island effects, carbon sequestration, and localized air pollution effects), and amenity values accru- ing to property owners as a result of increased


Chapter Two and Appendix F. The primary driver of these benefits are the esti- mates of (new) cycling and pedestrian behavior induced by proximity to a new amenity—in this case, a multipurpose recreational trail. Figure 6.1 shows single-family house


n in Table 6.3, tree planting results in reduced air pollution, carbon sequestration, and reduced energy use (urban heat island) as a result of biological processes and canopy shade, especially with respect to “green streets.” Typically, these benefi


Cumulative Approximate Air Pollution, Carbon, and Heat Island Effects from New Net Tree Plantings by Strategy Increment
($ Thousands per Year)
   Increment Strategy Net Total Trees
1 Channel 1/2 60
2 Basins 90
3 Swale/RSC 90
4 Paulson 140
5 Franksto


06
NOTE: Assumes properties affected are in a
with scenario. The 1 percent and 2.5 percent capitalized values are converted to annual values using a 3 percent discount rate. Average home value is assumed to be $134,071. Results are identical for all


eduction capital cost-effectiveness ($/gal.) Discounted economic benefit (2019$) Discounted life-cycle strategy cost (2019$)
Net present value (NPV) (2019$)
        Because the hydrologic results presented in the preceding chapter did not reveal any


 for regionwide CSO mitigation as a proxy for water-quality benefits in Negley Run (cost-avoidance approach; see Chapter Two).
To implement this approach, we used the average cost-effectiveness ($/gal.) from the updated ALCOSAN Clean Water Plan. The


rison Results
Combined Sewer-Overflow Reduction Cost-Effectiveness
We first compare results in terms of the cost-effectiveness of these investments toward the legally mandated overflow reduction goals. Figure 6.3 summarizes the cost-effectiveness of


ng to lower average annual rainfall and high CDAR capital costs) are still at or below $0.15/gal. This provides notable evidence to support PWSA proceeding with additional design increments for the lower portion of the watershed as part of a CSO comp


8A Lincoln-SLRB-25
8B Lincoln-SLRB-50
8C Lincoln-SLRBX-50
9A EastHills-SLRB-25
9B EastHills-SLRB-50
9C EastHills-SLRBX-50
(100)
(9)
11
37
40
40
37
40
37
39 NPV ($)
32
27
32
30
24
29
27
21 51
(50)
0
50
100
150
Benefit and cost (millions of discounted


could be undervalued. If these
Figure 6.5
Discounted Benefits and Costs by Strategy, 2013 Rainfall and Nominal Assumptions
Costs ($)
Benefits ($)
     (28)
(30) (21)
(33) (21) (36) (23) (41) (23)
(49) (27) (53) (29) (52) (24) (54) (28) (58) (31)
(56)


the scenario ensemble. NPV regret starts relatively high at Strategy 1,
8 More specifically, in each uncertain scenario, the best performing strategy among those compared is assigned a regret value of zero, and regret for each remaining strategy is m


further investments in local GSI in Homewood. Per Figure 6.3, there might be concern that these additional increments would yield comparatively
 Green Stormwater Infrastructure Economic Benefits and Costs 91
poor CSO cost-effectiveness, even if their


effectiveness. Alternately, if average annual rainfall in the near future more closely resembles the 2003‒2017 Recent Historical (41 inches) or Higher Total Rainfall (43 inches) estimates, the Max Build-Out strategy does not lead to higher than expec


trategies considered here are generally associated with at least competitive (and, at most, desirable) economic outcomes over a fairly wide range of future environmental conditions (rainfall) and economic estimates (estimates of the benefits and cost


while the uncertainty analysis incorporated variation in the key economic param- eters related to the overall level of annualized benefits as well as environmental conditions, the specific timing of costs and benefits, as well as the discount rate, w


 residents) were not included in the analysis. To the extent that these types of outcomes would be associated with adoption of a strategy, the costs of said strategy would be understated.
• Distributional aspects of the benefits and costs over the af


es for which NPV is negative. Coupled with the fact that the cobenefits are likely undervalued in the analy- sis, these results suggest that the additional services provided by GSI strategies are likely not only economically desirable for an assumed


nd assuming no additional investments in the combined sewer system. This estimate is in line with but some- what higher than prior past estimates. Our estimates are approximately 14 percent higher (986 MGY) when instead averaging model results from a


s effort provided some value in considering potential risks from heavy rainfall in terms of street or basement flooding. The relatively low computational cost allowed us to simulate thousands of different events and com- pare across several key dimen


presented in this analysis in a simplified, standardized way using the SWMM LID module in the UA-NR model and is not optimized for specific local site conditions. As a result, additional research will be needed to follow up on these results and deter


ars to be a robust and low-regret approach across a wide range of cost and rain- fall assumptions. When considering NPV and taking into account a broader range of coben- efits for local residents in Homewood and surrounding neighborhoods, however, st


e projects could be designed and sited to best address local flooding issues.
Continue to invest in distributed local GSI projects in Negley Run neighborhoods.
The cobenefits analysis presented here shows that GSI could provide notable benefits for r


101
102 Managing Heavy Rainfall with Green Infrastructure
drainage) modeling for key locations of interest, such as the Washington Boulevard corridor. In addition, coupled modeling building on this work could provide localized flood assessments at th


tary Authority, ALCOSAN Clean Water Plan, 2019e. As of September 29, 2020:
h t t p s : // w w w. a l c o s a n . o r g /d o c s /d e f a u l t- s o u r c e /c l e a n - w a t e r- p l a n - d o c u m e n t s /c w p - s e c t i o n - 4 . p d f ? s f v


 Computer-Assisted Approach to Scenario Discovery,” Technological Forecasting and Social Change, Vol. 77, No. 1, January 2010, pp. 34‒49.
Bureau of Reclamation, “Change in Discount Rate for Water Resources Planning,” Washington, D.C., December 17, 20


water Governance: The Need for a Paradigm Shift,” Environmental Management, Vol. 57, No. 5, 2016, pp. 1112‒1124.
Donat, Markus G., Andrew L. Lowry, Lisa V. Alexander, Paul A. O’Gorman, and Nicola Maher, “More Extreme Precipitation in the World’s Dry


ic Contribution to Global Occurrence of Heavy-Precipitation and High-Temperature Extremes,” Nature Climate Change, Vol. 5, No. 6, June 2015, pp. 560‒564.
Ford, Ariam, Rebecca Mizikar, Tom Mulholland, Masoud Sayles, Rachel Bowers, and Lydia Kramer, Gr


 An Analysis of National Bicycle and Pedestrian Documentation—Project Data from 2008 to 2010, Portland, Or., 2011.
Irwin, Nicholas B., H. Allen Klaiber, and Elena G. Irwin, “Do Stormwater Basins Generate Co-Benefits? Evidence from Baltimore County, M


heory to Practice, Dordrecht, Netherlands: Springer, 2019.
McGarity, Arthur, Fengwei Hung, Christina Rosan, Benjamin Hobbs, Megan Heckert, and Shandor Szalay, “Quantifying Benefits of Green Stormwater Infrastructure in Philadelphia,” World Environmen


ata Center: Pittsburgh ASOS, PA US Maximum Temperature,” 2020c. As of July 1, 2020: https://www.ncdc.noaa.gov/cdo-web/datasets/GHCND/stations/GHCND:USW00094823/detail
———, “National Climatic Data Center: Pittsburgh ASOS, PA US Minimum Temperature,” 2


oding on Road Transport: A Depth-Disruption Function,” Transportation Research Part D: Transport and Environment, Vol. 55, 2017, pp. 67‒81.
Price, Elizabeth, Taylor Holladay, and Lisa Wainger, Cost Analysis of Stormwater and Agricultural Practices fo


ation of Green Infrastructure in Buffalo: Rain Check 1.0, Buffalo, N.Y., 2018.
University Center for Social and Urban Research at University of Pittsburgh, “City of Pittsburgh Neighborhood Profiles Census 2010 Summary File 1 (SF1) Data,” 2011. As of


of Green Infrastructure,” StormwateReport, December 2, 2015. As of July 1, 2020:
https://stormwater.wef.org/2015/12/real-cost-green-infrastructure/
Western Pennsylvania Regional Data Center, “Allegheny County Property Assessments,” web page, undated.


timates of recreational, amenity, and other cobenefits to local residents; compare total benefits to costs; and explore potential trade-offs. Results show that a centralized system of stormwater management in Negley Run could yield cost-effective sew


     CORPORATION
MARJORY S. BLUMENTHAL, LAURA FRAADE-BLANAR, RYAN BEST, J. LUKE IRWIN
Safe Enough
Approaches to Assessing Acceptable Safety for Automated Vehicles
                                                           For more information on this


am
RAND Social and Economic Well-Being is a division of the RAND Corporation that seeks to actively improve the health and social and economic well-being of populations and communities throughout the world. This research was conducted in the Communit


re attention is being paid to processes—the kinds of steps taken by developers and how these steps are implemented—and what they indicate about AV safety. These processes include safety-relevant standards-setting activities, a growing emphasis on saf


ht serve as a transportation-system-wide goal as opposed to an AV-specific one. Thresholds can evolve.
  Approach
Measurement Leading measures Roadmanship Lagging measures
Process
Technical standards Government regulation Safety culture
Thresholds
Pr


f everyday products and experiences and weighs these risks. Communication is also important because, as with other technologies associated with significant risk, AV safety is a political issue and a technical one.
Mapping Agreement Among Stakeholders


onsortium could help, including by becoming a champion for advancing roadmanship measures.
Developers should forge uniform and transparent approaches to presenting evidence for meeting safety thresholds. There are differences across the sets of circu


ne Lappin, legal scholar Bryant Walker Smith, and a European technologist who preferred to remain anonymous.
As part of RAND’s quality assurance process, Liisa Ecola and Jane Lappin provided constructive feedback that helped us sharpen our analysis a


nventional vehicle production), industry analysts, safety and consumer advocacy organizations, government officials (federal, state, and local), safety researchers, and the general public (people expected to ride in or share the road with AVs). This


ow well an AV behaves in traffic.
As suggested by its title, Measuring Automated Vehicle Safety: Forging a Framework describes a framework that shows measurement possible in different settings (simulation, closed courses, and public roads with and wi


ior. As one person mused to the research team before everyday activities were constrained because of the pandemic:
What kinds of risk do we accept as we walk out the door, go down the street, get on a city bus, go into a store—what is the existing am


 adopt new technologies, such as seat belts, side airbags, and crumple zones—sometimes in response to regulatory standards and/or public pressure and sometimes voluntarily.
For example, Vision Zero is a global movement aimed at eliminating traffic fa


 and could feature fleet ownership. 4
 Perhaps the most important context is the environment (defined by geography, terrain, weather, lighting, traffic, roadway complexity, and more) in which the AV is tested and for which it is designed: the ODD. In


ard provides a taxonomy of automated driving capabilities, ranging from level 0 (no driving automation) to level 5 (fully automated driving with no human needed).
14 There are more than 250 million registered vehicles in the United States, a fleet wi


urces of evidence for AV safety using the RAND American Life Panel (ALP), a nationally representative sample of the U.S. population. The survey provided insight into how the general public weights different evidence types and sources relating to AV s


n Table 2.1, we present different ways to categorize AV safety approaches: safety as a measurement, safety as a process, and safety as a threshold.
 Table 2.1. Approaches for Assessing AV Safety
   Safety as a Measurement (Chapter 3)
Safety as a Thre


id and feasible. The former means that the approach must actually reflect safety (rather than, for example, driving function, public relations, or technology development). The latter means that it must be possible to generate evidence to support (or


describing the knowledge that safety measurements can yield.
How Measures Show Safety: Lagging and Leading Measures
Our 2018 report, Measuring Automated Vehicle Safety: Forging a Framework, articulated two streams of measurement: leading and lagging


this number” and assess if what happened in the past will continue to happen in the future.
Another said:
On the lagging side, what you’d like to see is a reduction of injurious and fatal crashes, and that will tell the final story on if it is safe o


o provide the public with critical analysis about how something is safe.
—————
The problem with disengagements: There isn’t enough context around them. You can’t be sure that a disengagement for [company A] is the same as for [company B], or even for


n aspirational leading measure for AV safety.
The original articulation of roadmanship focused on safe roadway citizenship. During the research for this project, stakeholder comments deepened the discussion of roadmanship:
When we’re talking about ro


ed to avoid a potentially dangerous situation, resulting in a leading-measure event occurring.
Table 3.1. Relationship Between Hard Braking and Appropriate ADS Action
    Danger Present
    A: appropriate reaction
   C: false negative (crash)
   Brak


aspects of roadmanship, nor does any measure cover all possible threats to safety that occur on the roadway. No one measure can be used in isolation; each measure’s contribution is best voiced in harmony. As one interviewee remarked:
Some of the conv


envelope (or through some other means) can generate rate-based measures, such as counts per intervention per VMT. It could also contribute to implementing other measures articulated in Table 3.2.32
Additionally, roadmanship measures are conceived of


racteristic
    Measures of Behavior
     Measures of Time
    Measures of Safeguard Engagement
    Measures of Probability
    Measures of User Perception
   Example
  Rapid acceleration or decelerationa
  Safety critical eventsb (combination of cra


were manually validated and coded by trained data reductionists." Johan Engström, Andrew Miller, Wenyan Huang, Susan Soccolich, Sahar Ghanipoor Machiani, Arash Jahangiri, Felix Dreger, and Joost de Winter, Behavior-Based Predictive Safety Analytics


eading measures do play a role, but we need to do more research to find out what is the most correlated with the lagging measures. This is hard to do.
———
AVs have different behaviors and different failure modes than human drivers.
———
Do you design


erpinnings hold. It follows that the results of leading and lagging measures should agree—or, as one interviewee put it, “it’s reasonable that fewer fatalities would also mean fewer injuries.” According to another:
Even though you’re meeting leading-


ntly, Safety II reflects the AV’s ability to adapt and respond to conditions, and Safety II shifts goals from the specific “don’t crash” to the broader “drive safely” and from “avoid that something goes wrong” to “ensure that everything goes right.”3


 after the update, which could result in degraded safety even if the update itself improves AV performance.
One option could be a measure for which events occur so frequently that a day’s worth of data would be informative, but such a measure would l


e last year and a half. At this point—it would be great if there were consensus, but, in its absence, we would entertain different methods.
However, this ad hoc approach muddies any effort to understand safety across the industry overall and the safe


nd other stakeholders point to processes or specific sets of steps that might be used by developers to assess safety. Processes can point to what some engineers call a positive trust balance. As explained in one interview:
The positive trust argument


raged or even required by government and by corporate policy. Options to demonstrate AV safety through process have been growing. As discussed in a later section, demonstrating compliance with a process is a way for developers to document the steps t


) who find common cause in establishing parameters for performance and design and for mechanisms that allow interoperability or consistency in training. One interviewee explained the need as follows:
[As a developer,] I should show what math I used a


ces in preferences for how prescriptive a standard should be—also adds delay.
Since Measuring Automated Vehicle Safety: Forging a Framework was published,50 the number and variety of AV standards have grown (Appendix C lists notable examples), as hav


y for Automated Driving Systems—Design, Verification and Validation Methods,” ISO/CD TR 4804, undated.
55 Underwriters Laboratories, “About Us,” webpage, undated-a.
56 Junko Yoshida, “Safe Autonomy: UL 4600 and How It Grew,” EE Times, April 2, 2020;


decisionmaking, industry will not know how safe is safe enough, and government will not have a tool to define what safe driving means. . . . This standard defines a technology-neutral formal model, parameterized so that the balance between safety an


e traffic and enforcement of traffic laws. The local and state levels are where one can find commitments to Vision Zero, focusing on the traffic system and not just individual vehicles (as noted in Chapter 1). At the state level, governments control


er if that happens through a sort of evolutionary process.
Even before the advent of the AVs now being tested, the development of FMVSS has been slow; the timetable built into the U.S. rulemaking process was noted by a variety of stakeholders.69 The


ng a floor for safety. Industry executives asserted that best practice will be at least comparable to that floor. For the public, a minimum expectation can provide assurance. This floor also limits the extent to which government officials need to con


uently pointed to a developer’s safety culture as an indicator of how it approaches safety-critical processes. Safety culture might be particularly helpful when specific processes break down or become disconnected from measures (see Chapter 3).
AV de


al standards, which could be bootstrapped by the best practices that AVSC develops.73
Insights can be gleaned from other domains. For example, the Federal Aviation Administration established the Safety Management System, a set of processes for managi


dge of safety, including how it is handled in a variety of industries, noted that, in transportation industries with comparatively few players (such as aviation and rail), safety culture extended to a willingness to share information about problems.7


 These thresholds exist along several dimensions.
79 These issues are discussed in later sections of this chapter. See “Safety as Achieving a Threshold Based on Human Driving Performance” and “Safety as Achieving a Threshold Based on an Absolute Safe


en they first enter commercial service but also in an ongoing manner. The former has been traditional for human-driven vehicles,83 but the latter is important for AVs given the possibility for ongoing over-the-air updates.84 With AVs, explained one i


ited by interviewees as a basis for a safety threshold of AVs (for example, making “comparisons to human drivers is necessary, at least in the short term”; “comparing to the baseline of a human driver is a good idea”; and an “AV needs to have the sam


han an average driver. Oddly, simple analysis shows that they are right. In terms of any crash, from fatalities to solely property
87 Fewtrell and Bartram, 2001.
88 Wishart et al., 2020. This option is somewhat predicated on the assumption that, as o


es were four-vehicle crashes, only 25,808,000 drivers (less than 12 percent of drivers) would have been involved in a crash.
94 The “trick” is in comparing mean with median and a population decimal rate with an individual’s whole number.
95 Insurance


”
A few definitions of safe drivers exist. We list these here with quotations from interviewees:
• Pick a percentage or numerical value better than average humans: “10, 20, 30 percent,” “a degree or order of magnitude better than humans,” “95 percent


 comparing of methods, instead of just looking at safety. We need to have a common definition and common methods.
Clear, transparent, and universally applied definitions and methods are required, especially for leading roadmanship measures (see Chapt


ar misses and other measures of roadmanship is naturalistic data on human-driven vehicles.101 Naturalistic driving data can provide otherwise-inaccessible insights into human factors specific to driver behavior, driver characteristics, pre-crash driv


re deeply and inextricably linked to ODDs. Regional differences exist in leading and lagging measures. Comparing on the state level, Mississippi had 23.12 crash fatalities per 100,000 people in 2017; New York had a rate of 5.03. But this rate is not


son (average, safe, or other) could be used. Consequently, because AVs develop in “islands of autonomy”—geographically specific and narrowly defined locales— they also might be most measurable in these select islands.
Safety as Achieving a Threshold


est might not be able to show performance to an acceptably high level.
Finally, validity is questionable. The degree to which passing or failing a driving test reflects overall driving ability is unclear. Although a human-driving test reflects abilit


as it can possibly be could involve evidence that the AV meets all existing best practices, including a variety of technical standards and other best-practice protocols (see Chapter 4). As one interviewee stated, “with regards to outcome, [I] don’t c


requirement of zero anything is conceptually fraught and might be practically infeasible.110 Accordingly, interviews with people concerned about the interpretation of a zero- death goal featured such comments as:
The absolute threshold should be zero


roads are right next to a sidewalk, and you may be going 40 to 45 mph. You assume those pedestrians will not enter the roadway, etc. The parameters for safe driving are making assumptions about these behaviors. For example, when following, you assum


r words, the new technology or system must include the same risk as the old technology or system.115 The theory is that one cannot get below this level. A hybrid between a threshold rooted in the average human driver and one rooted in the absolute go


ble fatal crash risk is a risk less than 1/20 of the above (or 3 x 10–6 person-years). Pegasus, 2019.
 51
enough? Several interviewees touched on the politics of the problem, as illustrated by the following quotation:
Industry hype and marketing say


erformance,” one of the main challenges of using leading events is that comparable human-driver data are rarely available. Using leading events in concert with absolute safety goals instead of thresholds based on human-driving behavior nullifies this


use it.
Definitions, by contrast, might benefit from uniformity: “To a certain extent, all need to be speaking the same language. And the language is important to not misrepresent to people what the vehicles do.” Because companies will have such dive


on what is doable given the sophistication of available technology, economic pressures, and what the public will accept.a ALARP, especially, tries to elucidate a level of risk between unacceptably high and acceptably low.b The natural corollary is th


nature, somewhere in between the two
• understand that some uncertainties must be accepted
• expect that resources will go to addressing the highest risks.
With this list in mind, phrasing an approach along the same lines as ARARA and ALARP might hav


lished and/or endorsed by
safety culture OR
Safety as achieving a threshold
Safety as achieving a threshold predicated on human driving
Safety as achieving a threshold predicated on technology
→ As safe as a safety culture can confirm OR
As safe as a


04.
125 Paul Slovic, “Perceptions of Risk: Reflections on the Psychometric Paradigm,” in Sheldon Krimsky and Dominic Golding, eds., Social Theories of Risk, New York: Praeger, 1990, p. 121.
  57
Psychometric Paradigm of Risk
The question of “how safe


.
128 Baruch Fischhoff, Paul Slovic, Sarah Lichtenstein, Stephen Read, and Barbara Combs, “How Safe Is Safe Enough? A Psychometric Study of Attitudes Towards Technological Risks and Benefits,” Policy Sciences, Vol. 9, No. 2, 1978.
 58
with an activit


ty—will all play large roles in the campaign to communicate to the public about AV safety. Framing messages in positive or negative language independently
129 For example, see Slovic et al., 2004.
130 Melissa L. Finucane, Ali Alhakami, Paul Slovic, a


commented on amusement parks: “If you go to an amusement park and get on a roller coaster, you assume that someone has checked that it is safe enough.” In the case of aviation and amusement parks, it is possible that the business design (e.g., paying


ce in the promotion of technology acceptance and adoption in areas in which negative outcomes, however rare, have the potential for severe harm. Acceptance of technology can be negatively affected by a negative reputation (of the technology or its pr


on of compatibility (as users develop a greater understanding of the abilities of the system).
Communication from the Automated Technology Itself
Communication of the automated capability of vehicles is an important way to address perceptions of risk


without specific training on the appropriate use or capabilities of the product or system. A recent survey found that only about one-half of new car buyers were offered system-specific training for their new vehicles by the dealership.144 In the same


ng conventions for 20 ADAS in five different categories.147
Convincing the Public
Sources of Communication—ALP Survey
Communication about the safety of AVs comes from a multitude of sources with varying public perceptions of trust and credibility. Ty


ormation is available from AV near-miss rates. This topic is further discussed in Appendix A.
To calculate the relative influence of each source, the safety messages and resulting perceived safety ratings were entered into a linear mixed model. Using


attern of similarities and differences between rank order and the regression coefficients from the implicit items (see Table 6.1). On average, participants were aware of which sources they most and least preferred. The implicit and explicit measures


Differing Sources for Messages About AV Safety
  Order of Preference
Safety Message Source (measure)
  Regression Coefficient (implicit)
Rank Order (explicit)
   AV crash rate
State or local government position Federal government position
AV near-mis


promote AV acceptance in light of the constructs and models discussed in this chapter. Early adopters will likely perceive greater benefits and lower risks associated with AVs than will individuals who are less receptive to new technologies. One surv


general public, industry stakeholders must be able to communicate their measure- and process-based safety approaches to government in ways that are clear and consolidated. According to Bryant Walker Smith:
The public and even regulators lack the reso


humans as opposed to errors from technology. Though the magnitude varies depending on the particulars of any given study on the topic, the general trend is clear: Individuals will not accept AVs until they are safer than human drivers. This expectat


of rare events, such as crashes.
158 Silberg, 2017.
  71
Table 7.1. Approaches to Assessing AV Safety Approach Takeaways
  Approach
Measurement • Leading measures
Roadmanship
Lagging measures
Measurement provides established and easily communicated e


t in company statements. Regardless of the source, AV communication should factor in the difficulty that people have in gauging risk accurately.
Progress will not come from selecting the best approach—there is no such thing. None of these approaches


practice. It is also, in effect, an individual’s recipe for a safety case (which we discuss later).
How Approaches Build on Each Other
Each of the approaches outlined in Table 7.1 builds on, supports, and contributes to the evidence used in others. I


andards—but it shouldn’t be part of the ultimate measurement of whether AVs are safe or not safe.
———
The reality is that, without metrics, where is the oversight to make sure people meet those metrics? Cars are self-certifying to what? A system that


4600) to the recognition that such broad approaches as the Federal Aviation Administration’s Safety Management System, which centers on safety cases, have value in AVs. Although safety cases began as a way of organizing and documenting engineering an


admanship even if that term was not used—the more general term, “leading measures,” effectively refers to roadmanship measures (due partially to a dearth of other options). Measurement uncertainties and the many pointers toward the need for an integr


e issue. That is why communication (discussed in the next section) is central to the question of whether AVs are acceptably safe. Developers and the AV industry depend on either facilitation or removal of barriers by government at different levels, a


that reality has been a motivation for regulation generally, regulators inevitably lag behind industry when it comes to understanding how the relevant science or technology works, another reality that was mentioned by a wide variety of stakeholders


d for sharing information associated with adverse incidents as case studies.171 In the absence of such sharing, National Transportation Safety Board investigations appear to play that role. Stakeholders continue to differ, as documented in the team’s


2 NHTSA, “New Test Tracking Tool,” webpage, undated-a.
 80
platform for data contributed voluntarily by developers and by states and localities that host testing. Stakeholders interviewed for the project struggled with the idea of a third-party revie


. Anderson, Nahom M. Beyene, Pavan Katkar, and Gregory Cyril Baumann, When Autonomous Vehicles Are Hacked, Who Is Liable? Santa Monica, Calif.: RAND Corporation, RR- 2654-RC, 2019.
174 Viscusi, 2006.
 81
improved. What is communicated—and how—will sh


abel Consumer Research,” Federal Register, Vol. 85, No. 23598, April 28, 2020b.
   Administration’s New Car Assessment Program (NCAP) created the 5-Star Safety Ratings Program to provide
 consumers with information about the crash protection and roll


e to respond better to new challenges after encountering a difficult situation than it was able to before.180 Associated assessment challenges might dwarf those of today, but meeting
177 Viscusi, 2006.
178 A survey by Partners for Automated Vehicle E


evelopers should collaborate with state and local leaders to bring their vehicles into communities around the country. This collaboration could include demonstrations of how AVs operate, things they can do that might improve on human perception and r


as administered online to ALP participants. The raw data will be made available online after an embargo period of one year.
Participants
Data were collected from 2,203 participants (ages 23–90 years, mean age = 58.42, standard deviation age = 13.87;


pt these as separate groups because these sources of information might not always agree with one another, and we wanted to allow for that information pattern in the structure of the survey (as we will describe). For example, in states where AV testin


b up indicates that the source mostly shows that AVs are safe; one red thumb down indicates that the source mostly shows that AVs are unsafe􏱋 and two red thumbs down indicates that the source strongly shows that AVs are unsafe.
In addition to the imp


ized coefficient; b = unstandardized coefficient; t = t-value; p = p-value.
0.25 4.46 0.22 3.99 0.16 2.99 0.16 2.98 0.10 1.84 0.06 1.08 0.05 1.02 0.03 0.55
As shown in Table A.1, although safety messages from all sources were positively related to pe


ge was associated with lower perceived safety of AVs, and males rated the perceived safety of AVs higher than females did.
Interestingly, the rank ordering of preference for message sources differed compared with the ordering provided by the standard


                                                                     Table A.2. Means and Standard Deviations of Perceived Safety, by Survey Item
   Safety Message Sources Ratings Number CR NM FVR FG SLG AV ADV FF M SD
Situation


state or local government official position. Positive and negative messages from these two general source categories were loosely contrasted in items 5 and 15 (Table A.2). These items were compared in a separate repeated- measure analysis of varianc


they had published that could be cited. Research team members took notes during the otherwise unrecorded conversations, and the multiple sets of notes made the records more complete.
Stakeholder Sample
Stratified expert (purposive) sampling was used,


 that enriched the discussion. In broad strokes, the set can be described as including AV technology developers (10–13), safety researchers and advocates (4– 8), AV industry and technology analysts (3–6), and government entities (7–8).
Interview and


gs, at one point extending to seven sublevels. Because the interviews were semistructured, the goal of interview coding was to organize the notes into a searchable format rather than for quantitative analysis.
 187 Dedoose Version 8.3.17, web applica


ign,” in Handbook of Reliability, Availability, Maintainability and Safety in Engineering Design, London: Springer, 2009.
191 Terje Aven and Vidar Kristensen, “Perspectives on Risk: Review and Discussion of the Basis for Establishing a Unified and Ho


9.
195 William D. Schulze, “Ethics, Economics and the Value of Safety,” in Societal Risk Assessment, Boston, Mass.:
Springer, 1980.
196 Lisa A. Robinson and James K. Hammitt, “Behavioral Economics and Regulatory Analysis,” Risk Analysis: An
Internati


odel (TAM), the Theory of Planned Behavior (TPB), and the Unified Theory of Acceptance and Use of Technology (UTAUT). Using a survey approach,
99
they found that each of these models individually explained 71 percent or more of the variability in beh


nsion of the UTAUT model, introducing the additional factors of hedonic motivation, price value, and habit as predictors of behavioral intention and, for the habit factor, technology use.
Perceptions of AVs
Among studies of technology attitudes, surv


 May 2018
Using an online survey and interviews, the authors compared the likelihood of acceptance and use of self-driving vehicles among individuals who use ride-hailing services (such as Uber and Lyft) with those who do not use such services. The a


 Iss. C, 2019
Researchers sought to identify age differences in AV acceptability from using a sample of residents of Tianjin, China. They found that younger respondents had more-positive attitudes and acceptance of AV compared with older respondents


o the more stable trait of personality, while safety climate is akin to the more transient state of mood.199 Because of its quantitative nature and focus on measurement, the state of research on safety climate is more developed than that of safety cu


9
This article provides a meta-analytic review of predictive capabilities of safety climate measures. Universal safety climate measures have more predictive power for non–injury- or non–accident-related adverse events, although industry-specific safe


 a systems failure.
ISO, “Road Vehicles—Functional Safety—Part 1: Vocabulary,” ISO 26262-1:2011, 2011
This standard focuses on the functional safety of electrical and electronic systems, particularly in the case of their malfunction or combination wi


and severity of adverse events. It clarifies regulations for aviation designed to balance risk and the severity of negative outcomes.
Antonius A. Lambregts, “Flight Envelope Protection for Automatic and Augmented Manual Control,” in Proceedings of th


“Consumer’s Decision-Making and Risk Perceptions Regarding Foods Produced with Biotechnology,” Journal of Consumer Policy, Vol. 21, 1998
This study showed that acceptability of genetically modified goods is influenced both by the magnitude and charac


aries/content/assets/cm/content/miscellaneous/adas-nomenclature.pdf
American National Standards Institute, “Project Initiation Notification System (PINS),” ANSI Standards Action, Vol. 51, No. 27, July 30, 2020. As of September 10, 2020: https://share


nd Gil Tal, “Uncovering Early Adopter’s Perceptions and Purchase Intentions of Automated Vehicles: Insights from Early Adopters of Electric Vehicles in California,” Transportation Research Part F: Traffic Psychology and Behaviour, Vol. 60, January 20


orvehiclesafety/teen_drivers/index.html
Chandraratna, Susantha, Nick Stamatiadis, and Arnold Stromberg, “Crash Involvement of Drivers with Multiple Crashes,” Accident Analysis & Prevention, Vol. 38, No. 3, June 2006, pp. 532–541. As of August 30, 202


–1427.
Dedoose Version 8.3.17, web application for managing, analyzing, and presenting qualitative and mixed-method research data, Los Angeles, Calif.: SocioCultural Research Consultants, 2020. As of June 23, 2020:
https://app.dedoose.com/App/?Versio


nents/
———, “Safety Management System (SMS),” webpage, June 21, 2019. As of August 31, 2020: https://www.faa.gov/about/initiatives/sms/
Federal Highway Administration, Surrogate Safety Assessment Model and Validation: Final Report, McLean, Va.: U.S.


 Safety Science, Vol. 46, 2008, pp. 1047–1066.
Hartwich, Franziska, Claudia Witzlack, Matthias Beggiato, and Josef F. Krems, “The First Impression Counts—A Combined Driving Simulator and Test Track Study on the Development of Trust and Acceptance of


f Surrogate Safety Indicators for Vulnerable Road Users: A Review of Surrogate Safety Indicators,” Transportation Reviews, Vol. 38, No. 5, 2018.
Junietz, Philipp Matthias, Microscopic and Macroscopic Risk Metrics for the Safety Validation of Automate


fety of Self-Driving Vehicles,” Reliability Engineering & System Safety, Vol. 185, Iss. C, 2019, pp. 341–347.
Lund, Frederick Hansen, “The Psychology of Belief: A Study of Its Emotional and Volitional Determinants,” Journal of Abnormal and Social Psy


al Governors Association, 2018. As of August 30, 2020: https://www.nga.org/wp-content/uploads/2019/09/Autonomous-Vehicle-Technology.pdf
National Highway Traffic Safety Administration, “New Test Tracking Tool,” webpage, undated- a. As of August 20, 20


ysical-systems
National Research Council, Software for Dependable Systems: Sufficient Evidence? Washington, D.C.: National Academies Press, 2007.
National Safety Council, “Happy Anniversary, Road to Zero!” Safety First, blog post, October 23, 2017. A


AND American Life Panel: Technical Description, Santa Monica, Calif.: RAND Corporation, RR-1651, 2017. As of September 20, 2020:
https://www.rand.org/pubs/research_reports/RR1651.html
Rae, Andrew J., Acceptable Residual Risk—Principles, Philosophies


Texas Cities,” Transportation Research Part F: Traffic Psychology and Behaviour, Vol. 62, April 2019, pp. 66–77.
Shalev-Schwartz, Shai, Shaked Shammah, and Amnon Shashua, On a Formal Model of Safe and Scalable Self-Driving Cars, arxiv.org, 2017. As o


 “Future of Driving,” webpage, undated. As of August 30, 2020: https://www.tesla.com/autopilot
Thong, James Y. L., and Xin Xu, “Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology,”


n.house.gov/imo/media/doc/2020.09.15%20FINAL%20737%20MAX%2 0Report%20for%20Public%20Release.pdf
Venkatesh, Viswanath, Michael G. Morris, Gordon B. Davis, and Fred D. Davis, “User Acceptance of Information Technology: Toward a Unified View,” MIS Quart


https://www.rand.org/pubs/research_reports/RR2654.html 127
Wishart, Jeffrey, Steven Como, Maria Elli, Brendan Russo, Jack Weast, Niraj Altekar, and Emmanuel James, Driving Safety Performance Assessments Metrics for ADS-Equipped Vehicles, SAE Internat


COVID-19 and the Experiences of Populations at Greater Risk
Description and Top-Line Summary Data—Wave 1, Summer 2020
KATHERINE GRACE CARMAN, ANITA CHANDRA, DELIA BUGLIARI, CHRISTOPHER NELSON, CAROLYN MILLER
 Sponsored by the Robert Wood Johnson Foun


rts will update these findings.
Researchers from RAND and RWJF jointly conducted the research reported here; the report is intended for individuals and organizations interested in learning more about public attitudes about a Culture of Health and how


care, do you
think it is easier or harder for them to get the care they need than it is for those who live in
urban areas, or is there not much of a difference? ....................................................................... 24
Q21. [w1_pocim


pectations: Understanding What Drives Health............................................... 27
Q29. [w1_drivers1-w1_driver4] Here is a list of some things that may affect people’s health and well-being. Please rate each on a scale from 1 to 5 where 1


................................................................................... 36
Q13. [w1_covidexp1- w1_covidexp14] Are you experiencing or have you experienced any of the following as a result of the coronavirus (COVID-19) pandemic or the resp


nd unfair treatment of some groups, such as where people get to live or who gets bank loans. This type of racism can be so embedded in the institutions and practices of society that it can still exist even if individuals don’t want to discriminate. .


tions: Trade-Offs and Perspectives.......................................................... 43
Q31. [w1_liberty] People’s right to move about freely is so important that it’s worth risking a substantial increase in coronavirus (COVID-19) infections


are recruited via probability-based sampling methods. Both provide computers and internet connections for respondents who do not have them at the time of panel recruitment. Both compensate respondents for their participation. Both panels collect dem


sity; Trinity College; the University of Connecticut; the University of Minnesota; and the Urban Institute for their contributions to the survey design process. We also thank the survey participants, who offered their perspectives on health and well-


safety, use of community and policy supports, concerns about equity in COVID-19 response and systemic racism, and health civic engagement, as well as views of the role of government, the role of the private sector, the determinants of health, how the


vey questions, as documented in Table 1.1. The NSHA was fielded in 2015 and 2018 (Carman et al., 2016; Carman et al., 2019) to capture how people in the United States think about, value, and prioritize issues of health, well-being, and health equity.


 75% of Californians Support Shelter in Place ‘as Long as Needed,’” California Health Care Foundation, April 24, 2020. As of October 9, 2020: https://www.chcf.org/blog/covid-19-tracking-poll-75- californians-support-shelter-place-as-long-as-needed/
P