ed by groups outside the NSDL.org team (Zia, 2008). The case of IA demonstrates that this was indeed possible.
Several reasons for the declines in activity of the NSDL Services projects are apparent.
First, even popular services showed a quick dec


egration may become increasingly important. For example, the NSDL  program’s plan to reposition itself for the future (Howe et al., 2011)
discussed the STEM Exchange, which is a system that enables education practitioners to integrate paradata and i


on iTunes project also expanded the idea of integration in an innovative way, by
successfully integrating NSDL content in a broadly used platform outside the NSDL
infrastructure. Given the range of widespread platforms, such as Facebook and Twitter


to touch on several of the broad themes related
to services that were discussed earlier. One relates to the trade-offs inherent in trying to control the digital resources and services being developed by projects, to ensure that they reuse materials


scientific accuracy, bias, advertising, design and usability, and other aspects of digital libraries. In McCown, Bollen, and Nelson (2005), public school teachers evaluated 38 search terms and search results from Google and NSDL, but participants wer


s trust in the resource and is an increasingly important measure of information retrieval effectiveness (Abdul-Rahman and Hailes, 1999; Karlgren, 2008; Ramachandran et al., 2009). The trustworthiness of the resource is particularly important on the w


earch functionality in the current NSDL portal are now aligned with these
recommendations. However, it is not evident from the description of relevance ranking provided to searchers (NSDL , n.d.-b) that the presence of filtering-element metadata cor


hool  educational level =
high school;
resource type = data
set, instructional
material
4 Classroom activity to
demonstrate pattern recognition in middle school mathematics  Topic  pattern recognition
mathematics classroom activity middle sc


., conditions 1–5). We applied appropriate filters to the first five results from each of these five search conditions and compared ratings of cumulative gain and discounted cumulative gain for the results from key word–only searches in NSDL and Goog


the eight searches was 124 for NSDL and 203 for Google. The total score for NSDL was 39 percent lower than the score from Google, showing that NSDL produced much less relevant results. Similarly, using a different approach for ranking relevance, McC


ers are considered together, the NSDL relevance
rankings are closer to those of Google. Table 6.5 shows discounted cumulative gain scores for results from the NSDL filtered search and Google. Results show much closer relevance scores. When filters a


 step.

97 Figure 6.1. Google Result Set for Animal Camouflage Photograph

SOURCE: Screenshot  of Google search results  for “animal camouflage photograph,”  2013.

98 Figure 6.2. NSDL Result Set for Animal Camouflage Photograph

SOURCE:


 Different sites led participants to different resources—e.g., resources reported by DLESE and MSP2 users were only sometimes in the set of resources reported by NSDL users,
and resources found by DLESE, MSP2, and NSDL users were only sometimes in t


and beyond. Second, the development of an improved middle school pathway presents an opportunity to explore whether there are usability differences as a function of grade-level orientation versus subject-matter domain, in comparison to generic starti


to complete. The criteria also ensured that we could compare grade-oriented pathways (specifically, MSP2, which focuses on all STEM topics taught in middle school), subject-oriented pathways (which vary based on the subject chosen), and NSDL.org.


tor degrees of freedom varied because of a
small number of missing responses.17 Results are shown in Figures 6.3–6.8.
Participants had an easier time finding resources for some tasks than others. Results
show a statistically significant effect for


ing points affected the diversity of host websites.23
Given that there were few effects for domain, we combined results across earth sciences and
physical sciences for our remaining analyses.
Google led to the widest range of results, followed by


URLs, and the number of pages viewed per site was generally less than for the text-plus-image task. Findings differed slightly in that participants in the NSDL starting point had the smallest maximum page views (nine) compared with the other startin


al in Table 6.8 show that NSDL users were more likely to find resources also found by pathway starters than the reverse, suggesting that pathway sites are more discoverable through NSDL than the reverse. Likewise, the differences between common resou


this is the case for some of the sites that were selected most frequently by Google users (e.g., youtube.com and quizlet.com). In addition, when users started their searches with Google, results did not include an NSDL-affiliated site until the 11th


ngs of authoritativeness did, F(3,91) = 5.15, p < 0.01.
A contrast showed substantially lower ratings of authoritativeness for resources in .com domains (M = 1.71 out of 3.0) compared with those in .edu, .org, and .gov domains (means ranged from 2.5


nt, but relevance ratings remained lower than those from Google. In addition, NSDL does not take full advantage of the metadata to rank the relevance of search results. It is common in digital library initiatives to rank information provided in metad


ture research should systematically review the content of the resources that users find. It also is critical to obtain feedback from users about the search experience. As noted, federal regulations on research limited our ability to investigate users


t cases reflecting different models of implementation. Our intent was to find field settings where lessons could be learned about successful implementation of cyberlearning approaches to advance STEM education.
We evaluated these full instantiations


ajors to become K–12 math and science teachers
and by increasing the number of strong STEM K–12 teachers in high-need school
districts.
• iCPALMS aims to create online tools and services to enable K–12 math and science teachers to integrate high-q


 of varied academic ability levels in mastering a science field.
• Support new standards-based curricula effectively for earth science instruction in DPS.
RAND’s prior formative evaluation study included a visit to the CCS and DPS site in a
prelim


guides. Curriculum units were framed in terms of standards, key concepts, and learning goals to be achieved. What was lacking were high-quality interactive and tailorable resources—the sorts of materials likely to engage diverse students with differi


 However, some aspects of the organizational context raised concerns among the districts.
119 First, although acquisition and use of CCS was free, which likely promoted adoption, a
nonnegligible amount of teacher time would be expended in develop


by the National Center for Atmospheric Research at UCAR and well-known to the technical team. The follow-on effort examined whether the system could readily accommodate a different science domain; the choice of physical sciences was driven in part by


space with their own contributions. By the start of the following academic year (2009–2010), the system was judged ready for rollout; all earth science teachers in DPS received four hours of training led by the CCS principal investigators and TAB mem


districts we visited, a key point is that the CCS interface is
intuitive, as is navigation. A second major consideration is that the resources CCS presents are selected from trusted providers of high-quality materials and have been vetted, so teache


utility of the resource ratings.
Outcomes
Effects on Teaching and Learning
Teacher uptake of CCS is widespread and voluntary in earth science instruction. CCS
researchers estimated that across the six adopting districts, about a third of them ar


l districts in particular.
The Noyce program was first authorized in 2002 through the NSF Authorization Act and
then reauthorized through America COMPETES in 2007 and 2010. The program provides funds
to universities for student instruction, academ


enjoy a rich virtual community and to access high-quality teaching resources from their classrooms, thus improving their retention and enhancing their professional development opportunities.
Noyce scholars also play a key role in developing and diss


olars see [digital
resources] more frequently. Some are virtual, others are in -person day or multi -
day workshops on how to use different resources. Those were very popular. And
when presenter is partnered with Noyce alum, that makes a big differ


imilar search engines—reflecting views about resource authoritativeness and usability. One participant noted, for example:
I do a lot of “Frankenstein ” stuff—sometimes I take individual photos,
sometimes labs, and piece it together.
I “Frankenst


folios, to information about effective classroom-management strategies, and to Noyce Voices, which is a Noyce-tailored portal to MERLOT Voices that helps Noyce scholars remain in contact with each other. Participants in our discussions noted that the


ask [the] master teacher,  “Do you think this would work? What would be
pitfalls? How can I make it better? ”
I was only in Noyce for a semester, but in that semester, I networked with in -
service teachers and professors, and I was involved in NS


bstacles
to uptake.
Variations in the Availability and Quality of Technology
The technology available in high-need classrooms varies significantly, and many teachers do
not have sufficient access to computers and projectors. Firewalls inhibit s


 exercises that would not otherwise be possible:
When you find videos, you find a way of teaching something you hadn ’t really
realized. It opens up your vocabulary for things. The way you say it to some kids
make s sense, and you have to say it f


ese [online] resources.
The resources are more critical once we are advanced. We can know how much
time the resources require, know if  it is appropriate.
Interviewees reported that digital resources also have positive effects on student
learni


ers.
• Establish a collaborative network of users.
• Build capacity for long-term sustainability.
In brief, CPALMS is a website that provides standards-based, instructional content for users,
and iCPALMS is a portal that allows users to customize


he development process accounts for the technical capabilities of diverse systems. For example, in many districts, computers in the schools use outdated Internet browsers because other school software is not compatible with browser updates. As a
res


training for early adopters, who would then spread the word among other potential contributors and end users. For example, the management team introduced CPALMS at conferences for district math and science curriculum specialists whose role it is to r


of the server were the only complaints that case study participants reported). This problem was resolved by obtaining funding through the Teacher Standards Instructional Tool grant to add servers and update software at the Florida DOE and transfer CP


n the case study provided comments that illustrate these benefits:
It’s great to know that teachers have a one -stop shop where they can put in the
benchmark, find a great aligned activity that ’s age appropriate, that ’s already
vetted and ready


ther teachers.
Other Effects
Although iCPALMS was still under development at the time of this case study, and the study
did not identify effects on learning other than statements about improved student engagement,
CPALMS and iCPALMS have demons


rocess in other districts and for other topical domains (physical science). CCS fostered technology readiness by providing all earth science teachers in DPS with hardware and Internet connectivity prior to rollout, in contrast to MERLOT and Noyce Sch


 In CPALMS and iCPALMS, the project
director was the primary champion, but a growing community of user-developers also
served to stimulate demand for digital resources.
• All three cases used participatory design of the system beyond the project’s


ces, and community members’ contributions in the form of creating, sharing, and vetting resources, plus support from partners, furthered the development of diverse collections of high-quality STEM instructional resources—thereby fueling additional de


he NSDL  portal has also helped to foster communities of developers and users who can share
ideas and resources, tools, and services.
143 These roles reflect the components of the NSDL logic model and also mirror its dynamic
processes: The NSDL p


e NSDL logic model. To our knowledge, however, CCS is one of the only NSDL projects that has approached this ideal. In the final chapter, we discuss some of the reasons that this goal has been so challenging for the NSDL program to achieve, and we id


arch from the top-level page for pathway portals. The NSDL portal may be
145 able to improve the relevance of search results by capitalizing on existing strengths—i.e., by
making use of its high-quality metadata, connection to educational standard


evelop their tools and services independently of other projects, and thus we did not see as much collaborative reuse and mixing of services as the NSDL logic model might have predicted.
146 To What Extent Did NSDL Produce a Growing and Vibrant Com


 program ended in FY 2011, and many of the NSDL pathways continue to
operate. As of December 2014, management of NSDL.org was transferred to the Institute for the Study of Knowledge Management in Education and is available at https://nsdl.oercommons


pand
collections to provide access to innovative content and genres and to present resources in
compelling and informative ways (e.g., images rather than text), particularly to maintain relevance to digitally savvy users. This poses a potentially t


t not implemented in NSDL sites. For well-established services, such as faceted search or browsing, it was reasonable that the program would expect funded projects to implement versions that could be integrated quickly into the NSDL platform. However


just passive consumers of digital resources, but this will impose additional management and monitoring costs on the library operators—assuming that they want to maintain high levels of quality.
Recommendations to Address Trade -offs
Some of the ten


ch the whole of
NSDL is greater than the sum of its parts; indeed, many of the project s’ products are highly
interdependent. For example, the NSDL portal and pathways are jointly generating composite metadata and paradata; in addition, resources a


cated case studies. At the same time, as we have noted throughout this report, a wider range of research targets and methods is needed for a thorough evaluation of NSDL or other digital initiatives.
Most important is evaluating impact on student en


stainability, such as development of strategic plans and funding for sustainability, and how the initiative contributes to the mission of the parent and sponsoring institutions (and vice versa). Data from parent and sponsoring institutions would addr


collection and analysis to a relatively few key projects and results, yielding more-complete information about outcomes, such the diversity of products, gaps, and redundancies in tools and collections and perhaps unexpected new topics of research and


it intends on STEM education for the long term. We examine the major components of the NSDL technical infrastructure and assess them according to standard and emerging rubrics for open-source software applications. The components that are evaluated a


Collection System)
and the Java-based Open Access Initiative (jOAI) Protocol for Metadata Harvesting (OAI-PMH)
application.
The NCS is a reference implementation of the Digital Collection System (DCS). The NCS is
a web-based metadata creation an


r)
associated with privileges within specific collections. Workflow control at the record level is
provided by statuses, with the ability to publish a metadata record in the NSDL Digital Discovery System (DDS) tied to a status indicating readiness


s of the organization evaluating the software. A popular model is the Navica Open Software Maturity Model (OSMM) (OSS Watch, 2010). The OSMM assesses maturity across six categories, with each category weighted according to the need of the organizatio


s broadly to a collection of metadata items. OAI-PMH applications may support data provision (responding to requests), harvesting (requesting metadata), or both (Open Archives Initiative, n.d.). The OAI implementation within the NCS, discussed above,


ty of Illinois at Urbana-Champaign Grainger Engineering Library, n.d.) and the OCLC WorldCat OAIster database (OCLC, n.d.), a union catalog of millions of metadata records harvested via OAI from OAI data providers.
The most important role for the jO


CPALMS  10 9 (90%)
MatDL  10 10 (100%)
MathDL  10 6 (60%)
MathLanding  10 10 (100%)
Middle School Portal  10 10 (100%)
SMILE  10 9 (90%)
Teacher’s Domain   10 10 (100%)
Teaching with Data  9 9 (100%)
Teach the Earth  10 10 (100%)
TO


ipt eye-readable text or JavaScript Object Notation (JSON) format, allowing developers to embed and reuse NSDL resources using a wide range of technologies, such as JavaScript, PHP (Personal Home Page), and JSP (JavaServer Pages) (John Weatherley, pe


 the components, particularly the scheduling of jOAI harvesting and the implementation of the DDS Search API. We evaluated search more extensively in Chapter Six , where we also provide recommendations for improving search
precision through the sett


omized Mediawiki. It included NSDL branding and was
integrated into the NSDL framework; new pages in the wiki were added to the NSDL repository ,
and plug- ins allowed the wiki editing tool to directly reference and incorporate content from the
NS


oted below. The NSDL Science Literacy Map service (http://strandmaps.nsdl.org/) continues to be used, with approximately 3,000 visitors a week between October 1, 2009, and
September 30, 2012.
Sustainability  The future of the Strand Map Service an


DL
Resource Center. Subsequent grants from NSF also offered professional development to teachers and evaluated the effectiveness of those efforts.
Subsequent
services/ tools or
uses None

168 Table C.5. NSDL on iTunesU
Service/Tool
Name


ake several modifications to CPALMS;
add servers to host
CPALMS/iCPALMS ; and
update software
Math an d Science
Partnerships/Bioscopes  2011 U.S. Department of Education (funded
through the State of Florida)  Teachers , along with arts and
sc


e used for research purposes only.  We will not link your email address
with your responses.  We will keep all responses confidential and will not retain your email
address after the study.
If you prefer a hard copy of the survey , along wi th a


th
o Once	 or	twice	a	term
o Less	often

174
7.	Please	check	all	of	the	ways	that	you	have	used	NSDL.
o Find	information	about	a	STEM	subject	to	plan	or	update	a	lesson
o Find	other	resources	to	use	for	delivering	a	lesson,	such	as	vid


sagree Disagree
Somewhat  Neither
Agree nor
Disagree Agree
Somewhat  Agree  Strongly
Agree
1 2 3 4 5 6 7
a. are	current. 	 o o o o o o o
b. are	comprehensive. 	 o o o o o o o
c. are	accurate. 	 o o o o o o o
d. reflect	real -world	content.


	school ’s	network	access	 (e.g.,
speed	and	reliability )	supports	my	use
of	NSDL	resources.	 	o o o o o o o
j. My	network	access	at	home	supports
my	use	of	NSDL	resources. 	o o o o o o o

179 12.	Please	rate	the	 following	statements	about


ee, ”	please	rate
these	descriptions	of	your	experience	with	technology.

 Strongly
Disagree Disagree Disagree
Somewhat  Neither
Agree nor
Disagree Agree
Somewhat  Agree  Strongly
Agree
	 1 2 3 4 5 6 7
a. If	I	heard	about	a	new	informatio


ternational Symposium on Database, Web and Cooperative Systems ,
1999. As of October 17, 2013:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.38.7934
Ackerman, S. L., K. Tebb, J. C. Stein, B. W. Frazee, G. W. Hendey, L. A. Schmidt, and R


ce of Creative Tension
with Technical Developments,” Bulletin of the IEEE TCDL , 2003. As of January 18, 2013:
http://www.ieee-tcdl.org/Bulletin/v1n1/blandford/blandford.html
Blue Ribbon Task Force on Sustainable Digital Preservation and Access, S


n Rules for Evaluating Open Source Software, Collaborative Consulting, 2004.
Downes, S., “Models for Sustainable Open Educational Resources,” Interdisciplinary Journal of
Knowledge and Learning Objects, Vol. 3, 2007, pp. 29–44.
Dupret, G., “Disc


ucation Development Center, Inc.: Waltham, Mass., 2005.
Harley, D., J. Henke, and D. Nasatir, Use and Users of Digital Resources: A Focus on
Undergraduate Education in the Humanities and Social Sciences , Berkeley: Center for
Studies in Higher Ed


Karlgren, J., “Changing the Subject—One Way of Measuring Trust in Information,” Workshop
on Novel Methodologies for Evaluation in Information Retrieval, Glasgow, March 30, 2008.
As of May 23, 2013: http://soda.swedish-ict.se/2763/
Kennedy, J., “A


Archives Info,
Vol. 1, No. 3, 2001–2002. As of December 19, 2016: http://www.archivesinfo.com/newsletter/Winter2001.pdf
Margaritopoulos, T., M. Margaritopoulos, I. Mavridis, and A. Manitsaris, “A Conceptual
Framework for Metadata Quality Assessmen


. As of January 18, 2013: http://www.tccgrp.com/pdfs/per_brief_tenkeys.pdf
Multimedia Educational Resource for Learning and Online Teaching, “Partner Communities and
Institutional Teaching Commons,” web page, 2009. As of March 6, 2017: http://info.


ion.
OCLC, “The OAIster Database,” web page, n.d. As of October 17, 2013:
http://www.oclc.org/oaister.en.html?urlm=168646
Open Archives Initiative, “Open Archives Initiative Protocol for Metadata Harvesting,
Interoperability Through Metadata Harv


nuary 18, 2013: http://hdl.handle.net/2142/145

195 Smith, A., Building and Sustaining Digital Collections: Models for Libraries and Museums,
Washington, D.C.: Council on Library and Information Resources, 2001. As of January 18,
2013: http://ww


009/11/best-practices-for-6-common-user-interface-elements/
Webster, F. E., “The Changing Role of Marketing in the Corporation,” Journal of Marketing,
Vol. 56, 1992, pp. 1–17.
Wenger-Trayner, E., and B. Wenger-Trayner, Introduction to Communities


SOCIAL AND ECONOMIC WELL-BEING
Updating the Costs of
Compliance for California’s Hospital Seismic Safety Standards
Appendix D. Regional Cost Analysis
Benjamin Lee Preston, Tom LaTourrette, James R. Broyles, R. J. Briggs,
David Catt, Christopher Ne


riteria, we defined analysis regions as individual counties with at least ten
hospitals and groups of counties containing fewer than ten hospitals each. For county groupings, we used the California economic markets defined by the California Employme


iring structural performance category (SPC) and nonstructural performance category (NPC) upgrades.  3 Table D.2. Regional Seismic Upgrade Costs
Analysis Region  Hospitals  Retrofit
Cost
($B) Escalated
Cost
4%/year  Demand Surge
Escalated Cost


CORPORATIONAppendix
The Welcome Baby Program
An Implementation and Outcomes
Evaluation
Sarah B. Hunter, M. Rebecca Kilburn, Teryn Mattox, Jill S. Cannon,
Terry Marsh, Melissa Felician, Maya Buenaventura, Lauren Davis,
Matthew CefaluLimited Print


was specifically designed by the research team
for this project. It was developed based on both a review of the Welcome Baby program materials that outlined the key elements of the program and meetings with F5LA, LABBN, and
the Welcome Baby sites a


 that
identifies you, including the audio recording of this interview, at the end of the study. Due to the
extensive data protection procedures we have in place there is minimal risk associated with participation in this interview. There is no dire


 recruitment
a. Is there anything about your site or organization that has helped your staff to recruit women postnatally? (e.g., bilingual hospital liaisons)
b. Is there a system in place to monitor outreach and recruitment activities? If so, 11


erience
Next, I would like to ask you about training resources and technical assistance
18. Can you describe your experience with PAC/LAC’s [Perinatal Advisory Council: Leadership, Advocacy, and Consultation’s] workshops and technical assistance?


Name ___________________ Date ___________________

Participant Name: ___________________ Welcome Baby Site ____________________
The purpose of this protocol is to guide the semi-structured interviews with Welcome Baby
Home Visiting staff (includi


ts.
If you have any questions or concerns about the study or this interview, you can contact
Sarah Hunter, the Principal Investigator or Jim Tebow from RAND’s Human Subjects Protection Committee. Their contact information is provided in the initia


ively?
12. Provision of service delivery/home visits
a. Is there anything about your site or organization that has helped to ensure that participants received their nurse visit? 16
 b. Is there anything about your site or organization that has he


ain communication challenges?
Staff Support
Now I will ask you some questions related to staff support and supervision
19. Do you receive support from your supervisor? If so:
a. What kind of support do you receive?
a. How often do you receive


 been enrolled in the Welcome Baby program.
Experience with Program
Program Enrollment
1. How did you learn about the Welcome Baby Program? (Ask where and when they were approached).
2. How was your enrollment experience?
3. Did you have any pr


ffsurvey@rand.org.
Please enter your 5 -digit survey password. ____________________
Screen 2 [ALL]
First 5 LA, has contracted the RAND Corporation, a nonprofit, nonpartisan, research organization, to conduct an
evaluation of the Welcome Baby p


@rand.org. If possible,
when you contact the Committee, please reference Study #2015 -0645.
Please indicate whether or not you agr ee to participate in this study.
______Yes, I agree to participate in this study
______No, I do not agree to par


 CDA
□ CHES
□ CIPSW
□ CLC
□ CLE
□ IBCLC/RLC
□ ISW   □ LCSW
□ LPN
□ LISW
□ LMFT
□ PHN
□ RN
□ Other, please specify: _____ ____
9. In what language(s) are you able to provide Welcome Baby services? [select all that apply]


know the exact date, please enter your best estimate .
__________ [clickable calendar]
If your supervisor has never observed one of your home visits, please leave the date field blank and check
the box below.
o My supervisor has never observed


30. Please indicate your agreement with the following statements using this scale:
[Strongly Agree; Agree; Neutral; Disagree; Strongly Disagree]
a. I believe that Welcome Baby services help the families I serve.
b. Welcome Baby program materia


enter
o Valley Presbyterian Hospital
o White Memorial Medical Center
Screen 27 [ALL]
37. What is your current Welcome Baby job title at <site name>? [select one]
If you do not see your exact title listed, please select the standard Welcome


is scale:
[Strongly Agree; Agree; Neutral; Disagree; Strongly Disagree]
a. The Welcome Baby new staff training (also called “Cohort Training”) prepared me to deliver
Welcome Baby as required by my current position.
b. The regional training ses


uding contact lists, protocols, forms, and
memoranda of understanding, were gathered from each site and abstracted for relevant information.
Abstraction content: The research team designed the abstraction form to cover the required
elements of th


Transportation

 36 Document Type   Referral Directory/Binder   Referral Protocols   Formal MOU
Checklist Items
Up-to-Date
English
Contact
Information
for Referral
Type  (Y/N)  Up-to-Date
Spanish
Conta


to any of these questions. The survey should take about
10–15 minutes.
In this next section, please answer some questions about the services you have received from Welcome
Baby. We are interested in your honest opinions, whether they are positive


 ¨ ¨
 40  Agree Disagree  Not sure
16. A baby with colic can cry for 20 or 30 minutes at a time,
no matter how much you try to comfort him (her).  ¨ ¨ ¨
17. Fathers are naturally clumsy when it comes to taking care
of babies.  ¨ ¨ ¨
18. Taking


aches, and Parent Coach Supervisors over the course of a client’s engagement with the
Welcome Baby program. Examples of information recorded in the SFDB are the types of content covered during home visits, measures of maternal and child health and s


This appendix provides an overview of our methods for calculating fidelity domain
achievement thresholds described in Chapter 4 of the report. Each table provides information on
one fidelity domain’s specifications for data, sample, method for crea


me of survey,  as applicable.
Dates covered by data  May–August 2017
Frequency of measurement  Once
Reporting process  Self-reported survey answers
Average across sites 101 of 117 staff (86%) met minimum requirements.
Additional notes • Ref


ervision than other staff positions surveyed.
• Four sites had a Staff Survey response rate less than 54%.
• Across sites, total Staff Survey sample size ranges from 2 –17.
• A site achieved domain fidelity if 100% of surveyed staff met minimum re


.7. Fidelity Domain Seven : Hospital Enrollment
Fidelity domain  Hospital Liaisons approach mothers who give birth at the hospital and
encourage them to enroll into Welcome Baby.
Data source  Monthly site data reports provided by LAB BN
Sample


ring these time periods:
• prenatal home engagement (up to 27 weeks)
• prenatal telephone assessment  (by 32 weeks of pregnancy )
• prenatal home engagement (by 38 weeks of pregnancy)
• postpartum hospital engagement
• postpartum Nurse home


iance Inventory, Bond subscale –4 items)
Sample 1,976 client visits (a client may have completed more than one survey) ;
Best Start families
Measure/indicator  All staff are mentored and trained to build positive relationships with their
client


tions.
• We did not include content related to referrals for services.
• We excluded visits designated “Other.”
• We base the site -level domain status on site achieving average
percentage or above based on Welcome Baby –wide average
across


pecifications for individual outcome measures reported in
Chapter 4.
Table D.1. Parenting Practices
Data source  Five mother -reported HOME inventory (Caldwell and Bradley, 1984)  items on
the Client Survey . Note that the full HOME inventory i


 D.3. Family Planning
Data source  SFDB
Sample  Data from 21,266 visits. We excluded the following from our analyses:
• family planning data from the post -NICU -discharge visits or visits
labeled “Other”
• data from clients not enrolled in


ck all that apply .” In a very small
number of cases, staff person checked a valid type of insurance for
the child (in most cases Medicaid) and also checked the “no insurance” box. In these cases, the child was marked as having insurance.

Table


me
• visits with missing home safety data
Measure/indicator  Percentage of visits where home safety inspection is completed with no
home safety issues identified
Dates covered by data January 2016 through December 2017
Frequency of measurement


ge of children above the ASQ- 3 cut -off on all five ASQ scales,
at a given visit
Dates covered by data  January 2016 through December 2017
Frequency of measurement  Every postpartum visit excluding the  hospital and post-NICU -discharge
visits


0.24 5,232
Percentagge of content  covered  –0.19 0.27 9,954
RN to Two -to-Four -Week Visit
Family Characteristics
Low income –0.24*  0.11 7,130
Low-birthweight child  0.26 0.21 7,082
Birth complications  0.15 0.15 7,132
First birth  –


ications  –0.19 0.11 5,380
First birth  –0.09 0.07 5,366
Enrolled prenatally 0.30***  0.09 5,380
 67  Coefficient
Estimate  Standard Error  Number of
Observations
Some English 0.66*** 0.12 5,374
No English  0.48***  0.11 5,374
Mother black


n #6 Technical Appendix
Table F.1. Outcomes by Time Point, Including Benchmarks
Outcome
Category Outcome
Measure  Data
Source Measure   Rates and Number of Observations
(bold = outcome analysis)
Benchmarks
 (sources below table)  Hospita


t
recommended
intervals SFDB  % with up -to-
date well-child
visits % N/A N/A 91.6 89.6 93.6 83.2 89.9 LAMB Survey (2010, 2012
combined): % of infants with any well- baby
checkup: 98.1% in LAC,
96.7% in BSC (2)
N N/A N/A 5,089 4,269 4,070 3,4


 co -sleeping  • No co -sleeping
Home visitor
workloads  Positive   • Immunizations up
to date  • Child insurance
• Immunizations
up to date
• No home safety
issues  • Back to sleep
Negative  • Family
planning  • Family planning  • Famil


, and J. Arthur Gillaspy, “Development and Validation of a Revised Short
Version of the Working Alliance Inventory,” Psychotherapy Research, Vol. 16, No. 1, 2006, pp. 12–25.
Larsen, Daniel L., C. Clifford Attkisson, William A. Hargreaves, and Tuan


Image credits—Cover: Violetastock/GettyImages; p. iv: digitalskillet/GettyImages; p. xvi: asiseeit/GettyImages;
p. 4: Debby Lewis-Harrison/GettyImages; p. 8: digitalskillet/GettyImages; p. 14: oscarhdez/GettyImages;p. 38: fstop123/GettyImages; p. 4


alth. Maltreated children fare worse than their peers on many important outcomes within these domains. The effects can persist and have long-term consequences into adulthood, including reduced labor market productivity, increased involvement with the


into young adulthood, manifested in numerous ways, including home-lessness, underemployment, criminal conviction, and substance abuse. Overall societal costs associated with child maltreatment may total $80 billion annually (Gelles and Perlman, 2012


tives. Increases in prevention lead to decreases in mal-treatment and improvements in young adult outcomes but do not affect the experiences of children who enter the system and result in small additional costs. Increases in treatment lead to improve


experiencing the different events in the model (e.g., maltreatment, referral to the system, placement in foster home). The baseline model, therefore, is intended to represent the current situation. We then used the model to assess the average effect


ilies involved with the child welfare system, whether to emphasize in-home options or out-of-home placements.
What Effects Do Individual Policies Have?
To understand the effects of individual policies, we estimated how the quantity and quality of pre


eventive services in the model, such as the Nurse-Family Partnership, have been shown to have a number of positive benefits for children and families beyond reducing the likelihood of maltreatment (Olds, 2006). These benefits include improved infant


Specifically, family preservation efforts led to changes in the young adult outcomes, with the largest changes when both quantity and quality of the policy were increased. The effects ranged from 0.4 to –9.8 percent for substance abuse, from –3.6 to


f the individual options that we considered achieves all these objectives.
We estimated the effect of a policy package that combined increases in
preventive services and treatment in the form of support for kinship care. The results are summarized a


ce of those children who are in the system and their outcomes in young adulthood. Family preservation and kinship care efforts also lead to reductions in the overall cost of the system because they promote placements that are less costly than the oth


ific jurisdictions would allow a number of important improvements, including more-specific informa-tion on pathways through the system; better data on children’s experiences in the system; and information on how the child welfare system interacts wit


 the
project. She was highly engaged and provided invaluable input throughout the entire process. We are particularly grateful for her assistance in identifying and connecting us with experts to provide input into the model.
Susan Gates, director of


ations at 37.4 percent (Kim et al., 2017) and lifetime substantiated maltreatment at 12.5 percent (Wildeman et al., 2014). Child maltreatment has been linked with child and family characteristics, such as a child’s age, race/ethnicity, gender, and sp


t to sepa-rate them when looking at outcomes (Masten and Wright, 1998).
Over time, the effects of maltreatment and child welfare system in-
volvement can persist, with effects on outcomes that last into young adulthood, including reduced labor marke


er comes from a mix of state and local funding (DeVooght and Cooper, 2013). This total likely does not include funding for all child maltreat-ment preventive services. The survey that generates these estimates asks about funding for these services pr


ult is a model that estimates the average
effect of implementing the policy in all jurisdictions across the country.
To simulate the effect for a particular state or locale, the model assump-tions and data would need to be changed to better reflect


maltreatment and detection component of the model until he or she leaves childhood.
System Pathways
The child welfare system pathways component of the model represents the possible pathways through the child welfare system for a child whose suspected


plex array of risk and protective factors ultimately influence how children fare over time. The outcomes are influenced by the circumstances that led to the maltreat-ment itself and, to some extent, the involvement with the child wel-
fare system. At


d the model to estimate the effect of implementing several
different policy, program, or practice changes.
These steps are briefly described below and in greater detail in
Appendix B, available on the project website (www.rand.org/child-welfare-mode


features, such as poverty rate.
1
To simulate the experiences of these children, we had to determine
their chances of experiencing each of the many child welfare events
and/or outcomes in our model. We pooled information from admin-
istrative data


S reports 18–22. We also reference the National Survey of Child and Adolescent Well-Being (NSCAW) Wave II Child Permanency report. With these data, our model simulates the childhood experiences for a cohort that is like the 2010 to 2015 cohort over t


typical goal for calibration is to get model outputs that are within the natural variance from year to year for
each metric. The 0.3-percent difference between our model outputs and the calibration targets is generally much smaller than the natural


ot directly account for all of the complexities in how
both risk and protective factors interact and the uncertainties about the causal pathways. However, it allows for an understanding of how child maltreatment and system involvement affect outcome


Some children experience multiple types of maltreatment (e.g., neglect and physical abuse); the estimates have been adjusted to account for this.
Efforts to prevent maltreatment will also occur during this period.
Collectively, our model estimates t


 1,000 children.
In higher-risk cases, the child is moved from the home to an out-of-
home placement. Our model projects that there will be 59 out-of-home
placements per 1,000 children. Most children who are removed from
the home will experience on


ixed, regular, adequate
nighttime residence by age 25. This includes adults living in travel accommodations, sharing the housing of
other persons due to hardship, having a nighttime residence that is not designed for ordinary use as a regular
slee


es in child welfare.
8   Substance abuse includes heavy drinking, defined as typically consuming five or more drinks three times per
week, and drug use, defined as use within the past ye ar of cocaine, amphetamines, hallucinogens, or heroin,
but n


d maltreatment.
Among promising targeted primary prevention approaches, home vis-
iting is the most widely studied. A number of assessments have found associations between prevention approaches and reductions in the risk factors for child maltreatme


t but assumes that
effectiveness does not change (Scenario A).9
•The s
econd scenario holds the quantity of services constant but
assumes that preventive services are of higher quality (Scenario B).We assume that the increase in the effectiveness is


agencies or community-based organizations may not
be included in this total. Over the same time period, the population of children in NCANDS-reporting states was approximately 67.7 million. Taken together, this equates to 33.9 preventive services fu


f prevention, the model estimates approximately
0.3 percent fewer investigations and 1.1 percent fewer substantia-
tions and out-of-home placements relative to baseline. With increases in both quantity and quality (Scenario C), we see somewhat large


1.6% –1.5% –1.6% –1.6%
–3.6%–3.4% –3.5%–3.6%FIGURE 4.6
Effects of Prevention Scenarios on Outcomes, Percentage Change
from Baseline26Family Preservation Services
Family preservation programs are designed to prevent out-of-home
placements or to reun


ot find that the program increased the likelihood that children would remain in the home, program participa-tion was associated with a higher rate of reunification for chil-dren in out-of-home care (Olsen, Laprade, and Holmes, 2015).
How the Policy I


reatment also increased slightly, both because the number of incidents increased and because of increased opportunities for detecting maltreatment when families receive family preservation services. For the quality scenario (B), both the average numb


r Family Preservation Scenarios30RAND  RR1775x1-4.9Scenario A Scenario B Scenario C
Prevention spending
Child welfare system spending
Net spending–0.3%
–9.9%–9.1%–6.7%–6.2%
–13.7%–12.7%–0.1%+0.2%FIGURE 4.9
Effects of Family Preservation Scenarios on


ounseling, respite care, and legal services (Lin, 2014). How and where services are offered varies significantly (for example, they can be peer to peer, provider-led, school-based, or community-based). Training and education interventions can support


anship as a permanent outcome;receipt of targeted preventive services;receipt of supports and services for transition to adulthoodTABLE 4.3
Kinship Care Scenarios33relatively little impact on detecting maltreatment in the population.
There is a very


first, which focuses on quantity, results in a 50-percent increase in the quantity of funded preventive services and in the number of children placed with kin (Scenario A). The second option focuses on quality by assuming that the preventive services


oodTABLE 4.4
Combined Prevention and Kinship Care Scenarios
RAND  RR1775x1-4.13Scenario A Scenario B Scenario C
Maltreatment episodes
Referrals to child welfare–1.6%–0.2%–0.6%
–4.1%–1.5%–0.2%FIGURE 4.13
Effects of Combined Approach of Prevention and


nge from Baseline
RAND  RR1775x1-4.16Scenario A Scenario B Scenario C
Homelessness
Underemployment
Substance abuseCriminal conviction–2.8%–1.6%
–2.3%
–3.3%–2.2%
–3.0%–2.5%
–5.8%–5.6%
–6.4%–5.2%–2.4%38
39CHAPTER FIVE
Discussion
Under current policy, f


vices are designed to prevent child maltreatment from occurring and thus reduce the number of children that ever enter the system. If a child has been maltreated, family preservation treatment efforts are aimed at providing services and supports that


ore children. The increase is partially offset by reductions in spending on screenings, investigations, services, and temporary placements as fewer children flow through the system. We assume that quality can be improved at no additional cost by real


ide services and supports to the kin caregivers to ultimately decrease time in care and increase the child’s chances of returning to the family. The increases in temporary kinship care and permanency outcomes with kin lead to small improvements in ou


ich is extremely com-plicated. Building such a model requires many assumptions. In many cases, the available data and literature do not provide as much infor-mation as would be ideal, and there is certainly room for reasonable disagreement about the


placement. However, it is difficult to say how much of the discrepancy represents real differences in the sources and how much is simply the result of not being able to make an apples-to-apples comparison between empirical data and the internal calcu


the effects of system experience on outcomes. To address this problem, we augmented the information we found in the literature by conducting data analyses designed to address this question specifically, but additional research in this area is needed


nformation on pathways through the system; better data on children’s experiences in the system; and information on how the child welfare system interacts with other related systems, such as education or criminal justice.
Finally, while the baseline m


pathways through the system, and consequences in a comprehensive quantitative model that can be used to simulate the potential impact of policy changes. The estimates presented here provide policymakers with a sense of the relative costs and benefit


rmer Foster Youth: Outcomes at Age 23 and 24,” Chicago, Ill.: Chapin Hall at the University of Chicago, 2009.
Courtney, M. E., B. Needell, and F. Wulczyn, “Unintended
Consequences of the Push for Accountability: The Case of National Child Welfare Pe


es Bell Associates, Profiles of the Active Title IV-E Child Welfare
Waiver Demonstrations , Arlington, Va., 2016.
Jones Harden, B. J., “Safety and Stability for Foster Children: A
Developmental Perspective,” The Future of Children , Vol. 14, No. 1,


ly Partnership: An Evidence‐Based
Preventive Intervention,” Infant Mental Health Journal , Vol. 27,
No. 1, 2006, pp. 5–25.
Olds, D. L., J. Eckenrode, C. R. Henderson, Jr., H. Kitzman, J. Powers,
R. Cole, K. Sidora, P. Morris, L. M. Pettitt, and D.


hildren’s Bureau, Child Maltreatment 2014 ,
Washington, D.C., 2016.
Washington State Institute for Public Policy, Intensive Family
Preservation Programs: Program Fidelity Influences Effectiveness—Revised , Olympia, Wash., 2006.
Wildeman, C., N. Ema


Discount Justice
State Court Belt-Tightening in
an Era of Fiscal Austerity
Michael D. Greenberg, Samantha Cherney
CORPORATIONLimited Print and Electronic Distribution Rights
This document and trademark(s) contained herein are protected by law. This


participants included judges, state court administrators, prominent mem -
bers of the bar, and legal scholars.
This report shares key issues and topics from the January 2015 conference sessions. This
document is not a transcript. Rather, it summari


mitted to the public interest.
RAND Ventures is a vehicle for investing in policy solutions. Philanthropic contributions
support our ability to take the long view, tackle tough and often-controversial topics, and share our findings in innovative and


....................................... 12
Do Courts Risk Being Delegitimized When Challenging the Constitutionality of Budget Cuts
Imposed by Other Branches of State Government?  .................................................... 13
What Role Can


, state court administrators, prominent members of the bar, and legal scholars.
Keynote speeches at the conference were given by Tani G. Cantil-Sakauye, the Chief
Justice of California, and by William T. Robinson III, co-chair of the American Bar A


nd all those who engaged in the conference discussions. We would particularly like to
thank the following participants for their contribution to the panels and the discussion, with -
out which the conference would not have been feasible: Joseph Dohe


d by chapters, which correspond to the panel discussions
and speeches that took place during the conference. Each of the panel discussions is sum -
marized by the major themes that were discussed in the order that they were covered at the conference


nd somebody asked me, “What are you working on now, and what excites you about it?” And I replied, “There are two things that have been on my mind lately. One of them is the institutional administration and governance of state court systems. The othe


most elegant text, and the best Constitution. But at the end of the day, if you’re not willing to put money into the institutions that actually carry out the work, the documents are capable of becoming empty. The promises are capable of being unfulf


it comes to the money side of the picture. This is true with regard to the courts and the judiciary, as it is with regard to lots of other kinds of institutions.4    Discount Justice: State Court Belt-Tightening in an Era of Fiscal Austerity
It turns


dea of the Minnesota Supreme Court; Judge Carolyn Kuhl of the Los Angeles Superior Court; Mary McQueen, president of the National Center for State Courts; and Michael Greenberg, senior behavioral scientist and then director of the RAND Center for Cor


e funding mechanism worked well during the temporary surge in foreclosure filings, but it then caused the courts to go into the red when foreclosures started to tail off several years later. McQueen and Greenberg also talked about the pros and cons o


urt system in Minnesota. She observed that when the state transi -
tioned from county funding to state funding in 2005, the Chief Justice decided to share her governance authority over the courts by creating the Judicial Council, which is a 25-member


en shifted to address the court system governance models and choices of
other states. McQueen described the governance systems in Oregon and Florida. In Oregon, she pointed out, the move to centralize funding for the courts by the state legislature


d that that financing mech -
anism ties back to governance, since the flow of money also reflects who has decisionmaking authority over the money. Greenberg concluded that Ohio illustrates how complex the local budgeting and governance picture actual


he impact on the system, Kuhl described morale, security, and geographic
access challenges that have resulted from the resourcing cuts. She also described several func -
tional measures suggesting impairments in the speed and volume of cases process


bmit their budget proposals for the governor’s approval prior to reaching the legislature (a practice akin to that of an execu -
tive branch agency) or else by being subject to detailed line-item appropriations that restrict the courts in shifting th


obert Peck then spoke to the latter question and about the response of the courts them -
selves to funding shortfalls that putatively threatened their constitutional function. Peck observed that there has been a series of cases addressing the status


opriation below the level at which it had been prior to 1991. This was a compromise result, but it generated lots of ill will and distrust between the branches of government in New York.
Lippman concluded that the New York experience in 1991 invites


summarized this another way by emphasizing that public recognition of the importance of the courts, and of keeping them fully functional by keeping them fully funded, is often central in these kinds of disputes—and lack of public recognition is probl


o litigants. This got noticed by the newspapers and by the legislature. The courts made the decision to con -
tinue allocating money to legal aid for the poor, based on the belief that this was constitution -
ally imperative, even while staff were be


Equal Protection Problem?
A member of the audience observed that the burdens of the budget shortfall in California
courts have fallen heavily on personal injury cases, such that delays and geographic problems in accessing the courts have become part


o say that a shutdown response is not only not unprecedented, but that it is almost a necessity when budget cuts become draconian. In those situations, Lippman said that the courts run through all the options presented by Judge Kuhl in the earlier se


 the judiciary is a unified system, there is a lot of autonomy built into it at the local, trial court level.
Can You Describe the Transition from Local to State Funding of the
Judiciary?
Sungaila followed up with a question about the transition fro


choices to make about each disposable dollar and that she must make the best case for why they should invest in the judicial branch. This takes time, education, and exposure to the courts. Now when she goes to the Capitol, Cantil-Sakauye does not as


the broader, long-term future of the judicial branch. Cantil-Sakauye shared a term she coined to reflect her vision: “Access 3-D.” The first element of three-dimensional access is remote access to the courts. Litigants should be able to file, search,


t of their own districts: how many judicial officers are employed, how many cases there are, where budgets have been cut, and how many people have been laid off. In those snapshots, the Judicial Council also tries to include anecdotal information to


but a significant amount of money had been spent on it, and it had been cited as wasteful. Several Keynote Address: Tani G. Cantil-Sakauye, Chief Justice of California    21
counties were prepared to implement it, but it would have taken millions of


filings have declined.
As for criminal filing fees, Cantil-Sakauye explained that current thinking posits that law
enforcement personnel were laid off during the recession, which led to fewer arrests and crimi -
nal reports and, ultimately, fewer fi


lawyers and litigants, noting the cost of such technol -
ogies as the CourtCall remote appearance system. Cantil-Sakauye acknowledged the possibil -
ity of adverse impact from such advances and affirmed that the judiciary cannot be improved on the ba


riefly summarize the empiri -
cal research that has been done on the economic impact of state courts and also to recreate the intellectual journey that NCSC undertook during 2009–2014 in attempting to help the courts with their funding crisis, by way


een little formal evaluation work done on these to quantify and validate their impact. Clarke went on to suggest that even in theory, their chief impact may have less to do with their economic effects than with their impact on rule of law. In turn, C


rts are likely to become increasingly reliant on remote technology to adjudicate cases in the service of cost-cutting efforts.
Eagly began by describing the basic concept of remote adjudication, which (in the immi -
gration context) involves a detain


tigants. However, she also went on to describe her qualitative work to investigate the impact of remote adjudication on immigration respondents who are being held in remote locations and to assess why those persons may become disengaged from the proc


bserved that this theme had already come up in bits and pieces in the earlier panel discussion in the conference. The pilot qualitative study that Greenberg and McGovern did looked specifically at the court systems in Utah, California, Massachusetts,


e states that have adopted it? What has the impact been on access to justice? What are the implications of remit -
tances from judicial fees and fines to a state’s general fund? Is funding for the courts more or less resilient for court systems in wh


 of the reasons why empirical research on courts
and litigation is difficult to conduct and to interpret across studies. One problem that he pointed out is the difficulty in simply designating a unit of observation, like a “case” or a “trial.” These


ative dispute resolution schemes. Kritzer agreed with the premise of the question and observed that the vast majority of trial-like proceedings and adjudications are happening outside the court system, as in the context of unemployment proceedings, S


Community in California Been Affected by the Funding Crisis That the Courts Are Facing?
Holden commended the court leadership for doing as much as they can under difficult cir -
cumstances and noted that the funding crisis has affected businesses dif


sinesses have, particularly in resolving complex cases in an efficient manner. In sum, Holden opined that there is a need for stronger funding by the legislature to the courts overall, including the complex litigation courts, to improve the timelines


tigation    35
Sandefur suggested that these dynamics create two basic problems for the courts. First,
there are high rates of default in civil matters, simply because people do not realize that they
are supposed to show up and do not understand th


of lack of diversity in the field of mediation, which he explained as resulting in large part from the difficulties for young people trying to break into the field. In turn, he described those difficulties as being exacerbated by submarket rates for


 to Sandefur,
asking her about the importance of the 92 percent of justiciable problems reported in her survey that nevertheless did not generate any court system involvement. The audience member asked whether the courts should be concerned about th


n a manner with which they are okay. Kichaven also pointed out that similar systems are being developed in Europe to handle other categories of civil matters beyond divorce. 39CHAPTER SEVEN
Final Keynote Address
The concluding address of the conferen


 we were kidding earlier, I think during the lunch break several of us [said], when’s
the last time you went to a cocktail party and somebody came up to you and said, “Let’s talk 40    Discount Justice: State Court Belt-Tightening in an Era of Fisca


e should take on a cause that had historical significance and merited continuity of leadership. We wouldn’t worry about who got the credit. We would be focused on the opportunity to serve and to make a difference over a period of time. And we chose t


ge, the most experience, the most perspective; and yet, for whatever reason, it remains one of our biggest challenges to figure out how to get our judges out into the com -
munities. In politics we say, “Make friends before you need them.” And I get


are in crisis. Courthouses must be
open to keep the public peace and tranquility. Courthouses must be open for the wheels of justice to turn.
We’re committed. We need to be more missionary-like, I respectfully submit, and enlist
more of our sister


3:00 p.m. Break

3:15 p.m.  Panel #4:  Access to Justice and B2B Litigation

Moderator: Geoffrey McGovern
• Rebecca Sandefur, University of Illinois College of Law
• Craig Holden, Lewis Brisbois  Bisgaard & Smith
• Jeff Kichaven, Jeff Kicha


Incorporating Resilience
into Transportation Planning and Assessment
SARAH WEILANT, AARON STRONG, BENJAMIN M. MILLER
SOCIAL AND ECONOMIC WELL-BEING
Prepared for the Transportation Research BoardLimited Print and Electronic Distribution Rights
This d


ation system assets: transportation activities, transportation outputs and outcomes, and, finally, socioeconomic outcomes. This logic model, together with more-traditional views of resilience, informed the development of the absorptive capacity, rest


neau et al., 2003, Framework ........................................ 79

     viii Tables
Table 2.1. Resilience Metrics, Based on Bruneau and Colleagues’ Components of Resilience .... 8
Table 3.1. Potential Hazards to Resilience ................


te more
aspects of resilience:
• Expand the objectives and scope of the framework to include shocks and stresses that are
not directly tied to climate change, including cyberattacks.
• Broaden the asset data to include human and equipment assets


n Research Board
TTTR Truck Travel Time Reliability
VAF Vulnerability Assessment and Adaptation Framework
VMT vehicle miles traveled

     xii
   1 1. Introduction
The Federal Highway Administration’s (FHWA’s) Vulnerability Assessment and Ad


luable locations. Overlaying these ideas on a more traditional characterization of resilience could contradict or miss key aspects of the value of the transportation system in times of stress or shock. Therefore, our approach for better integrating r


ting organizations, academics, and others who focus on resilience in transportation. This approach informed our conceptual framework and provided context for resilience that can be used to modify the implementation of the VAF. Our conceptual framewor


ach to resilience. Each of the AREA dimensions of resilience suggests a means through which to increase the resilience of the system, although by different methods. Importantly, the networked nature of the transportation system means that the aim of


er to glean insights from those facing different geographies and contexts, although some organizations did not respond or declined to participate. Stakeholders that were interviewed provided valuable information about how transportation planners and


pected new challenges to arise in the future, such as autonomous vehicle adaptation, coordination among organizations for long-term planning with a resilience focus, increased wear and tear on infrastructure, climate change impacts, and sustainable f


em while maintaining functions in the presence of a shock, and (3) reducing the time to recovery to normal functioning, which might be different from pre-event functioning. These themes translate into three capacities at the community or regional lev


 withstand and recover from incidents” (Tennessee DOT, 2015, p. 4). As
   7 noted by Dix and colleagues, 2018, p. 5, state DOTs and MPOs differ on how they propose to
improve their resilience capabilities: “Some emphasize the importance of system


lternate routes
Available capacity on alternate routes
Congestion on alternate routes
Availability of alternate routes, such as g raph theory connectivity score
Transit alternatives
Adjacent park -and-ride lots
Resourcefulness  Safety ser


ete qualities of those segments. Preferences among different qualities of the roadway segments are determined by the preferences incorporated into the model.
Planners care about how “critical” a roadway segment is in terms of usage, availability of


lanning process. The FAST Act adds resilience to sections of the U.S. Code (particularly Title 23, Highways, and Title 49, Transportation). For example, 49 C.F.R. § 5303 (a)(1), Metropolitan Transportation Planning, now seeks “to encourage and promot


n a major transportation route is closed. The Colorado DOT, for example, has been proactive in thinking about system resilience rather than solely focusing on hardening infrastructure. States also are required to develop a risk-based asset management


In particular, they highlight metrics focused on job access, such as cumulative opportunity and gravity metrics (discussed in Chapter 4). Jobs are not the only destination users might want to access. Nicholls, 2001, looks at access to public parks as


such as achieving environmental objectives or supporting emergency management capabilities. Indeed, 49 C.F.R. § 5303 (a)(1) states that resilient surface transportation systems should be encouraged “while minimizing transportation-related fuel consu


gions, such as
permafrost melt, coastal erosion, and volcanic activity, can affect both transportation planning and a variety of environmental benefits ( duVair, Wickizer, and Burer, 2003).
   14 Who Benefits from Transportation Infrastructure?


h mobility options to providing citizens with access to a system, even when that system faces shocks or stresses.
   15 3. Conceptual Framework
In this chapter, as well as in Chapters 4 and 5, we discuss our conceptual framework, which
expands


nsportation organizations are doing, specifically focusing on hazards associated with climate change and extreme weather events. We use the VAF as a starting point for incorporating resilience into transportation planning. As illustrated in Figure 3.


systems approach, the benefits of prioritized or
increased investments in the transportation system likely will increase the probability that the
transportation system stays at or near a preferred state of well-being. In this case, we would note th


ctivities). These outputs then lead to higher-level outcomes, which are system achievements, including desired or observed changes for the transportation system and in the context of the system of systems. As noted earlier, the left side of the figur


frastructure assets, roads, bridges, and tunnels that are vulnerable to risk. Information on the social system also is needed, including on education; health services; and economic systems, such as businesses, in order to understand the criticality a


tation system.
Transportation System Outputs
In an ideal system, the inputs and activities will achieve three main outputs. The first output is
access to critical social and economic systems. The transportation infrastructure provides access to
c


om transportation infrastructure; and those who lacked options with affordable fees for service for public transit and community-led ride sharing services, or personal vehicles. Collaboration and service provision for all populations are important el


ansportation
systems and the aspects of resilience that can be used to achieve well -being (Figure 3.3).
Specifically, we describe the capacities needed to improve the resilience of the system.
What This System M eans for Resilience and What Is N


of Resilience
As discussed in Chapter 2 and in Appendix B, Bruneau and colleagues’ 2003 framework defines
a resilient system as having the following characteristics:
• reflective: continuously evolving
• robust: anticipating potential failures
•


unity or by providing multimodal access to vulnerable and remote populations. Adaptive capacity investments could include adding roads that provide redundancy so that the system can handle more traffic, reducing potential congestion during both disru


red critical according to the Colorado DOT criticality index. In essence, criticality is how important a segment is to the movement of goods and people, while providing access to vulnerable populations. This is consistent with the definition of criti


e system. Additionally, the criticality of different assets and how they are distributed across the community provides a means for understanding equitable access to the transportation system. Long-term transportation planning should consider the suit


ible outcomes listed in Chapter 3. Assessing resilience first requires defining the desired level of service the transportation system should be able to maintain when faced with specific hazards that result in stresses or shocks. For example, transpo


resilience for different elements of
the transportation system (see Figure 3.4 in Chapter 3). Metrics based on existing or desired data and information can be mapped to each element of the logic model to help planners clarify what they are measuring


systems to make more-informed decisions that contribute to a resilient transportation system. The metrics described in this section include some of those documented in Chapter 2 and new metrics for consideration.
Measuring Inputs and Activities
In


 a cyber disruption interferes with their normal ability to regulate traffic flow. Planners also should consider the return periods of risks that are most relevant to their regions.
Transportation planners frequently invest in activities to increase


hysical resources set aside for known disruptions, such as snow, fire, cyber system disruptions, or congestion.
Having response plans in place can greatly improve the speed and effectiveness of efforts to
restore transportation capacity. Some of th


ined and
supported roles in disasters
Measures of
communities’ communication
capabilities  Amount  of time it takes to contact all community
residents in the immediate aftermath of an event
Percentage of community residents that can be
conta


tive Capacity Inputs  and Activities
Step of Logic
Model  Category  Sample Me trics
Inputs  Availability of alternate
routes and alternative mode choices  The distance to alternative routes
Friability , or the change in the population -weighted


nors’ Institute on Community Design, 2017.
Different groups might have different access to the transportation system during times of shock
or stress. Such metrics as heat vulnerability ind exes, as described in Madrigano et al., 2015, can
highligh


fferent times. Discussions of such metrics can be found in Venter, 2016; Governors’ Institute on Community Design, 2017; and Dix et al., 2018. These metrics are useful for assessing the absorptive capacity for traffic flow and congestion, either as a


l highway system
Pavement and bridge condition on local roads
Condition and availability of sidewalks, crosswalks, and bicycle
infrastructure
Reliability metrics  Measures of resilience (Zhang et al. , 2010)
Truck Travel Time Reliability (TTT


es in travel costs, travel time, or travel speed before and after an event. Absorptive capacity is concerned with keeping this change as small as possible, while restorative capacity is concerned with having
any changes return to normal operating le


given area
or on a given route
Number (or rate) of alcohol -related incidents  that occur
each year in a given area or on a given route
Number (or rate) of tr uck-related incidents that occur
each year in a given area or on a given route
Sea


this problem, it is important to measure the cost of transportation and cost of living for system users, particularly vulnerable or low-income populations. Such metrics include measures of housing costs, single-mode or multimodal transportation costs


e the hazards that will be considered as part of
the vulnerability assessment. The VAF is designed to consider climate change vulnerability but it can be expanded to consider shocks and stresses that are not directly tied to climate change. In


ncrease the resilience of the system. By having alternate routes or modes available or by expanding capacity, an individual asset would become less critical to the functioning of the system as a whole. This would increase the resilience of the system


ties. The AREA approach to resilience highlights approaches
that subject-matter experts could use to incorporate resilience from a systems-level perspective. Given the subjective nature of the stakeholder input process, many of the suggested improve


is
difficult to analyze as a whole but that should be considered through the AREA approach. Thus, capacity and equity concerns can be considered jointly, and solutions can be identified that take into account the different aspects of the AREA approa


y could be more cost-effective and increase the resilience of the entire system. The focus should be on the resilience of the network rather than of an asset. Having options across the spectrum of capacities is important, given the scarce resources t


ience into the transportation system requires a change in perspective from
protecting every asset to a systems-level view. It is not simply the direct transportation infrastructure
that is important in building resilience; it is also the human and


e costs and benefits of resilience when making long-term investments in highway and transportation infrastructure. The interviews also provided insight into how stakeholders think about and understand transportation resilience, which might influence


el DOTs  and MPOs. Invitations
were emailed to participants; the recruitment email is included at the end of this Appendix.
Final Sample and Interview Timeline
We conducted nine interviews at eight organizations over three months. We acknowledge


more than one conversation, as well as detailed examples of problems, successes, or recommendations related to planning for transportation resilience.
What Stakeholders Are Saying About Transportat ion Planning  and Resilience
In this section, we


or all transportation
projects that receive federal funding ( U.S. Department of Transportation, Federal  Transit Administration, 2019).
   53 current status of roads, their maintenance, and future construction. MPOs also might interact with
mul


erations for resilience.
Finally, stakeholders noted a few additional influencers in transportation planning and
investment with whom they might interact only tangentially or not at all. This includes such organizations as local or grassroots advo


events, was mentioned. Stakeholders noted that it was important that this information be shared with MPOs in all locations affected by hurricanes to help plan for future stresses and shocks.
In terms of interaction and communication with other sect


gn, it will be important to understand the implications of those options in a broader, long-term context.
Transportation Funding
With the exception of pilot projects funded by FHWA (U.S. Department of Transportation,
FHWA,  2018c), none of the in


recreation,
daycare, [and] senior centers. You can’t get anything done in the community if [the]
system isn’t functioning properly.
Transportation planning and infrastructure influence the location of housing facilities, utilities,
and other ele


 because of changing U.S. tariffs
• jurisdictional overlap, lack of clarity about who owns certain road infrastructure and who can—or who understands the need to—act to address issues that affect multiple jurisdictions
• unclear roles and responsi


work in
order to understand how transportation can affect the resilience of bigger systems and entire regions. Finally, one stakeholder noted that transportation resilience is the ability to learn from experience to better respond with the assets pr


ods. Increased communication with the public and improved warning systems would provide them with the ability to make more-informed decisions about what to do in certain events.
Data and Measur ement
The stakeholders we interviewed were not collec


ent resilience strategies for the short, medium, and long term
• the need to understand the connectivity between transportation programs and systems during events so that appropriate plans result in sustained movement of people and goods
• the ne


ortation resilience and improving opportunities for MPOs to consult each other for guidance on how to meet those standards. The stakeholders suggested
that the standards should extend beyond transportation agencies in their applicability and
should


pant name],
Hello, my name is [name] and I am a/an [title] at the RAND Corporation (www.rand.org). I am
conducting a study on behalf of the Transportation Research Board (TRB) to help them develop an analytic framework for the Federal Highway Admi


participation are minimal.
Do you have any questions about our confidentiality procedures before we begin? [If yes, respond
to all questions. If no, proceed with discussion.]
General Background Questions
1. Can you describe your role at [respond


ticipate, prepare for, and adapt to changing conditions and withstand, respond to, and rapidly
recover from disruptions, along with a reduction in overall vulnerability.]
Based on that [if not already answered],
12. What do you want to be resili


urces based on
searches using widely available online databases and keywords relevant to each of the topic areas.
We searched Google Scholar and Web of Science using the following keywords: “resilience,” “resilience framework,” “resilience conceptu


red by the movement of the system from one equilibrium state to another, and the length of time it takes for the system to rebalance once the stressor has been removed. Holling, 1973, was the first to transfer these ideas from the physical sciences t


capacity to deal with future stressors, whereas mitigation efforts are limited to minimizing and repairing damage after the event (i.e., recovery).
Components of Resilience
There is a growing convergence of the definitions of resilience used in di


cal
system  The capacity of a system to absorb disturbance and reorganize while undergoing change so as to retain essentially the
same function, structure , and feedbacks —and therefore  the same identity
Adger, 2000 Social  The ability of commun


 Norris et al. , 2008, p. 129.

   71 Resilience Frameworks in the Literature
Many of the nuances in definitions of resilience arise when developing frameworks for
analyzing resilience and community risk. A system-of-systems approach disaggreg


conomy and society, and
infrastructure and environment through the seven qualities of resilient cities described earlier.
This framework integrates the individual systems through various channels.
Figure B.1. The City Resilience Framework

Acco


he system is dysfunctional at this point, it can, again, take one of two  paths. The system’s ability to adapt to the changed environment, together with the
system’s ability to recover, determines whether the system can adjust to the changed environ


em sends to firms and households. In other words, how individuals and

   75 firms behave before, during, and after a shock (or in the presence of a stressor) are mitigating
factors that influence the resilience of the system.
Although Rose, 2004


base
4. policies
5. implementation
6. monitoring and evaluation
7. internal consistency
8. organization and presentation
9. interorganizational coordination
10. compliance.
The first six principles contribute to the seventh, internal consiste


incorporated into a larger, community-level analysis that considers the joint determination of the larger system. In addition to those two points of analysis, the framework explicitly incorporates decision support as a subsystem within the larger sy


o other interdependent systems—to that subsystem. This approach is very similar to that of Rose, 2004, and others, who have used input-output and computable general equilibrium–type models that can simulate the interdependencies in the supply chain.


 actors, environments, purposes ,
and disciplines involved.  Instead, the landscape of resilience indicators is just as
diverse as the systems, communities, or disasters that are studied .
It is difficult to develop a single system of metrics that


ional, community capital, environmental, educational, and health systems that were identified in Cutter, 2016.
In addition to the systems discussed in the aforementioned reviews, there are various
resilience indexes that have been considered. For


 of the indicator systems is that if the subsystems are resilient, then the
system is resilient. This is a significant assumption and one that misses a point often made in the
resilience literature: There are cascading consequences across the syste


atrix that summarizes the strength and presence of a connection between two subsystems can be used as a weighting matrix in the development of an index of resilience that comes from indicators of resilience of the subsystems. Rather than relying on s


, pp. 1403–1409.
Adger, W. Neil, “Social and Ecological Resilience: Are They Related?” Progress in Human
Geography, Vol. 24, No. 3, 2000, pp. 347–364.
Ahmed, Rashid, Mohamed Seedat, Ashley van Niekerk, and Samed Bulbulia, “Discerning
Community R


s of Coordinating Human
Service Transportation and Transit Services, Washington, D.C.: Transportation Research Board, Report 91, 2003. As of August 5, 2019: http://onlinepubs.trb.org/onlinepubs/tcrp/tcrp_rpt_91.pdf
Butler, Lisa D., Leslie A. Morlan


8–606.
Cutter, Susan, Bryan Boruff, and W. Lynn Shirley, “Social Vulnerability to Environmental
Hazards,” Social Science Quarterly, Vol. 84, No. 2, June 2003, pp. 242–261.
da Silva, Jo, and Braulio Eduardo Morera, City Resilience Framework, Unit


rd, NCHRP Synthesis 527, 2018.
   89 Francis, Royce, and Behailu Bekera, “A Metric and Frameworks for Resilience Analysis of
Engineered and Infrastructure Systems,” Reliability Engineering and System Safety, Vol.
121, January 2014, pp. 90–103.
G


ty and Emergency Management, Vol. 6, No. 1, 2009, pp. 1–48.
Kimhi, Shaul, and Michal Shamai, “Community Resilience and the Impact of Stress: Adult
Response to Israel’s Withdrawal from Lebanon,” Journal of Community Psychology, Vol.
32, No. 4, July


North America,” Transport Policy, Vol. 37, January 2015, pp. 167–176.
Marshall, Wesley E., Alejandro Henao, and Rachel Bronson, Building a Framework for
Transportation Resiliency and Evaluating the Resiliency Benefits of Light Rail Transit in Denve


gy Use and Greenhouse Gas Emissions: A Longitudinal Study from Vancouver, Canada,” Transportation Research Part D: Transport and Environment, Vol. 62, 2018, pp. 715–725.
Nicholls, Sarah, “Measuring the Accessibility and Equity of Public Parks: A Cas


 Act, Washington, D.C., December
4, 2015.
Public Law 115-254, Federal Aviation Administration Reauthorization Act, Division D, Disaster
Recovery Reform, 2018.
   94 Putnam-Walkerly, Kris, and Elizabeth Russell, “What the Heck Does ‘Equity’ Mean?


enter, The Wharton School, University of Pennsylvania, Working Paper No. 2018-01, March 2018. As of August 5, 2019: https://web-oup.s3-us-gov-west -1.amazonaws.com/showc/assets/File/
CIRI_Tonn_Improving%20US%20Transportation%20Infrastructure%20Resil


in/erelief.cfm
U.S. Department of Transportation, Federal Transit Administration, “Transportation
Improvement Program (TIP),” webpage, March 11, 2019. As of March 11, 2019: https://www.transit.dot.gov/regulations-and-guidance/transportation-plannin


CORPORATIONResearch Report
The Welcome Baby Program
An Implementation and Outcomes
Evaluation
Sarah B. Hunter, M. Rebecca Kilburn, Teryn Mattox, Jill S. Cannon,
Terry Marsh, Melissa Felician, Maya Buenaventura, Lauren Davis,
Matthew CefaluLimite


icy concerns that are influenced by social
and behavioral actions and systems that affect well-being. For more information, email
sbp@rand.org.
RAND Social and Economic Well-Being is a division of the RAND Corporation that seeks to
actively impr


 Question 1: To What Extent Are Sites Implementing Welcome Baby to Fidelity?  ............. 19
Evaluation Question 2: Is There Variability in Site s’ Ability to Reach Fidelity to Welcome Baby?  ..... 23
Evaluation Question 3: How Are Sites Maintain


ty are born healthy, maintain a healthy weight, are safe from abuse and
neglect, and enter kindergarten ready to succeed in school and life (F5LA, 2014a). Over the last
decade, F5LA has developed a unique approach to supporting families through a t


n was designed to help F5LA decide how to best allocate resources for
training, program monitoring, and other program management activities. The evaluation findings
are also likely to be valuable to other home visiting programs, as the fidelity com


ndated.
F5LA developed a Welcome Baby Fidelity Framework (see Table S.1; Appendix A) to guide
the implementation of the program, and the framework specifies central components of the
program in terms of staff training and expertise, number and ti


ent
Focus
Groups  Staff
Survey  Referral
Documents  Client
Survey  Stronger
Families
Databasea
1. To what extent are
sites implementing
Welcome Baby to
fidelity?    X  X X
2. Is there variability in
sites’ ability to reach fidelity to


aining Community Resource and Referral Networks? What, If
Any, Gaps Exist in These Networks?
The completeness of referral directories varied widely across sites. Very few sites had
developed the organizational infrastructure to facilitate successf


 leaving the program.
6. To What Extent Do Participants Achieve Short- and Intermediate-Term Outcomes?
We examined Welcome Baby participants’ outcomes across all sites and visits for 12
outcome measures. Where regional or national benchmarks were


 is particularly important because reflective supervision was related to several positive participant outcomes, such as well-child
visits, immunizations, home safety, and safe sleeping practices. Reflective supervision also
appears to negatively re


pment of
detailed protocols for all high-priority referral types, including public benefits, alcohol, smoking, and drug treatment (all protocols should include the provision of client confidentiality). F5LA
should also support sites in developing a


 Agniel, Mollie Rudnick, Christian Lopez, Patricia Lozano, Anamarie Auger, and Lance Tan. Rachana Seelam and Rodger Madison were the study
programmers and assisted with the data safeguarding plan. We appreciate Tiffany Hruby’s help
preparing this d


the quantity of births and diversity of families, it is not surprising that child and family
outcomes are highly variable. There have been gains in some areas over the last decade—for
instance, the rate of third-grade students scoring at or above p


s to use a
family-centered, strength-based approach to accomplish the following objectives: ensuring
parents can provide enriching structured and nurturing environments, strengthening parental
resilience and self-efficacy, ensuring that children a


act evaluation should be conducted.   3 Figure 1. 1. Welcome Baby Logic Model: Family & Child Level

SOURCE: F5LA, “ First 5 LA Welcome Baby Logic Model: Family & Child Level,” fact sheet, undated.
NOTE: PHQ = Patient Health Questionnaire; ASQ =


dy of the early implementation of the Welcome Baby expansion in the
remaining Best Start sites was conducted in March 2014. Through semistructured interviews with a variety of key stakeholders, researchers found that stakeholders were enthusiastic a


 our approach to examining
implementation and outcomes, results from the evaluation, implications, and conclusions.
Chapter 2 provides an overview of the Welcome Baby program. Chapter 3 describes the data
sources used for each of the evaluation qu


lp them better
connect with their child.
Soon after delivery, a Hospital Liaison conducts a risk assessment with women interested in
receiving home visitation services. The level of risk identified through the Modified Bridges for Newborns Screen


1
Asian 1.92
Mother ’s risk  factors
Average Bridges Score  (standard deviation)  43.27*
(16.33)
Percentage first time mothers 36.60
Child ’s risk factors
Percentage gestational age < 37 weeks at birth  7.26
Percentage low birth weight


treach Specialists develop and maintain key community contacts, conduct client intakes, and follow up with potential clients.
Figure 2.2 depicts an organizational chart for a typical site.
Figure 2.2. Typical Welcome Baby Site Organizational Chart


m families, the Welcome Baby pilot study, and extensive meetings with other
stakeholders (LABBN, 2008).
Table 2.3. The Welcome Baby Fidelity Domains
Fidelity Domain  Description
1. Staff Qualifications  Staff meet minimum requirements
2. Staf


of Welcome Baby stakeholders to ensure that multiple perspectives are included
• pilot test all instruments before deploying
• design data collection, analysis, and reporting procedures to facilitate sustainability by
Welcome Baby stakeholders for


ion on the staffing- and service delivery–related domains,
and client focus groups helped to explain variation in the participant-related domains (e.g., participant perception of the relationship).
 14 Table 3 .2. Quant itative Data Sources That


on allowed us to collect qualitatively rich
information about staff perceptions of support for Welcome Baby program implementation,
including factors that staff perceived as influencing program delivery. Key domains in the
interview protocol inclu


variability in sites’ abil ity to reach fidelity to Welcome
Baby? If so, what factors account for this variability?

Topic areas (components of
fidelity framework)
 • staff qualifications
• staff training
• supervisory requirements
• ref


m outcomes?
7. What are the relationships between program fid elity outcomes
and participant outcomes?
Topic areas (components
of fidelity framework)  • parent perceptions of the relationship with the Welcome Baby
providers (part of fidelity


ut every site achieved fidelity in at least two domains (supervisory
requirements and participant perception of the relationship). Individual sites achieved fidelity in
18 to 80 percent of measured domains, with an average of 48 percent.
Methods


, Parent Coach
Supervis ors, and RNs met minimum
client engagement requirements,
based on number of visits in  last week
worked prior to survey  78% of staff
D7. Hospital
Enrollment  Site reports
(12 months
per site)  January–
December
201


y
shading and “N/A” indicates we did not have data for that site and domain. Additionally, we
exclude from the summary the three domains for which we did not have available data for site-
level analysis (domains 6, 12, and 14). Eight of the 12 site


ts and participant perceptions of
the relationship), all assessed sites achieved fidelity thresholds with the Welcome Baby model;
three sites achieved one domain each (one site met staff qualifications, one met reflective
supervision, and one met


alysis. Data were coded independently by research staff using a
 24 constant comparative process. Initial coding was done on notes from 25 of the earliest interviews
conducted, using a conventional content analysis approach and allowing the conten


 the 11 measured domains
(domains for staff qualifications, reflective supervision, and hospital enrollment), only one site
met fidelity thresholds. For one domain (participant perception of the relationship), all 12 sites
met fidelity thresholds;


blic
health certification and six months of home visiting experience. That’s hard to find, especially the home visiting experience. It would be easier to get [RNs] if they could
shadow for training and get that home visiting experience that way.


allenging because on
training days that are back to back it really cuts into seeing clients. We have to see
32–40 [clients] and if I miss two to three days in a month for training, I can only see
32 at the most. It cuts into my numbers when we hav


is supervisory
requirement.
 28 Domain 4: Reflective Supervision
Across the five staff positions, an average of 61 percent of all staff met minimum individual
and group reflective supervision requirements in the past three months (Table 4.5). O


rcent
meeting workload requirements (Table 4.6). We note that threshold achievement rates were
calculated based on survey responses for the number of client visits conducted in the last week
worked, which we multiplied by four to determine number


three to four clients a day and
stretching themselves.
Domain 6: Prenatal Enrollment
Although we were unable to assess whether sites met the fidelity threshold on this domain
using quantitative data, staff were asked in the interviews whether th


cember 2017 .
Domain 7: Qualitative Findings
Consistent with the quantitative findings, staff reported more challenges in missing some
clients at the hospital (the “approach” requirement of this domain) than in enrolling families once
approached


 losing a client.
Although the largest drop-off in program participation occurs between the hospital and home
visits, we did not find consistent themes addressing this time point.
Domain 9: Timing of Service Delivery
The fidelity threshold was b


er examples about these points).
Domain 10: Referrals to C ommunity Services
The fidelity threshold was based on rates at or above the Welcome Baby–wide average,
which was 25 percent of referrals as reported in SFDB data. We included referrals ma


a are not subject to external validation. Additional information
on how this measure was specified can be found in Appendix C. Most recommended content
items from the Welcome Baby Fidelity Framework were captured in the SFDB data, but in a few
cas


ted documents, we
 37 identified gaps in referral networks or procedures that may be preventing the proper
implementation or successful completion of referrals.
Using this approach, we requested the following documents from each site’s Program
M


ral types.
 39 Table 4.13. C ategories of Information Gathered for Primary and Secondary Referral Types
Document Type  Primary Referral Types  Secondary Referral Types
Referral directory  Contact information present in a centrally
maintained


gure 4.2). One site did not
have a referral directory, while two additional sites had referral directories with less than 50
percent of categories covered. Across the primary referral types, with respect to smoking
cessation and SSDI/Supplemental


l types
 42 consideration indicates the presence of a clear and present danger to an individual or to
others (Alliance of Information and Referral Systems, 2017).
• Role of home visitor in making referral: Does the protocol require staff to assist


re not made because there were no resources in the
community to refer families. The most common responses were the lack of mental health care
resources, especially for clients with Medi-Cal or with no insurance, and limited housing service supports


obtain qualitative feedback, we conducted 20 focus groups (in Spanish and English) that
included 140 participants across the 12 sites. Similar to the site interviews, field notes taken
during the focus groups were reviewed against the digital recor


 to the nine-month visit). The size of the sample across some of the sites precluded us from running
statistical tests, but the observed average values and standard deviations showed consistently
positive scores with low variability across the site


 requires a lot of
information. Here with Welcome Baby it was fast and did not take much time.
• It was pretty fun, she explained a lot of cool things about the program.
• It was a good experience, the lady who enrolled me was really nice, and she


 a question—it would be much easier.
• I would like to get the information in a way that has more visuals. The book that we get
has too much information [that is difficult to digest].
Another commonly expressed program improvement recommendation


g me with information.
Program participants also felt that Parent Coaches gave them the confidence they needed,
often assured them that they were doing a good job, and cared not only about the baby but also
about them (as moms) and about their oth


ient or visit characteristics were associated with continuing to the next visit.
Methods
The dependent variable in the model was the visit. The visits considered were
• the RN visit
• the two-to-four-week visit
• the two-month visit
• the thre


ferent types of families. The fidelity variable was measured at the previous visit—for example, staff qualifications at the hospital visit for the “hospital to RN”
transition analysis. Not accounting for these differences may bias the estimates of t


ns.
Relationship Between Family Characteristics and Program Retention
We examined demographic/family, birth, and visit characteristics to better understand
program retention. Across all visits, receiving prenatal services was the only characterist


–0.31 –0.17 –0.17
Age >34 –0.02 0.00 0.26 0.13 0.19
First birth  0.11 –0.10 –0.01 –0.09 –0.08
Low-birthweight
child 0.28 0.26 –0.22 –0.21 –0.26
Birth complications  0.26 0.15 0.19 –0.19 –0.19
Previous Visit  Variables
Worked full  time 0.08 N


sit Transition
Fidelity Component  Hospital to
RN Visit  RN to Two -to-
Four -Week
Visit Two-to-Four -
Week to Two-
Month Visit  Two-Month
to Three -to-
Four -Month
Visit Three -to-
Four -Month
to Nine-
Month Visit
D1. Provider met
minimum


nsight into how Welcome Baby clients fare relative to these
external benchmarks. While we do not have the data to assess the degree to which the samples in
these other data sets are comparable to the Welcome Baby sample, they use similar age groups


About how often does your child have
a chance to get ou t of the house?  5 = every day
4 = 4 or more times a week
3 = a few times a week
2 = about once a week
1 = a few times a month or less  Child gets out of the house at least 4
times pe


 measured using a modified version of the
KIDI (MacPhee, 1981). Several items on the KIDI were modified, and many items were
removed entirely from the survey at the request of F5LA, such that the full KIDI scale was not
kept intact. The full list


ternal depression,
using data from the SFDB, the Welcome Baby administrative database described in Chapter 3.
Family planning was measured using the home visitor’s report of whether the client used any
form of family planning or contraception. Th


e 5
Site 1Site 9Site 6Site 7Site 3
Site 8
Site 10
Site 4
Site 12Site 11
Site 2
PRAMS (2004– 2006) NSFG (2006 –2010)
 63 nurse should administer the PHQ-9 (Kroenke, Spitzer, and Williams, 2001), which is a more
detailed evaluation for assessing mild


h visits (36 percent in
Welcome Baby versus 38 percent in the LAMB), and slightly more likely to be incorporating
breastfeeding in some capacity into their infant feeding routine than the average for Los Angeles
County at both the 72-hour (93 perc


e was no recent benchmark for children’s insurance coverage at hospital exit, or at a specific
time in the first year.
Figure 4.12. Percentage of  Clients Whose Infant Is Covered by Health Insurance, by Visit , with Los
Angeles County Benchmark


home safety inspection was completed with no issues identified. Home visitors then provide
tailored home safety education to the client according to needs identified.
The vast majority of home safety issues identified fall in the category of inade


nchmarks

SOURCES : SFDB  for Welcome Baby ( n = 4,958, January 2016– December  2017); Los Angeles County Department
of Public Health for LAMB, 2014 (n  = 6,035) .
Child Development
As part of the Welcome Baby curriculum, Parent Coaches adminis


 For the five
postpartum visits beginning with the RN visit, we estimated a set of models for 12 outcomes
with these fidelity components. The outcomes were positive parenting practices, parent
knowledge of infant development, use of family plannin


nt for
clustering. For the few continuous outcomes in our analysis, we used the mixed command in
STATA. Also as described in the methods for Evaluation Question 5, we used the Benjamini-
Hochberg procedure with a false discovery rate of 0.10 to adj


taff training
and workload standards were associated with lower rates of family planning. Further, meeting
the home visitor workload fidelity threshold was associated with lower use of family planning
for the last four visits, one of the most cons


e
relationships across all the outcomes for the five visits.
Overall we find little evidence to support a relationship between the fidelity components and
client outcomes. This finding could be because of a true null effect or to the analyses bei


idelity in at least two domains  (supervisory requirements and participant
perception of the relationship). Within each domain, between 18 and 80 percent of sites met the fidelity threshold.
2. Is there variability in sites’ ability to reach fidel


but a lower likelihood of staying in the program for later visits.  For the
2-to-4-week visit and later visits, adherence to Welcome Baby fidelity standards is related to lower rates of
participant s leaving the program.
6. To what extent do part


here the sites on average scored well, but we did not have any
benchmarks to compare to the Welcome Baby site performance, including
•parenting knowledge
•immunization rates
•home safety.
 78 Recommendations for Improving the Welcome Baby Program


frequently, and staff across positions and sites
questioned the quality and value of reflective supervision. Since reflective supervision appeared
to negatively relate to retention between the hospital and RN visit, an examination of the quality
o


ompleting four or more postpartum visits, there are potentially lessons
learned from staff at the high-performing sites that could help improve performance at other
sites.
Referrals to  Community Services ( Domain 10)
Site performance on this f


ition (i.e., Evaluation Question 5), we examined both
participant characteristics and performance on the fidelity domains. The practical value of the
findings from these analyses is twofold. Knowing which family characteristics are associated
with