 court order provides a Yahoo
email address for an individual who was to meet the victim at a casino shortly before he was murdered. Given the tenuous tie to this email address, the investigator wonders what legal authorities might be utilized to re


, she does not know what apps may have been on the POI’s phone. To pursue this line of attack, she returns to MIKE’s homepage and again uses MIKE’s query wizard to query for all mobile apps that collect GPS location.
13 Th  is query (Figure 2.23) tur


ed—the courts, law enforcement, and the commercial enterprises.
All the data MIKE used in this illustration, and the illustrations that follow, were derived
from previous analyses (independent of any particular investigation) of the commercial enti-


esponses to requests from those who lack a good grasp of this mate-rial may be delayed because the commercial entity or the courts find them to be overly broad or improperly specified. Here, MIKE could be used in two ways for training purposes. First


 the public and the press become increasingly concerned about geolocation, a state
legislator asks one of his staffers to learn more about geolocation and privacy, to figure out how
to best address the issue. This staffer is unfamiliar with the pri


er to a page describing how ECPA regulates law enforcement access to basic subscriber information (Figure 2.28). She notes the type of information protected by ECPA in this instance (e.g., a street address) and realizes that this is different from th


To determine what kinds of protection this law provides, the analyst then clicks on the
link in Figure 2.30 for the required legal document. Th  is brings her to Figure 2.31, which explains the requirements for obtaining an order for installation of


w private corpo-rations, not just law enforcement, use information. For each app, she also investigates the types of encryption used, the retention length, and resolution.
Th  e analyst recalls the article she read that described the criminal investi


cts her to the page in Figure 2.38.
From this description, she notices that cell site location data not only seem less specifi  c
in general but that the level of specifi  city of the data varies. However, Geoloqi is capable of providing fairly deta


o
explore uncertainty and diff  ering opinions in the law. For example, if a privacy advocate adds case law suggesting a broader interpretation of the Fourth Amendment, an assistant district attorney could be motivated to add information about the c


 hired principally to manage the company’s patent portfolio, although she did craft the Protected Gardens privacy policy. WaldGardn Figure 2.37
Geoloqi
RAND RR1482-2.37
Geoloqi’s
Website
Types of Location
InformationExplanation of Geoloqi
Figure 2.


e request and be reimbursed for the work required. He also wants to know what he can tell his customers.
The Analysis
LG assumes that the guidance from her CEO means that staff members are to help if they can (i.e., do not fight the order) but not to


s information than it gives to advertisers, so no notifi  cation of custom-
ers should be necessary.
Th  e state investigator swallows hard, agrees to pay WaldGardn the $15,000, and two
weeks later is rewarded with the names of two Gardns whose (as


 rewarded with several corporate guide-
lines for law enforcement (Figure 2.44).
In her own legal research, LG uses policies gleaned from guidelines other corporations
have issued. After using both MIKE and traditional legal resources to analyze the


ages
data context for WaldGardnUser’s Guide to MIKE    47
Summary
Th  is example has illustrated how MIKE can support the commercial sector of the mobile
ecosystem by providing guidance for how to respond to court orders and reducing the time devot


th
what MIKE is and how to use it. For more information these topics, see Chapter Two.
Curating and Updating MIKE
Entering Data and Maintaining the Map
MIKE is flexible in how data can be entered. A user can focus on entering legal statutes, cases,


typing, a list of previously used inputs will appear in a drop-down
Figure 3.1
Obtaining Information About Entries on Forms
RAND RR1482-3.1
Hovering
Mouse Over
Blue Question
Mark Opens
Pop-Up with
ExplanationCurator’s Guide to MIKE    51
box.2


n about how to inter -
pret ambiguous situations. The law is particularly problematic in this regard. Law enforce-ment professions and privacy advocates often disagree about policies and can cite different court decisions to support their points of v


data contexts54    Helping Law Enforcement Use Data from Mobile Applications
Structured information is added via a series of labeled boxes just below the “General
Information” box. Many of these are self-explanatory, with places to enter the name o


text is accessed in different ways, depending on whether the user is adding a new data context or updating an existing one. A user wanting to add a new data context must do so on the associated mobile app’s page (as described earlier). Once added, th


 Encryp
tion is for describing how the data are encrypted. If no information is available on
whether the data are encrypted, this row should be left blank. A user who knows what kind of encryption is used but is unsure how to describe it can press


ead, MIKE populates appropriate legal information automatically based on the parameters entered in the data context. Figure 3.6 shows a completed data context page. Th  e legal information is in a box on the right. Th  is legal information is not man


specifi  c details about the fi  rm itself. “Activities” will generate a drop-down list of suggested inputs if the user presses the down arrow on the keyboard.
5 Real name  refers to how an app characterizes the data type. Of course, that name may be


nformation about the fi  rm based on other entries already in
MIKE. For example, even though the user did not specify the mobile apps Facebook operates
in the form in Figure 3.9, a table describing these apps automatically appeared on the page in F


itions in it, and each defi  nition links to a separate page pertaining to that defi  nition. Th  e user can then add a description of how the term is defi  ned to the defi  nition page.
Finally, the user can enter the diff  erent legal contexts. Eac


ular type of entity they can obtain it from. Correctly defining legal contexts is crucial for ensuring that MIKE connects the appropriate legal information with the appropriate technical information. Figure 3.14 shows the form used to update the lega


ry and case law.
• Policy issue discussed  allows specifi  cation of any policy issues that the user thinks are impli-
cated by this particular legal context. Th  is enables MIKE to create pages capturing all statutes and cases related to a particula


ded as
Unstructured
InformationCurator’s Guide to MIKE    67
statute and the name of the document being described (separated by a slash mark). Figure 3.15
is the form for updating the entry for “Court Order Authorizing Intercept.”
Each form provid


s to webpages outside MIKE that pertain to that situation. However, once the user saves any changes, the completed page referring to each law enforcement situation also includes tables summarizing the relevant statutory and case law (Figure 3.17). Th


 to the court that issued the ruling on the particular case being added to
MIKE. A drop-down menu is available to assist in adding the relevant court.
• Constitutional provisions interpreted  refers to the constitutional articles and amendments
dis


E if changes to the structure of MIKE negatively affect MIKE’s functionality.
Instructional Videos
The MIKE prototype includes a number of instructional videos to help new users understand the system. These include a broad overview of what MIKE is, a


el 4 data are sent from the app server to some other server. Users are unlikely to be
aware o
f this data transfer, and there is no way for them to know that it occurs (aside
from a possible mention in the firm’s privacy policy). The transfer is on


lt it is to extract the information about an app using the particular source. The “Technical Difficulty” column is for the programming skill necessary to use the data source.
The “Scope” and “Coverage” columns describe the total number of apps that c


re to capture all external data transmissions. TaintDroid (Enck et al., 2010) is an example.High High Full Complete
Investigative reportsReports have shown which data are transmitted by particular apps (see, for example, “What They Know—Mobile,” 2013


file. This will not say what data are being sent. However, APIs usually care about only a few things (device ID, ZIP code, GPS location, etc.)Low Medium to
highFull Possible
App to API data setSome companies have lists of which apps use which APIs.


ctions against government searches—although this is probably too ambitious and may overwhelm every user of MIKE except for policy analysts.
Broadening the scope of legal information has two advantages. First, MIKE will become
more useful to its inte


esearcher to develop
coding procedures prior to analyzing the documents. These coding procedures are rules for assessing the meaning of the laws: Rather than reading cases individually to determine their substance and importance, the researcher eval


ny different laws, to allow the user to research and com -
pare laws across jurisdictions.
Addressing Conflicting Interpretation of the Law
Although electronic surveillance law is evolving rapidly, there will always be areas of ambiguity where applic


mobile app ecosystem can be difficult to obtain. Fully populating MIKE will require researchers with expertise in technology, commerce, and law to identify and exploit potential sources of information. Particularly as MIKE is being initially develope


s—could be extended to use Wikipedia’s moderation tools (e.g., blogs to dispute the accuracy or completeness of content). The moderator or editor needs to be neutral so that valid alternative interpretations of ambiguous data by stakeholder groups ar


ormation is stored within the devices or by the commercial entities and how law enforcement can use that information. MIKE also needed to convey the equally com -
plex arrangement of federal, state and local laws, and the jurisprudence that governs a


epiction is likely to be brittle. While static maps may be directionally correct, they are not easily extended in response to changes in the technical, commercial, and legal concepts they describe and are ill-suited to the aims of this project. Howev


rket segments and representative firms to further investigate.Architect’s Guide to MIKE    83
From the academic world, one deserves special note (Basole, 2009a; Basole, 2009b). Basole
aggregated two main data sets of relationships between firms and


ide to legal issues directly related to law enforce-
ment searches of the mobile devices, apps, and services across a selection of state and fed -
1 The four approaches Kerr, 2007, describes are as follows: (1) the probabilistic model, (2) the privat


set of these views.
The prototype’s purely text and hyperlink approach forgoes graphic representations of
information and relationships. Graphics provide powerful ways to understand information and relationships, and it should be feasible to extract


rs to define additional relation-
ships between data objects
(e.g., documents and/or webpages) to create a knowledge structure
that allows computers to
extract useful information automatically. Fundamentally, semantic
web technologies work by
li


at is required for it to be displayed on the new owner’s product list and deleted from the prior owner’s product list.
Much more complex queries based on the relationships stored in MIKE are possible. For
instance, one can generate a display of all


ls about sharing emerge, e.g., through a subpoena, it would be useful to capture this informa -
tion to inform others that the data are, in fact, not shared. MIKE is a proof of concept, and this refinement is something that should be consider by an i


elationship Diagram
7
3615 16
186
1413
1917
5
31
211
128
109
95
4
RAND RR1482-4.2Data type
Resolution
Legal context StatuteLegal
de/f_initions
RetentionThird-party
libraryFirmEncryptionRequired legal
documentLaw enforcement
situation
Storage modeMobi


ther the statute applies to data stored locally or remotely.
15 Has required legal documentIndicates what legal document is required to lawfully gain access to the data protected by the statute.
16 Has applicable situation Specifies what legal situat


 hierarchy  includes the types of services firms perform in the mobile ecosys -
tem.
• The product
s hierarchy  includes physical or digital things traded in the ecosystem.
• The organi
zation hierarchy  includes social groups.
• The context
 and con


his hierarchy organizes the activities that firms engage in. The purpose of the activity catego -
rization is to link firms engaged in specific activities to legal statutes containing rules applicable to that activity. Sometimes, that linkage is dire


 interfaces, and the mobile apps.
• Legal goo
ds organizes federal and state-specific statutes and legal process documents.
• Physic
al goods organizes the physical platforms, such as the cell towers and the makes and
models of phones.
Organizationa


pts in the future.
All collected data items are assumed to have both a data context and a legal context, with
some conditions overlapping the two contexts.
Miscellaneous
Since MIKE is implemented using open-source software supporting web apps, it al


n and various rules governing access to this infor -
mation may lead to inefficient requests for information, thus requiring them to expend unnec -
essary time and money searching for information. Policy advocates were interested in a way to demonstr


ll represented. In contrast, policy advocates wanted a broad range of apps represented, rather than many examples of apps that fundamen -
tally perform the same functions. Their perspective was that a range of apps would better allow them to use MIKE


tion was not whether society should value privacy or public safety more highly but rather how to simultaneously provide meaningful privacy protections and facilitate law enforcement investigations. While stakeholder groups may disagree about the opti


customers from hackers, i.e., protecting personally iden-tifiable information.•
 Policy gr
oups discussed their interest
in having clear standards for obtaining information, to protect privacy rights.Stakeholder Reactions to the Wiki    97Table A.2


 law en
forcement officer stated
that “there are times when it could be extremely valuable.”
•
 Law enfo
rcement officers described
themselves as very likely to use MIKE, provided that it contained useful information.
•
 One law
 enforcement office


ntrol
MIKE Mobile Information Knowledge Ecosystem
POI person of interest
POP Post Office Protocol
SMS Short Message Service
URL Uniform Resource Locator101Glossary
ad exchange service Firm that allows websites to “sell” advertising space to the
high


r Vehicles, real property records, buying habits, etc. The brokerage then sells that information to other brokerages or businesses.
deep packet inspection Extension of packet inspection to include examination of both
identifying information and cont


 transmission of packets of information
on the internet. These rules are implemented via messaging protocols acting at the different layers of communication: application, transport, and network. A fourth layer—the link layer—is outside the internet


d services themselves to be part of the mobile ecosystem,)
onion routing (e.g., The Onion Router)A technique to make communication over the internet resistant to traffic analysis and eavesdropping, using successive layers of encryption to hide not on


provides a service to a mobile application, e.g., the ability to make a payment, archive data, display information on a geographic map, or determine restaurants in proximity to the phone. 106    Helping Law Enforcement Use Data from Mobile Applicatio


rred to as a web address.
web traffic analytics The measurement, collection, analysis, and reporting of a website’s use, usually to judge the popularity of a specific website or webpage. The data can also be used to assess and improve the effectivene


y Monitoring on Smartphones,” Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation ,
2010.
Gershowitz, Adam M., “The iPhone Meets the Fourth Amendment,” UCLA Law Review , Vol. 56, No. 1,
2008, pp. 27–58.Hall, Mark


ps 2012 Roadmap,” January 30, 2012. As of February 5, 2013:
https://wiki.mozilla.org/Apps/Roadmap
Stewart, Jim, “Crow’s Feet Are Best,” The Data Administration Newsletter , June 1, 2008. As of May 23, 2014:
http://www.tdan.com/view-articles/7474Sve


Real-Time Crime Centers in
Chicago
Evaluation of the Chicago Police Department’s
Strategic Decision Support Centers
JOHN S. HOLLYWOOD, KENNETH N. MCKAY, DULANI WOODS,
DENIS AGNIEL
SOCIAL AND ECONOMIC WELL-BEING
Prepared for the Bureau of Justice


g crime.
RAND Social and Economic Well-Being is a division of the RAND Corporation that
seeks to actively improve the health and social and economic well-being of populations and communities throughout the world. This research was conducted in the


Shot detection, which includes fielded sen -
sors, costs $65,000 per square mile covered. From a labor perspective, the civilian crime ana -
lysts in each SDSC are new hires; other staffing reflects reassignments of existing personnel.
RTCCs typical


knowledge: what is or has happened, what it means, and what it might imply in the future. The briefings then include an initial discussion during which participants identify and agree on priorities for watches, areas, and specific missions.
3.
 Ma
k


cruiting and retention of SDSC staff at all levels needs to be bolstered to increase expertise. SDSCs also need to adopt continuous improvement and change management measures to support the improvement of operations over time. Governing policies and


rate systems. For the SDSC’s app for officers in the field, we recommend cre-ating a “what’s important now” screen displaying the most pressing information, rather than having different types of information on separate screens only.
Besides technolo


ustice Assistance
CCTV closed-circuit television
CPD Chicago Police Department
CVRM Crime and Victimization Risk Model
FBI Federal Bureau of Investigation
IIT Illinois Institute of Technology
OODA Observe, Orient, Decide, Act
POD Police Observation D


software (Vlahos, 2012; Jouvenal, 2016). There have also been evaluations of specific technologies that were present in RTCCs, 2    Real-Time Crime Centers in Chicago
Table 1.1
Examples of Real-Time Crime Centers
Agency RTCC Emphasis
Albuquerque,
N


her cameras, license plate readers, automated police vehicle locator data, a gunshot detection system, call-for-service data, predictive analysis tools, a “dashboard” displaying key crime statistics, and operators in a central center as well as tools


providing situational awareness during emergencies and major public events, as well as providing real-time and investigative information to officers responding to crimes. Technologies include more than 340 cameras and an ability to integrate with pr


ication is increasing the number of vehicle theft arrests. It also includes a large data warehouse of police data, corrections and probation data, open-source records data, license plate readers, and tracking and facial recognition software. It is st


erguson, Mo., and subsequent civil unrest began a new period in which law enforcement agencies have come under heavy scrutiny for the lethal use of force and broader problems with community trust and legitimacy. The 2015 death of Freddie Gray while i


 to the police. As researchers looking at the nationwide increase in homicides in 2015 noted, “Lack of trust and confidence in the police militates against crime preven -
tion as a co‐production of police and community and may promote violence as a f


ns: Laquan McDonald shooting video released; heavy protests follow through February 2016.Introduction    7
the CPD to the BJA National Public Safety Partnership to assist with developing a new crime
reduction strategy to address the crime surge. The


e receiving SDSCs first. The initial schedule of SDSC deployments was as follows:
•
 Ja
nuary 7, 2017: District 7
• Fe
bruary 17, 2017: District 11
• Ma
rch 15, 2017: Districts 6, 9, 10, and 15.
As shown in Figure 1.1, these districts had the highest


orting responses to violent crimes, crime prevention efforts, and actions to reduce response times.
•
 Su
pport the development of crime reduction strategies and day-to-day missions (assigning
resources to carry out specific policing tasks).
• Su
pp


1.7 shows a sample predictive policing display.
The predictive algorithm identifies locations to focus on using a combination of inputs,
including historical crime data, broken out by time of day and day of week; weather; the pres -
ence of feature


ar to each other, with more co-arrest links resulting in greater similarities (Illinois Institute of Tech -
nology [IIT], 2019). We observed that the CVRM was primarily employed as a static model, with updated risk scores made available to the CPD by


icago
When assessing the SDSCs’ processes, we compared their processes to known best prac -
tices and assessment frameworks related to (1) examining whether and how information tech -
nology improvements lead (or fail to lead) to better situational a


 studies in which we worked with expert panels to determine needs for law enforcement to use emerging technologies. A signifi -
cant part of the panel discussions concerned identifying needed protections to use the tech -
nologies safely, legally, an


 reviews (COMPSTAT is a performance management system for police departments), and (3) trigger follow-on analyses.
c.
 Ca
pture decisions made in the briefings and create action items to implement them,
both for those in the room and not in the room


e briefing to share within their area of authority (e.g., priorities, missions).
c. Id
entify operational directives and supporting information for SDSC personnel to
share using the room’s communication channels.Process Evaluation    19
SDSC Operati


tion should acquire to become proficient in an area of expertise; Rosenstock, Johnston, and Anderson, 2000, provides a survey of management-related maturity models. The six levels in Table 2.1 align with incremental increases in added value, accompli


 of any automated or imbedded tools that turn the data into information (basic analyses, aggregate reports). The room personnel request certain statistical processing of data, grouping of associated data, or specific formatting of the data.
Level 4 (


lly automated way. Human judgment and critical thinking are needed, especially in rapidly changing situations in which disjointed information and situational knowledge are dynamically pulled together to arrive at a decision or insight.
The levels in


 staff are able to do their tasks.
• Po
lice officer (Part B)
 – Th
ere are nine tasks or duties listed and all align with Level 1 .
• Cr
ime analyst (Part C)
 –Pe
rform data analytics, formulate strategies, recommend missions ( Level 4  and Level 5


 is done and what the room personnel do. For rooms
to evolve and get better, the rooms’ personnel need to use some of their time and resources to identify shortfalls in the tools and processes used by the team.
•
 Ef
fectiveness focuses on the quali


 support regular reviews of best
practices by internal and external experts.
In our observations and discussions, we found evidence of such inward-looking problem-
solving, with much of it being ad hoc and independently driven rather than through re


ensure that each topic is learned at the appropriate, desired level. This process might include a one-to-two-day visit in a room before receiving the official training. By seeing the tools and processes in context, the individuals can ask relevant qu


aspects. For example, this would include some, but not the majority, of the relevant and potential personnel flows. This would include some support for other units but perhaps more of a hands-off role, rather than in-person, everyday support. This w


he outputs from any hub must be trusted and valued and must have credibility. Although there are many factors that can affect the trust, value, and credibility—the ultimate factors that affect the performance of a room—there are some common elements.


potter, and camera display, with effective over -
watch support limited to those places with cameras and acoustic sensors. Post-event analysis tools were also present, with results delivered to the field via the app, briefings, and roll calls. Figure


t is needed to make the best use of the
technology
• Pe
rsonnel who understand field operations and activities
• Pe
rsonnel who understand the specific nuances of the situation (e.g., the specifics of their
district)
• Pe
rsonnel who are available


m’s objectives
 –Pa
rtially present; some staff (crime analysts) received training from the University of
Chicago’s Crime Lab, and as noted, there was some ongoing collaboration, dissemina -
tion (newsletter), and training (all-hands meetings). We a


ngle formula to ensure that innovations are sustained or will reach their
long-term potential. However, using our prior research, we can identify common reasons why innovations such as the SDSC rooms do not sustain results, and we can identify simil


w superior does not understand the intent, history, and value of the innovation, it is likely that the inno -
vation will not receive the care and nourishment it requires.
5.
 Th
e innovation and its sustainment and evolution are not integrated into


s static. This is a major risk and con -
cern. There are some activities that currently exist (e.g., the SDSC newsletter, the docu -
ment repository in the SDSC network area, the biweekly meetings of the analysts and Process Evaluation    33
supervis


empt. Because Genetec Citigraf recently announced that they will integrate ShotSpot-ter displays, this shortcoming will likely be addressed directly by the vendor (Genetec, 2017).
SDSC staff indicated that they also use Caboodle (a locally developed


ctly monitoring ShotSpotter alerts at the SDSC (rather than at the citywide Office of Emergency Management Center),
 they are able to quickly get
cameras trained on the area to begin gathering video evidence. They are also able to get patrol officer


 not clear how much the social-networking tool was being used in prac -
tice in SDSC operations. This tool was not commonly referenced during SDSC daily briefings, which tended to reflect the fact that the official focus of SDSC operations was on pat


gs in 2018. Thus, the CVRM was reasonably effec -
tive at identifying a subset of those at a highly elevated risk for being a PTV, and interventions with the roughly 10,000 people in the highest risk categories could potentially preempt about one-qua


terven -
tions for those at high risk. To start with, given Chicago’s low clearance rates for shootings, the CVRM is de facto a victim prediction system . As noted in a RAND study of the initial ver -
sion of the CVRM (then called the SSL; Saunders,


e Minority Report , and/or that the
model constituted computer-assisted discrimination. An article from technology news website The Verge, titled “The Minority Report: Chicago’s New Police Computer Predicts Crimes, but Is It Racist?” is just one esp


otections covering the full range of data collection and man -
2  The Priority Criminal Justice Needs Initiative, a project for the National Institute of Justice, has worked with several
expert panels on law enforcement’s needs for emerging informat


y rights protections. These included a panel on social media and social network analysis
(Hollywood, Vermeer, et al., 2018a) and a panel on video analytics and sensor fusion (Hollywood, Vermeer, et al., 2018b).Figure 2.3
Information Protections Fram


e
ospatial predictive policing software. HunchLab’s (now ShotSpotter Missions)
developers assert that they are transparent about how the algorithm works and that the tool uses only place-based data (no data about individuals) and only forecasts plac


n National Institute of Justice panel on social media and social network analysis, one comment was “just because you have 500 Facebook friends doesn’t mean you have a real relationship with more than a few of them” (Hollywood, Vermeer, et al., 2018a,


tics systems—there is a need for more formal community engagement to develop norms for how cameras will be placed, how data and analyses will be used in making those decisions, and how the cameras should be used.
•
 CV
RM.  Descriptions of the most


ilar models) to the SDSC evaluation, we first note that the inputs needed to get to improved understanding, decisionmaking, and performance are not limited to technology in these frameworks. A more complete set of inputs is shown in Figure 2.4.
The


 in Figure 2.5.
Policing also has decision cycle models. The best known of these are the SARA model for
implementing problem-oriented policing (Scanning, Analysis, Response, and Assessment; Eck and Spelman, 1987) and the predictive policing cycle (P


 and promising crime-reduction strategies. Examples included the following:
• Mu
ch of the daily briefings process was about identifying areas with crime problems,
increasing resources in those areas, and then following up on the deployments. Hot sp


ctor
units on scene faster than waiting for citywide dispatch.
The SDSCs directly contribute to the second factor in the following way:
• Te
chnology such as HunchLab, as well as straightforward eyeballing of crime locations,
hot spot mapping, and


g in-depth information about locations and persons over the radio as officers were en route or staging.
Again, we did not assess officer safety outcomes directly. Note that assessing officer safety
outcomes statistically is difficult, because assau


intervals (bars)
in crimes, by type and period.
Of the 40 models (ten crime types by four periods), 15 showed significant reductions in
average monthly crime counts after districts added SDSCs. Robberies, burglaries, and aggra -
vated batteries sh


ges in crimes for each district, for both the 2016–2018
period and the 2013–2018 period. Figures 3.4, 3.5, and 3.6 show estimated crime changes by crime type.
As shown in Figures 3.3–3.5, the estimated crime changes include increases and also
more


2.0 1.0Theft
Shooting
Robbery
Motor vehicle
theft
Homicide
Criminal sexual
assault
Burglary
Battery
Assault
1.5
Estimated intervention effectCrime Rate Outcomes Evaluation    57
Figure 3.7
Estimated Intervention Effects, by District (7, 8, 9, and 10)


ly planning cycles, and supporting longer-term opera -
tions to address chronic crime problems and hold specified offenders accountable. In general, we found evidence that SDSCs were supporting much higher levels of awareness and decision -
making th


tions.  Although the team did not perform detailed technical secu -
rity, privacy, and civil rights protections evaluations, we did assess alignment of SDSC prac -
tices and operations with key protections. In general, we did not identify major warni


Findings
We statistically assessed the impacts of adding SDSCs for their principal objective: reducing
crime. SDSCs were found in this experiment to be effective, overall, at reducing crime. In 40 models, districts’ average monthly crime counts decl


, there are three needed organi -
zational knowledge components, as follows:
1.
 th
e institutional foundation to provide a basic level of knowledge (including policies
and SOPs, covered below, along with infrastructure and formal training)
2. per


ns as well as the frequency of these situations, all of which can be summarized in training sessions. (Here, database  is a bit of a misnomer:
A searchable document will serve initially, with a database created as knowledge docu -
mentation accrues.


raining sessions involving hands-on experience in controlled situations.
These sessions would lead to doing the actual task in a “real” situation with the “real” pressures and context, either supervised or in a parallel virtual exercise.
•
 As p
art


n the field,
organizational knowledge, and crime analyses
• da
ily briefings, covering reviews of what has been learned, how that knowledge will be
used, and what is likely coming up and how to respond, especially how upcoming risks will be address


xt target might be for its evolution (Figure 4.1). The direction of planned, conscious evolution should be toward the lower right, meaning more integration and more functional capability.
It is important to progress from Level 1 to 6 and from Stage 1


and address those issues throughout the project, using both formal and informal means.
• Th
e core sequence of implementing change is: (1) why do it, what is the problem, (2) what
needs to be done, (3) figuring out how it needs to be done, (4) ac


er range of problem-oriented policing solutions. The Better Policing Toolkit’s guide on problem-oriented policing introduces core
strategies and methods that are the most promising in making major-crime reductions (Holly -
wood, Lauland, et al., 201


f use was officers providing cards with personal email and phone numbers as another way to get tips. The commander during the observation period described sometimes being woken up by calls but said that the tips received were worth immediately forwar


ese tools (especially
HunchLab) sometimes were left off to the side. If technically feasible, we recommend integrat -
ing ShotSpotter’s shot location hits and HunchLab’s predictive squares (along with any infor -
mation HunchLab can provide on what


 eye toward tools and supporting methods that (1) are based on recommend -
ing specific services and interventions, and (2) are integrated with service providers, includ -
ing treatment providers, outside of the CPD. Transparency and community partic


 as a promising model for improving law enforcement agen -
cies’ awareness of their communities, improving their decisionmaking, and carrying out more effective and more efficient operations that lead to crime reductions and other policing ben -
efit


t measures to support improving operations over time.
5. Recruiting and retention measures are needed.•
 Bo
lster the recruiting and retention of SDSC staff at all levels, furthering
expertise.
• SD
SC supervisors should be as senior as possible, i


m/pdfs/hunchlab/HunchLab-Under-the-Hood.pdfBauer, Kelly, “Bizarre ‘Stay Fetal’ T-Shirt Shows Baby Police Officer, Mocks Rahm,” DNAInfo , April 7, 2016.
As of June 19, 2019:
https://www.dnainfo.com/chicago/20160407/downtown/bizarre-stay-fetal-t-shi


t.ojp.gov/CAT/Resource/125
Cheetham, Robert, “Why We Sold HunchLab,” Azavea Blog , January 23, 2019. As of June 20, 2019:
https://www.azavea.com/blog/2019/01/23/why-we-sold-hunchlab/Chicago Police Department, “Police Observation Devices (PODs),” we


 , Washington, D.C.: Office of Force Transformation, April 1,
2005.
Genetec, “Genetec Inc. Integrates ShotSpotter Gunshot Detection Technology in Security Center,” press
release, Montreal, Canada, March 30, 201 7. As of June 2 7, 2019:
https://ww


earch Agenda That Includes
Business Cases, Privacy and Civil Rights Protections, and Needs for Innovation , Santa Monica, Calif.: RAND
Corporation, RR-2619-NIJ, 2018b. As of September 18, 2019:
https://www.rand.org/pubs/research_reports/RR2619.htm


Appointment Necessary,’” in Brian T Denton, ed., Handbook of Healthcare Operations Management: Methods and Applications , New York: Springer, 2013,
pp. 349–386.
McKay, Kenneth N., Fatih Safa Erenay, and B. Wang, Innovation Maturity Study , Toronto:


licing Cause the Increase in Homicide Rates?” Criminology
& Public Policy , Vol. 18, No. 1, February 2019, pp. 51–75.
Rosenstock, Christian, Robert S. Johnston, and Larry M. Anderson, Maturity Model Implementation and Use: A Case Study , paper prese


n
We Maximize Both Fairness and Effectiveness in the New Proactive Policing?” University of Chicago Legal Forum , Vol. 2016, Article 16, 2016. As of June 20, 2019:
http://chicagounbound.uchicago.edu/uclf/vol2016/iss1/16
Williams, Timothy, “Can 30,


Research Report
Local Evaluation Report for Los Angeles
County ’s Mentally Ill Offender Crime
Reduction (MIOCR) Program
Sarah B. Hunter, Maya Buenaventura, Matthew Cefalu
RAND Justice Policy
C O R P O R A T I O NLimited Print and Electronic Dis


which spans both criminal and civil justice system issues with such topics as public safety, effective policing, police–community relations, drug policy and enforcement, corrections policy, use of technology in law enforcement, tort reform, catastro


County MIOCR
Participants and Other Populations ....................................................................................... 31
Table 4.9. Alcohol Use Disorders Identification Test–Consumption (AUDIT-C) Results for
Los Angeles County M


itions. To reduce the risk of recidivism and improve outcomes for individuals suffering from mental health disorders who are returning to the community after incarceration, California’s Board of State and Community Corrections issued awards (called M


ear program maintained or improved their
reported mental health and substance use status. Program graduates also demonstrated
improvements in health care insurance status, benefit establishment, and housing stability. Data on criminal justice invol


ion Test–Consumption
BSCC California’s Board of State and Community Corrections
CDC HQROL-4 Centers for Disease Control and Prevention Health-Related Quality of
Life Survey
CI confidence interval
COMPAS Correctional Offender Management Profilin


ifornia Department of Corrections and Rehabilitation, 2015).  Silver,
Cohen, and Spodak (1989) analyzed five-year re-arrest rates among a Maryland state prison cohort and found that 73.3 percent of individuals experiencing mental illness were re-arr


llargeon, Hoge, and Penn, 2010; Messina et al., 2004; Osher and Steadman, 2007; Peters and Bekman, 2007). Lack of access to treatment and other support results in higher rates of recidivism,
homelessness, and poor health outcomes (Baillargeon, Hoge,


ions and next steps from this work. Each of following chapters has been organized to include the topics specified in the local evaluation reporting requirements by the BSCC in its communication to local MIOCR grantees.

   4 2. Project  Descripti


rograms. An average GR  case consists of one person, living alone, with no income or resources .   5 Table 2.1. This program logic model was initially drafted by the evaluation team. Program
stakeholders, including staff from the Department of Healt


housing/ICMS,
substance use disorder treatment, and/or other community services (as applicable to individual client)
 • Improved health,
mental health and
substance use
status
• Decreased hospitalizations
and emergency
department visits
•


s were prepared while in custody. Jail-in reach services were provided, and   8 individuals received a coordinated release to a community-based service provider to ensure
linkages to community services upon reentry. In sum, the MIOCR team services i


 LAC’s DHS. DHS established a MIOCR Project Workgroup,
composed of members from the District Attorney’s office, Sheriff’s Department, DMH, Department of Public Health (which includes the SAPC unit), Probation Department, Public Defender, and Alterna


der treatment through referral by the SAPC. The program also
included follow-up by a community worker.
These project components were tracked in an Excel-based project tracking worksheet, as further
detailed below. We also collected information on


t completion, reentry plan establishment, benefit application activities, and linkage to FSP, permanent supportive housing/ICMS, and/or substance use disorder treatment. Staff also tracked dates of referral, assessment, service referrals, and release


llment using validated tools. As further detailed below, the Centers for Disease Control and Prevention’s standard four-item set of “Healthy Days” core questions was used to collect data on self-perceived physical and mental health; the Mental Health


ems from the 36-Item Short-Form Health Survey (SF-36), which measures mental distress and mental well-being. The MHI-5 is a tool used to screen for overall psychological well-being and symptoms of anxiety and depression. The five questions on the MHI


a high level of security during the time of initial recruitment.
Thibodeau et al. (2009) administered the AUDIT-C to a convenience sample of 49 men
incarcerated in a minimum-security Wisconsin state prison. The AUDIT-C was administered at
one-mont


, 2013; Bolano et al., 2016; Evren et al., 2014; Humphreys et al., 2017; Izenberg et al., 2014; Mukherjee et al., 2016; Myers et al., 2018; Owens, Rogers, and Whitesell,
2011; Shiu-Yee et al., 2018; Thibodeau et al., 2009; Walker, 2011). For example


. Standard linear regression models assume that observations are independent. However, the surveys were administered to the same individuals at baseline, 6 months, and 12 months. Thus, observations are not independent. Because multiple observations w


tcomes and program costs. Finally, we provide conclusions and next steps regarding our study findings.
Results
Recruitment and Enrollment
During the grant period, 152 individuals were referred to the MIOCR program. Of those, 64
percent were enr


 Center  5.6%  0.0%
Homeless Initiative D2  1.9%  1.0%
Century Regional Detention Facility  1.9%  0.0%
Whole Person Care  1.9%  0.0%
Client request  1.9%  0.0%
Forensic Inpatient Unit  1.9%  0.0%
Office of Diversion and Reentry  0.0%  3.1


ipants (7 percent) reported visiting the emergency room in the 12 months prior to enrollment in the MIOCR program. Most participants (74 percent) reported not being employed in the 12 months prior to enrollment in the MIOCR program. A little less tha


r more than one
category.    24 Process Evaluation Results
As noted previously, the primary project components were (1) generate referrals to the program,
(2) provide jail-based reentry services, and (3) provide linkage to and follow-up with commu


ntial participant. If the individual was accepted to the program, then a coordinated release was set up so that the contracted program staff would be present at the time of the participant’s release for transport and/or acceptance into their program.


   26 Table 4.5. Reasons for Program Termination and Days in the Program for All Program Dropouts
(N=45)
	All	Dropouts
N %
Termination reason
Client declined  10 22.2%
Moved out of LAC 2 4.4%
Needed higher level of care 2 4.4%
Patton


rom jail to receipt of publicly funded substance use treatment in the future.
Other program barriers experienced relate to the provision of in-jail services and
characteristics of program participants. More specifically, there were some delays in t


 for
how many days during the past 30 days was your mental
health not good?  –6.69 0.008  –9.1 0.017
CDCHRQOL 4: During the past 30 days, for about how many
days did poor physical or mental health keep you from doing
your usual activities, such


des evidence that if bias exists in the full analysis, it appears to be attenuating the estimated effect toward zero. This means that the full analysis may be conservative in its estimation (not biasing the results to favor larger impacts—if anything


 Other Populations
 General U.S.
Population
(BRFSS)  Brent and
Maschi
(2015) Gjelsvik
et al. (2014)  LAC
MIOCR
Baseline
(N=97) LAC
MIOCR
6 Months
(N=43) LAC
MIOCR
12 Months
(N=22)
General health is (Centers for Disease Control a


al., 2015). These statistics are included in Table 4.8. For comparison, summary statistics from the MHI-5 data from Karus et al. (2007) (male inmates seriously ill with HIV/AIDS at two large jails in New
Orleans and Los Angeles) and Wilkinson et al.


ulations at baseline, with about one-third scoring in the positive range for a probable alcohol use disorder. As noted earlier, scores substantially declined at the follow-up time points, with about 21 percent in the positive range for a probable dis


9 0–8 0–4 0–9
Jail days
Mean  13.3 6.1 8.1 20.5
Standard deviation  31.7 16.5 22.7 40.6
Range  0–153 0–74 0–99 0–153
12-month post -period (only includes the 52 individuals who have 12 full months of post -
enrollment data)
 All at
12 M


.9; SD = 1.8; 95-percent CI = 0.2–1.6; post-period mean = 0.4; SD = 1.2; 95-percent CI = –0.07–0.9; t = 1.2; df = 27; p-
value = 0.2286).
• There was a decrease in the mean number of jail days, but this decrease was not statistically significant (p


eipt of Medi- Cal or Other Health Care Insurance Among L os Angeles County
MIOCR Participants by Program Status

The increase in the proportion of graduates who had some form of health care insurance from
12 months pre-enrollment to 6 months po


till
Enrolled All
Dropouts
N 98 30 23 45
12-month pre -period (includes all 98 enrollees)
Employed/stipend (N) 26 6 9 11
Employed/stipend (%) 26.53%  20.0%  39.1%  24.4%
Program completion (includes 30 individuals who graduated)
Employed/s


t may receive 3, 30, 60, or 90 days of residential substance use treatment or receive one to three visits a week from an FSP provider. Taking into account the enrollment of 98 individuals, and subtracting out the evaluation costs ($150,000), the prog


herefore these results should be interpreted with caution in terms of the causal link between program participation and outcomes. In addition, the current program attrition rate of 60 percent suggests there may be some room for improving the connecti


 GR 12 4 2 5 1
% with GR 40.0 28.6 50.0 45.5 100.0
NOTE: ODR SHP = Office of Diversion and Reentry Supportive Housing Program; SAPC = Substance Abuse Prevention and Control residential substance abuse
t


dhcs.ca.gov/provgovpart/Documents/Waiver%20Renewal/FinalDraftMediCal2020WaiverRenewal.pdf
California Healthline, “Health Care Revamp at the L.A. County Jails,” March 1, 2018. As of July
24, 2018: https://californiahealthline.org/news/health-care-re


th, 2017.
Hunter, Sarah B., Melody Harvey, Brian Briscombe, and Matt Cefalu, Evaluation of Housing for
Health Permanent Supportive Housing Program, Santa Monica, Calif.: RAND Corporation, RR-1694-BRC, 2017. As of August 13, 2018: https://www.rand.o


ctors Associated with Interest in Receiving Prison-Based Methadone Maintenance Therapy in Malaysia,” Drug and Alcohol Dependence, Vol. 164, 2016, pp. 120–127.
Myers, Janet J., Mi-Suk Kang Dufour, Kimberly A. Koester, Mark Morewitz, Rebecca Packard,


8: https://www.americashealthrankings.org/explore/annual/measure/MentalHealth/state/ALL
———, “Poor Physical Health Days in United States in 2017,” 2018b. As of July 24, 2018:
https://www.americashealthrankings.org/explore/annual/measure/PhysicalHea


CORPORATION
Feasibility of a Survey
Panel of Criminal Justice Agencies in Small, Rural, Tribal, and Border Areas
Jessica Saunders, Meagan Cahill, Jirka TaylorLimited Print and Electronic Distribution Rights
This document and trademark(s) contained h


cy
The research reported here was conducted in the RAND Justice Policy Program, which spans
both criminal and civil justice system issues with such topics as public safety, effective policing,
police–community relations, drug policy and enforceme


nfidence Intervals with 5-Percent Margin of Error .......................... 26

Table A.1. Strategies Used to Improve Response Rates in National Panel Surveys .................... 35
  Table B.1. How Old Is Your Record Management System, in Year


ystems, jails, and community corrections
agencies for other projects, thus increasing SRTB representation in research.
To determine whether this idea is feasible, the JIC conducted several tasks across multiple
phases. First, we used a convenience


University’s Center for Violence Prevention and Community Safety. We are grateful to all of the
staff who participated in the development and implementation of this project, including our
RAND colleagues Andrew Morral, Liisa Ecola, Josh Mallet, a


iorities specific to SRTB agencies. The purpose of the JIC is to gather information on the challenges facing SRTB agencies, identify relevant technology solutions that can address those challenges, and assess the solutions as they are implemented in


ipation,
voting  behavior,  and  public  opinion   Online  and
face-to-face
interviews
PSIDb   More  than  24,000
residents  (10,000
families)  Repeated  interviews
every  two  years   Family  economics,
demography ,  and
health


udinal use of survey panels also enables an analysis
of change in variables of interest over time (Berrington, Smith, and Sturgis, 2006). The same is true for repeated cross-sectional surveys not utilizing a panel (i.e., surveys that ask a new indep


pondents via a panel on short notice is especially useful, given the rapid pace of technology development, and would therefore enable the JIC to collect information as current as possible.
This report describes the JIC’s efforts to create a survey


sworn law enforcement officers. This definition is similar to previously published definitions, which have defined agency size based on either the number of personnel employed or the residential population served. For example, a previous study define


rature: The Rural Policy Research
Institute identified over 15 definitions of rural in use by federal programs. In general, this term
refers to areas with low-density populations at a specified distance from large cities and metropolitan areas. We


nderstanding how they are similar to or different from non-SRTB agencies. While we did not do so for the feasibility surveys, an option going forward is to provide open enrollment to other types of agencies to use as comparison groups, which we discu


t, and data were analyzed using Microsoft Excel. The results were depicted using different types of graphs and emailed to everyone who received the survey (i.e., all potential respondents in the sample, n = 148).
During this time period, we worked o


was sent out in March 2016, along with the results from the previous round.
We selected the topic of mental health needs across the different types of criminal justice systems agencies. This month, a few individuals (3.5 percent) asked to be removed


vitations with individualized links, allowing the JIC to match
responses to individual respondents without requiring additional logins from respondents
•create personalized surveys based on either agency type (small, rural, tribal, border) orsector (


 each participant to classify them
into the correct categories—small, rural, tribal, border (or other) AND law enforcement,
courts, institutional corrections, or community corrections? The categories are not always
mutually exclusive, as some indi


uit agencies into the panel study. We decided to focus on a subset of the target population—small law enforcement departments—because it has a known sampling frame (e.g., a list of all eligible agencies) that allows us to randomly assign different po


ents operate
to inform better policymaking
6.This is sponsored by the National Institute of Justice
ENROLL NOW!
Survey questions will concern  agency  policies,  operations, staffing,  use of technology, and
related  topics. Each  month, you


ny questions  about  the
JIC survey  panel,  please  contact  Jessica  Saunders  (jsaunder@rand.org)
Please  visit tinyurl.com/JICEnrollByMail  to enroll  in the panel.

Thank  you,
Jessica  Saunders  and Meagan Cahill

 20 Condition  Thr


___________________________
 Please answer the following questions about your primary law enforcement agency’s
characteristics:
�  Agency jurisdiction:
o  drop down menu state/county/city
o  other: fill in the blank
�  number of sworn full-tim


 populations: SRTB law enforcement, SRTB courts, SRTB institutional corrections,
and SRTB community corrections
3.  four populations: small criminal justice agencies, rural criminal justice agencies, tribal criminal justice agencies, and border cr


es except for the tribal populations. Thus, there is a sampling frame available for a probability sample for only ten of the 16 agency types; the others would require the development of a sampling frame.
In addition, it is also more expensive and d


 to construct.
How to construct a stratified sample:
1.  specify the population
2.  specify the feature/characteristic to stratify on
3.  list the population according to the stratification
4.  calculate the desired sample size per stratum
5.


r analytic reasons to require equal numbers across different strata, such as having an equal sample size across populations to use in different analyses.
How to construct quota sample:
1.  specify the population
2.  specify the feature/characteris


bias (Groves and Peytcheva, 2008). Correspondingly, some researchers have suggested focusing efforts on increases in the effective sample size and nonresponse bias analyses, rather than response-rate maximization (e.g. Davern, 2013; Halbesleben and W


t have incentives (like the panel we tried to create), so it is hard to anticipate whether a panel study with this population would continue to get such high response rates. Still, the question of incentives for JIC panel participants merits further


ondents receive an honorarium based on the length of the survey.
6
The proposed panel survey of SRTB criminal justice agencies is a short survey of only two
minutes per month, so a large monetary incentive would not be appropriate, but research has


ating a narrowly focused and useful survey can set the groundwork for expanding by demonstrating the survey’s utility.
We would recommend starting with law enforcement or institutional corrections for two
reasons:
1.  There are already sampling fr


e  payments
Monetary  compensation  for  completion  of  interview    £10  per  adult
respondent   €5  per  adult
respondent   $60– 100  per
respondent   $Australian
30–60  $50– $110  per
respondent   $65  per
respond


x
Promise  respondents  confidentiality  of  responses    x   x   x   x   x   x
Bonuses  to  interviewers/teams  for  achieving  response -
rate  goals  or  working  additional  hours   x   x   x   x   x   x
Ongoing  monitoring  of  resp


68– 2013) .  N/A  =  not  applicable.

 39 Appendix  B:  Results  from   the  Email  and  Web -Based  Surveys
January  2016  Survey
The first law enforcement survey found that departments held mixed opinions of their current
records


y, almost 75 percent use some sort of remote alcohol monitoring devices with their clients.
Table  B.7.  How  Many  Hours  Do  Your  Officers  Spend  Commuting  Per  Week?
Response  Frequency
(%)
0–3  hours   45.5
4–10  hours   36.4
11–20


.15.  To  Whom  D o  You  Allow  Access  to  Video  Visitation?  (Q uestion  Applicable  to  56
Percent  of  Agencies)
Response  Frequency
(%)
Family  members  and  legal  professionals    56
Family  members  only    44
 None of the comm


court  require  a  mental  health  assessment/screening  for  defendants ?  40  60
Is  the  training  on  mental  health  issues  adequate  for  judges?   37  63
Is  the  training  on  mental  health  issues  adequate  for  other  staff?    14  8


at, in comparison with 2007, their budgets, workload and staff numbers had all increased. On average, reporting agencies applied for 2.8 grants, with one reporting applying for 35. They received an average of 2.6 grants, with one agency receiving 34.


 Piekarski, Kumar Rao, Douglas Rivers, Randall K. Thomas, and Dan Zahs, “AAPOR Report on Online Panels,” Public Opinion Quarterly, Vol. 74, No. 4, 2010, pp. 711–781.
Baker, Reg, J. Michael Brick, Nancy A. Bates, Mike Battaglia, Mick P. Couper, Jill


015, pp. 43–59.
Cochran, William G., Sampling Techniques, 3rd ed., New York: John Wiley & Sons, 1977.
Collins, Pamela, Research for Practice: Law Enforcement Technology—Are Small and Rural
Agencies Equipped and Trained? Washington, D.C.: National


p. 103–124.
Hillygus, D. Sunshine, and Steven A. Snell, Longitudinal Surveys: Issues and Opportunities,
Durham, N.C.: Duke University Initiative on Survey Methodology, August 8, 2015. As of March 6, 2017: https://sites.duke.edu/ssnell/files/2015/07


ility
Between Probability Sample Estimates and Nonprobability Web Samples Estimates,” Proceedings of the 2013 Federal Committee on Statistical Methodology (FCSM) Research Conference, 2014. As of March 24, 2017: https://s3.amazonaws.com/sitesusa/wp-c


Methods in Social and Behavioral Research, Thousand Oaks, Calif.: Sage Publications, 2003, pp. 671–702.
Teddlie, Charles, and Fen Yu, “Mixed Methods Sampling: A Typology with Examples,” Journal
of Mixed Methods Research, Vol. 1, No. 1, 2007, pp. 77


An Agenda to Advance Integrative
Resilience Research and Practice
Key Themes From a Resilience Roundtable
Joie Acosta, Anita Chandra, Jaime Madrigano
February 2017An Agenda to Advance Integrative Resilience Research and Practice  ii Preface
Resilien


es  1
 Definitions and Drivers of Resilience  3
 Challenges to Advancing the Field of Resilience  5
 A Need for a Shared Agenda Going Forward  8
2. Resilience Roundtable to Generate Agenda Priorities  10
 Purpose of This Agenda  14
3. Stresses: A Res


683.An Agenda to Advance Integrative Resilience Research and Practice  v Acknowledgments
The authors are greatly appreciative for the time and insights of the many
contributors to this report. In particular, we highlight the contribution of Tracy Co


 Integrative Resilience Research and Practice  vii of the various types of impacts into a useful metric lag behind. What is needed is a
shared agenda, going forward, that will promote research in resilience in a way that builds across
 disciplines.


cipants met in breakout
sessions to discuss each topic
and offer recommendations
in four areas: facilitation of
ongoing multisectoral and
interdisciplinary dialogue
on resilience; expansion or
enhancement of policies,
programs and services;


ithin communities and from one community to another). This model challenges the normative thinking that every observed effect has an observable cause and that the whole can be understood by studying the pieces.The Roundtable participants
noted the c


le discussions,
participants underscored many of the challenges confronting the development of a robust workforce that can address the need for more transdisciplinary and integrated action. Resilience leadership development was a leading priority ac


to help its residents achieve
good health and prosperity, while struggling with the challenges of limited opportunity, chronic violence, a history of inequity, and other stresses. Now imagine Anytown is hit with a hurricane that devastates its coast


h year (1990 to 1999), to an average of 56 per year (2000 to 2009) (Linds
ay & McCarth, 2015). Income inequality is at the highest
level for the past 50 years, with the richest 10 percent of the population now making over nine times more than the po


practitioners have begun to identify the key factors that promote community resilience, such as the application of long-term recovery plans, the active engagement of nongovernmental or civil society organizations, and adherence to principles of socia


 is recognized as being culturally and contextually dependent
(ASPR , 2015), a number of factors that drive individual resilience are common across
populations, including those operating on individual, family, school, and community levels.
On the


levels (Zolkoski & Bullock, 2012).
The evolving fields of emergency management and disaster risk reduction, in particular,
have supported the development of the community resilience field. Emergency management started with a focus on hazard and vul


as much attention in resilience discussions.
While this
growing inquiry
into the factors
that influence
resilience and
shape resilient
communities
represents a
step forward
in building the
evidence base
in resilience
research and
po


s as resilience action and how those actions address long-standing social challenges are central to the resilience dialogue (e.g., how to create policies that adapt to climate change in ways that are fair across neighborhoods). Y et, too often, resil


 more coherent and robust picture on the factors and mechanisms that propel resilience development. FIGURE 1
Moving to Transdisciplinary Science
PsychologyResilience Science
AnthropologySocial ScienceResilience
Science
Anthro-
pologyPsychology Sociol


g
resilience in developmental and ecological processes with attention to cultural influences on resilience; 3) intervening in resilience; and 4) resilience research on multiple systems levels
 — the broader field of resilience research and practice


have had two challenges. They either have been highly specialized to specific sectors only or have offered important but numerous insights and recommendations, making it difficult to prioritize which directions to pursue across disciplines and secto


factors in resilience. The RWJF-RAND team purposefully selected experts who often meet in specialized resilience discussions but do not convene together.
* Available online at: www.rwjf.org/en/library/features/resilience-convening-2016.html2. Resili


eting themes using real-time illustration (example in Figure 2).The topics were selected
because they offered key
points of intersection or
synergy between the fields
of individual and community
resilience. Actions across
these topic areas woul


aturally
occurring innovations and programs that
have been evaluated57% Systems
Develop a minimalist set of measures Ensure measures capture response to
acute stress and presence of capabilities
and capacities to deal with chronic stresses of all


es. If
 the workforce does not understand resilience principles and frameworks, we cannot
move research, policy, and practice  effectively.
As referenced earlier, some disciplines, like psyc
hology, have already outlined waves of
resilience resear


ody. While beneficial in responding to acute stress, repeated activation of this adaptive response over time can lead to negative consequences (Karatsoreos & McEwen, 2011; Selye, 1936). Recently, stress has been viewed within the context of allostasi


required for communities, families, and individuals based on their stress background. Accumulating evidence suggests that health disparities can be partially attributed to allostatic overload resulting from exposure to multiple physical, social, and


s on issues of enduring stress and how that relates to persistent inequity, a
 criticism in the resilience backlash mentioned earlier (chapter 1) (Allmark et al., 2014).
Theme Recommendations
The insights and discussion from the Resilience Roundtabl


nes and sectors together (National Research Council, 2009).
While there has been greater movement toward developing comprehensive resilience
indices, these measurement tools are still limited in their ability to capture stress across a community’s


of resilience that may be unique to a particular community. Community involvement is also crucial to the collection of new and emerging data types (e.g., social media or community experiential data) as well as the dissemination of any data collection


nterventions for multiple stresses, and these interventions can then be pilot tested in communities where they can have the largest
 impact.
Recommendation 3: Focus on narratives, storytelling, and digital media to increase
understanding within and


y is needed. Further, it has been argued While some have argued that the decline
in American informal and formal civic
organizations is, at least, partially due to a
rise in media consumption, others have made
the case that popular culture is a


ding to environmental shocks or stress); and connectivity across a range of scales (i.e., between individuals within communities and from one community to another) (Andries, Folke, Walker, & Ostrom,
 2013).
In complex systems theory, the relationshi


lience outcomes.
They also engaged in dialogue about what policies can support resilience in a systems
framework, and what policies historically served as impediments across systems to advancing resilience (e.g., residential segregation and related


can be highly unpredictable (since relationships between parts of the system are not linear), an important first step is mapping out the key actors and intersection points that comprise the boundaries of a community and then beginning to identify the


d
administrative partnerships and return on investment for investing in partnership activities.
With this model in mind, one framework to achieve resilience and well-being is
through an “enterprise” approach, which harnesses a full range of governm


though the nature of resilience ‘interventions’ (i.e., policies,
practices, plans) may look different than traditional programs (e.g., cascading shifts in policy changes from state to local, organizational partnership development), some processes ma


improvement of essential community services such as education or health care (Committee on Increasing National Resilience to Hazards and Disasters & Committee on Science Engineering and Public Policy, 2012a). While federal policies that promote resil


es should reflect the ‘business case’ for resilience and be useful to practitioners,
policymakers, and investors so that they are able to understand the risk and ROI in resilience An Agenda to Advance Integrative Resilience Research and Practice  28


arrative insights and pertinent facts about changing patterns in the system and inform decisions about changes to system structure or coordination (Altintas et al., 2015). Alignment of data systems across organizations requires changes to governance


pronged approach to planning, design, and management of public spaces to promote health and
well-being by leveraging community assets and inspiration (McCann, 2002).5. WorkforceResilience Can
Be Built by
Strengthening
Community
Leadership, the B


nowing how to “set the table.” In other words, new leaders need to bring together diverse stakeholders in effective ways. Further, leaders and the broader workforce must know how to acknowledge the depth of some of the equity and cultural values that


nd methodology. In team science analysis, translational strategies that can help with progressing a field include developing integrative conceptualizations of team science processes and outcomes, exploring the impact of interpersonal processes and le


behavioral sciences must have some understanding of how the built or physical environmental affects adaptive capacity. This focus does not negate the benefit of a new, transdisciplinary field of resilience that can stand alone, but does ensure that r


and difficulty in working at multiple ecological levels (e.g., federal/local) (Fawcett, Schultz, Watson-Thompson, Fox, &
 Bremby,  2010).
This analysis adapted the IO
M
framework for collaborative action to outline
recommendations for that field g


ived in disaster response and recovery or have been effective at addressing compounding stress (Boin & Hart, 2003). The extent to whic
h
this systems or multisectoral thinking can be operationalized will not only inform the earlier recommendation ab


 any remaining residents with unmet needs so that they can get assistance
through Rebuild. One year after the disaster, Anytown hosts a community celebration of
their resilience. The Anytown Assets festival becomes an annual tradition where residen


o a full assessment of acute and chronic or enduring stresses facing some populations and some communities. Equity was also noted as critical to understanding the functioning, outcomes, and returns to developing resilience-oriented systems.
As summa


s frameworks
 for action. An Agenda to Advance Integrative Resilience Research and Practice  38 TABLE 2
Recommendations to Advance the Field of Resilience by Theme
  THEME RECOMMENDATIONS
Stresses:
Resilience should
be applied across a wide
vari


 given. The meeting sparked discussion, but by no means completed the analysis and dialogue.
As such, more work is needed to further unpack points of intersection where
resilience-oriented leaders can synchronize and align efforts. Further, given t


Ecology and Society, 18(2), 8–18.
Aranda, K., & Hart, A. (2015). Resilient moves: Tinkering with practice theory to generate new ways of
thinking about using resilience. Health:, 19(4), 355–371.
Armistead, C., Pettigrew, P., & Aves, S. (2007). Exp


mmunity. American Journal of Community Psychology, 47(3–4), 217-235. doi:10.1007/
s10464-010-9399-5
Buizer, M., Arts, B., & Kok, K. (2011). Governance, scale and the environment: The importance of
recognizing knowledge claims in transdisciplinary ar


nd
resilience as indirect measures of reliability in the framework of water distribution network design. Procedia Engineering, 70, 379–388.
Cutter, S. L., Ahearn, J. A., Amadei, B., Crawford, P., Eide, E. A., Galloway, G. E., … Schoch-Spana, M.
(2


 E. T., McDonald-Madden, E. V. E., Puotinen, M. L., & Possingham, H. P. (2008). Should we
protect the strong or the weak? Risk, resilience, and the selection of marine protected areas. Conservation Biology, 22(6), 1619–1629.
Glass, T. A., Goodman,


s in Cognitive Sciences, 15(12), 576–584.
Karlin, S. (August 22, 2016). Louisiana residents without flood insurance face uncertainty. Retrieved
from www.reuters.com/article/us-usa-weather-idUSKCN10W0UG
Keeley, B. (2015). Income inequality: The gap


tic load. Metabolism, 55, S20–S23.
McEwen, B. S., & Tucker, P. (2011). Critical biological pathways for chronic psychosocial stress and
research opportunities to advance the consideration of stress in chemical risk assessment. American Journal of P


conomic Co-operation and Development. (2016). Measuring well-being and
progress: Well-being research. Retrieved from www.oecd.org/statistics/measuring-well-being-and-progress.htm
Ortiz de Mandojana, N., & Bansal, P. (2015). The long term benefits of


.
Social and Personality Psychology Compass, 5(9), 634–652.
Selye, H. (1936). A syndrome produced by diverse nocuous agents. Nature, 138(3476), 32. Sempier, T. T., Swann, D. L., Emmer, R., Sempier, S. H., & Schneider, M. (2010 [online]). Coastal
c


n and Youth Services Review, 33(9), 1742–1748.
United Nations. (2016). Sustainable development goals. Retrieved from www.un.org/
sustainabledevelopment/sustainable-development-goals/
United Nations Office for Disaster Risk Reduction. (2005). Hyogo f


n and families.
It teaches children and families to understand their emotional
reactions, communicate more clearly, solve problems more
effectively, and set and achieve their goals. With this training, children and families learn not just to meet


son (e.g., the Chief
Resilience Officer) or entity (e.g., Office of Resilience) to braid
efforts to build resilience capabilities in a cross-department or
agency manner.
Convening: The Annual Global Forum on Urban Resilience and
Adaptation, hoste


iciency• Programs to support
neighbor-to-neighbor
networks
• Community block planningCitizen engagement: In New York City, the Department of
Emer
gency Management coordinates the NYC Citizen Corps,
an initiative that seeks to make communities saf


to offer both insurance and resilience benefits for disaster-prone cities. For
example, as part of Oxfam’s R4 Rural Resilience Initiative, SwissRe
is allowing farmers along the horn of Africa to pay for crop insurance using their own labor.An Agend


 Integrative Resilience Research and Practice  52 • Finding a powerful story and underscoring the urgency or need for joint action were
suggested as ways to address challenges related to competing priorities. In particular,
attendees noted that dev


lience
language relevant to researchers, practitioners, and policymakers?
 »How do we take groups that focus on climate and neighborhood vulnerability
and combine them with organizations focused on adverse childhood experiences,

which work mainl


ategies to inform this thinking from the inside out
(e.g., starting at the grassroots level)?
• How do we build will and demand among communities of all cultures to call for
resources and norms that make them more resilient?
Theme 4: Resilience can


re
components of effective resilience strategy29%
Create a unified picture Demonstrate individual, family, and community/
systems connections across the stress continuum29%
Link with cultural norms Link with organizational and cultural norms and
so


r for the Study of Social Policy Amy Fine
Center for the Study of Social Policy Lisbeth Schorr
ChangeLab Solutions Marcie Ashe
ChangeLab Solutions Heather Wooten
City of Pittsburgh Grant Ervin
Columbia University Jeff Schlegelmilch
Columbia Universit


hoch-Spana
UPMC Center for Health Security Tom Inglesby
UPROSE Elizabeth Yeampierre
U.S. Chamber of Commerce Foundation Gerald McSwiggan
U.S. Department of Health and Human Services Natalie Grant
U.S. Department of Homeland Security Jeffrey Stiefel
W


resilience research or practice take into account the
interconnections between organizations, families, and individuals in your community?
• T o what extent does your work on resilience rely on or leverage collaborations with
colleagues and organ


Access to Medical
Treatment for Injured Workers in California
Year 1 Annual Report
Andrew W. Mulcahy, Madeline B. Doyle, Rosalie Malsberger,
Kandice A. KapinosJUSTICE, INFRASTRUCTURE, AND ENVIRONMENTLimited Print and Electronic Distribution Rights


ions. All its reports are subject to peer review and disseminated widely to policymakers, practitioners in law and business, other researchers, and the public.
ICJ is part of RAND Justice Policy within RAND Justice, Infrastructure , and Environment


 Timeliness to Care —First Non -ED E&M Visit, by
Injury Year and Body Part or Condition  ............................................................................ 48
Table A.2. Number of Observations for Timeliness to Care —First E&M Visit with


etermination process ; and monitoring of medical provider networks (MPNs ).
Many of these recent policy changes might have affected access to care for injured workers,
including changes in the availability of appropriate providers and services ; t


hose populations.
Because access to health  care is one of the key dimensions of health care quality, there has
been significant interest in refining measurements to improve construct validity (Donabedian, 1966; Berenson, Pronovost, and Krumholz,


orting tool used  by the DIR to collect comprehensive information from
WC claims administrators about initial reports of injuries and all medical billing related to WC
care.
In examining trends over time, we aim to highlight potential access -to


ng WC patients .
Utilization and Payments  per Provider
Measures of the number of claims  (i.e., injuries) per provider , the number of bill lines  (i.e.,
discrete medical services) per provider , and payments per provider are important complemen


 likely driven by changes in very large practices or health systems treating many injured workers. Changes in the m edian  claims per provider  were
much lower for all three measures. For example, the median number of claims per provider was 2 in 20


crease in the median wait time between injury and any E&M
visit, from 2 days in 2010 to 3 days in 2014.
• The pattern is similar between Northern and Southern California , but there were
differences in timeliness of care between types of injury .


umber
of claims, bill lines, and payments per TIN. Furthermore, we expect the number of TINs treating
injured workers to decrease over time because of consolidation in the broader health care system ,
rather t han because of decisions to stop trea


nt of Industrial Relations, Division
of Workers’ Compensation, under contract number 41336064, for which Irina Nemirovsky and
Genet Daba served as the project coordinators . This report was reviewed according to RAND’s
Standard for High- Quality R


h as the
availability of appropriate practitioners  and services and timeliness of care. For example,
practitioners might have changed the way that they provide services to injured workers (or
started or stopped seeing injured workers at all) as a


lthough we discuss potential explanations for the trends observed here, our analyses should be considered descriptive , as we are not conducting causal analysis.
Medical bill and administrative data offer an important but limited window into access


lations,
because patients with better access to care systems are more likely to receive comprehensive,
higher- quality care and are therefore more likely to experience better outcomes ( Kominski et al. ,
2006).
The term access, as it applies to


oxy measures —such as utilization of services, specialist
referral patterns, provider type, care setting, purpose of encounter, and the frequency or continuity of care —can be used to assess access to adequate or appropriate care. However,   5 focus


e access to medical treatment from 2011  to 2013. This more recent study found
that the number of injured workers who received medical services in the WC  system decreased
30 percent from 2007 to 2012, followed by a slight increase in 2013, and fou


5 A PCP is a provider whose billing specialty is  multispecialty group practice, family medicine/general practice,
internal m edicine,  occupational medicine, or chiropractor.    7 5. visit elsewhere.
Injured workers flow from an ini


l injuries in WCIS for tha t calendar year, respectively . In the second approach —the
injury- year approach —we first identif ied all injuries in a calendar year and then analyzed WCIS
bill line records with service dates within 12 months of the i


articipating providers  as those who submitted
WC medical bills to WCIS for that year  and for whom we have a provider identification  (TIN).
To examine the supply of appropriate providers, we analyzed  trends in participating providers
over time


r of injury. We primarily report  means (or medians , when  the distributions were skewed) .
We tested for significant differences in  means  over time and  across i njury types  using two-sided
statistical tests and a significance level of 0.01 ,


 WC
system or to all of providers’ care to injured workers. Finally, some WCIS records are not linked
to a claim identifier (JCN) ; therefore,  we cannot observe injury type and other fields from the
administrative data. F or these records and fo


pation, Utiliza tion, and Payments
In this chapter, we present trends in provider participation, utilization , and payments . All else
equal, a greater number of providers treating injured workers likely represents better access to
care. Measures


(increased by 670).

8 Some specialties —such as family medicine/general practice —might have been more likely to have fewer
practitioners per billing entity or might  have been more likely to consolidate over time compared with others. The


king
Bonferroni adjustments  for multiple hypothes is testing, these means would not be considered
statistically significantly different  at conventional levels .10 Only among occupational m edicine
providers did we find an increase in the number


cupational medicine physicians and therapists into occupational medicine because there were typically
fewer  than 200 therapists per year. We combined physical medicine and rehabilitation providers and physical
therapists  into physical therapy bec


tion drug utilization , is 4.32 , versus  6.97
among those who were using prescription drugs. Therefore , the average use of services among
injured workers actually getting care is much higher. However, an examination of changes in
conditional mea


 prosthetics/orthotics) (all p < 0.01 ). Some
of these changes are consistent with higher payment rates implemented through the transition to
RBRVS in 2014. Again, it is not clear from these analyses whether SB 863 provisions or other
factors are


s following the injury date.
Sample Sizes
In Table 5.1, we present sample sizes , using the broadest definition of first visit (e.g., in any
setting with any provider) , of the number of injured workers per year for each injury category.
We exa


able A.2 for sample sizes.
* Statistically different from the 2014 median in the same injury (row) at p  < 0.001.
    26 Table 5.4. Days from Injury to First E&M Visit with a P CP, by Injury Year and Body Part or
Condition

2010  2011  2012


was significantly greater than in the southern region ( p < 0.001) , and the wait time for out -of-
state receipt of care was  significantly greater than care received in the state ( p < 0.001) ( see
Figure 5.1 ).
Figure  5.1. Median Days from Inju


7. Mean PCPs Seen in First 12 Months of Injury, by Injury Year and Body Part or Condition
 2010  2011  2012  2013  2014
All injuries   1.25* 1.25* 1.28* 1.24* 1.21
Low-back pain 1.32* 1.32* 1.35* 1.30* 1.28
Shoulder injuries  1.34* 1.33* 1.36*


s in providers  (see Chapter Three) .
There was variation across the site of injury, but the time trends were relatively similar across
the three different measures of first visit. Injured workers with low -back injuries were typically
seen withi


and by the study conditions. In
Table 6.1 , we present the distribution of injured workers in each pathway in each injury year,
from 2010 to 2013. The shaded bars signify the relative percent age of workers within each year  10% 11% 11% 11%61%65% 6


010  2011  2012  2013
1–3 ED à Admit  0 0 0 0
7–9 PCP E&M  à Specialist  11 12 12 13
10–12 PCP E&M à Nonspecialist  4 5 5 5
15–17 Non-PCP E&M  à Specialist  7 8 8 9
18–20 Non-PCP E&M à Nonspecialist  5 5 6 6
23 Other office/HOPD à  PCP  3 3 3


o ha d the same path but start ed at
a specialist’s  or non- PCP’s office (pathway 15). This suggests that , for injured workers who
started in one of the two main entry points (physician or outpatient), there does not appear to be a
significant d


er year to almost 20  for
injured workers with at least three visits , depending on the pathway.
We also examined pathways by main type of injury. The most common pathway—start ing
with an E&M visit with a PCP, followed by a specialist visit and


ishes between different groups of
practitioners that separately bill for services. A TIN may represent a single practitioner (e.g., a solo physician
practice) or many individual practitioners (e.g., a large physician practice or delivery system). I


 and any E&M visit , from 2 days in 2010 to 3 days in 2014.
The pattern is similar between Northern and Southern California , but there were differences
in timeliness of care between types of injury.
Monitoring Provider Churn
Injured workers m


—the TIN —distinguishes between different groups of practitioners
that separately bill for se rvices. A TIN may represent a single practitioner (e.g., in the case of a
solo physician practice) or many individual practitioners (e.g., in the case of


 to
monitor access in the C alifornia  WC system over time , given the  numerous recent and potential
future changes in California and U.S. health care systems , such as health insurance coverage
expansions; changes to the Patient Protection and A


tions for Timeliness to Care —First Non -ED E&M Visit, by Injury Year
and Region

2010  2011  2012  2013  2014
Northern region  81,792  78,462  84,704  79,743  93,487
Southern region  161,456  155,763  164,933  156,445  183,852
Out of state


n-PCP
E&M
Office/HOPD  Specialist  None  2,198  2,072  2,027  2,545
18 Non-PCP
E&M office/
HOPD  Nonspecialist  PCP
office/HOPD  1,303  1,387  1,437  1,646
19 Non-PCP
E&M office/
HOPD  Nonspecialist  Other service 42 57 86 76
20 Non-PCP


2018:
https://www.aamc.org/download/447152/data/californiaprofile.pdf
Benjamin , D. J., J. O. Berger, M. Johannesson, B. A. Nosek, E. -J. Wagenmakers, R.  Berk, K. A.
Bollen, B. Brembs, L. Brown, C. Camerer,  et al. , “Redefine Statistical Signif


are,”  The Milbank Memorial Fund
Quarterly , Vol. 44, No. 3, 1966, pp. 166–206.
Haddad, S., and K. Mohindra, “Access, Opportunities and Communities: Ingredients for Health
Equity in the South,” paper presented at the Public Health and Internationa


my of Social Insurance, Workers’ Compensation: Benefits, Coverage, and Costs ,
Washington, D.C., October 2017. As of August 17, 2018:
https://www.nasi.org/sites/default/files/research/NASI_Workers%20Comp%20Report%202017_web.pdf
Peters, D. H., A.


Evaluation of the Jinan
City Water Ecological  Development Implementation Plan and Recommendations  for Improvement
David G. Groves, Debra Knopman, James Syme, Nidhi Kalra, Zhimin Mao
CORPORATIONLimited Print and Electronic Distribution Rights
This


development; infrastructure financing; energy policy; urban planning and the role of public–private partnerships; transportation policy; climate response, mitigation, and adaptation; environmental sustainability; and water resource management and co


 (in years) Predicted by IPCC SREX Report  ....................... 99
 8.9. Rive
r Stage Predicted for Various 24-Precipitation Return Periods  ....................... 100
 A.1. Proj
ections of Agricultural Demand Provided by Jinan Water Resources Bu


ime Series of Precipitation and Stage Data  ........................................ 95
 8.2. 24-H
our Precipitation Return Period, Intensity, and Predicted River Stage  .............. 98
 8.3. Futu
re 24-Hour Precipitation Return Period, Intensity,


r in this kind of thinking. This has served the region well, but as the city’s water needs approach—and in some cases exceed—the available supplies, large uncertainties about future conditions become increasingly important. Specifically, the possibil


confidence in results based on a limited number of scenarios, manage surprise, and use quantitative analysis in situations when key data are missing or imprecise.
RAND’s technical road map therefore includes the following elements:
•
 Intr
oduce RDM


ple helps to explain the difference between RDM and more-traditional engi -
neering analysis. Suppose a new reservoir is to be built to increase storage in the Jinan system. How large should the reservoir be, how should it be operated, and what is th


 to increase in the future.
In contrast, the green line shows the performance of the system with a reservoir with the
same capacity as before but, in this case, additional investments in interconnections or changes in operations throughout the syst


esentation of the full span of precipitation and temperature effects generated from the climate models. Over the period from 2000 to 2050, changes in precipita -
tion are estimated to range from –100 millimeters (mm) to almost 300 mm annually; change


n
Strategy Name Description
1 Current Projects Current state of Jinan system
2 Completion of IP Includes Current Projects and completion of IP projects
3 IP and Increased Water Reuse
(IPR)In addition to IP, includes increase of reuse rates in distri


sidential and industrial sectors) or climate effects on demand (agricultural sector) by 2035. For each sector, efficiency standards were evaluated at a 100-per -
cent implementation level (full implementation) and a 50-percent implementation level. F


strategies across a wide range of possible future climate and demand projections that could reduce future vulnerabilities. Our focus was on longer-term trends, spanning the period between the present and 2050, and the implications of these trends fo


tions and assumptions of a continuation of current infrastructure and management. Note that demand and supply prefer -
ences were set within the Jinan WEAP model following multiple discussions with the JWRB technical team. Demand priorities have the


ulnerable if average unmet residential demand from
Figure S.5
Summary of Unmet Residential Demand and Vulnerability Across All Futures and Groundwater Assumptions
Current 50% Current 50% Current 50%Low Demand High Demand Medium DemandYellow River
Al


sensitive to the climate projections when the initial storage is assumed to be very large. In effect, the groundwater basins absorb most of the climate sensitivity, a reason why we do see climate effects (in the form of unmet demand) in the residenti


 residential unmet demand for the same strategies as in
Figure S.6 across all futures. Each symbol represents a particular combination of climate pro -
jection, demand projection, Yellow River allotment, and groundwater storage. Many symbols fall on


in data collection will have a beneficial effect on analysis and the integ -
rity of the Jinan WEAP model. As JWRB further refines the data behind the demand and supply nodes and other model components, results could shift.
At the same time, the Jin


tion of IP
IPR
IPR + WQIPR + SAGMIPR + WQ + SAGM
IPR + Residential Efficiency
IPR + Residential Efficiency + SAGMIPR + Res/Ind Efficiency
IPR + EffAll
IPR + EffAll + SAGM
IPR + EffAll + WQIPR + EffAll + WQ + SAGM
IPR + 50Eff
IPR + 50Eff + WQIPR + 50E


processes.
How Will Future Demand and Climate Conditions Challenge Jinan’s System?
When uncertainties are considered about future climate conditions, demand, future Yellow River allocations, and initial storage levels in groundwater basins, model re


e outcomes if conditions in
the future were to take a different course then the one on which project designs were based.
How Will JWRB’s Implementation Plan Address Future Climate and Other Uncertainties?
At the present time and for the next severa


et demand almost entirely. Vulnerabilities to the industrial sector can be completely eliminated if all efficiency improvements are implemented, water reuse is implemented, and the WQ strategy for the Xiaoqing River is available as a backstop.
•
 Th


y uncertainties that drive the analysis. Having the appro -
priate modeling tools for the decision at hand is essential.
• RDM pr
ovides a structured approach to analysis of future conditions and strategies for
reducing future vulnerabilities and sh


(姜爱华， 济南市水利局科技办公室， 主任 ), who led the JWRB technical team supporting
the project. Other team members included:
• Zhan
g Baoxiang, professor of the Shandong Water Resources Research Institute
(张保祥， 山东省水利科学研究院， 研究员， 教授 )
• Mao Qia
o, vice director o


lly, we would not have been able to conduct this project without the wisdom and
guidance of Mu Dan Ping, RAND’s business adviser for our work in China.xxxvAbbreviations
50Eff 50-Percent Efficiency
CCAFS Climate Change, Agriculture, and Food Security


ter quality, improving aquatic ecosystem health, reducing damage from flooding, and increasing water use efficiency. As of this writing, most of the construction under the Pilot Scheme has been completed. Understanding how Jinan’s complex hydrologic


fined subsurface chambers, eventually discharging to the surface in a system of 733 natural springs (Kang, Jin, and Qin, 2011; Zhao, 2015; Wu, Xing, and Zhou, 2009).
Recharge occurs primarily in the southern mountains and is largely attributed to dir


d water system and continuous flow of spring waters.
• Devel
op healthy and aesthetically appealing water ecology and environmental systems.
• Achi
eve high water quality, maintain ecological flow of rivers, and a “beautiful scenic
view.”
• Devel
op


ential demand is unacceptable.
Groundwater is the preferred source of water for residential use. Hence, continuous discharge of water from the city’s springs is one of the more visible signs of success in maintaining a sustainable yield of groundwat


 critical to ensuring that Jinan’s water management goals are achieved.
To focus our analysis, RAND sought guidance from JWRB’s experts and refined the ini -
tial set of research questions during a November 2015 workshop in Jinan and follow-up discus


 with historical and current data
 Develop future projections of factors driving change
 Calibrate model
 Evaluate current system for different baseline futures
 Evaluate system with additional Implementation Plan projects
 Consider flood i


cientifically sound approach to analysis that is particularly
well-suited for considering long-term challenges to the performance of a regional water manage-ment system with large uncertainties about future climate changes, changes in demand across


B to apply these methods in future work.
With respect to capacity building, RAND built and ran a simulation model, then trans -
ferred it to JWRB for future use, built and demonstrated use of a decision support tool and then transferred it to JWRB fo


 and oper -
ational strategies, simulate operations, and determine the extent to which shortages would be reduced under the different potential sizes and strategies. Finally, decision makers would choose a design storage capacity that does best under


enario and limited
strategies were considered In RDM analysis,
this option would
be more desirable Current system con/f_iguration
Addition of reservoir with X capacityAddition of reservoir with X capacity + other changes in system 12    Evaluation


e the term strategy  to refer to potential future projects, policies, or combi -
nations of projects and policies that could affect water management in the Jinan system.
Performance Metrics
Table 2.2 lists the key performance metrics that RAND identi


 would have liked to analyze Jinan’s vulnerability to flooding under a wide range of
climate scenarios (as represented by differences between precipitation and temperature histori -
cal averages generated by global climate models). However, we were


dard Data displayed for 2013 to 2015
in Chapter Three
Reduce flood damage Change in recurrence intervals of extreme flood eventsDescription of method  in Chapter EightApproach to Analysis and Visualization of Long-Term Performance    15
cultural—are


ls
• incr
eased supply by drawing from the Xiaoqing River if water quality standards are met.16    Evaluation of the Jinan City Water Ecological Development Implementation Plan
Relationships
The focus of this study was the development of a WEAP model


t and for more-detailed exploration of the results. The use of a Planning Tool is part of RAND’s approach of using analysis to support deliberation over choices, as recommended by the National Research Council (2009), and used to support high profile


via the tabs at the top of the screen or via a dropdown menu in the upper left.
Figure 2.6
Screenshot of the Jinan Water Resources Bureau Planning Tool
RAND RR1682-2.6
Figure 2.5Planning Tool and Approach to Deliberation over Trade-Offs
RAND RR1682-2


 estimates demands and supplies over time on a monthly basis using historical data derived from reference documents provided to the study team by the JWRB (see Appendix A) and mathematical relationships described in this chap -
ter and Appendixes C a


r 698.73
Shanghe Shanghe County Groundwater and the Tuhai River 758.47
Licheng Licheng District Yellow River, Xiaoqing River, and Jinxiuchuan, Wohushan, and Langmaoshan reservoirs281.00
Changqing Changqing District Groundwater and the Yellow River 22


ion 2004–2014 Significant source of surface water for Jinan Urban Center
Nandasha 1 Luzhuang Drainage
Station2011–2014 Significant source of groundwater recharge and seasonal flow into the Yellow River
Xinghua 1 Xinglin inflow 1972–2014 Significant


ussions with JWRB, this supply was included in the model as an annual allocation of 100 million cubic meters (MCM) per year, with 83 MCM per year for Jinan Urban Center and 17 MCM per year for Zhangqiu. An expected, doubling of this allocation to 200


ential demands
Jiangshuiquan 1,050,000 191.80 Near
completionXinglong
ReservoirQuanfu River, Longquan Reservoir
Jinxiuchuan 35,920,000 251.00 Complete Yufu River Jinan and Licheng residential demand, Xinglong Reservoir
Langmaoshan 10,940,000 187.00


n Reservoir
4  Spring discharge at these nodes were calibrated assuming cyclical flow under cycled Historical Climate conditions with
no trends in increasing discharge. Calibration of these springs is discussed later in this chapter. Table 3.6
Prim


riables in the Jinan WEAP model to be consistent with the climate projec -
tions (see Chapter Six). Thus, the multivariate regression models translate climate change into changes in supply availability. A detailed explanation of the development of th


hat the water quality of Xiaoqing River (fourth through seventh row in Figure 3.3) failed to meet water quality standards in virtually all months from 2013 to 2015 for all sampling loca -
tions, rendering the River an unusable supply. If the Xiaoqing


 to have validity, historical data used throughout the model need to have the same record lengths; otherwise, the WEAP calculations that match supply to demand in each month and year would not be using internally consistent data.
5 For
the Jinan WEA


e estimated certain quantities required in the model based on observa -
tional records, consumption statistics, and simulated reservoir operations in an effort to best reflect observed supply delivery records. For example, we applied this approach to


Water in this catchment, which lies directly below part of Changqing, continues across the Mashan fault—which is a low permeability fault close to the Yellow River—to arrive in the Jinan Spring Water Catch -
ment. The Jinan Spring Water Catchment inc


d for which we lacked historical data. For the period in which observational discharge records are available, Figure 3.6 shows that simulated aggre-gate discharge is similar in amplitude and phase to observed discharge. Additionally, outside of the p


 Bai Springs.
District-Level Splits in Supply Between Surface and Groundwater
Ensuring that quantities of surface and groundwater deliveries by district were representative of observed quantities was an important element of the calibration process fo


20 2010 2050
YearBai Springs Baimai Springs Baotu SpringsDevelopment and Calibration of the Jinan WEAP Model    35
Figure 3.9
Calibration for Jinxiuchuan Reservoir Operations
RAND RR1682-3.90.4
0.30.20.1
0Storage volume, 108 m3
2014 2012 2010 2016
Ye


and 2030. Since these projections implicitly included historically observed years, they were adjusted to match the most recently available observed data year, which was 2014. To perform the adjustment, we first estimated the projected demand for the


elivered) as Population Scenario 2, we did not include it in subsequent analysis described in later chapters.
Instead, we developed population scenarios using population data from the United
Nations Department of Economic and Social Affairs (UNESA,


lic supply water use rates of 75 gallons (283.9 liters) per person per day.
1 Note that the projection is driven primarily by water use rates rising to
parity with New York City, not due to an assumption of population growth. Note, however that even


wth rate of 7 percent from 2015 to 2020 and 5 percent from 2020 to 2030. These growth rates were used to estimate demand through 2020. However, to eliminate extreme growth from industrial value projections going forward, projections after 2020 were r


fective irrigation area in 10,000 monetary units, Qg is the water demand (m3 per monetary unit), and γ is the irriga -
tion water usage ratio. A description of estimates of change in agricultural demand under different climate projections is
provide


, and residential demand projections used to develop these three projections are shown in Table 4.4.
Climate Projections
This study uses a range of estimates of future climate based on simulations from a set of global
climate models (GCMs) developed