formal consent decrees (i.e., legal agreements in which an agency agrees to make specified changes, with the implementation of those changes overseen by a court) (U.S. Department of Justice, 2016). In the first five fiscal years of the Obama adminis


er year in the 1920s, which was the 12    Identifying High-Priority Technology and Other Needs for Improving Law Enforcement Operations and Outcomes
deadliest decade in U.S. law enforcement history (National Law Enforcement Officers Memo -
rial Fund,


inhabitants, 1996–2015.”
5  Federal Bureau of Investigation, Crime in the United States , 2015, Table 12, “Crime Trends, by Population Group,
2014–2015.”
6  Of the 1,532 additional homicides recorded in 2015 as compared to 2014, these four cities


dent in Ferguson. While the evi -
dence for a national Ferguson effect is lacking, some cities may be seeing increases consistent with the Ferguson effect hypothesis, particularly cities with large African-American popula -
tions.
8 Further research


daunting in the current environment, in which negative perceptions of the police, more opportunities in the private sector, and sometimes archaic police hiring policies and standards are making it difficult to attract new recruits (Wilson et al., 201


is of digital evidence not only possible but also crucial for solving crime (Goodison, Davis, and Jackson, 2015). And in today’s “CSI” world, digital evidence is becoming increas -
ingly important—indeed, expected—by juries in many criminal cases. Ho


elopment and training
FacilitiesVehicles
• Personal clothing, protection,
or augmentation
• Weapons and force• Ground
• Aircraft• Watercraft• Associated technologies
• Internal access control• Internal environmental control• Internal physical infras


s of training at all levels.
3. Facility Operations and Population Services , which includes all aspects of maintain -
ing the buildings and structures, as well as the related security. This topic also includes management of infrastructure and logis


5, p. 3)
It is not clear whether these changes have made police more effective. Evaluation research
on police technology has tended to focus more on operation and outputs—for example,
whether a technology works and makes a process faster—than on i


resented in each subcategory.
Information Collection
The collection of information is vital to performing day-to-day law enforcement operations, and currently many information collection tools are available to law enforcement. Basic data collection


g, 2015). Farrar and Ariel (2103) found that body-worn cameras were associated with a 50 percent decrease in use-of-force incidents. Similarly, a University of Cambridge study found a 93 percent decrease in complaints against the police after officer


G), and wireless Internet from both public and private hosts.
From an information delivery perspective, we have seen an upsurge in the number of
vendors offering systems with common operational picture-type displays, which show anno -
tated maps, t


ork has been done to improve the sharing and use of information. Feder -
ally supported examples alone include, but are not limited to, the following:
• The National Information Exchange Model (NIEM) is a common technical framework for sharing struct


vely involved in information sharing, to include sponsoring several IEPD standards packages; it maintains a center on social media, as well.
At a lower level, we have seen an upsurge in vendors offering information integration
services.
However, th


ledge Development and Training
DOJ and other organizations have put a great deal of resources into creating online resources for disseminating law enforcement–related knowledge. As the first of just a few examples, NIJ maintains the CrimeSolutions.g


nsic backlogs, although most, except for the general Paul Coverdell Forensic Science Improvement Law Enforcement Technology and Practice Today    25Table 3.1
Sample Technology and Practice Resources for Law Enforcement Agencies
Need Resource Descript


Center for Problem-Oriented Policing (University at Albany/Office of Community Oriented Policing Services)Repository of over 100 guides with ideas to address different types of crime-generating problemshttp://www.popcenter.org/26    Identifying High-


The National Institute for Occupational Safety and Health (2016) maintains a portal on motor vehicle risks and potential solutions.
This domain does cover unmanned or remotely piloted vehicles—aerial, ground, and on
the water. Unmanned aerial vehic


nducted energy weapons to its offi -
cers, for example (D’Onofrio, 2016).
Prior Law Enforcement Needs: The First Law Enforcement Advisory Panel
(LEAP 1) and Related Studies
There have been a number of studies on law enforcement agencies’ needs for


rds, common operational pictures, health monitoring systems, and law enforcement process initiatives than there was “supply.” Finally, top themes in keynote addresses in IACP conferences from 2011 to 2014 were also reflected as top-ranking needs, of


stions have included: How can sponsors make practitioners better
aware of technology information and training? How can sponsors make it easier for practi -
tioners to find the information they need? How can the presentation of information be better


s for policies, procedures, and guidance related
to video surveillance, sensor systems, and cybersecurity. This continues to be an area where RAND sees high demand from both practitioners and external organizations for additional guidance and knowle


eployable biometric devices with a range of biometric sensors and connectivity needed to positively identify some -
one within a minute; a device that can extract photos, texts, and other electronic evidence from
5  Recent articles starting to exami


a new technology into their organizations and operations?
• Are there any people in your organization who publish articles to inform other agencies on how to integrate new technologies into their organizations and operations?
• Who comes to mind when


lists also discussed how the operational context of policing has changed since the original LEAP in 2013. Panelists then identified issues facing law enforcement, focusing on issues that go beyond what LEAP 1 covered in 2013, and brainstormed and dev


ds that rose to the top of the highest tier likely merit par -
ticular attention and focus. Needs in the middle-rated group are in Tier 2, and needs in the bottom-rated group are in Tier 3.
These steps are explained in more detail below. Appendix C


ower, with lowering costs being weighted, on average, far lower than the others.
The 2013–2014 LEAP 1 showed a somewhat different pattern of weights on the objec -
tives. That panel rated reducing crime as the most important, followed by improving co


unity rela -
tions and uses of force. At the same time, budgetary pressures have been reduced since the height of the 2008 recession. We find it likely that these changes in priorities reflect changes in the social, political, and budgetary environme


AP 2. At the end of both sessions of LEAP 2, we had 82 needs from the Operational session and 72 needs from the Strategic and Administrative session.
Prioritizing Needs
The first step in prioritization was to capture the objectives that each need sup


areas as
• cost to the fielding agencies
• policy and politics, notably including security, privacy, and civil rights concerns
• culture
• human factors (solution not too hard to use).
High-rated solutions were seen as likely to succeed in the field,


.5
Strategic and Administrative Needs, by Overall Score and Tier
0 200 400 600 800 1,000 1,200 1,400
0 10 20 30 40 50 60 70 80 Median objective-weighted expected value
Need rank
RAND RR1814-4.5Top
Tier 1Tier 2Tier 3
Figure 4.6
LEAP 2 Needs, by Pr


rove
trustLower
costsImprove
responsesImprove
competencies
RAND RR1814-4.7Number of needs
Weight
Law enforcement objective
Tier 1                 Tier 2                 Tier 3                 WeightIdentifying and Prioritizing Innovation Needs in T


ment, and dispatch center (i.e., PSAP) operations. Consistent with this panel’s emphasis on improving community trust, there also was a dramatic increase in needs related to communicating with the public. There were also slight increases in needs for


ment but that support only one or two objectives. If improvement on those objectives was viewed as particularly important by the panelists, then neglecting to identify those needs would potentially miss an opportunity to do so.
Another way to look at


 assessments of community policing implementations to determine which ones are more successful.
S
Evidence-based management practices are often not generalizable across different agency sizes and environments.Examine ways to better present to decisio


g that can be read and understood quickly.
O
Training best (and worst) practices and instructional design are either not well known or not implemented widely.Develop a taxonomy or set of categories and supporting information that can be used to desig


and disseminate partnerships with universities, community colleges, and federal training programs (enrollment limited) that already provide training and support on Internet investigations, to make it a routine part of training and operations. Traini


ansparency may give the public the incorrect impression of what is actually going on (e.g., “shots fired” ended up being firecrackers, ”dangerous area” is a busy shopping mall).Develop best practices surrounding ensuring that the appropriate levels o


imes it is optimized for fire and emergency medical services [EMS]).Research the locally optimal sets of questions (with branching) to gather the critical information and be able to dispatch law enforcement, firefighters, and EMS.
S
Agencies unwillin


Opportunity) Associated Need
Solve more casesS Insufficient data exist to adequately measure the impact of forensic backlogs and other bottlenecks on criminal justice system efficiency and throughput.Do a study to measure the impact of forensic backl


 needs that can be considered “low-hanging fruit,” which are needs
that had top-rated median risk scores (defined as technical feasibility score × operational feasi -
bility score). All low-hanging fruit needs were already captured either as Tier 1


nforcement’s sharing and use of
information.
• About 15 percent were needs to improve law enforcement’s forensic capabilities.
1  Figure 5.1 reflects both Tier 1 needs and special priority needs, as discussed in Chapter Four.Figure 5.1
Building the


arting with the repository and working outward.
Need for a Research Repository
Panelists noted that important research and technology results are not widely known by the practitioner community. In this top-ranked need, they called for a repository th


ivacy,
civil rights, and cybersecurity protections for new technologies. LEAP 2 had few top-rated needs on these topics; instead, the discussion focused on privacy, civil rights, and security as attributes that needed to be embedded by default. One


ctices, and may be more realistic. One potential example cited was public commentary by law enforcement officials; while a need for more and better training was cited, it was also acknowledged that it can be difficult to foster the instincts needed f


). Change management strategies will need to be able to respond to such objections directly.
Desires for change management specifically included calls for help in evolving agency
structure, hiring, retention, and human resources. For example, paneli


nelists noted, the problems are clearer than the
solutions. Technology would seem to be an obvious means to help support improved relations; however, while the potential for new technology to create new problems is clear (for example, the failure to


sion was that agencies need to actually listen to the feedback, “truly listening” to concerns from the community, and to create a feedback loop so that the community would have evidence that law enforcement had heard their concerns, even if law enfor


d that since agencies already do not feel they are able to train offi -
cers sufficiently on topics such as technology, even on the basics, new tools and systems can leave officers worn out and overloaded.
To address this, panelists called for metho


l evidence indefinitely may result in both higher facility costs and higher labor costs to find, access, and manage all of the evidence when needed. They even noted that problems in finding and maintaining evidence in large stores could result in pro


es for selecting gear, carrying gear, and informing the public of why different types of personal gear are needed.
Technologies to Reduce Casualties: Incapacitation Devices
Panelists noted that current less-lethal weapons, including conducted energy


ticipants citing problematic consolidations of fire and police dispatch. Participants wanted operational and financial efficiencies but also wanted to retain ownership, direction, and control of important assets. Participants also called for research


need to be coordinated with the development and education efforts of DHS, such as with DHS’s First Responders Group (U.S. Department of Homeland Security, First Responders Group, 2016). Needs to reduce crime were well-covered in LEAP 1, and a number


rials to produce an initial taxonomy.
Police-community relations: strategiesR&E on sectoral/community policing practicesReview prior references to produce a quick guide on what seems to work best.
Measures for evaluating community-relations activitie


e law enforcement discipline as a whole.
Law enforcement today is facing a number of challenges, including erosion in public trust
and confidence, a rise in homicides and other violent crime that includes a spike in attacks on officers, and continu


ty (California)
Gary Woodruff  Depu ty Chief, Lawrence (Indiana) Police Department
Harlan Yu  Princip al, Upturn (D.C.)
Mike Yu   Dete ctive, Montgomery County (Maryland) Police Department69APPENDIX B
Pre-Meeting Questionnaire
Thank you for agreeing


s include tracking the numbers of sick days, long-term leave days, and health-related departures from agencies over time.
Reduce Casualties in the Line of DutyReduce the numbers of serious or fatal injuries to law enforcement, bystanders, and suspect


uding lethal, less-lethal, restraints, and related technologies
•Management, Personnel Development, and Training —including departmental poli -
cies, strategy and tactics, training, education, management and business process, andother concerns.
Quest


as you would like)
1.
2.
3.
Q7. What problems or shortfalls exist in the area of management, personnel, develop -
ment, and training  that hurt LE agencies’ ability to achieve their missions effectively and
efficiently? (List as many or as few as


, and Re-Rating the Needs
The ratings process is a variant of RAND’s Delphi method, which was first introduced in the
1950s (for example, Dalkey and Helmer, 1963). As RAND’s portal on the Delphi method notes,
The method entails a group of experts wh


list j , E(V
ij), is:
E(Vij)= CkIk
k=18
∑⎛
⎝⎜⎞
⎠⎟(MijXTijXOij).
Here,
• The first summation in the formula reflects the weighted number of objectives that the
need supports. Ck represents the weight for each objective (recall this was generated by


ntial outcomes of an effort to meet a need with a real-world benefits perspec -
tive. The approach provides for much more rigor than simply asking panelists how important each need is.
Categorizing the Needs into Priority Tiers
We used a hierarchical


, “1” represents the need that had the
highest score, and “82” represents the need that had the lowest score. The pattern of branches
shows when each
need or group of needs was combined into a larger group, with the length of
each branch
reflect


alue score for each need. Dendrogram
generated using Wessa (2016).
RAND RR1814-D.1Tier 3
(55–82)
Cut: 470
Tier 2
(30–54)
Tier 1
(1–29)
Cut: 665
Tier 1+
(1–7)
Cut: 1,03082    Identifying High-Priority Technology and Other Needs for Improving Law Enfo


ng
2. Social media strategies and tools for interacting with the public
 ii. Fiel d analytic tools or test technologies
1. Drug and alcohol screens and tests
 iii. Labo ratory tools and techniques for evidence analysis (forensics technologies)
1. Chr


ecision dashboards (e.g., COMPSTAT)
2. Tablet or mobile applications for presentation of data to officers
2. Vehicles
a. Ground
i. Automobiles
1. Patrol cars
2. Patrol SUVs
 ii. Moto rcycles
1. Patrol motorcycles
2. Patrol tricycles
iii. Bicycles
iv.


 center design
6. Real-time crime centersFull Law Enforcement Technology Taxonomy    91
7. For jail design and related technologies, see “Fostering innovation in
community and institutional corrections”
 ii. Fur nishings and contents
1. Visitor inte


ing
 iii. Pol icies and knowledge for carrying out roles
1. Standard operating procedures and department specific policies
2. Special operations plan templates and tools
e. Specialist/technologist knowledge development and trainingFull Law Enforcemen


current situation.***
O75 1 *Doctrine and Strategy for Carrying Out Agency MissionsBest practices on sectoral policing with community engagement elementsIssue: Overcoming “us versus them” mentalities between departments and communities.
Need: Develo


entions are effectiveIssue: Evidence-based management practices are often not generalizable across different agency sizes and environments.
Need: Evaluation reports need to show information about
when and where interventions/technologies are more ar


2 Acquisition and Technology DecisionmakingUse cases for portable fingerprint scannersIssue: It is difficult to verify identities when individuals lie about who they are.
Need: Collect and disseminate agency use cases and
experience with portable fi


enforcement personnelIssue: Law enforcement agencies are often “training grounds” for individuals who leave for larger agencies/salaries (especially in highly technical fields). As a result, law enforcement agencies have difficulty retaining experien


gy for Carrying Out Agency MissionsIdentify sources of assistance for agencies to measure their effectivenessIssue: There is not a common understanding of the definition of what it means for agencies to be effective.
Need: Identify sources of assista


yIssue: Reduce information overload and lack of accessibility of academic/technical papers.
Need: Academics need to work with practitioners to create
documents and training that can be read and understood quickly.* * *
O91 1 Practices and Tactics T


t between standards for recording and standards for public disclosure, which is having a de facto affect on cases.
Need: Set up a collaborative effort to identify promising
legal provisions and policies around video footage and other law enforcemen


ble F.1—continuedFull List of Needs from LEAP 2    105ID Tier Top Category Title Description
Reduce Crime
Solve Cases
Improve HealthReduce CasualtiesImprove TrustLower CostsReduce CompetenciesImprove Responses
S127 3 Firearm Range Technologies Study


ers, etc.) that are most likely to get support. **
S129 2 Staff Equipment Storage and MaintenanceStudy costs of storing excess evidenceIssue: Significant amounts of excess evidence are being retained well beyond retention standards.
Need: Study the c


 linking, forensic evidence, and tracking tools to solve more crimes.
Need: Research and dissemination on integration and
filtering of crime analysis and forensic analysis data to collectively solve crimes.* *
O17 2 Computational Tools R&D on impro


esulting in delayed justice and wasting other law enforcement resources.
Need: Examine the potential effects of “sharing
arrangements” to optimize forensic analyst labor across state and local demands. * **Table F.1—continuedFull List of Needs from


Monitoring R&D on “going dark” and
workaroundsIssue: End-to-end encryption supports both improved protection of law enforcement and the public and concealment of criminal activity.
Need: Research and dissemination on data on the nature
and extent


ratory Tools and Techniques for Evidence Analysis (Forensics Technologies)Partner with universities on studying forensic backlogsIssue: Forensics backlogs at labs (especially state and federal) are resulting in delayed justice and wasting other law e


e vendors with identifying information needed by officers at different points on a callIssue: Many information displays result in information overload for responding officers.
Need: Assist software vendors with identifying the pieces
of information


cement dataIssue: Open data/transparency may give the public the incorrect impression of what is actually going on (“shots fired” ended up being firecrackers, ”dangerous area” is a busy shopping mall).
Need: Develop best practices for ensuring that t


om LEAP 2    115ID Tier Top Category Title Description
Reduce Crime
Solve Cases
Improve HealthReduce CasualtiesImprove TrustLower CostsReduce CompetenciesImprove Responses
O29 2 Information Presentation
Tools and DashboardsR&D on interfaces for prep


esistant to facilitating data interchange).
Need: Have DOJ publish model interoperability language
that can be readily dropped into requests for proposals for new RMSs. Language will need to support being configured for different sizes and types of


interchange standards.* *
S55 2 System Integration and Information SharingModel records and evidence lawsIssue: Records and evidence laws are out of date—many were written in the paper records era.
Need: Develop model laws that can be easily adapted


 Develop data interchange standards (including
business rules) to facilitate making the connection between law enforcement records and court records.* *
S99 3 IT Systems for Managing Mission-Related DataStandards for RMS/CAD systems to include built


esImprove Responses
S63 3 Information Security Case studies on how agencies
have implemented cyber protectionsIssue: Law enforcement agencies are still susceptible to cyber attacks.
Need: Disseminate case study vignettes that describe the
successe


 more consistent incapacitation effectsIssue: Need more-reliable and safer incapacitation weapons than the status quo (conducted energy weapons, beanbags, pepperballs).
Need: Research to develop weapons and practices that
have consistent effects in


cene lighting.* *
O151 3 Pursuit Management Research on remote vehicle immobilization technologiesIssue: Insufficient pursuit mitigation/vehicle immobilization technologies to address changes in pursuit policies.
Need: Research on remote immobilizat


BC News, July 9, 2016. As of June 7, 2017: http://www.nbcnews.com/tech/tech-news/police-shootings-test-new-era-violent-social-media-video-n605366
Berg, Mark T., Eric A. Stewart, Jonathan Intravia, Patricia Y. Warren, and Ronald L. Simons, “Cynical
S


 Profiling,” Journal of Contemporary Criminal Justice ,
Vol. 28, No. 2, 2012, pp. 206–227.
Comey, Jim, “Message from the Director,” in Federal Bureau of Investigation, Crime in the United States,
2014. As of June 7, 2017: https://ucr.fbi.gov/crime-


rts References    127
Durham Constabulary, “Sir Robert Peel’s Principles of Law Enforcement,” 1829. As of June 12, 2017:
https://www.durham.police.uk/About-Us/Documents/Peels_Principles_Of_Law_Enforcement.pdf
The Economist, “Police Departments Strug


 (RMSs) as They Pertain to FBI Programs and Systems , Washington, D.C.: U.S. Department of Justice,
no date. As of December 5, 2016: https://www.fbi.gov/file-repository/law-enforcement-records-management-system.pdf/view
Federal Bureau of Investigat


ation/attachments/MPD%20Annual%20Report%202015_lowres_0.pdf
Groff, Elizabeth R., and Tom McEwen, Identifying and Measuring the Effects of Information Technologies on
Law Enforcement Agencies , Washington, D.C.: Office of Community Oriented Policing


nvironment_Playbook.pdf
International Association of Chiefs of Police, Law Enforcement Priorities for Public Safety: Identifying Critical
Technology Needs , Alexandria, Va., 2005.
———, “Recommended Guidelines for the Use of Unmanned Aircraft,” Alex


oads/2016/09/
21-Sep-16-Supt-Johnson-Announced-Major-Hiring-Plan.pdf
Jones, Jeffrey M., “In U.S., Confidence in Police Lowest in 22 Years,” Gallup, Inc., June 19, 2015. As of June 7,
2017:
http://www.gallup.com/poll/183704/confidence-police-lowe


ice
Use of Force: An Examination at Micro and Meso Levels,” Journal of Criminal Justice , Vol. 42, No. 6, 2014,
pp. 491–499.
LEITSC— See Law Enforcement Information Technology Standards Council.
Levenson, Laurie L., “Police Corruption and New Mode


ril 13, 2017:
http://www.policeforum.org/assets/docs/Free_Online_Documents/Police_Evaluation/
the%20bottom%20line%20of%20policing%202003.pdf Morin, Rich, Kim Parker, Renee Stepler, and Andrew Mercer, Behind the Badge: Amid Protests and Calls for


cademies Press, 2004. Newport, Frank, “U.S. Confidence in Police Recovers from Last Year’s Low,” Gallup, Inc., June 14, 2016.
As of April 26, 2017: http://www.gallup.com/poll/192701/confidence-police-recovers-last-year-low.aspx
NIJ— See National I


d Administration of Justice, The Challenge of Crime in a Free
Society , Washington, D.C.: U.S. Government Printing Office, 1967.
“Protests Against Police Violence Continue Across U.S.,” CBS News, July 11, 2016.
Public Law 106-297, Death in Custody R


afety ,
Santa Monica, Calif.: RAND Corporation, RR-1488-NIJ, 2016. As of June 8, 2017: https://www.rand.org/pubs/research_reports/RR1488.html
Shaw, Clifford, and Henry McKay, Juvenile Delinquency and Urban Areas , Chicago: University of Chicago
Pre


an.org/features/body-camera-update/index.html
U.S. Department of Homeland Security, First Responders Group, “First Responders,” DHS.gov, 2016. As of
November 3, 2016:
https://www.dhs.gov/science-and-technology/first-responders#
U.S. Department o


e–Community Relations in a Majority-Black
City,”  Journal of Research in Crime and Delinquency , Vol. 45, No. 4, 2008, pp. 398–428.
Wessa, P., “Hierarchical Clustering (v1.0.3),” Free Statistics Software (v1.1.23-r7), Office for Research Developmen


ponding to an active shooter. There is also a need for a review of technologies that might improve ofﬁcers’
health.
CORPORATION


Measuring Automated
Vehicle Safety
Forging a Framework
Laura Fraade-Blanar, Marjory S. Blumenthal, James M. Anderson, Nidhi Kalra
CORPORATIONLimited Print and Electronic Distribution Rights
This document and trademark(s) contained herein are protect


y held nature of AV data limits the details of what is made public or shared between companies and with the government. The report focuses on identifying key concepts and illuminating the kinds of measurements that might be made and communicated.
T


sure (Denominators), and Comparisons
for AV Safety Metrics and Potential Rating of Characteristics (outlined in Table 3.3) ........ 41

 viii Summary
The race to introduce automated vehicles (AVs) to consumers and communities is premised
in p


 (simulation, closed courses, and public roads) and deployment (on public roads).
The practice that we are calling demonstration
(while acknowledging that there are other uses
of that term) is common in product development. For AVs, it involves en


and long terms). A third type of measurement, around standards, processes, procedures, and design, is not included because it can be applied at lower levels of vehicle composition (i.e., not for complete vehicles) that evolve constantly, and because


their concerns on the
safety of the public, not on how development is progressing per se (which is the
developer’s concern).
• The opportunity to leverage a demonstration stage as a time for communication outside a company about safety (e.g., to p


so benefited from discussions with research organizations, including California PATH at the University of California, Berkeley; Exponent; kVA; the Insurance Institute for Highway Safety; and the PEGASUS Project. Conversations with Advocates for Highw


n computer-
based technologies, how to think about, measure, and discuss AV safety is not obvious. That is no small irony; proponents of AVs have emphasized their potential to be safer than the conventional vehicles they are expected to displace. In


ufacturing and mechanical engineering roots. Companies focusing on mobility as a service (e.g., ride-share services) have also entered the fray because of the potential synergy of AVs with new service models. Cross-fertilization of different communit


h as those responsible for perception processes
and for planning how the AV will move through the environment it perceives.
The macro-level systems of AV composition consist of the vehicle itself, its occupants
(themselves representing the comple


teady stream of feedback, new inputs, and discussion.
A principal constraint on the articulation of very specific metrics is the absence of publicly
available data.
13 AVs’ computer-based technologies collect and analyze vast amounts of data that


h vehicle technology and the
larger roadway ecosystem are evolving—what is judged a safe car today might not represent safety tomorrow. As reflected in media coverage and congressional hearings, the average consumer would like to compare the safety


and measures aimed at reducing the risk of accidents. . . . Traffic safety is an indicator of [the] road transport system, as a consequence of the
interaction of factors that determine i ts operation.”
d
• “In most technical contexts, safety is de


rred relative to the number of people exposed to the
potential for harm? For motor vehicles, exposure-based rates of risk have been measured based on population (e.g., per licensed driver or registered vehicle), amount of driving (e.g., vehicle mile


otective roadside
objects
Post-crash • First-aid skills
• Access to medics  • Ease of access
• Fire risk  • Rescue facilities
• Congestion
SOURCE: Peden et al., 2004.

There are a variety of views on how safe AVs should be before commer


hardware and software) in the face of a malfunction (a kind of functional safety).
29 It also
provides a framework for evaluating associated tools, such as simulators. ISO 26262 starts at the
vehicle level, and addresses testing of separate compone


p safety and security requirements for the system design. Probability
Risk Assessments and Fault Tree Analysis focus on hardware -specific failures within a system and can
assist subcomponent designers in understanding how their electronics affect


ndustry can agree to as
broadly beneficial.39 Deep consideration of policy mechanisms for motor vehicle safety is
beyond the scope of this report.
Special Aspects of Level 4 AVs
As with other levels of automated driving systems, the development


hip than consumer ownership).

41 NHTSA (2017b) outlines the role of ODDs. The industry is guided in more detail by SAE J3016 (SAE Mobilus,
2018).
42 SAE Mobilus, 2018.
43 One pair of researchers


ety Considerations  Extent to Which
Public Is Exposed  How Risk Is Controlled
Artificial settings:
simulationa  There is no risk because automated
driving is not occurring in a real environment  Not at risk • No risk exists. The
environment is


nd best practices)?
2. External validity:47 Is simulated AV performance consistent with real-world
performance under the same and other parameters (e.g., if the simulation assumes full sun
at a 90-degree angle, does the AV system perform similarl


hysical environment, although environments could be limited and/or highly controlled.53
Many closed courses exist around the world—some for exclusive use by a given developer or
OEM, others available on a membership basis or other paid use. In 2017


er situations that fall within the ODD but in which the AV cannot function, and evaluate
AV system performance after upgrades can be explored on public roads. Driving in closed courses and simulation provide additional settings to explore and circle


However, these loops risk myopia without the following:
• procedures that introduce faults (a process referred to as fault injection59) into the AV
system hardware, software subsystems, or communication between subsystems to test
robustness
•


deployment stage, where the product is released to the public (Figure 3.3). Sometimes, particularly for software-based technology, refinement and revisions (some characterized as upgrades) continue during deployment. For AVs, safety in development a


res can be generated from scenarios in artificial or public
road settings. Many developmental pathways exist, from extensive testing in a private campus

66 SAE Mobilus (20 18) outlines such fallback


me able to show that the AV does not
present a danger to the general public—especially people, but also animals and
property outside the vehicle.
¾ During demonstration, the goal is to show publicly that the AV is safe for commercial deployment o


velop “a generally accepted and
standardized procedure, for the testing and approval of automated driving functions.” PEGASUS Research Proje ct,
homepage, undated -a; PEGASUS Research Project, undated- b.
77 Koopman and Wagner, 2018.

 25 cou


 and Measuring Systems
to “ensure an independent further development and long -lasting maintenance of the standard.” Association for
Standardization of Automation and Measuring Systems, undated.
79 California Code of Regulatio ns, undated; Nowako


ubcomponents are focusing safety requirements around significant mishaps that
must be avoided while the lower -level functions  can perform as intended beyond the  safety constraints.

By appropriately defining requirements and performing specifi


col by developers,
who can be expected to design for satisfying the protocol—which is different from addressing
real-world requirements. Demonstration does not prove the vehicle is safe; at best, it shows the vehicle is not unsafe.
87
Deployment


elopment and demonstration take place in artificial
settings and on public roads, deployment only on public roads. Safety in development,
demonstration, and deployment can be assessed by multiple measures, with different interpretations and differe


outcome, but in

90 The Abbreviated Injury Scale is maintained by the Associati on for the Advancement of Automotive Medicine
(undated).
91 Government of Alberta, 2015; Occupational Safety and Healt


dated.
94 California Department of Motor Vehicles, undated.
95 California Department of Motor Vehicles, 2017.
96 Felton, 2018.
97 Prescott, 2017.
98 That reality, echoing situations observed with aviation automation, has motivated the rise


tment of Motor
Vehicles. Waymo, 2016.
100 Infractions could also include violations or misdemeanors. Exact terminology differs between states. Within this
report, infractions  refers to noncriminal violations of state and local traffic law.
101


nite number of driving
behavior rules is almost always violated. Somehow, safety envelopes were violated—one of the
RSS rules was not met by the AV or another road user. There could be other attractive ways to define the safety envelopes and
other


bjective; physics -based; a vailable using current technology; currently in use in simulation and public road
settings
Weaknesses  Fails to distinguish initiators from responders; incomplete; might not serve as a proxy for all crash types;
relati


crashes resulting in property damage over a certain cost, crashes resulting in injury, in severe injury (Abbreviated Injury Scale 3+), in death, etc. As an additional layer, outcomes can be classified by crash types or configurations. Measures that i


driving on public roads with a safety driver, meaning the simulated rate had low validity. However, if the simulation were run 5,000 times, always with the same result, the simulated rate could have high reliability. Similarly, safety in the develop


use of an excellent AV system performance or disengagement—is the developer’s purview. In the demonstration stage on closed courses and public roads, disengagement becomes informative if a human safety driver is involved (compared with an unmanned ve


engagements are not informative) from
demonstration (when disengagements are highly informative and central to a comprehensive view of safety). The demonstration stage as outlined in the report doe s not currently exist. Until the demonstration stag


h a measure might have high validity, medium reliability, high feasibility, and high non -
manipula tability.
• In demonstration using simulation s, the measure reflects the count or rate per given simulation (or per
scenario) of simula ted crashe


ne metric that has high non-manipulatability but medium validity and feasibility with another metric that has high validity and feasibility but medium non-manipulatability might present a coherent safety argument.
At every stage in each setting, th


xtent of penetration)  Per hour driving
(on public roads)  All roadway types, in all weather, at all
speeds, with clear lane markings and no pedestrians, bicyclists, or scooters detected Same model –year conventional
vehicles with ADAS, excluding


erwise, the comparison is
biased. This concern gives weight to the question of which conventional vehicles should be
included. Contrasting an HAV to a 12-year-old car (the current average vehicle age on the road in 2016
116) is biased because of th


Level 4 vehicles because the system is expected to be able to operate (monitor the driving
environment; steer; accelerate or decelerate; and respond effectively where it cannot do those
things— i.e., fall back) without the aid of a human driver wit


ng on what consumers need to understand, what regulators need to know, and what information companies need to communicate internally—or, possibly, externally within the industry. Although taxonomic guidelines found in SAE J3016 map levels of automati


ow for differing lengths of time to be analyzed, but still require time to be bucketed. Bucketing exposure requires developers to decide when the system has changed sufficiently to start measuring safety afresh (i.e., to start a new bucket).
One po


there will be larger delays and larger variations in when the upgrade is downloaded.
Figure 3.6. Decay of Influence of Time Period n over Time

The Ecosystem
Thus far, safety has been considered at the system level. As Figure 1.2 shows, the eco


e in immediate proximity to or contact with, elevating the safety of the entire fleet.
127
An ecological study could be implemented in many ways. The simplest option is to compare
a safety metric for location X to location Y, where X has AVs and Y


mendations aimed at furthering broad public dialogue about the safety of vehicles that are highly (and eventually fully) automated.
AVs in the Broad Motor Vehicle Context
Beyond passenger vehicles, automation is of interest for trucks and buses. A


t (e.g., the closed course of the American Center for Mobility
134); so do
efforts to promote development and use of common scenarios (e.g., the PEGASUS project),
common testing code and procedures (e.g., Voyage’s Open Autonomous Safety Initiative1


sultant information
3. how the system interacted with the external environment
4. safety lagging or leading events
5. cybersecurity incidents (which will be discussed in next section).143

Theoretically, all data could be shared between compani


 have a history of buying, evaluating, and rating automobiles. A new
data set containing detailed crash reconstructions for all lagging-measure events at or above a
given severity would enhance understanding of how AVs fail and what happens when th


on rely at least in part on some kind of connectivity with
other vehicles, infrastructure, or the internet. This connectivity is probably necessary, but it also increases cybersecurity risks. Even for conventional vehicles, over-the-air communicatio


apers on
vulnerabilities, threats, and attacks on automotive systems.
151 Consumer Reports has weighed in on this issue. Barry, 2018.
152 For a discussion of how AV cybers ecurity lapses might implicate liability, see Winkelman et al., forthcomi


154 Winkelman et al., forthcoming.
155 Evtimov et al., 2017.
156 The process of learning by doing, combining different modes of development and testing to capture unanticipated
circumstances and challenges, was documente


stem’s website (undated) underscores that associated reporting is “Confidential.
Voluntary. Non -punitive.” Other reporting channels for safety incidents in aviation include the Federal Aviation
Administration Hotline  (2018) and the National Trans


mportantly, effective risk communications can lead to the safer use of the vehicles. If there are particular circumstances that lead to even small amounts of increased risk, warning users of these risks could lead to increased vigilance and reduced c


Vs, as Chao noted. When elevators
were initially developed, there was considerable public fear. Being supported by unseen machinery while riding in a box that had the potential of dropping one to one’s doom likely elicited considerable fear. To addr


 more miles driven) . Demonstration is undertaken
by testing through simulation and on closed courses and public roads with safety drivers—the
settings discussed in this report. In closed courses and on public roads, demonstration protocols must co


ote a safety culture. The great race to develop safe and practical AVs should contribute to the overall goal of safe motor vehicle environments. This report’s framework is intended to assist in that process. At the same time, it is important to ackno


 these early results requires care and
caution because of differences in severity threshold and a lack of generalizability beyond the
geographies, time periods, and manufacturers included. Additionally, results associated with the AVs reflect the c


206e1bb49731bf3.pdf
Association for the Advancement of Automotive Medicine, “Overview,” webpage, undated. As
of September 4, 2018:  https://www.aaam.org/abbreviated-injury-scale -ais
Association for Standardization of Automation and Measuring Syst


al Report of Autonomous Vehicle Disengagements,
undated.
———, “Autonomous Vehicle Disengagement Reports 2017,” webpage, 2017. As of September
4, 2018: https://www.dmv.ca.gov/portal/dmv/detail/vr/autonomous/disengagement_report_2017
Centers for D


7.
———, “FAA Hotline Reporting Form,” August 13, 2018. As of September 5, 2018:
https://hotline.faa.gov
Federal Highway Administration, Surrogate Safety Assessment Model and Validation: Final
Report, McLean, Va.: U.S. Department of Transportation


of September 4, 2018: https://www.theverge.com/2018/1/10/16874410/ voyage-self -driving-cars-villages-florida-retirement-communities
Hayward, John C., “Near Miss Determination Through Use of a Scale of Danger,” in
Proceedings of the 51st Annual Mee


n.
 67 Jha, Saurabh, Subho Sankar Banerjee, James Cyriac, Zbigniew Kalbarczyk, and Ravishankar K.
Iyer, AVFI, Fault Injection for Autonomous Vehicles, Luxembourg, IEEE/IFIP International
Conference on Dependable Systems and Networks, June 2018. A


th authors, June 11–August 24, 2018.
Laris, Michael, “Federal Researchers Are Using Data from Waze and Maryland to Try to Predict
Road Dangers,” Washington Post, 2018.
Le Coze, Jean-Chrisophe, Kenneth Pettersen, and Teemu Reiman, “The Foundations


rporation, PE-241-RC, 2017. As of September 2, 2018: https://www.rand.org/pubs/perspectives/PE241.html
Möller, Niklas, Sven Ove Hansson, and Martin Peterson, “Safety Is More Than the Antonym of
Risk,” Journal of Applied Philosophy, Vol. 23, No. 4,


files/pre-crash_scenario_typology-final_pdf_version_5-2-07.pdf
———, Safety-Related Defects and Automated Safety Technologies , Washington, D.C.: U.S.
Department of Transportation, NHTSA Enforcement Guidance Bulletin 2016-02, Docket No. NHTSA-2016-0


ournal of the Transportation Research Board, Vol. 2559, 2016, pp. 65–72.
 72 Nowakowski, Christopher, Steven E. Shladover, Ching-Yao Chan, and Han-Shue Tan,
“Development of California Regulations to Govern Testing and Operation of Automated
Driv


:
Kluwer, 1991, pp. 175–217.
Roose, Kevin, “Can Ford Turn Itself into a Tech Company?” New York Times Magazine,
November 9, 2017. As of August 31, 2018: https://www.nytimes.com/interactive/2017/11/09/magazine/tech-design-autonomous-future-cars-det


of-passenger-cars-in-the-united-states/
Stern, Paul C., and Harvey V. Fineberg, eds., Understanding Risk: Informing Decisions in a
Democratic Society, Washington, D.C.: Committee on Risk Characterization, National Research Council, 1996.
Stilgoe


d Prevention, Vol. 71, October 2014, pp. 82–92. As of September 11, 2018: https://www.sciencedirect.com/science/article/pii/S0001457514001365
 76 Waymo, Report on Autonomous Mode Disengagements for Waymo Self-Driving Vehicles in
California, Mounta


a new kind of measurement.  While acknowledging that the closely held nature of AV data limits the amount of data that are made public or shared between companies and with the government, the report highlights the kinds of information that could be p


Zero Days,
Thousands of Nights
The Life and Times of Zero-Day
Vulnerabilities and Their Exploits
Lillian Ablon, Andy Bogart
CORPORATION
Limited Print and Electronic Distribution Rights
This document and trademark(s) contained herein are protected b


ap might justify retention. But without information on the over -
lap, or concrete metrics based on actual data, it is challenging to make a well-informed decision about stockpiling.
To address this question, RAND obtained rare access to a dataset of


widely to policymakers, practitioners in law and business, other researchers, and the public.
The ICJ is part of RAND Justice, Infrastructure, and Environment, a division of
the RAND Corporation dedicated to improving policy- and decisionmaking in a


earchers: Who Looks for Vulnerabilities?  ........................ 73
C. How Mitigations Have Affected Exploitability: Heap Versus Stack
Exploitation Case Study  ....................................................................... 79
D. Close Co


Smoothed Hazard Function  ............................................................ 91
  F.3. 365-
Day Time Interval, with Percentages per Group and Interval  ............... 96
Tables
 3.1.  Description of Various Life Statuses  .................


en the zero-day vulnerabilities or exploits the U.S. government keeps and those its adversaries are stockpiling. If both sides have the same stockpiles, then some argue that there is little point to keeping them private—whereas a smaller overlap migh


 as alive (publicly unknown) or dead (publicly
known) may be misleading and too simplistic
Common practice is to classify a vulnerability simply as alive  (publicly unknown) or
dead (publicly known); however, our analysis revealed that there are s


ties—in particular the ones that exploits are created for gray, or government, market use—are likely old.
While our data showed that a short life is 1.5 years, this might be long enough for
most vulnerability researchers.
Finding #3: No characteri


interval (i.e., all of our data in one time interval) yielded a 40 per -
cent overlap. With the exception of the 14-year interval, our data show a relatively low collision rate. This may be because those in the private exploitation space are looking


lities and develop -
ing exploits in-house if a long life is desired.
Exploit development time does not appear to have an influence on the lifespan or
survival time of an exploit.
At the most basic level, any serious attacker can likely get an affo


it does not come from labor, but rather the value inherent in them and the lack of supply. Other tangible costs (acquiring products to find the vulnerabilities in, setting up test infrastructure, main -
taining and porting the exploit to work on mult


chwarz, Caolionn O’Connell, Igor Mikolic-Torreira, and Sasha Romanosky provided invaluable insights and feedback. We also thank Ernesto Amaral for his exper -
tise and input on methodologies for evaluating demographic data, Andrea Golay for her thoug


reate an exploit for a zero-day vulnerability.
Little Is Known About the Extent, Use, Benefit, or Harm of Zero-Day
Exploits
Software vendors aim to create flawless software, but this is aspirational. Devices, net -
works, systems, and processes in


the file system) does not mean the implementation is available (perhaps there is a fixed/static file that is not part of any other directory, thus, nothing an attacker enters will ever get to that desired file location) (Libicki, Ablon, and Webb, 20


 used in some operations while still a zero-day.
Beyond these high-profile examples, little is known about the true extent, use,
benefit, and harm of zero-day exploits. Discussions are often speculative or based on what is discovered after the vulne


atch against anyone using it, but it may also leave a critical population vulnerable. Furthermore, who finds the zero-day makes a difference—for example, an affected vendor or a bug hunter working on the vendor’s behalf, or cybercriminals or other ne


and version of code, how often it gets patched, how long a vulnerability should be held before it is made public, and who the target is.
There has been much discussion on this topic: Following the discovery and disclo -
sure of the Heartbleed vulner


vendor)?12
It may be additionally helpful to know:
4. Cost:  W
hat is the cost to develop an exploit for the vulnerability (e.g., in order
to help set purchase price)?
Answering these questions can be useful in determining which vulnerabilities to


ng zero-day exploits.
Other groups that may be interested in the research results include:
•
 Those who a
re actively looking for vulnerabilities and developing exploits for
offensive purposes. The results could help them be more efficient in develo


t is no longer considered a potential harm to the public.
But this view is largely oversimplified. Vulnerabilities can remain privately known
but get quietly patched (so thus are no longer considered zero-day); conversely, vul -
nerabilities can be


represent those that might use zero-days against the interests of Blue. Furthermore, there is an assumption
is that, if a vulnerability is private, it is (1) in use and (2) in use by a nation-state or criminal. But vulnerability researchers we spoke


ad been cre-ated, and if and when the vulnerability was found externally by a third party. It also contains information on the analyst who wrote the exploit and the relevant security bulletin (where applicable).
The dataset contained numerous catego


erabilities
and public knowledge vulnerabilities (i.e., the teal overlap between the blue and green in Figure 1.4).
Our research focuses on the characteristics of vulnerabilities that are in the over -
lap between Blue (or a reasonable proxy for Blu


find vulnerabilities, they
must determine whether or not they are able to develop an exploit. Many factors can come into play when decid -
ing whether an exploit will be developed to take advantage of the vulnerability. For example, there may be oth


t reliable for multiple
versions, but this timeline was not captured.
25  These dates are fairly accurate, as exploit developers tend to be quite excited, proud, and eager to share with
their colleagues when one of their exploits is fully function


 size for
analysis, as it meant that we could use only those with complete information for per -
forming survival analysis and calculation of expected lifespan. In the cases where we had only the month and year timestamp for the birth date, we defau


ty researchers (Appendix B), a case study on how mitigations have affected exploitability (Appendix C), an investigation of close collisions (Appendix D), a first order look at cost and pricing considerations for zero-day exploits (Appendix E), and a


 core develop -
ment team), the “blocker bug” submitted for patch allows the introduction.
3  For example, the introduction of call flow graphing in Windows 10.18    Zero Days, Thousands of Nights: The Life and Times of Zero-Day Vulnerabilities and T


 emerged that would assist in exploitation.
5  For the cases in our data where no exploit was created (although exploitable vulnerabilities were found),
approximately half were not exploited because certain capabilities were required or conditions


  Here we are talking about reliability for exploits, not implants or “payloads.” Reliability is desired for exploits,
but it is not always guaranteed. Reliability for an implant, however, is critical. While an exploit provides initial access, an im


ind and what needs to be cleaned up). For more details about the exploit development cycle and some factors that go into exploit development, see Appendix A, “The Exploit Development Cycle.”
People in the Zero-Day Vulnerability Space
Some vulnerabili


country), “black hat” cybercriminals, or hob -
byists with various motives.
18 The most sophisticated aim to not only find zero-day
vulnerabilities but also to create fully functional exploits for them.
While exact numbers are unknown, many estimat


s in the lowest tier is at least an order of magnitude larger than in the highest tier. For example, there are more than 26,000 researchers on the BugCrowd platform (though it is unclear how many are active bug hunters, as only ten have collectively


e government market. More Discussion of Zero-Day Vulnerabilities    23
cases, able to avoid detection. As such, PoC exploits are generally an end goal for white
hat researchers, whereas the gray and black markets deal in fully functioning exploits.2


as a “master of the dark arts” to get custom-
ers. Once a vulnerability researcher gained traction, she or he could shift to the white market.
25  One vulnerability research group shared that they had three to four entities to which they sell; anoth


-in-the-middle” refers to an attacker who is in the middle of the legitimate
sender and receiver, relaying and possibly altering the communications between the two parties.More Discussion of Zero-Day Vulnerabilities    25
Exploit Developers (and Exp


ion-based model, whereby subscribers get regular information about zero-
day vulnerabilities that have not yet been disclosed. Often, subscribers use the feed to test their own products against zero-day vulnerabilities to further secure them, or use


rability or vulnerabilities that the exploit is taking advantage of. An exploit dying means that one of the previously unknown vulnerabilities (if multiple exist) gets discovered.
1
1  It could also have a patch available, but at the most basic level


sing terms such as “dead” may be misleading, because the vast majority of cyber actions
use these “dead” N-Day vulnerabilities. One expert we spoke with suggested we use the terminology “a vulner -
ability previously undescribed in literature,” espe


 the BUSBY vulnerability researchers found that it could provide privilege escalation—something much more severe.
4  For example, one of BUSBY’s exploits had a vulnerability that had been found in a 32-bit version of a system
but was not exploitable


so a security patch will never be issued).
Security Patch A vulnerability found by a third party and recognized as a security vulnerability; an advisory, patch, and/or CVE has been issued.
Killed by BUSBY A vulnerability publicly disclosed by the pri


abilities). Figure 3.4
Vulnerabilities Found by BUSBY Each Year, by Current Life Status (n = 207)
RAND RR1751-3.40 5 10 15 20 25 30
2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Number of vulnerabilitiesUncertain
Code refacto


at had a date of birth (n = 127).
8
8  Seven of our exploits that died had no birthdate recorded, as their vulnerabilities were purchased by a third
party. We do have date of maturity for those exploits (i.e., the date when a fully functional explo


 plots, we obtained the median survival time—
exploits that had a life status of Immortal; however, because survival analysis and life expectancy analysis are
typically done on humans or things that are all expected to die or fail at some point, we


ause there were more data points to draw from, providing greater precision in the estimate of sur -
vival probability.
11
After initial detection, any given exploit (and thus its vulnerabilities) has a median
survival time of 5.07 years (95 percent


t developer) to detection by outside partyAnalysis of the Data    37
1. Vulnerability type:  Mem ory Corruption, Memory Mismanagement, Logic,
Mixed or Other
2. Platfo
rm affected:  Linux, OSX, Open Source, PHP, Unix-based (excluding
Linux), Windows


is analysis on their own, so they were combined.
15  Curves that end in a horizontal line indicate that the last observed value was alive; curves that end in a verti-
cal line indicate that the last observed value was death (i.e., public disclosure)


r data, based on its assumption of a constant hazard function, which our data appear to exhibit. Further details about the hazard function and plots of our unadjusted hazard estimates and parametric model comparisons can be found in Appendix F, “Addi


ies) is expected to live 6.90 years (with 95
percent confidence interval: 5.39 to 8.84 years).18
Proportional Hazards Regression Modeling
To assess whether any particular characteristic of an exploit contributes to a lower or higher probability of


or without adjustment for the other characteristics. Without more data, we
cannot statistically determine whether any characteristic of an exploit indicates a long or short life. The high p-values may be a result of a true absence of effect, or a re


rich debate that many have spoken and written about. Schneier (2014) believes that vulnerabilities are plentiful (which one might interpret as dense), and as such the overlap is relatively small; Geer (2014) states that vulnerabilities are sparse en


In particular, those who rely on zero-day exploits for offensive or defensive purposes, such as penetration testing or trying out a new detection tool, may want to know how long they can expect to go undetected by their target during a campaign. Vul


efore, the fewer number of points in the higher diagonal bands is
artificially lower than what the complete plot would display, if we had had additional follow-up time available.
Because this analysis does not take into account censoring, we cannot


unable to determine the dates that those exploits were found.
26  Certainly, there are many times when the use or usefulness of an exploit does not rely on its vulnerabilities
being zero-day. But for the sake of argument, we consider the cases where


o outstanding trends for those exploits (n = 16) that had an exploit development time of more
than 90 days: Over 56 percent are still unknown (with one Immortal, and one Code Refactor), and roughly 31 percent are known and have had security patches.


 of exploits
Time to develop an exploit (days)55
21 31
13
5 7 5 5
1 4
0 0 12
0–10 10–20 20–30 30–40 40–50 50–60 60–70 70–80 80–90 90–
100 100–
110 110–120 >120 Analysis of the Data    49
We also evaluated exploit development time compared with


ghts: The Life and Times of Zero-Day Vulnerabilities and Their Exploits
Implications for Defense
Oversimplifying vulnerabilities as either alive (publicly unknown) or dead (publicly
known) may be creating a barrier for vulnerability-detection effort


 help continuously bring new researchers to looking at code already deemed clean.
Finding and patching vulnerabilities in code becomes harder with software reuse
between platforms and, in particular, with the Internet of Things, as more code is reu


osed, it often leads to the
discovery and disclosure of other similar vulnerabilities; this could be a factor in increasing collisions. 54    Zero Days, Thousands of Nights: The Life and Times of Zero-Day Vulnerabilities and Their Exploits
On the ot


icator of a long life was a vulnerability
being found in a class that had already been thoroughly examined by other vulnerability researchers (e.g., font bugs). Other vulnerability researchers we spoke to agreed with this supposition.
3  Of course,


rate may be
4  This is a prime example of how data can be used to support a variety of viewpoints, in particular a high versus
low collision rate. Those that have argued that the overlap is small (or large) do not typically specify the time
interv


ing a product to the public, or study the techniques of how offense finds vulnerabilities. Additionally, defenders could consider tackling software security by treating all code as insecure, and investing
5  There are many analogies for this swarm m


al (with a median of 22 days, minimum of 1 day, and maximum of 955 days).
The cost to develop (and, relatedly, the value or price of) an exploit can rely on
many factors: the time to find a viable zero-day vulnerability (research time), the time to


ly.
In our data, vulnerabilities purchased from external third parties had a shorter
lifespan (average life of 1.4 years). This may be an argument for finding vulnerabilities and developing exploits in-house if a long life is desired.
Governments c


 code bases and previous versions—exploitable vulnerabilities may still exist, and older products may still be in use around in the world.
To beat the software testing programs (“fuzzers”) and bug hunters searching for
vulnerabilities in the public


aracteristics not necessarily mutually exclusive). If another vulnerability usu -
ally exists, then the level of protection consumers gain from a researcher disclosing a vulnerability may be seen as modest, and some may conclude that stockpiling zero


 the affected vendor finds a vulnerability, it may not say as much. Fur -
thermore, some researchers choose to not to disclose vulnerabilities they have found.
There are several reasons why an exploit developer or bug hunter may have found
a vulner


nd another might focus on web-based applications and server-side software. Others are not as discriminating.
14 Because
our data came from one source, not all vulnerability or product types are represented in the data (e.g., indigenous versus commer


nvolved in discus -
sions. We only scratched the surface of what is possible to investigate with data. More
research threads exist. For example:
•
 What (
if any) characteristics of zero-day vulnerabilities indicate a long or short
life, and what


ro-day?65APPENDIX A
The Exploit Development Cycle
The exploit development process consists of many steps, each of which can go through
multiple iterations.
Phase 1: Discovery and Verification
In the first phase, vulnerability researchers search for


 are created equal. Some have the ability to go deeper than others, and some find different bugs
depending on their method. For example, AFL (American Fuzzy Lop) finds bugs quickly by tracking down every
“if statement” in a code base.
2  Taint ana


concept). Once narrowed down, the researcher determines exactly what can be done with the crash (e.g., remote code execution, information leak, crash a different process), the potential impact caused by the crash, and other requirements needed to red


not exist in an operational environment. In an operational setting, the target system may be under heavy load, and different language packs and dynamic link libraries (dll’s) may be loaded differ -
ently. Thus, any lab-based assumptions about process


y of estimates for the percentage of crashes or triggered bugs that would turn into useful, exploitable vulnerabilities: 0.5–1 per -
cent, 15–20 percent, 25–30 percent. One vulnerability researcher shared that that as high as 50 percent of vulnerabil


anyone can write fully func -
tional exploits. Conference talks, reports, and books can outline how an exploit was created. But taking that information and writing an exploit for a new vulnerability still requires skill. Some liken it to the culinary


publicly
unknown—and alive—for a long time) may depend on the ability of the exploit devel -
oper to go several layers deeper into code spaces than normal fuzzing (which is typi -
cally on the surface).
Some vulnerability researchers are better tha


g a vulnerability. Figure B.2 shows that the rate of devel -
opment is not steady: In some years, a researcher may not find or develop an exploit.
From these data, it appears that there is a high frequency of early developers (e.g.,
researchers U,


 career of vulnerability researchers in the gray, or government, markets with the life-time of bug hunters in the white markets (i.e., bug bounties, vendors, and groups such as Google’s Project Zero).
4  Interestingly, white hat bug hunters who inve


nted, since some exploits developed and vulnerabilities found were not reported or recorded in our data due to sensitivities or ongoing operational use at the time of collection.
6  The low count for August is statistically significant, while the hi


d manipulate than the heap).
1 Because of this, exploits based
on stack overflows were more common, and, as such, security vendors focused on cre-ating mitigations for the stack (e.g., stack cookies, stack canaries, stack address space layout random


an exploit devel -
oper examining the details of a newly released vulnerability advisory, and finding other similar—and still publicly unknown (i.e., alive)—vulnerabilities.
Twenty-four of BUSBY’s exploits are close collisions. As of our information


e are correct, or getting the right configurations and infrastructure set up. For those who fuzz, once the fuzzing tool is set up, it can be very fast to find vulnerabilities—one person gave an estimate of
1  This included two teams of exploit devel


e developers to keep
looking. Even then, the research is not continuous. If the exploit developers have not found anything after a few
months of looking (and cycling through a handful of exploit developers), they may wait until a new version of the


sts (premium of a high-demand, low-supply product, etc.) can cause the price to rise dramatically. And, to be clear, our analysis does not take into account the cost or time of failing to find vulnerabilities or to develop a fully function -
ing expl


rabilities claimed that gray market prices of exploits for the Tor Browser Bundle were two orders of magnitude higher than prices paid on the white market for Firefox vulnerabilities. Others we spoke to noted that it is often common practice for expl


pe a sandbox, or remotely jailbreak [for mobile devices], then the payout is greater).
In the end, for those who sell their exploits, the entity that purchases the vul -
nerability can often be the ultimate decider of what to purchase and for how mu


Kaplan-Meier curve, though they tend to overestimate the Kaplan-Meier curve in the first two years. Subsequently, each approximates the Kaplan-Meier curve’s general shape, with differing rates of decay starting in year five. We chose the exponential


the exponential model-based estimates of expected lifetime.
We used Cox proportional hazards regression modeling to determine the hazard
ratio for each characteristic.
The hazard function itself can be interpreted as the instantaneous probability o


a common model. In the partially adjusted model, we removed the “platform affected” category, because source type is heavily correlated to the platform affected (e.g., all Windows is closed source) and just performed regression adjusting for vulnerab


AOL, Ethereal, Adobe, Alt-N Tech -
nologies, CryptoCat, and RealPlayer/RealServer, as well as some lesser-known vendors.
Table G.1
Data Frequency Counts
Row Labels Count
Life Status
Living 66
Code Refactor 21
Immortal 13
Killed by BUSBY 8
Other—featu


ies and Their Exploits
Row Labels Count
Remote code injection 1
SQLi 1
Stack overflow 40
Stack overflow (x2); memory 1
Stack overflow; heap overflow 1
Type confusion/object mismanagement 1
Unsecure environment variables (so can eventually take
advan


am attempts to put data in a memory area past a buffer (Open Web Application Security Project [OWASP], undated-a).
bug Software flaw.104    Zero Days, Thousands of Nights: The Life and Times of Zero-Day Vulnerabilities and Their Exploits
bug bounty A


n vulnerability for the version of the product it was created for; that product is no longer main-tained (so a security patch will never be issued).
implant A program that solidifies and maintains access initially pro-vided by an exploit (i.e., achie


htly controlled environment that restricts permissions
on what can be run (to prevent malicious code from execut-ing or accessing something it should not, for example).
sandbox escape Code that can escape a sandbox.
stack A data area or buffer used


toc9
BugCrowd, “What’s a Bug Worth?” 2015. As of January 30, 2017:
https://pages.bugcrowd.com/whats-a-bug-worth-2015
———, “The State of Bug Bounty,” June 2016. As of January 30, 2017:
https://pages.bugcrowd.com/hubfs/PDFs/state-of-bug-bounty-20


, 2017: http://www.reuters.com/article/us-cyber-blackhat-apple-idUSKCN10F2TX
Geer, Dan, “Cybersecurity as Realpolitik,” BlackHat, August 6, 2014. As of January 30, 2017:
http://geer.tinho.net/geer.blackhat.6viii14.txt
Gellman, Barton, and Ellen Nak


de: Platinum Edition ,
New York: John Wiley & Sons, 2006. Kuehn, Andreas, and Milton Mueller, “Analyzing Bug Bounty Programs: An Institutional
Perspective on the Economics of Software Vulnerabilities,” paper presented at 2014 TPRC: The 42nd Researc


Market , presentation given at RSA Conference 2015, San Francisco, April 20–24, 2015. As
of January 30, 2017:  https://www.rsaconference.com/writable/presentations/file_upload/ht-t08-the-wolves-of-vuln-street-the-1st-dynamic-systems-model-of-the-0da


d-hackers-fix-cybersecurity-holes-or-exploit-them/371197/
Schryen, Guido, and Eliot Rich, “Increasing Software Security Through Open Source or Closed
Source Development? Empirics Suggest That We Have Asked the Wrong Question,” paper presented at th


tems,” in Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security , New York: ACM, 2015.
Zorabedian, John, “How a Hacking Team Got Hacked,” Naked Security , April 2016. As of
January 30, 2017:  https://nakedsecurity.so


CORPORATIONKnowing More, but
Accomplishing What?
Developing Approaches to Measure
the Effects of Information-Sharing on  Criminal Justice Outcomes
Brian A. Jackson, Lane F. Burgette, Caroline Stevens, Claude Messan
Setodji, Erinn Herberman, Steph


ial for governmental boundaries to inhibit those efforts, because criminals frequently do not limit their activities to the jurisdiction of a single law enforcement agency.
Although there are high-profile success stories in which data-sharing assiste


orts to measure its effects on performance have been
few and far between. Anecdotal success stories of cases in which data from sharing systems were critical to cracking a case or quickly stopping a crime in progress are easy to identify. But even t


 over -
all crime rates—is a starting point to gauge whether more data-sharing helps improve those measures. Other possible measures include the speed or probability of arrests or of officers locating and interviewing a suspect. As in some previous e


Three Sets of Measures Linking Sharing to Criminal Justice Outcomes
In collaboration with ARJIS and informed by insights from its users, we tested three
approaches for measuring the effects of information-sharing on criminal justice out -
comes. Th


re than one agency. And the time to the next contact with a different agency was shorter as well: 1.3 years for the group with ONS records versus 1.7 years for the control group without such records. Furthermore, individuals with ONS records were mor


o iden -
tify crime series across jurisdictions. Although our analysis was suggestive that BOLO notifications increased arrests for cases in which a BOLO was used, those effects did not meet the requirements for statistical significance.
Information


verage monthly number of cross-
 jur
isdictional xiv    Knowing More, but Accomplishing What? Measuring the Effects of Information-Sharing
arrests increasing from 0.044 to 0.048, while the estimate for the probability of making
a cross-jurisdictiona


lp practitioners across the justice system do their jobs better. Thinking now about which data should be recorded and kept to make that sort of high-resolution analysis of the effects of criminal justice data on outcomes possible would make a down pa


nd the threat of terrorism were frequently the focus of
debate about information-sharing, such sharing is just as important for other functions in the criminal justice system—specifically, data moving across jurisdictional bound -
aries so that a po


organization of criminal justice in the United
States (as well as the broader U.S. homeland security enterprise, which includes mul -
tiple federal, state, local, and tribal agencies with many distinct missions), individual organizations are often n


ocalities).
2
Other efforts exist across the country at the state and local levels or designed
around the specific information-sharing needs of particular areas (e.g., the National Capital Region surrounding and including Washington, D.C.). These sh


f Police, undated).
3
While it would be difficult to estimate the resources that have been spent put -
ting information-sharing systems and initiatives in place and supporting their oper -
ation, national expenditures on information-sharing have bee


tigating crimes; detecting threats; and making decisions about charging, sentencing, or taking other criminal justice actions involve myriad decisions, the quality of which are related to the completeness, relevance, and currency of the data availabl


the citizens who are its subjects, and depending on how the sharing systems are designed, the information might lose its context—and its meaning and implications might therefore shift—as a result of being shared (Jackson et al., 2017). The accuracy o


ormation-Sharing?    7
saging system and sought to connect its use to arrests, case characteristics, and other
measures (which showed mixed results).3 Koper et al. (2015)’s study on technology
applications in law enforcement evaluated an internal l


eading in the criminal justice system, such user perception studies sought
to link the extent of system use to outcome effects. For example, looking at detectives, Danziger and Kraemer (1985, p. 200) linked use of computing to users’ reports of “cas


 on outcomes of interest limited the ability of both critics and pro -
ponents to support their arguments and to have a productive debate on these systems and capabilities.
Logically Linking Information-Sharing to Outcomes for Developing
Measures
Fo


olving more crimes), and the efficiency of such activity (e.g., solving them more quickly) may therefore contribute to improvement in broader (and potentially more societally impor -
tant) measures. Reflecting the complexity of the environment in whi


 his behavior might have justified doing so (e.g., the officer may learn that the subject has a mental illness that causes him to react aggressively, so rather than arrest the sub -
ject, the officer refers him to treatment). Such actions might be re


d the former measure would capture the fact that rapid contact with suspects could have benefits for containing escalating situations and more rapidly clearing cases by arrest, as well as more rapidly resolving suspicion of individuals who turned out


stem will not be valuable.
10 In practice, these two effects will be mixed—because getting data to an investigator or officer faster (efficiency)
may also help him or her identify and successfully apprehend the perpetrator (effectiveness).12    Kno


nformation-
 sha
ring—whose effects would be expected to be observable at the indi -
vidual case level—would also be straightforward to assess even for systems that have already been implemented, assuming comparable groups of cases in which the shari


st
crime, there may be one critical dot (or a small number of dots) that makes the picture make sense. 14    Knowing More, but Accomplishing What? Measuring the Effects of Information-Sharing
sible people to contact who were unproductive in solving


tunities for policing to prevent it.
Unlike the simple and intermediate cases, where the question was how to appro -
priately distinguish the different mechanisms for how information-sharing affects deci -
sions and outcomes, in this broader case, di


d into the system in different ways (or different information-sharing features may be relevant to different people), making it possible to distinguish different outcomes. For cases, investigating officers might use the system more or less, or differe


ata in the field, mapping, crime analysis tools, and an enter -
prise system of applications that help users solve crimes and identify offenders. In this capacity, ARJIS has developed replicable open source technical solutions, policies, and governan


 and officers in managerial roles. (Detectives made up slightly more than a quarter, and managerial officers made up less than a quarter.)
Table 3.1
Summary of ARJIS Tools
Class Tool or Feature Description
User pull User query of linked databases
(


 particularly help -
ful would have been resolved without the availability of cross-jurisdictional information-
 shari
ng.1
The ARJIS users we interviewed varied considerably in the circumstances in
which they worked, with variation even among pract


f the cross-jurisdictional systems is prominent for many users, it also demonstrates the evaluation challenge presented when an information-sharing system is not the sole system available to criminal justice practitioners in an area. One driver of th


h not all interviewees used ARJIS tools the same amount or for the same
things, nearly every respondent credited the tools with increases in their productivity.
Some estimated that the tools provided by ARJIS at least doubled their productivity, sa


 like information-sharing—
whether within single agencies or across agency boundaries—requires differences in the level of the intervention (ideally with all other factors constant) to make it pos -
sible to look for differences in outcomes. Thinking


outinely record created opportunities for this type of evaluation in some cases and,
in others, seriously complicated the effort. We return to those lessons at the end of the report, because they can inform thinking about system design for areas tha


th respect
to different suspects or offenders could be used to identify effects for a subset of criminal justice outcomes. This analysis option is most relevant for case clear -
ance (e.g., where more searches regarding suspects in a case might lead


ls operating across jurisdictional lines could also provide evidence for the effect of information-sharing efforts.
Because this research effort was based in a geographic area that has had cross-
jurisdictional information-sharing in place for decad


fic loca-tions of officer deployment over time) that were necessary to do some analyses. In other cases, limitations in the way data are captured and integrated into the cross-

jur
isdictional information-sharing systems made it difficult or imposs


ring of information (for notifications related to investigations), the type of record examined in our evaluation experiment pushed information to the user doing the searching, making specific information more prominent and thus reducing the chance it


 bends down in Figure  2.
1.
13 The complete analysis is documented in Kovalchik et al., 2017.Developing and Testing Measures with the Automated Regional Justice Information System    27
One set of measures we tested focused on the role of this ARJI


that ONS records caused the differences. The observed effects may be driven by systematic differences between the two groups that we could not observe; for example, if the offenders with ONS records were higher risk than those without, such a differe


LO notifications’ effects on crime cases,
several effects might be observable.
14 Effects at the case level (e.g., a BOLO issued about
a crime case might increase the chance it was cleared) and at the suspect level (e.g., a BOLO citing a person as


tive underestimates of their actual effects.
In an effort to further control for any differences that the matching process did
not address between suspects in BOLO cases and suspects in the comparison cases, we compared the pre-post differences in


 improved information-sharing is addressing the fact that criminal activity crosses jurisdictional boundaries; for example, a single person or group may commit crimes in many suburbs of a metropolitan area.
16 ARJIS users
employed BOLO notifications


 perc
ent offended in adjoining jurisdictions, and 7  perc
ent offended in jurisdictions of
nonadjoining police forces (Wiles and Costello, 2000, p. 26). As cited previously, simi -
lar work in the  Net
herlands indicated that offenders who moved ac


tomated Regional Justice Information System    33
submitted to ARJIS that the arrest was related to a warrant originating in another
jurisdiction. If a suspect’s cross-jurisdictional arrest occurring after a crime case was not
associated with a wa


s, narrowing the matching criteria), the effects were in the same directions but got weaker. At the most restrictive—looking at cases that involved warrants—an increase from zero to ten queries was associated with the average number of cross-jurisdic


tered during the project also produced a different set of results: lessons on how systems could be designed to make this sort of effort more straightforward by building in fea -
tures or addressing key problems. These lessons would make it easier to


ta about individuals, addresses, or vehicles to be more prominently displayed when a system user runs a search. Nothing in available data sets
21 This issue was raised by Scott (2006, p. 130) in his examination of an information-sharing system in Fl


rovide very useful data; for example, in Scott (2016, p. 123), this sort of success tagging was used to try to measure the “organizational distance” between the agency that was the source of data and the agency reporting a success based on its use. S


rview report that contributed to an investigation or the fact that an arrest or booking record is tied to a crime case that already existed in a local agency system. Differences in the way that agencies’ individual systems are designed can significan


useful enough for evaluation to merit saving even if there is not a strong operational reason to maintain the data. In this effort, this situation arose in our attempt to measure the effects of ONASAS 38    Knowing More, but Accomplishing What? Measu


’ data. Even differences in simpler procedures (e.g., departmental conventions for assigning case numbers and whether approaches were the same across participants in the system) could make matching cases, suspects, or arrests across different jurisdi


evel
 ana
lyses like this one but would also make it possible to explore more-detailed ques -
tions. For example, measuring which types of data contribute most to solving cases could make it possible to target investments to provide the highest crim


effect of jurisdictional boundaries on agency activities and effectiveness. Data-sharing via BOLO notifications was associ -
ated with increases in the number of crime cases connected to suspects—an important outcome given the role of serial offender


and how they might affect evaluation design. Initial interviewees were nominated by organizations that use ARJIS systems and included individuals from different criminal justice roles. Interviewees were asked for suggestions of others whose perspecti


 important these systems are to what you do, we want to
get an estimate of how much you use information from criminal justice IT sys -
tems (whether your agencies’ or interagency information-sharing via ARJIS) in the course of doing your work.
Ther


tant/
useful for your Top 3 tasks? How was it used? Is data from ARJIS more fre-quently a source of new leads/information or a way to confirm information from other sources—or a mix of both?
 – Ind
ividual criminal history
 – Inc
ident descriptive i


s didn’t exist?
 ◦In th
at “world without ARJIS,” how do you think the outcome of the situa -
tion would have differed?
i. Wou
ld the chance of resolving it been different?
ii. Wou
ld the speed of resolving it been different?49Bibliography
Agrawal,


the Effects of Information-Sharing
Danziger, James N., and Kenneth L. Kraemer, “Computerized Data-Based Systems and Productivity
Among Professional Workers: The Case of Detectives,” Public Administration Review , Vol.  45
, No.  1,
J
anuary–Februar


 Center for Best Practices, July 2009. As of
August 11, 2017:  https://www.nga.org/files/live/sites/NGA/files/pdf/0907JUSTICEINFOSHARING.PDF
Information Sharing Environment, “The Role of PM-ISE,” web page, undated. As of August 11,
2017:  https://w


ion of Serial Offenders’ Crime Locations on Their Probability of Arrest,” European Journal of Criminology , Vol.
 10
, No.  2, 2
013, pp.  16
8–186.
Lammers, Marre, Wim Bernasco, and Henk Elffers, “How Long Do Offenders Escape Arrest? Using DNA Trace


al Mobility in Criminal Histories , presentation slides, U.S.
Department of State, Bureau of Justice Statistics, October 30, 2012. As of August 11, 2017:
http://www.crj.org/page/-/cjifiles/ICRN2_Cross-JurisdictionalMobilityinCriminalHistories.pdf


ut Needs to More Accurately Account for Federal Funding Provided to Centers , Washington, D.C., GAO-15-155,
November 2014.
Wellford, Charles, and James Cronin, An Analysis of Variables Affecting the Clearance of Homicides: A
Multistate Study , Wash


CORPORATION
Reducing Coastal
Flood Risk with a Lake Pontchartrain Barrier
Jordan R. Fischbach, David R. Johnson, Edmundo Molina-PerezLimited Print and Electronic Distribution Rights
This document and trademark(s) contained herein are protected by la


ding Lake Pontchartrain. It is also intended for similar audiences in Mississippi whom a proposed barrier might affect.
The research described in this report directly applies an updated version of the CLARA
model developed as part of CPRA’s 2017 Mod


e
Heights for Highway 90 with Its Existing Crown and Low Gates, 100-Year Flood
Interval  ............................................................................................. 23
 3.6. Expec
ted Annual Damage, by Alignment and Louisiana Regi


3.6. Incre
ase in Damage, by Annual Exceedance Probability Interval for Coastal
Mississippi  .......................................................................................... 33ixSummary
Louisiana’s coastal communities face a range of chal


th median expected annual damage reduc -
tion benefits ranging from $1.2 billion to $1.4 billion per year (Table S.2). A barrier could also
lower future design height requirements for the Greater New Orleans hurricane protection system along the sou


 implementation as part of the final 2017 Coastal Master Plan.xiiiAcknowledgments
This document describes a preliminary investigation that was designed to guide analysis in
the 2017 Coastal Master Plan. We gratefully acknowledge the support and feed


from
extreme coastal storms. Tropical storms and hurricanes making landfall nearby can push storm surge and high waves into populated areas, damaging or destroying assets, disrupting economic activity and basic services, and threatening the health a


Barrier
populated areas. In particular, given the proximity to the border with Mississippi, a primary
concern was that a new barrier could induce additional flooding and flood damage to Mis -
sissippi coastal communities, such as Bay St. Louis, Gulf


, 2017 [hereafter FJK, 2017]). We developed results to guide CPRA’s selection of one or more barrier alignments to consider in the 2017 Coastal Master Plan analysis. As a result, we describe only damage reduction benefits and potential induced damage


the Louisiana and Mississippi coastlines. Along the coast, CLARA labels different areas as unenclosed (those with no levees, floodwalls, or other barriers or with structures that do not fully enclose the population at risk) or enclosed  (those with


nd
value of the assets at risk from flooding and calculates damage in dollars as a function of
3 AEPs are statistical estimates of the flooding and damage expected to recur with a certain probability in each year. For
example, the 1-percent or “10


n, and Jackson counties in Mississippi. During the model improvement process, we added grid points for these counties, and Figure 1.1 in Chapter One shows the combined Louisiana and Mississippi domain considered. We identified no enclosed areas for c


 with the full 232-storm set as a baseline. We conducted two iterations of the analy -
sis and shared and discussed results with CPRA and Arcadis through in-person and phone meetings. Conclusions from the analysis can be summarized as follows:
5
• Th


 of assets at risk to include coastal Mississippi, drawing on the
most-recent data sources available.
• The CLAR
A 2.0 geospatial grid requires a higher level of geospatial fidelity than the pre-
vious analysis, motivating a move away from data sour


y includes the length of road (in road-miles) that falls within the vicinity of each grid point.
Agriculture
For this study, we did not include crops grown in the coastal region in the updated asset inven -
tory for Mississippi. The CLARA team analyz


tion Type Occupation Type Code Damage Category Code Equivalent GBS Code
One story 1STY Residential RES1
Two story 2STY Residential RES1
Mobile home MOBHOME Mobile home RES4
Multiunit MULTI-UNIT Multiunit RES2
Commercial COM Commercial COM
Industrial


illion in value and appear as 0.0 in the table because of rounding.Methods and Data Sources    13
intervening time span. In addition, this simplifying assumption allows for easier interpretation
of analysis results when looking at a single scenario


certainty developed for 2017 and documented in Attachment C3-25 to the 2017 Coastal Master Plan (FJK, 2017). In this instance, parametric uncertainty  refers to uncertainty
or noise associated with flood depth and damage assessment that emerges from


xisting road grade to a large 24.5-ft. barrier with gates. The lower alignments would be designed to reduce the volume of storm surge entering the lake but not necessarily to prevent surge and wave overtopping (i.e., they would deliberately allow som


ps Showing Two Proposed Alignment Configurations
SOURCE: Arcadis.
NOTE: Locations are approximate, and the scales of the maps are not equal.
RAND RR1988-2.2Hwy90/10/10 CSX/10/10
Road crest set to a constant elevationof 10 ft.Tie into the existingSlid


utside HSDRRS East, including Slidell and Plaquemines, also show extensive 100-year flood depths in the FWOA.18    Reducing Coastal Flood Risk with a Lake Pontchartrain Barrier
Figure 3.1
Future Without Action Flood Depths for the Study Region, 100-Y


ge, by Region and Asset Class, in Billions of Constant 2015 Dollars
2.1
0.5
0.10.5
1.6
NOTE: The stacked bars show the 50th percentiles (medians), and the lines show the 10th- to
90th-percentile ranges.
RAND RR1988-3.3RegionSt. Tammany
Orleans and J


erms of both EAD and damage by AEP interval. The subsequent section then details potential negative effects on Mississippi coastal parishes.
Note that this section summarizes results for four of the five barrier alignments consid -
ered. We do not de


are mixed in parishes that are partially inside and partially outside of the barrier, however. For example, average 100-year depth reduction in portions of St. Tammany Parish without an enclosed protection system (i.e., nearly the whole parish except


t because of the assumption about future levee lifts and
maintenance previously described. Similarly, we observe no increases for enclosed areas of Plaquemines Parish (East Bank). This is because enclosed areas of Plaquemines already overtop and flo


t different geographic locations in each region (polder).Average change in surge and wave elevation (ft)
–5 3Results from Proposed Barrier Alignments    25
icant wave height (right pane) for points surrounding the protection system with Hwy90/0/2
in


elds $1.8 billion in remaining damage ($1.2 billion to $2.5 billion). Despite a much taller barrier, Hwy90/24.5/24.5 shows remaining EAD similar to that of Hwy90/0/2. This is caused by induced damage in some areas, particu -
larly outlying portions o


ent to finalizing the analysis in this report, but we did not apply it here.Figure 3.6Expected Annual Damage, by Alignment and Louisiana Region, in Billions of Constant 2015 Dollars
0.1
0.1
0.1 0.10.10.5
0.2
0.1 0.10.10.5
0.20.2 0.20.2
2.1
1.5 1.4 1.


uisiana study region, we see that EAD reduction
at the median across the Louisiana study area ranges from $1.2 billion to $1.4 billion from the different alignments, with Hwy90/10/10 ranking highest in terms of net EAD reduction. Interestingly, Hwy9


); red shading indicates induced
damage (disbene
fit). Fragility scenario = MTTG.
RAND RR1988-T3.3Change in EAD (millions of constant 2015 $)
–50 700Results from Proposed Barrier Alignments    29
Table 3.4
Damage Reduction or Inducement, by Annual E


ven at the 500-year interval.
Potential Induced Flood Damage in Coastal Mississippi
One of the primary concerns about a new Lake Pontchartrain barrier is the potential that
a new alignment, by preventing storm surge from entering the lake, would ins


t results as Figure  3.8 but also includes percentage increase and is broken out further
by type of asset. According to Table 3.5, the percentage increase in EAD from a barrier in Hancock County ranges from 3.2 percent (Hwy90/0/2) to 9.5 percent (Hw


Table 3.6
Increase in Damage, by Annual Exceedance Probability Interval for Coastal Mississippi
Return
Interval, in Years CountyAlignment
Hwy90/0/2 Hwy90/10/10 CSX/10/10 Hwy90/24.5/24.5
Total Delta Percentage Total Delta Percentage Total Delta Perce


inducement with a particular barrier configuration in place. Figure 3.9, for instance, shows a map of median 100-year damage in the FWOA (top pane) and change in median 100-year damage with Hwy90/0/2 in place (bottom pane). The color scales are logar


ences in performance between Hwy90/10/10 and CSX/10/10, however, and these differences occur largely in the immediate vicinity of the barrier. The dis -
tinction among the project benefits is reduced further when taking parametric uncertainty into ac


ertain future scenario reflecting plausible sea level rise, land subsidence conditions, and asset growth 50 years from today. It also does not formally consider other decision metrics, such as project cost and environmental impact. However, the analy


chbach,
Kevin Hanegan, Kristy Lewis, David Lindquist, J. Alex McCorquodale, Michael Poff, Hugh Roberts, Jenni Schindler, Jenneke M. Visser, Zhanxian Wang, Yushi Wang, and Eric White, 2017 Coastal Master Plan ,
Appendix C, Chapter 4: Model Outcomes


Information System, “County Data Download,” updated April 2016.
Referenced August 15, 2014. As of August 15, 2014: http://www.maris.state.ms.us/HTM/DownloadData/County.html
Peyronnin, Natalie, Mandy Green, Carol Parsons Richards, Alaina Owens, Denis


 CORPORATION
Helping Law Enforcement
Use Data from  Mobile Applications
A Guide to the Prototype Mobile Information
and Knowledge Ecosystem (MIKE) Tool
Edward Balkovich, Don Prosnitz, Steven C. Isley, Anne Boustead, Bonnie TriezenbergLimited Print


rototype
that we developed. It is intended to serve as users’ manual for MIKE, by explaining its prin -
ciples, illustrating its potential uses, and describing how to add content. Although this tool is not widely available, our goal in writing this


the Mobile Information and Knowledge Ecosystem (MIKE). This prototype is not widely distributed. The companion volume, Electronic Surveillance of Mobile Devices: Understanding the Mobile Ecosys -
tem and Applicable Surveillance Law (Balkovich et al.,


nt, operation, and/or use. Chapter Two describes how MIKE can be used and is intended for those who adopt and utilize MIKE. Chapter Three describes how MIKE can be edited and maintained and is intended for those who are responsible for curat -
ing MI


ights and civil liberties. Neither is good. Law enforcement needs to be as well informed as possible when making decisions about the use of electronic surveillance.
2 While some argue
that the mobile ecosystem has created a “golden age” of surveilla


the mobile ecosystem, we
developed a tool—the Mobile Information and Knowledge Ecosystem (MIKE)—that system -
atically relates the commercial entities that collect data about mobile device users to the types of data they collect, the rules governing


entation and use. We explored MIKE’s utility by demonstrating its use to stake-holders and asking for their feedback (see Appendix A). Preliminary assessments from repre-sentatives of these three stakeholder communities—law enforcement, commercial fi


 law. While this volume was not written to provide an introduction to any of the above, the glossary provides specific definitions for terminology used in this volume and may be useful for those less familiar with these topics. Note, however,
6 Thes


nd the utility of MIKE. Chap -
ter Four is a reference guide for MIKE and provides an architectural overview.
This chapter describes MIKE and illustrates ways to use it. MIKE is intended to be an
interactive tool, and it is therefore difficult to fu


arch for particular types of information. MIKE’s homepage also provides access to help functions.
Figure 2.1
MIKE’s Homepage
RAND RR1482-2.1
Search Bar
Help
Quick Links
Query
FunctionUser’s Guide to MIKE    7
MIKE contains pages pertaining to corpor


n on the phone is diff  erent from that used on a server. It would probably
be useful for an implementation of MIKE to incorporate such details.Figure 2.3
MIKE’s Description of a Mobile App
RAND RR1482-2.3
Technical
Requirements
Funding
ModelEstim


legal issues she must address to access the
data. Th  e example is highly stylized to illustrate many of MIKE’s features and does not pre-sume that this is the only way the investigation might proceed or that the indicated pathways are the only (or


hat Gmail does in fact maintain MAC addresses and device IDs, along with log information about Gmail accounts (the “Data Contexts” table in the lower center of Figure 2.7).
6
We entered the types of information apps gather ( data contexts ) into MIKE


ed. Th  is is typically done to manage licenses and enable the rein-
stallation of a deleted app. Th  ese app-tracking apps are only aware of the apps downloaded from the store associated with the app, e.g., Google or Apple. Figure 2.6
Initial Query


m’s device from Google.
Clicking on 2703(d) reminds the investigator about what is needed to obtain such an
order (Figure 2.13). In case of a 2703(d) order, “specifi  c and articulable facts . . . reasonable grounds .  .  . are relevant and material


elping Law Enforcement Use Data from Mobile Applications
• Google Calendar
• Foursquare
• SquareCash.
 Th  e investigator now has an email account associated with “a” phone, but needs to
confi  rm that it is in fact the victim’s phone. Matching a pi


then off  ers a drop-down listing several choices. She choses the Global Positioning System (GPS), which is likely to be more precise and accurate than Cell Site Location, and runs the query. Th  is produces the list in Figure 2.17.
Th  e GPS locatio