 or by noticing sus- picious activity, for example). On the other hand, some studies have shown that those with higher levels of education often believe that they are less vulnerable (Wash and Rader, 2015).
13 See, for example, Google Trends search a


health history, cannot be changed, and chang- ing a Social Security number, or personal information, such as a physi- cal home address and mother’s maiden name, can be slow, complex, and comes at a great cost. From an attacker’s perspective, these la


15) found that less than 10 percent of victims purchase identity-theft protection (4 per- cent) or purchased identity-theft insurance or used a credit monitoring service (6 percent).20 One possible explanation for the low percentages, especially comp


es of Assistance (n = 998)?
    Overall
If credit card information was taken
If health information was taken
If information other than credit card or health information was taken
Yes No
60% 30% 61% 31% 80% 12% 58% 30%
Not Sure
10% 8% 8% 12%
  Figure


the company any- more.”
• did not seem necessary: “I checked my account and nothing was missing”; “I wasn’t interested.”
• seemed like a hassle or too much work: “Laziness”; “Basic Apathy”; “Hassle.”
• no time, or forgetfulness: “I did not have time


itudes Toward Data Breach Notifications
process of enrollment could be made easier, and clear benefits could be better communicated.
Consumer Attrition
A critical corporate and policy issue concerns consumer attrition in the event of a data breach: A


 a Ponemon study that found that 20 percent of information technology security practitioners reported that they were unable to deter- mine whether a breach was ever resolved (Ponemon Institute, 2015).
That the overwhelming majority of consumers (89 p


nsumer Financial Protection Bureau, 2013). Further, the number of consumers who file police reports or take legal action when they become fraud victims is decreasing, although many consumers claim to install antispyware or firewalls on their computer


ngly, those in the 18–34 age category were significantly less likely to be extreme responders than older age groups. Those with graduate degrees were significantly less likely to be extreme responders than those with less than a high school diploma.


umer Satisfaction with Company Response to the Breach
Companies can engage in a variety of post-breach activities in an attempt to assuage customers, employees, potential plaintiffs, federal and state regulators, and, in certain cases, Congress. Howe


ld take were donating money to organizations that promote cyber- security and simply apologizing to those affected.
When offered the opportunity to elaborate on their responses, respondents commonly described the following:
• provide advice on what I


U.S. Census Bureau, undated).
 39
40 Consumer Attitudes Toward Data Breach Notifications
ongoing law enforcement investigations. Thus, the optimal time for consumer notification of a data breach remains unclear.2
Sixty-two percent of respondents acce


 actual knowledge.
When examining customer reactions and sentiments following a breach, we found that consumer attrition is only 11 percent, and con- sumers seem to appreciate companies that respond and appear to take responsibility for breaches they


0 customers (Levin, 2015). In May 2015, Representatives Randy Neugebauer and John Carney introduced a bill, the Data Security Act of 2015, that would help standardize breach notifications from financial institutions and retailers. At the same time, m


., suspicious behavior on credit card, locked out of accounts, etc.)
(Opt2) Yes—I was notified by a 3rd party company (e.g., my bank or credit card company told me)
(Opt3) Yes—I heard media reports about the data loss (Opt4) No—I first learned of thi


l, and 5 represents an actions that would greatly improve your satisfaction, please rate the following actions in terms of how much they would affect your satisfaction with a company’s response following a loss/theft of your information.
Apologize to


 Method of Notification
 Category
Postgraduate Race/ethnicity
From the Company Directly
53%
Identified Suspicious Activity
12%
Notified by Third Party/Bank
14%
Heard Media Reports
36%1,2
30%3 33% 19%1 29%
30%3,4 31%3,4 16%+
3%+
 Non-Hispanic 56%4 10


ast
Midwest South West
Education
High school or less Some college Bachelor’s Postgraduate
Race/ethnicity
Non-Hispanic white
Non-Hispanic black
Hispanic/Latino 326 16% Non-Hispanic other 112 6%
NOTE: Percentage totals might not sum to 100 due to round


w Forms Improve Consumer Understanding, Aid Comparison Shopping, and Help Prevent Surprises,” fact sheet, November 20, 2013. As of March 6, 2015: http://files.consumerfinance.gov/f/201311_cfpb_factsheet_kbyo_testing.pdf
Garrison, L., M. Hastak, J. M.


gton, D.C.: Pew Research Center, May 20, 2015. As of February 8, 2016:
h t t p : / / w w w . p e w i n t e r n e t . o r g / 2 0 15 / 0 5 / 2 0 / americans-attitudes-about-privacy-security-and-surveillance/
Microsoft Corporate Blogs, “#MSFTCOSO POV:


 . l e g i n f o . c a . g o v / p u b / 15 - 1 6 / b i l l / s e n /
s b _ 0 5 5 1 - 0 6 0 0 / s b _ 5 7 0 _ c f a _ 2 0 15 0 5 2 2 _ 15 1 6 1 2 _ s e n _ fl o o r . h t m l
Shey, Heidi, “Market Overview: Customer Data Breach Notification and Respon


ers of free credit monitoring; (4) only 11 percent of respondents stopped dealing with the affected company following a breach;
(5) 32 percent of respondents reported no costs of the breach and any inconvenience it garnered, while, among those report


 Fostering
Innovation
in the U.S.
Court System
Identifying High-Priority Technology and Other Needs for Improving Court Operations and Outcomes
Brian A. Jackson, Duren Banks, John S. Hollywood, Dulani Woods, Amanda Royal, Patrick W. Woodson, Nicole J


one product of that effort, completed as a joint effort of the RAND Corpo- ration and RTI International. It presents the results of the Courts Advisory Panel, a group con- vened in fiscal year 2015 as part of the NLECTC Priority Criminal Justice Need


bout this report should be sent to the project leader, Brian A. Jackson at Brian_Jackson@rand.org. For more information about the Justice Policy Program, see www.rand.org/jie/justice-policy or contact the director at justice@rand.org. For more infor-


 . . . . . . . ...
.
. 48 . .58 . 66 . 68 . .78 . 94
Summary
 Society relies on the judicial system to play numerous roles. It is the link between law enforce- ment and the corrections system and serves as a check on their power over citizens. Beyond


rnment broadly defined should do (such as address funding constraints that limit the ability to innovate), and things the public needs to do (such as advocate for changes in law to address current roadblocks to better court performance).
ix
x Fosteri


 or low-hanging fruit were identified and pulled out.
The court innovation agenda is dominated by information and communications needs (orange icons, top left), which made up the majority not just of the high-priority needs (falling on and within the


es of communication, and social media
Associated Need
High High Priority Value
Low- Hanging Fruit
 Adopt commercial alerting tools, u which are available but not widely
used.
Design systems with backup capabilities and prioritize technology support t


equipped to handle,
reflecting the different needs of
Make the investments needed to allow connectivity, and explore new technologies that make it easier to install wireless Internet in older court buildings.
different types of courtrooms.a
Develop t


t a data standard that does not conform to their processes.
Train clerks who are entering data u to provide enough detail and
granularity to facilitate judges’ tasks
and activities, including descriptive
file names and semantic context information to


he outset.a
Make broader use of standards
for information-sharing to allow compatibility (criminal justice coordinating councils are a potential model to drive change).
Develop analysis tools or entities responsible for assessing the implications of


public.
Pursue statutory authority or court procedural rule authority for specialists to appear via video presence to increase efficiency of staff usage.a
Create governance structures that limit the level of autonomy that elected judges can have; tha


level category. Full categorization of needs is included in Appendix E. a This need is associated with two categories and is included twice in the table.
From the needs that were rated highly enough for inclusion in our U.S. courts innovation agenda,


mats for digital data used in courts to avoid incompatibility problems. Addi- tional needs that were included in the agenda because they were rated as high value or low-hanging fruit also fell in this theme, including the need to adopt data standards


nologies, rather than always requiring technology providers to adapt tools to the historical ways that courts have functioned.
• Using technology for notification and public communication. Beyond the need for public notification in the context of eme


ng High-Priority Technology and Other Needs
able. Judges and others need better tools to quickly parse and understand larger and more- complicated bodies of data related to cases, and to integrate data from multiple criminal justice systems to inform


 larger system might have more resources available to acquire new technology, but imple- mentation might be more tractable in a smaller system that has less of an investment in legacy systems or fewer staff to train.
It is also important to frankly a


e in identifying and inviting members of the Courts Advisory Panel. Two anonymous peer reviewers from the National Institute of Jus- tice and Geoffrey McGovern of RAND provided comments on the draft manuscript.
We would also like to acknowledge the c


In this effort, we view innovation broadly, including both incremen- tal changes, where agencies improve on current practices, and transformational change, which makes it possible for agencies to pursue their objectives in new ways or make more-radic


utures in detail, simulations that seek to analyze large numbers of possible futures, expert elicitation methods that attempt to leverage the knowl- edge and intuition of many individuals to build collaborative group estimates, and structured analyti


he results of that questionnaire as a foundation, attend a one-day, in-person workshop at the RAND Corporation’s Washington, D.C.–area office. The workshop featured a structured elicitation process to explore both current and potential near- term pro


sory panel members and process, additional methodological detail, and the full list of needs from the panel.
CHAPTER TWO
The State of the U.S. Court System Today
The court system is one of three main components of the U.S. criminal justice system, wh


ch report directly up to the superior court, while others report directly to the court of appeals (see Figure 2.1). In an effort to facilitate more-uniform procedures within states, many state court systems are “unified,” allowing a single entity to


isdiction courts rarely conduct trials. Data from state court records, prosecutor’s offices, and the federal judiciary indicate that less than 5 percent of all felony charges are ultimately disposed through a trial (Rosenmerkel, Durose, and Farole, 2


Justice Statistics, justice expenditures in 2012 totaled more than
Figure 2.2
Typical Resolution of Felony Defendants Arraigned in State Courts in the 75 Largest U.S. Counties
The State of the U.S. Court System Today 7
  Dismissed (25%)
 Diversion or


nd people management. The goal was to get a snapshot of issues that were on the minds of practitioners from a range of courts and roles within the court system, and to capture challenges faced in areas as disparate as technology use and human resourc


relieving pressure on court systems that have been dealing with high case burdens for many years, this raises potential questions about equitable access to justice for all populations. The use of alternative dispute resolution mechanisms (e.g., arbit


stem (Bureau of Justice Statistics, 2012). Staffing con- straints and the resulting increase in caseloads has been particularly problematic in individ- ual components of the judicial system, including indigent defense organizations (see Lefstein, 201


em-solving courts that focus on drugs, domestic violence, mental health, veterans, re- entry, teens, prostitution, and the homeless. Other states, such as Wyoming, Mississippi, New Jersey, and North Dakota, have only a drug court (Strickland, Schauff


 and Economic Disparities in Case Processing and Outcomes
There is increasing concern in the United States about the fairness of the justice system, espe- cially about the effect that bias—whether knowing or not—in the enforcement of laws, pros- ecut


The inability to post bail can make it more difficult for individuals to fulfill work, family, and other obligations, which can magnify consequences of incarceration. As described earlier, 97 percent of federal criminal prosecutions are resolved by


vide access to files and resources to assist them.
Courts and Changing Technology
Technology can play a role in helping courts perform their functions, but shifts in technology in both society and the criminal justice system can also create challenge


technological availability, (2) could threaten the trial process when jurors and others use those technologies to do their own research outside of what is presented in testimony, and (3) when those in the courtroom take photos or video of proceeding


by the absence of resources to upgrade or replace them (58 percent of respondents).
The issues posed by new technology can be particularly challenging when considering the unique court environment and its barriers to adopting and using new technologi


s distinct needs and requirements. The functioning of courts involves judges, court administrators, clerks, attorneys, and case litigants. Each group has its own requirements for access to information, data that must be collected and managed, and act


pellate review of trial proceedings (Lederer, 2000).
As cited previously in our panel results, challenges faced by court systems in building and maintaining internal human capital can also complicate innovation efforts. Societal shifts toward Interne


e of U.S. courts and the potential for innovations to improve performance, the path to realizing the potential of technology and meeting the challenges faced by the U.S. court system starts with the technology and practice environment as it exists to


omy in this work for application to courts.
There are five central categories within the taxonomy:
• facility operations and population services
• person-worn equipment and weapons/force
• information and communications
• doctrine, tactics, managemen


pment and training category captures knowledge and resources for
2 New branches were needed to capture primary record-keeping methods and tools (which would include court reporting technologies) and public information provision and training for crimi


t records and other documents
Biometrics and other sensors
(see also, information collection) attorneyin-fdraestfruectnurde ant
Information presentation tools and dashboards
Voice
• •
•
Judicial bench information delivery and interfaces
Tablet and mo


ologtichainl g monitoring)
Specialized task technologies
Restraint technologies
Education or information delivery
Delivering services to population
Health care delivery
See information collection for biometric and other sensors for information system


ncial management systems
• Court human resource management systems
Tactics and practices
Technology-mediated training tools
Computational tools
Electronic document management systems Digital evidence management systems Case management systems
Fine an


d in court facilities, but also the technologies that prosecutors, private attorneys, other criminal justice practitioners, and even members of a jury might have and use during court activities.
Nevertheless, some survey data are available. The Ameri


t from Illinois. As a result, while the data appear to be the best available on tech- nology use in courts across the country, they must be interpreted with caution.
5 It should be noted that there is a literature describing court practice, technolog


he mobile devices of the registered parties.
Prior to the implementation of [the program], domestic violence victims had to travel to multiple locations to obtain a protective order. . . . Having to travel publicly to multiple locations, usually foll


sing from 10 percent to 37 percent) (Poje, 2014; see also discussion in Heerboth, 2013). As discussed in Chapter Two, technology advances within society as a whole are increasing expectations for information technology capability in courts—and that c


 and Schneider, 2007), but little data are available on what courts are actually doing.8
Information Collection
In contrast to other components of the criminal justice system—where collection of new information is central to their roles—courts are ge


 then, in the text for each topic, relate the functional areas to the categories in our comprehensive taxonomy.
10 Security-related infrastructure and facility technologies and practice are discussed later in this chapter.
 Based on NCSC’s review,11


rts and court participants—often central in civil proceedings—is the acquisition and analysis of electronic information in case preparation, termed e-discovery.14 It is defined as
11 NCSC identified a sample of 77 courts of different sizes, geographi


e-discovery productions found that 73 percent of e-discovery costs were spent on document review. Computer-categorized document review techniques, such as predictive coding, identify at least as many documents of interest as tra- ditional eyes-on re


it was available, although a much smaller percentage (31 percent) indicated that it was always available in all courtrooms (Wiggins, 2004). More- sophisticated versions of such software can make transcript-in-progress tools available to court partici


paper-based records (Delehandy et al., 2014; Lederer, 2000). Comprehensive recent data on the extent of adoption of digital recording are not available, but in a 2002 survey of federal courtrooms, 18 percent reported being equipped with digital audio


ining data from an individual’s criminal history with other factors—such as antisocial personality patterns, procriminal attitudes, substance abuse, procrime social supports, poor relationships, school/work failure, and lack of prosocial recreational


such management can support judges in making well-informed rulings quickly. It can also allow courts, criminal justice agencies, and community agencies to collaborate, improve services, and solve community problems. In our taxonomy of criminal justic


e
• accounting (including front counter, cashier, back office, and general ledger functions)
• security
• management and statistical reports. (NCSC, 2001a, pp. 2–3)
The link to accounting functions can also enable caseflow management systems to per-


ions could be ongoing, given the continued use of paper or physical submission of some materials or evidence. The extent of adoption of these technologies is not clear, but in the survey of Ohio courts, 51 percent of the courts had some type of imagi


rovide efficient check-in processes in order to streamline the juror system. These tools also allow for last-minute recorded information pertaining to trials to be delivered to the prospective jurors (NCSC, undated d). According to a 2014 survey, 63


on electronically, 13 reported that all information exchanges were electronic (i.e., no information was exchanged via paper). Twenty-four responding courts solely or predominantly used real-time information exchange, 13 solely or predominantly used E


ed sets of people to trans- mit information. Jury management systems (discussed previously) can be linked to notifica- tion systems to remind jurors of their summons and help ensure juror attendance. Some of these systems make it possible for jurors


 that guides self-represented litigants through a series of questions (Sandman and Rawdon, 2014). Based on the answers provided, the program will prepare personalized court forms that are ready to be served and filed.
23 For example, Clarke, 2015.
24


a challenge and an oppor- tunity for court systems. When many independent actors bring their own communications devices to the courtroom, it creates infrastructure demands. At the same time, the proliferation of mobile devices has given courts an opp


ransport defendants or prisoners from jails, which also decreases risk and saves money. However, doing so has raised concerns about whether telepresence is compatible with fairness of the judicial process and the right of the accused to face their ac


Court System: Identifying High-Priority Technology and Other Needs
case delays to use an interpreter who is physically present, video remote language interpreta- tion is a solid option and “may be the best alternative for many rarer languages” (Clark


tation. Data from the ABA survey of attorneys suggest that these cameras are available in many courts but are not widely used (from 2011 to 2014, between 17 and 22 percent of respondents indicated that they are available [Poje, 2014].) “Enhanced whit


tween 12 and 27 percent indicated that there were capabilities for them to connect their own devices to court presentation systems (from projection screens, which were available in more than half of courtrooms, to individual monitors for trial partic


 and the knowledge associated with criminal justice functions. And the route for implementing that doctrine or those tactics in the court system is the training that provides the necessary knowledge for court participants.
In the literature and in fe


it was not neces- sary and between one-fourth and one-third indicated that the courtrooms they practiced in did not utilize technology or that courtroom technology was not relevant to their practice, reasons cited among the remainder of respondents


standards for criminal justice information systems already exist to meet this need (see IJIS Institute, undated; NCSC, 2001b). For instance, the Global Information Sharing Initiative created a standard for data exchange models called the Global Justi


r physical infrastructure used for such applications— including designs that provide single points of access to enable screening, lighting and other technologies at the perimeter of buildings (e.g., applying crime prevention through environ- mental d


accounting for paper files.
Person-Worn Equipment and Weapons/Force
In our taxonomy, the person-worn equipment category covers the equipment and tools (from general use equipment to specialized technology) that criminal justice practitioners use in c


viduals or offenders from place to place, we have covered those roles and functions in our analysis of the corrections sector and therefore omitted them from this work in the interest of clarity.
Court Technology and Practice Today 43
 CHAPTER FOUR
F


nvolving language interpretation, jury and jury selection, long-distance court proceedings, public defense systems, and the training and education of judges and court staff.
• Such organizations as ABA, the National Association for Court Management,


address how evidence can come to courts in many different electronic file formats) and (2) steps that could allow courts to take advantage of new opportunities that improvements in technology have made possible. In our work, we therefore define a co


preparation and presentation, conducted primarily by attorneys or litigants to pre- pare for a case and to present evidence pertaining to that case at court hearings and trial. This area also includes judicial functions to rule on motions or evidence


fy potential innovations that might relate to one or more areas of concern or opportunity. This avoided focusing on a single ideal future for the court system. For example, some of the problems that are recognized today are, at their heart, resource


es efficiently and fairly.
Improve the competencies of staff through training, education, and readiness. Ensure an independent, impartial, and accountable judiciary.
    From Courts Today to Courts Tomorrow: Identifying and Prioritizing Innovation Ne


 to vary, each panelist assigned a rating for each need on a scale of 1 to 9, where 1 meant the need would not be valuable and 9 meant that meeting it could result in a 20-percent or greater improvement in performance or efficiency.6
3 One issue high


ds identified are practical for existing practice and to identify needs that analysts who are not as intimately familiar with the sector might not. The central challenge in these efforts is that the topics discussed—and, therefore, the needs generate


approach only implicitly includes considerations of cost. The probability of adoption of a solution or technology (our operational success measure) implicitly includes cost, because more-expensive solutions are less likely to be implemented broadly.


 of the need was too high, too low, or correct as is. This final round of prioritiza-
7 Our approach to identifying and presenting the needs in tiers also had the potential to minimize the effect of this truncation.
8 The process of consolidation and


technology information to judges and how to recruit and keep information technology staff in court organizations. Needs in this area ranged from the very broad (e.g., the need for training and capability-building for court staff to address the fact


se associated with specialty courts needing to share data with a wider range of orga- nizations and service providers. Needs in this area also included ways that information- sharing could contribute to resolving important problems in courts today. F


er tools for evaluating system security.
• Notification and communication. Because court proceedings rely on all the required par- ticipants appearing at the same time, when litigants or defendants do not show up, it cre- ates schedule problems and c


ely under doctrine, tactics, management, and behavioral knowl- edge development and training, with the majority of those needs falling into our management/
From Courts Today to Courts Tomorrow: Identifying and Prioritizing Innovation Needs 55
Figure


formation provided by the Courts Advisory Panel members (on how valuable a need would be to the court system if met, how technically straightforward it would be to achieve,
Figure 4.2
Percentage of Needs Contributing to Each Court Objective
      Per


rs in one of the needs that bridges categories. Table 4.2 presents the top-tier needs by top-level taxonomy category. The dominance of infor- mation and communication needs in the top-priority list is very different from the category breakdown of the


tems when they go down. Develop, exercise, and implement response plans to address technology failure.a, b
Ensure that electronic and other court data have robust backups and that courts have sufficient control over the data storage to permit this.c


dges, lawyers, and others)
Minimal or nonexistent wireless Internet and bandwidth in many court buildings
Poor access to complete information to inform bail decisions
Poor access to complete information to inform bail decisions
Due process concerns a


to support sentencing. The availability and quality of information available to judges for making bail decisions has also been an enduring concern, with discussion in the 1988 Office of Technology Assessment report examining technology’s potential fo


to aid in locating information later.
Explore whether features of technology systems provide opportunities to better meet the timeliness goals of the justice system (versus just focusing on existing technology and what
it can do).
Pursue statutory au


ween public access and maintaining sufficient court security
Backlogs in forensic laboratories and the slow processing of evidence delaying justice
Vendor systems that try to simultaneously meet the needs of multiple stakeholders (e.g., judges, couns


h other needs that were not ranked as high priority) highlights a key technology development challenge for courts and a situation where stakeholder concerns and best practices can sometimes pull in multiple directions. The involvement of stakeholders


tem-level tools to assess the cascading effects of changes in criminal justice policy and practice.
During discussion and the prioritization process, panel members commented on indi- vidual needs, providing additional context and nuance based on thei


rmation- sharing
 IT systems for managing mission-related data
 External communications
 Fixed location communications (video)
 Information management (including sharing)
 Information delivery (including communications)
 Information technology— basic


ted to serious barri- ers: “Confrontation Clause objections can prevent this for now”; and “Easy to do technologi- cally, harder to do if there are legal objections.”12 Focusing specifically on remote appearances of crime lab staff, one participant e


e main prioritization discussed above includes consid- eration of value (and the number of court objectives where that value could be realized), likeli- hood of success, and likelihood of broad adoption of the innovation. Although this calculation pr


both sides in a dispute that educates pro se litigants. These needs actually scored quite highly on all three measures in our calculations, but because they were relevant to only one or two of the court objectives, their overall expected values were


nd for all sides of disputes. Such calculations would also have implications for time standards for court operations (Steelman, 2010). See also NCSC, undated a.
c Such role-based and other access- or disclosure-control issues are general issues acros


ations and maintenance costs are captured in acquisition decisions and included in out-year budgets.g
  NOTE: Needs are grouped by their top-level taxonomy category. Full categorization of needs is included in Appendix E. Needs that were already incl


ore whether features of technology systems provide opportunities to better meet the timeliness goals of the justice system (versus just focusing on existing technology and what it can do).
Design systems that are capable of capturing unstructured but


s. Communication between courts and other criminal justice agencies and community support groups is essential to increasing the effectiveness and efficiency of the court system. Having updated, well-developed standards that are widely implemented wou


s relatively evenly split between the information and communications category and the doctrine, knowledge, and training category, the high-priority needs were dominated by information and communications issues. That dominance is even stronger when
 7


e top-tier needs addressed con- cerns about the ability of courts to maintain security and to be prepared for emergencies and other incidents. With respect to security, the need for standards and performance measures for security at different locatio


mation of citizens in court records. Tools for better data protection appeared in both the high-value and low-hanging fruit lists.
Addressing basic technology shortfalls in today’s courts. Panel members raised a number of concerns about the technolog


urt. Social media is a tool that so many members of the public use, and they increasingly assume that they should be able to communicate with government this way. However, courts need guidelines and materials to ensure that both they and citizens und


U.S. Court System
This effort, aimed at the national level, sought to frame an innovation agenda for the court system writ large. The value of the result will be driven by application and by how individual agencies or organizations use the identified


nizations with needs and roles to play in court innovation. This effort has provided a set of high-priority needs that rose to the top of our panel’s deliberations, as well as a broader—and much longer—set of innovation options that represent opportu


formation Officer Massachusetts Trial Court Boston, Mass.
David K. Byers
Administrative Director Arizona Supreme Court Phoenix, Ariz.
Kay Chopard Cohen
Executive Director
National District Attorneys Association Alexandria, Va.
The Honorable Amy Daven


ctitioners National Criminal Justice Association Washington, D.C.
Robert A. Zastany
Executive Director
Administrative Office of the Nineteenth Judicial Circuit Waukegan, Ill.
Courts Advisory Panel Members 79
    APPENDIX B
Courts Advisory Panel Pre-M


s unimportant would get a score of 0.
Each objective can have any number. For example, if you think all these objectives are equally important, all would be scored as 100. If you think they are each of different levels of importance, each score would


nformation per- taining to a specific case from the courts to or between external entities, such as the prosecutor’s office, defense counsel, law enforcement, pretrial services or other supervi- sory agencies, and the public. Functions here include c


enting technologies to support [NAME OF FUNC- TIONAL AREA] in your court system (or the court systems with which you are most familiar)? Please indicate whether each obstacle is major, minor, or not an obstacle in your experience. [Check boxes for ea


vidual level (e.g., ensuring due process and justice for an individual accused of a crime or party to litigation) or at the societal level (e.g., providing access to justice uniformly for all citizens or protecting the public at large by administerin


iving a weight from 100 (if it was equally important to the highest-priority objective) to 0 (if it was viewed as unimportant). These scores were then averaged across all panelists’ responses to produce a set of weights on the objectives. The results


that could be addressed through research or science and technology investment—that is, the specific needs. The panelists also identified which court objectives each need supported (by marking a simple “yes” if the need supported the objective). In al


ctics, management, and behavioral knowledge development and training
Facility operations and population services
Objective
       Percentage of identified needs
         Judges
Attorneys
Working group
Court administrators
and the doctrine, tactics, m


 2).
Detailed Methodology 93
94 Fostering Innovation in the U.S. Court System: Identifying High-Priority Technology and Other Needs
Table D.1
Example Original and Combined Needs
  Original
Combined
  Problem or Opportunity
Technology reliability is a


te storage.
  Figure D.4
Number of Original Needs Consolidated to Produce Combined Needs
80
70
Number of resulting combined needs
0
RAND RR1255-D.4
Prioritizing Court Needs
Detailed Methodology 95
     60
50 40 30 20 10
           12345
Number of ori


kthroughs if an effort to meet the need is successful, multiplied by the probability that such efforts will be technically and operationally successful. Put another way, the score for a need is determined by how beneficial it will be in achieving one


he data points into sets 1, 2, . . . K so that the following measure is minimized:
min∑K ∑x−μ2. i=1 xjinseti j i
Here, μi is the center, or average, of cluster i. This measure, also called the within-cluster sum of squares or cluster cohesion, is one


ly, it is as difficult as an NP (nondeterministic polynomial time) problem.
Detailed Methodology 97
 98 Fostering Innovation in the U.S. Court System: Identifying High-Priority Technology and Other Needs
Figure D.5
Ranges of Expected Values for Tiere


the score for a need broke through the top or bottom of its existing tier (e.g., a Tier 3 need’s normalized score increased above the top of the Tier 3 range), its rank was changed to the tier one rank up or down, even if the new score did not fully


sociated Need
Examine technologies to help organize and analyze large volumes of more-complicated information. Though some commercial tools
are available, courts need a
better understanding of how new technology could help manage the effects of digit


icy decisions, ranging from increases in criminal justice capacity to pushes for efficiencies in the system
Concerns about the suitability of the risk and needs assessments that help determine sentencing and release from the corrections system
Concer


y of examples, see Johnson, 2009; Clarke, 2015; and Cabral et al., 2012.
104 Fostering Innovation in the U.S. Court System: Identifying High-Priority Technology and Other Needs
Table E.1—Continued
 Category and Subcategory
Internal data collection: O


ailable for addressing backlogs for processing DNA (deoxyribonucleic acid) evidence (NIJ, 2015b). The broader issue of case backlogs was explored in the National Research Council’s review of forensic science in the United States (Committee on Identif


s for improving appearance rates (Legal Design Lab, 2015).
k The need for up-to-date source lists is widely recognized; is included in relevant standards, such as those published by ABA (2005); and has been examined in surveys of courts that explore


meliness goals of the justice system (versus just focusing on existing technology and what it can do).
High Value
Low- Hanging Fruit
 External communications
u
n For example, Rule 5.2 of the Federal Rules of Civil Procedure includes such guidelines (


: Video
Problem or Opportunity
Maintaining continuity of operations during natural or other events
Due process concerns about remote appearances in judicial proceedings
Limited resources for prosecutors and public defenders (e.g., not enough attorney


ird-party information as a result of technology, which is so integrated into the lives of defendants, victims, and police, creating challenges for both prosecutors and defenders
Space constraints (physical and virtual) for managing records and eviden


lop centralized standards for 1 authenticating electronic documents. Examples of implementation
are available, but no practice is universally adopted.
Build into record entry systems rules 2 or expert tools that help ensure complete data entry and ap


d be digitized).
Define requirements that some 3 categories of judicial work product
need to be confidential, even if they
are stored in the same system with official public files.a
In the absence of redaction, develop better ways to protect some sen


g this).a
Full List of Court Needs 111
 Category and Subcategory
Problem or Opportunity
Concerns about data quality within the court system as
a result of inconsistency in the way data are entered, limits in clerk knowledge, hiring of individuals wit


been shared appropriately.bb
Promote a common understanding 2 of what it means to share digital evidence (e.g., footage from
a security camera at a private
site). Does it mean offering the opportunity to access evidence (similar to physical evidence


e use of such tools as encryption.a
Develop standards for evaluating the 1 security of cloud storage providers to both inform decisions and assuage concerns.
Take advantage of records becoming 3 electronic and, rather than having to gather and ship r


a and mobile devices, which enable disclosure of information and cyber harassment
Courts’ tendency to be reactive to technology changes rather than plan ahead
Limited resources for prosecutors and public defenders (e.g., not enough attorneys, too hig


eline, could meet this need.
116 Fostering Innovation in the U.S. Court System: Identifying High-Priority Technology and Other Needs
Table E.2—Continued
 Category and Subcategory
Problem or Opportunity
Vendor systems that do not meet needs
Vendor sys


or 2 modify business processes that can
be adopted across stakeholders (or
match business processes to existing software capabilities), making it
possible for technologies to function effectively (examples of consensus development and ranking process


ng, and match processes
to the goals they are trying to achieve.a
Modify planning and funding processes to ensure that operations and maintenance costs are captured in acquisition decisions and included in out-year budgets.
2 u
c Organizational effor


ts who cannot afford
lawyers or procedures that allow
victim advocates to take a more central role in the trial process.a
e NCSC maintains a compendium of state rules on bulk data sale (NCSC, undated e). In 2005, SEARCH, the National Consortium for J


esources to better inform 2 judges’ decisions when appointing counsel (and to inform the public
about these decisions), including an assessment of the costs and benefits
of appointing counsel early versus late.
Adopt business process reengineering 1


 to decisionmaking.a
Determine caseload standards
to make it possible to perform h assessments across jurisdictions.
2 u
3
Update job requirements for court 2 management staff to create a body
of employees whose technology skills
are sophisticated en


erformance measures for effective security
for different types of courts and locations within a court to minimize intrusiveness for court participants, staff, and the public.a
Increase training and build the capability and expertise to addressj chang


instructions to ensure that those using the technology (lawyers, court officers, etc.) understand how to usea it, reducing the probability of errors.
 Societal/legal knowledge development and innovation
Develop standards and better 2 technological to


y corrections technologies, such as continuous alcohol monitors and Global a Positioning System monitors.
Pursue statutory authority or 1 court procedural rule authority
for specialists to appear via video presence to increase efficiency of
Develop i


 or fail
Challenges keeping information technology staff up to speed on emerging technology
Develop more exercises and drills 1 to determine likelihood of success,
such as using red teams, performing testing, and actually operating from backup sites


ine best practices to address 2 this misbehavior, which could range
from educational efforts to simple prohibition of mobile devices.d
Design court infrastructure to be 3 more welcoming and engaging.a
Develop standard lists of basic 1 technology that


merican Bar Association, “Court Funding Best Practices,” web page, undated. As of March 28, 2016: http://www.americanbar.org/groups/committees/ american_judicial_system/task_force_on_the_preservation_of_the_justice_system/Court_Funding_Toolkit/ ABA_T


Carrie A. Schneider, “Emergency Management in the Courts: Trends After September 11 and Hurricane Katrina,” The Justice System Journal, 2007, pp. 20–35.
Black, Nicole, “Lawyers, Cloud Computing, and Innovation: How Cloud Computing Facilitates Innovat


er 2004. As of March 28, 2016: http://www.proselex.net/Documents/JMI%20ELECTRONIC%20FILING.pdf
CCJ/COSCA Joint Committee on Court Security and Emergency Preparedness, CCJ/COSCA Court Security Handbook: Ten Essential Elements for Court Security and Em


ings-in-the-united-states/
Cowan, Bradley M., “Children in the Courtroom: Essential Strategies for Effective Testimony by Child Victims of Sexual Abuse,” The Army Lawyer, February 2013, pp. 4–16.
Crandall, Nancy, Making a Business Case for Automated


ter for State Courts, 2013.
Federal Judicial Center, Advisory Committee on Evidence Rules: Meeting and Symposium, Chicago, October 9, 2015. As of March 28, 2016:
http://w w w.uscourts.gov/file/18432/
Flango, Carol R., Deborah W. Smith, Nora E. Sydow,


09,
pp. 1–21.
Hardenbergh, Don, Robert Tobin, Sr., and Chang-Ming Yeh, The Courthouse: A Planning and Design Guide for Court Facilities, Williamsburg, Va.: National Center for State Courts, 1991.
Heerboth, Cynthia K., “Technology and Its Effects on t


 2009, pp. 259–283.
Joint Technology Committee, Making the Case for Judicial Tools, JTC Resource Bulletin, 2014.
Judicial Council of California, California Trial Court Facilities Standards 2011, San Francisco, 2011. ———, Video Remote Technology in Ca


://scholarship.law.wm.edu/facpubs/635
———, “Court Technology: For Trial Lawyers the Future Is Now,” Criminal Justice, 2004a, pp. 14–21. ———, “Introduction: What Have We Wrought?” William & Mary Bill of Rights Journal, Vol. 12, No. 3,
2004b, pp. 637–6


74–179.
McMillan, James E., Using Technology to Improve Customer Service: Trends 2007, Williamsburg, Va.: National Center for State Courts, 2007.
———, Videoconferencing Survey 2010 Results, Williamsburg, Va.: National Center for State Courts, 2010. A


ies/Courthouse-Design-and-Finance/Resource-Guide.aspx
———, “Court Website Design: Courts with Mobile Applications,” web page, undated c. As of March 28, 2016: http://www.ncsc.org/Topics/Media/Court-Websites/State-Links.aspx?cat=Courts%20with%20Mobile


au/law-professionals/wearable-technology-courtroom/
Office of the State Attorney, Fifteenth Judicial Circuit, Digital Evidence and Document Standards, Palm Beach County, Fla., June 2011. As of March 28, 2016: http://www.sa15.state.fl.us/stateattorney


thod.html
———, “Priority Criminal Justice Needs Initiative,” web page, Santa Monica, Calif., undated b. As of March 28, 2016:
http://www.rand.org/jie/justice-policy/projects/priority-criminal-justice-needs.html
Randall, Ronald, James A. Woods, and Ro


inal Justice Record Information, Sacramento, Calif., 2005. As of March 28, 2016: http://www.search.org/files/pdf/RNTFCSCJRI.pdf
Shapiro, Joseph, “As Court Fees Rise, the Poor Are Paying the Price,” NPR News, 2014. As of March 28, 2016:
http://w w w.n


ronic Evidence: Electronic Evidence Procedures,” August 1, 2012. As of March 28, 2016: http://www.tnmb.uscourts.gov/documents/EEP_APPs.pdf
U.S. Congress, Office of Technology Assessment, Criminal Justice, New Technologies, and the Constitution, OTA-C


ocial Media,” American Journal of Trial Advocacy, Vol. 36, 2013, pp. 641–653.
Zorza, Richard, and David Udell, “New Roles for Non-Lawyers to Increase Access to Justice,” Fordham Urban Law Journal, Vol. 41, No. 4, May 2014, pp. 1259–1316.
Society reli


  Jan Osburg, Philip S. Antón, Frank Camm, Jeremy M. Eckhause, Jaime L. Hastings, Jakub Hlávka, James G. Kallimani, Thomas Light,
Chad J. R. Ohlandt, Douglas Shontz, Abbie Tingstad, Jia Xu
 CAPABILITIES, NEEDS, AND MANAGEMENT OPTIONS FOR NASA’S AER


nterest to those respon- sible for managing NASA flight research infrastructure and capabilities, aeronautics researchers, and the broader oversight community in the Office of Management and Budget, the Office of Science and Technology Policy, and Co


nts about this report should be sent to the project leader, Jan Osburg (Jan_Osburg@rand.org). For more information about the Science, Tech- nology, and Policy program, see www.rand.org/jie/stp or contact the director at stp@rand.org.
Contents
 Prefac


creases in ARMD budgets that are expected to persist over the foreseeable future, budget levels over the next ten years similarly cannot be predicted precisely.
To address this challenge, we developed high-level needs projections for three different


ng the portion of the ARMD budget that is spent on flight research
3. Making the ARMD budget for flight research go further by:
– improving the efficiency of NASA’s flight research enterprise
– taking different approaches to adjusting flight research


capa- bilities are available and modernized to facilitate progress. Given the extensive breadth of ARMD’s SIP, an explicit prioritization of capabilities (as later described in MO-04) could inform possible infrastructure investments—as well as manag


ectation of stable funding, rather than having to plan for contingencies. Projects funded with the next $100 million could be designated as “con- fident” and managers could prepare in similar fashion. On the other hand, projects for which the funds a


of poten- tial users. A well-publicized annual survey could be instituted, complemented by less frequent long-term examinations like the Decadal Surveys compiled by the National Academies of Science. Leadership prioritization is key to planning, inte


versely affect pro- gram budgets and lead to delays—even collateral effects on other programs. Explicitly and effectively balancing such risks across all elements involved in flight research (air- craft, airspace, ranges, ground infrastructure, and w


o may have concepts that would benefit from small amounts of flight research but who do not have sufficient funding for flight research (e.g., in a similar way that NASA allocates supercomputing time to researchers). For example, many academics use R


trate return on investment by articulating benefits, especially in monetary terms, to explain their value (MO‐15). Whenever possible, NASA should continue to explain potential benefits in both domain and monetary terms. NASA funding ultimately depend


lka, David “Nils” Larson, Brad Flick, Sean McMorrow, Steve Schmidt, Albion H. Bowers, David A. Samuels, Chauncey C. Williams, and J. Brett Swanson. At Ames Research Center, we thank Michael Aftosmis, Roy Williams, Munro Dearing, Huy Tran, and Matthew


hout the study and for reviewing an earlier version of this report. We are also indebted to William Welser IV, for his constructive review of this report. Tom LaTourrette, Anita Chandra, and Marjory Blumenthal provided greatly appreciated quality ass


l
UAS unmanned aircraft systems UHBR ultra-high bypass ratio
UK United Kingdom
USAF U.S. Air Force
WFF Wallops Flight Facility
Abbreviations xxi
CHAPTER ONE
Introduction
 Flight research, flight testing, and flight safety analysis are essential compo


s the use of flight test is specifically indicated.
 1
2 Expanding Flight Research
RAND Corporation to assess its flight research and flight test needs and related capa- bilities, identifying gaps and management options (MOs) that support evolving AR


ecisely.
Despite these limitations, by using a futures-based approach, we were able to identify broader strategic direction, needed capabilities, and MOs.
Aeronautics Research Needs over the Next Ten Years
The 2015 ARMD SIP provided the foundation fo


rior work (e.g., Anton, Gritton, et al., 2004a; Anton, Johnson, et al., 2004b), we examined the conceptual range of needs and capabilities, rather than specific details (e.g., the predicted number of flight hours needed for specific aircraft out to t


We compared the needs with the capabilities as a way to identify any gaps or excess. Given the uncertainties already outlined, this gap analysis was at the level of research domains in the SIP thrust areas mapped against the five areas of flight rese


ina and Russia, with which close collaboration in aeronautics research is less likely in the foreseeable future.
The timing of this study imposed certain limitations as well. At the time of the kick-off meeting, ARMD had not yet published its SIP nor


tion tools. Based on this, we generate a set of three estimates for future needs, representing steady, reduced, and increased demands for flight research.
Second, we review flight research capabilities based on existing NASA flight and ground test as


ter One), we did include UAS-related flight research needs because they feature prominently in NASA ARMD’s SIP and generally also involve non-UAS capabilities; e.g., manned safety chase planes.
3.
Support to space exploration–related efforts (from mi


um UAS to establish proof of concept and mature the underlying technologies. Technologies with the most promise are scaled to larger aircraft (manned or unmanned) and flight tested. In some cases the goal is autonomous medium UAS operating in the Nat


he develop- ment of next-generation fighters (although such collaborations are outside the scope of the current SIP).
In the international arena, Japan and nations within the European Union (EU) have civil aviation research with portfolios similar to


- tion (Tinoco, 2008), as does pushing past TRL 5 (defined as system and component
12 Expanding Flight Research
prototype validation in the expected operating environment). When considering the impact of M&S development on flight research, the primar


parallelism with heterogeneous, specialized hardware. This may have negative implications for code generality and computation efficiency. On the algorithmic front, the state-of-the-art industrial CFD remains the unsteady RANS (URANS) codes based on


e.g., motion, lighting). Moreover, faithful representations of vehicle behavior are fundamentally dependent on our ability to accurately characterize aircraft aerodynamics, dynamic response, and control logic across the flight regime, all of which ar


ay have implications for flight research. For example, the most-profound advances in aviation efficiency in the last half-century have come from improvements in propulsion, specifically through the development and refine- ment of the high bypass turb


ht research, ground testing still provides controlled environments that can be more heavily instrumented but ground testing no longer benefits from industrial scales of efficiency, given the modern role of CFD. As a result, flight research is more co


fect), the potential futures help explore how these factors might evolve, identify management challenges, and ultimately serve as a backdrop against which to illustrate how the MOs we later develop could help ARMD regardless of what occurs in the fut


likely to benefit most directly from such a technology demonstrator face potential questions of environmen- tal compatibility and economic viability. Hence, it may be reasonable to expect that the supersonic thrust has to cope with additional, nontec


sed resources for flight research. This future would include full, robust, stable research in all six SIP thrust areas, additional investments in modifi- able testbed aircraft, and initiation of major investments in both subscale and full- scale X-pl


ables real-time analysis of flight test data and decisionmaking on envelope expansion and other efforts during a flight
• ground-based flight simulation and flight research laboratories to enable risk reduction prior to test flights
• maintenance and


search Capabilities
NASA’s fleet of more than 60 aircraft—including the aircraft used for flight research— is distributed across several NASA centers. NASA research aircraft are used for both aeronautics flight research and airborne science research.


1H
KSC
3× UH1H
SOURCE: Based on NASA, 2015a.
NOTE: Aircraft of particular relevance to ARMD are highlighted in bold.
JSC = Johnson Space Center; KSC = Kennedy Space Center; WFF (GSFC) = Wallops Flight Facility (Goddard Space Flight Center).
RAND RR13


rthiness, aircraft/cruise missile, and crossover/atmospheric vehicles (aeronautical aspects). Flight test range assets include the Ridley Mission Con- trol Center and Birk Flight Test Facility, supersonic flight corridors, reconnaissance test ranges,


afe testing environment and for overseeing that test site operation occurs under strict safety standards.
Flight Research Capabilities of U.S. Industry and Academia
Industry and academia focus on T&E and R&D, respectively. Both sectors offer sub- son


chinery Facility.
NASA supports flight research in academia, and some academic researchers use NASA facilities for testing. Although this relationship might extend further to NASA taking advantage of university flight research equipment and pilots, t


ve to long-term planning and investing, since they are able to accumulate funding across multiple fiscal years, have more autonomy in setting their own research agendas, and are not prohibited from “competing” with commercial enterprises.
During our


will replace their Lear 25 with a Lear 35, and LaRC is pursuing the acquisition of another HU-25C. JSC’s C-9B is already scheduled for retirement, as is one of their T-38s. On the commercial side, the fate of Boeing’s current EcoDemonstrators is unc


on 20E (D‐CMET) (D‐ADLR) G550
    Boeing:
Demonstrators 757, 787, Eco‐ 747 engine 757 engine P&W, GE: Honeywell: testbeds
testbeds Calspan:
G‐III, Learjet
      MQ‐1** MQ‐9** RQ‐4*,**
C‐12 C‐130
B‐2 B‐52 C‐5 C‐17
KC‐135
        WFF
B‐200
(N8NA)
(N439


 Cessna C206 SR‐22 (N501NA) Columbia Cirrus 300
(N525NA, HU‐25C TBD?)
       GRC
(N608NA)
(N601NA)
T‐34C
S‐3B
  DHC‐6
(N607NA)
Lear 35
Lear 25*
(N616NA)
       ARC
Sierra**
(N707NA)
       AFRC
B‐200 (N7NA, N801NA) (N870NA) (N149AF)+ Ikhana**
TG‐14
(


II
SOFIA
         Low Subsonic
(M<0.3)
Subsonic
(0.3<M<0.7)
Transonic
(0.7<M<1.2)
  International
 Industry Partners (Examples)
 Aircraft with Potential Utility for ARMD Flight Research—Ten Years Out
NASA
DoD
 Table 2.3
Maximum Speed
 NASA ARMD Fligh


nomy for Aviation
Transformation
6
         Legend: Available now Can acquire or modify during project ramp-up Cannot acquire in advance N/A NOTE: AC=aircraft; ACTE=adaptive conformal trailing edge; cf.=comparable to; GA=General Aviation; GE=General


h using rotorcraft. One company the research team talked to mentioned some interest in a vertical-lift consortium, but that would also require budgetary stability. Finally, some international flight research organiza- tions such as DLR, JAXA, and NRC


 in Chapter Four.
Excess
We found no current, strategic excess flight research capabilities that do not map to one or more SIP thrusts.
Summary of Findings
ARMD has substantial infrastructure supporting flight research in funded areas. While there ar


are usually significantly cheaper and faster to build than full-size platforms, and enable char- acterization of some—but, due to scaling issues, not all—aerodynamic effects in flight. They also are subject to more-restrictive volume and power const


on study on the U.S. Antarctic Program concluded that 90 percent of funds went toward logistics costs and only 10 percent toward the science operations that the logistics efforts are supposed to enable (U.S. Antarctic Program Blue Ribbon Panel, 2012)


         SOURCES: NASA, 2015a, and previous AMD Annual Aircraft Reports.
RAND RR1361-3.3
Additional Factors Affecting ARMD Flight Research 41
Figure 3.4
AFRC Aircraft Utilization, June 2009–May 2014
800
700
600
500 Five-year cumulative flight hours 4


ith flight research managers at the four NASA centers involved in flight research, the workforce is aging at the same time that employment caps and lags makes hiring replacements difficult, if not impossible. At some centers (but not across NASA), th


research is needed to raise the TRL of more aeronautics tech- nologies to where they can be incorporated into commercial products (for examples, see Klamper, 2009; Warwick, 2015; NASA, 2015c; NRC, 2012). Related options are thus included, as they ma


anagement Option
MO‐01 Planning integration MO‐02a Strategic investment MO‐03a Acquire‐at‐need
MO‐04 Increase budget certainty
MO‐05 Increase cooperation
MO‐06a Increase partnering
MO‐07 Space/military tie‐in MO‐08a Annual needs survey
MO‐09 Cross‐ce


orate into a specific project, researchers should at least create a technology transfer road map showing at what point flight research would play a role in advancing the technology in question toward higher TRLs (and, ultimately, toward transfer to i


ubsequent tech-
1 TRL is measured on a scale from 1 to 9, with TRL 1 (“basic principles observed”) being the lowest level of readiness and TRL 9 (“actual system flight proven”) being the highest.
 nology transfer, which increases the impact of ARMD r


ated capabilities. Such cycles of revolving or staggered activity, including associated mothballing and restoration of flight research capabilities, could be synchronized with NASA’s long-term planning, such as the Aero- nautics Roadmap (NASA, 2015g,


Another contributor to increasing flight research is ensuring that sufficient infra- structure capabilities are available to facilitate progress, since inadequate equipment and
2 As suggested above, the magnitude of the costs associated with each of


ly given the lull in need.
Potential positive effects: Reducing active aircraft inventory directly decreases cost of ownership, which allows for increased investment in infrastructure improvements.
Potential negative effects: Reducing active aircraft


planning and prioritiza- tion has been accomplished (MO-01, MO-04), then gap and market analysis can be focused on potential opportunities.
Discussion: As discussed in the previous section, when utilization is projected to be low for long periods of


duty in a configuration consistent with a current DoD version than it took for the WB-57.
This is true whether the aircraft remains in active use or is mothballed for poten- tial resurrection. If a NASA aircraft effectively becomes obsolete while in


volves ramp-up cost (both for hardware and for workforce retraining). Thus, the success of this approach depends on the stability of the strategic plans on which deci- sions are based, and on the availability of ramp-up funding when needed.6 Savings


ate” for each year, then identify the top 60 percent of projects as “high confidence,” the next 20 percent as “likely,” and the last 20 percent as “possible.”
Rough level of effort: Modest. Once longer-range plans and cost estimates exist, then ranki


. In fact, better communication may be all
Management Options 55
56 Expanding Flight Research
that is required. While we heard from some sources at ARMD HQ that such planning certainty exists, some center-based researchers were unaware of this and we


l as NASA’s Office of General Counsel, approve the final plans and arrangements.
Who would implement the MO? Project and program managers with center infrastructure managers would handle implementation.
Implementation steps: Specific steps vary by ca


s, including through proper planning of any outreach activities and publications, something particularly important in foster- ing long-term partnerships
Management Options 57
58 Expanding Flight Research
• making sure research that is brought in is i


s do not meet criteria for appropriateness; however, collaboration interest can also be an indicator for changing community needs (MO-08).
Effects on other stakeholders: Cooperative research can facilitate research and may incur financial reimburseme


the high cost of supersonic capabilities. Time to flight can be reduced if
Management Options 59
60 Expanding Flight Research
the partner has a needed capability ready to go. The four scenarios explored in Chapter Five offer details specific to partn


given that ARMD does not have primary responsibility for these research areas. However, areas such as hypersonics have well- established coordination panels.
Discussion: A different strategic approach for pursuing increased budgets is to tie ARMD fli


(or a centralized, matrixed organization would, in the case of MO-09).
7 See, for example, the Aerial Regional-scale Environmental Survey of Mars (ARES) proposal for a “Mars Scout” mission, although it was not funded for implementation (NASA, 2011c).


e efficiency of available dollars is critical. NASA, like many organizations facing persistent budget shortfalls, has already been working on increasing its efficiencies, but we identified several related MOs for consideration. The options also inclu


director. This structure sets up a natural tendency for staff to advocate for local capabilities when talking with programs about future research. Some organizations framed this as a friendly competition to provide support to programs.
A new cross-ce


s such as intellectual property decisions and international policies. Such a function could be performed by a new, separate office, but having it integrated with the internal matrixed organization would provide for a convenient “one-stop shop” and fa


 Initial investigations, however, could be modest, explor- atory research endeavors aimed at developing efficient, practical approaches for aug- menting existing strategic efforts.
Discussion: Flight research is subject to interacting execution risks


hose risks should lead those projects to explicitly develop risk- mitigation plans for how they would handle budgetary instability and the effect of that instability on infrastructure investments needed for their research. For example, those projects


1 cannot be implemented by ARMD alone, because it would affect NASA center and NASA HQ assets and roles.
Reducing the number of backup chase aircraft would trade increased schedule risk for cost reductions through reduced infrastructure redundancy. N


.1 Average 90.8
SOURCE: Horn, 2014, chart 22
FY 2013 Fleet Availability (%)
97.3 99.2 94.7 84.4 90.6 91.8
Number of Aircraft in Fleet
1 1 2 2 3
—
Low Fleet Availability Rate (%)
   97
   99
   95
   66
   91
   91
High Fleet Availability Rate (%)
  1


lity of aircraft behaves as though the breakdown rate—or nonavailability rate—for each aircraft is binomially distrib- uted. Table 4.5 shows the resulting relationships between the number of support aircraft that are prepared for a flight test and th


led data for the reliability of each unique tail number in the fleet, could support decisionmaking on potential fleet size reduction options by helping balance cost of ownership against the risk of potential schedule delays. However, the latter can b


request process, possibly in conjunction with the annual user survey (MO-08) or using SMD’s Science Operations Flight Request System (SOFRS). Establish a selec- tion board (which could reach across NASA through the matrix managed structure, MO-09) an


potentially highly valuable flight research, thus reducing process times (MO-13) and cost to the researcher while increasing benefits relative to cost. Further- more, it would allow researchers to gain experience with and understand the value of fli


performing flight research at NASA can make their needs known (MO-08).
It could also involve designating single points of contact who can help research- ers navigate the required processes and paperwork (for example, the approach used by DLR and the


available. Therefore, this option entails improving the storage and archiving of flight research data and derived results, facili- tating U.S. researcher access to that data in order to increase the value that can be extracted from existing flight re


ral hard drives—that needed to be archived, this would result in storage charges of $2,400 per year.
Potential positive effects: Improving access can save cost and time needed for finding and accessing relevant flight research data. It can also reduc


ain potential benefits in both domain and monetary terms.
Perhaps the most direct approach to justifying increased flight research is to describe its relative value as ROI. In addition to explaining the potential benefits in qualitative terms, moneti


ue) plus the monetary value ($1.8 billion) better informs nonspecialists of the costs and benefits of NASA’s research. These figures can
11 The Environmental Protection Agency estimates the social cost of carbon dioxide to be about $41.15 per ton of


 fixed nonrecurring costs of maintaining unique flight research capabilities by spreading them across a larger user base.
Potential negative effects: Expanding BCA and promoting the results will incur a certain cost, mainly in labor, and credible mon


uld be unacceptable to the owner). However, leasing is not viable for testbed aircraft (because they usu- ally require major modifications) or supersonic aircraft (because generally they are not commercially available for lease). Thus, while leasing


aintain capabilities that NASA does not have, and collaboration as well as partnering can therefore be a viable option (MO-05 and MO-06). However, full outsourcing to inter- national organizations would require NASA to accept significant risk and int


rch.
The scenarios represent examples of past projects and those of plausible future work.
1. ACTE—a conformal trailing edge flap utilizing nonrigid wing flaps mounted to a large business jet–class aircraft
2. LBFD—supersonic transport technology res


ning partnering alternatives and other MOs. Alternative arrangements change the overall cost, timeliness, and safety of the scenarios. The alternatives allow for a systematic assessment of alternative capability sources, which can illustrate partneri


echnology and covered some of cost for the testbed aircraft, the first two alter- native scenarios provide contrast to that partnering arrangement. The all-NASA and all- DoD alternatives primarily transfer the cost to one or the other, and the latter


 aircraft
  Increases by $300,000
and all costs transferred to DoD
  No significant change
  No significant change
  No significant change
  No significant change
  No partnering with DoD
  No partnering with NASA
             NOTE: Green=substantial


lly the LBFD would be flown in the confines of a test range but, given adequate confidence, could be flown in the NAS. The first alternative has NASA not partnering in order to maximize public benefit of a technology developed for minimizing sonic bo


ost, time, and other considerations (such as partnering opportunities). The ratings are shown in Table 5.6.
Ultimately, the acquisition cost of a low-boom supersonic demonstrator, expected to be in the millions of dollars, outweighs the costs of supp


either industry or other government entities and a contrasting simple alternative of going it alone. Table 5.7 shows the alternatives that were considered, with the ratings provided in Table 5.8.
The two expanded partnering alternatives have the adva


l increased risk of test failures due to multiple safety standards
No significant change
      Significant partnering with industry and academia
Lack of partnering, but more flexibility to release intellectual property
Partnering with DoD and FAA


ake use of partnerships around the world
  SOURCES: NASA, 2012; Office of the Undersecretary of Defense, 2013.
NOTES: Amounts converted to FY 2015 dollars. Assumes 15–20 sorties. The flight test aircraft is a four‐ engine aircraft, for which acquisit


Partnering can transfer costs but not necessarily reduce them. Government part- ners are unlikely to provide flight research capabilities that are more cost-effective or more capable than those maintained by NASA. When costs are transferred to indust


1361-5.4
                   Compatibility Matrix for Management Options
Legend: Management options are compatible Management options are compatible and have positive synergies
risks; MO-11=Adjust number of chase planes; MO-12=Voucher system; MO-13=St


name.
Assessment of Management Options 95
 96 Expanding Flight Research
Figure 5.5
SME Ratings of 15 Management Options, by Relevance
Management option
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
  5
4 SME rating (1 = no relevance; 5 = high relevance) 3 2
1


10; NRC, 2012; Pav- lock, 2013; Merlin, 2013) and from interviews with NASA researchers and managers. We then performed a gap analysis by mapping these need categories against NASA’s existing capabilities.
We found that NASA has capabilities in most


tated every few years. (MO-02, MO-04, MO-15)
– Based on the resulting focus areas, consider divestment or reduction in readi- ness of flight research capabilities that are not currently needed. For example, the supersonic aircraft are one category of


rts of explor- atory research through a voucher program for internal and external researchers. This will also leverage external research and may foster interest in additional collaboration. (MO-05, MO-12)
Articulate benefits in monetary terms to faci


nontraditional” commercial partners, insights from and access to new markets), whereas universities can offer opportunities to work with researchers who have experience designing experiments, publishing, and exploring new directions in flight researc


tomated aerial refueling efforts and sense-and-avoid testing. It has a number of aircraft available for research, including four Learjet “in flight simulators” that can be modified with programmable flight systems, as well as a Gulfstream G-3. It als


 to a PDR [preliminary design review]) and a test readi- ness review stage, which includes a risk and hazard analysis.
Finally, we interviewed some flight research organizations outside of the United States that expressed interest in partnering with


enter into and perform such contracts, leases, cooperative agree- ments, or other transactions as may be necessary in the conduct of its work and on such terms as it may deem appropriate, with any agency or instrumentality of the United States, or w


may have to be explored further to expand partnering between ARMD and external—particularly commercial—organizations. In infor- mal conversations with members of the research team, several industry representatives voiced concerns about partnering wi


tracts providing long-term support of the QinetiQ five-meter pressurized wind tunnel in the UK (QinetiQ, 2013b), and the 25-year Long-Term Partnering Agree- ment between the MOD and QinetiQ Group running from 2003 to 2028. Under the latter agreement,


e entire matrixed organization. Such shifts in authority, responsibility, and benefits is why the exact structure of the resulting organization would require further analysis and negotiation among the stakeholders to address concerns, statutory and r


mpetitor has fixed costs that it must cover in its price to remain in business. More competitors thus means higher total fixed costs that have to be spread out over the customer base, especially in areas such as flight research, where fixed costs are


an individual link within the greater value chain to improvement of these final products. “Improvement” involves achieving better matches among the attributes of the final products and what their users want in terms of cost, reliability, and other p


e in place, efforts to increase competition can actually degrade coordination and induce partners to withhold information and sacrifice long-term mutual gains for short-term opportunistic behavior.
Within organizations, efforts to coordinate specific


ive participation from each center’s operation group rather than having a single local contact that can represent and reach out to other centers’ capabilities. If NASA chose to pursue this MO, then different options for how the matrixed organization


anagement: Not a Structure, a Frame of Mind,” Harvard Business Review, Vol. 68, No. 4, 1990, pp. 138–145. As of March 12, 2016: https://hbr.org/1990/07/matrix-management-not-a-structure-a-frame-of-mind/ar/1
113
114 Expanding Flight Research
Bilstein,


ment for Army Science and Technology Programs Under Uncertainties, Santa Monica, Calif.: RAND Corporation, MG-979-A, 2011. As of April 15, 2016: http://w w w.rand.org/pubs/monographs/MG979.html
Creech, Gray, “NASA Quiet Sonic Boom Research Effort End


puterworld.com/article/2599804/ cloud-computing-nasa-launches-massive-cloud-migration.html
Gawdiak, Yuri, Gregory Carr, Jeremy Eckhause, Robert Hemm, Mark Narkus-Kramer, and Jeffrey Johnson, “JPDO Evaluation of NextGen Operational Benefits,” 11th AIA


had J. R. Ohlandt, and Jan Osburg, Pricing Strategies for NASA Wind-Tunnel Facilities, Santa Monica, Calif.: RAND Corporation, TR-999-NASA, 2011. As of August 19, 2015: http://www.rand.org/pubs/technical_reports/TR999
Long, Dou, Jeremy Eckhause, and


earch,” 2014a. As of January 8, 2016:
http://science.nasa.gov/media/medialibrary/2014/12/05/ NASA_Plan_for_increasing_access_to_results_of_federally_funded_research.pdf
———, NASA’s Socio-Economic Impacts Aligned with the 2014 Strategic Plan, Washingt


/files/fy_2017_budget_estimates.pdf
———, “ARMD Aircraft Req FY15-20 for AMD May 2015 V5.xlsx,” Excel spreadsheet, provided by ARMD, January 20, 2016b.
———, “NASA Innovative Advanced Concepts (NIAC) Program,” updated March 24, 2016c. As of April 26, 2


MB—See Office of Management and Budget.
ONERA, 2014 Annual Report, Palaiseau Cedex, France, 2015. As of March 28, 2016: http://w3.onera.fr/rapport-annuel/sites/w3.onera.fr.rapport-annuel/files/documents/ onera-ra2014-va.pdf
OSTP—See Office of Science


r 27, 2006. As of June 6, 2016:
http://www.dfrc.nasa.gov/Gallery/Photo/Quiet_Spike/Large/ED06-0184-23.jpg
Tinoco, Edward N., “Validation and Minimizing CFD Uncertainty for Commercial Aircraft Applications,” 26th AIAA Applied Aerodynamics Conference,


6:
h t t p s : / / w w w . w h i t e h o u s e . g o v / t h e - p r e s s - o ffi c e / 2 0 15 / 0 7 / 2 9 / executive-order-creating-national-strategic-computing-initiative
Womack, James P., Danie T. Jones, and Daniel Roos, The Machine That Changed


 CORPORATION
Designing Compensation Programs for Individuals and Households After Man-Made and Natural Disasters in the United States
 Steven Garber
For more information on this publication, visit www.rand.org/t/RR1005
  Library of Congress Catalogin


tion and inform policy choices about liability, compensation, regula- tion, risk management, and insurance. The institute builds on a long
iii
iv Designing Compensation Programs for Individuals and Households After Disasters
tradition of RAND Corpora


the seven PVCPs were responses to intentional acts—four mass shootings (Aurora, Colorado; Newtown, Connecticut; Tucson, Arizona; and Virginia Tech’s campus) and an act of terrorism (the Boston Marathon bombings). Another responded to an accident (at


P designers can follow some practical sugges- tions to help accomplish their goals fairly, quickly, and efficiently. Such suggestions include the following:
1. Focus on the well-being of the victims of the disaster and on meeting obligations to funde


s for their contributions to the report. First, I thank the two technical reviewers: Robert L. Rabin (Stanford Law School) and Nicholas M. Pace (RAND). Second, I am grateful for comments from several members of two advisory boards at RAND— namely, th


ndividuals and households for personal injuries, property damage, and death.
One of the most important questions related to this topic asks how such a compensation program should be designed. And designing a VCP requires answering several subquestion


, for example, Faure and Hartlief, 2006; Faure, 2007). There are several ex ante compensa- tion programs in the United States—that is, programs created and designed before the rel- evant injuries occur. Examples of ex ante schedules of compensation i


enses, such as costs of temporary housing. This assistance is not intended to provide compensation for losses by individuals or house- holds (in contrast to public entities, which are often compensated through FEMA for infra- structure damage).
7 See


sed as a condition for receiving com- pensation from a victim compensation fund (VCF), such compensation may be viewed as a “collateral” source in tort and thereby reduce dam- ages available from tort actions.
The doctrines and processes of the U.S.


vel GVCPs, including one created in
1 See Table 4.1 in Chapter Four.
2 While the 9/11 VCF is widely described as unprecedented, publicly funded compensation
predates the terrorist attacks on September 11 (Landis, 1998). For example, the United States


(Feinberg, 2005, pp. 34–37; Feinberg, 2012, p. 63). Moreover, the requirement that compensation received from collateral sources be deducted from compensation amounts available from the fund (so-called “offsets” for collateral sources) became very p


oxic exposures. In relation to the traumatic fatal and nonfatal injuries addressed by the original fund, then, the 9/11 VCF of 2011 raises difficult issues involving injury causation, as well as the potential for compensable injuries becoming manifes


clude tort actions against private parties. VCP design decisions not determined by the legislation were delegated to a panel of three special masters. The state reached settlements with all 179 eligible victims, for a total of roughly $36.5 million i


35,000 for each death claim. For nonfatal injury claims, the amount of compensation depended on numbers of days and nights the victim was hospitalized—specifically, $25,000 for ten or more days and nights of hospitalization, $7,500 for between four a


rograms
 This chapter considers seven PVCPs since 2007, detailing their known design features in a table and then comparing and contrasting those features.
Table 4.1 provides overviews of seven PVCPs set up in response to man-made disasters since 200


tion at the Kenneth R. Feinberg • • Aurora Victim Relief • • Special Master:
12 fatalities, 70 others injured
movie theater, Aurora, Colo. Shooting at
July 20, 2012
other funds established Psychological injuries relinquish tort rights to for individu


ollar value of points to respectively processed
  About $77 million avail- tion Compensation Fund, Reno Air Racing Associa- • claiming deadline Octo- (almost 1 year after the Kenneth R. Feinberg established August 2012 able to compensate; Administrat


families of the • o $40,000 and tuition and fee o $90,000 and tuition and fee o $10,000 or tuition and fee o $10,000 or tuition and fee waiver to those in specific classrooms and not covered • $6.5 million, distributed the last • (for mental trauma)


clusion of some types of injury; for exam- ple, Aurora victims who were not hospitalized and Boston Marathon victims with only psychological injuries were ineligible for payments. Moreover, physically injured Boston victims who were not hospital- ize


As reported in Table 4.1, the Boston Marathon bombing and Aurora shooting PVCPs pro- vided compensation payments within four and five months of these events, respectively, and payments were distributed from the Hokie Spirit Memorial Fund less than se


(Dixon and Stern, 2004, pp. 138–140).
The existence of multiple sources of compensation can greatly complicate the tasks of VCP designers and impede them from achiev- ing their goals. First, for VCP designers who seek to distribute their funds to pro


administrative costs. Design- ing a VCP when funds are limited is much more challenging than when funds are unlimited. Thus, the discussion that follows focuses on situations with limited funds,1 and it is assumed that available funds do not suffice


 as well as what
2 Of course, PVCP designers are likely to receive such comments even if they do not invite them.
 deadlines they must meet. Designers must also develop forms for claimants to use to apply for compensation. In addition, claiming pro-


ern to VCP designers are appar- ent from what designers and onlookers emphasize—namely, the extent of fairness, speed of compensation, and size of transaction costs. As we will see, there are conflicts or trade-offs in jointly pursuing these outcomes


zontal equity. As will become apparent, conceptualizing and apply- ing notions of fairness leave lots of room for discretion, even for pro- gram designers who choose to focus on these three aspects of fairness.
Social Welfare or Tort?
A helpful persp


rvingness and horizontal equity). Such indicators reflect different interpretations of need—involving dif- ferent levels of complexity—that VCP administrators could measure.
The simplest indicator of need listed in the table is a claimant’s total los


ies of injury based on pain and suffering,
future medical expenses, and so on.
   Horizontal equity
Concept: Victims in identical or similar circumstances should be treated identically or similarly.
Value judgment: People in identical or similar situ


ample, most victims of terrorist acts or mass shootings cannot be viewed as knowingly put- ting themselves in harm’s way.
 in harm’s way.7 When determining eligibility and amounts of com- pensation, many VCP designers are likely to conclude that whet


nsurance for personal injury. The simplest plausible indicator—and the first listed for deserv- ingness in Table 6.1—might be whether a victim had any insurance relevant to the losses caused by the disaster. This indicator might be too crude for VCP


rity. In designing and implementing the 9/11 VCF of 2001, special master Feinberg found the horizontal equity principle appealing in theory but unhelpful in practice. In particular, he writes, “I tried to treat similar claims alike. But I soon learne


ategories reflect respected value judgments about what factors make victims more and less alike for the purposes of VCP design. The third listed indicator would refine the second indicator by considering—in addition to days hospitalized—pain and suff


a- tional indicators of fairness are adopted.
Fairness to Nonvictims
While most of the concern about fairness in designing VCPs centers on being fair to victims, VCP designers may also have substantial con- cerns about being fair to nonvictims, such


erjury that all information contained in and submitted with the Eligibility and Compensation Forms is true and accurate. False statements or claims made in connection with this application may result in fines, imprisonment and/or any other remedy ava


ant to limit the transaction costs involved in designing and administering VCPs. These costs include the salaries and wages paid to program designers, administrators, claims proces- sors, and auditors; travel expenses; and costs associated with creat


likely delays processing claims and remitting payments.
Activities that increase transaction costs may, however, be cru- cial to addressing other policy concerns. For example, requiring more- extensive documentation of eligibility for compensation an


nowledge of the true probability of experi- encing losses of various magnitudes.
 experimental evidence and observed patterns of behavior in real-world settings.19
Financial Self-Protection
As the uptake and coverage levels of relevant kinds of insur


across disasters in the number and nature of injuries, differences in budgets across VCPs, and extensive uncertainty about such key issues as how much compensation victims have received or will receive from other sources. In short, when it comes to


ments that many would view as very generous. In reflecting on his experience as the administrator of the 9/11 VCF of 2001, the Hokie Spirit Memorial Fund, and the GCCF, Feinberg states that establish- ing a VCP in response to a disaster creates “a s


y, design- ers may have obligations to those who provide a VCP’s funds—namely, taxpayers and donors—and to those who created the VCP and del- egated remaining design decisions (for example, to a special master). How these obligations are best balance


e likely to confront in designing most, if not all, VCPs. But providing specific recommendations about complete designs or key features of high-quality designs seems unrealistic. For instance, the best design decisions—and the appropriateness of part


nd size of transaction costs—with fairly small sacrifices in other outcomes. These suggestions for VCP designers are as follows:
1. Focus on the well-being of the victims of the disaster and on meeting obligations to funders and upstream VCP creators


cover the administrative costs of the program. Rightly or wrongly, this is money to which victims are likely to feel entitled. For example, try to find businesses that are willing to cover administrative costs separately, and encourage dona- tions o


signers face complex tasks with no correct solutions. The design- ers are often monitored, criticized, second-guessed, and challenged by various stakeholders. The most experienced and prominent designer of U.S. VCPs has cautioned that “no good deed g


013/04/11/nyregion/ foundation-to-distribute-4-million-to-families-affected-by-newtown-massacre. html
Associated Press, “Newtown Struggling to Divvy Up Donations for Sandy Hook Shooting Victims,” Insurance Journal, July 2, 2013a. As of January 19, 20


torney’s Office, Victim Services Division, undated. As of May 23, 2016: https://www.ncjtc.org/CONF/Ovcconf/AttMat/ Understanding%20Victim%20Compensation_Burbank_Handout.pdf
Campbell, Alex, “$13.2M Settlement Stalls in Indian Stage Collapse,” Indianap


 pp. 719–768.
Disis, Jill, “Stage Collapse Victim Asks Appeals Court to Throw Out State’s Liability Cap,” Indianapolis Star, December 16, 2014. As of June 10, 2015: h t t p : // w w w.w h a s11. c o m /s t o r y /n e w s / l o c a l / i n d i a n a /


com/articles/97244/ minn-reaches-deals-with-bridge-collapse-victims
Indiana State Fair Commission, “Indiana State Fair Relief Fund—Final Protocol,” undated. As of May 23, 2016: http://www.thefederation.org/documents/05.Protocol%20%20FINAL-Clean.pdf
R


h Paper Series, No. 567, 2014.
Mullenix, Linda S., and Kristen B. Stewart, “The September 11th Victim Compensation Fund: Fund Approaches to Resolving Mass Litigation,” Connecticut Insurance Law Journal, Vol. 9, 2002–2003, pp. 121–152.
Nagareda, Richa


tember 6, 2015. As of February 25, 2016: http://www.vcf.gov/pdf/VCFProgramStatistics09092015.pdf
Stafford, Dave, “AG: State Fair Stage Collapse Victim Payments Completed,” Indiana Lawyer, December 20, 2012. As of September 5, 2014: http://www.theindi


ng and Financing Recovery from Natural Disasters in the United States, Philadelphia, Pa., March 2008.
References 69
Man-made and natural disasters (such as mass shootings, terrorist attacks, flooding, and hurricanes) occur fairly frequently in the Un


  CORPORATION
Systematic Reviews for
Occupational Safety and
Health Questions
Resources for Evidence Synthesis
Susanne Hempel, Lea Xenakis, Marjorie Danz
For more information on this publication, visit www.rand.org/t/RR1463
 Published by the RAND Cor


rch on the methodology of evidence synthesis.
RAND Infrastructure Resilience and Environmental Policy
The research reported here was conducted in the RAND Infrastructure Resilience and Environmental Policy program, which performs analyses on urbaniza


a particular challenge, typically exceeding the scope of traditional evidence-based medicine systematic review approaches. Furthermore, systematic reviews that support guidance and recommendation documents need to consider mechanisms to ensure trans


s, and other sources. The chapter outlines the development of search strategies and the use of citation management software. It covers formulating eligibility criteria for the review, outlines the inclusion screening process, and introduces a literat


Table S.1.
Table S.1. Online Resources
Resource Link Chapter One: “Introduction and Report Methods”
    Cochrane Handbook for Systematic Reviews of Interventions
Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group


 evidence-based reports
Abstrackr, software for automated citation processing
Weka data mining algorithms
USPSTF procedure manual
HLWIKI International
http://www.prisma-statement.org/Extensions/Protocols.aspx http://www.crd.york.ac.uk/PROSPERO http:/


ICF International, Dragon, online tool for systematic reviews to support risk assessments
Guide to Community Preventive Service, guidance for data extraction
AHRQ Slide Training Module for data extraction
Newcastle-Ottawa Scale for observational stud


l for public commenting
http://www.ahrq.gov/research/findings/evidence-based- reports/search.html https://www.federalregister.gov/articles/search?conditions%5Bterm %5D=Request+for+information&commit=Go https://effectivehealthcare.ahrq.gov/index.cfm/r


Healthcare Research and Quality CDSR Cochrane Database of Systematic Reviews DARE Database of Abstracts of Reviews of Effects EPA U.S. Environmental Protection Agency
EPC Evidence-based Practice Center GRADE
IOM
NIOSH National Institute for Occupatio


ber of relevant resources are available, including published guidance from established organizations describing their procedures (Higgins and Green, 2011, and Agency for Healthcare Research and Quality [AHRQ], 2014). However, while researchers conduc


e conducted for a variety of purposes, but those supporting recommendation and policy documents have unique requirements. In particular, they need to embrace mechanisms to ensure transparency of the evidence summary as well as the recommendations sup


adj3 evidence).ti. NOT “weight loss”.ti; “body weight”.ti; “birth weight”.ti; “low weight”.ti., integrative.ti., “high-grade”.ti.] The search was performed in MEDLINE (Ovid) and identified 704 citations.
Furthermore, we elicited input from ten key in


.
 4
Objective
The report aims to provide resources for systematic reviews in occupational safety and health. It provides an overview of general steps in systematic reviews with a particular focus on occupational safety and health-relevant topics.
Th


ture flow with reasons for exclusion). Hence, the aspects of the recommendations that should be based on a systematic literature search should be determined in advance with resources and utility in mind.
Ideally, a full systematic review should be co


the database since March 2015. The entries are also indexed in PubMed Health, a publicly available database maintained by the U.S. National Library of Medicine (NLM). Another key resource is the collection of existing systematic reviews registered wi


Review Questions
As outlined at the beginning of this chapter, for the systematic review, it is very useful to formulate specific review questions. The review questions will determine the purpose and refine the scope of the systematic review. A key p


actice for many topic areas. Key informant interviews should reflect the perspectives of those who would make a decision based on the subsequent recommendations as well as those who would be affected by those decisions (AHRQ, 2014). An expert panel s


en limiting the technical expert panel to nine members. To reduce the workload, panel members should be able to restrict their input to areas in which they feel qualified.
Identifying stakeholders that may help refine the systematic review question(s


section and the methods of the systematic review:
• Rationale
− should describe the review in the context of what is already known.
• Objectives
− should address explicit review questions.
• Eligibility criteria
− should cover inclusion and exclusion


dards for systematic reviews (Eden et al., 2011).
While the PRISMA statement covers essential elements expected and often required by scientific journals and repositories, published (publicly available) protocols may include additional elements and s


teristic and is not part of the systematic review (e.g., the review does not summarize the research on effects of exposure as such). The mechanism of action linking the exposure to outcomes is shown in the framework
  13
to help the reader understand


ys (Ankley et al., 2010). It may also draw on the International Agency for Research on Cancer framework to assess the strength of the available evidence that an agent could alter the incidence of cancer in humans, which combines exposure data, cancer


s. Multiple sources may be different electronic databases (e.g., PubMed and PsycINFO) that cover the topic of interest. Other sources may be reference mining (e.g., systematically searching the bibliographic references of pertinent reviews and studie


methods. The 11 items in the tool were derived by applying a large list of quality items to an empirical sample of reviews and then applying principal-component analysis to identify underlying factors. The items cover an a priori design (question and


terature published after a certain benchmark study). Considering the different audiences and consumers of the subsequent guidelines document, not all users of the recommendation issued by NIOSH or other agencies may be interested in the full methodol


them particularly well suited to respond to questions regarding the evidence in a timely manner (Ganann, Ciliska, and Thomas, 2010). Timely products are especially important when developments require that advice is available quickly, for example, for


The criteria for using rapid-review methodology in the given examples included the need to synthesize evidence in a timely manner to inform emergent decisions faced by decisionmakers because of the nature of the review topic, limited resources that


, and Thomas, 2010, and Tricco et al., 2015) outlined methods that allowed for faster review production. Methods focused on the amount of literature to be reviewed, while other rapid review teams applied strict inclusion criteria to the retrieved sea


ng.
− Apply strict inclusion criteria to the publication type (e.g., peer-reviewed research).
• Document and assess included studies of scalability options:
− Minimize data extraction (number of variables).
− Minimize coding/abstraction (e.g., using


o establish the baseline risk in the study population, the relative frequency of the outcome cannot be determined. Determining which studies should be eligible for inclusion should be discussed in sufficient detail to ensure that the most-appropriate


a extraction is prone to errors, or can one reviewer be trusted to do it? How difficult is it to assess the quality of the studies (do two independent reviewers come to the same conclusion)? Is the quality of evidence assessment likely to be controve


ical literature citations. The continuously expanding NLM platform includes a number of other smaller collections (e.g., PubChem Substance, a database for chemical structure information). Other large databases include the Web of Science, Scopus (broa


published by the Campbell Collaboration. This information would be missed when limiting the search to major electronic databases targeting traditional academic publishing channels. There are other aspects to be considered, but one is the ability to s


blished data have to be citable and released upon request. Contacting known researchers for additional, unpublished data may also be an option. Response success is limited and depends on several factors.
Caveats for including unpublished data and gre


ack publications associated with the study.
A way to quickly locate the most relevant studies is to ask content experts. This can be very useful when setting up the systematic review. The identified articles can be used to validate the search strateg


his step is useful even if an information specialist executes the search.
Figure 4.1. Search Venn Diagram
For each component, key terms and synonyms (e.g., harm, adverse event, adverse effect, safety risk, side effect) need to be identified. The scop


ccupational health in 1991. However, this tag has been shown to be insufficient in identifying specific occupational safety and health research such as occupational health intervention studies (Verbeek and Van Dijk, 2006). A key limitation of relying


state until which date existing literature has been reviewed.
The purpose of developing a search strategy is to identify all pertinent citations. Searches can be evaluated by recall and retrieval rates (e.g., how many relevant studies are included in


tion that will be included in the review. However, technically, these are eligibility criteria. Eligibility criteria define what publications need to report in order to be eligible for inclusion in the review. It needs to be made transparent which st


can be an intervention or an exposure. The eligible intervention(s) and exposure(s) need to be described in detail. It may be necessary to define eligible interventions/exposures or list components that define the interventions/exposure (e.g., interv


can make the literature findings more applicable to current working conditions, even when the effective savings in reviewable literature is limited (see discussion in previous section). Ideally, the eligible data timing should be linked to a benchmar