to ensure that all relevant studies have been
33
reviewed, and what is relevant has been determined before the results of the included research studies were known to the reviewers.
Flow Diagram
Documentation of the application of eligibility criteria


n that the extraction process is labor intensive and prone to errors (which requires additional steps to ensure accuracy). Data extraction in systematic reviews is a means to an end; it is not a final product. The data-extraction form and the evidenc


analysis), differences in extracted detail (e.g., point estimate together with confidence intervals or p values), and differences in extracted information (e.g., when deciding which result is important) to a minimum. A mock-up of an evidence table f


ias assessment, the results of the assessment may either be reported in a separate table or incorporated, at least in summary format, in the evidence table.
As discussed in the previous section of this chapter, there is a trade-off between presenting


 the assessment to increase transparency.
Critical appraisal may focus on the internal and external validity of the study.
Internal Validity
Internal validity refers to the conduct, internal logic, and methodological rigor of a research study and ass


− Attrition bias addresses systematic differences in the loss of participants (e.g., incomplete follow-up and differential attrition in the study and control groups); drop- offs may be systematically different from study completers. Intention to trea


rventions and the Comparative Effective Rules Methods Guide provide resources for critical appraisal for a broad range of study types.
External Validity
External validity concerns the generalizability of results shown in individual study samples. It


ome focuses on how closely linked the study outcome is to the outcome of interest (see discussion about intermediate outcomes in the “Analytic Framework” section in Chapter Three). The assessment of the external validity of the timing may assess the


s risk of bias criteria to human and nonhuman studies separately. For human studies, these include (1) recruitment strategy,
(2) blinding, (3) exposure assessment, (4) confounding, (5) incomplete outcome data, (6) selective outcome reporting, (7) con


rnatively, the critical appraisal assessment can be documented across all included studies. Figure 5.1 is an example showing the proportion of studies with a high risk of bias, unclear risk of bias, or low risk of bias for ten selected risk-of-bias c


ented to increase transparency for the reader of the systematic review.
Data Access
Increasingly, scientific journals and government agencies require that the data used in research studies are made accessible to the reader. This requirement also appl


amework for Rating Evidence in Public Health (PRECEPT) suggests grouping evidence first based on four general domains (disease burden/level of contamination, risk factors, diagnostics, and interventions questions) (Harder et al., 2015). Within domain


sic meta-analytic techniques, and the user will be prompted by a program wizard. It can be used, for example, to compute rates or the mean and standard deviation for two groups across studies to arrive at a summary estimate. It computes a range of ef


hod to structure data across studies.
Other Approaches to Summarize Results Across Studies
Other ways to numerically summarize results across studies include the reporting of the range of study results. Computing the mean or the median of the metric


n particular when sample sizes are small and results are only based on a handful of participants. The table should summarize the results, ideally both in terms of the direction of effect and the size of the effect. The table is organized by outcomes.


tatement can be graded. This section differentiates criteria to evaluate a body of evidence, the starting point for evaluations, and the level of quality of evidence.
49
GRADE
 Criteria to Evaluate a Body of Evidence
A 2002 systematic review of syste


e varies by exposure dose). In that case, separate evidence statements for the different scenarios may be warranted.
50
• Indirectness
− Evidence can be indirect because study participants may differ from those of interest in the review, the interven


idual confounding
− This domain considers whether we can be confident that confounding would reduce, rather than explain, the reported effect. Or when no effect was observed, that all plausible residual confounding would suggest a spurious effect.
Of


ystematic review, and the criteria may need to be adapted according to the evaluated evidence. Apart from adaptations of individual grading criteria, there are also body-of-evidence evaluation adaptations that use different sets of criteria, or addit


emonstrated effect or would suggest a spurious effect if no effect was observed; and use of theory/logic models, assessment of contexts, reporting of implementation process, and fidelity of implementation. In addition, for observational studies, the


the grading system.
Evidence Grading Starting-Point Adaptations
Several research and policy areas have opted to revise the trial-centered starting point in body-of- evidence evaluations. Van Staa and colleagues (2008) make a strong case that, when ev


substantially
different from the estimate of the effect. • Very low quality of evidence
− We have very little confidence in the effect estimate: The true effect is likely to be substantially different from the estimate of effect.
In contrast, the AHR


factor, to arrive at the summary. In the standard GRADE approach, quality of evidence can be downgraded by one (e.g., from high to moderate) or by two (e.g., from high to low), depending on the extent to which the issue affects the confidence in the


e or in the presence or lack of research in some lines of evidence. An example is in vitro studies indicating the potential for serious adverse effects on humans, while no human-participant study exists that has empirically demonstrated a risk to hum


works resulted in the formulation of 17 best practices for integrating and evaluating evidence (Rhomberg et al., 2013). Best practices relevant to different lines of evidence include:
• Assess all human and nonhuman data relevant to mode of action, t


ine factors should be considered a definitive interpretation of cause and effect, and none were required to be present for cause and effect to be determined. However, he believed that together, they could be used to help decide whether any other inte


 to humans.
− Group 3 applies to bodies of evidence where evidence is inadequate in humans and
inadequate or limited in experimental animals.
• Group 4: The agent is probably not carcinogenic to humans.
− Group 4 is used for agents for which there is


integrate results across outcomes and lines of evidence. The methodology involves seven steps: (1) formulate the problem and develop protocol, (2) search for and select studies for inclusion, (3) extract data from studies, (4) assess internal validi


concentrate on the evidence for preventative mechanisms. The appropriate level of differentiation will depend on the scope of the systematic review and the recommendations. The level of differentiation and the level of detail that will be used to ans


vels (e.g., high, moderate, low, insufficient quality of evidence). The approach should be outlined in the systematic review protocol and not be decided after research results have been reviewed. Furthermore, given the subjectivity involved in the pr


tudy eligibility criteria, appraisal and synthesis methods, results, limitations, conclusions, implications, and the PROSPERO registration number.
• Introduction
− rationale (in the context of what is already known)
− objective (using the PICO framew


. Elements such as the flow diagram, evidence table, and summary of findings table are designed to provide a clear overview to summarize the available evidence. A concise synthesis is of particular importance when the systematic review is supporting


de effects known in clinical intervention research. However, side effects of
 66
recommendations could be social or socioeconomic in nature. For example, recommended safety procedures will be associated with required time and resources and may have c


it had been omitted from the evidence summaries or was new and critical for decisionmaking, the GRADE approach was to be used to grade the quality of evidence and the strength of recommendations, recommendations were to be based on a consensus of the


s are asked to individually rank order or rate their judgment. Summary estimates of the effects of alternatives can be represented as a balance sheet depicting benefits and harms for each strategy and can help illustrate the trade offs (Berg, Atkins,


lance between desirable and undesirable effects, values and preferences, and costs or resource allocation needed (Guyatt et al., 2008). Factors that could weaken the strength of a recommendation in the WHO Rapid Advice Guidelines (Schünemann et al.,


different standards and preferences for presenting conclusions and recommendations and linking the evidence and recommendations. A study surveying occupational exposure limits for metals and other mining-related chemicals identified a number of publi


posure groups (Schünemann et al. 2007a). The exact wording of the recommendations and the implications for the end user have to be critically reviewed. This is particularly important for occupational safety and health applications that do not repres


commendations. While several guideline documents exist that can generally guide systematic reviews in occupational safety and health research, we identified several unique requirements, such as the integration of different lines of evidence that woul


onstantly published, the new studies may either confirm the systematic review conclusions or challenge them.
Determining whether a systematic review is out of date presents a key challenge for producers of systematic reviews. The AHRQ EPC program use


 System to Assess the Need for Updating Systematic Reviews,” Systematic Review, Vol. 2, 2013, p. 104.
AHRQ—See Agency for Healthcare Research and Quality.
Alli, Benjamin O., Fundamental Principles of Occupational Health And Safety, 2nd ed., Geneva,
S


t S. Thompson, Steven H. Woolf, Vilma G. Carande-Kulis, Laurie Anderson, Alan R. Hinman, David V. McQueen, Steven M. Teutsch, and Jeffrey R. Harris, “Developing an Evidence-Based Guide to Community Preventive Services—Methods,” American Journal of Pr


ovember 12, 1994, pp. 1286–1291.
Dyrvig, Anne-Kristine, Kristian Kidholm, Oke Gerke, and Hindrik Vondeling, “Checklists for External Validity: A Systematic Review,” Journal of Evaluation in Clinical Practice,
Vol. 20, No. 6, December 2014, pp. 857–86


 Liberati, and Holger J. Schünemann, “Going from Evidence to Recommendations,” British Medical Journal, Vol. 336, No. 7652, May 2008, pp. 1049– 1051.
Haber, Lynne T., and Andrew Maier, “Scientific Criteria Used for the Development of Occupational Ex


linical Epidemiology, Vol. 66, No. 9, September 2013, pp. 973–981.
Hempel, Susanne, Melinda Maggard-Gibbons, David K. Nguyen, Aaron J. Dawes, Isomi Miake- Lye, Jessica M. Beroes, Marika J. Booth, Jeremy N. V. Miles, Roberta Shanman, and Paul G. Sheke


Research Methodology, Vol. 11, 2011, p. 23.
Horsley, Tanya, Orvie Dingwall, and Margaret Sampson, “Checking Reference Lists to Find Additional Studies for Systematic Reviews,” Cochrane Database of Systematic Reviews, Vol. 8, August 2011, p. Mr000026.


Sara, Kristin Konnyu, Rob Cushman, Jeremy Grimshaw, and David Moher, “Evidence Summaries: The Evolution of a Rapid Review Approach,” Systematic Reviews, Vol. 1, No. 10, 2012, p. 10.
Khodyakov, Dmitry, Susanne Hempel, Lisa Rubenstein, Paul Shekelle, R


onal Exposure Limit for Inorganic Borates Using a Weight of Evidence Approach,” Regulatory Toxicology and Pharmacology, Vol. 68, No. 3, 2014, pp. 424–437.
82
Matthews, Edwin J., Naomi L. Kruhlak, R. Daniel Benz, and Joseph F. Contrera, “A Comprehensi


ce Evaluation of the Human Carcinogenicity of Toluene Diisocyanate,” Critical Reviews in Toxicology, Vol. 43, No. 5, 2013, pp. 391–435.
Rhomberg, Lorenz. R., Lisa A. Bailey, and Julie E. Goodman, “Hypothesis-Based Weight of Evidence: A Tool for Evalu


 7, No. 1, January 2007a, pp. 21–31.
Schünemann, Holger J., Suzanne R. Hill, Meetali Kakad, Gunn E. Vist, Richard Bellamy, Lauren Stockman, Torbjørn Fosen Wisløff, Chris Del Mar, Frederick Hayden, Timothy M. Uyeki, Jeremy Farrar, Yazdan Yazdanpanah,


Davidson, “Building Better Guidelines with BRIDGE-Wiz: Development and Evaluation of a Software Assistant to Promote Clarity, Transparency, and Implementability,” Journal of the American Medical Informatics Association, Vol. 19, No. 1, January–Februa


No. 7, Geneva, Switzerland: World Health Organization, 2006.
Verbeek, J., J. Salmi, I. Pasternack, M. Jauhiainen, I. Laamanen, F. Schaafsma, C. Hulshof, and F. van Dijk, “A Search Strategy for Occupational Health Intervention Studies,” Occupational a


hrane Database of Systematic Reviews, No. 11, 2011.
Zaza, Stephanie, Linda K. Wright–De Agüero, Peter A. Briss, Benedict I. Truman, David P. Hopkins, Michael H. Hennessy, Daniel M. Sosin, Laurie Anderson, Vilma Carande-Kulis, Steven M. Teutsch, and


  CORPORATION
Los Angeles County
Juvenile Justice Crime
Prevention Act
Fiscal Year 2014–2015 Report Terry Fain, Susan Turner, Mauri Matsuda
For more information on this publication, visit www.rand.org/t/RR1458
  Published by the RAND Corporation, San


evaluation of the county’s JJCPA programs, including analyzing data and reporting findings to the BSCC. This report summarizes the fiscal year 2014–2015 findings reported to the BSCC, as well as additional program informa- tion gathered by the Los An


Santa Monica, Calif.: RAND Corpora- tion, TR-498-LACPD, 2007
• Susan Turner, Terry Fain, John MacDonald, and Amber Sehgal, with Jitahadi Imara, Felicia Cotton, Davida Davies, and Apryl Harris, Los Angeles County Juvenile Justice Crime Prevention Act:


 Participants........................................................................................
ix
x Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
S.2. Programs in the Three JJCPA FY 2014–2015 Initiative


 . . . . . . . . . . . . . . . . . . . . . . . . .
2.9. Means, Differences in Differences, Odds Ratios, and Confidence Intervals for Outcomes for Youth Substance Abuse Intervention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .


Justice Costs for Gender-Specific Community. . . . . . . . . . . . . . . . . . . . . . . . .
3.6. Estimated Juvenile Justice Costs for High Risk/High Need . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.7. Estimated Juvenile Justice


ogram to provide a stable funding source for juvenile programs that have proven effective in curbing crime among at-risk youths2 and young offenders (BSCC, 2016). The legislature asked counties to submit plans to the state for funding to identify pro


zes. For programs that used a pre–post evaluation, we used McNemar’s test to determine significance for arrests and incarcerations. For pre–post comparisons of secondary outcomes, such as risk and strength scores, we used a difference-of- means test


nt, working with BOC (and later with the Corrections Standards Authority and the BSCC), defined supplemental outcomes specific to each program, which it also reports to the BSCC annually.
Some discussion of the big six is in order. The BSCC does not


initia- tive and from multiple programs, within or across initiatives, and concurrently or consecu- tively. Probation counts a given juvenile as a participant within each program from which he or she receives services and could therefore count that j


–2014 SBMS-AR participants Routine probationers
Summary xvii
  MH MST SNC
1,081 FY 2013–2014 MH participants 68 MST-identified near misses 40 SNC-identified near misses
 NOTE: We limited near misses for MST and SNC to those with characteristics compa


a positive outcome. Using the previous year’s cohort as a compar- ison group for this year’s program youths was suggested by the BSCC. We do not deter- mine whether a given outcome is “good” or “bad.” We simply report whether a pre–post comparison s


For the most part, difference-in-differences analyses supported simple comparisons
between groups.
• With the exception of SBHS-PROB and SBMS-PROB, programs that used contempo-
raneous comparison groups were small and showed no significant difference


le. Echoing the results for MH participants, the two groups did not differ significantly on any of the big six outcomes. The difference-in-differences analyses for MH also found no significant differences between the two groups on any of the big six


owing program entry over that of the previous term. School suspensions and expulsions also dropped, although the differences were not always statistically significant. Among participants in the school-based programs, test scores were significantly hi


in-Differences Analyses
For seven Los Angeles County JJCPA programs (GSCOMM, HRHN, IOW, MH, SBHS-AR, SBMS-AR, and YSA), the county evaluates outcomes by comparing the current cohort’s results and those of the previous year’s cohort, with the goal of


eas the Enhanced Services to High-Risk/High-Need Youths initiative cost $2,800 per participant served, and the Enhanced School- and Community-Based Services initiative spent $725 per participant. Differences between initiatives in estimated mean cost


were in the MH program.
As one might expect, mean overall juvenile justice costs for JJCPA participants were generally higher in the six months after program entry ($11,436) than in the six months prior to program entry ($8,598), primarily because of


an net cost for each juvenile justice cost—i.e., the mean difference between the cost in the six months before entering the program and the six months after entering. As one might expect, mean costs differ noticeably among the three initiatives. The


.
Another limitation of this report is that, although we can determine statistical signifi- cance for a given outcome, we have no way to judge the raw numbers as “good” or “bad.”
Probation extracted data used to compute outcome measures from its data


n to modify programs based on the integration of evaluation findings and
Summary xxvii
xxviii Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
effective juvenile justice practices. Differences in outcomes between


ne or more JJCPA programs according to an assessment of that juvenile’s need for services.
1 In earlier years, a few JJCPA programs also targeted youths in juvenile camps but now have only two programs admin- istered within juvenile halls and none in


n youths in reference groups. Program and comparison-group youths did not differ significantly in their rates of completion of probation, completion of restitution, completion of community service, or probation violations (BSCC, 2016).
JJCPA in the C


l as on a comparison group for each group of program partici- pants, which it uses as a reference for measuring program success.
The most preferable research design is experimental, in which researchers randomly assign participants to either a treatm


gns to BOC that used quasi-experimental methods. BOC subsequently approved these designs. Whenever pos- sible, comparison groups included youths with characteristics similar to those of program participants—either routine probationers, probationers i


 outcome measures, extracted them from the on-site database, and sent them to RAND for analysis. Appendix A provides additional details on construction of the comparison groups.
RAND researchers verified the comparability of comparison groups for SBH


groups for MST were youths who had similar characteristics to program partici- pants but who were not accepted into the program, usually because of lack of Medi-Cal coverage needed to cover the cost of program participation or because they were rece


rom the Los Angeles County Department of Mental Health (DMH), Probation, the courts, and law enforcement. This report focuses not on the perfor- mance of individual CBOs or individual county agencies in providing services to JJCPA pro- grams but on t


e that the current year’s cohort had a more favorable outcome (i.e., improved more), while an upper bound of the CI less than 1 indicates a less favorable outcome (i.e., improved less).
In our discussion of outcomes for all of the programs that use t


ails of outcomes for each program
• Appendix E: details of outcomes for each program, by participant gender
• Appendix F: details of outcomes for each program, by cluster. Los Angeles County admin-
isters probation in five areas called clusters, whic


 FY 2014–2015 and the number of participants who received services in each program. Table 2.2 shows the number of par-
1 A given youth can participate in more than one JJCPA program, and a single youth can participate in the same program more than on


 them for the program in previous years, severe enough after SNC changed its qualifica- tion criteria.
 MH MST SNC
1,081 FY 2013–2014 MH participants 68 MST-identified near misses 40 SNC-identified near misses
II. Enhanced Services to High-Risk/High-


 lists the programs that constitute the Enhanced Mental Health Ser- vices initiative, along with a description of the comparison group for each program.
Table 2.3
JJCPA Programs and Comparison Groups in the Enhanced Mental Health Services Initiative


very small cell sizes.
 Current JJCPA Programs and FY 2014–2015 Outcome Measures 13
On entry into juvenile hall, DMH professional staff screen detained minors. The staff employ the Massachusetts Youth Screening Instrument and a structured interview.


ased on research available at that time. Whenever possible, we have attempted to supplement these older research reports with more-recent research findings. We have not removed the older citations, however, because they form the original evidence bas


from those of the previous cohort, and a difference-in-differences analysis shows that there is not a huge change between baseline and follow-up rates for either cohort. The stated goal of the program is that the current year’s cohort perform at lea


asures.
 16 Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
Table 2.4
Means, Differences in Differences, Odds Ratios, and Confidence Intervals for Outcomes for Mental Health
    Outcome Baseline
Arrest 45.79
Inc


stently with social-ecological models of behavior and findings from causal modeling studies of delinquency and drug use, MST posits that multiple factors determine youth antiso-
Current JJCPA Programs and FY 2014–2015 Outcome Measures 17
cial behavio


between the program and comparison-group youths, especially with such small samples. The two groups had similar gender distributions, with male participants making up 77.9 percent of the MST participants and 77.8 percent of the comparison group. Mean


bation Department and the court
• comprehensive mental health screening and evaluation by a multidisciplinary team
• an individualized mental health treatment plan
• court- and Probation-monitored case-management processes.
Evidence Base for the Prog


he juvenile drug court model was soon generalized to address concerns other than drug use. The goals of juvenile courts are to do the following:
• Provide immediate intervention, treatment, and structure in the lives of juveniles through ongoing, act


79.0a 21.0a
29.0 7.9 63.2 0.0
 Mean age (years) 15.6 Gender (%)
Male 57.5a
Female 42.5a Race and ethnicity (%)
Black 20.0 White 15.0 Hispanic 65.0 Other 0.0
SOURCE: Analysis of data from Probation’s database.
NOTE: Percentages might not sum to 100 b


e highest need for services. Programs and services in this initiative are the GSCOMM, HRHN, and YSA programs. Table 2.6 lists the programs in this initiative and briefly describes the comparison group for each program.
 Current JJCPA Programs and FY


vices are consistent with the Office of Juvenile Justice and Delinquency Prevention’s (OJJDP’s) gender-specific programming and principles of prevention, early intervention, and aftercare services (Greene, Peters, and Associates and Northwest Regiona


practitioners called the Girls Study Group, with the specific purpose of understanding and responding to delin- quency among female juveniles. This group subsequently published findings that both sup- ported and expanded on the earlier OJJDP work on


ichever came first (30.3). Figure 2.4
26 Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
presents big six outcomes, with details for all outcomes shown in Table D.4 in Appendix D. Cluster and gender data were no


ioral skill training across domains—family, peer, school, and neighborhood) and incor- porates the phase model of Functional Family Therapy (FFT). Additionally, such programs as MST and multidimensional-treatment foster care (MTFC) place a strong emp


 with prosocial peers (Henggeler et al., 1998).
• FFT: FFT is a family-based prevention and intervention program that has been applied successfully in a variety of contexts to treat a range of these high-risk youths and their families. It was develop


rvices be available not only on weekdays but also in the evenings and on weekends.
– a balance of incentives and graduated consequences. Intensive supervision is likely to uncover numerous technical violations and program infractions. The IAP model i


nts’ responsibility and leadership skills, creating opportunities that are age and stage appropriate, and building a sense of self and group.
The HRHN employment components are based on many of the design elements in JOB- START and the recommended pr


 2.5
Outcomes for High Risk/High Need, FY 2014–2015
NOTE: A star indicates a statistically significant difference (p < 0.05) between the two groups.
Difference-in-Differences Analyses
As with all JJCPA programs that used the previous year’s cohorts a


nt provider has access to inpatient services as needed.
Program goals are to reduce crime and antisocial behavior and reduce the number of participants with positive drug tests. YSA providers work collaboratively with school-based DPOs in developing


g family substance use and deviant peer networks. Programs must also employ empirically validated therapies and address co-occurring conditions, such as learning disabilities and other mental health disorders.
YSA’s approach incorporates many of the


percent CIs is less than 1 and the upper bound is greater than 1, we conclude that the two cohorts did not differ significantly. We could not compute odds ratios for successful completion of probation or successful comple- tion of community service b


ly at baseline in arrest rates, so a difference-in-differences analysis indicated that the change from baseline to follow-up for the FY 2014–2015 cohort was significantly greater than for the FY 2013–2014 cohort. For the YSA program, FY 2014–2015 par


ists the programs in this initiative and briefly describes the comparison group for each program.
Current JJCPA Programs and FY 2014–2015 Outcome Measures 37
Table 2.10
Programs and Comparison Groups in the Enhanced School- and Community-Based Servic


a difference-of-means test to evaluate statistical significance.
       38 Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
ensure that youths who are at risk of truancy or excessive absences attend school. The p


riate corrective action. The OJJDP bulletin on the Juvenile Accountability Block Grants program (Gramckow and Tompkins, 1999) cites the ACT pro- gram and presents it as one model of an approach and program that holds juvenile offenders accountable fo


y services to probationers and at-risk youths
• increase school attendance and performance
• reduce crime rates in the housing units.
The HB program places DPOs at selected public housing developments to provide day services and supervision for proba


FT and MST, the HB program places emphasis on
• identifying factors in the adolescent’s and family’s social networks that are linked with antisocial behavior
• developing and reinforcing family strengths
• intervening with delinquent peer groups thro


ram. The participants meet weekly, in sessions that professional writers lead, to write and critique their written work with others in the group. The program guides participants both in their writing and in their discussion of their written work, pro


uth during incarceration, may improve their attitudes towards reading, and influences academic and vocational outcomes fol- lowing incarceration” (p. 4).
Comparison Group and Reference Period
The comparison group for the current year’s IOW participan


.608 1.809 0.975–3.354
1.226 0.788–1.910 1.855 0.819–4.203
1.225 0.899–1.669
      the previous year to the current year. A negative value in
County of Los Angeles Department of Parks and Recreation and City of Los Angeles Depart- ment of Recreation


 will become serious. However, reducing risk factors and enhancing protective factors, such as positive social orientation, prosocial bonding, and clear and positive standards of behavior, can delay or pre- vent delinquency (OJJDP, 1995).
Communities


ity safety. Participants include probationers and at-risk youths in 85 school service areas whom school-based DPOs accept into the program. These DPOs are assigned and placed on school campuses with a focus on monitoring school attendance, behavior,


 the LARRC (Turner, Fain, and Sehgal,
Current JJCPA Programs and FY 2014–2015 Outcome Measures 47
2005b; Turner and Fain, 2006). The LARRC is based on the what-works research. Further, school-based DPOs enhance strength-based training, including trai


(p < 0.05).
For both SBHS-AR participants and comparison-group youths, we measured big six outcomes during the six months following entry into the program. For supplemental school outcomes—attendance, suspensions, and expulsions—we compared program p


School At-Risk Youths
SBHS-AR uses program participants from the previous year as a comparison group, so we have included difference-in-differences analyses for this program. For arrest and incarceration outcomes in the SBHS-AR program, Table 2.13 sh


i is the weight for each individual and the sum is across all individuals in the group.
w2 i
52
Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
Table 2.14
Factors Used to Match School-Based Probation Supervision


x D.
Figure 2.10
Outcomes for School-Based Probation Supervision for High School Probationers, FY 2014–2015
NOTE: A star indicates a statistically significant difference (p < 0.05) between the two groups.
As we noted in Chapter One, Los Angeles Count


ion Supervision for Middle School At-Risk Youths in FY 2014–2015 and Those in FY 2013–2014
 Factor
FY 2014–2015
FY 2013–2014
12.4 60.3
16.9 2.1 73.9 7.2
28.6 18.8a 26.0 16.9a
9.7
 Mean age (years) Male (%)
Race or ethnicity (%)
12.5 58.7
Black 14.0 W


- Risk Youths, by Cluster, FY 2014–2015
 Current JJCPA Programs and FY 2014–2015 Outcome Measures 59
Difference-in-Differences Analyses for School-Based Probation Supervision for Middle School At-Risk Youths
We include difference-in-differences analy


served feature might
20 We used the statistical technique of propensity-score weighting to obtain weights for comparison-group youths so that their characteristics matched those of the program participants. We included only probationers with valid da


of restitution. As in the SBHS-PROB program, comparison-group youths had lower rates of probation violations (3.6 percent) than SBMS-PROB participants
Current JJCPA Programs and FY 2014–2015 Outcome Measures 61
(13.5 percent). For big six outcomes, s


tal outcomes as well.
Three of the programs in this initiative—IOW, SBHS-AR, and SBMS-AR—used the previous year’s program participants as comparison groups. In all three programs, difference- in-differences analyses agreed with a simple comparison of


ed within juvenile halls, we measure costs during the six months prior to hall entry and six months following hall exit for the hall stay during which program services were received.
  65
66 Los Angeles County Juvenile Justice Crime Prevention Act: F


ed that a sheriff’s department juvenile arrest cost $2,082.72, calculated as 4.5 hours of deputy generalists at $128.18 per hour and 4.5 hours of a deputy’s time at $135.39 per hour for arrest, report writing, and transport; 4.5 hours of a deputy’s t


loyee benefit increases. Additionally, the daily populations of halls and camps have decreased sig- nificantly without commensurate decreases in costs, thereby resulting in increased costs per probationer (Bryant, 2015).
Program Cost
We calculated th


 Fiscal Year 2014–2015 Report
We also assume that program costs in the six months before someone enters a program are 0. This is a deliberately conservative estimate because participants might have actually received other services during that period,


uced savings ($1,427 per individual). These savings were not enough to offset the very high program costs— the highest per capita program cost of any Los Angeles County JJCPA program—as well as increased supervision and court costs in the follow-up.


   Juvenile
Justice Unit Cost Cost ($)
Arrest 2,114.74
Camp 677.37
Court 3,394.00
Juvenile 770.76 hall
Program 42.97 Supervision 7.43 Total
Unit Units
Arrest 0.54 Day 2.18 Appear. 0.99 Day 3.98
Day 0.00 Day 106.54
Baseline
Median Mean ($) ($)
1,152 0


sitive number in a difference column indicates that the cost was lower in the six months after beginning the program than in the six months before beginning. A negative number indicates that the cost was higher after entering the program than before


raged $21,425 in the baseline and $25,375 in the follow-up, a difference of $3,950 per participant.
    Baseline
Median Mean ($) ($)
1,578 2,115 2,597 0 4,055 3,394
12,362 771
Program 0.52
Supervision 7.43
Total 21,425 8,173
Follow-Up
Mean ($)
1,111


r baseline or follow-up, and costs for all other components were slightly higher in the follow-up than in the baseline period. Mean gain in school attendance ($388 per youth) was not enough to offset all the other costs, resulting in an overall mean


cost was higher after entering the program than before entering. Var. = Variable. Appear. = Appearance.
was lower in the six months after
Estimated Juvenile Justice Costs for JJCPA Participants 83
Estimated Costs for School-Based Probation Supervisio


.47 1,608 4.17 3,215
Follow-Up
Mean ($)
529 0 627 2,049
1,307 597 – 433 4,719
Difference ($)
   Juvenile Unit Cost
Justice Cost ($) Unit
Arrest 2,114.74 Arrest
Camp 677.37 Day
Court 3,394.00 Appear.
Juvenile 770.76 Day hall
Program 7.51 Day
Supervisi


 the “Difference of Means, in Dollars” column indicates that the mean cost was lower in the six months after beginning the program
  Mean Estimated Cost per Participant, Participants Served, and Cost Differences, by JJCPA Program, FY 2014–2015
Table


2 –923
Estimated Juvenile Justice Costs for JJCPA Participants 87
Estimated Juvenile Justice Cost Savings, by Initiative
For each of the three FY 2014–2015 initiatives, Table 3.17 shows the estimated mean net cost for each juvenile justice cost—i.e.,


 youths, such as IOW and MH, will always have
88 Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
relatively high supervision costs, making these programs look worse on these cost comparisons for supervision. Arr


e halls (MH and IOW), we measure outcomes and costs in the six months prior to hall entry and in the six months following release from the hall.
  89
90
Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
• • •
•
•


erences except for pre–post improvement in school attendance for MST participants. Primarily because of the smallness of samples, changes in all other supplemental outcomes were not statistically significant.
None of the three programs in the Enhance


nificantly different for the two groups. For the programs that used educational measures as supplemen- tal outcomes, school attendance improved significantly in the term following program entry compared with the previous term. School suspensions and


n the GSCOMM, HRHN, and IOW programs had positive results for supplemental outcomes.
The three programs that utilized pre–post comparison designs—ACT, HB, and PARKS— primarily targeted at-risk youths, so the only reportable big six outcomes were arre


 — —
 this table indicates that the FY 2014–2015 cohort had a significantly more positive result.
NOTE: FY 2014–2015 in
A dash indicates no significant difference between the two cohorts. n.a. = not applicable.
Among the programs that used the previo


l, there is a consistent pattern of program participants meet- ing program goals.
Supplemental outcomes also show very similar results from year to year, with almost all follow-up measures significantly more positive than baseline measures. However,


ll costs in the follow-up period than in the baseline period.
Limitations of This Evaluation
Comparison-Group Youths Versus Program Participants
As with any evaluation, our assessment of the JJCPA program in Los Angeles County has some inherent limit


ine and a follow-up evaluation.
We also did not have access to how certain scales used for supplemental outcomes (e.g., strength, risk, and barrier scores for the school-based programs, and family functioning for HRHN) were constructed or to the just


lt of two trends: (1) a steady decline in juvenile arrest rates since 2007 (Office of the Attorney General, undated [b]) and (2) the Los
Angeles County Probation Department’s deliberate strategy of devoting an increasing number of resources to at-ris


signed a gang-avoidance order.
We calculate propensity-score weights by performing a logistical regression to predict whether a given youth is in the treatment group or the comparison group. The independent variables are those on which we will match


s result in sustained petitions by the court. Therefore, Probation considers arrest an important indicator with this caveat and qualifier.
3. violation of probation: As with arrests, violations are a key indicator of recidivism and delinquency. Howev


rvices for Groups
Star View Children and Family Services Tarzana Treatment Centers
Primary Service Offered
Gender-specific services HRHN employment services Substance abuse treatment MST
Substance abuse treatment MST
MST
Substance abuse treatment Sub


omparison group consists of youths who qualified for MST in FY 2012–2013, FY 2013–2014, or
FY 2014–2015 but did not participate in the program and were agreed on by MST staff, Probation Department staff, and RAND staff. The MST team identified these


onths after that or when the youth exited the program, whichever came first.
a Difference is statistically significant (p < 0.05).
108 Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2014–2015 Report
Table D.5
Outcomes for High


cy, FY 2014–2015
     BSCC-Mandated Outcome
Arrest
Incarceration
Completion of probation
Completion of restitution
Completion of community service
Probation violation
BSCC Supplemental Outcome
School absences
Baseline
Number Percentage
18 0.34a 0 0.0


Size
0.12a 1,761
       NOTE: The comparison group consists of all program participants whose outcomes the program reported for the previous fiscal year (FY 2013–2014). We measured mandated outcomes during the six months after the youth exited juveni


ntered the program and for the first complete academic period after the youth entered the program. We measured strength and barrier outcomes when the youth entered the program and six months after the youth entered the program or when the youth exite


17.79a 598 4.37a 598
             NOTE: The comparison group consists of all program participants whose outcomes we reported for the previous fiscal year (FY 2013–2014). We measured mandated outcomes during the six months after the youth entered the


ipants
  BSCC-Mandated Outcome
Arrest
Incarceration
Completion of probation
Completion of restitution
Completion of community service
Probation violation
Number Percentage
5 33.33 1 6.67 2 16.67 3 42.86 2 20.00
0 0.00
Sample Size
15 15 12
7 10
12
Num


the genders of six participants in this program. n.a. = not applicable.
Table E.7
Outcomes for School-Based Probation Supervision for Middle School Probationers, FY 2014–2015
   Female Participants
Male Participants
  BSCC-Mandated Outcome
Arrest
Inc


omes for School-Based Probation Supervision for Middle School At-Risk Youths, FY 2014–2015
      Cluster
12345
          Sample Sample Sample
% Size % Size % Size %
Arrest
Incarceration 0.75
Completion of n.a. probation
Completion of n.a. restitution


TOTAL CHECKED NUMBERS)
Ethnicity:
  PRE TEST DATE:
POST TEST DATE:
  RISKS
Poor Social Skills Physical Health Problems Violent
Anti-Social Behavior
No Community Ties Unsafe Neighborhoods Prior Arrest History
Prior Runaway
Current Abuse
Past Abuse
Neg


stem Resourceful
Parent Acknowledges Willingness to Work on Problems
TOTAL CHECKS
TOTAL STRENGTHS (SUM TOTAL CHECKED NUMBERS)
PRE
POST
PRE POST
             Acknowledgments
 We are grateful to Sarah B. Hunter of the RAND Corporation and to Carly B. D


le Justice and Delinquency Prevention, September 1994. As of March 8, 2016: http://purl.access.gpo.gov/GPO/LPS36019
American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders: DSM-IV, Washington, D.C., 1994.
Aos, Steve, R


3. As of March 8, 2016: http://www.futureofchildren.org/futureofchildren/publications/docs/18_02_08.pdf
Cocozza, Joseph J., and Jennie L. Shufelt, Juvenile Mental Health Courts: An Emerging Strategy, Delmar, N.Y.: National Center for Mental Health an


uth Confined to Juvenile Corrections,” Journal of Correctional Education, Vol. 53, No. 4, December 2002, pp. 139–144.
Dryfoos, Joy G., Adolescents at Risk: Prevalence and Prevention, New York: Oxford University Press, 1990. Emling, Kelly, chief deput


 Justice, Office of Justice Programs, Bureau of Justice Assistance, April 2000. As of March 8, 2016:
https://www.ncjrs.gov/html/bja/mentalhealth/contents.html
Gramckow, Heike P., and Elena Tompkins, Enhancing Prosecutors’ Ability to Combat and Preven


ms, Office of Juvenile Justice and Delinquency Prevention, October 1994. As of March 8, 2016:
http://www.ncjrs.gov/pdffiles/lit.pdf
Huddleston, West, and Douglas B. Marlowe, Painting the Current Picture: A National Report on Drug Courts and Other Pro


arch 8, 2016:
http://www.policechiefmagazine.org/magazine/ index.cfm?fuseaction=display_arch&article_id=1584&issue_id=82008
Miller, Ted R., Mark A. Cohen, and Brian Wiersema, Victim Costs and Consequences: A New Look, Washington, D.C.: U.S. Departmen


cal Year 2014–2015 Report
Sexton, Thomas L., and James F. Alexander, “Functional Family Therapy: A Mature Clinical Model,” in Mike Sexton, Tom Weeks, and Gerald Robbins, eds., Handbook of Family Therapy, New York: Brunner/Routledge, 2003, pp. 323–348


partment, Los Angeles County Juvenile Justice Crime Prevention Act: Fiscal Year 2005– 2006 Report, Santa Monica, Calif.: RAND Corporation, TR-498-LACPD, 2007. As of March 8, 2016: http://www.rand.org/pubs/technical_reports/TR498.html
Underwood, Lee,


  CORPORATION
Implementing a Drug Formulary for California’s Workers’ Compensation Program
Barbara O. Wynn, Christine Buttorff, Erika Meza, Erin A. Taylor, Andrew W. Mulcahy
For more information on this publication, visit www.rand.org/t/RR1560
 Publi


ivate foun- dations. All its reports are subject to peer review and disseminated widely to policymakers, practitioners in law and business, other researchers, and the public.
ICJ is part of RAND Justice Policy within RAND Justice, Infrastructure, and


13 . . . . . . . . . . . . . . . . . . . . . .
2.3. Trends in Therapeutic Class Share of Payments in WCIS, 2007–2013 . . . . . . . . . . . . . . . . . . . .
2.4. Trends in Average Payment per Drug Line in WCIS, 2007–2013. . . . . . . . . . . . . . .


lls (lines) by Preferred Status for Knee, Shoulder,
and Low-Back Injuries, by Formulary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3. Summary of ACOEM Recommendations for Low-B


 later than July 1, 2017. The legislation intended that the formulary further “the goal of providing appropriate medications expeditiously while minimizing administrative burden and associated costs.”
A carefully structured formulary can reinforce th


e utilization and spending; (2) use of substitut-
ix
x Implementing a Drug Formulary for California’s Workers’ Compensation Program
able generics instead of brand-name drugs, except when there is a clinical rationale for prescrib- ing the brand; (3)


process. Instead, the formulary policies were developed by US Script, a pharmacy benefit management company, and approved by Workforce Safety and Insurance’s Pharmacy and Therapeutics Committee. The Delaware formulary is derived from Delaware’s Medi


veral reasons, the ODG formulary would be easier to implement. It is already in use in several WC programs and incorporates PR recommendations. The formulary drug listing uses a simple yes-or-no structure to indicate whether PR is required for most d


umes that PR rules would be based on specified criteria and that condition-specific PR rules would be established when needed.
Requires developing PR requirements based on specific criteria and clinical review
and integrating the different guidelines


with PR recommendations, any conflicts between applicable ODG and ACOEM recommendations for individual drugs would need to be resolved so that it is clear which guideline has precedence.4 However, the ongoing administrative burden is likely to be le


formulary structure and ground rules, it will be important to establish, through rulemaking, ancillary policies governing how the formulary will be imple- mented and integrated with the medical necessity dispute-resolution process and the OMFS. Tabl


 simultaneously with changes in the formulary without rulemaking.
Table S.2
Summary of Major Policy Development Steps
Summary xv
  Step
Establish ancillary formulary policies
Make conforming changes to UR and independent medical review (IMR) rules
Ma


t from input from other Department of Industrial Relations staff, including Ray Meister and Melissa Hicks at Division of Workers’ Compensation and Amy Coombe in the Office of the Director.
We also appreciate the support that we received from individu


rather than its generic name). Brand-name drugs are often single-source drugs following their initial Food and Drug Administration (FDA) approval because of patent protection and FDA-granted regulatory exclusivity. Later, generic versions of some bra


tion that processes and pays drug claims, administers drug benefits and formularies, negotiates with drug manufacturers on price, and provides other services to payers.
This is group of preferred pharmacies for a payer (insurer or self- insured emplo


eral mechanisms, including purchasing WC insurance from commercial insurance companies or from the California State Compensation Insurance Fund (a public nonprofit carrier) or by setting up a self-insured employer fund. DWC, within the Department of


ly unnecessary drug therapies, raising additional concerns about the administrative burden imposed by current prescribing practices on the UR/IMR medical necessity dispute- resolution process (RAND Corporation analysis of 2014 IMR decisions).
In resp


e process a physician uses to seek approval for a proposed drug therapy before prescribing and dispens- ing it. We use PR throughout this report to describe the comparable policies in other states, although, in other contexts, preauthorization and pr


y net- work. The drug formulary and standards adopted by the administrative director are applicable to drugs provided by a network pharmacy under contract with a payer (or a pharmaceutical benefits manager [PBM] under contract with the payer).
The fo


g formulary. The third report will describe a monitoring framework for tracking the actual impacts of implementing the drug formulary.
Guiding Assumptions
Several assumptions about how DWC will design and implement the drug formulary under- pin our m


y over developing a new formulary that is specific to the California WC program. However, because the MTUS draws on multiple sources for its treatment guide- lines, some modifications may be required for consistency with the MTUS. Moreover, the polic


n at a DWC public hearing held February 17, 2016, to discuss formulary design and implementation issues. The criteria reflect the Labor Code requirements for the formulary and operational require- ments for successfully implementing a formulary consi


cluded in the formulary?
• When should PR be required? What criteria should be used to classify drugs as requir-
ing PR? Should the classification apply across the board to the drug or differentiate by
condition?
• Should there be a first-fill policy


s include the types of drugs and care settings the formu- lary addresses, utilization management tools (PR, point-of-sale processing, retrospective review), the medical necessity dispute-resolution process, first-fill policies following an injury, an


drugs account for a small share of prescription drug volume and payments in California WC. We merged these data to the TRUVEN Health Analytics RED BOOK database by NDC to assign NDCs to therapeutic classes and to flag NDCs as “brand” or “generic,” o


 laxatives.
RAND RR1560-2.1
2 The Workers’ Compensation Insurance Rating Bureau of California collects and reports medical expenditures by insur- ers (Workers’ Compensation Insurance Rating Bureau, 2016). We converted the bureau’s estimate of pharmac


antiulcerants, e.g., omeprazole.
RAND RR1560-2.2
12 Implementing a Drug Formulary for California’s Workers’ Compensation Program
Figure 2.3
Trends in Therapeutic Class Share of Payments in WCIS, 2007–2013
30
25
20
15
10
5
0
2011 2012 2013
2009 2010
2


 Book” lists generic drugs that the FDA has assigned an “A” substitutable designation, which signifies that the reference drug and generic are substitutable (FDA, 2016a).3
Over time, price competition often drives generic prices down significantly be


Code §4073 allows a pharmacist to substitute another drug product with the same active chemical ingredients of the same strength, quantity, and dosage form and of the same generic drug name.
4 The rules governing in-office dispensing are found in §41


(55 percent) of tramadol billed lines are medical rather than pharmacy lines.
6 Prior WCRI and the California Workers’ Compensation Institute work suggests that unit prices for physician-dispensed
drugs are higher than unit prices for pharmacy-dispen


al., 2012; Tao et al., 2015; White et al., 2012). The experience in Florida following its prohibition of physician dispensing of Schedule II and Schedule III opioids suggests that there may be overprescrib- ing of physician-dispensed opioid drugs. Af


rnia’s Business and Professions Code §4052(a)(1) limits physi- cian dispensing of compounded drugs to a 72-hour supply. The most commonly prescribed compounded drugs for WC patients are topical analgesic creams and lotions.
In 2012, AB 378 (Solorio)


rview of Outpatient Drugs Furnished to Injured Workers 19
A recent WCRI report highlights a new concern with physician dispensing. The report indicates that the OMFS is now being undermined by physician dispensing of “specialty” drugs (FDA-approved d


name:
• For simple prescriptions, the Medi-Cal price is the applicable unit price times the number of units plus a dispensing fee of $7.25 for patients who are not in a nursing home. Most unit prices are based on 83 percent of the average wholesale p


cription drug.
The Medi-Cal fee schedule is limited to covered drugs provided by drug manufacturers that participate in the federal Medicaid drug rebate program; as a result, not all drugs provided to WC patients appear in the Medi-Cal fee schedule.1


ate only in the federal rebate program.
 22 Implementing a Drug Formulary for California’s Workers’ Compensation Program
Figure 2.13
Share of Outpatient Drug Lines and Payments Without Medi-Cal Prices, 2013
 Percentage of drug lines Percentage of dru


hen and hydrocodone bitartrate
Sildenafil citrate
Tadalafil
Menthol
Hydroxytryptophan/magnesium/melatonin/ pyridoxine/tryptophan
Docusate sodium Hyaluronate sodium
Chondroitin sulfate/dimethyl sulfone/ glucosamine sulfate
Modafinil
Docusate sodium an


to the PBM. Although the details vary, payer contracts with PBMs often require the PBM to assume the cost for a drug that is dispensed before the compensability determination if the claim is subsequently denied or if UR deter- mines the drug to be me


assume liability and pay a network pharmacy for the drug. If the employer limits its cov- erage to pharmacy network providers, a nonnetwork pharmacy may file a lien to recover its charge for the prescription.
Claims Administrator or PBM
The roles the


shed to injured workers from 2007 to 2013 are matters of concern:
• We found a 43-percent increase in the average number of drug line items per WC claim and a 100-percent increase in average drug payments per injured worker from 2007 to 2013.
• While


each criterion. When information was not publicly available, we did attempt to contact the formulary developers for more information on particular policies. Evaluation of the quality of the treatment guidelines and drug recommendations was beyond th


incorporate methods for dealing with legacy claims,1 particularly for drugs being removed from a formulary.
The fifth criterion specifies the resulting formulary be easily accessible to the end users, such as patients, physicians, pharmacists, and pa


t evaluates evidence-based recommendations of the Medicare Contract Drug Advisory Committee and negotiates prices with manufacturers.
Type of Drug Listing
Categorizes drugs as preferred (no PR), PR required, or noncovered.
Lists drugs by condition, d


on State’s P&T committee then makes recommendations for inclusion on the formu- lary. The P&T committee considers the drug without regard to its cost. However, drug costs are a factor for the final decisions on which drugs will be listed in the Washi


tioner prescribes a nonpreferred drug.
American College of Occupational and Environmental Medicine
ACOEM develops treatment guidelines for specific WC conditions. Drug recommendations are developed as part of the treatment guidelines, using several k


delines in June 2016 amid con- cerns from the National Guideline Clearinghouse that it was unable to verify that each ODG treatment guideline topic met new criteria for systematic evidence review. WLDI has issued an updated statement of methodology t


ed to cover benzodiaze- pines, hypnotics, and antipsychotics. Changes in the formulary are based on recommendations from a P&T committee. The rules state that the P&T committee will base its evaluations and recommendations on “current medical literat


e dispensing applies.
Medi-Cal
The Medi-Cal contract drug list is the formulary for California’s fee-for-service Medicaid beneficiaries. The listed drugs do not require PR (Maiuro, Coronado, and Edwards, 2009). Medi-Cal uses the list to leverage sign


to develop the L&I formulary drugs, including opioids, in a process that is not transparent, and the public cannot formally comment on these changes before they are adopted.
Comparison of Existing Drug Formularies 33
34 Implementing a Drug Formulary


rporates evidence medical condition.
reviewed literature, editorial committee for Incorporates peer- ranking system and decisionmaking. evaluated with
evidence-based
Reliance on
criteria
ACOEM clinical guidelines but has not updated them to include r


all mean quality score for ODG was 3.50 compared to 4.75 for ACOEM, based on publicly available infor- mation about the development methods (Nuckols et al., 2014).3 Because the same guideline development process applies to all clinical topics, the as


back and shoulders. As a result, the ACOEM formu- lary should be more closely aligned with the MTUS than with the other formularies. However, the MTUS ACOEM-based guidelines predate the versions of the treatment guidelines from which the ACOEM formu


nput. A PBM helps translate the recom- mendations into the formulary by including, for example, notes to prescribers or reviewers on particular drug ingredients.
ODG documents do not outline a role for a PBM in developing their formulary drug recomme


require that the P&T committee review the formulary annually and publish changes through a rulemaking process that makes the updating process more cumbersome than those of other formularies (BWC, 2014).
Medi-Cal does not have a regularly scheduled up


is publicly available and is organized by drug name. The Ohio for- mulary specifies the covered form, whether PR is required, and any limitations on dosage strengths or duration. By adding the drug limitations to the drug listing, the Ohio formulary


Chapter Four by examining by the differences in the drug list- ings and PR recommendations.
• Transparency in the decisionmaking process. We found that some parts of the processes for all the formularies are transparent to the public or users of the


edient, such as acetaminophen, underlies a variety of different drug products. For example, acetaminophen comes in a variety of strengths (the amount of the active ingredient), forms (tablet, capsule, syrup), and generic store brands or name brands (


 compensable injury as “denied” (or, essentially, not covered). An exception may be requested through PR if a non– work-related condition directly slows the patient’s recovery from the work-related condition.
ACOEM lists whether the active ingredient


, without further elaboration on whether all forms of this active ingredient are included in the listing. Washington L&I’s preferred drug listing includes oral forms only, broadly defined to include oral, solid, and liquid formulations. In contrast,


d Medi-Cal use First Databank; ACOEM and Ohio BWC use Medispan; and ODG develops its own classifications based on how the drugs are used for WC purposes. For the analyses in this chapter, we used the high-volume active ingredients for the prescrip- t


a- tions of OTC drug ingredients, such as capsaicin, menthol, and methyl salicylate (a pain-relief ointment or lotion). The combination may not be addressed in the formulary, but the individ- ual ingredients may be. For example, both ACOEM and ODG ad


y includes drugs that are adminis- tered in a care setting.
46 Implementing a Drug Formulary for California’s Workers’ Compensation Program
Differences in Preferred Drugs
We next examine whether differences in preferred drugs (which we define as drug


 drug is recommended but unclear on whether the drug is a first- or second- line therapy; whether the drug is not recommended; and finally, whether the drug has no recommendation. These recommendations can vary by the specific
2 The ODG makes some pr


ugs are being dispensed. Analyzing by volume gives an idea of the actual number of PR requests that would be filed under each formulary if it were implemented in California without additional modifications.
We assigned the NDCs reported in the WCIS f


nes) Requiring
   (% of lines)
Not in
Not Requiring Requiring
   Not
Not in
Requiring Requiring
Not in
  Table 4.2—Continued
% of Knee, Shoulder, and Low- Back Lines in WCIS
Acetaminophen/oxycodone
Acetaminophen
hydrochloride
phosphate
50 Implementin


prescribed drug is not consistent with the treatment guidelines. This approach puts the onus on the prescribing physician to know the specifics of the treatment guidelines; in contrast, any variations in the drug recommendations across conditions in


Drug
Recommendation Total No
Recommended Total Not
Yes No* No*
Yes
2
0
32 0 1
Etodolac (Lodine)
Yes
2
0
32 0 1
Famotidine, ibuprofen (Duexis)
Fenoprofen calcium (Nalfon)
Yes
2
0
31 0 1
Yes*
2
1
32 0 1
Yes
2
0
32 1 0
Indomethacin (Indocin®)
Yes
2
0
32


emented with other high-volume or high- expenditure drugs used in the California WC program. If the default policy is that a drug is nonpreferred and thus requires PR if it is not on the listing, the focus should be on whether there are other first-l


 topic was further informed by our interviews with state officials and California stakeholders, presentations at the DWC public hearing, and available literature on specific topics. The topics we cover in this chapter include the following:
• whether


er the formulary should apply in outpatient care settings, such as a doctor’s office or clinic. Typically, drugs are administered to the patient in care settings, while drugs are dispensed by a pharmacy or a physician to be taken by or administered t


body of a patient . . . by injection, inhalation, ingestion, or other means.” Section 4024 defines dispense as furnish- ing drugs on prescription from a physician or nonphysician practitioner or a physician or nonphysician practitioner acting within


ity to develop IDDS-specific rules regarding the frequency and circumstances for which PR is required for refills. If no changes in the PR rules are considered appropriate, there is no reason to specifi- cally address IDDS in the formulary rules.
Dru


ition.
    laries to therapeutic classes that are commonly prescribed for WC conditions and provide an exceptions process to obtain noncovered drugs.
Because treatment guidelines address only drug therapies considered for common WC conditions, the fo


he injured worker’s permanent ability to regain maximum function.” The UR decision must be made within 72 hours. An expedited appeal for IMR requires submission of medical documentation with 24 hours and a decision within three days of receipt of the


rization. Chapter Three discusses the available information on the processes and criteria used to establish drug-specific PR policies for the drug formularies.
Regardless of the formulary structure that DWC adopts, the policies of states adopt- ing t


the ODG
formulary.
PR is required for
• drugs with an “N” status on the ODG Drug Formulary
• any compounded drug containing an ingredient with “N” status
• any investigational or experimental drug
• intrathecal drug regimens and refills involving cha


tions. This could be accomplished by requiring PR for off-label drug usage that is not addressed by the MTUS. The MTUS Medical Evidence Search Sequence would then be used to deter- mine whether off-label usage that is not addressed by the treatment g


forcement is more likely to occur through PBMs under contract with the payer. Other than Tennessee, we did not find state policies pertaining to point-of-sale enforcement. This is not surprising; arrange- ments between the various entities involved i


ect to retrospective review for medical necessity under the state’s process applicable to other medical services. A physician who prescribes services that exceed, are not recommended, or are not addressed by the treatment guidelines is required to pr


ocess. The medical director’s determi- nation is effective until a final IMR determination is issued. In Texas, the volume of medical interlocutory orders during initial implementation was small. For California, a separate appeal
64 Implementing a Dr


he conditions listed in the notice of injury. No more than one drug per therapeutic class is permitted, and no compounded drugs are eligible for the first-fill policy. PR is required for formulary drugs that are not on the first-fill drug listing. Th


ensed for outpatient use.
In addition to Ohio and Washington, several nonformulary states (Montana, New York, Rhode Island, and West Virginia) do not cover physician-dispensed drugs through their prac- tice acts or specifically for their WC programs


ispensed unless the prescriber includes “dispensed as written” or similar language on a prescription for a brand-name prod- uct, Oklahoma and Texas require that the prescriber maintain documentation in the injured worker’s medical record (see Table 5


as a documented systemic reaction consistent with a medication allergy and has tried generic drugs.
Therapeutic Interchange
Therapeutic interchange occurs when prescribers authorize patients to receive a drug similar to the drug initially prescribed.


exclusion from the new formulary. Therefore, states implementing a WC formulary have established a transition process for “legacy” claims (claims with dates of injury occurring before the new formulary implementation). The transition process allows t


ions affected by a formulary.
10 Instead of providing for a transition policy, Oklahoma’s formulary is effective only for injuries occurring on or after Feb- ruary 1, 2014. The effect is to provide injured workers with two standards of care, dependin


d provide a process for considering cost- saving opportunities through therapeutic interchange.
Updating Process
California’s process for updating its drug formulary will depend on how the formulary is structured and the reliance that is placed on ex


ed changes. The final update is posted on BWC’s website. Since the formulary was adopted in 2011, it has been updated eight times.
Updates to the Washington State and L&I formularies are handled differently, but nei- ther involves rulemaking. For the


t of BWC’s pharmacy program director. The rules also charge the P&T committee with reviewing the professional performance of pharmacies and BWC’s PBM. BWC is proposing to add review of provider prescribing patterns and practices to the P&T committee’


al beneficiaries. While technically not a P&T committee, the MCDAC is charged with providing scientific and medical analysis and recommendations to the Department of Health Care Services regarding the addition or deletion of any drug from the contrac


f- interest policies for state Medicaid programs, based on four high-level criteria: (1) whether the policy was available to the public, (2) the requirements for disclosure, (3) the policies associated with recusal associated with conflict of interes


nterest with the sponsor and manufacturers. OIG issued five recommendations encouraging CMS to strengthen oversight of the P&T com- mittees.3 In February 2015, CMS issued a final rule incorporating some of OIG’s recommen- dations into regulation. CMS


 this topic. 1
Establishing a single, integrated formulary that is consistent with the MTUS requires one of two approaches: (1) Adopt both the formulary and guidelines of either ACOEM or ODG, or (2) develop a California-specific WC formulary (which w


adjustors could concentrate on the portions that have been updated, but if a new guideline is implemented, they would need to become familiar with an entirely new guideline. An updated guideline is also likely to require less-extensive changes in bil


nt. Some clinical topics have not been updated since 2011, but revisions are in progress.
Updated MTUS Guidelines and Formulary
Depends on updating decisions. Revised ACOEM-based guidelines and 2016 versions of ODG guidelines for chronic pain and DWC


ngredients that are addressed in the treatment guidelines, (2) estab- lishing PR requirements for formulary drugs based on the relevant MTUS treatment guide- lines, (3) organizing a formulary drug listing by active ingredient, and (4) operationalizin


nditures (exclusive of bulk ingredients used in compounding). Taken together, they account for 83 percent of total drug expenditures.
Operationalize Steps for Three Formulary Alternatives
In this section, we discuss major activities that would be nee


rmulary’s implementation so that the P&T commit- tee will not be in place to consider the initial drug recommendations. Our analyses in Chapter Four suggest that the initial criteria could be relatively straightforward to apply, i.e., drug ther- apie


t different PR requirements. Some differences seem minor and could probably be eliminated (e.g., drug has no recommendation for a condition or the recommendation does not specify first-line therapy when the drug is recommended as a first-line therapy


rtunity for public comment on the process and results. After the formulary is imple- mented, the P&T committee could review and recommend any changes in the PR require- ments as part of its update process.
Appendix B compares ODG and RAND-derived ACO


etermine what drug alternatives might be available under the MTUS for drug ingredients with similar therapeutic purposes. They are also needed if any PR requirements are listed at the therapeutic class level rather than at the drug ingredient level.


ived under a first-fill policy.
Link Online Formulary Listing to Treatment Guidelines
Linking the formulary back to treatment guidelines in an easy way to facilitate the use of the guidelines among prescribers. ODG has an online drug formulary listin


tant to establish, through rulemaking, policies governing how the formulary will be implemented and integrated with the medical necessity dispute-resolution process and the OMFS. Table 7.3 summarizes the implementation policies that will need to be a


defini- tion works only if the rules specify that PR is required for any drug not included in the for- mulary drug listing or not addressed by the MTUS. For the latter, the rule should emphasize that the prescriber should describe why the drug is med


ubstantially higher prices than their OTC counterparts with similar drug ingredients (Healthesystems, 2013). Intrathecal drugs are usually compounded sterile solutions but should be mentioned separately because there may be occasions when a single dr


injury to which the policy applies, the drugs that may be prescribed and dispensed under the policy, any limits on the duration of the initial prescription, and any other conditions on cov- erage under a first-fill policy. The policies that other sta


ork contracts to the formulary but also to specify that “such contracts may not limit the availability of medications otherwise pre- scribed pursuant to the formulary based on whether the pharmacy services are provided within or outside a medical pro


substances prescribed beyond the initial supply must be filled by a pharmacy (Nevada Legislature, 2015). A policy that restricts physician-dispensing to certain drugs is likely to be more straightforward to implement than a rule that provides for phy


vidence-based.” Policy decisions in implementing this policy include the following:
• Whether the policy applies any time a generic drug is available or only when there are multiple sources for the generic drug. For example, the Medicaid FUL is trigg


se require PR. We do not recommend either policy because of the complexities of implementing them across the board. However, a general policy supporting prescribing of OTC drugs when medi- cally appropriate and less costly could lend support to payer


dations and review of OMFS allowances and average daily dosages, the administrative director could limit recommended drugs to the less costly alternatives among the drugs with similar thera- peutic effects. Because California is a multipayer state an


y or modifying the PR recommendation. This issue is discussed in greater detail later, under “Medical Evidence Search Sequence.”
Transparency and public input into the updating process should be facilitated through the rules adopted for the P&T commi


a’s Workers’ Compensation Program
of medically appropriate drugs. Timely updates in the underlying treatment guidelines should avert this situation. Outdated guidelines are less likely to occur with ODG-based guidelines, which are updated on a flow b


 or pharmacy—is liable for the cost of the prescription.
3. A drug bill is presented after the prescription is dispensed, and no retrospective review occurs (i.e., the prescription passes bill-processing screens), or retrospective review deter- mines


uld include updated information on the patient’s condition(s) (diagnoses).
• The section of the form dealing with plan of treatment would separate drug therapies from other therapies.
• The section on drug therapies includes checkboxes to indicate wh


s for drug therapies may address any concerns that the medical appropriateness of drug thera- pies may be determined using outdated medical information. For the WC formulary, more- recent medical evidence could potentially affect two different polici


gs, and nonlisted drugs. In this situation, if a retrospective review concludes that the drug is not medically appropriate, the payer should not be liable for the cost of the pre- scription. In the second situation, a drug that otherwise does not req


This could involve, for example, eliminating the dispensing fee for physician-dispensed drugs and capping the allowance for a brand-name drug at the price for the generic equivalent. The rules for pricing drugs that do not have Medi-Cal prices shoul


 a lien to recover payment that has been denied because the dispenser is not in the payer’s MPN or pharmacy network if there is a dispute over the propriety of treating outside the MPN or pharmacy network.
 Preliminary Implementation Plan 97
   Compa


there is a UR decision to modify If the prescriber did not submit an RFA, the designated entity medication does not pass the PBM’s prepayment screens. or deny the medication.
to initiate the UR process.
   Retrospective review
process
98 Implementing


hat passes its prepayment screens for consistency with the guidelines and advises the pharmacy of its findings. If modifications are appropriate, the arrangements between the claims administrator or PBM and the pharmacy will determine which entity is


manu- facturer, product, and form of the active ingredient. While drugs are often prescribed at the active ingredient and dosage level, drugs are priced and paid at the NDC level. Our assump- tions create differences from how the formularies are ope


f a first-line therapy is not effective, the recommendations cannot be equated to the ODG “Yes” or “No” requirements for PR.
For purposes of comparing the PR requirements across conditions in ACOEM, we estab- lished the following rules, which are for


o so, we calculated the percentage of recommendations for which PR applied across all the drug’s recommendations for a given condition. We did this calculation by condition because a given active ingredient could have different PR criteria for anothe


ions only, all the NDCs were marked as nonpreferred. Other active ingredients had
Detailed Methods and Results for Chapter Four 103
104 Implementing a Drug Formulary for California’s Workers’ Compensation Program
dosage, step therapy, or quantity lim


d in compounding and drugs with deactivated NDCs were excluded from the numerator and denominator for the
calculation of the percent of WCIS payments mentioned in each formulary for each active ingredient.
ODG Ohio Medi-Cal
ACOEM
WCIS Washington
99.6


of drug recommendations were chosen because they form the basis for the majority of MTUS guidelines currently in use in California.
Table B.1 compares, for ACOEM and ODG, the active ingredients accounting for more than 1 percent of paid bills in WCIS


s for chronic pain treatments not covered in the body part chapters for
individuals with chronic pain.
If DWC decides to retain the current guideline structure, the actual PR requirements
would depend on how the ACOEM-based drug recommendations are o


Celecoxib
112 Implementing a Drug Formulary for California’s Workers’ Compensation Program
  Precedence? That Guideline ODG and In Pref. or In Pref. or Guideline with Pref. or Not in ACOEM Not aabab b In Both ODG ACOEM ACOEM ODG Not
Percentage of Li


mitriptyline
hydrochloride
Depends on NP chronic pain
d
0 —
0 1 NP
1
Benzodiazepines
(anxiolytic)
Anxiolytic,
sedatives,
hypnotics
Lorazepam
SOURCE: RAND analysis of ACOEM and ODG drug recommendations, as well as pharmacy claims from WCIS.
Example Fo


nsation Information System (WCIS),” website, 2016a. As of August 4, 2016: http://www.dir.ca.gov/dwc/WCIS.htm
———, “Workers’ Compensation Information System (WCIS): First Reports of Injury (FROI) and Subsequent Reports of Injury (SROI) Data and Report


nmental Medicine’s Occupational Medicine Practice Guidelines, Second Edition,” Journal of Occupational and Environmental Medicine, Vol. 50, No. 3, 2008, pp. 282–295.
Healthesystems, “Topical Analgesics: Expensive and Avoidable,” RxInformer, Fall 2013


d drug substitution—Exceptions—Notice—Limited restrictions, Olympia: Washington State Legislature, 2009.
Swedlow, Alex, and Eileen Auen, “Current Trends in Compound Drug Utilization in the California Workers’ Compensation System,” research note, Oakl


 and the FDA: Questions and Answers,” webpage, 2015. As of July 27, 2016: http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/PharmacyCompounding/ ucm339764.htm
———, “Orange Book: Approved Drug Products with Therapeutic Equivalence Evalu


orker’s Compensation Claim Cost in the State of Michigan,” Journal of Occupational and Environmental Medicine, Vol. 54, No. 8, August 2012, pp. 948–953.
Wiffen, Philip J., Sheena Derry, R. Andrew Moore, Dominic Aldington, Peter Cole, Andrew S. C.
Ric


  CORPORATION
Assessment of the Consolidation of the Australian Customs and Border Protection Service (ACBPS) with the Department of Immigration and Border Protection (DIBP)
 Daniel M. Gerstein, Karen Edwards, Dulani Woods, Julie Newell, Jennifer D.


eas in particu- lar: (1) intelligence, (2) investigations, (3) detention, (4) integrity and corruption, and (5) learning and development. Performance as measured through the documents and interviews highlighted systemic shortfalls that contributed to


nnounced the integration of the Austra- lian Customs and Border Protection Service (ACBPS) and the Department of Immi- gration and Border Protection (DIBP) into a single government department, including the formation of the Australian Border Force (A


 to 2005
documented the shortfalls and provided insights and recommendations for
improving the respective organisations.
– Shortfalls were observed across each of the five areas that were examined in
depth (i.e., intelligence, investigations, detenti


al noteworthy programs were included almost immediately to send a strong
signal to the workforce and the public concerning the changes under way.
– Examples of changes include transitioning to a single uniform for the Austra- lian Border Force; stand


he analy- sis: (1) intelligence, (2) investigations, (3) detention, (4) integrity and corruption, and (5) learning and development. Some of the more-noteworthy shortfalls across the five areas have been consolidated in Table S.1.
Specific Findings: O


 2010; for Immigration, the requirement for all officers to acquire a minimum of baseline security clear- ance through the Australian Government Security Vetting Agency was not uni- versally enforced
• There was no standardised training and developme


ing conditions
• Little direct quantitative evidence of capability-building, yet important qualitative evidence (e.g., common human resources and financial systems, ABF College development)
• Strong preexecution plan developed, but shortfalls noted i


leave government altogether. In terms of workforce acceptance of the integration, a consistent characterisation in the senior- leader interviews was that 30 percent of staff were appreciative of the change, 50 per- cent remained on the fence, and 20


chieve their goals largely rests on the ability of the leadership to keep up the momentum, bring the workforce along, develop a unified DIBP culture, and ensure that the capacities that have been built are able to survive.
Acknowledgements
 This work


e in the integrated organisation to harness the strengths of both former agen- cies? What had been done to mitigate the weaknesses of the two former organisations?
1 The DIBP has undergone a number of name changes since 2001. From 2001 to 2006, it wa


, the development of an evaluation plan (in coordination with the DIBP), an analysis of the existing data concerning the inte- gration progress to date, and an overall assessment of the performance of the DIBP. Formal deliverables were provided in ac


ndividually, indicating that reform was both necessary and inevi- table. So while the integration was a vehicle for achieving change and gaining opera- tional effectiveness, a transformation of the two organisations regardless of integration would ha


her Immigration nor Customs had the professionalisation required of a modern border-management organisation. The need for transformation of the mission space had been well documented over almost a decade.
• With implementation of the integration form


 procedures for border-management, professionalisation, and talent-management efforts.
• The one-year time frame for completing the integration was assessed to be overly ambitious by several of those interviewed. The challenges of integrating two org


sations. Difficulties were made more profound by the lack of a centralised data-storage capacity.
Other common themes from the interviews were the lack of a professional intel- ligence workforce and the differences in cultures with regard to the view


stems.
• Customs conducted a number of classified Strategic Intelligence Assessments.
• Concerning the differences in cultures between the organisations, one inter- viewee noted that Immigration was a facilitative, social-policy organisation and not


ed interdiction,” which includes the use of technology in targeting.5 This report details the importance of technological enhancements to keep pace with the growing opera- tional tempo and increasing sophistication of the potential violators.
Career-


. The broad nature of the review encompasses the full range of intelligence functions, including additional calls for information-sharing and employing technology to more effectively and efficiently manage customs, immigration, and border issues. For


plement changes to develop an intelligence function. These early efforts translated into an ability to target suspect shipments at the operational level.
Because Customs had developed a more robust intelligence function, when the organisations were c


er-map- ping, training, learning, and education programs
– the systems-related investment in devel- opment of a future single intelligence data storage system
– the hiring of intelligence professionals to build capacity
The Case for Change 13
      H


The OSA was combined with the mandatory baseline minimum clearance level. In Immigration, the requirement for all officers to acquire a minimum of baseline security clearance through the Australian Government Security Vetting Agency (AGSVA) was not e


ich cost AUD 700 million, was identified as a wasted opportunity to reform several areas, including investigations. Had the program been successful, it would have connected several disparate systems throughout Immigration.
Findings After Integration


was developed after the integration addresses the need for DIBP officials to have a clear understanding of their powers and demonstrate this competency in several key areas, including investigations.15 Finally, a May 2016 document—Making Children Sa


ternal and external investigations
• Lacked emphasis on proper procedures
for conducting the range of investigations required of a customs, immigration, and • border organisation
• Recommended that training programs for compliance and investigations


 on the organisation, and, following behind the Palmer Report, it sent a shockwave through the portfolio that lasted for many years.
For example, the report noted the need for a “compulsory checklist [to] be com- pleted to record the actioning of a r


, a 2011 report, the DIAC’s Strategic Plan 2011–12, continued to highlight concerns with “administering immigration detention.”21 In 2014, an inter- nal audit titled the Review of Onshore Detention Facilities Maintenance and Asset Man- agement identi


l 2015.
The Case for Change 19
 20 Assessment of the Consolidation of the ACBPS with the DIBP
Findings After Integration
Efforts to reform the detention area are well documented in the February 2016 Deten- tion Remediation Report (Classified) develop


 Canberra, March 2016.
 put in place a capability in the immigration detention network to address potential child abuse.
Efforts are also under way to ensure that the roles and authorities of officers within the detention system are understood. This


ystemic issues in its dealings with people at the border and in the detention network, many of the same systemic issues were identified in later reports, including the Moss Review and the
The Case for Change 21
22 Assessment of the Consolidation of t


f corrup- tion risks in that organisation was extremely low and largely limited to overseas opera- tions. This resulted in blindness and a less-than-robust response to the risks of corrup- tion in immigration decisionmaking.
 29 DIBP, MIR Summary—Nau


sing was centralised in the AGSVA. At this time, the requirement for all individuals to obtain a minimum baseline clear- ance was continued, albeit via the AGSVA, with the additional “fit and proper person” checks continuing to be undertaken by Custo


egic Plan to Build Stronger Migration and Visa Services, Can- berra, 2010; DIAC, Strategic Plan 2011–12, Canberra, 1 July 2011.
31 See Australian Public Service Commission, “Security,” in Foundations of Government, Canberra, 23 May 2012.
32 Australia


rt (Interim Report), Canberra, February 2013.
34 Australian Commission for Law Enforcement Integrity, Operation Heritage (Final Report)—A Joint Investiga- tion of Alleged Corrupt Conduct Among Officers of the Australian Customs and Border Protection


on issues were plaguing both organ- isations. Although some reforms were put in place prior to integration, in the post- integration phase and carrying through into the more comprehensive reform period, a
36 Australian government, Customs Reform Boar


part of the integration-reform professionalisation:
• Implement better record and case management of sensitive matters by utilising Custom’s upgraded IT system capability.
• Implement a collection-management strategy in association with the Intellige


rous areas.
In addition to the DIBP Leadership Framework, the leader-development road map was established in May 2016. One element of this road map is highlighting the various points in an officer’s career where integrity training and development wil


recommendations after the investigation of corruption allegations through the Operation Heritage and Myrrh reports. Although a reform programme to address the integrity issues began in 2013, they will continue and be challenging, even in the integra


y developed drug and alcohol policy in May 2015 has been implemented
• A leader-development road map was devel- oped in May 2016
• Online ethics and conduct training was developed
• Integrity conversations were incorporated into workforce education
T


 work- force issues.
Other important development shortfalls were in areas previously discussed, such as integrity and anticorruption and general professionalisation. Of note, the two organ- isations had very different cultures, which could complicate


public confidence. The roots of this change can be seen in several of the documents that highlighted shortfalls in training and education, as well as deci- sionmaking skills.
Also highlighted were requirements for different types of training, learnin


tioned from being highly focused on specialty areas to including greater emphasis on general law-enforcement issues, individual development and profession- alisation, and broader roles and responsibilities. This was also seen in the interviews that w


k package. Both the visibility this topic is getting and the ongoing efforts bode well for long-term success. However, it remains too early to assess whether these programmes will ultimately be successful.
Establishment of a board to monitor this are


demonstrated the need for change within Australia’s border management. The five areas examined—(1) intelligence, (2) investiga- tions, (3) detention, (4) integrity and corruption, and (5) learning and development— provided ample evidence of shortfal


ntegration began (and the resulting reinvigoration of the effort several months later) should serve as a warning of the potential for the effort to stall if leadership engagement does not continue.
The Case for Change 37
CHAPTER THREE
Assessing Opera


rocess designed to build organisational capacity
Examination of the reorganisation planning for pre- and post-execution
Capabilities for monitoring the progress of the reorganisation
Assessment
• No observed loss in effectiveness across opera- tional


filling the goals of the integration. However, a noted shortfall is the nearly exclusive focus on operational metrics at the expense of strategic and reorganisation metrics and the omission of some key DIBP mission elements, such as cybersecurity and


––2
–63003
311–1
2– 1 – 1 14 14 – – – 8413– 22–––
0– – – – 2– – – 2 31020
130 64 29 15 22
  Outcome 1: Protect Australia’s sovereignty, security, and safety by
managing its border, including managing the stay and departure of all noncitizens.
 Progra


gramming, requirement-generation, and bud- geting functions of the department; although in the early stages, these efforts should improve the department’s operational capacity across the DIBP mission space. Com- bining management functions was design


portant to sustaining momentum for the reorganisation, as will elevating the responsibility for reorganisation to the deputy secretary level.
However, several important shortfalls in the new metrics have been identified. First, they do not consider t


ltural and personnel integration, and capability-building (including in planning, programming,
requirement-generation, and budgeting functions), even after such a short period.
In looking to the future, the most-critical elements for progress in the


and lack of consistent leadership engagement.
– The use of a system approach to completing the integration and subsequent reform will be important to ensuring that the comprehensive reform envi- sioned is achieved. This is especially important given


ation and Citizenship Policy Division (DIBP)
First Assistant Secretary International Division (DIBP)
First Assistant Secretary Traveller Customs and Industry Policy Division (DIBP)
A/g Deputy Commissioner Operations (ABF)
Assistant Commissioner Strat


ing a professional customs, immigration, and border organisation?
b. Other assessments concerning the reorganisation?
c. Anything else you would like to add?
ANNEX D
Assessment Plan: Operational Effectiveness
 Organisational Analysis
Using an objecti


n this annex corresponds to the five assessment categories in the GAO report: operational effectiveness and efficiency, capability, cultural and person- nel, action plan, and monitoring progress. This annex includes additional information and assessm


nd crews refused immigration clearance
Number of detections of undeclared firearms
Number of visa cancellations
2014, Quarter 4
65 557.94
1,482 306 4,283
2015, Quarter 1
384 4,033.33
585
805 12,806
2015, Quarter 2
123 1,053.23
964
495 16,687
2015, Qu


hich include such areas as improving the department’s processes, budgets (and revenues), material expenditures (and modernisation of equipment), personnel, and training.
The development of the 2016 Strategy Handbook demonstrates a commitment to build


lt from savings from eliminating the IT management organisation organic to Customs. The increased capacity would result from increased information-sharing and eliminating an unnecessary IT seam between the former two departments’ IT sections.
The dev


a- bility were identified. The categories included border force, future traveller, trusted trader, case management, data intelligence, mail modernisation, and infrastructure. They encompassed a broad range of activities, from development and fielding


s. This too became an important signal to the organisation and helped to build a unified culture even at these lower tactical levels. Such teams can also have benefits in operational effective- ness and efficiency.
Although many early efforts to buil


agency.’”8 The commission’s recommenda- tion was based on two anticipated outcomes, “the potential to generate significant savings and provide the ‘optimal structure’ to pursue a more effective approach to border protection through ‘a series of inte


assess- ing whether the high-level objectives that served as the rationale for the reorgan- isation, as well as the reorganisation action plan’s objectives, were achieved. This shortfall was recognised, and metrics have been developed for many of the


- sion support, and ultimately resource allocation.
Additionally, given the implementation issues discovered shortly after integration was initiated on 1 July 2015, the DIBP has instituted the Integration Action Tracker, specifically to monitor integ


regards.
– Operational versus reorganisation focus: In the long term, such an operation-
ally focused list might be appropriate; however, given the expressed desire to
monitor the progress of the reorganisation, it is likely not adequate.
– Operation


izenship
DIBP Department of Immigration and Border Protection GAO U.S. Government Accountability Office
IMA illegal maritime arrival
IT information technology
OSA Organisational Suitability Assessment
PSM Public Service Medal
75
References
 Assistant


icy Institute, 30 Janaury 2009. As of 29 August 2016: https://www.aspi.org.au/publications/a-safer-australia-meeting-the-challenges-of-homeland-security- by-anthony-bergin/Policy_Analysis36.pdf
Boston Consulting Group, Reform and Integration: Measure


ist, Canberra, 2016.
———, Vocational Competency Profile: Investigations Compliance and Regulation, Border Force, Canberra, 2016.
Department of Immigration and Citizenship, The DIAC Strategic Plan 2010–11: A Strategic Plan to Build Stronger Migration


ess the degree to which the new organisation has been able to mitigate weaknesses and reinforce strengths.
The integration of the DIBP reflects an important inflection point in the history of customs, immigration, and border-management activities wit


  Using Future Broadband Communications Technologies to Strengthen Law Enforcement
John S. Hollywood, Dulani Woods, Andrew Lauland, Sean E. Goodison, Thomas J. Wilson, and Brian A. Jackson
 Forthcoming broadband communications technologies could prov


. This will require
a governance model that can make these decisions and get the message out to the individual departments. Need to develop an equivalent of a certificate authority for authentication for public safety communications.
2
law enforcemen


ommu- nications, today the focus is on electronic data, ranging from life-critical information about persons and locations of interest to vehicle tracking data. Tomorrow, communications net- works will have to handle ever-growing amounts of data from


ng beyond “next shift/next purchase” communications issues and helping shape law enforcement communications in the 2020s. The key objective of the work- shop is to inform key science and technology (S&T) develop- ment and deployment efforts, with the


 and health data.
− Near-real-time video feeds from deployable closed-
circuit TV (CCTV) systems. The studies called for a relatively small number of deployed cameras in support of specific operations, not a general video surveillance network.
− Exch


take more than a minute in total.
• Live transmission of a “small” number of high-definition video feeds simultaneously.
An Initial Set of Emerging Technologies
To provide the workshop with a set of initial ideas for technolo- gies to discuss, NIJ an


thered UAS that can act as a local area relay, with the tether providing persistent power that can keep the UAS in the air for long periods of time.
• Improvements to satellite communications. Minoli (2015) provides a recent survey.
• Improvements an


ons across the network. Includes identity credentials, access manage- ment, other security measures, and network management functions.
• Operational considerations—cutting across the techni- cal layers, these are operational challenges for efficient


x B (available at www.rand.org/t/RR1462). In brief, needs were prioritized based on their expected value, which combines assessments of how much value a solution to each need might be to law enforcement, how technically feasible developing a solution


For each need, we show the problem to be solved (“Problem”) and the call for a spe- cific solution (“Need).” Note that some operational problems are addressed by multiple specific needs, meaning that some problem descriptions are repeated multiple ti


nd for what purposes? What apps/ functionalities do we want to put on officers’ devices?
Problem: Scale of data being collected/ exchanged expected to increase dramatically.
Need to provide agencies with guidance on how to acquire, manage, and use mi


Need better ways to do user authentication—make it easy for individual users, such that, for example, if a public safety officer leaves their device behind it is locked, but also accessible by individuals from other agencies.
Need to solve the federa


ilter and prioritize core types of data coming into PSAPs and operations centers and provide useful products to the field. Need to explore leveraging analytics from the carriers to identify incidents of high interest. Would involve creating archives


apacity issues occur not only during incidents that are universally viewed as special circum- stances or emergencies (e.g., Boston Marathon bombing) but also during times that would traditionally be considered nor- mal daily operations (e.g., high-vo


vice and wireless Internet access points that the devices have been manually configured to access.
Network management in this future environment will cease to be about managing specific sets of equipment (or bandwidth usage permitted under a specific


 best support law enforcement requirements. Common types
of services (push-to-talk voice over Internet, maps, etc.); data flows responders need on a day-to-day basis; and their sources, qualities, and attributes need to be identified. There was also


Flood. Data storage and archiving will be a huge technical, legal, and policy issue. Participants noted that there is a sense that all of the footage being generated from new body cameras “probably has to go into evidentiary archive” and that the vol


sed this area of research on grounds that it was unlikely that small law enforcement–focused projects would improve on huge com- mercial investments in antenna development.
A Larger Debate: Commercial vs. Customized Devices. While not R&D needs per s


 meet changing user needs.
High-level Network Management Issues. Participants noted that while long-range forecasting of demand is important, and there may be some relevant cases to study today, demand will always come down to a management issue. One