Suggestions for Evaluation of Programs Designed To Reduce Recidivism among Juvenile Justice- Involved Youth
*Hide Yamatani
University Of Pittsburgh, USA
Submission: February 17, 2017; Published: March 22, 2017
*Corresponding author: Hide Yamatani, Center on Race and Social Problems, University Of Pittsburgh, School of Social Work, PA. 15260, Pittsburgh, USA, Tel: 412-624-6300; Email: hzy@pitt.edu
How to cite this article: Hide Y. Suggestions for Evaluation of Programs Designed To Reduce Recidivism among Juvenile Justice-Involved Youth. J Forensic Sci & Criminal Inves. 2017; 2(3): 555587. DOI: 10.19080/JFSCI.2017.02.555587
Abstract
This paper discusses suggested evaluation designs for of criminal justice programs focused on reducing the recidivism rate among juvenile. Various concerns related to traditional experimental and quasi-experimental designs are examined. Evaluation consultants and doctoral students who are planning to study potential impact of criminal justice social interventions are highly recommended to consider advantages and disadvantages of possible evaluation designs. It is essential that program evaluators are able to identify areas in need of improvement, and specify evidence of the program benefit to youth involved in the juvenile justice system. Rigorous program evaluation can help establish these benchmarks and also provide agencies with feedback needed to improve implementation and optimize program efficacy. This paper also notes that the mission of evaluators and researchers should be focused on generating accurate and highly beneficial findings for program participants. In other words, the greatest beneficiaries of human service evaluation and research products should be those individuals in need (e.g., youth involved in the juvenile justice system).
Keywords: Criminal justice programs; Evaluation design; Juvenile youth; Criminal justice impact; Recidivism measurement
Introduction
Outcomes for high-risk youth in the juvenile justice system are known to be poor, leading to unrealized potential and high rates of recidivism, at considerable cost to the commonwealth and the Nation [1-3]. It is essential that novel programs are developed to improve these outcomes, and that such programs demonstrate evidence of their benefit to youth involved in the juvenile justice system. Rigorous program evaluation can help establish these benchmarks and also provide agencies with feedback needed to improve implementation and optimize program efficacy.
General approach to rigorous program evaluation
The School of Social Work’s research team has been conducting evaluation research in the criminal justice field for over ten years. Our approach is to partner with the agency being evaluated. Evaluation consultants that elect not to collaborate with the organization being analyzed may simply incorporate their own personal biases into their evaluation design and interpretations of findings. Such a practice can lead to unverified and incorrect assertions, erroneous projections, and unsuitable recommendations. Unquestioned acceptance of isolated and remotely generated evaluation reports that contain a semblance of "objectivity" without verifications can engender hefty liabilities for clients in need, their service providers, and the funding organizations. Thus, human service evaluators should begin by creating a forthright rapport with the program organizations in order to jointly collaborate on scientific evaluation goals and method.
An intensive sharing of ideas between the researcher and organization is especially important during the development phase of data collection protocol, which requires open communication with the service staff members to:
- ensure a practical and well-integrated research design that incorporates considerations of unique operational challenges and administrative-level insights; and
- Prevent evaluation research errors of Type III (asking the wrong questions), Type IV (pontificating on trivial findings over important findings), and Type V (anticipating errors in reference to "good practice" based on effect size and achieved effect size).
For a more in-depth discussion of how to avoid such evaluation errors by implementing collaboration-based research, see the attached journal article [4].
Based on more than thirty years of experience as an evaluative researcher, the author may assume that academic researchers often underestimate the values and benefits of social service intervention programs. Such repercussion stems from the limited scope and magnitude of selected outcome indicators, which only expose a small portion of the panoramic benefits that often accompany the mission-driven and dedicated human service providers. The attainments of intangible benefits (e.g., "understanding of consequences associated with criminal behavior," "hope for the future," "self-esteem and perceived efficacy," "career aspiration," etc.) are typically ignored because they are invisible and cannot adequately be translated into dollar value of benefits. However, it is important to understand that directly measurable program benefits (e.g., rates of truancy and recidivism reduction, improved employment status, enrolment in training programs and secondary schools, etc.) are unattainable without the achievement of the intangible intervention effects (via intervening or mediator process roles). Additionally, inexperienced evaluation researchers are often short on understanding the enormity of the serious risks, disadvantages, and crises that continually challenge the program participants.
It should also be noted that the mission of social work evaluators and researchers should be focused on generating accurate and highly beneficial findings for program participants. In other words, the greatest beneficiaries of human service evaluation and research products should be those individuals in need (e.g., youth involved in the juvenile justice system). A commitment to such a research mission helps to avoid potential conflicts of interest between the corroborating researchers and host organizations, funding agencies, and other key stakeholders. It is also mandatory that research products be developed with rigor and generated through ethical treatment of study participants in reference to three major principles: protection of individuals' rights, social justice, and utilitarian perspectives [5]. Therefore, based on this understanding, it is essential to establish an active, trust-centered, and scientifically sound collaboration between the evaluator and the organization in order to generate accurate and meaningful evaluation products.
Potential risks of using randomized evaluation methods
The randomized evaluation method is generally viewed as the most powerful scientific methodology for assessment of a selected intervention's effectiveness. However, randomized- controlled studies continue to be the "exception" rather than the rule in criminal justice research [6]. This is because randomization often falls short of client consent and the potentially impractical nature of the data collection process to be adopted by the intervention providers. A randomization- based intervention assignment of participants often requires clear communication about potentially differential benefits and remains open to the participants’ first choice of service group without coercion, baiting, harm, or deceit. If the participant (or parent/guardian of the participant) wishes to enroll in a different program after commencement of the assigned group, such requests must be accepted with sincerity and support. In other words, in human service evaluation studies, the participants rule-not the researchers. Another concern with randomized evaluation design is related to its near impossibility of achieving sample equivalence between assigned groups beyond the initial participant assignment period. Typical evaluation studies that require 12 months or more of an intervention period cannot guarantee that their social environment will remain static, or that influence between selected participant groups will be equitably and evenly distributed.
Unless the participants are in captivity, there is no way for the researchers to control varying and potentially uneven external influences among study groups (e.g., dynamic impact of the individuals' peer group relationships, potential spillover effects between two separate study groups living in same neighborhood, influences stemming from ever-changing community environments, effects of fluctuating emotional support by extended family members, shifting facilitation by other community-based organizations, faith groups, etc.) In other words, in the real world, it is highly unlikely that postrandomization effects will be equitably dispersed for both intervention and control groups. Thus, under such conditions, researchers will not be able to tell precisely to what degree intervention outcomes are due solely to the selected intervention or how contextual influences may have intensified or diffused the intended intervention outcomes. However, the value of the intervention under assessment can become obvious if and when it is powerful enough to overcome and diffuse extraneous differences. For example, if the quality of the mentorship is consistent and strong enough, study findings will point toward its significant utility and value.
A case example of measuring mentoring impact
Even with the aforementioned concerns, an evaluation project’s best approach is to implement a randomized assignment of the participants for evaluative comparison. A primary ethical concern with randomized assignments is the presence of true clinical equipoise between experimental conditions. In some cases, this cannot be established, as prior evidence heavily indicates a benefit of one intervention over the other (e.g., in the case of cancer screening). However, mentoring interventions, for example, have equivocal support in various fields and limited information is available on their efficacy in juvenile justice populations. In the field of mental health, mentoring interventions have become popular [7], but randomized trials have not supported their efficacy for improving outcomes [8]. As such, an evaluator may assume a null-hypothesis-that there are no significant benefits of mentorship, even though such intervention arrangements are becoming very popular among human service providers. Given this context, true clinical equipoise appears to exist, and if participants are fully informed and voluntarily participating in the evaluation, there should be no significant institutional review board (IRB) issues associated with the randomization option in this case. However, evaluation based on randomization must ensure that there are no significant spillover effects (e.g., between groups sharing information regarding intervention insights or participating in the same activity events organized by the mentors).
This means that the host organization must be adamant about avoiding potential between-group spillover liabilities- they must actively monitor their youth participants' intervention schedules and social events and achieve the necessary level of fidelity as specified by the evaluation protocol. Thus, to successfully institute such a demanding evaluation process, one option is to limit the number of evaluation sites in order to allow for close monitoring and direct guidance by the evaluation team. The actual sample size of the evaluation interviews will vary depending on the number of major variables to be included for the assessment. In any case, one must carefully estimate the number of participants needed to avoid Type II errors (attaining an error of false non- significance). For example, an hypothetically anticipated 20% improvement among mentored former juveniles compared to those without mentorship (with 85% statistical power level and alpha set at 0.05), it would require 222 individuals for randomly assigned two-group comparison to avoid committing type II error [9]. The major statistical assessment technique will depend on the attained type of data distribution (parametric versus non-parametric), which will be examined based on Shapiro-Wilk and Kolmogorov- Smirnov tests of data distribution patterns. An assessment of the normality of data is a prerequisite for many data sets generated by human service projects, because bell-shaped data distribution is an underlying major assumption in parametric statistical tests.
Thus, ifthe data distribution assessment supports a normality pattern, the evaluation can utilize parametric statistical procedures, such as the Pearson correlation coefficient, t-test, dependent t-test, analysis of variance, multiple regression, and survival analysis (Kaplan-Meier analysis)-depending on the sample group profile and type of major variable comparisons to be examined [10]. However, data involving social service participants may not meet the data normality requirement. In such a case, major non-parametric statistical procedures can be employed, such as the Chi-Square test, Fisher’s Exact test, MannWhitney U test, Wilcoxon Rank Sum test, Wilcoxon Signed Rank test, Kruskal-Wallis test, Friedman test, McNemar test for binary variables, or logistic regression-again, depending on the sample group profile and type of variable comparisons to be analyzed [10].
Risks associated with Quasi-experimental methods
Quasi-experimental methods offer various techniques for sample group selection and data collection strategies, but they are often criticized for non-attainment of sample comparability between the treatment and comparison groups (if selected), and can therefore fall short on measuring the sole effects of the intervention. In other words, an inability to institute pure randomized sample assignments to one program or the other leads to a "soft" comparison-even though pure randomized methods are difficult to institute in the real-world studies of afflicted or victimized individuals or populations in need. However, it should be noted that typical mission of the human service program is to assist afflicted participants in reference to a set number of program goals and objectives, and needs to assess the extent of their goal achievements. Thus, it is often questioned as to why one would expend limited resources to measure whether the extent of the participants' achievements are due "solely" to the selected intervention. In the human service world, no single intervention is powerful enough to override the participants’ various other needs, as well as their variety of contextual influences (e.g., household poverty, drug addiction among parents/guardians, physical health issues, community-wide violence, positive influence of new peer group, teachers taking the role of counselor, etc.).
Therefore, based on a utilitarian view, answers needed in evaluation are to document to what extent the participants improved from the baseline to after discharge-there is no need to further invest limited resources on additional costly analyses based on controls of extraneous or contextual variables. A common quasi-experimental method compares outcome-related variables (representing measurable program goals) based on data sets at intake, discharge, and six months after discharge-progressive repeated measurements and analysis. More specifically, this quasi-experimental method calls for paired sample groups (comparing the same individuals over time) with repeated measurement design involving the following steps:
i. Baseline assessment-participant profile measurement at intake/admission as baseline data on demographics, as well as variables representing the intervention's goals and objectives (e.g., history of arrests and conviction, criminal attitude and anger management, drug and alcohol usage, social skills and abilities, understanding about restorative justice and truancy rates, behavioral and mental health status, overall physical health status, future career aspirations, etc.) to be collaboratively selected and specified by the organizational stakeholders and evaluation team.
ii. Assessment at discharge-suggested assessment areas for discussion with youth advocate programs (YAP), State officials, and other key stakeholders, as well as measurements of successful completion rates of aforementioned outcome indicators.
iii. Post-discharge outcome assessment-based on previous experience, thirty days after discharge is too early for intervention impacts to surface among the participants. Six months after discharge is a more reasonable assessment period. Suggested measurement variables for discussion with YAP, State officials, and other key stakeholders include measurements of rates of successful outcome indicators as noted above.
iv. Cost-benefit/savings analysis-open to collaborative discussion and decision-making, this analysis involves a comparison between the cost of program operation and estimated amounts of tangible benefits (or savings) associated with program intervention. The cost-savings analysis may be based on analytic strategies reported by the Pennsylvania Preventive Research Center (2007), Washington State Institute for Public Policy (2007), Urban Institute (2006), or Juvenile Justice Evaluation Center (2002).
More specifically, the comparative assessment can consist of the following potential cost-saving areas:
a) Average cost of institutional or jail stay,
b) Processing offenders in the criminal justice system,
c) Service needs savings,
d) Crime victimization reduction. Another potential benefit assessment may include a comparative estimation based on the completion of high school graduates versus non-graduates on lifetime earnings [11].
If a quasi-experimental method is selected, it is suggested that research design include a triangulation method in the evaluation process. As described by Thurmond [12], triangulation method calls for the combination of two or more data sources, investigators, methodological approaches, theoretical perspectives, or analytical methods within the same study. These combinations can be called data triangulation, investigator triangulation, methodological triangulation, theoretical triangulation, or analytical triangulation.
Conclusion
Although rarely discussed in program evaluation literature, the triangulation method helps to increase the validity, strength, and interpretative potential of evaluation findings, while helping to decrease investigator biases and provide multiple perspectives-a system that adheres to strong internal, external, convergent, and theoretical validity. It should be noted that author's research projects have been utilizing triangulation methodology for the past few decades. For example, the study on unveiling patterns of salary inequity offers methodological triangulation [13], and studies including the Allegheny County jail collaborative cost-benefit evaluation study [14], the Allegheny County child welfare worker caseload study [15], an article on contemporary social policy analysis methods [5], and the study on employing people with disabilities [16] are based on data triangulation methods. As discussed in the randomized assessment strategy section, the statistical assessment technique will depend on the attained type of data distribution-parametric versus non-parametric distribution. If the data distribution indicates normality, parametric statistical procedures will be utilized; otherwise, major non-parametric statistical procedures will be employed.
References
- Seigle E, Walsh N, Weber J (2014) Core Principles for Reducing Recidivism and Improving Other Outcomes for Youth in the Juvenile Justice System. The National Reentry Resource Center and Justice Center, USA.
- Gottesman D, Schwarz S (2011) Juvenile Justice in the U.S. Facts for Policymakers. National Center for Children in Poverty. USA.
- Holman B, Ziedenberg J (2011) The Dangers of Detention: The Impact of Incarcerating Youth in Detention and Other Secure Facilities. Justice Policy Institute, USA.
- Yamatani H, Mann A, Feit M (2013) Avoiding Type III, IV and V Errors through Collaborative research. J Evid Based Soc Work 10(4): 358-364.
- Yamatani H, Feit M, Mann (2013) Contemporary Social Policy Analysis Methods: An Incorporation of Ethical Principals and Implementation Processes. Journal of Human Behavior in the Social Environment 23(7): 817-823.
- Weisburd D (2016) The "care package," prison domestic violence programs and recidivism: a quasi-experimental study. Experimental Criminology 12: 563.
- Davidson L, Chinman M, Sells D, & Rowe M (2006) Peer support among adults with serious mental illness: a report from the field. Schizophrenia Bulletin 32(3): 443-450.
- Lloyd-Evans B, Mayo-Wilson E, Harrison B, Istead H, Brown E, et al. (2014) A systematic review and meta-analysis of randomized controlled trials of peer support for people with severe mental illness. Res Soc Work Pract 24(2): 192-212.
- Kane SP (2017) Sample Size Calculator. Clin Calc.
- IBM SPSS Statistics for Windows (2016) IBM SPSS Statistics for Windows, Version 24.0. IBM Corp. USA.
- Yamatani H, Solveig S (2011) Rescuing U.S. Criminal Justice System: An Efficacy of Collaborative Social Service System. Social Work 56(1): 5361.
- Thurmond VA (2001) The Point of Triangulation. Journal of Nursing Scholarship 33(3): 253-258.
- Yamatani H (2006) Unveiling Patterns of Salary Inequity: Suggested Measurement Strategy for Health Care Organizations. Journal of Health and Social Policy 21(4): 95-108.
- Yamatani H (2008) Overview Report of Allegheny County Jail Collaborative Evaluation Findings. Center on Race and Social Problems, School of Social Work, University of Pittsburgh, USA.
- Yamatani H, Engel R, Spjeldnes S (2009) Child Welfare Worker Caseload: What's Just Right? Social Work, 54(4): 361-368.
- Yamatani H (2012) The Program for Offenders: Comprehensive Evaluation and Cost/Benefit Analysis of a Community Corrections Facility. National Institute of Corrections, U.S. Department of Justice, USA.