Roca devotes 100 percent of its expenses to a program designed to help “high-risk”
young men ages 17 to 24 move out of violence and poverty. Its fundraising materials
accurately reflect that mission and spending breakdown.
Is the causal logic plausible? *
Roca’s Web site outlines a three-phase “intervention model” to help its graduates
get jobs and stay out of jail. In the first phase, the charity recruits young people into
the program, offering a range of job-training and educational programs. In the second phase, it builds “transformational relationships” with its clients, offering more
intensive programs. In the third phase, it offers formal vocational training and other
career help so they can “move themselves out of violence and into jobs.”
Is there some indication of how much of what action is required to produce
Roca’s 2013-17 strategic business plan fleshes out its timeline for achieving its
goals during a two-year intensive program, followed by two years of follow-up services. It specifies how many times a week clients should meet with youth workers or attend various programs, depending on how long they have been enrolled. It also outlines a series of activities that clients should complete during the first two years, including two industry-recognized job-certification trainings, three mock job interviews,
10 work-readiness workshops, and four pregnancy-prevention workshops.
Is there some indication that [its plans to achieve its goals are] based on rea-
In the paper “The Evolution of Roca’s Intervention Model,” Roca explains how it has
studied, adapted, and put into place elements of behavior-change programs that
have proven effective. It says it has aligned its approach with the “Eight Evidence-Based Principles of Effective Intervention” that were identified by the Crime and Justice Institute, a research center at the nonprofit Community Resources for Justice.
Are there specified measures (indicators) to be collected and a plan to do so?
Roca’s 2013-17 strategic business plan says its program is designed to achieve
three things that will save the public money: reduce incarcerations of young men,
make sure they comply with parole or probation conditions, and prepare them for
economic independence through jobs and education. It highlights statistics that it
has collected to show progress in those three areas: In 2012, 90 percent of the
graduates of Roca’s two-year intensive program had not been rearrested, none had
violated probation or parole, and 79 percent were on track to retain jobs for at least
Does the charity publish evaluation reports that cover the results
of its program at least every five years?
Roca regularly assesses its performance and seeks outside evaluations. Charity
Navigator specifically cites a report published by Chapin Hall, a policy-research center at the University of Chicago that the charity hired to conduct an extensive study
of its impact—partly to prepare for participation in a Massachusetts program that
will pay for Roca’s services if they produce specified results. The 2012 report outlined a strategic plan for measuring and reporting on the group’s performance.
Are those evaluation reports based on recognized techniques
to understand their results?
Chapin Hall’s report, prepared by its Center for State Child Welfare Data, outlines
recognized techniques such as collecting data, strengthening continuous-quality-im-provement processes, setting up statistical models for assessing impact, and establishing random and control groups to compare outcomes of Roca’s graduates with
those of similar young men who did not enroll in the charity’s programs.
Either in the evaluation report or subsequently, does the charity explain
what, if anything, it is changing as a result of the findings in the evalua-
The Chapin Hall report outlines ways to conduct “formative evaluations,” or collect
preliminary data so Roca can adjust its strategy if it is not achieving its desired outcomes.
* In other words, does the charity’s explanation of how it plans to achieve its goals seem possible or likely?
Charity Navigator, a ratings group that many do-
nors consult before deciding where to give, now
awards charities up to four stars based on their fi-
nancial health and their accountability policies.
But in a new approach it dubs CN 3.0, it has also
begun analyzing whether organizations measure the
results of their programs and let the public know
about those results. Its assessments will not affect
a charity’s star ratings until at least 2016, but the
findings will be posted online for anyone to see.
The watchdog has so far evaluated more than
200 family- and children’s-services organizations
on their results reporting, including Roca, a non-
profit in Chelsea, Mass., that provides educational
and career services to young men who have been
involved in gangs or criminal activities.
Roca, a four-star charity that gives priority to
measuring what works, earned eight positive marks
out of 14 questions—much higher than most charities that have been reviewed.
“We’re really committed to driving the outcomes
and having an impact,” says Anisha Chablani, Roca’s chief knowledge officer. “When we started
down this road internally, our mantra was, Either we
should help young people to get these outcomes or
we should shut our doors.”
‘Pay for Success’
Roca has been selected to participate in a “pay
for success” pilot program operated by the State
of Massachusetts, which will pay for the charity’s
services only if it meets specific goals. It is now
setting up a randomized controlled trial—
considered the gold standard of program evaluation—to
study how its graduates have fared compared with
a group of similar young men who did not receive
However, even Roca got all Xs from Charity Navigator in a section called “constituent voice,” which
evaluates whether a charity is getting feedback
from the people it serves about its effectiveness.
Ms. Chablani says the charity does seek client
feedback but has no formal process in place—and
is pondering how to strengthen its efforts. She
says it is challenging to find the right methods and
timing in dealing with young men who often find
it difficult to engage with Roca staff, especially at
“If we collected it in the first year of engagement,
it would be highly skewed because we’re pushing
them to change behaviors,” she says. “They might
say, ‘I hate Roca, it doesn’t make sense,’ but it
might be based on where they are in the program.”
Following are the check marks that Roca won in
a CN 3.0 assessment and the documents Charity
Navigator cited to explain its results.
Ratings of 1 Group