December 16, 2011

Building a Culture of Continuous Improvement

With jobs at the top of the national agenda, public and private funders are intent on finding effective job training programs. But what data demonstrate effectiveness, and how can programs use data to improve what they do?

With Fry Foundation support, a path breaking collaborative of employment training programs is building the answers.

At first blush, measuring the effectiveness of job training and placement programs seems easy: how many people got jobs, and did they hold onto them? But such simple measures beg a whole set of questions. How skilled were the candidates in the first place? Did they learn anything that helped them qualify for jobs? Did the job offer a decent skill match, reasonable pay, room for advancement?

Demonstrating effectiveness has long bedeviled employment training programs. The question is more important now than ever. Unemployment hovers around 9 percent, yet employers complain about lack of adequately skilled workers. With prospects for economic recovery uncertain, identifying and expanding solid programs that build workers’ skills has become a national imperative.

Taking up the challenge is Public/Private Ventures (PPV), a national nonprofit that uses research to improve programs serving low-income people. Through its Performance Benchmarking Project, PPV works to establish national performance standards for employment services, help funders compare outcomes, and help providers use data to improve what they do.

With support from the Fry Foundation and other local funders, PPV last year convened 22 Chicago providers to explore together how they can measure what they’re doing, examine the data to identify what is and is not working, and act on the results. The goal is to build what Senior Program Director Marty Miles calls “a culture of continuous improvement.”

i.c.stars, which trains young people for careers in technology, joined the collaborative even though, as a self-described “geek” organization, they were already measuring everything that moves. That was part of the problem, says President and co-founder Sandee Kastrul: people all over the organization had created tracking systems, but there was no “big picture” view. Just as frustrating, there was no way to measure the accuracy of i.c.stars’ core insight, that young people who have overcome challenging life situations often have the resiliency it takes to succeed in the tech world.

Drawing from its own experiences and the needs of others in the collaborative, i.c.stars created software that collects data on candidates, referral sources, programs, and outcomes and creates a dashboard that organizations can use to track progress. A private consulting group, spurred by an employee who came from i.c.stars, is now marketing the software to help other nonprofits improve their performance. Meanwhile, says Miles, i.c.stars’ interest in “resiliency” led to a thoughtful conversation about measuring intangibles that sparked creative thinking among others in the collaborative as well.

A different measurement challenge faces Chicago House. Having started out providing end-of-life support for people with HIV/AIDS, Chicago House needed a jobs strategy once clients started living longer, says Employment Program Manager Cheryl Potts. Clients can face challenges similar to people served by other training providers: homelessness, substance abuse, poor job history. But HIV status creates its own employment barriers: needing to maintain public benefits to fund medications, getting time away from work to administer them—and facing down the HIV stigma.

“We can’t build relationships with employers and contact them about candidates, because if we do that, we are outing their HIV status,” Potts explains. Instead, Chicago House concentrates on job readiness, determining employment goals, overcoming barriers such as housing or substance abuse, offering volunteer and transitional jobs opportunities—everything up to the point of placement. This configuration creates its own measurement problems, as Chicago House seeks to demonstrate the social return on investment”—how it makes a difference in people’s lives.

Through the PPV collaborative, says Potts, “I was able to steal a lot of information—how to create a database, do data collection, motivate staff. I set up a dashboard and review it monthly, to see how we’re performing against our grants, or against other agencies.” The data lead directly to improvements. Seeing clients drop out of transitional jobs at its bakery, the program collected data showing that people were falling back into substance abuse once they got their first paycheck; now the program intervenes to help people over that milestone.

In a second phase of the collaborative that began in fall 2011, Chicago House, i.c.stars, and a dozen other providers will work to deepen such internal performance improvement practices. They will bring in staff teams and identify very specific focus areas, like what kinds of people benefit most from their services, how long training should last, keeping people engaged, working with employers. Together they hope to create the small but continuous day-to-day improvements that can really transform an organization over time.

PPV will be watching the progress and documenting what modifications most affect performance. Those lessons will be used by the Chicago funders who are supporting the collaborative to create a common set of definitions to help both funders and grantees evaluate performance. Ultimately PPV plans to use the results, along with those from a similar initiative in New York, to help improve performance management across the field and share the lessons with federal policymakers seeking to strengthen job training programs.