Evaluation

November 2003 DRAFT - "Program Owned Evaluation" as a Leadership and Management Tool in Civil Legal Services Programs

This report is a NLADA/AARP collaboration compiled by Martha Bergmark, Wayne Moore, Ken Smith, John Tull and John Scanlon.

• DRAFT – Full report (pdf, 838 Kb)
• Case 1: Mary Asbury, Executive Director, Legal Aid Society of Greater Cincinnati (pdf, 123 Kb)
• Case 2: Eric Avildsen, Executive Director, Vermont Legal Aid (pdf, 46 Kb)
• Case 3: Jeanne Charn, Executive Director, Hale and Dorr Legal Clinic (pdf, 160 Kb)
• Case 4: Ross Dolloff, Executive Director, Neighborhood Legal Services, Lynn and Lawrence, Massachusetts (pdf, 133 Kb)
• Case 5: Chuck Greenfield, Executive Director, Legal Services of Northern Virginia (pdf, 116 Kb)
• Case 6: Rhonda B. Lipkin, Deputy Director, Legal Aid Bureau of Maryland (pdf, 73 Kb)
• Case 7: Jan May, Director, Legal Counsel for the Elderly (pdf, 47 Kb)
• Case 8: Neil McBride, General Counsel Legal Aid Society of Middle Tennessee and the Cumberlands (pdf, 100 Kb)
• Case 9: Irene Morales, Executive Director, Inland Counties Legal Services (pdf, 72 Kb)
• Case 10: Robert Oakley, Executive Director, Northwestern Legal Services (pdf, 49 Kb)

Hotline Outcomes Assessment Study

The Project for the Future of Equal Justice has engaged the Center for Policy Research, Denver to conduct a study of the effectiveness of centralized telephone legal advice, brief service, and referral systems (“Hotlines”) in the delivery of civil legal assistance. The Center for Policy Research is a non-profit research firm that specializes in planning, developing, and testing projects to improve the effectiveness and efficiency of the justice system.

The Hotline Outcomes Assessment Study is being managed by Julia Gordon, senior staff attorney at the Center for Law and Social Policy, and Bob Echols, a legal services consultant.  The study is being guided by an Advisory Committee .

Phase I

The study consisted of three phases.  Phase I, completed in March 2000, used existing data to compare “before” and “after” caseload statistics in programs that had adopted Hotline systems to determine the effect of the Hotline system on the number of clients served and the levels of brief and extended services.  Both a summary of the Phase I conclusions and the full The Hotline Outcomes Assessment Study Final Report - Phase I (msword, 336 Kb) are available.

Phase II

Phase II of the study was conducted in the spring and summer of 2000 to explore the feasibility of answering these types of questions using survey research techniques. The specific objectives of Phase II were to develop and test a data collection procedure and instrument that would generate reliable information about Hotline outcomes. A draft data collection instrument was designed that could be administered over the telephone by non-lawyers to Hotline clients about their problems, their experience with the Hotline, and the outcomes of their calls; and it was tested at two Hotlines, Statewide Legal Services of Connecticut (SLS) and the Northwest Justice Project Coordinated Legal Education, Advice and Referral (CLEAR). Phase II led to a number of conclusions about the feasibility of assessing outcomes to legal problems using survey research techniques.

Based on the pre-test results gleaned in Phase II, PFEJ funded a full-scale survey of Hotline clients to learn more about whether clients understand the advice they are given, whether they follow up on it, and whether they realize a satisfactory resolution of their problems.

Phase III

Phase III of the Study researched whether clients understand the advice they are given by Hotlines, whether they follow up on it, and whether they realize a satisfactory resolution of their problems. The study methodology included:

• Generating samples of callers at five legal Hotlines that were representative of the total universe of clients served at legal services programs;
• Conducting telephone interviews with 2,034 callers three to six months after they contacted the Hotlines and eliciting their general reactions to the Hotlines, as well as the specific outcomes of their cases;
• Having experienced legal services lawyers generate both factual and evaluative assessments of outcomes, which were based on a review of case files and interview notes, including verbatim responses to questions about legal outcomes; and
• Analyzing the resulting data set to produce profiles of callers across the five sites and outcome patterns with special attention to the client, case, and advice characteristics of cases with favorable and unfavorable outcome patterns.

Both the The Hotline Outcomes Assessment Study - Final Report Phase III (pdf, 701 Kb) and the Executive Summary (pdf, 137 Kb) are available.

ADVISORY COMMITTEE

Jonathan Asher
Colorado Legal Services

Sandy DeMent
Advisory Communications Systems, Inc.

John Eidleman
Legal Services Corporation

Michael Genz
Legal Services Corporation

Wayne Moore
AARP Foundation

Susan Reif
Georgia Legal Services Program

MaryAnn Sarosi
Consultant

Documents linked from this page are in the PDF format and require the free Adobe Acrobat Reader in order to view or print them. Click to download the Acrobat Reader software.