Meaningful Assessment and Continuous Improvement: Using the

To view this page ensure that Adobe Flash Player version 9.0.124 or greater is installed.

Please login or register to make a comment!

23 Harvard Family Research Project The Evaluation Exchange X 1 Claudia Weisburd and Rhe McLaughlin of Foundations, Inc., describe their Quality Assurance System for program im- provement. S ince 1992, Foundations, Inc., has operated extended-day enrichment programs and provided technical assistance to schools, school districts, and other education and commu- nity organizations. Working primarily with children from low- income communities and those who serve them, Foundations seeks to improve program performance and enhance student achievement in school and during non-school hours.

In 1999, after a rapid expansion in our after school opera- tions, we at Foundations found ourselves confronted with what is now a widespread challenge: maintaining program quality across states, sites, and highly varied on-the-ground conditions. Snapshot evaluations or one-time ratings were clearly insuffi- cient. We needed a system that would establish the highest stan- dards and at the same time help staff attain and maintain those standards through a process of tailored, targeted, and continu- ous improvement.

After three years of piloting and revising, Foundations developed the Quality Assurance System (QAS) evaluation tool. The QAS is an assessment tool that can be tailored to indi- vidual programs to build a comprehensive picture for planning program improvement. It is designed to ... more. less.

show change over time, instead of merely showing where a program happens to be on assessment day. Using the QAS, programs begin with an initial assessment. Based on the results, assessors and program staff identify areas in need of improvement and develop specific im- provement strategies. The program then conducts a follow-up assessment, paying particular attention to those elements tar- geted for improvement. The QAS is designed to be used as either a self-assessment tool or a tool that outside evaluators can use. (Outside assessment offers the opportunity for program-specific expert help. By comparing programs across sites, the reports generated through outside assessment create a picture of how well programs measure up against each other; these reports may also be particularly useful for formal reporting requirements.) Offering Flexibility While Ensuring High Standards The design of the QAS is based on our belief that if staff are to embrace assessment-based improvement plans, they must see the process as relevant, contextual, and potentially helpful. A hallmark of the QAS therefore is its flexibility. The tool starts cfrom where programs are, d by assessing their missions, con- tent, and resources. At the same time, it emphasizes best prac- tices for after school. This combination leads to high quality, re- alistic, and site-specific improvement planning. The two-part program profile assesses the context of the program. Part I addresses items essential to all after school pro- grams, including staffing, facilities, and health and safety. These cprogram-basics building blocks d are examined for all pro- grams regardless of particular program content. Part II narrows the lens to analyze cprogram-focus building blocks, d such as program mission, target population, and ac- tivities, with specific sections applicable to academic programs, recreation programs, and youth development. The assessor us- ing the QAS completes only those sections corresponding to specific program goals. This program-focus section allows pro- viders to tailor the assessment to the actual goals and activities of their programs. This specificity frees programs from being penalized for failing to provide content that was not part of their mission. For example, a recreation program will not re- ceive a lower rating because it scores poorly in academic con- tent, if academics were not part of the program goals. While the QAS deliberately accommodates a range of pro- gram types, it establishes clear, high standards through the close assessment of building blocks. Each building block is divided into composite elements that are evaluated on a scale of 1 to 4. Element scores are summed to provide a total score for each building block. Within the staffing building block, for example, the assessor looks at elements such as qualifications, atten- dance, and staff-child ratios. The level of detail provided by these elements and building blocks allows staff to easily identify both program strengths and weaknesses. This information can then guide targeted improvement planning. Current and Future Progress Since the debut of QAS in April 2003, over 150 programs have participated in its processes, which have involved approximately 1,000 staff members in evaluation, reflection, and program im- provement. With the launch of a web-based version of the tool (slated for spring 2004), sites will have the ability to collect data, analyze findings, plan change, and produce reports online. By blending meaningful assessment with continuous improve- ment, the QAS has encouraged and will continue to support the development and maintenance of high quality after school pro- grams across diverse sites. For more information about evaluation of the Foundations, Inc., after school programs, visit the HFRP Out-of-School Time Program Eval- uation Database at www.gse.harvard.edu/hfrp/projects/afterschool/ evaldatabase.html. Claudia Weisburd, Ph.D. Director of Special Projects Email: cweisburd@foundationsinc.org Rhe McLaughlin Director of Evaluation Email: rmclaughlin@foundationsinc.org Moorestown West Corporate Center 2 Executive Drive, Ste. 1 Moorestown, NJ 08057-4245 Tel: 856-533-1600 Website: www.foundationsinc.org Meaningful Assessment and Continuous Improvement: Using the Foundations Quality Assurance System > promising practices

less