Nine Exemplary and Thirty-three Promising Programs Identified by an Expert Panel - DRAFT
The Safe and Drug-Free Schools and Communities Program (SDFSC) of the U.S. Department of Education used an expert panel process to identify programs that should be promoted nationally as promising and exemplary. A 15-member Expert Panel oversaw a valid and reliable process for identifying effective school-based programs that promote healthy students and safe, disciplined, and drug-free schools. The Department and the Expert Panel on Safe, Disciplined and Drug-Free Schools have identified nine exemplary and thirty-three promising programs.
The Department will disseminate information about the programs and encourage their use in new sites. The Expert Panel initiative is a way of enhancing programs aimed at developing the health and well being of our nation's young people by making schools and communities aware of programs that have proven their effectiveness when judged against rigorous criteria.
A publication which features the 42 programs identified as either promising or exemplary by the U.S. Department of Education's Safe, Disciplined and Drug-Free Schools Expert Panel will be published soon. Information about the publication's availability will be posted on the SDFSC web site at:
http://www.ed.gov/offices/OESE/SDFS
The Safe and Drug-Free Schools Program hopes to institute a mechanism to continuously identify effective strategies, approaches and programs for strengthening the health and well being of youth.
Introduction
In 1994, Congress directed the Office of Educational Research and Improvement (OERI) to establish "panels of appropriate qualified experts and practitioners" to evaluate educational programs and recommend to the Secretary of Education those programs that should be designated as exemplary or promising. The purpose of these panels is to provide teachers, administrators, policymakers, and parents with solid information on the quality and effectiveness of programs and materials so that they can make better-informed decisions in their efforts to improve the quality of student learning.
The Safe and Drug-Free Schools Program (SDFS), in cooperation with the Office of Educational Research and Improvement, established the Safe, Disciplined, and Drug-Free Schools Expert Panel in May 1998. The 16-member Expert Panel was composed of educators, researchers, evaluators, program developers, and representatives from local and state education agencies, businesses, institutions of higher education, and from medical and legal communities. Its task was to develop and oversee a process for identifying and designating as promising and exemplary programs that promote safe, disciplined, and drug-free schools. The Expert Panel initiative is a way of enhancing prevention programming by making schools and communities aware of programs that have proven their effectiveness when judged against rigorous criteria. The activity is also in keeping with the "Principles of Effectiveness" governing recipients' use of funds received under the Safe and Drug-Free Schools and Communities Act, State Grants Program.
The Review Process
The Panel initially met to set up the process for making determinations and to establish the criteria under which programs would be reviewed. While there are no nationally accepted standards for safe, disciplined, and drug-free schools programs, the Panel drew heavily on the considerable research on "what works" in prevention programming in combating both substance use and violence among youth. The Panel developed seven criteria, under the four "criteria categories" provided in the legislation, for judging the efficacy and quality of programs that would be submitted for their review and consideration. These criteria follow this introductory section.
The Expert Panel had an open and widely publicized submission process that encouraged applications from any program that might meet the review criteria. A total of 132 programs were submitted. The review process consisted of several stages. In the first stage, 19 individuals with expertise in safe, disciplined, and drug-free schools programming as well as research and evaluation, were trained and formed a pool of field reviewers. Each program submitting a complete and appropriate submission was reviewed and scored for evidence of efficacy by two reviewers from this pool.
Programs with high scores on the evidence of efficacy criterion were then considered by two additional field reviewers. Reviewers in this second stage of the review process were selected from a different pool of 40 individuals, all having expertise in safe, disciplined and drug-free schools programming. The second phase review included the criteria categories of quality of program, educational significance, and usefulness to others. Programs that received low scores on the initial evidence of efficacy review were reviewed in this second phase by one field reviewer for the purpose of providing substantive feedback to the submitter.
Programs receiving high scores in the first two stages of review were subsequently sent for further review by two members of a group of national experts in evaluation/research design and analysis. This group of reviewers was established by the Department of Education to review the strength of evidence of program effects for all Expert Panels.
The full Expert Panel then reviewed all the programs, along with the ratings and comments of the reviewers, to determine which programs to recommend to the Secretary of Education as exemplary or promising.
This publication provides descriptions of the nine exemplary and 33 promising programs designated by the Secretary of Education in 2000. Contact information for each program is also provided. In the program summaries that follow, the sections "Program Description" and "Professional Development Resources and Program Costs" were prepared based on information provided by the developers. The remaining sections - "Program Quality" and "Evidence of Efficacy" - are based on the assessments of the reviewers and panelists.
Evaluation Criteria
The following criteria and indicators were used to evaluate the safe, disciplined, and drug-free schools programs submitted to the Expert Panel.
- Evidence of Efficacy
Criterion 1. The program reports relevant evidence of efficacy/effectiveness based on a methodologically sound evaluation.
Condition a.
The program evaluation indicates a measurable difference in outcomes that is based on statistical significance testing or a credible indicator of magnitude of effect. Relevant outcomes are factors related to making schools safe, disciplined, and drug-free: reducing substance use, violence, and other conduct problems and positive changes in scientifically-established risk and protective factors for these problems.
Condition b.
The program evaluation used a design and analysis that adequately controls for threats to internal validity, including attrition.
*NOTE: Some evaluation designs do not meet the criteria for Exemplary or Promising status. Such designs include the following
- pre-post designs without comparison groups;
- one-time, post-test only, comparison studies without randomization or other efforts to control threats to internal validity; and
- case studies without comparisons.
Condition c.
The program evaluation used reliable and valid outcome measures.
*NOTE: Some evaluation measures do not meet the criteria for Exemplary or Promising status. Such measures of program effects include the following:
- judgments based on clinical experience; and
- authoritative evidence such as reports by expert committees and testimonials.
Condition d.
The program evaluation used analyses appropriate to the data.
- Quality of Program
Criterion 2 (Goals). The program's goals with respect to changing behavior and/or risk and protective factors are clear and appropriate for the intended population and setting.
Condition a.
The program's goals are explicit and clearly stated.
Condition b.
The program's goals are appropriate to the intended population and setting.
Criterion 3 (Rationale). The rationale underlying the program is clearly stated, and the program's content and processes are aligned with its goals.
Condition a.
The rationale (for example, logic model, theory) underlying the program is clearly stated and includes appropriate documentation (for example, literature reviews and previous research).
Condition b.
The program's content and processes are aligned with its goals.
Criterion 4 (Content Appropriateness). *The program's content takes into consideration the characteristics of the intended population and setting
(for example, developmental stage, motivational readiness, ethnicity, gender, socioeconomic status,language, disabilities, culture) and the needs implied by
these characteristics.
*NOTE: Content appropriateness will be determined on the basis of the application narrative and the program materials submitted.
Criterion 5 (Implementation Methods). The program implementation process effectively engages the intended population.
Condition a.
The program provides a relevant rationale to participants for its implementation.
Condition b.
The program actively engages the intended population.
Condition c.
The program attends to participants' prior knowledge, attitudes, and commonly held conceptions.
Condition d.
The program implementation methods promote participants' collaboration, discourse, and reflection.
Where applicable:
Condition e.
The methods foster the use and application of skills.
Condition f.
The program promotes multiple approaches to learning. C. Educational Significance
Criterion 6. The application describes how the program is integrated into schools' educational missions.
- Usefulness to Others
Criterion 7 (Replicability). The program provides necessary information and guidance for replication in other appropriate settings.
Condition a.
The program clearly outlines the essential conditions required to replicate it with fidelity in other settings (for example, strategies, resources, implementation plans, materials, etc.).
Condition b.
The program includes guidelines and materials for training and supporting those who are to replicate it.
Design and content © WestEd
|
printer version