Program Evaluation Project
Kimberly Owens
University of Oregon
Family & Human Services 495
February 21, 2012
Evaluation
Overview
Senior Project
For the purposes of this program
evaluation, I will be conducting the program evaluation of my senior
project. My senior project is based on a
case study featuring two students who attend Adams Elementary School. The case study is intended to group two peers
together, one student in the Life Skills Program who is at the 3rd
grade level, and one student in the general education 5th grade
classroom. The purpose of the project is
to help the student in the Life Skills Program increase her gross motor
abilities, as well as increase her comfort and confidence in social settings. By pairing the two students together, the
purpose is to naturally increase her social skills while working towards
outlined goals with a peer. Although
this is a single case study, the target population for future contexts is a peer
mentorship between any groups of students at Adams Elementary, to increase
certain skills in a social setting. Also,
because this study is implemented during school recess, other students will
likely volunteer themselves to be apart of their social interactions and
activities they are apart of. For
example, the particular skill that week may be to increase balance and
coordination, so a game of hopscotch will inherently involve other students.
The project will last the duration
of twelve weeks, during the first recess of each day during the 12 weeks. Each recess session lasts fifteen minutes and
I will facilitate the fifteen minutes to optimize the time allotted. The duration of the study is concentrated
upon this particular study. However, in
future peer mentorships, the duration would most likely occur throughout the
school year, with the group of students meeting once a week.
Certain complexities are worked into
this project due to the sensitivity of implementing a project into a structured
school setting. There are multiple
components because there are a total of three main classrooms students are
apart of and so involving all of the teachers is a priority of the project so
that their input and observations are noted.
Another complexity is that there are a few different goals and intended
outcomes with this project, and only one goal of increasing gross motor skills
that can be tested with direct before and after results.
Purpose of Program Evaluation
The purpose of this program evaluation is
to measure effectiveness of a peer mentorship program at Adams. This project is only a case study and so it
is a model of what a future buddy system could look like. Data is required in order to implement a permanent
model of the peer mentorship case study.
The evaluation will answer certain questions that will answer whether my
project is feasible and realistic to implement at this time. I will be utilizing a specific approach
called the Transformational Evaluation because it uses mixed methods; it has a
social justice focus, as well as inclusion and self-improvement (In Class
Lecture, 1/24/12).
Evaluation questions.
The broad questions that this program
evaluation is based off focus on the gross motor outcomes and her social skill
comfort and confidence. The questions
also look at the possibility of creating a long-term peer mentorship program at
the school. For example, these questions
are: Does the time allotted allow for
activities to be facilitated and carried through? Do both students benefit from a peer
mentorship program? Did the student
working on gross motor skills make improvement on at least five out of ten
tasks that were apart of her Independent Education Program (IEP) plan? Has the student shown progress with her
ability to socialize with her peers in the classroom? Do the students engage in
a similar manner during the lunch recess or do they go off in the separate ways
when a facilitator is not present?
Program Staff & Stakeholder Involvement
The teachers are a critical
component for designing and implementing in terms of the program evaluation. Not only do the teachers play a significant
role in the evaluation process, other professionals such as the occupational
therapist (OT) and the speech pathologist also play a role in the evaluation. The OT understands gross motor skills
development and so her involvement is critical for the gross motor
assessment. Also, the speech pathologist
works with the student weekly on her verbal skills and so the speech pathologist
will be able to participate in a survey of before and after questions to see if
the student has improved verbally as well as socially, for example, whether the
student speaks more clearly, or loudly after time. The Special Education Director, who is the
lead in the Life Skills Program, is a very important aspect of the program
evaluation because she can speak to the student’s IEP goals that include gross
motor skills. Not only can she speak to
the IEP goals, but she is the professional who teaches the student daily and
would be able to evaluate her progress in all aspects. The program staff will be involved with
completing surveys of their observations of the student as well as provide
their overall input on the success of the project and any possibilities of a long-term
implementation of a peer mentorship program at Adams. Their participation is crucial in order to
evaluate the program successfully.
In order to determine the success of the
senior project, the input of other professionals and family members is
important. The school administrator may
be concerned with how well a peer mentorship program would operate long-term
and how well the different components would operate in the future (Unrau,
Gabor, Grinnell, 2001). The
administrator would provide feedback on whether she believes this project would
be useful in her school and the details of the framework of a future permanent
program. Students’ family members are
also key stakeholders in the program evaluation because they ultimately will determine
if they want their student to be involved with a peer mentorship program, and
so their participation in the evaluation and their feedback is critical. Due to the sensitivity of students in the
Life Skills Program, families need to be fully notified of the project and of
the program evaluation. The families
determine if the project meets their student’s needs or if they prefer a
different route to meet their student’s needs.
Cultural Differences and Diversity
The project in itself
focuses on cultural differences and diversity.
The purpose of the project is to include students with differences in a
common place that naturally focuses on social integration. The student’s social nature needs to be taken
into consideration when measuring her comfort and confidence in social
situations. Sensitivity needs to be
taken into account when determining what her abilities are in a social
environment because her capabilities may mean something completely different
when interacting with people she is familiar with. She may not feel comfortable in this
situation and so the project may not be an ideal situation for her to work on
her social skills and gaging how she does in the situation. For the program evaluation, it is important
to be sensitive towards utilizing the evaluation. Due to the fact that the student’s gross
motor skills are apart of her confidential IEP plan, the outcomes of these
goals need to be taken into consideration.
The program evaluation can be used to inform staff and educate people
but confidentiality needs to be highly prioritized (In class lecture,
1/31/12).
The transformative evaluation is strengths
based. This program evaluation will
highlight both student’s strengths in ways that the student otherwise would not
have the opportunity to show to their teachers. During integration of the student in Life
Skills, it is important that she feels comfortable when interacting with her
peers in the general education classroom. The involvement of the student’s families is a
cultural and diversity issue as well and the program evaluation needs to be
sensitive to their wants and needs.
Data
Collection Tools and Strategies
Types of Data
There
will be three types of data that will be collected that will coincide with the
prior evaluation questions. Due to the
nature of the senior project and considering it is a single case study,
feedback from the stakeholders involved as well as participants will be
extremely important in order to expand and continue the program in the
future. Feedback is not a traditional
method of intake data, however, while it is collected as an outcome, it can
reflect the true perspective of the client (Unrau et al., 2001). Understanding whether the client enjoyed the
project as well as gained new skills from the project are two critical aspects
of the program evaluation that can be determined by collecting feedback.
Another
focal point of the data collection is the service statistics. From the program evaluation it will be
significant to know whether or not the services provided by the project were
successful. In order to implement the
project long term, administrators and teachers will need to understand that the
services are realistic and useful. The
professionals overseeing the project will need data to ensure that the results
obtained correspond to the original expectations set forth (Unrau et al.,
2001).
The
student results in the senior project are the most significant factors to
consider in the program evaluation. The
purpose of the senior project is intended to increase the student’s ability in
gross motor skills as well as provide more comfort to engage socially. Client outcomes are the main purpose of
social service organizations and their reason for existence (Urau et al.,
2001). In relation to the evaluation
questions, student outcomes correspond to the questions about the progress the
student has made socially and with her gross motor capabilities. It is crucial to take client outcome data for
the senior project because it will note whether it is a realistic goal to implement
the program in the future.
Strategies
The first data strategy
that the program evaluation will focus on is existing data. The Individualized Education Program (IEP)
plan will serve as the existing data to obtain information on the abilities and
goals of the student in the Life Skills Program. Through the IEP, the program evaluation will
determine before and after assessments about the student’s capabilities and any
improvement she has made. It is
important to have the prior knowledge of her abilities to know how much growth
has been made, if any. Since the
existing data was created for another purpose other than the senior project,
the information of gross motor skills will be unbiased (Unrau et al.,
2001).
The second strategy the program evaluation
will use to collect data will be observation checklists. It is crucial for the project facilitator to
take notes diligently about the activities each day and progress that was made
throughout the project. Observation is
particularly important for the social skills aspect. Considering a child’s social environment
affects how a child will engage socially, observation will help to measure
social comfort and confidence.
Observation differs from other forms of data collection because it
depends on person observing to record what was seen (Urau et al., 2001). Structured observation would entail paying
close attention to detail and watching the two students amongst the rest of the
students during recess. The observing period is flexible as to whether it is a
natural or artificial setting, however, the time period must be predetermined
(Urau et al., 2001). In this case of the
peer mentorship, observing allows the flexibility to obtain the data that is
needed in order to determine program success.
The observer will take qualitative data so that social skills can be
fairly recorded and the gross motor abilities can be taken with quantitative
data so the student’s achievement can be measured precisely (Urau et al.,
2001).
Lastly, the third strategy used for the
program evaluation in order to sufficiently collect data is through a
survey. The survey will be given to the
professionals who work directly with the student in the Life Skills Program to
measure outside assessment of success.
Surveys serve as the method to collect the opinions of a collective
group. With surveys come challenges of
reliability and validity of the questions asked (Urau et al., 2001). In order to reduce the challenges of using
surveys, there will be more open-ended questions than closed-ended questions to
ensure that the people taking the survey have the opportunity to provide their
insight and opinions.
The three strategies to collect data were
chosen because the project is unique in the sense that it is a case study and
requires flexibility in the program evaluation.
Existing data provides the framework for the goals that the project will
build upon to help the student reach their certain IEP goals. It is important to have the existing data so
that the facilitator understands the foundation the student is working
upon. The observation is crucial to
determine the reality of the project and to collect data on various aspects of
the project. Lastly, the survey will
involve all professionals who work directly with the student which ensures that
the student is optimizing the services she receives in the classroom and builds
upon them participating in the project during recess.
The observation forms will provide
information on whether the project is successful throughout the duration of the
12 weeks. For example, if the student
gravitates towards the fifth grade peer, and has a positive outlook on recess
time and participating in the activities, I will know that the project is
progressing in a positive direction. If
the student is making progress with her gross motor skills, the project is also
working towards the desired outcomes. Teachers’
input about her social skills will also determine whether or not the project is
of benefit to the students. Not until the
end of the 12-week project, will it be known whether or not the project was
successful and feasible to implement in the future.
Instruments Used
See
Attached (page 12)
Data Collection
The data will be collected in various
methods depending on the type of data that is collected. For example, the facilitator will write the
observations each day after the fifteen minutes of the recess activities. The facilitator will note the important
details, how the students reacted to the activities, and the performance and
engagement of the students involved. The
teachers directly working with the student will also be asked prior to starting
the project to write observations throughout the 12-week period if he or she
recognizes any changes in behavior or gross motor ability. The survey will be given to the family of the
student in the Life Skills Program to fill out with the help of their student,
the teachers, and the peer mentor student.
I will be monitoring the observation data collection throughout the 12-week
period to make adjustments to the project if needed. I will collect the data weekly and combine
any observations made by myself and by the teachers to list after each of my
weekly observations. The survey will be
used to determine if there is the possibility of implementing a full program
based on peer mentorships.
Data
Analysis and Reporting
Analyzing Data
I will summarize and analyze the data
collected by using the existing data as a benchmark for where the student was with
her gross motor abilities. By referring
to my observation forms, I will determine the progress the student has made
each week and by the end of the project.
At the end of week 12, I will write observations based on the existing
data assessments in order to measure the student’s progress. I will be the sole person responsible for
summarizing my observations. However, I
will discuss the other’s observations with them to ensure my interpretation of
their observations. From that point on,
I will also summarize their observations and discuss with the lead teacher
about the progress made from the existing data through the end of the
project. I will refer to the lead
teacher to summarize the surveys and to interpret the family’s answers to the
questions.
Reporting Results
I will report the results of the program evaluation
by showing a graph from the surveys.
However, because this is a case study, the results will be more
qualitative based than quantitative based that can be shown visually with
graphs, etc.
Program Evaluation Audiences
The audiences for the completed program
evaluation are the students’ teachers at Adams, the school administrator, the
students’ families, as well as the audience during the Senior Project
Presentations. The professionals at
Adams are the focus of determining whether the project will be implemented in
the future. It is important for the data
to be precise and simple for people to digest and understand clearly.
The data will be used to improve the
project in many ways. For one, it will
determine whether the project will continue on and who will be its focus. For example, as of now the project is
designed to pair peers together who are from the general education classroom
setting and another student who is from the Life Skills Program. However, this project can easily be
implemented for students who vary in age, so for example, a fifth grader paired
with a second grader. In this situation,
the focus of the project would be for students to be peer leaders and tutor
that child with reading. The data will
also determine the use of this peer mentorship for other students and to
understand if the program needs to be case study focused or if more than two
students can be involved at one time. Overall,
the data will be used to implement the project into a long-term basis that
meets students’ needs and by having a case study model to base it off of, it
can hopefully expand and be successful in the future.
Observation Checklist
Name of Observer: __________________
Date: ___________________
Week: __________________
Students Participating:
__________________________________________________
Activity:
Gross Motor Skill:
Observations made (Include data of gross
motor skills):
Any other notes
References
Unrau, Y., Gabor, P. & Grinnell, R.
(2001). Evaluation in Human Services.
Belmont, CA: Wadsworth/Thomson Learning.
