Evaluation is one of the key components of the design for instruction. In the evaluation models presented by Reiser and Dempsey, they said, “The major purpose of the evaluation was to determine the value or worth of the innovation that was being developed” (p. 96). The terms of formative and summative evaluation were coined in the 60’s and included in most instructional design models of that time. There were two key features noted in early instructional design models, objectives and the learners. Reiser and Dempsey expressed, “Formative evaluation focuses on learners’ ability to learn from the instruction, and to enjoy it” (p. 97).
Other Evaluation Models
The Logic Model
The Logic Model incorporates the theory of working backward, starting with the outcomes desired (Frechtling, 2007). Frye and Hemmer (2012, p. 294) report, “Some US funding agencies require this model when evaluating a program” (Frechtling, 2007). The Logic Model is considered to be a linear approach in educational planning evaluation (Frye, Ann W. & Hemmer, Paul A., 2012). This model is similar to that of Stufflebeam’s CIPP evaluation model, “but focuses on the change process and the system within which the educational innovation is embedded” (p. 294). The four essentials elements to this model include inputs, activities, outputs, and outcomes. The inputs are considered the resources available to a program. The activities could include “strategies, innovations or changes planned for the education program”(Frye, Ann W. & Hemmer, Paul A., 2012). The third component of output is a result of the activity (a “product”). The demonstration of new knowledge or a learned skill can be included in the final element of the outcome. Again, these outcomes can be specific to an individual or by group(s). The environment, or the context, also plays a vital role in the evaluation of a program. The impact of a program is also an extended component to consider during evaluation.
Outcome-Based Evaluation Model
Outcome-based evaluation (OBE) involves the expression of goals wanting to be achieved by use of the program and services and reflects the effectiveness and efficiency to the referenced outcome. Wang clarifies that “OBE focuses on the benefits gained by clients from using the program, and look at whether the client’s needs are met by the activities of the organization” (p.134). Those benefits could include an increase in knowledge, attitude change, a skill learned, and/or a form of change in the condition.
In the OBE evaluation, there are four types of evaluation used most often. The evaluations include program, effectiveness, impact, and policy.
The program evaluation involves three steps. The establishment of a baseline reflecting current outcomes. The desired outcomes are included in the second step with the organization indicating goals yearned to achieve. The final step to this evaluation is aligning the services to meet the outcomes of the organization. Wang says, “In order for a program evaluation to be successful, an organization must be willing to foster a culture of change, be accountable in developing strategic plans and performance goals, and develop a monitoring system (Mika, 1996).
The effectiveness evaluation is similar to the program evaluation with the exception of its judgment of accountability and outcomes to the comparison. This model is considered more a systematics approach. The analysis of its five steps includes performance goals, purpose and comparison condition, methodology, data collection and analysis, and person and organization-referenced outcomes (Wang, 2010, p. 145).
The impact evaluation, “focuses on the program’s impacts and determines whether those impacts can be attributed to the intervention being employed or the services being evaluation,” says Wang (2010, p. 146). In this evaluation, a comparison group is used with the evaluator measuring the difference in results, if any noted.
In a policy evaluation, the equity, efficiency, and effectiveness are all evaluated in the outcomes whether employed individually, by the program, or as a whole system. Five steps are included in this evaluation. The goals of the policy are identified, as well as the values, expression of intended outcomes are provided, the evaluation of the outcome, and feedback is provided as the final step.
Many programs are scrutinized for the results and it impacts many organizations. Wang says, “They are using outcome-based data as a basis for making changes to improve services and increase accountability” (p. 148).
To Sum Up
Wang said it clearly, “No evaluation model can be all things to all people in all situations,” (p.148). He proceeded to share the advantages and disadvantages of each model explored this week in our reading Trends and Issues in Instructional Design and Technology. The case of the matter is finding the most trusted model that relates the verifiable results of the learner in measuring the objectives or goals mastered. In his conclusion regarding program evaluation, Wang remarked, “Evaluation models that provide practical tools and easily understood directions leading to verifiable results in a cost-efficient manner will be of the utmost importance” (p. 149).
I want to be able to effectively put the model to practice while not interfering with its validity. Depending on the need for evaluation, whether it be for quantitative measures or for program evaluations and identifying areas for improvement, I would most likely refer to the Kirkpatrick or CIPP Model for future reference. I’ll use the model that examines the program related to the goals and that of the learners. Their needs hold the highest stakes!
Other Questions and Measures to Consider
When considering the implementation of an instructional program, one important aspect to be considered is the amount of training provided to the facilitator. Not only considering the onset of the program but what will continuous support, if any, look like? What happens when those formative assessments show little to no growth for some learners? What support will be available in providing intervention? A few other components to consider during an evaluation include; the training knowledge of that instructor (Is he applying the new knowledge of his learning of this new program(lack of transfer)?), the environment, and the attendance of the students should all be considered.
Performance
A performance problem in my area would be tardiness and absenteeism. It was a school-wide problem. From my own experience, there were many students with more than 15 tardiness and 20+ absences. With gaps in learning, as a result from missing instruction, education growth regressed or didn't advance. A non-instructional solution put in place by the administration was an incentive program for those classes that reported on time and with no marked absences. At different levels of meeting goals, classes earned prizes (i.e. an extra recess, popcorn celebration, etc.). It was an attempt to encourage attendance while promoting teamwork in the classroom. I can’t help but wonder about the other uncontrollable factors in this problem of tardiness and absenteeism, that stems from the environment outside the walls of a school. Is it possible to consider (with educators) the parents and community members as stakeholders in the human performance problem of our students? The value of knowledge can be underrated in some environments. When considering this problem, one has to think about tacit and common knowledge, or lack of, as the root of the problem. Therefore, better support systems related to performance and knowledge would especially be beneficial in this case, where we consider everyone a stakeholder in the performance of our learners.
References
Frechtling J., (2007). Logic modeling methods in program evaluation. San Francisco: John Wiley & Sons.
Frye, Ann W. & Hemmer, Paul A. (2012). Program evaluation models and related theories: AMEE Guide No. 67. Medical Teacher 34, e288-e299. DOI: 10.3109/0142159X.2012.668637 Retrieved from https://pdfs.semanticscholar.org/1667/a06366e1f78159fe724c1115963eb5aba1df.pdf
Mika, K.L. (1996). Program outcome evaluation: A step by step handbook. Milwaukee, WI: Families International.
Reiser, Robert A. & Dempsey, John V. (2012). Trends and Issues in Instructional Design and Technology (3rd ed.). Boston, MA: Pearson Education, Inc.
Wang, Victor C.X. (2010). Evaluation Models for Evaluating Educational Programs. In Assessing and Evaluating Adult Learning in Career and Technical Education (Chapter 7). Retrieved from http://web.utk.edu/~ewbrewer/pdf/books/Evaluations%20Models%20for%20Evaluation.pdf
I found your description of the logic model to be well-written and provided a great amount of detail about this model. In one of my graduate courses, we participated in a backwards thinking exercise. I found this to be very applicable to how we plan for instruction as educators. I like the Outcome Based Evaluation Model as well when considering the impact on clients. Our current district began pushing new initiatives in what they call “customer service” or the way we serve parents and students in the district. At first, this way of thinking was very contradictory to the field of education in my opinion, but over time, I began to see how it aligned to the business model of serving customers. The OBE model would be helpful in evaluating how well our “clients” are served in the education realm. Furthermore, it would be very important to consider what steps would be taken if the evaluation of our customer service practices showed our performance to be subpar as you mention in the other considerations section of your blog post. On a final note, your description of tardiness/absenteeism is a great example of human performance, and I couldn’t help but agree with the role of stakeholders in this problem. Maybe an incentive program that benefited both the students and other stakeholders would yield greater success.
ReplyDeleteThe learning that is taking place as I read each person’s take on various models is abundant. I can to learn about other models, some of which I had not found in my research, and they help me to have a clear picture of how each type plays into an organizations business plan. Your reference to the model’s explanation of “policy evaluation, the equity, and effectiveness are all evaluated in the outcomes whether employed individually, by the program, or as a whole system,” clarifies how these models help employers to evaluation the performance of the workforce. The wonder of this whole process is in the outcome to have that ideal worker who will be an asset to the company and to help produce a marketable product. As you quoted from Wang “they are using outcome-based data as a basis for making changes to improve services and increase accountability.”
ReplyDeleteVery good point made about the amount of training needed to implement a program. Far too many organizations buy into a program without adequate training for the employees who must use that program. We see it far too many times in our own school districts. This problem of lack of training is found more often in the public sector, public utilities companies, school districts, public transits, etc.; than in private organizations where loss of time in non-effective training is important to them.
Absenteeism and tardiness is a real problem for employers as well. School is expected to be the training ground so that children will learn the value of promptness and attendance. It’s not only the huge gaps in learning that exist but their lack of learning the value of attendance and punctuality.
You are so right about those outside influences, such as: lazy parenting; children observing parents who choose not to go to their own employment because they didn’t feel good or found something else that they wanted to do and abused their company’s time off policy. Children learn best by example. Along with parents who allow children to have any excuse to be late or miss classes if they don’t want to get up for school. This opens a whole new set of problems.
Parents are getting younger and younger and they never had a complete raising; now they are left to raise their children without the proper skills. It’s a vicious cycle that must be addressed because it is causing the employer millions of dollars per year (HR Hero, 2017). The problem is so broad that the government Workforce Commission is considering ways to address industries concerns. On their site, they list the issues and possible strategies that employers can use to correct the problem (OFM U. S., 2012). You are so right “everyone is a stakeholder in the performance of our learners.”
References
HR Hero. (2017, June). Absenteeism and Attendance of Employees. Retrieved from Hero.com: http://topics.hrhero.com/absenteeism-and-attendance/#
OFM U. S. (2012, June). Access Washington. Retrieved from Office of Financial Management : http://hr.ofm.wa.gov/workforce-data-planning/research-resources/best-practices-and-benchmarks/managing-attendance-issues
Overall, your blog was in-depth and clearly written. I developed a deeper understanding of the Logic Model and Outcome-Based Evaluation Model due to your reflection. I agree, that when looking at the affordances of a program, we need to look at the training that is provided to the educator. I have found that on my campus teachers are not using some of the digital resources that are offered due to lack of training or continued support. This is truly a disservice to our teachers and students. A program will never be successful if there is not proper education or clear expectations on it.
ReplyDelete