One aspect of training evaluation is to seek small improvements in course design, delivery, and transfer with the objective of increasing impact. Continuous improvement should be one of the tools that underpins the philosophies of training evaluation. Through constant study and revision of evaluation results, better training occurs, resulting in increased impact (value to the business). Continuous Improvement is a set of activities designed to bring gradual but continual improvement to training results through constant review. The principles that underlie continuous improvement using training evaluation are the following: Focus on the stakeholder. Training is designed, developed, and delivered to meet stakeholder requirements and needs. To accomplish this, stakeholders were solicited in the planning phase and their requirements were built into the evaluation. A decision is then made by stakeholders as to proceed with training, based on the attractiveness of benefits (value) to be gained. Training results are then measured from the perspective of the stakeholder. A preventive approach. "Do it right the first time" instead of correcting through evaluation (inspection) if possible. This requires a systematic instructional design process (such as ADDIE [Analysis Design Development Implementation Evaluation]) to ensure that the training will deliver the results that stakeholders require. Management by data. Fact rather than supposition drives continuous improvement actions. Evaluation data communicated in the dashboards against respective Success Gates—direct decision in improvements and outcomes. Commit to ongoing improvement. Each time that Success Gates are reached, consider setting new and improved success measures. Training Evaluation is not a project or a task with a definite end point, but an ongoing commitment to seek out opportunities to deliver better results. Cross-functional problem solving. Training problems do not usually fall neatly into a single function such as design, development, delivery, management support, etc. They usually call for solutions that cut horizontally across organizational functions. People with different responsibilities and talents need to develop habits of working together to improve training and its results. Constancy of leadership commitment. A commitment to using evaluation as continuous improvement must be a way of life, built into the fabric of the company. In most organizations, this represents a fundamental change in culture and values. Tenacious and visible commitment of top leadership to evaluation is required if employees inside and outside of the training function are to be motivated to make the investment to change. Previous posts in this series: Getting Started Questions Evaluation will Answer Elements in an Evaluation Plan Evaluation Deliverables & Schedule Performing the Evaluation Coming up in this series of posts: Training evaluation organizational readiness As always, I would enjoy hearing your thoughts on this topic.

Many times I have T&D professionals ask me the steps I take when evaluating programs.  Here are the steps I follow: But this is only the framework for conducting an evaluation, you should create an Evaluation Project Plan.  This is a formal, approved document used to guide both project execution and project control. The primary uses of the Evaluation Project Plan are to document planning assumptions and decisions, facilitate communication among stakeholders, and document approved scope, cost, and schedule baselines. Currently I am about to begin Level 3 Evaluation project for a client.  Here is the project plan we just agreed to: This gives you an idea of the time per task/milestone and the activities (tasks) for each step in the framework.

Failure to do proper training evaluation planning causes poor efforts that usually include rework, the project being over budget, and delays.  The Plan phase is the starting point of the training evaluation.  Planning an evaluation is specifying what and how you’re going to evaluate by reviewing the BOC-1 program, developing the project’s mission, creating the evaluation objectives & deliverables, developing the questions the evaluation will answer, recording assumptions, documenting boundaries, identifying stakeholders, developing the strategy, estimated costs & timelines, and performing risk assessment.  From this a detailed evaluation project plan with milestones is created. I begin all my evaluations with with a detailed plan in which I include a steering committee to build it.  The steps I use with the steering committee are: Determine the evaluation methodology Understand the course being evaluated Mission of the evaluation Objectives & deliverables Questions the evaluation will answer Assumptions Boundaries Stakeholders On-the-job behaviors Beliefs Strategies (data collection, data analysis, reporting) Timeline (project plan & key milestones) Over the next few weeks I will post how to do each step.  I hope you find them useful and encourage you to give me your thoughts.

Just had a new article published with one of my best clients: Michael Yarter in Chief Learning Officer titled: How to Set Up an Internal Training Evaluation Department.  Michael and I have both set up internal evaluation departments and collaborated on what we call the "imperatives" for doing this properly to write this article. I hope you find it useful and would enjoy hearing your comments and experiences on how you have done this.

When you embark on creating a training evaluation plan one of the first steps is to determine the process or how the evaluation will be performed.  Whatever evaluation method you elect to use (Predictive Evaluation, The Kirkpatrick Model, The Success Case Model), the process to implement needs to be defined and accepted.  The process I use is: The Plan phase is the starting point of the training evaluation. Planning an evaluation is specifying what and how you’re going to evaluate by reviewing the program, developing the project’s mission, creating the evaluation objectives & deliverables, developing the questions the evaluation will answer, recording assumptions, documenting boundaries, identifying stakeholders, developing the strategy, estimated costs & timelines, and performing risk assessment. From this a detailed evaluation project plan with milestones is created. Developing evaluation procedures involves taking the approved evaluation plan from phase 1 and drafting the techniques, procedures, and instruments necessary to implement the plan. This phase of the training evaluation of entails (collect phase) gathering data to address the evaluation questions created in the evaluation plan. In the analyze phase the meaning of collected data are interpreted. This phase uses the collected data to determine the conclusions for which the data supports and the amount of support the data supplies for, or against, those conclusions. Quantitative & qualitative data analysis procedures are employed. The purpose of this phase (report) is to create the vehicles to communicate training evaluation information to stakeholders so that they can make use of the evaluation information for decision-making or purposes of accountability. This phase will generate an evaluation report and an executive dashboard detailing the results of the evaluation and the recommendations for the city to maximize the program’s impact. As always, I look forward to you comments and ways to improve this approach.

Last night I had the honor of speaking at the ISPI Atlanta chapter with Denise Traicoff from the Centers for Disease Control.  I did my usual speech on my Predictive Evaluation Model (PE) and Denise followed on how she and her team had implemented PE on one of their programs. She told the story of how her team had been given the assignment to teach leadership & management skills to groups of volunteers from around the world to go in a eradicate polio in countries.  It was a compelling story that kept all in attendance transfixed on her talk. Sandi Brown, Dave Basarab, and Denise Traicoff She shared how on her flight to the World Health Organization to begin the needs analysis she was reading my book.  She thought that due to nature of her course "the world deserved more than just happy sheets".  As she read, she got more and more excited about PE and shared that the could provide more rigor to her course measurement.  What made me proud was that she implemented Intention and Adoption evaluation from just reading the book.  What ever author wants. Denise and her team, including Sandi Brown (pictured), have now successfully implemented PE on three sessions and will continue to use it.  But PE is not the story here.  It is Denise and her team - what they have done.  Even more fun, was that last night was the first time we had met face-to-face even though we both live in Atlanta. I am honored to be associated with these talented people.  So in a small way sometime we do make a difference.  I had my moment last night.  Thank you so much Denise.  

The mission of an evaluation documents What we going to do? For whom are we going to do it? The mission itself defines what you are doing - how you’re going to do it is Evaluation Plan.  An example of an evaluation mission is…   As the word "mission" implies, the mission statement clearly states the evaluation’s goal. With this goal in mind, all stakeholders have an understanding of the evaluation direction and purpose.  A mission statement is one of the simplest and most basic necessities of a training evaluation, yet it is frequently overlooked by evaluators.

Once a mission statement (see this blog) has been developed, you can write your evaluation objectives. Note that objectives are much more specific than the mission statement itself and defines results that must be achieved in order for the overall mission to be accomplished. Also, an objective defines the desired end result. Deliverables are the items that will be produced by the training evaluation.  Items such as reports, dashboards, executive presentations, etc. are products typically named here. As example of objectives and deliverables in an evaluation plan: The following two questions are useful both in setting objectives and monitoring progress: What is the desired outcome? This is called the outcome frame. It helps you keep focused on the result we are trying to achieve, rather than on the effort expended to get there. How will we know when we achieve it?  This is the evidence question. This question is very useful for establishing exit criteria for objectives that cannot be quantified. What are your thoughts regarding objectives and deliverables with respect to training evaluation efforts?  

Now that you understand the course you are evaluating, have an agreed upon mission, objectives, and deliverables you can turn to the questions this evaluation will answer.  These are the questions that we will answer after we complete the evaluation. They drive the evaluation strategy and activity and must be aligned with the mission, objectives, and deliverables. Some examples for a transfer evaluation: What behaviors have been successfully transferred to the job? What is the rate of transfer?  What should it be? What is the profile of a graduate who has successful transfer? By creating these questions it keeps your evaluation focused on the things that matter and avoids scope creep that may occur.

Few evaluations begin with absolute certainty. If we had to wait for absolute certainty, most evaluations would never get off the ground. As evaluations are planned and executed, some facts and issues are known, others must be estimated. Estimation is an art, with many fine points to finesse between certainty and wishful thinking. You can’t just hope you have the resources you need to do the evaluation, and you can’t wait until every resource is available to begin. You have to manage and mitigate using informed assumptions. Assumptions fill in the gaps between known proven facts and total guesswork.  Each assumption is an "educated guess", a likely condition, circumstance or event, presumed known and true in the absence of absolute certainty.  Once identified, these assumptions and shape our evaluation plan. Consider this example: A defined budget is a fact.  i.e.  $10,000 has been allocated to complete a given evaluation. The belief that the budget is sufficient to complete the project on time and as required is an assumption.  This assumption should not be a guess.  It should be the result of a planned, verified budget estimate. From initiation to closure, assumptions set the stage for evaluation planning and execution.  As the evaluation is planned, assumptions are used to define and shape tasks, schedules, resource assignments and budget allocations.  As such, each is used to manage an otherwise uncertain future, laying out a roadmap for how the evaluation will proceed. So assume carefully…but document those assumptions as backup. Answer this question: What assumptions have already been made about the evaluation? What assumptions have you made regarding a training evaluation?  I would enjoy hearing your thoughts.