Training effectiveness
Refers to the benefits that the company and the trainees receive from training.
Training outcomes or criteria
Refer to measures that the trainer and the company use to evaluate training programs.
Training Evaluation
Refers to the process of collecting the outcomes needed to determine whether training is effective.
Evaluation Design
Refers to the collection of information-- including what, when, how, and from whom-- that will be used to determine the effectiveness of the training program.
Formative Evaluation
Refers to the evaluation of training that takes place during program design and development. Helps ensure that: the training program is well organized and runs smoothly and trainees learn and are satisfied with the program.
Pilot Testing
Refers to the process of previewing the training program with potential trainees and managers or with other customers.
Summative evaluation
Refers to an evaluation conducted to determine the extent to which trainees have changed as a result of participating in the training program.
The evaluationg process
- 1. Conduct a Needs Analysis
- 2. Develop Measurable Learning Objectives and Analyze Transfer of Training
- 3. Develop Outcome Measures
- 4. Choose and Evaluation Strategy
- 5. Plan and Execute the Evaluation
Kirkpatrick's Four-level Framework of Evaluation Criteria
- 4. Results: Business results achieved by trainees
- 3. Behavior: Improvement of behavior on the job
- 2. Learning: Acquisition of knowledge, skills, attitudes, behavior
- 1. Reactions: Trainee satisfaction
Reaction Outcomes
Refer to trainees perceptions of the program, including the facilities, trainers, and content.
Evaluation Outcomes
- Reactions
- Learning or Cognitive
- Behavior and Skills
- Affective
- Results
- Return on Investment
Cognitive Outcomes
are used to determine the degree to which trainees are familiar with principles, facts, techniques, procedures, or processes emphasized in the training program.
Skill-Based outcomes
Are used to assess the level of technical or motor skills and behaviors.
Affective Outcomes
Include attitudes and motivation.
Results
Are used to determine the training program's payoff for the company
Return on Investment (ROI)
Refers to comparing the training monetary benefits with the cost of the training.
Direct costs
Include: salaries and benefits for all employees involved in training
Indirect costs
Are not related directly to the design, development, or delivery of the training program. General office supplies, facilities, equipment, and related expenses.
Benefits
Are the value that the company gains from the training program
-
Criteria relevance
Refers to the extent to which training outcomes are related to the learned capabilities emphasized in the training program.
Criterion contamination
Refers to the extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions.
Criterion Deficiency
Refers to the failure to measure training outcomes that were emphasized in the training objectives.
Reliability
Refers to the degree to which outcomes can be measured consistently over time.
Discrimination
Refers to the degree to which trainees' perfomance on the outcome actually reflects true differences in performance.
Practicality
Refers to the ease with which the outcome measures can be collected.
Training Evaluation Practices
Training Program Objectives and Their Implications for Evaluation
Threats to validity
Refer to factors that will lead and evaluator to question either the believability of the study results or the extent to which the evaluation results are generalizable to other groups of trainees and situations
Threats to validity
- Internal Validity: Company, Persons, Outcome Measures.
- External Validity: Reaction to pretest, reaction to evaluation, interactions of selection and training, interaction of methods.
Pretraining measure
Baseline of the outcome.
Posttraining measure
Measure of outcome after training
Comparison group
Refers to a group of employees who participate in the evaluation study but do not attend the training program
Hawthorne Effect
Refers to employees in an evaluation study performing at a high level simply because of the attention they are receiving.
Random Assignment
Refers to assigning employees to the training or comparison group on the basis of chance.
Comparison of Evaluation Designs
- Posttest Only
- Pretest/posttest
- Posttest only with comparison group
- Pretest/posttest with comparison group
- Time series
- Time series with comparison group and reversal
- Solomon Four-group
Factors that influence the type of evaluation design
Change potential, Importance
Scale, Purpose of training, Organization Culture
Expertise, Cost, Time frame
Cost Benefit Analysis
The process of determining the economic benefits of a training program using accounting methods that look at training costs and benefits.
Utility Analysis
Cost benefit analysis method that involves assessing the dollar value of training based on estimates of the difference in job performance betwee trained and untrained employees, the number of individuals trained, the length of time a training program is expected to influence performance, and the variability in job performance in the untrained group of employees.
Success cases
Refer to concrete examples of the impact of training that show how learning has led to results that the company finds worthwhile and the managers find credible.
Training Metrics
Expenditure/employee, learning hours recieved/employee, expenditure as a % of payroll, expenditure as a % of revenue, cost/learning hour received, % of expenditures for external services, Learning hours received/training and development staff member, average % of learning activities outsourced, average % of learning contant by content area, Average % of learning hours provided via different delivery methods.
Is a process of previewing the training program with potential trainees and managers with other customer?
6-18 DeterminingWhether Outcomes areAppropriate Criteria Relevance The extent to which training outcomes are related to the learned capabilities emphasized in the training program. Criterion contamination - the extent that training outcomes measure inappropriate capabilities or are affected by extraneous conditions.
Training evaluation is the systematic process of collecting information and using that information to improve your training. Evaluation provides feedback to help you identify if your training achieved your intended outcomes, and helps you make decisions about future trainings.
trainees' perceptions of the program, including the facilities, trainers, and content (Reaction outcomes are often referred to as a measure of "creature comfort."
Level 1: Reaction
The first level of criteria is “reaction,” which measures whether learners find the training engaging, favorable, and relevant to their jobs. This level is most commonly assessed by an after-training survey (often referred to as a “smile sheet”) that asks students to rate their experience.