Program Goals and Measures Program Goals and Measures
How to set a program’s goals and the measures that show it is working.
What this presentation covers
What this presentation covers
What this presentation covers
Start from a theory of change
Write down the change you want
MENTOR asks every program for a theory of change: how mentoring leads to results.
- Inputs mentors, staff, time
- Outputs matches made, meetings held
- Short-term outcomes skills, confidence, network
- Long-term outcomes retention, promotion
The four levels of evaluation
Measure on four levels, not one
Kirkpatrick’s model runs from how people react to the results that follow.
- Reaction was it useful?
- Learning what did they gain?
- Behaviour what do they do differently?
- Results what changed for the organisation?
Measuring more than satisfaction
Ask about usefulness, not only enjoyment
In 34 training studies, usefulness ratings tracked use on the job better than liking did.
- Did you enjoy it? a weaker link to use on the job
- Was it useful to you? a stronger link to use on the job
How often pairs meet
Count meetings: they are the first sign
Across 173 samples, meeting more often went with more psychosocial support.
Logging every meeting
Log each meeting the same way
The standard asks for each meeting’s date, length and activity, plus regular check-ins.
- Date day of the meeting
- Length in minutes
- Activity what the pair worked on
- Check-in twice in month one, then monthly
Define each measure
Define every measure before launch
Write each formula, source and target down, so every report counts the same way.
- Participation active pairs ÷ matched pairs
- Meeting rate meetings per pair per month
- Satisfaction average rating, 1 to 5
- Retention still here ÷ started, at 12 months
Leading and lagging measures
Watch early signs, report later results
Meetings and relationship quality show within weeks, retention and promotion take years.
- Leading sign-ups, meetings, relationship quality
- Lagging retention, promotion, skills in use
Measuring retention
Retention shows up years later
Among over 1,000 Army officers, mentoring went with less actual turnover ten years on.
Career outcomes like pay
Keep targets for pay modest
A meta-analysis found mentoring helps careers, but effects on objective outcomes were small.
- Objective pay: small effects
- Subjective career satisfaction
Taking a baseline
Take a baseline before the first match
A youth mentoring trial surveyed both groups at baseline and again 18 months later.
- The same survey before and after
- A comparison group when you can
- Records you already keep, like retention
- A fixed date for the follow-up
Sources
The research behind this presentation
- MENTOR: The National Mentoring Partnership (2015) Elements of Effective Practice for Mentoring, Fourth Edition MENTOR, Boston
- Kirkpatrick, D. L. & Kirkpatrick, J. D. (2006) Evaluating Training Programs: The Four Levels (3rd ed.) Berrett-Koehler, San Francisco
- Alliger, G. M., Tannenbaum, S. I., Bennett, W., Traver, H. & Shotland, A. (1997) A meta-analysis of the relations among training criteria Personnel Psychology, 50(2), 341-358
- Eby, L. T., Allen, T. D., Hoffman, B. J., Baranik, L. E., Sauer, J. B., Baldwin, S., Morrison, M. A., Kinkade, K. M., Maher, C. P., Curtis, S. & Evans, S. C. (2013) An interdisciplinary meta-analysis of the potential antecedents, correlates, and consequences of protégé perceptions of mentoring Psychological Bulletin, 139(2), 441-476
- Payne, S. C. & Huffman, A. H. (2005) A longitudinal examination of the influence of mentoring on organizational commitment and turnover Academy of Management Journal, 48(1), 158-168
- Allen, T. D., Eby, L. T., Poteet, M. L., Lentz, E. & Lima, L. (2004) Career benefits associated with mentoring for protégés: A meta-analysis Journal of Applied Psychology, 89(1), 127-136
- Grossman, J. B. & Rhodes, J. E. (2002) The test of time: Predictors and effects of duration in youth mentoring relationships American Journal of Community Psychology, 30(2), 199-219
Choose few measures and fix them early
Set the goal, the baseline and the target before the first pair meets.