“Keeping it Real”: An Evaluation Audit of Five Years of Youth-Led Program Evaluation

YouthREX Research Summaries ask Just Six Questions of research publications on key youth issues. These summaries get at what the youth sector needs to know in two pages or less!

1. What is the research about?
Adult centrism (an emphasis on adult perspectives and processes) can limit the relevance of social services for youth. Research suggests that participatory action (a democratic research process that answers questions by engaging relevant stakeholders) can significantly improve youth engagement. In particular, Youth Participatory Evaluation (YPE), which engages youth in the evaluation process, can allow programs to gain crucial insight into young people’s perspectives.

This article is about the evaluation of Stand Up! Help Out! (SUHO), an after-school youth leadership development program that focuses on helping youth respond to the challenges of living in disadvantaged urban neighbourhoods. The research examines YPE methods by summarizing lessons learned in three key areas:

i) how youth were engaged as program evaluators using a peer interview process;

ii) central features of the evaluation process; and

iii) how pitfalls were overcome to maximize the authenticity of youth’s evaluations.

2. Where did the research take place?
The research took place in Chicago, Illinois, in the United States.

3. Who is this research about?
This research is about African American youth living in under-resourced urban neighbourhoods.

“Youth-led program evaluation had other advantages. It created democracy within the program, optimized youth engagement, met youths’ needs for autonomy, allowed youths’ voices to be heard, and provided a way to add fun to the typically dull process of evaluation” (p. 296).

4. How was this research done?
The researchers conducted an evaluation audit in order to report on SUHO program evaluation processes in the three areas mentioned above. The evaluation team reviewed program evaluations from 2006 to 2011.

The program evaluations used qualitative methods, including interviews and focus groups. Youth developed interview questions, conducted qualitative interviews with their peers, and contributed to data analysis. A total of 203 audio-recorded interviews were transcribed and made available for the evaluation audit.

Youth members of the research team also conducted focus groups with youth who engaged in the evaluation process as either interviewer or informant. Focus group and interview data were analyzed to identify best practices for YPE.

5. What are the key findings?
There were six areas of major findings:

i) Incorporating Youth Feedback into Service Design
As the central goal of the evaluation was to seek feedback for improvement, it was critical to respond to young people’s suggestions and concerns.

ii) Youth-Developed Interview Process
The youth-developed interview process was improved based on feedback from youth and instructors, and a review of collected data. The modified questions served to elicit more complete responses (i.e., more specific suggestions for program change). Questions also aimed to elicit negative comments, so that even youth who ‘enjoyed’ everything and had no apprehensions about the program could identify drawbacks for others (e.g., talk about a part of the program they did not enjoy).

iii) Youth as Informants
The evaluations showed that youth have diverse informant styles. Eight categories of informant styles were identified:

a) standard informants give answers that are sufficient, but lack depth

b) articulate informants show genuine interest in the research process, and provide detailed responses

c) inconsistent informants offer conflicting responses

d) playful informants are more interested in eliciting a response from the interviewer than contributing to the evaluation

e) informants with learning disabilities have difficulties understanding questions and verbally processing their responses

f) apathetic informants are indifferent and give limited responses

g) rushing informants attempt to speed up the interview

h) nervous informants struggle to articulate their experiences

Training interviewers to manage different informant styles can improve the quality of data. Youth who participated in the evaluation process suggested that interviews should be shorter in length (6-10 minutes, with 10-12 questions), and that informants should be assured that data is considered valuable and kept confidential.

iv) Youth as Interviewers
The evaluations showed the importance of selecting interviewers carefully, and providing basic training. Training should include information on how to ask questions, develop rapport, and manage different informant styles. Seven interviewer personalities were described:

a) rapport-building interviewers make the informant feel comfortable and are able to elicit fuller responses

b) clarifiers use various techniques to prompt fuller responses (e.g., repetition, asking for examples)

c) leading interviewers provide informants with potential responses

d) impatient interviewers react when a respondent fails to answer the question clearly or concisely

e) playful interviewers give informants a ‘hard time’

f) rushing interviewers are not concerned with the quality of the answers, and sometimes skip questions

g) exhausted interviewers experience a decline in interview quality after conducting a number of consecutive interviews

v) Youth Feedback on the Participatory Evaluation Process
Young people’s feedback on the participatory evaluation process was generally positive. The majority of youth enjoyed being interviewed; they felt heard, cared about, and valued because they knew their feedback would be used. They were able to recognize the value of the program evaluation and felt empowered by its youth-led approach. 

vi) Youth Feedback on the Use of Youth Interviewers
Most of the youth preferred having a youth interviewer. Adult interviewers evoked feelings of nervousness, and youth felt more comfortable with a peer. There was a feeling that adult interviewers would have elicited more cautious feedback, and that the interview process would have been different (e.g., adults would have more questions, and demand more answers or explanations).

6. Why does it matter for youth work?
This research suggests that YPE is a practical, cost-effective approach that can benefit both youth and program staff. It is an effective way to create democracy in programs while optimizing youth engagement. YPE meets young people’s need for autonomy, and offers an enjoyable way to hear youth voices in the evaluation process, which is often perceived as dull.

When conducting evaluations, organizations should prepare youth interviewers by implementing role plays of the interview process, and integrating different informant styles. Effective training for youth interviewers should:

• discuss the process and meaning of evaluation,
• emphasize the importance of building rapport with respondents,
• prepare youth for the types of scenarios they may encounter with respondents,
• teach the process of asking follow-up questions, and
• develop sensitivity to informant styles.

Categorised in: Research Summary