Results from a Beaufort County School District-administered survey on mentoring programs at the district's high schools indicate students in Strive to Excel are mostly happy with the program and believe it has been a positive influence on their lives.
The survey was prompted by recent revelations that the president of Strive, Hilton Head Island High School football coach Tim Singleton, has operated with little financial oversight or board governance in recent years. Former Board of Education member Joan Deery also has argued the district should do more to measure the effectiveness of programs it allows in its schools.
The survey is the district's first step in evaluating the mentoring programs and likely its first attempt at a systemic study of such services, district instructional service chief Sean Alford said.
In all, 170 students -- 128 of them with Strive to Excel at Hilton Head Island high and middle schools -- from six mentoring programs took the survey. Alford said that is about an 80 to 90 percent participation rate, achieved because the survey was administered during school hours. Assistant principals took students enrolled in the programs to the computer lab and gave them time to complete the survey. The only students who didn't answer the survey were ones who were absent that day, Alford said.
Alford said he and another district staff member with a background in program evaluation crafted the 10 questions. Most asked students to rate the usefulness of aspects of their mentoring program on a scale ranging from "greatly" to "not at all"; other questions were open-ended.
That rate of return should ensure results are not biased, according to Tammiee Dickenson, director of the Office of Program Evaluation at the College of Education at the University of South Carolina, who specializes in measuring the effectiveness of educational programs. She added that the questions seemed aimed at getting a snapshot of perceptions and that its usefulness in measuring a program's overall effectiveness is limited. Typically, a scientific survey would have 30 or more questions, she said.
The district identified participants by requesting lists of students enrolled in the programs from principals' secretaries.
Strive to Excel's participants are a combination of students who express interest in the program and students recruited to it. Some respondents indicated they no longer go to Strive meetings, but the survey was aimed primarily at active participants.
WHAT STRIVE RESPONDENTS SAID
Strive attempts to improve self-esteem, provide academic enrichment and prepare participants for post-secondary education and careers, according to its website.
Strive students at Hilton Head High said help with SAT vocabulary words, talks about college and discussions about their futures were the most helpful aspects of the program. Most of the 56 high school Strive students said they couldn't think of anything that wasn't helpful about the program.
Many suggested the program meet more often or during the school day; others wondered why the group wasn't taking more college visits.
The high school students said they had noticed average improvement in their grades, and Strive had greatly increased their desire to go to college.
At the middle school, 72 Strive students also gave the program high marks, saying it had greatly increased their desire to attend college and given them confidence to make post-secondary education plans.
Middle-school Strive participants said they found motivational talks to be the most helpful.
They suggested that spending more time in the program would make it better.
SURVEY DOESN'T PROVE EFFECTIVENESS
Alford said the district won't try to extrapolate too much from the survey results and that it is not finished with its evaluation of mentoring programs. For instance, he plans to examine participants' grades and survey their parents, too.
"We appreciate the subjective response, but we want to take a look at it objectively to see whether or not they are doing better," Alford said. "... Our goal is to monitor the impact of these programs on our students. It gives us a good picture of what the students think about their participation. For us, that's a good place to start."
At last week's Board of Education meeting, Superintendent Valerie Truesdale reported that Strive must vacate its office space in Hilton Head High by Dec. 31. It will still be offered free meeting space in the building.
Truesdale's report also indicated the district would require Strive to show proof each year that it was accomplishing its goals. Its report should include:
That report will be generated by Strive and not a third-party evaluator, district spokesman Jim Foster said.
Strive will also be required to submit federal tax forms and annual financial statements to the district. Foster said the district will examine whether Strive is operating in the black, how it is spending money and if it is conducting audits.
Proving a mentoring program is effective isn't easy, experts say.
Public/Private Ventures, a nonprofit research organization that has studied the effectiveness of mentoring programs such as Big Brothers Big Sisters, published a brief in 2009 that details ways to measure success. It cautions against participant-satisfaction surveys, saying they may not always be accurate.
"As many teachers say, the most enjoyable class may not be the class that promotes the most learning," it says.
It also cautions that children are more likely to give answers they think adults want to hear and may let their feelings about their mentor influence their responses.
Alford said he was confident students taking the district survey were truthful and not simply giving answers adults wanted to hear, in part because the program officials did not administer the test and were not told it was going to be given. He said the questions were designed to keep respondents from equating fun with effectiveness.
Dickenson said reassuring the students that results would be anonymous probably reduced the risk the answers were biased.
The Public/Private Ventures brief suggests collecting responses from as many stakeholders as possible -- students, parents, mentors and teachers -- while also collecting concrete data, such as test scores, grades, or instances of disciplinary action against the student.
Dickenson agreed. She suggested surveying teachers about changes in Strive students and tracking kids through their post-secondary career to see whether they stayed in college -- a task she admitted isn't easy.
The district has not considered surveying teachers, Alford said.
Dickenson said that in a perfect world, a control group would be formed to compare to Strive participants. That group would consist of students similar to those enrolled in Strive in terms of demographics, she said. Then, the Strive participants' outcomes could be compared to those of their counterparts who weren't mentored.
The Public/Private Ventures brief also suggests a comparison group be used. Random assignment -- assigning students randomly to either receive or not receive a mentor -- is the strongest way to prove a program is working. But that isn't always possible or ethical in mentoring programs.
If nothing else, Dickenson suggested the program should be evaluated by an independent party.
"For research-credibility purposes, that's something important," she said. "Usually when you have an internal report, they're more likely to be biased."
Follow reporter Rachel Heaton at twitter.com/HomeroomBft.