The main way tech-savvy educators find out about ed-tech tools is from web searches, but nearly half of them also rely heavily on vendor reports for information on those products, according to a new survey.
In the study of 1,124 teachers, school administrators, district staff, and technology leaders, over half (57 percent) indicated that their local education leaders rarely or never discuss ed-tech research.
The respondents think research is important. In fact, three-fourths of them said it is part of their jobs.
When they talk about research, these educators generally do so under specific conditions. About 70 percent said it’s when they’re discussing tools recommended by colleagues or, collaboratively, during planning time (64 percent.)
Of those who reported being personally involved in procurement decisions, 65 percent said that they discuss ed-tech research with colleagues when working on a committee responsible for making procurement decisions about it, and 57 percent discuss such research with colleagues when seeking approval to use a new tool or product.
The respondents to the survey—from 50 states, D.C., and four territories—were more likely to be involved with ed-tech procurement than a general group of educators would be, according to Brandon Olszewski, director of research at ISTE, which distributed the survey with JEX to 18 organizations. That’s possibly because 60 percent of them are part of ISTE, and 28 percent from ASCD.
The strong representation of respondents who play a role in procurement “is both a limitation and strength of the survey,” Olszewski said, because a general population of educators is less likely to be involved with researching and purchasing ed-tech products. But since this sample did have a preponderance of those who are, the message from the research is: “They don’t have a place they can consistently go for evidence they consider reliable,” he said.
Dubious View of Vendor Research
Getting product information educators can trust is a challenge, according to the “Education Research Perspectives Survey,” which was fielded in May. More than three-quarters of respondents (76 percent) felt that ed-tech vendors were not qualified to conduct valid research about their products, yet 48 percent look at vendors’ sites to get information about those tools.
Interestingly, what researchers often consider the highest level of evidence from the U.S. Department of Education’s What Works Clearinghouse isn’t necessarily a “go to” source for them. Eighty-one percent of this “plugged-in” group does not regularly review research from the Institute of Education Sciences, the study found. The IES, part of the U.S. Department of Education, is often considered the “gold standard” of research, and its What Works Clearinghouse invites educators to “find what works” based on the evidence.
The overall findings echo elements of earlier research, which show a lack of willingness to seek out peer-reviewed research. In a study it commissioned three years ago, Google learned that educators are strongly influenced by their peers when they want information about ed-tech. Last year, another study found that only 11 percent of district administrators and teachers said they would outright reject buying or adopting an ed-tech product if it lacked peer-reviewed research behind it.
Educators—particularly those involved in procurement—believe that the organizations best equipped to conduct research are first and foremost, local school districts (67 percent), followed closely by research organizations (65 percent) and education non-profits (63 percent.)
“We continue to spend billions on these products without the information we need to make good decisions,” said Bart Epstein, the president and CEO of the Jefferson Education Exchange, in an interview about the survey. The survey’s findings “reinforce the need for new approaches to gathering and sharing reliable data about education technology,” he said in the release about it.
Without what they consider reliable information about ed-tech products, educators’ “number one source of data is peers, word-of-mouth,” said Joseph South, the chief learning officer for ISTE, in an interview. The result? Generally, ed-tech products “are chosen based on popularity, not efficacy,” he said.
South, like Epstein, called for a collaboration to fill the gap between what’s available to help schools make research-based decisions.
The Jefferson Education Exchange and ISTE are both working to address the issues raised in their research. The two organizations are collaborating with other groups, like Digital Promise, to elevate the issue of sharing reliable information about ed-tech products. Most recently, it was a topic of discussion at the EdClusters conference in Philadelphia.
ISTE also opened a community-driven review platform called the ISTE Edtech Advisor earlier this year. This resource is designed to give members of ISTE insight into which tools, technology, and apps are most likely to best meet their learning objectives.
And JEX, launched in February, is a nonprofit organization working to improve how schools choose, procure and implement ed-tech. It plans to pay teachers for reviews of ed-tech products.
Epstein said educators are right to be wary of vendor-sponsored research. When a company does research for the purpose of supporting the sale of its product, “it is naturally incentivized” to select among the schools where its product is implemented and working best, said Epstein. “As a result, educational technology purchasers are often led to believe—erroneously—that they can expect the same result in their local environment [which may be] wildly different.”
Denis Newman, whose company wrote the Guidelines for Conducting and Reporting EdTech Impact Research in U.S. K-12 Schools, an industry-backed best practices report, has said it can be challenging for school leaders to surface this kind of information even at the What Works Clearinghouse.
Educators searching the clearinghouse for research on subgroups that mirror their school’s demographics can find it difficult to discover a match, he said. They can’t easily find the differential impact of a program on a certain subgroup of students, because that information is buried in the report. In its summary reviews of research, the clearinghouse only shares the overall average effect, according to Newman, CEO of Empirical Education, Inc., a Silicon Valley-based research and development company that has conducted research that is also stored in the What Works Clearinghouse.
“I often point out a big project we did for a regional lab in the Southeast,” he said in an interview for a previous story. The study in Alabama found that a STEM program had a small overall effect on math. “But it didn’t work as well for the black kids as for the white kids,” he said, referring to the subgroup impact analysis. That information is included in the report, on page 299.
Epstein said further research is needed to understand more about how often—and what percentage of—educators review IES-supported research and use that research to inform their decision-making processes. “Stay tuned for more on that,” he said.
Mark Schneider, the first permanent director of the IES in four years, seems to be aware of some of the limitations. In an interview with Sarah Sparks published last month, he said that the agency is exploring whether to use the What Works Clearinghouse to promote research that offers a strong cost-benefit or evidence of longterm results, rather than just looking over a year or two of evaluation.
- Peer-Reviewed Research Not a ‘Must Have’ for Many K-12 Officials
- New ‘Education Exchange’ Will Pay Teachers for Reviews of Ed-Tech Products
- The 3 Questions Researchers Want Educators to Ask About Ed-Tech
- Research Guidance for Ed-Tech Industry Updated on Usage, Other Criteria for ESSA Evidence Levels
- U.S. Education Innovation Cluster Leaders Gather to Support Advancement in Teaching and Learning
- Google Research Shows ‘Word of Mouth’ Fuels Ed-Tech Decisions
- Who Calls the Shots in School Purchasing? Vendors, K-12 Officials Disagree
- New Education Research Chief Signals More Focus on College Over K-12 Studies