One of the nation’s two main common-core testing consortia has ordered an independent review of an ambitious test-delivery platform blamed by a vendor for contributing to disruptions on statewide exams in Nevada.
The Smarter Balanced Assessment Consortium’s evaluation will examine whether the test-delivery system created for its member states by the American Institutes for Research has lived up to expectations laid out in the vendor’s contract.
Consortium officials say they are now in the process of negotiating an agreement with a software company to conduct the review.
When the state of Nevada weathered a series of breakdowns in trying to give statewide tests earlier this year, the state’s testing vendor at the time, Measured Progress, argued that some of the fault belonged to the AIR for delays in delivering its testing platform. The AIR disagreed, saying any delays were minimal, and the blame was misdirected.
The contract between Smarter Balanced and the AIR is worth $14 million, about $6 million of which is for work associated with the platform, the vendor said.
In a statement, AIR officials said they welcomed the Smarter Balanced review, and stand by their work.
“We are confident the open source system as delivered exceeded the contract requirements in many ways,” the AIR said. “The review is probably helpful given some of the events and discussions surrounding the use of the system by other vendors whose understanding of the system may have been incomplete.”
Montana’s state schools chief Denise Juneau also publicly pointed to problems with transferring a bank of test items between the AIR and Measured Progress, in describing exam disruptions in her state this year.
Worries about the platform that emerged in those two states, as well as in North Dakota, another state that faced testing disruptions, prompted Smarter Balanced to set in motion the review, said Tony Alpert, executive director of the consortium, in an interview.
“There were concerns members had about [whether] the test delivery system was doing what we thought it should,” Alpert said, adding that “the main focus of [the review] is to fulfill the vision that will allow states to use this resource.”
Smarter Balanced is leaving decisions about administration of its exams to individual states, such as Nevada, which had hired Measured Progress for that work. (The contract has since changed hands.)
The AIR’s contract was to develop the test-delivery system for Smarter Balanced states, which often requires the vendor to coordinate its work with that of other testing organizations.
High Expectations for Platform
Based in Washington, the AIR has emerged as a major player in the world of assessment in recent years, with broad experience helping states with online testing.
The test-delivery platform has been envisioned by Smarter Balanced officials as providing a cutting-edge set of online assessment features for states. The platform or engine is software designed to deliver tests in states and their individual schools—ideally, meshing smoothly with the various technologies used to give the exams as it collects responses from students and facilitates the scoring of assessments.
The “open-source” nature of the platform was meant to allow other companies and developers to build upon and enhance it, because the code for the technology is publicly available.
Smarter Balanced has hoped the open-source platform will encourage innovations in testing across states, and eventually streamline the assessment process for schools, and ultimately teachers and students, Alpert said. The consortium continues to maintain those lofty ambitions, said Alpert, who described the open-source platform’s potential as “incredibly important.”
Even so, conducting an independent review of the AIR system was warranted, given states’ concerns, the Smarter Balanced official said.
“There were some great successes,” Alpert said of states’ initial experiences with the platform, “and there were some frustrating challenges.”
The consortium earlier this year released a request for proposals through the University of California, Los Angeles for vendors to conduct the review. (Smarter Balanced is an independent operating unit of UCLA.) Smarter Balanced officials say negotiations are underway with a vendor, who they would not identify, to do the evaluation.
In a letter sent this week to state school superintendents in Montana, Nevada, and North Dakota, Smarter Balanced Deputy Executive Director Luci Willits said the goal of the review was to “evaluate the quality and performance” of the test-delivery system.
“We know the spring rollout was rocky, and we don’t want to replicate that experience in the future,” Willits said.
North Dakota state schools chief Kirsten Baesler, in a statement released today, said her office had pressed Smarter Balanced to review the testing platform and take other steps to ensure that future tests went more smoothly. Baesler said Smarter Balanced had agreed to refund $92,320 in membership fees to the organization, to compensate for the testing problems.