|He then welcomed Dr. Marilyn Howard, Superintendent of Public
Instruction who will inform the committee about state testing.
Chairman Schroeder said he was especially interested in knowing if the
tests that are given are the correct tests. When the tests were developed,
were they subject to a scientific analysis? Do they provide the best
information? The Chairman said he hopes his questions can be
Inserted in these minutes is a copy of Dr. Howard’s talk.
Comments to Senate Education Committee by Dr. Marilyn Howard
Senator Schroeder and committee members:
A year ago, the State Board of Education proposed, and the Legislature
through the rule-making process, a state-testing plan that, in a nutshell,
continued some of the state’s existing tests and replaced one multiple-choice test
For years, we used the Iowa Test of Basic Skills, which measured reading,
language arts, math, social studies, science, and research skills of students in
grades 3 through 11 once a year .
Under the new plan, we discontinued the Iowa Test and substituted the ISAT, a
multiple-choice test that assesses reading, language arts, and math twice a year
for students in grades 2 through 10. Essentially, what the state did was to swap
one “off-the-shelf, pre-constructed” test for another “off-the-shelf, pre-constructed” test.
As its testing contractor, the State Board of Education picked the Northwest
Evaluation Association. The idea was to modify the NWEA’s “levels” test for use
in Idaho schools.
Let me take a moment to talk about a “levels” testing. This is a test that adapts to
each student’s level of performance. For instance, a fifth grader who answers a
question correctly will be given a more difficult question, and if he answers
correctly again, he gets a tougher question, and so on until his “level” or plateau
is found. The same would be true for the 5th grader who misses a question; he
gets an “easier” question and so on. A 5th grade teacher may learn that she has
students performing at an 8th grade level, others who may be at a 3rd-grade
level, and others who are right where they should be at a 5th-grade level.
Because the test adapts to the level of individual student, every test is
This choice -working with the NWEA’s levels testing -was popular because
more than 30 school districts already were using their local funds to buy the
“level” test in addition to the state-paid-for ITBS. At that time, a little over half our
student population was taking the level test.
So the idea was to use the levels testing as the basis for the new ISAT, and for
students to take the test twice: once in the fall to give teachers a way to plan
instruction based on each student’s needs, and again in the spring as a year-end
evaluation of how effective instruction had been.
Then we hit our first problem. The U.S. Department of Education would not
accept the adaptive testing format selected by the State Board.
That’s because the federal focus is on making sure every student is meeting
grade-level standards. In other words, every student should be asked the same
questions so that we know which 5th graders are reading at the 5th grade level
and which are not. The levels test, on the other hand, doesn’t measure that.
Instead, it measures where the student really is. You can think of the federal
requirement as a “yes or no” answer, whereas levels testing gives you a “where
are you” kind of answer.
The U.S. Department of Education’s stamp of approval is critical. That’s because
the No Child Left Behind Act requires states to set standards for students and
then measure whether students are meeting those standards. Since most of our
money for programs for disadvantaged students comes from the federal
government, we have to comply with the federal act.
During the year, the then-State Board of Education President made repeated
attempts to approach federal officials, including U.S. Department of Education
Under Secretary George Hansen, with the hopes of having the adaptive testing
model approved. The Northwest Association also spoke to federal officials about
its levels testing. Those efforts were unsuccessful –in fact they led to an official
statement that the “adaptive” model (the levels testing) would not be acceptable.
All of this came to a head last spring. To make sure that Idaho would continue to
receive federal education aid, the State Department of Education entered into a
three-year compliance agreement with the U.S. Department to create tests that
would be acceptable and ultimately bring the state into compliance with No Child
Left Behind. That meant the State Department had to sign a separate
memorandum of understanding with the Northwest Association to make sure it
would help us meet those federal requirements.
That’s an overview of the past year. Where are we today?
The State Board of Education bought the NWEA’s test, which means that we buy
access to the items in the NWEA test bank. When the test is delivered
electronically, every child’s test could be different than that of his or her
classmates. The items are the ones that NWEA has used in other schools in
various states, and NWEA has determined the level of difficulty of the items so
the computer can do a form of random access of items. The goal is to see if
students answer more difficult questions in the winter than they did the previous
fall and if, in the spring, the computer is able to place even more difficult
questions before them. If learning is progressing, the students’ scores should
have a higher numerical value each time the test is given if learning is
When the student takes this test in Idaho, we call it an ISAT .
Then, as required by our compliance agreement with the feds, we began
developing the federally-required tests, which we will pilot this spring. The
compliance agreement gives us three years to construct and phase in the tests.
This year we will test fourth, eighth and l0th graders and in the ensuing years
third, fifth, sixth, and seventh will be added.
Last, we were able to satisfy both objectives -the state’s need for levels testing to
see whether students were making progress, and the federal need to find out
whether students are meeting our standards for their age and grade -without
over-testing students. The solution was to take advantage of the computer
format and create what we are calling a blended test.
Ultimately, this one blended test, given in the spring to grades 3-8, will include
both components: the standards measurement and the levels testing. The result
should be good information to the extent of which students are meeting our
standards and good information on student progress during the school year.
Bob West said Senator Schroeder asked for information about how the tests
created, how the tests are being evaluated, and what the tests are suppose to
measure. Since the spring ISAT is really two tests blended into one, I’ll talk about
I’ll start with the part of the spring ISAT that is being developed to meet federal
requirements. This is the piece of the ISAT for which the State Department of
Education is directly responsible.
The questions for l0th, 8th, and 4th, grade tests are being written or reviewed by
Idaho teachers and the department’s experts in reading, language arts, and
math, all of them working with the Northwest Association. These questions are
linked directly to Idaho’s Achievement Standards.
The same process will be used to develop the “federal” tests for 3rd, 5th, 6th, and
7th grades during the next two years. This will give us the information the feds
want – that is, reliable information about whether Idaho students are meeting
Idaho’s academic standards for their grade and age.
As for the fall ISAT and “non federal” portions of the spring ISAT, it is important
to remember that the state bought a preexisting test that was originally developed
by the Northwest Association for a different purpose. Its questions are drawn
from the Northwest Association’s bank of questions and they are aligned to the
association’s learning continuum -in other words, to the association’s own
expectations of what students should know, and when they should know it.
That raises an obvious question: Does the NWEA continuum match the Idaho
No, not exactly. That’s one of the reasons the department has contracted with the
Northwest Association to create a customized document called the “Idaho
Learning Continuum.” This will be available to districts in the next few weeks.
That and other aides developed by the State Department will be powerful tools to
show the connections among the standards, the assessments, local curriculum,
and instruction. It is our hope that these will help teachers recognize that our goal
is not to teach the “test” but really to measure a student’s mastery of the
The next question had to do with how we are evaluating the ISAT.
Again I’ll start with the “federal” portion of the spring ISAT.
Under the compliance agreement I signed last spring, the State Department must
provide the federal department evidence that the test is reliable and valid,
meaning it consistently predicts student achievement and accurately measures
whether students are meeting a standard.
The tests also must be free of bias. Test questions can’t be written in a way that
could limit the performance of specific groups of students.
The questions must be cognitively complex. The students have to do more than
just regurgitate facts and figures. They must show they can apply what they know
to solve problems, analyze information, and so on.
The test must be suitable for use with special needs students and if not,
appropriate accommodations such as large print or Braille tests must be made
In addition, we must have the data disaggregated so we can report on the
performance of each subgroup.
To do all this, the State Department of Education is working with a Technical
Advisory Panel of national experts, and with the Northwest Regional Educational
Laboratory, and with the Northwest Evaluation Association to produce the studies
and research required.
As for the state portions of the ISAT, the State Board of Education has not
discussed pursuing reliability, validity and bias studies. This has been of some
concern to the department, because the federal rules require that all tests given
to all students must meet the criteria I just mentioned: be reliable and valid, free
of bias, and cognitively complex. Eventually I believe we will have to deal with
this for the state portion of the ISAT, just as we have for the federal portion.
The State Department has asked the vendor, NWEA, to provide tests that meet
these expectations and proof that they do so. This is asking the company to
provide assurances and products that they have not had to provide in the past.
The result has been revealing both of the capacity of the vendor to deliver and
the willingness of the vendor to meet our needs -which are outlined in the signed
memorandum of agreement with the company.
Bob West said you also want to know how we administer the ISAT. The State
Board of Education’s goal is for all students who are able to do so, to take the
test on the computer. The computer really is the best format for this type of test
because it adapts quickly to the student’s responses and immediately scores the
test. It’s quicker and more efficient.
During the fall, the majority of students took the ISAT on computer, but a
sizeable chunk also took a paper form of the test because of technology
difficulties. This spring, we expect to have about 96 percent of all students taking
the test on computer. Soon, I hope, we will use paper tests only to help special
needs students who may need that option.
Finally, Bob said you also wanted me to talk about my concerns about the cut
scores or proficiency levels the State Board of Education adopted for the ISAT
earlier this month. You probably know that I was unsuccessful in asking for the
board to delay this decision. In fact, the vote was 7 -1, and I was the cheese
A “cut score” is a way we categorize students according to how they did on the
test. You all know this from your own school days. In most classes, a 93 or above
was an ” A “, 85 to 93 a “B” and so on. ISAT cut scores should tell us which
students are considered below basic, which are basic, which are proficient, and
This is old hat to us: we did the same thing for the Idaho Reading Indicator to
decide which students were below grade level, which were above grade, and
which fell in between.
The reason I was the lone vote against adopting cut scores for the ISAT is that I
don’t have the information I need to say with confidence that these scores will tell
us what we want to know about student performance. You may have been told
that my reluctance to set cut scores is some philosophical distinction between
norm reference testing and standards based testing. That is not true. Rather, I
believe the board skipped essential steps that would help to ensure a thorough
and thoughtful decision on ISAT scores.
This is a complicated issue and I’ll skip the long explanation and just say this: At
the end, what we really want to know is whether our students are meeting our
standards. I don’t think we have enough information yet to answer that question.
As I summarized the test development and blending of state and federal
requirements, it may have sounded like a neat and tidy package, but the reality is
that implementation has been messy and many areas are still “under
In launching a new test, we expected challenges, but in the interest of full
disclosure, I want to share some of the difficulties we faced and continue to
wrestle with as we assist schools in meeting state and federal testing
One obvious problem has been trying to decide which decisions are policy and
which are implementation.
The State Board has pretty much taken charge of the testing program, selecting
contractor and adding people to its staff.
Still, the State Department has had to provide more technical and supplemental
services than we anticipated to support the state’s testing contractor. NWEA is a
fledgling company and is having to create products new to them. We have to fill
in the gaps with our own experts.
For example, the department has provided technology support for everything
directly helping districts get their computers online to give the test, to creating the
“electronic” formats for reporting data and ensuring the security and privacy of
data transmitted from district to the testing contractor.
Another example: the department’s reading, language arts, and math specialists
have been called on to develop test items, lead teacher workgroups, proofread
and edit test questions.
And our special education staff also has done significant work to help address
concerns for special needs students. We do see some relief in sight because the
Northwest Association has recently added staff who are knowledgeable in these
areas. This should ease the workload on the department’s staff.
The other wrinkle this past year has been the mixed messages and in some
cases a complete 180-degree change in direction in the “oversight” of the State
Board of Education.
Early on, the department was told to work with the Northwest Association to
implement the ISAT. No problem: that just meant the department would be doing
what it had been successfully doing for more than 20 years –running the day-to-day operations of the state’s assessment plan.
And keep in mind that the state’s assessment plan included not just the new
ISAT, but also a couple of old-timers: the Direct Writing and the Direct Math
We scheduled meetings with superintendents, principals, testing coordinators,
created brochures for parents, provided materials to support school districts in
communicating with parents and the public, prepared joint news releases, and so
But since about last mid-fall, the board began shifting from oversight to day-to-
day operations. Our staff has been told to quit scheduling training and meetings
with educators. And we refer all ISAT questions to the board office.
My assumption can only be that the board intended for the department to
this program only until it could get other staff on board -which it has done. The
board’s academic officer, a position that has traditionally provided oversight for
higher education academic programs, now is overseeing K-12 issues including
the ISAT. The board has replaced its achievement standards coordinator with an
assessment program manager, who among other duties, has been directed to
organize the training and meetings the department has traditionally done. In fact,
I had to write a special memo asking to have the state department’s testing
expert added to the board’s assessment commission -and, frankly, I still don’t
know whether that’s going to happen.
No wonder school districts are confused about who is doing what. School
districts -teachers and administrators and testing coordinators- are used to
calling the department for assistance or advice or direction. But now when they
call to report problems or ask questions, depending on what test they are asking
about, we might be able to answer- if the question is about direct writing or direct
math or maybe about the federal testing requirements -or we might have to refer
them to the board office if the question is about the state portion of the ISAT.
We’ve always taken it for granted that Idaho had an efficient system for K-12
issues, but it’s not working that neatly anymore.
I didn’t come here today to ask you to sort through these differences. But you’ve
all seen the newspaper reports and you know we’ve had problems.
The reality is that you can’t drive a car with two sets of hands on the wheel and
perhaps that’s why we have seen swerves in the past year. In time, I hope, we
will get a clearer picture of what constitutes policy, which is clearly the board’s
purview, and what constitutes program implementation, which is what we have a
state department of education for.
Last but not least, I also have concerns about the costs of this testing.
I mentioned earlier that school districts were happy about the levels testing
decision, because about half of the state’s students- in 33 districts and 3 charter
schools-were already using them. The cost for about 140,000 students, from
local funds, was about $600,000.
Now, the State Board of Education is paying about $3 million to test about
I want to stress that is just for the state’s portion of the ISAT -the levels testing
portion. It does not include the federal portion of the test, which measures the
extent to which Idaho students are meeting Idaho standards.
I can’t report to you on how that money was spent since it is a State Board of
Education responsibility. I requested a detailed accounting of how state funds
were used because the department must assure the U.S. Department of
Education that we are not spending federal funds to “operate” our state
The $4 million in federal funds that the department receives has to be spent on
the federal portion, and not on the state portion. That $4 million will pay for the
federal portion of the spring ISAT -both the tests we have now and the ones we
will be developing- and for the federally-required assurances on reliability,
validity, cognitive complexity, and on whether the tests are free of bias, and for
federally required tests to track progress of limited English proficient students,
and for three new science tests that are required by 2005. We’re making that $4
million stretch as far as we can.
In other words, the federal piece -how the state department is using the $4
million- is pretty clear. I don’t have enough information to evaluate the state’s
portion, although I’m sure the board’s staff could give you all the information you
need. As we move forward, I will be watching this intently. The State Department
has a good track record of getting a lot of bang for our buck, and I believe that
should extend to contracts with test vendors.
Now, despite the problems I’ve mentioned, and despite how complicated all of
this seems, I want all of us to keep an eye on our real goal, which is to use all of
these tools to improve student learning and to ensure that students who struggle
get the help they need when they need it.
We may be hitting a few bumps along the way, but I’m convinced that in the end
we’ll have a meaningful, relevant, sensible and defensible testing program.
Thank you for your time and I’m here to answer any questions you have.