Showing posts with label NCLB. Show all posts
Showing posts with label NCLB. Show all posts

Sunday, November 8, 2009

A Lesson from Oklahoma

Earlier this fall, the Oklahoma Council for Public Affairs released the results from the survey it had commissioned about the civics knowledge of high school students, which were very alarming. Only 23% identified George Washington as the first president, 29% identified the president as being in charge of the executive branch, and 43% that Democrats and Republicans being the two major parties in America (10% identified the two parties as Communist and Republican). OCPA decried the results as proof of the failure of Oklahoma's educational system, and the results of their survey were used in many major publications.

But, almost as soon as they were released, questions began to be raised. Could it really be that only 2.8% of the 1000 polled high school students, as the survey claimed, could pass the test (which is a meager 6 out of 10 correct)? And none would get 8-10 correct? In a random distribution that would bring about 600 college-bound students and 50 gifted students, none got 8-10 correct?

This raised some questions, most notably by statistician Nate Silver. Even starting with the assumption that only 23% of the sampled students knew about Washington, the results still looked fabricated. Simply put, the distribution of student scores matches almost identically to a homogeneous distribution of probability. However, students are not homogeneous... a student that gets the first three right is much more likely to get #4 right than one who got the first three wrong.

Silver wasn't the only one, as he mentioned today. State representative Ed Cannaday, a former educator, also thought something was fishy. He conducted the same survey in school districts in his own congressional district within Oklahoma (N = 325). And, he found an entirely different set of results. In fact, 98% identified George Washington as the first president, 85% identified the president as being in charge of the executive branch, and 95% identified the correct two political parties. In fact, the average score in Cannaday's survey was 7.8 correct out of 10, very striking considering the OPCA survey said none in 1000 scored more than 7 correct.

POLITICS INVOLVED
While Silver doesn't focus specifically on it, still what is clear is the subtext of OCPA being a conservative group pushing educational policy changes. And, by the reach of the survey's results, which landed prominent places in Time, Newsweek, and the USA Today, it can be said the survey was successful, however dubious. Now that headlines have blared how Oklahoma's public schools are failing miserably, the damage is done, and a page 12B follow-up article will not do anything about it.

The lesson here is twofold. One, that in an era of data-driven decision making, standardized assessment results still run secondary to sensationalized opinion polls in the effort to sway public opinion. And two, that schools appear to be not off-grounds for fabricated politicization. Which means one thing...

If you are an Iowan representative, you better be extra critical of any external data used to describe the achievement of Iowa's students.

And if you are a newspaper, like let's say a Register in the state capital, you should be equally cautious.

Using standardized data as a measuring stick for how well the nation's schools are doing is troubling enough for many reasons. It becomes infinitely worse when using non-standardized survey data.

There are a million reasons behind the data, all the way down to whether the students had any breakfast this morning, to whether the test had cultural bias within it, to whether a teacher happened to give the students the answers. Now we have to add nefarious purposes of the testing provider to the list.

Friday, April 3, 2009

Better Data Needed

Our neighbors to the north were in the news, with some harsh (but very thought-provoking) criticism of NCLB. From Minnesota 2020:

Last fall, the prestigious publication Education Week hosted an on-line chat about the federal No Child Left Behind law. One of the panelists was David Figlio, a professor at Northwestern University and a Research Associate at the National Bureau of Economic Research.

Ellen Solek of East Haddam, Conn., asked if Figlio was aware “of any current research that has, or is being conducted that determines correlation (if any) between K-12 student test scores, accountability, and future success in the workplace?”

This is a magnificent question because it goes to the heart of NCLB and how it relates to every Minnesotan. The question is simple: What difference does NCLB make?

Figlio doesn’t really have an answer. First, he says this: “It’s too early to know about the effects of accountability on workplace success.” Then he says “there have been a number of studies that have linked K-12 test scores to labor market outcomes as adults,” but then adds “these papers use data that are decades old, however.”


Minnesota 2020 is asking the right question... what difference does NCLB make? Sure, it gives you objective data to draw conclusions from, but are they the data we need? In simple words, don't we need authentic standardized assessment?

Which brings me back to the comment Shawne Berrian left on a post of mine several weeks ago.

Here is the research we need to conduct. Instead of debating about what will bring about success, let's go find successful people. Use the basis of peer nominations about who is the best in your profession. And choose all professions... mechanics, auctioneers, librarians, soldiers, not just doctors and engineers. Then, choose people who aren't successful in those professions.

Then test them. Give them standardized content-specific tests, such as the Iowa Test of Educational Development. Then give them a standardized assessment which measures the 21st century skills (PISA). And, look at the correlation.

This is what Wiggins and McTighe did. And, they found what made the difference were the 21st century skills, not the content skills.

This is a very interesting point. We always define best practice based on what elicits improvements in student achievement scores, but what if this has stunted other practices that "didn't bring about ITED gains" that would actually help out students be successful? There is a movement towards starting with the end goal in education and designing backwards... maybe we didn't go far enough to start the design process.

(h/t to Education Futures for the link)

Saturday, March 21, 2009

Fact Checking Obama's Educational Speech

The president made a lengthy speech on education last week at the U.S. Hispanic Chamber of Commerce. The overall immediate intent of the speech was to outline some of the pressing issues for education of Hispanics today, and how those issues are hurting the community succeed. But Obama went beyond focusing on the Hispanic population and looked at the "educational epidemic" as a whole.

FactCheck.org has an excellent analysis of Obama's speech. In it, while there are some points that Obama's information is accurate, they found several inaccuracies:

We certainly wouldn't argue that education can't be improved, but some of the figures Obama used painted a bleaker picture than actually exists:
  • The high school dropout rate hasn't "tripled in the past 30 years," as Obama claimed. According to the Department of Education, it has actually declined by a third.
  • Eighth-grade math scores haven't "fallen" to ninth place compared with other countries. U.S. scores have climbed to that ranking from as low as 28th place in 1995.
  • Obama also set a goal "of having the highest proportion of college graduates in the world" by 2020. But in terms of bachelor's degrees, we're nearly there. The U.S. is already second only to Norway in the percentage of adults age 25 to 64 with a four-year degree, and trails by just 1 percentage point.
All of this is in direct contrast with the words of George Bush a year ago, who boasted of the nation's success since passing the NCLB legislation. This has sent the blogosphere ablaze with postulates of why Obama said these items, many being quick to identify Obama as a foe to public education.

I'm not so sure of this; given that Obama has repeatedly asked for more money for public education, this seems to be plain old politics at its "best". Obama isn't going to gather much support for increased spending in education with the statistics that show the U.S. is doing fairly well. And Bush isn't going to use statistics that make his landmark public education bill look like a failure. The truth is somewhere in the between.

FactCheck.org's analysis does raise some questions. Even if I'm right and Obama's major intention was to garner support for funding, will there be unintended consequences of the speech, ostracizing those in the educational community? Will he continue to cherry-pick data as other presidents have, using it for political purposes?

And, how can we do apples-to-apples comparisons using TIMSS data? FactCheck pointed out well that when you compare front-running Norway to Massachusetts, where population and per-pupil expenditures are roughly equal, Massachusetts leads Norway in graduation. When comparing the U.S. to the European Union (again, a more apples-to-apples comparison) the U.S. leads in all but one category. Plus, as we've discussed, the TIMSS data is not necessarily the most valid measure to begin with.

Wednesday, January 28, 2009

Program Evaluation of 1:1 Environments

More food for thought from the day at Newell-Fonda...

At a couple different times during the day, one member asked a good question: what is the impact of the 1:1 on student achievement. The Newell-Fonda district, of course can't answer that quantitatively, being that they just started implementation at the beginning of the year. And while teachers indicated they saw improvement in student's performance in their own courses, critics could counter by arguing that's subjective data from a partial source.

Leaving Newell, I had a conversation with Steve Linduska, a colleague of mine at Heartland. One of the things we discussed is the interesting dynamic of a 1:1. You have an empassioned district (and in this case, partnering with an empassioned vendor, Apple). And at this district, the superintendent is also the high school principal, and probably has half a dozen other hats, like curriculum, public relations, school finance, and running the scoreboard at athletic events. Simply put, you don't have someone who can take an objective look at whether it is working, because they are too involved.

But as Steve pointed out, that's just half the dynamic. The other half is how this would be measured, which in Iowa, has been resoundedly via ITEDs. But, the benefits of a 1:1 do not appear solely (or I'd suggest even primarily) in ITEDs. What about visual literacy? Creativity? Synthesization? Presentational skills? Adaptability and problem solving? Not bubble friendly.

Mark Pullen put this well:

I think one of the biggest unspoken messages that No Child Left Behind has sent to teachers and students across the country is this: If something can’t be easily benchmarked, it isn’t worth teaching or learning.

The trouble is that most of the things that really matter in education (and life) aren’t benchmarkable.

Jeff Dicks, the Newell Superintendent, hit on that as well, as he replied that a junior in his school had suddenly taken a career interest in information technology, which doesn't appear in a bubble sheet.

Yet, it is essential that we do evaluate our programs to know if they are working, and do so objectively, authentically. I'd edit Pullen's last statement to say "The trouble is most of the things that really matter in education aren't easily benchmarkable". They still must be measured. What are assessments we can create that measure this reliably? This is more than just Newell-Fonda's task, it is Iowa's (and the nation's).


Sunday, September 28, 2008

Call For Action: Authentic Standardized Assessment

We need better standardized assessments. Which is truly an indictment on two things. As many would quickly assume, this is an indictment on the ITBS and ITEDs. But it is equally an indictment on the areas those tests don't cover, which are then covered by inadequate locally-made exams (if at all). Neither are appropriate.

In Iowa, we (mainly) measure our proficiency with the Iowa Test of Basic Skills and the Iowa Test of Educational Development, or ITBS and ITEDs. Published by Riverside at the University of Iowa, they have long been used as a norm-referenced test to measure student achievement in the classroom. With the advent of No Child Left Behind, the ITEDs/ITBS were suddenly used as the official achievement tests for accountability purposes. If students did not perform well on the tests, schools now faced consequences.

While there are tests in social studies, reading materials, and language, more and more schools are not taking those tests, as only math, reading, and science are required. Many outspoken critics of NCLB have mentioned that testing becomes a shell game, as schools teach to the test (or more linguistically correct, teach the test), and marginalize other curriculum for the sake of proficiency. More importantly in my estimation, they marginalize other students. Iowa requires being at the 41st percentile to be considered proficient, and a school needs a large proportion of its students to be proficient (79.3% in 11th grade, for example). From a statistical point of view, the students who are most likely to make a difference for a school are those students who are between the 20th-50th percentile. Many districts are implementing programs like Second Chance Reading, or Remedial Math to help those students, while not addressing the needs of the solidly proficient, let alone those who are in the top 10 or bottom 10 percentile. This is the equivalent to Obama and McCain putting all their money and resources into Ohio, Pennsylvania, Colorado, and Virginia, because the other states don't matter.

In James Popham's The Truth About Testing, he points out several problems with using tests like the ITBS/ITEDs as measures for accountability. They are a snapshot in time and could easily be thrown off by external factors (like whether the student eat breakfast that day). By being a normed-reference test, they require testing elements that separate students for validity, or in other words, questions need to have students miss. While NCLB wants all students to get the question right, Riverside would consider the questions awful if everyone got them right. (The test has not been re-normed since being used for accountability, by the way.) Moreover, tests have large amounts of bias for intelligence and social-economic status that cannot be distilled. While Popham says standardized data are useful, they just are not reliable for accountability.

In brief, I not only agree with Popham, I could also add several other reasons why they are not valuable. But that's not the subject of my post. I strongly feel schools should be accountable, and that schools should use standardized tests. They just need to be authentic ones.

There isn't a more un-authentic test than a multiple choice test. When was the last time you took one? Seriously? High school? What job requires proficiency via multiple choice tests?

We show our proficiency in the world through our performance. Can you compose an essay that illustrates the reasons why I should be for a particular argument? Can you create a pamphlet that gives the reader instructions on how to complete the task? Can you draw a conclusion through a set of laboratory experiments? Can you grow strawberries in your agriculture class? Multiple choice, at best, is a tool for formative assessment, and is primarily overused by teachers because it is wildly convenient.

And, that's why we use multiple choice tests for accountability... because they are convenient. A scantron can score them. To those who would say convenience has to be considered from a financial standpoint, I could not object more. Flat out, that convenience hurts kids.

So I propose a call for action. Let's establish quality authentic assessments for all our schools. To do this, we must have standards that we deem are essential for the world. The Iowa Core Curriculum, I feel, will do that for us. But unless we then have quality authentic assessments, how will we know how students are performing?

Let's look at an example. When I say a student "must be able to use technology to solve a variety of problems" as my educational standard, how do I know students are successful at this? The two options that are currently out there are inadequate. An ITED-esque standardized test won't tell us. But moreover, a locally-made criterion-test (aka "a teacher test") won't tell us either. We have no way knowing how well a school's instruction compares to other schools with locally made tests.

Right now, technology literacy, as defined by the federal government, must be measured in 8th grade. And it is a joke. I've seen quite a few school districts who use a random assignment or a class grade as the proficiency exam. These districts have no way of being able to tell me "Yes, Johnny has met the standards for technology and is ready for the world". We need a performance-based set of examinations that address the technology standards. Every technology teacher in the state will be aware of them, and therefore will help their students prepare for those. It creates the most apples-for-apples comparison out there. And, technology is not the only field. Writing, financial literacy, civics, scientific thinking, physical education, and the fine arts all need this as well. Until then, we continue to mis-prepare our students for the world.

Wednesday, September 24, 2008

Public Perception of Failing Schools

As part of my reading, I've come across the following from Sam Harris in an editorial in Newsweek (9/20/08):

The next administration must immediately confront issues like nuclear proliferation, ongoing wars in Iraq and Afghanistan (and covert wars elsewhere), global climate change, a convulsing economy, Russian belligerence, the rise of China, emerging epidemics, Islamism on a hundred fronts, a defunct United Nations, the deterioration of American schools, failures of energy, infrastructure and Internet security … .

Interestingly enough, Harris is not focusing on education at all in this article, it is rather a commentary on vice-presidential nominee Sarah Palin. But I believe the very insignificant mentioning of education paints a picture I've seen a lot of this political season. In the litany of issues the next president will face, the fact that schools are failing is a given for Harris, without any data or exposition and support necessary. Harris isn't alone, of course. Consider this editorial from the New York Times...

I urge all voters to send a message to the candidates: Tell them they must stand up to the special interests that oppose fundamental reforms because America's failing schools are a national crisis that can only be solved with strong leadership.

And from John McCain's official website...

The deplorable status of preparation for our children, particularly in comparison with the rest of the industrialized world, does not allow us the luxury of eliminating options in our educational repertoire.

And from the Washington Post...

There is no question that No Child Left Behind has brought accountability to America's classrooms... But Mr. Miller is right in saying that students are still not achieving as they should...

It is hard to find any rebuttal to this in the national media. There is no request for data to support these claims as there would be for claims about a "broken health care system" or "foreign policy that violates human rights".

It has long been postulated by Gerald Bracey as well as others that this was the grand intention of No Child Left Behind: create an impossible accountability system that creates the perception in America's people that their schools are failures. I can't speak for the originators' intention of the legislation, but that looks to be the end result. Even Obama, derided by his opponents as "being in the pocket of the teacher unions" argues that America's schools are in need of massive reform.

There are two veins of thought, then, for educators at this point. One, let's admit the reality of the situation and learn to work with a re-shaped world view. Or two, let's be strategically organized to counteract the perception that is out there. Option one is depressing, option two is monumental.

Before we dismiss option one, consider some other scapegoats of Americans. One of my personal friends in Decorah is the owner of a gas station. He is a respected member of the community... a small business owner raising a successful family. People in Decorah think he is an honest man. Of course, the same people in Decorah, like everyone nationwide, curse under their breath the oil companies, and how the prices at the community's gas stations are nothing less than price gouging. There is a dichotomy between how we perceive the ambiguous oil company and how we perceive the actual person. As is often the case, getting to truly know an enemy often makes the enemy disappear.

When I asked him about this, he would shrug and say, "That's the way it is. If people knew about how much money I was making, they wouldn't be objecting." He mentioned that there was some truth in CEO's of oil companies being overly wealthy, but there would be thousands of angry investors if the oil company decided not to work for as big a profit. Plus, he has gotten to know several of the people on the ground floor, the oil truck drivers and the distribution managers. They too are in the same boat, lambasted as a collective whole, but respected when people know them on an individual basis.

I'd say congresspeople are in the same boat. The U.S. Congress as a whole is fighting approval ratings in the teens currently. And while their pay is above the median for wages in this country, those are not 9 to 5 positions. You have to work full time. Seriously, it makes you wonder why anyone would want to become a senator. The answer is, while the Congress as a whole is unpopular, individual senators are widely popular in their districts. It is hard to unseat an incumbent (no further proof needed than Harkin and Grassley). Ask yourself, do you have an unfavorable view of Congress, and yet vote Harkin and Grassley in every time? The reality, as objective and non-partisan as I can be, is that those two senators are doing a good job for Iowa, at least in the eyes of the majority of Iowans who have gotten to know them pretty well.

Education is the exact same. In Gallup's March 2008 survey of public perception of public schools, they asked Americans what grade they would give to schools as a country, schools locally, and schools their child attends. The most recent results show the same pattern as for oil and congress.

When asked about schools in the country as a whole, 44% gave schools a C, 18% gave schools a D or an F (that alone brings into question the perpetuation of the "America's failing schools" meme that is unchecked right now, as more people give an A-B than a D-F).

However, when asked about schools in their area, 30% gave schools a C and 16% gave schools a D or an F. And most telling, when asked about schools their child attends, 14% gave schools a C and 9% a D or an F. A whopping 72% gave their own schools an A or a B.

Perhaps we need to take comfort in the truth of this "collective whole effect". Go ahead and rant against us in the media. We know you love us on an individual basis. Perhaps we have to get over ourselves acknowledge there will always be continuous disrespect just like other professions... it is the task we have. Perhaps we need to reframe our focus, not on the public's perception, but rather back to the original task of increasing student performance.

Unfortunately, there is great anger on the behalf of educators, certainly the ones that I know. They, most understandably, won't just accept that "disrespect happens". As a whole, educators are an emotional, passionate bunch... that's why we became teachers, it is what makes us good for students. Disrespect hurts. And while it is possible to work as people in congress or owners of gas station do, and maybe even in our best interests to, it isn't likely that we adopt that approach.

Which brings us to option #2. If our goal is changing public perception, then this is truly where we are failing... all of us. From NEA and ISEA to SAI to IASB to the DE and AEAs and LEAs. We are doing something wrong.

Jamie Vollmer, he of the infamous Blueberry Story and one of the people who I genuinely respect in this area, has this to say about our fight. When institutions try to go from #2 to #1, they can't do it by extolling their own virtues. The public will say, "Hey, we already have #1... why do we need you?" No, instead they need to do what Pepsi and Apple have done, show you what is wrong with #1. Research shows it is the only way to vault ahead. Research also shows that the way for #1 to combat this is either to ignore the attacks as insignificant, or if the threats are significant, attack back at #2. If #1 just extolls their own virtues, it will begin to plant a seed of doubt in the public's mind that will grow as attacks continue to mount.

The problem for schools is, unlike Coke and Windows, we can't attack back. Not because we're too good for that, but rather because the attacks do not come from competitors (charter schools), but rather because they come from people in the public. Schools cannot push back against the public... it would be a disaster. So instead, people grow uncomfortable when educators push back on the charter schools who didn't start the fight.

Vollmer argues that there is nothing more we can do than a grassroots educational campaign. Full blown, over the backyard fence, stop and talk in aisle 3 of the supermarket, setting the record straight. Listening to people's concerns and then inviting them to see what we are doing in school. Let seeing be believing.

This is unbelievably daunting. It is what we are failing at, perhaps due to lack of organization on this type of campaign. But if we don't like the alternative, we'll have to start.