Saturday, September 1, 2012

The Insanity of the Marzano Evaluation System...

You have to laugh at the state of education in Florida (and to some extent, the US).  While NAEP scores continue to rise throughout the country for 30+ years, you still have people screaming for the latest fad to save us.  In the case of Florida, they bought into Marzano's "causal" framework.

Let's start with the most obvious problem:  it's ridiculously convoluted.  If I were asked to create a parody of an evaluation system, I couldn't possibly make something more hilarious than Marzano.  Even they state it "works best" with a year of planning and training.  What the hell kind of evaluation system requires a year of EITHER, yet alone both?  (Answer:  A very bad one.)  If you are going to invest this kind of time (and the resources) into a program, you better have extremely high expectations for it.  Sadly, pretty much everyone with a modicum of intelligence knows that this is not going to change anything (yet alone significantly for the better).  It shouldn't be necessary to have to tell ostensibly well-educated people this, but a simpler (and thus more easily/better understood) system will work better than a complicated one.

How many "elements" (items) are in it?  The answer is 60, in four "domains" (edu-speak for "areas"; you can't charge $23/book for using simple language!).  The evaluation system would be better as a bulleted list of "suggestions"--it would have saved millions of dollars.  In that role (a list of "good suggestions"), Marzano is just fine.

But, surely it will be massively successful; after all, Marzano's system has been evaluated by none other than Marzano himself--just check out Research Base and Validation Studies on the Marzano Evaluation Model, April 2011.

To quote this (presumably) non-peer reviewed work, "[The] Marzano Evaluation Model is based on a number of previous, related works that include: What Works in Schools (Marzano, 2003), Classroom Instruction that Works (Marzano, Pickering, & Pollock, 2001), Classroom Management that Works (Marzano, Pickering, & Marzano, 2003), Classroom Assessment and Grading that Work (Marzano, 2006), The Art and Science of Teaching (Marzano, 2007), Effective Supervision: Supporting the Art and Science of Teaching (Marzano, Frontier, & Livingston, 2011)."

That's right--you know Marzano's framework will work because it's based on Marzano's work!  (Also, Phillip Morris would like you to know that smoking increases your libido, cures cancer, and prevents male-pattern baldness.)  Check out the section entitled The Research Base from Which the Model Was Developed:

"Each of the works (cited above) from which the model was developed report substantial research on the elements they address. For example, The Art and Science of Teaching includes over 25 tables reporting the research on the various elements of Domain 1."

I don't believe I've ever seen the number of tables in a book cited as evidence of the book's quality, i.e., "You know it's good because there's a lot of it."   Apparently taking lots of (often old and often poor-quality) semi-related data and mashing it together makes good research.  Only in education research is this even remotely possible.

36 comments:

  1. "The evaluation system would be better as a bulleted list of "suggestions"--it would have saved millions of dollars. In that role (a list of "good suggestions"), Marzano is just fine."

    This is exactly right. Many of the mandated garbage in the education system fits this description (LFS immediately comes to mind).

    ReplyDelete
    Replies
    1. It's sad, especially with recent budget shortfalls, that the politicians continue to push this worthless--and expensive and time-consuming--garbage onto us.

      Had to look up LFS; fortunately, my district (or at least my school) has been spared this bunkum for now.

      Delete
  2. I keep hearing "only 2% of teachers are innovating/highly effective" according to Marzano. Sounds like it was more like an investment on the part of those who were scared about paying any merit pay.

    ReplyDelete
    Replies
    1. I only found one reference to the "2%" and that was in his "Race to the Top" "whitepaper" (read: Paper that looks academic, but really isn't any more so than this blog...in fact, given that it's Marzano, I'd say my blog is probably more credible simply on the basis that any positive number is greater than zero). In it they state,

      "Marzano (2009) has noted that a teacher who is classified as “most effective” (i.e., at the 98th percentile in terms of his or her pedagogical skill)
      will be expected to produce student achievement that
      is 54 percentile points higher than the achievement
      produced by a teacher who is classified as “least
      effective” (i.e., at the 2nd percentile in terms of his or her pedagogical skill)"

      A 54 percentage point increase? That's just laughably unrealistic. One of these days I'm going to have to slog through some more of Marzano's pseudo-research (aka "meta analysis") to try to figure out just how incompetently the numbers were crunched. In the mean time, I'll leave it to Dr. Wexler's analysis:

      http://www.amazon.com/review/RYB2CUDKIISEP

      PS Notice how surprisingly few reviews Marzano's books have?

      Delete
    2. "Sounds like it was more like an investment on the part of those who were scared about paying any merit pay. "

      And yes, that's definitely a benefit for administration/politicians: Not only will this "improve" the schools, but it all-but-guarantees not having to pay out a lot of money in the process.

      I'm willing to bet Dr. Marzano's proctologist helped him find that 2% figure.

      Delete
  3. A bit more of why I don't trust anything from "experts" such as Marzano:

    http://ventingmycynicism.blogspot.com/2012/10/marzanos-database.html

    ReplyDelete
  4. Thank goodness, I found someone else who would dare to question the master, the guru, the end all of education. While he is making millions establishing his brand, public school educators have been thrown under the bus, and he is at the wheel, driving forward and backwards impressing every school administrator in ear shot of the screams of agony wrenching out from under the bus! I will listen to him when he comes into my school and spends one entire year implementing personally his "strategies" of effective classroom management, etc. Who evaluates his work and dares to test it to prove it more than a quick fix, pie in the sky phenomena. Let's throw Hunter in the mix with it, shall we?

    ReplyDelete
    Replies
    1. School administration (particularly at the district and state levels) are proof positive that there's a sucker born every minute...

      And good luck getting the "esteemed" Marzano to spend time in a real classroom and not one picked out for photo ops. ;)

      Delete
  5. I'm so glad that I found this site! My school here in Queensland Australia is all fired-up about this model, while I thought what a HUGE waste of human resources. Our Principal is a true disciple and flew to a workshop in the U.S.A. to really get soaked in Mazano strategies. We just this afternoon had an hour long meeting on one point; a design question that otherwise would have been a two minute casual discussion. I thought I was the only one on earth that thought this is all a great temple built of cardboard and wishing it would all blow away. I have taught for 25 years and have never seen such a complicated, response-funneled, mind-directed, time-consuming waste of time.

    ReplyDelete
    Replies
    1. Sorry I didn't see your reply sooner; haven't used the blog much. But I'm more sorry for all the horrible things my country has exported to the world--Marzano may be the worst yet. Marzano is just a waste of time and money--anyone with common sense can see that. Unfortunately, the policy makers at the state-level are hardly known for doing anything sensical.

      Delete
    2. Thank for the sympathy! But it's now got worse! Seminars are big time here now. The principal just returned from a $600 two day seminar and stated that it was so great to hear other principals speaking the same Marzano talk, (couldn't any of them tell its repackaged much more of the same?). The principal added that other schools love it, even those teachers who have taught for 30 years! WHAT! I don't believe that, I am sure it has to do with the questions one asked about this 'strategy', because when I ask teachers what they think they can't stand the amount of time it takes to respond to the 'design questions'. All that the Marzano system does is gather stats quickly for portfolios, waste a lot of time, install a weird lingo into staff meetings and worst of all creates a great divide. We now have the disciples and the commoners. The beguiled believers and the frustrated teachers who want to get onto the real tasks involved with this profession.

      Delete
  6. Mary Wilburn March 6 2014 - This too will pass but unfortunately not untl it has driven all of the good teachers out of education. Marzano is such a bunch of convoluted bunk leavng little to no room for individuality in teaching styles.......how boring for our students. Really....60 things we have to do every lesson to be good teachers. Why not. Make it 65 tell treachers how to all smile....walk....sit.....stand....and take a #@*$ exactly the same way so the can get a check in the right collumn. Young administrators are all hyped on Marzano talk like cult members. It is a pathetic situation.

    ReplyDelete
    Replies
    1. Sorry for the late reply. Yes, Marzano is convoluted and has at best no return on investment. The research behind it is basically incestuous ("You know Marzano's system works because it's based on...Marzano's research!"). If the "effect sizes" actually meant what Marzano, Hattie, and others claim, any teacher could just do one, two, maybe three of these strategies and their students would be waaaaay better than average.

      As for the administrators, I once made the mistake of asking an in-depth question about effect sizes; I'd never seen someone so blatantly "caught". I'm willing to bet asking just one question these poor people don't have a talking point for (i.e., a detailed or technical question) will reveal the same thing--that they really have no idea what's underpinning Marzano's system.

      Delete
  7. We teachers are intelligent. Lets try to take action against this guy and other legislature. (Maybe senators, firemen and policemen also need cuts and job evaluations,) why are public teachers being targeted so heavily? do you think it has to do with pension money????? Vote these people out, its still a democracy. If enough people write, sign petitions to their unions, maybe we can turn this. We can't take this lying down! Use your voting and legislation rights.

    ReplyDelete
  8. Any chance that I could get a copy of Marzanos database that you copied and pasted

    ReplyDelete
    Replies
    1. http://depositfiles.com/files/tisvbfmpd

      Delete
  9. You are an idiot and likely what is wrong with teaching - your childish outbursts have stalled educational reform and will continue to. Stop believing you know everything.

    ReplyDelete
    Replies
    1. "...likely what is wrong with teaching..."

      The overwhelming majority of my students, their parents, and my administrators seem to really like the job I do. But I'm sure you, random internet denizen, know better than these hundreds of people. (I hope you now see the irony/hypocrisy of your final statement of, "Stop believing you know everything.")

      "your childish outbursts have stalled educational reform and will continue to"

      1. I certainly hope so but I sincerely doubt it. Outside of fellow teachers--who have essentially no influence on education policy--I doubt anyone of genuine influence has or will ever read this. But I can dream.

      2. Let's suppose you're actually right. What does it tell you about the facts--and millions upon millions of dollars--behind "educational reform" if all it takes to stall it is a "childish outburst?"

      "Stop believing you know everything."

      I certainly don't believe I know everything, but having survived four years of engineering school, several years of teaching, and extensive reading on the subject, I know enough to know when people and ideas are hopelessly, ridiculously incorrect. Though in the case of Marzano, all you really need is common sense. (If that's not enough and you are statistically inclined, go look at his database. Be prepared to laugh and cry.)

      Delete
    2. Agreed. The mantra of "evidence-based" is a ritualized slogan in my neck of the woods. Sadly, most have no clue what evidence-based even means. When I inquire about the plethora[sic] of peer-reviewed studies that justify a thorough and complete erasure of current systems while simultaneously implementing Marzano as a county evaluation tool... I get crickets. This thing has become a religion.

      Delete
    3. Yup. It's "too big to fail" now. Too many people would have to admit they just wasted (tens? hundreds? of) millions of tax dollars throughout the state. And of course, they'd have to be re-assigned to do something else...

      Thanks for commenting! I really need to update this blog at some point...

      http://education.modernsense.net/2013/01/what-do-banks-and-bad-ideas-in.html

      Delete
    4. As Trash? I'm sure it's all about the Benjamins. So who gets kickbacks for implementing this trash?

      Delete
    5. Yep--it's all about the $$$, not education.

      Delete
  10. Well, I am a retired high school teacher. I knew that as soon as my school adopted Marzano, I was going to go the way of the Dodo. He is a fraud, plain and simple. At best his 41 elements comprise either nonsensical suggestions or suggestions that are so common sense, most teachers have been doing them for years. Ii am out and now he can peddle his snake oil to my replacement.

    ReplyDelete
    Replies
    1. Yeah, my school seems to be slowly realizing that we've spent an ungodly amount of time on it and have gotten absolutely nothing in return. One AP is still drinking the Kool Aid, but most of the administration recognizes this for the bunkum it is.

      Delete
  11. I am retired and glad that my replacement will now have to put up with the Marzano crap. He is a fraud and many of my teacher cohorts know it, but they have to pretend to go along or lose their jobs. Meanwhile, Marzano and his half-baked cronies just get richer.

    ReplyDelete
    Replies
    1. Yup...little to do with education, a lot to do with making $$$. Congrats on the retirement!

      Delete
    2. Well, I've been on this site before, but I thought I would put in my two cents worth one more time. Now that I have been retired for a while and working part time at a local college, I rarely hear about Marzano. I am not sure if his brand of fraud has passed the way of the dino or what? But, I do know that education now has new problems in the form of funding. Higher Ed has bigger issues as do all schools in my state (MO) -- lack of money, consolidation of programs, cutting of faculty and staff. I do think that Marzano's main focus was always on the K-8 level and only sometime toward 9-12.Marzano never worked well with the high-achieving AP classes I taught. I'm not trying to be a snob, just stating an observation. Now I teach everyone from high to low ACT. My teaching style has had to adapt. Other than studying Marzano as just another educational theorist in an ed department, I never hear of him anymore. Good riddance!

      Delete
    3. Yes indeed. What a scam!

      Delete
  12. Marzano hasn't gone away, retired AP high school teacher. I'm dealing with his boatload of fraudulent measures here in ME where I teach art. Such a collossal waste of our time and taxpayer money, but you know that Bob Marzano is cashing in big time. I still love teaching but this stuff makes me want to get out a.s.a.p.

    ReplyDelete
    Replies
    1. PS Thanks for hosting this site and giving us folks who are still in the trenches a place to vent. I just wish I could speak truth to power and tell my admins what I think of this Marzano business.

      Delete
  13. Thank you so much for this blog. I have been teaching in Florida for almost 10 years and for about 6 of those years we have been dealing with this Marzano garbage. Hundreds of teachers in my district have either retired early, quit for another district not using this foolishness or have left the profession altogether. It's really sad. I don't think this should have ever been used as an evaluation tool. It's just a possible resource that teachers can use to help them get stronger in a specific area by tracking their progress from beginning to innovating in that area. It shouldn't be the actual tool itself. Now the Marzano center has released a more consolidated version of the model so instead of 41 elements in domain 1 from example there are now 20-something. They're trying to make it work when they should just throw it away. But of course the money spent on it is what they are thinking about. It's ludicrous. I pray that next school year is my last one in the classroom and am even taking additional graduate courses so that it can happen.

    ReplyDelete
  14. Marzano are not alone! There are many other fraudsters like McRel. They're sending, to every school,"well trained" and "experienced instructors". Usually, they'll try to teach/train you on the same topics and basics, learned by a new student teacher in their first semester in the teaching program. WTF? And don't forget about the other bull poop with the teacher coaches! We don't have money for the classroom, but there's plenty to pay these impostors.

    ReplyDelete
  15. I've been searching the internet for something to tell me how to do week on a marzano evaluation. Somehow u doing this and a light bulb went off. Maybe u should do what I know to be best for my kids and screw getting highly effective. Add long as i get to keep my job. Thanks for saving my sanity!

    ReplyDelete
  16. In my opinion Marzano is crap. His theories are ridiculous and they treat children as mentally retarded, unfortunately they begin to act like mentally handicapped too. The evaluation system is very general and it doesn't adjust to the classroom/teacher needs but to a general scenario that many times doesn't exist.

    ReplyDelete
  17. So discouraged with having to "prove" I do all of the items in every domain. Three more years teaching for me, if I can hold out that long. It's sad, because I used to really love my job. Now, I can't stand it. Thanks, Marzano

    ReplyDelete
  18. To put it quite simply, Marzano is a hack. The guy taught ONE year and couldn't cut it, so he found a way to make money telling others how to do what he couldn't. For all his preaching about being 'highly engaging', ever sit through his video 'teaching'? A monotone droning voice as he SITS ON A COUCH! How about applying all those 'methods' to yourself! I taught in FloriDUH for 18 years and was thrilled to get the hell outta there. Marzano IS A HACK WHO KNOWS NOTHING!

    ReplyDelete