Noise in Marking

Writing – The Most Important Factor in GPA

Writing is the most important factor when it comes to general grades and GPA. Why? Professors are not as good at marking as you would like to believe. I’ve written a couple of articles about this in the past, and Kahneman & Co. have elucidated why.

In their recent book Noise: A Flaw in Human Judgment, Kahneman, Sibony, and Sustein focus on the variability amongst experts when it comes to subjective judgments. The New York Times had this to say about this book:

“From the Nobel Prize-winning author of Thinking, Fast and Slow and the coauthor of Nudge, a revolutionary exploration of why people make bad judgments and how to make better ones–”a tour de force”.”

Noise can be defined as the difference in the outcome of judgments when it comes to “expert” judges. Professors are considered “expert” judges when it comes to grading papers. The variability between professors’ marking is huge, just like the variability between doctors’ diagnoses, judges’ sentencing, insurance underwriters’ estimates (one of their examples), and almost anywhere else that subjective judgments are made. In fact, research has shown that in many fields, the noise range is 50% or over. That means that if you have a piece of work graded by Professor 1, and you get a “B”, you are just as likely to get an “A” from Professor 2 and a “C” from Professor 3. Clearly, your grade is largely determined by who grades it.

As Sibony points out:

“One of the mottos of the book is wherever there is judgment, there is noise. And more of it than you think.”.

At this point, you might want to just throw your hands in the air and say, “Since it is all down to the luck of the draw, I give up!”. But don’t! There is something that you can do. I’ve studied this kind of thing for a long time, and all is not lost.

There are some aspects about marking papers that we know and by knowing these yourself, you can decrease the noise in the marking of your papers across professors.

One thing that you can’t control but you can wish for (luck of the draw on this one) is the paper that is marked just before yours. The contrast effect means that if the paper before yours is really, really bad, yours will look brilliant in comparison (unless yours is even worse). However, if your paper follows the best writer in the class, you’re going to look much worse by comparison. The fix for this problem is for professors to mark each paper three times, not knowing the previous marks given, shuffling all the papers between each marking, and then taking the average of the three marks. Sorry – it isn’t ever going to happen.

The best thing for you to do is to make sure that your paper is written beautifully. I was a good writer early in my undergraduate years. I was told this by two of my professors (who have since become friends). However, I was also told that I needed to improve my writing ability in my final year… advice that I did not take. I let things glide. Gliding along isn’t something you can do when there are others working at becoming better, no matter how few. The real killer was when I was told by my MSc supervisor that I needed to take some time and focus on improving my writing ability if I wanted to do well on my Ph.D.

Good writing ability is a skill and skills aren’t something that you ever stop learning. I know. I’ve written hundreds of articles and I’m still getting better. On my wall, just above my desk is a quote by Isaac Bashevis Singer, Nobel Prize winner in Literature in 1978:

“I believe in miracles in every area of life except writing. Experience has shown me that there are no miracles in writing. The only thing that produces good writing is hard work.”

Improving my writing ability didn’t just happen. I sought out help and then practiced, practiced, practiced. One of the greatest compliments I ever received was from my Ph.D. supervisor (my former MSc supervisor). I gave him a draft of something I had written and when I went in to talk to him about it, he said, “When I said you needed to improve your writing ability, I didn’t think you would ever go out and get someone else to write your work for you. You need to go back and rewrite this yourself”. He didn’t mean it as a compliment, but it was.

My research career wandered from cognitive psychology, cognitive neuroscience, and finally settled around the cognitive neuroscience underlying how people learn and how that science maps on to educational practice. A real journey with a shocking end. The Science of Learning is not systematically used in education, but that’s another story for another day.

One of the things that I did during that time was to develop a method of teaching and learning that relied on the principles underlying The Science of Learning. As I wrote in a previous post, one of the principles woven into the method is the Audience Effect. The Audience Effect has a profound effect on writing ability (and other communication skills). Almost every student I have taught using the method, which I now call the Socelor method, has seen a significant improvement in their writing ability.

If you wonder how improving your writing ability improves grades across the board, it is simple. Most students gain an understanding of whatever concept they are writing about. When a marker is reading the work, if the reading is fluent and easy to read and understand, the paper gets a good grade. Every time the marker furrows a brow or has to go back and reread a passage for clarification the grade drops. It’s that simple. Poor grammar and spelling cause small winces. Taken together, these impressions move a grade downward from wherever the marker started from at the beginning of the process. Grading is a subtractive experience with markers looking for the reasons to drop marks rather than add marks. That’s just the way it is.

There are other things involved in the marking process that affect grades, but writing ability is the number one influence.

As I wrote in my earlier article, at Socelor, we provide a model of learning that combines the force of active social learning and the opportunity of peer imitation to massively improve your writing ability. Something that to my knowledge, is not available anywhere else.

Come see for yourself at Socelor.com. You won’t be sorry.