Thursday, October 04, 2007

What Peer Review Actually Means

Do you ever wonder if peer reviewed journals are the big deal that elitist academics try to make them out to be? For example, here is a definition of a peer reviewed journal:

Peer review (known as refereeing in some academic fields) is a process of subjecting an author's scholarly work, research or ideas to the scrutiny of others who are experts in the same field. It is used primarily by editors to select and to screen submitted manuscripts, and by funding agencies, to decide the awarding of grants. The peer review process aims to make authors meet the standards of their discipline, and of science in general. Publications and awards that have not undergone peer review are likely to be regarded with suspicion by scholars and professionals in many fields. Even refereed journals, however, can contain errors.


Not only can refereed journals contain errors, they can contain "experts" who make errors about acceptance and rejection of articles based on their prejudice and fear of ideas that do not meet with their preconceived ideas of how the world should work. I experienced this first hand with an article I wrote with some colleagues a few years ago and submitted to The Journal of Legal Education whom we had contacted and they said they would be interested in such a piece. We thought an article on violence prevention would be helpful to law professors but apparently, the reviewers had other ideas. I have no problem accepting a rejection for an article but when I saw the two reviews our rejection was based on, I must say, I lost respect for this particular journal's decision to allow such lame reviews.

One review was just lame stating that our analysis of violence prevention was just "simplistic"--a buzz word for "I don't agree with your viewpoint"--and we needed to get more information from administrators and deans (the very people who have no clue about violence!) with regards to their experience with violence to get more insight etc. Not sure that would help but okay. The second "expert" review reads more like a rant than a review; the funny thing is that I can just imagine the fury of this guy who wrote a long tirade full of typos and I could just imagine his fingers clacking away angrily as he typed. Here are a few highlights from this "expert" review:

The article "Anger and Violence on Campus" is not worthy of publication. Reading it, I was reminded of the premises of so-called war on terrorism-terrorists are evil people who need to be contained or eliminated. Never mind trying to understand the underlying reasons people might resort to violence, there is something fundamentally wrong with them so we need to figure out who they are and keep them from causing harm to others. This paper takes a similar superficial look at violence on campus- those who act violently are fundamentally disturbed people who need to be identified, contained, and if possible eliminated or screened form our campuses. At no point do the author(s) follow-up on their initial observation that "vengeful individuals sought to address perceived grievances" Rather than examining the sorts of perceived grievances people sought to address, they focus exclusively on the failure of institutions to recognize "clues of impending violence" and their inability to take preventative actions.....{long rant}

I will not even bother going into detail in my concerns about the sorts of admissions screening and record keeping that are suggested at the end of the paper-the authors) must have been taking instruction from the Justice department since there is not even any recognition that there might be some issues of rights involved here.


I have no problem being rejected but I do have a problem with the bias that this reviewer shows, indicating that we are "taking instruction from the Justice department" because he does not agree with us. Why say this? I can just imagine the "I hate Bush stickers" plastered on his office door. Anyway, good news, our article has been resubmitted and been accepted by another equally prestigious law review so obviously it is not as "unworthy of publication" as the biased reviewer seemed to think.

I am not retelling this story to get revenge, I merely want to point out that peer review can often mean gate keeper to those ideas that some academics want kept out of print. Diversity of ideas often means little to some elitists when it comes to ideas that do not fit in with with their preconceived notions of how the world should be. This is not a scholarly critique so much as a political rant. We get it, he doesn't like Bush, but what does this have to do with violence in the classroom? It is, to say the least, disappointing.

22 Comments:

Blogger Cham said...

I see a reference to the Justice Department from the information you provided, but I don't see anything portion of the review that outlines the writer's hatred of Bush or his bumper sticker collection. Maybe it was in part of the analysis you didn't post.

I guess the Journal of Legal Education is free to publish any article of its choosing, it can use just about any reason for against acceptance of any submissions. Just about everything in life is biased, luckily for you there is more than one journal. "Diversity of Ideas" sounds good, but is most often a pipe dream.

And just let you know, I am in 100% agreement with you. Anyone who enters a room with the idea to cause harm to others should be removed immediately. I'm not all that interested into delving into their underlying reasons, their lousy childhood or inner demons. That is a job for someone else.

9:52 AM, October 04, 2007  
Blogger DADvocate said...

Is there any evidence that understanding the "underlying reasons" or root causes for violence or other socially deviant behavior reduces the occurence of that behavior?

It seems for several decades we've been trying to understand the root causes of violence, poverty, crime, etc. in order to lower the rates in which these things occur. But, I see little, if any, success of understanding the root causes alleviating the problems.

10:41 AM, October 04, 2007  
Blogger Danny said...

I thnk that this "peer-reviewer" who sees your paper being influenced by the Justice Dept, is missing the entire point. What is "searching for the root cause" do, wehn one has to confront a violent person bent on causing mayhem?
This reviewer would probably spend countless hours deriding thse who would suggest taking decisive action against violent people. Yet , if confronted with a violence, he/she will probably turtle up,and cry and whine for mercy.
Very typical of many a modern-day scholar, who is all about pontificating about issues and calling for more "dialogue" and "discourse" and calling for resolving of "root causes" and other such academic claptrap.
yet al lthis does nothing to answer the basic question asked- how to deal withviolentand potentially violent people and their misbehavior.
Why? Becasue most academics have no answers on how to real with real-life problems.
All they do is talk and talk and talk ad nauseaum.

10:49 AM, October 04, 2007  
Blogger SGT Ted said...

The "search for the root causes" is leftist boilerplate for "it's not the perpetrators fault, something(poverty, injustice, fill-in-the-blank-ism) MADE him do it. We must then blame the victims for their hate/uncaring/bigotry/excuse-du-jour. especially the rich, white male ones."

12:39 PM, October 04, 2007  
Blogger Earnest Iconoclast said...

I'm fine with the idea of looking for the root causes. But you do that after the threat has been neutralized. In the case of a potentially violent student, you get him out of the classroom and somewhere where he can't hurt people. THEN you might try to figure out the source of his violence. Of course, you can't honestly look for the root cause unless you accept that "He's just an inherently violent person" is at least a possibility.

EI

1:09 PM, October 04, 2007  
Blogger Serket said...

I hope the review process is more accurate than this for history and hard sciences.

2:10 PM, October 04, 2007  
Blogger Helen said...

Serket,

I hope so too! Or else the science community is in big trouble. However, look at global warming, it seemed like for a while, anyone who did not go along with global warming was a pariah but that (hopefully) seems to be changing.

To all:

I think that part of the problem is with professors and/or administrators and others who appease or overlook aberrant behavior in the classroom and do not take indicators of potential violence very seriously. Violence does not occur in a vaccum, it typically escalates one step at a time, and therefore is often preventable before it reaches a final lethal stage. If educators can understand at what point to intervene, they can often deescalate a bad situation.

3:03 PM, October 04, 2007  
Blogger David Foster said...

"root causes"...in manufacturing, the mantra ask why five times is often used. This reflects the reality that there is a *hierarchy* of causation, not a single "root cause." In the example, if you neglected to clean up the oil while searching for the "root cause," someone would likely slip on the floor and hurt themselves. If you didn't replace the gasket because you were too busy with analysis, the machine would likely lose all its oil and destroy itself.

I wonder how the abilities of the typical (non-scientific) academic at cause-and-effect reasoning compare with the abilities of the typical mechanic or shop foreman.

4:49 PM, October 04, 2007  
Blogger Locomotive Breath said...

The review process in the hard sciences is a little goofy too and a lot more fad driven than you might think.

In my former career as an academic (electrical engineering) I wrote many proposals.

One of my favorite reviews said my proposed project was "Not innovative, not feasible". In other words, "it's done all the time but you can't make it work".

Feh.

6:13 PM, October 04, 2007  
Blogger Cham said...

I do have a little story regarding violent behavior on a college campus. When I was in college way way way back circa 1980 I lived in a dorm on campus, sophmore year. One of my suitemates graduated midyear and the school provided a replacment. Enter Missy, who was on some sort of program for behaviorally-challenged bad girls. The idea was to place this highly violent and temperamental girl with massive anger issues into a dorm of mild-mannered Jewish girls. Missy showed up with a large knife collection which she stored by stabbing them into the sheetrock. She enjoyed threatening just about everybody, but for some reason kept her distance from me. She also brought her boyfriend to live with all of us, he was on the lam from the law. Her roommate immediately vacated the premises in fear for her life. I went to the Resident Assistant in search of assistance. The RA calmly explained that we should be "tolerant" and that Missy had special needs.

I also mentioned the situation to my father, who immediately called campus police. The campus police had a different opinion than the RA. Missy was gone in 60 seconds, along with her boyfriend and the knife collection. Although the two did leave a large amount of venison steaks in the communal refrigerator. I wonder whatever happened to Missy.

7:57 PM, October 04, 2007  
Blogger Unknown said...

Helen,

The Last Psychiatrist just did a great post related to this topic: http://thelastpsychiatrist.com/2007/09/ten_things_wrong_with_medical.html

9:46 PM, October 04, 2007  
Blogger Charles Montgomery said...

attribution error...

someone who doesn't like your piece is "lame" and you imagine "I hate Bush stickers" plastered on his office door."

How is that analysis different from what you think they did to you?

you're the referee. ;-)

12:30 AM, October 05, 2007  
Blogger Helen said...

Ipsedixiter,

Thanks for the link--my favorite line about peer review from the last psychiatrist was "Better example: imagine the four members of the current Administration "peer reviewed" news stories for the NY Times." That's about right.

5:43 AM, October 05, 2007  
Blogger DADvocate said...

for some reason kept her distance from me.

Cham - anyone who reads your post on Gunpowder Bridges knows why she kept her distance. She knew she couldn't intimidate you.

9:50 AM, October 05, 2007  
Blogger Unknown said...

Ah yes, been there, done that, both as a submitter and a reviewer. Peer review, like scholarship overall, has declined markedly in the last twenty years.

2:07 PM, October 05, 2007  
Blogger Sid said...

As to bias, I am reminded of an email joke. You are asked to imagine yourself and your daughter suddenly confronted by a knife-wielding psycho. You have a handgun. If you are a Democrat/Liberal/Vegetarian/Insert Group of Your Bias, you begin to ask all sorts of politically guided questions about his motivations, gun owenership, urban decay, society, government, yadda yadda yadda... If you are a Republican/Conservative/Group You Wish to Praise, the answer is "Bam". If you are Southern/Redneck/Group You Wish to Exalt, the answer is "Bam, bam, bam, bam, repeatedly, sounds of reloading, more gunshots".

The point is immediate violence must be confronted. Afterwards, much later, and from a safe location, we can look beyond the immediate case to underlying causes.

As to peer-reviewing, it only works if the reviewers are objective. Are they willing to hear the argument and judge based on the evidence provided? I have listened to several good graduate professors that were vexed due to reviews that had nothing to say about methodology, scholarship, or true academic grounds. Rather, the reviewers were concerned about publishing the article as presented because of the "implications" or because the article challenged accepted principles in the field.

I do not see this as limited to social sciences. As Dr. Helen pointed out, some scientists have paid a significant price for publishing against the party line on matters such as global warming.

12:44 PM, October 06, 2007  
Blogger TheLastPsychiatrist said...

Ah, peer review. It's much worse than you think.

Ten Things Wrong With Medical Journals

Peer review is only number 6.

3:09 PM, October 07, 2007  
Blogger TheLastPsychiatrist said...

Kill me. Should have read all the comments first.

That said, I did update the article since those comments. (Advertising.)

3:10 PM, October 07, 2007  
Anonymous Anonymous said...

I was a grad student in the hard sciences. While no one quotes the DoJ, peer review isn't really any better there.

First, the profs refuse to do the actual reviewing; they give that work to their grad students. Second, although it's supposed to be blind, you can't be assured of your anonymity. Further, by definition you know the research in your field, and you know who is writing the paper you're reading. Why? Because you heard them give a talk on this topic at last year's conference, as real journal articles come out much later than the immediate results. So, since you know who wrote it, you will never ever ever badly critique someone famous, because you will never want to be found out, and can't afford to ruin a potential future relationship. So there's no incentive to kill any sloppy work by anyone famous, but there's lots of incentive to kill a competitor whose work is not yet famous.

12:47 AM, October 08, 2007  
Anonymous Anonymous said...

It scares me to think that I once seriously considered becoming an English professor. I thought I would be like Tolkien or something - an old tweedy guy in a comfy position writing little papers about stuff only two or three other people were interested in.

I'm glad I became a sewer inspector instead.

12:11 PM, October 09, 2007  
Anonymous Anonymous said...

視訊做愛視訊美女無碼A片情色影劇kyo成人動漫tt1069同志交友網ut同志交友網微風成人論壇6k聊天室日本 avdvd 介紹免費觀賞UT視訊美女交友..........................

11:37 PM, May 19, 2009  
Anonymous Anonymous said...

4u成人358成人影城358成人影城3388影片區3388影片區25xxx成人影城25xxx成人影城24h免費成人頻道24h免費成人頻道視訊交友173視訊聊天173視訊聊天173liveshow視訊美女173liveshow視訊美女視訊聊天室v6 0080視訊聊天室080視訊聊天室0401 影音視訊聊天室0401 影音視訊聊天室

3:16 AM, June 08, 2009  

Post a Comment

<< Home