As I understand, in BJCP events there are two judges that judge a beer. Do they discuss the beer together when assigning scores or do they judge the beer completely independent from each other?
Sometimes there are 3 judges. It can work either way. Generally, each will score the beer independently, then will discuss their impressions and scores. Sometimes they discuss as they judge.
It just seems to me that one judge could influence another judge easily if they are discussing the beer while they’re judging. It seems often that they match up an experienced judge with a novice, which I can understand. But, my point is does this put pressure on the less experienced judge to agree or reinforce the more experienced judge’s impressions about the beer?
I try to tell anyone I judge with if they don’t detect what I do that is fine. We all perceive things at different threshold levels and taste is based upon experience. If you have never had or experienced something you will have no taste or smell memory of it.
Experienced judges should not be pressuring anyone to do anything but judge at a rapid pace.
Having stewarded competitions over the years, I’m always amazed at how much the judges of a beer are in basic agreement. Basically a good beer is a good beer and a bad one is a bad one.
What I’ve usually seen is that the judges will complete the scoresheets independently, then at the end they’ll compare notes to see if there’s anything they missed or, you know, look for opportunities to learn something. They’re usually pretty close so only minor tweaks are needed if any.
Yes, and this is exactly how it’s supposed to be done per the BJCP guidelines for how to judge.
What I’ve seen is a lot of the discussion is around interpretation of the style, which helps quantify the score adjustments appropriately, or finding the right descriptors for a shared perception. It can be helpful to talk to help verbalize what you perceive.
Tweaks to scores or verbiage can result. But it’s not typical to be off by too much with the other judges at the table. And it should not be that the experienced judge influences or dictates the scores for the newer judges.
I’m sure there are exceptions where that occurs, but it’s against what we are taught, and I believe it’s the exception.
Judges should be within 5 points if each other. Sometimes there are three judges, but usually two. You judge the beers independent of each other, then discuss, and adjust scores as necessary.
Thanks for all of your feedback. This year I’ve entered more comps than my entire 6 years of brewing and I am trying to understand the process. I hope to steward a comp soon which will give me much more insight and maybe try my hand at becoming a BJCP judge in a year or so.
BJCP guidelines judges them to comment and score independently, then discuss. Discussion usually changes scores, but not comments. Lower ranked judges are expected to be more willing to change their scores if there is a large difference, but usually they come to an agreement.
As a BJCP judge, I try not to influence my judging partners in any way, but sometimes it can’t be avoided. It’s best to allow your judging partner to complete his assessment and then discuss the results. I’ve judged beers that have been infected to the point that they were undrinkable, and this may require some discussion prior to finishing the judging process. So, there are exceptions. But, as a general rule it’s best to discuss the beer after the score sheets are completed.
So I got to thinking some more about this… Personally I disagree that scoresheets between different judges should necessarily fall within X points of each other. With decent judges, they usually will agree, but this should NOT be forced to within X points. Not ever. Honestly… who’s to say that one judge is “more right” than another? I might only be Recognized but I’ve read scoresheets from National rank judges who claimed they could SMELL astringency in the beer… baloney! Just because I haven’t bothered to re-test in 6 or 7 years to attain Certified or National or whatever doesn’t mean I should allow myself to be too easily influenced by a higher ranked judge’s opinions. If I think the other guy is wrong, I ain’t changing my score, and I would expect the same the other way around as well. I think there needs to be a place for bullheads like me. Who’s to say I’m wrong and another guy is right? My opinions are just that – my opinions. I get mad when the comp organizer says I need to be within 5 points of Joe Schmoe who I don’t even know when I think he’s way off. I’m sure the same is true the other way around when Bubba Gump with zero experience thinks he’s tasted the best beer in the universe, gives it a 40, and I score the same beer a 21. These are opinions. We try to get as close as we can, and like I say, with good judges, we usually will.
Perhaps my real wish is that we could get a whole lot more feedback from any one competition. One Recognized BJCP judge plus a Gump just ain’t cutting it. I wish we could get feedback from 3 or 4 BJCP ranked judges at every comp. Perhaps not feasible today, but maybe in 50 years it will be common. That’d be sweet. Otherwise the only way to know if your beer is really REALLY good is to enter each beer into at least 3 competitions. And that gets real expensive real quick. So anyway…
;D
I agree! I have also thought that this would allow for some interesting statistical analysis of judging and judges. If a particular judge is consistently 5 points above the other 2-3 judges she is working with perhaps her rank needs to be re-examined?
As dmtaylor mentions, this is not going to happen any time soon but we can always hope.
interesting…I put my American Barley Wine this year in three different comps all within a time frame of 2-3 months. I’ve received scores from 23-43 on the same beer…yeah a 20 point difference from BJCP judges. The 43 was from the highest ranking judge, so I 'll have to agree with him 
Initially the judging is independent by 2-3 judges, and we don’t talk while we fill out the scoresheets. Grunts, moans, ‘wow!’ remarks are deeply frowned upon. We should be doing it uninfluenced by the other judge. Definitely no table talk while the sheets are filled out. After arriving at a number we then compare and try to get the scores within some range which is instructed to us by the judge director prior to the start of judging. Usually it’s 5-7 points max.
After discussion, the judges might make a few quick additions or edits to their scoresheets .
And it’s true, after a couple beers the judges will often end up in lockstep and after judging independently will find that they are making the same comments and scoring them the same.
If you really want to learn about how beer is judged, volunteer to steward at your local homebrew competition. They are always looking for help. It’s a great way to improve your brewing too, with the knowledge you pick up from hearing the entries discussed by the judge panel. Generally, the stewards are poured a sample right along with the judges.
It’s up to the judge director to match up judging styles and personalities. So if it were up to me, I’d put you with someone equally bullheaded and let you duke it out! Nothing wrong at all with a heated debate over a beer, as long as the comments go on the scoresheet too, so the brewer gets the benefit of the debate.
I’d rather get two well written scoresheets from any rank of judges than four poorly written ones. It’s up to the judge director to keep an eye one these and make sure the brewers are getting their money’s worth from the judges. You should be able to tell what’s going on with your beer from two good scoresheets.
Lockstep is interesting. To the extent that judges are trying to mirror each other then lockstep is not necessarily a good thing. To the extent that judges are in lockstep because they have a common (hopefully correct) understanding of the style, which is often not the case initially, lockstep is generally good.
Dan,
Send me a bottle of your barleywine, I’ll sit down with two or three other judges and give you some feedback for free! :) LOL!
What were the individual scores? Was the 23 an outlier?
Any chance you had bottle to bottle variability?
It’s also possible that the competitions didn’t temper the beers correctly. An ice cold barleywine can come across thin and bitter, even though the judges are supposed to ensure the beer is served at the proper temperature.
Some competitions are more stingy with points too, and some are more generous. A lot of times it comes from the judge director reminding the judges at the kick off speech that they shouldn’t be afraid to give points out in the 40’s, and be sure they use the entire range possible. Without such encouragement, I see it all the time, scores range from 20 to high 30’s and that’s it. So we’re judging beers over an 18 point range instead of a 35 point range, from 15-50 (in general, we’re limited to 15 as the minimum so as not to be hurtful). So some of the variability you see is from that. But within the competition they should be consistent, just more or less stingy with points.
It just take a few beers to get dialed sometimes. But they should be in agreement on what the style should smell/look/taste/feel like. If one thinks a style is one way, and another thinks the style is another way it would be tough to come to any consensus. That’s why the style guidelines are so great.
As far as mirroring one another, that would mean that they’re tasting the same things, and comparing the beer to the same standard for the style, and coming to the same conclusion and score. So yes, as long as the standard they’re using (the style guideline) is correct, then mirroring is good. If they’re in lockstep and using the wrong standard then that’s bad!