Just watched an episode of Brain Games an saw a test that struck me as very interesting - they had two panels examine a set of products. The women’s panel examined and rated 5 nylon stockings and the men’s panel examined and rated 5 pairs of blue jeans - both were using a given criteria set by which to judge the items.
Interestingly, the items were identical! But the judges were not told that they were the same and rated them differently. It made me wonder if a panel of BJCP judges were set up the same way, how well they would do analyzing the same beer?
When told about the test some of the subjects then acknowledged that they thought they were the same, while others still felt that there were legitimate differences. The point is, if you expect there to be discernable differences, your brain can trick you into finding them…
Almost everyone who has entered the same beer in multiple competitions has a story about how their beer got a 40 in one competition and a 20 in a different competition a short time later. So I suppose it is certainly possible.
I think this is more like the “I don’t want to look stupid” effect. You’re given 5 identical items and told to rate them. If you say they are identical, you run the risk of being wrong and showing that you can’t discern between them. Then you look stupid. Since you don’t want to look stupid, you “find” differences.
I think the ‘finding differences’ is sub-conscious. But you have to be very confident in your own abilities to tell the person organizing that they are wrong.
this is why a proper triangle test has two stages.
first you eliminate those judges who simply get it wrong (say all are the same, all are different when two are the same etc.) and then use the feedback from the judges who got the differences right.
One time I entered a Munich helles as itself and a blonde ale, the helles took BOS and the blonde took honorable mention in BOS. The judges I talked to swore they tasted different. I bottled them at the exact same time, same keg, etc.
There’s an entire chapter in “Experimental Brewing” dedicated to evaluation, and we examine this phenomenon in some depth. This experiment has been repeated many times with the same results.
I do that with my lambics. It’s always a crap-shoot as to if the lambic entered as a lambic will win or the lambic entered as a gueuze will win. I haven’t found a pattern yet, and I don’t think I will. I will be entering a different “trick beer” into the KCBM XXXI Competition, same beer, two categories - just to see what happens.
I’ve had one of my beers tasted twice by the same judge in two different competitions just a few weeks apart, with two completely different scores. I believe it was a Baltic porter or an old ale or something like that. The first time he gave it mid-30s, and the second time a 17. Also worth noting, the first time he was judging with a good reasonable judge, and the second time with an arrogant butthole who gave it a 13. I am a judge myself so I knew that my beer deserved a score in the 30s. I hate buttholes. Think they can influence the world in any way they see fit even when they are clearly wrong. But of course it’s also the other guy’s fault for letting himself be influenced by a butthole. At least I know his true opinion.
It’s also interesting judging your own beer blind in a competition or whatever when you don’t know that it’s your own. With your bias aside, you can get some really great opinions from your own self on your own beer. Impossible when you know that it’s your own. Pretty cool stuff.
In two different comps I had an odd thing happen with the same entry: one judge said no hop aroma, the other said too much hop aroma in each comp. Just can’t please some people.
Regarding judging your own beer blind a few months ago I had an entry with a wild yeast infection. It wasn’t until I tasted it thinking I was having someone else’s beer that I noticed how bad it was.
Perception is a weird thing.
Isn’t there a bjcp rule about the lowest score being 15? 13 seems just mean.
13 is BJCP’s low, though anything in the teens must be terrible and I think many competitions/judges don’t go below 17 or 19. I mean once you’re that low, the actual score doesn’t matter anyway.
A great prank would be to send somebody scoresheets with a score under 10.
I’m just getting started. Zero experience at official beer judging. I have evaluated people though. Its not easy but having objective standardized guidelines helps. If an observable element is required and it’s not there, opinion is not needed. I think where the problem lies is as already mentioned, but also when the evaluator puts him or herself into the mix. What I mean is when it goes from “check out this beer” to “check me out, judging this beer”.
Sometimes I think you just have to risk being the dummy and say “they seem the same to me.”
I still enter contests but I wonder why…If I do well I second guess the judges and if I do poorly I second guess the judges. Judging beer can be so subjective and as others have pointed out influenced by a myriad of factors. Just remember the most important judge is YOU and whether you are happy with your beer!
There isn’t really a BJCP low score. Competitions can choose to set a courtesy score if they wish and many choose 13 as this score. Basically they don’t want people getting butt hurt with scores of like 5… Not that in the grand scheme of things there is really a difference between 5 and 13…
I’m currently enrolled in the CraftBrewers Apprenticeship through the American Brewers Guild, and just went through a whole lecture on sensory evaluation. It was some pretty interesting stuff, about the difference between sensory analysts (trained evaluators) versus organoleptic tasters (untrained, emotional tasters) and how they attempted to quantify the evaluations as much as possible. They also talked about sensory biases and how they affected the evaluations.
In short, there’s a WHOLE LOT of factors that can influence beer judging- many of which are outside the control of the brewer.
