I wish to share a thought with the forum after reading for the nth time that somebody interprets the researchers not finding a significant difference as the treatments being the same, or that it makes no difference whether I do X vs if I do Y, because this experiment found no difference.
It seems important then to state that failing to find a significant difference can be related to many other factors, for example sample size not being large enough, confounding variables having a larger weight vs the experimental variable, etc.
In other words, failing to prove that there is a 95% probability that treatments are different, does NOT mean that there is a 95% probability that treatments are similar, or even 50%, or any probability. We just do not know…
Good point.
I’ll again repeat (ad nauseum) that in my opinion, casual experiments run by homebrewers should only be aiming for like 80-85% confidence, not 95%. In a non-laboratory setting, it’s just way too easy for us to screw it up. In other words, if we were to find, say 80% likelihood that maybe a variable makes a difference, I would have about 80% confidence that that were true… never 95% confidence, not in an experiment run by homebrewers.
Someone will correct me where I’m wrong, and that’s fine. I’m okay with being wrong. Cheers all.
Something that goes along with this…the key to science is repeatability. If you do an experiment once, you have a data point. If it gets done repeatedly, by different people, and most of them get the same result, then you may be on to something. That’s why I always encourage people to try things themselves. It’s also why at EB we have a team of IGORs, so that we have multiple brewers each holding tastings with multiple people.
Let’s not forget that this forum is unique in that it is impossible to 100% prove anything here. To attempt to do so is both folly and honorable. It’s what we do. And it’s not necessarily a bad thing. After all, our end product is ultimately judged entirety subjectively
This
DOUBLE this!
Much of what I get out of the various experiments is that is really hard to run a proper experiment particularly one with a triangle test.
All of the above! And I really love the thread title, Mr. Twain. 
This is very true. Homebrewers cite to a given experiment as conclusive proof of a given point but the conclusion rarely has such a broad application. Generally the results are some evidence of a particular conclusion under a particular set of circumstances.
…with a particular set of people’s tongues
Accept all inputs with grains of salt. Always with grains of salt. Nobody nobody NOBODY is perfect and knows everything about everything. NOBODY.
Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.
Which is my point anout repeatability
Including the ones we do? What would you consider an adequate number? I think the fewest brewers we’ve had was maybe 5 and each of those had around 7 tasters.
Before I retired I was exposed to GFSS. You Gould spend as much time, or more, on he design of the experiment as doing the experiment.
Mostly referring to exbeeriments. When only 12 correct tasters are needed to provide significance, it just doesn’t seem like enough data.
Am I correct, in that yours involve multiple individuals brewing separate batches? To me, that throws in a lot of other variables like water, fermentation profiles, OG, FG, etc. I am not that familiar with your process so maybe you control those aspects closely.
EDIT - I am not calling anyone’s process into question as I myself am not very scientific. I would be the last person to say anyone is doing it wrong or they should do it another way. Just some basic observations that influence my perspective.
I agree with all of the above for the most part… ultimately we are talking about individual data points, under “home-brew conditions” as opposed to “lab conditions.” And - the “results” are generally being gathered subjectively. I don’t think anyone should be looking at most of these tests/experiments as “published science.”
That said - I think that Brulosophy and Denny/Drew seem to always/almost always go out of their way to reinforce that they are not declaring anything “certain” or “factual” - rather simply sharing: “here is what we did, here is why we did it this way, here is how we evaluated it, here is what we found… check it out yourself and see what you think.”
In that regard - I really enjoy having a bunch of people checking things out and sharing what they find. It allows me to potentially agree and further feel comfortable with my own observations on a topic. Or, perhaps it inspires me to check something out further that I was curious about. Or - maybe it is just something I don’t care that much about and I ignore it.
At the end of the day - there will likely never be any one test or observation that leads to an entirely new world of brewing. However, the cumulative power of people sharing information with each other eventually has the ability to move everyone ahead incrementally over time.
I’ll bet he’s referring more to the big B-to-the-y with the umlaut.
EDIT: Ah yes, I see it above now.
I think experiments/exbeeriments are great for the hobby, as long as the experiment process is explained well, and that we take it as it should be - not as gospel but as a data point. In my mind, enough similar data points performed in a coherent fashion constitute a likelihood in my mind. Not always. I also like reading the posted professional literature and studies as well as a means to decide what I feel is applicable/feasible, or just for the sake of knowing. But what trumps it all for me is using my senses to eval my beer. After all, I don’t drink studies or experiments.
You know what? I find them interesting to read and to think about. Sometimes there’s useful takeaways. Do some critical thinking and take away what you will.
Anyone looking to these sorts of experiments for definitive proof of something is looking to hard.
If you don’t like how the experiment was set up, disregard it.
Whatever. Didn’t cost you nothing anyway.