5 Initial steps towards analysing scientific ‘fact’!
A high protein diet can be as harmful as smoking 20 cigarettes a day…….
Creatine killed Jonah Lomu………
We have all heard the headlines apparently based on legitimate studies and/or reports which have ‘proved’ these points beyond doubt. Firstly allow me to add a disclaimer: I am by no means saying that there is no truth whatsoever behind such headlines. However I feel it is important that we, as educated professionals, take a step back before we accept these statements as fact.
The Rogue Institute is here to teach students how to interpret these statements so that they can, throughout their careers, recognise legitimate scientific research from ridiculous claims. I will keep the following guide specific to scientific studies and I remind the reader that it is only an introductory guide to analysing them and suggests a few questions we should ask before we condemn any supplement, food group or training methodology.
Recognise that no one study can prove anything categorically.
With regard to science: the more we learn the less we know. For almost every study out there, we can find one which seemingly disproves its conclusions. Research becomes widely accepted only when numerous studies have been conducted on various populations accounting for most conceivable variables using different methods. Be aware that there are always variables. If the research is based on food diaries, what guarantee do we have that the participants didn’t forget something they ate? If it relies upon self-reporting of any kind the study can automatically be weakened.
This is when meta-analyses can become very useful. A meta-analysis considers not just one study but the body of research in the area. A useful starting point is PubMed or Medline, when looking for such. Meta-analyses consider the conclusions of all or most peer-reviewed, published studies in one area and statistically analyse their conclusions to establish validity or lack thereof. These can be hugely useful when determined whether or not you should advocate a certain course of action to clients.
Ketogenic dieting is a prime example of a body of work seemingly advocating a considerably extreme type of diet not to mention counter- intuitive to some. However it should be noted that many of the studies in the area are done on clinical populations including participants with conditions like obesity, epilepsy and diabetes. It is important to consider also the origin of the research idea in the first instance. The ketogenic diet for example was originally used in the treatment of epilepsy and in more recent years its efficacy has been considered on other populations. The origins of the research area is often an important factor in its development.
Ask yourself ‘who sponsored and/or paid for this study?’
Often it is easy to miss the fact that a study has been sponsored by a food or drinks company, a medical company or other corporate organization which coincidentally have a vested interest in the conclusions found. A study in the British Medical Journal published only last month (January 2017) concluded that ‘financial ties of principal investigators were independently associated with positive clinical trial results.’
This was also a contributing factor in the debacle that was the initial research into the low fat diet which ‘proved’ that low fat diets would prevent heart disease. It has now been widely accepted that the original research by Ancel Keys was defended by the sugar companies who ostracised sceptics for their own ends. Nina Teicholz illustrated this convincingly in her book ‘The Big Fat Surprise’ which was only published in 2014. Sugar companies could not risk their very lucrative product being exposed.
Consider the comparisons and study design used!
Firstly: how on earth can protein intake be compared to smoking?
High protein diets can be difficult on the digestive system and smoking causes no end of trouble for the respiratory system but to compare the two is nonsensical as they both effect the body in totally different ways. The point I’m making here is that these comparisons to not equate to each other and this should be considered when we read the headlines.
How large was the population studied? Was it large enough to guarantee statistical significance? Statistical validity, can only be proven if a substantial number of participants take part in a study. This is not to say that qualitative studies on small population, or indeed individuals, do not contribute to a body of research but it must be said that a great deal of further research is needed to flesh out suggestions made in such studies. The duration of the study should also be considered. Double blind longitudinal studies are widely accepted to be the most valid type of study design. So ask yourself, how close to this design the study under consideration is.
Consider the conclusions!
Try to find any legitimate peer reviewed study which does not recommend further research. You will be a long time looking. No legitimate researcher would categorically state that their research has been proven beyond doubt, at least not on paper. While most researchers are understandable invested in their work and will defend it, few are arrogant enough to profess to being the one and only expert in their field.
Where are these studies published?
What is the impact factor of the Journal? Certain Journals have been around for many years and have cemented their place in academic practise. The Lancet for example was first published in 1823. Studies published on academic search engines like PubMed and Medline can usually also be trusted to be legitimate sources. The higher the Impact factor of the Journal the stronger its influence is deemed to be. The British Medical Journal is a well-known journal and its impact factor is currently at 19. While this system is not without controversy it is a relatively reliable way of establishing legitimacy of a study.
To conclude: Don’t take anything as fact! As educated professionals it is our responsibility to question what we hear in the media. To recognize legitimate practices from temporary fads is not any easy thing, and it is made more difficult when those our clients come to for advice simply jump on bandwagons. If you have read the literature and come to believe in a certain methodology you will be able to defend your choices and prescriptions.