Can we trust this?

Finding and evaluating information.

3090 words (that’s a lot!) : estimated reading time: 15 mins, 27 secs. 

A guide

There are arguably many ways that personal research could be done; I’ve tried to approach this from an exercise professional’s perspective. Hopefully, some or all of this information will be helpful in some way. You may want to skip over the sections that don’t apply to you. Please share the link!

1. Don’t be a magpie!

Magpies are easily distracted and attracted to bright shiny things. Your time is valuable, so be discerning about what captures your attention and what you search for! I’m as guilty as the next person of reading ‘snippets’ and consuming ‘sound bites’, so I get a lot of headlines and not a lot of detail. I think the key is that if anything seems like it might be important or have some meaning for me – READ BEYOND! Go deeper than those attention grabbing headlines, and try to understand the back stories.

2. Frame your question carefully.

Lawson (1984) argues that many professionals don’t put enough effort into ‘problem setting’. A lot of time can be wasted trying to solve a problem that is perceived but that may not actually exist. Or, it could be that we’ve simply misinterpreted or misunderstood an issue. The media certainly doesn’t help with its constant barrage of things we need to be worried about (e.g. sleeping with a light on!). Framing your problem is a first-line reality check. Is this really a problem that needs to be addressed or prioritised? How will solving this influence my work or guide my practice? For example, I had an email alert last week on customised cold-water immersion (CWI) protocols for athlete recovery (Zandvoort et al, 2018). I don’t currently consider recovery problematic (see my blog on fatigue) and I’m ambivalent about CWI. So my reality check says that I really don’t need to spend time following that link. If I was interested in CWI, I would probably frame a question that would consider my athlete preferences, their access to, and time available for CWI. My search would want to focus on those factors.

3. Perform a stocktake.

Your existing knowledge serves as your initial information filter, so it’s useful to consider what you already know about a topic. Even if it’s not a lot, summarise what you already know and think about that topic. What’s your gut feeling? What else do you need to know to be able to evaluate any information and findings? Try to identify and acknowledge any biases that you may hold. A stocktake process will also help you to identify keywords that you can use to narrow your search so that any search results are on-topic. For example with CWI, my master’s research was on blood flow with cooling (injury management) so I’ve remained somewhat interested in cold immersion. I’ve also seen elite athletes commit to contrast baths and cold water immersion and then abandon it after a few seasons. So I’m immediately thinking that the results can’t be that impressive if professional athletes are walking away from this practice. I know CWI takes resources and takes time out of everybody’s day. I’ve also read the latest research that suggests that inflammation might actually be important – for injury healing and hypertrophy. So I can admit to having built up a pretty strong bias against CWI being a practical solution. Any research abstract is going to have to promise major gains for me to read any further.

4. What type of evidence?

Decide on the nature of the information that you are seeking. Ask how will answering this guide my future practice? That should help you narrow your interest down to things like outcomes, details on exercise methods and protocols, client satisfaction, personal stories and experiences, recommendations and guidelines, or possible practical applications? For our CWI example, I’m going to be looking for research evidence demonstrating performance gains, ease of use and a high level of athlete compliance. So I’m going to be less interested in physiological mechanisms and more interested in practical applications.

Considering the type of evidence I’m interested in and the results from my stocktake, I should be able to generate a list of keywords to guide my search. My question – is cold water immersion well tolerated and does it improve performance? I need to identify the main concepts – cold – sports– performance + tolerance. I can grow and improve my keyword list by thinking laterally and considering possible synonyms and antonyms – ice, cryotherapy, hydrotherapy, fatigue, and inflammation. I can also include acronyms and abbreviations – CWI and DOMs. I now have a list to start my search. I’ll probably start with cold + sports + performance, but I have a list of alternates that I can use to refine and improve my search. If you are easily distracted like me, having a good keyword list can narrow your searches and remove that temptation to follow whatever pops up.

5. Identify the information sources that you trust.

When I graduated my only resources were my textbooks, my course notes, a few articles I’d managed to photocopy, and anything I’d picked up from observing others. There was little else within my reach, and most local libraries were hopelessly out of date. Today there are so many choices that your information sources will depend on accessibility, your views on trustworthiness, and your personal comfort.

Academic journals might be the preferred information source for researchers, but these are often inaccessible to practitioners. Given the information explosion, it is costly and impractical to be able to scan all of the relevant journals in our domain. There are numerous commercial databases available but these are costly.Free(ish) alternatives include Google Scholar, BASE, and Mendeley.Google Scholar is an easy and free application for searching the academic literature. Google Scholar will return numerous results and some of those will be directly relevant and offer accessible downloadable papers in PDF format. Even where articles are not available, clicking through will take you to the study abstract which will often have enough detail for you to make an initial assessment. My searches usually start out with 2-3 keywords and will sometimes expand to 5 keywords to help refine my search even further. You will find that the drop-down box in the browser top left-hand corner has a couple of useful links. You can create an alert so that you are notified by email about any new results from your search. The other useful link is the advanced search where you can specify different searches such as an exact phrase or sequence of words. I don’t want to know about patents and I’m not interested in who has referenced this topic, so I will narrow the results by unchecking the ‘include patents’ and ‘include citations’ boxes. I also will normally check ‘since 2015’, so that Google Scholar only returns the last 5 years of publications. My logic is that anything meaningful should show up in the reference lists of the more recent papers.

We often want to search for details on recommended guidelines, structured programmes, and specific exercises and these are more likely to be found in textbooks or as web-based information. Textbooks are becoming a forgotten resource for contemporary professionals, yet they remain a great resource for traditional and well-established information. Most are out of date by the time they appear on bookshelves, so even in our digital age textbooks are rarely cutting edge, but sometimes that’s enough.

Web information is something to approach with caution. I don’t use web-based information often, but if I do I’ll normally restrict my searches to trustworthy sites. For me, this means the corporate websites of official organisations (e.g. a Heart Foundation), government organisations (e.g. CDC, the Center for Disease Control) or recognised professional authorities (e.g ASCM or NSCA). These sites will usually have information and publications available for the general public and for professionals. This information should (normally) be well vetted, although these organisations often have a reputation for being relatively conservative, consensus-driven and slow-moving. The ACSM and NSCA have both been taken to task for not moving with contemporary exercise prescription principles, understanding, and practices. But I’m aware of those limitations when I read their guidelines. For example, New Zealand’s Accident Compensation Corporation still promotes cool down as part of injury prevention. While cooling down after activity is a nice habit to promote, clearly any injury, damage or inflammation is not going to be positively influenced by post-exercise physical activity (Van Hooren & Peake, 2018). So cooling down for injury prevention remains a tough sell.

While I don’t search for information from mainstream news organisations (or their web sites) they occasionally identify useful information. Although the concept of objective journalism appears to be a thing of the past, I like to think exercise and health information is immune to political influences. I’m aware that news organisations like to sensationalise stories, but as ‘reputable’ organisations they usually provide a trail back to the original research. For example, my morning read alerted me to research showing that breakfast is not helpful for weight loss! So, I followed the research trail and it turns out it was a meta-analysis (a statistical analysis of the analysis from 12 previous studies ) that found a small difference (0.44 kg) in weight favouring participants who had skipped breakfast rather than follow the advice that a good breakfast prevented overeating!! Sorry, but less than 500 grams is not going to convince me to go hungry! (Sievert et al, 2019)

Other sources are magazines, non-mainstream media, social media, industry publications, industry conferences (with workshops and training opportunities), blogs (like this one) and public forums (e.g. Quora or Reddit). While my web searches will often turn up industry, commercial and personal websites, I choose to never click through to these. Simply put, I don’t trust them as they are usually more interested in convincing me than educating me (see filtering advice below). Remember anyone can publish anything they want on the internet, true or false!!

6. Search.

That’s right, just go ahead and search using your keywords and selecting the sources that you trust. Searching requires a bit of trial and error. So try different combinations of keywords to help focus and filter your search so that the results match your question and the type of information that you are seeking. In my searches, I am usually aiming for information saturation; an admittedly arbitrary endpoint. What I mean by that is that when I get to the point where I am seeing consistency and agreement on a topic with no real contradictions, I’ve probably reached saturation and I’ll stop my search.

7. Be strategic in collecting information.

I’ve got two tips here. I’ve learned the hard way that it’s important to keep a note of any useful sources. You may read something contrary and later on you are no longer sure that you read or understood it correctly, so you’d like to go back and check. I keep a list of references or the URLs (Uniform Resource Locator or website address). You can find the URL in the address bar of your web browser – copy and paste it somewhere. The second tip is to make notes as you read. Note any key points and make sure that you can link them back to their source (URL or reference). I’ve wasted so many hours hunting for an elusive statement that I was sure was important but can no longer remember the exact wording, and can’t remember where I found it. Your research trail and research notes don’t need to be formal or extensive, they are for your future use only. My strategy is that if I think a source is likely to be useful, I copy and paste the reference and/or URL and then make brief notes or cut and paste bullet points under that reference to track my thought processes.

8. Filtering and validating the information.

This is where your professional competency really comes to the fore. The filtering process very much depends on your knowledge and understanding. For this, you need to turn your intuition onto it’s highest setting and put on your strongest pair of skeptical glasses. It’s helpful to make routine checks on things like the author’s credentials, the source or site, and the content and its structure (MacDonald & Seel, 2019). Questions to ask are;

• Who wrote it?: Is the author clearly identified and named? Are their qualifications and the organisation that they represent included? Is the organisation reputable? Is the author’s name familiar? Is their contact information listed? Have they published on this topic before?
• What’s the level of the information? Is the information pitched at the level that you’re seeking? You want information that is going to have practical significance. Highly specialised, technical, and theoretical information will probably be of limited use, just as very simplistic information intended for non-professionals may also have minimal value.
• Is it up to date? Is it dated with a suitably recent publication date? Does the information refer to current understanding of the topic?
• Is the information accurate? Separating ‘drivel’ from ‘bona fide’ information can be a challenge. Look at the style of writing. Does it come across as factual or opinion-based? Does the writing refer to what is already known on the topic and acknowledge different interpretations? Does it reference the most recent research on the topic? Do the arguments presented appear to be based on good evidence and use solid logic? Does the writer’s point of view appear objective or is there bias and emotional language? Is a reference list provided? Are key references acknowledged? Does the information rely on being an authority rather than presenting credible evidence and arguments? Authority information will generally suggest that the information is correct because the writer is an authority, or it’s based on celebrity endorsement – those sorts of appeals to authority are often not knowledge-based.
• How good is the research? If this is research-based information I want to check the research question, the methods and the organisation of the research. If it’s an intervention (exercise/training) I will glance at the research design, the number, and type of participants. I’ll be honest, I don’t pay a lot of attention to the statistical analysis but I will take more notice of raw data or group means. I will look at baseline measures and try to gauge the magnitude of any reported changes. I will take note of significance, but I’m more interested in the mean change observed and individual variability of changes. As an exercise professional, I’m always interested in individual variability because that we deal with individuals (not populations) on a daily basis. So I’ll be looking to see if there appear to be high responders and low responders. I will then normally jump to the discussion and the authors’ explanations for what they have observed or think has happened. In particular, I’m interested in the logic of their arguments and whether they have inferred causation where there may only be an association. I’ll put my 10th man hat on for a moment and take an adversarial approach to the study, trying to find reasons to refute the authors’ arguments. I’ll make comparisons with what I already know and think. I’ll then try (really hard) to put all of my biases and beliefs aside and ask – is this something that I should be including in my practice? Is this likely to make a difference for my clients? Am I simply going to be using my clients as personal lab rats to try out something that MIGHT work? An acid test for me is how does this stand up to scrutiny when the first principles of exercise (overload, progression, etc) are considered? I also need to ask ‘how will this make the people that I work with, feel?
• Is it well presented? Is the information well written with excellent grammar and a consistent style? Is there any advertising or endorsement of products or services? (I’m out if there is!) Is the information trying to ‘funnel’ you towards buying, joining or subscribing? (I’m running away screaming now!)

9. Contemplate, cross-check and network.

Now it’s important to take time to contemplate what you have learned from your personal research. Try to cross-check information from different sources and also with your current knowledge and understanding of the topic. Put your 10th man persona on and try to find any discrepancies within the information, and also discrepancies between theory and practice. Put trust in your intuition – if it seems too good to be true, then it likely is. Use colleagues or a community of practice if you can. Share your research findings and your interpretation of those findings, and seek the opinions and feedback of others.

10. Plan how to make use of your information.

Any time you encounter new information you have the opportunity to react to it in different ways. You can ignore or engage with it. You can reflect on it, dissect it, and interpret it. You could accept it, contemplate it, dispute it, or maybe discount it. You can seize on it and adopt it as part of your practice. You may prefer to adapt it to suit your context. Regardless, you will have considered the information in some way and as a professional made a decision on how this new information could benefit your practice. Planning my include figuring out how you will use the new information, who you will use it with, and when you will use it.

11. Implement and evaluate.

If you are going to make changes to your practice you should also have a strategy for evaluating the application of any information. This might be formal or informal, but by reflecting on the success or otherwise, the usefulness and practical application of any new information can be reevaluated and help inform your future practice.

In summary, although researching new information may seem like an intimidating, time-consuming and complex process, it’s possible to make it simple, methodical and fulfilling. An exercise professional dealing with new information is reliant on their explicit (foundation knowledge) and tacit (understanding based on ongoing experiences) knowledge. As exercise professionals, we should have a natural curiosity and inquisitiveness about our practice if we want to remain current. When time permits, that spirit of inquiry should lead to personal research which could take different forms. Regardless of the process adopted, exercise professionals need to develop and utilise good information filters. These will help encourage skepticism, critical thought and encourage us to take note of apparent discrepancies. Sharing ideas with your network of professional contacts before making changes to your practice is recommended – your friends and colleagues are a great final test!

Best, Phil

References

1. Lawson, H.A. (1984) Problem-Setting for Physical Education and Sport. Quest 36; 48-60.
2. MacDonald, W. B. (Academic Skills Centre), Seel, J. (Library, both University of Toronto) Research Using the Internet https://advice.writing.utoronto.ca/researching/research-using-internet/ (Accessed 7 Feb 2019)
3. Sievert, K., Hussain, S.M., Page, M.J. et al (2019) Effect of breakfast on weight and energy intake: systematic review and meta-analysis of randomised controlled trials. BMJ 364:l42
4. Van Hooren, B. & Peake, J.M.(2018) Do We Need a Cool-Down After Exercise? A Narrative Review of the Psychophysiological Effects and the Effects on Performance, Injuries and the Long-Term Adaptive Response. Sports Med 48(7): 1575 -1595.
5. Zandvoort, C.S., de Zwart, J.R., van Keeken, B.L. et al (2018) A customised cold-water immersion protocol favours one- size-fits-all protocols in improving acute performance recovery, European Journal of Sport Science, 18:1, 54-61.