How To Assess Evidence And Know What Works

Updated June 6, 2021

  • Do ThunderShirts® work for noise phobia?
  • Does turmeric or rosehip help canine arthritis?
  • Does acupuncture work in dogs?
  • Do dental chews clean teeth?
  • Do pheromones help anxiety?

If we took a negative attitude to natural remedies we’d never advance our knowledge. That doesn’t mean they all work though. Just because a product is available doesn’t mean it’s either effective or safe.

I’m sorry that I won’t be able to answer these questions. There are just too many brands, choices and options for me to ever do that.

What I do hope to do is show you how to get the right information to find the answers. How do you avoid bad advice and outright scams to do the best you can for your pets? You have to go to the evidence…

First you need to know why all evidence isn’t trustworthy.

The Dark Arts Are Alive & Well

Harry Potter Avada Kedavra GIF - Find & Share on GIPHY

Deliberate falsehood is depressingly common, and very hard to tackle. Expelliarmus won’t help against the Belle Gibsons of the world.

All you can do is keep your guard up. No matter how attractive the story they tell, if people won’t or can’t provide evidence then refuse to have anything to do with them.

Not everyone is malicious though. Most ineffective treatments are promoted by people who honestly believe they work. How does this happen?

Human Reasoning Is Rubbish

It’s about knowing the limitations we all share in human reasoning. For a smart species, we can make some pretty dumb decisions.

I love this list of cognitive biases that affect how we judge evidence. If you’re like me you’ll be nodding at a few of them, such as confirmation bias. I certainly don’t trust my gut instincts on whether a treatment works without hard evidence.

It’s the job of good science to weed out the human element.

Cherrypicking Is A Thing

Science alone isn’t enough. Cherrypickers are everywhere.

Science produces so many studies that inevitably some outliers will produce conflicting results. These get seized upon by those peddling the bizarre.

You name a crackpot theory, you’ll find people cherrypicking studies to support it. Unless you know about the rest of the field, these people are very convincing.

Anecdote Is Not Evidence

Anecdote is like a swear word in science, but it just means a brief story, like “My dog had a vaccine and the next week he got sick” or “I gave rhubarb to my dog with cancer and now he’s cured”. Don’t do that, by the way.

Anecdotes or testimonials give a personal face to a product claim and are very effective marketing tools. I hope you can see how prone they will be to chance events and unrecognised bias despite being sincerely believed.

The plural of anecdote is not evidence. Ten dogs treated with rhubarb means nothing without analysing:

  • How many were treated in total
  • How many got better just with a placebo
  • What the odds are that the difference was by chance alone

A recent example of this problem is the debate over the safety of Bravecto.

The Placebo Effect Fools Us All

Placebo? Wait, you say. That’s a human thing; animals can’t get the placebo effect.

Here are three recent studies that explain it better than I can.

Jaeger et al. evaluated gold bead therapy as a treatment for canine hip dysplasia. They demonstrated that when owners believed that their dog was receiving the active treatment, they reported a significantly greater improvement in pain signs, regardless of which treatment was actually used.

Muñana et al. studied seizure medications in canine epilepsy. They found that placebo administration was associated with an estimated reduction in seizure frequency of 26–46% in the population of dogs studied.

Gruen et al. found that owners giving pain control or placebo to cats reported the same improvement in their cats’ signs of pain.

We call what is happening here the ‘caregiver placebo effect’. It’s carers seeing improvement where none exists. We all do it, and yes, vets get the credit, which is nice (for us).

How To Find Evidence-Based Treatments

  1. Know how to find good science and read scientific and research papers critically. This is such a big topic we’ve written a separate detailed page on this. Read about Researching Animal Treatments And Diseases here. I also answer the turmeric question there.
  2. Trial medications with a clinical attitude. When I put The Puss on pain meds, I deliberately kept it secret from the rest of the family so they were ‘blinded’ in a mini-study. Then I waited to see what they said, which was “What’s with the cat?”
  3. Try to use things you can measure. For example, before starting an arthritis treatment, try timing the regular walk five times. Then time it when on treatment another five and compare the results. For The Puss, it was that she could climb trees again that gave me the black and white answer I needed.
  4. Do trials both on and off medication, if it’s safe to stop. The study on pain control in cats found that the owners of the cats receiving the pain medication were able to detect a worsening of their condition when they were switched to the placebo. These are the same owners who couldn’t see any difference between placebo and ‘real’ treatments when they were started.
  5. And of course, trust an expert in the field, like a university-based academic. ‘Expert’ is like a dirty word these days thanks mostly to people with extreme views. No-one else is across all the studies and no one understands the field better. An expert is also likely to have no financial incentives for his or her opinion.

With a critical attitude, it’s possible to overcome the faults we all share in our reasoning and get the best for our pets. You may also end up knowing more about a topic than I do. Which I’ll try not to mind.

References

Gruen, M. E., Griffith, E., Thomson, A., Simpson, W., & Lascelles, B. D. X. (2014). Detection of clinically relevant pain relief in cats with degenerative joint disease associated pain. Journal of Veterinary Internal Medicine, 28(2), 346-350.

Muñana, K. R., Zhang, D., & Patterson, E. E. (2010). Placebo effect in canine epilepsy trials. Journal of veterinary internal medicine, 24(1), 166-170.

Jaeger, G. T., Larsen, S., & Moe, L. (2005). Stratification, blinding and placebo effect in a randomized, double blind placebo-controlled clinical trial of gold bead implantation in dogs with hip dysplasia. Acta Veterinaria Scandinavica, 46(2), 1.

Have something to add? Comments (if open) will appear within 24 hours.
By Andrew Spanner BVSc(Hons) MVetStud, a vet in Adelaide, Australia. Meet his team here.