Yeah, The problem is always that very few, very few indeed, medical procedure rises to the prior that parachutes have... but anti-EBM people somehow have a surprisingly long list of such procedures, a list that rather resembles the list they would've made before EBM began insisting on RCTs.

I like EBM. I would love to live in a world in which it is practiced properly. One of the biggest problems with EBM, though, is that what often is advertised EBM, actually isn't.

My ex was a medical doctor, and for fun I used to read NEMJ and BMJ articles through her subscription. If you trace through references, you often find a (reasonably rigorous) study with reasonable effect observed on a group 40 norwegian women aged 40-50; which in a later article referencing it is assumed to apply to all women in that age group; and through the years is assumed to apply to all women in all age groups. And that's the part that is "easy" to trace (except no one ever does, and it's not really easy)

Additionally, the statistical evidence is inherently weaker than is presented (in a way which no one can actually evaluate) due to the lack of accounting for negative results and unreported trials.

Also, accepted login in EBM is not acceptable from a math perspective. Seth Robert's recent blog post gives a striking example: http://blog.sethroberts.net/2012/10/11/jama-jumps-to-conclus... ; Tell someone about YOUR (n=1, but better controlled than most medical tests!) experiment with Vitamin D, and they'll likely counter with "Oh, but that has been refuted. It's just placebo". That's a big percentage of how EBM is used in practice. And that's the part I don't like.

> Seth Robert's recent blog post gives a striking example: http://blog.sethroberts.net/2012/10/11/jama-jumps-to-conclus.... ; Tell someone about YOUR (n=1, but better controlled than most medical tests!) experiment with Vitamin D, and they'll likely counter with "Oh, but that has been refuted. It's just placebo". That's a big percentage of how EBM is used in practice.

Actually, it's funny that you bring that up. Yes, my own vitamin D self-experiments were more rigorous than Roberts's vitamin D experiments/anecdata, and both of my self-experiments used blinding precisely to avoid the placebo refutation, so anyone trying to refute them so simplistically would be engaged in poor reasoning; but Roberts is not innocent himself of poor reasoning himself!

I pointed out in his previous post on that study that his criticism makes no sense since the injection happened once a month, which implies the following dilemma (if we believe his criticism): the effect from bad timing should either be undetectable due to contaminating only 1/31 of the data (1 day out of each month), or if the negative effect is persistent over many days, renders every single anecdote & self-experiment (including mine) completely unreliable trash since none of them were expecting an effect which contaminates so many days. So either his criticism is wrong or his entire collection of data is worthless. (Besides that, plenty of vitamin D studies have shown benefits while completely ignoring administration time. What's sauce for the goose is sauce for the gander.)

So maybe EBM is abused in practice... but I don't see your examples showing this.

> So maybe EBM is abused in practice... but I don't see your examples showing this.

I was giving a (paraphrased) example from my history - it was 10 years ago, and I didn't keep my notes.

But Seth's example is typical: A result (negative in this case) applicable very narrowly is assumed to apply very broadly (By a top practictioner in the field, and his view is unlikely to be rejected; most readers will take his conclusion without trying to confirm or criticize the rationale).

> the effect from bad timing should either be undetectable due to contaminating only 1/31 of the data (1 day out of each month), or if the negative effect is persistent over many days, renders every single anecdote & self-experiment (including mine) completely unreliable trash since none of them were expecting an effect which contaminates so many days.

I don't follow you; In this experiment, they gave 100,000IU once a month; meaning that there must be a sharp rise in the beginning, followed by a decay which we believe (from other experiments) to happen slowly over a month.

If the effects are modulated by the _delta_ in vitamin D, rather than the _level_ (e.g. if there's a phase-locked-loop somewhere), then Seth's reasoning applies perfectly well: In the JAMA experiment, there is a (very small) constant negative delta, and nothing for a biological PLL synthesizer to "lock on" to - whereas with your experiment (and other anecdata collected by Seth), the sleep cycle can be locked onto the Vitamin D delta (with 12 hour delay), and all would make sense.

> so anyone trying to refute them so simplistically would be engaged in poor reasoning;

Yes, that's exactly my point. Most people who think they follow EBM are actually engaged in poor reasoning, and even worse meta-reasoning, as in: "The fact that other people did not notice that this evidence is inadequte, is evidence that YOUR reasoning is flawed". Yes, more than one person with an MD/PhD told me that if what I claim about how B12 is produced is right, they would have learned about it in school; even after they relented and looked it up themselves!

Furthermore, most placebo-controlled double-blinded experiments are not actually blinded: the substance or procedure being tested is known to have an effect (e.g. dry mouth) but it is not known if it has the desired effect. The placebo is known to have no effect (e.g. sugar pills). The patient therefore knows when they got the real thing (though they're still in the dark if they got the placebo).

The only way this is ever controlled for (if at all), is by comparing to an accepted substance/procedure used for the same purpose - which might, itself, be subject to the same kind of bias.

So, to cut a long story short ... The ideal of EBM is nice. The practice is very, very far away, but is assumed by most EBM proponents to have all the properties of the ideal.

> In the JAMA experiment, there is a (very small) constant negative delta, and nothing for a biological PLL synthesizer to "lock on"

Exactly. With nothing to lock onto, there should be no effect or a 'very small' effect since people will continue their normal circadian rhythms and light exposure levels which zeitgebers will, just as usual, adjust things daily - if that delta over the entire month even matters, which it probably doesn't since it's fat-soluble and should be released gradually at need from fat stores. To say that there should be a strong benefit to colds and this tiny non-existent effect explains why we don't see it (without also explaining why this doesn't eliminate all other benefits observed from vitamin D, since use of injections is common in vitamin D experiments) is just totally obvious special pleading by Roberts.

(And yes, the relevant factor cannot be the level, because that is flagrantly inconsistent with my self-experiments where the level was kept constant and only the timing was varied.)

> Furthermore, most placebo-controlled double-blinded experiments are not actually blinded: the substance or procedure being tested is known to have an effect (e.g. dry mouth) but it is not known if it has the desired effect. The placebo is known to have no effect (e.g. sugar pills). The patient therefore knows when they got the real thing (though they're still in the dark if they got the placebo).

'Most'? Cite.

I guess I can't support 'most'. It did happen in two out of three experiments for which I actually had intimate details.

But this is one of those things that will never get a trustworthy citation, and yet is probable (replace "most" with "significantly many" to increase probability according to your own probability measure).

A drug tested today on humans has, with high probability, been shown to be safe, but most importantly, effective, in an animal model - or is already known to be effective and safe in humans for another use.

Note the term "effective". That means that it is known to have a measurable effect. For many drugs, it is therefore probable that the (human) subject receiving the treatment can tell that it isn't placebo.

Specifically, quite a few psychiatric drugs are known to cause obesity, compulsive thoughts and lactation (even in males). Niacin is known to cause flushing (at lower levels, it is not externally visible, but the sensation is still there). Viagra was discovered during an attempt to treat hypertension. I'm sure the subjects could tell it from the placebo. Similarly for Minoxidil.

To actually assess how prevalent this is, you need at least access to notes made during the experiment, or worse - you need to do measurements and stats that weren't made in the first place (and which making may only make the results less promising; which is a problem in for-profit development)