<< Growing Up Is Hard... | Main | Taking It Easy on Pot in Colorado >>

How Do You Conduct a Phony Pot Survey?

By sneaking in at the very end that it's simply a collection of responses from "site visitors" who wanted to put in their two cents.

Sentencing Law and Policy has up an entry touting a supposed poll by WebMD.  I went to WebMD's site and found its article, which begins as follows:

A majority of doctors say that medical marijuana should be legalized nationally and that it can deliver real benefits to patients, a new survey by WebMD/Medscape finds.

WebMD's web site for health professionals surveyed 1,544 doctors as more than 10 states consider bills to legalize medical marijuana. It is already legal in 21 states and Washington, DC. 

Now this got me a little suspicious, since (1) the AMA only recently came out against legalization, saying point-blank that pot is a "dangerous drug," and (2) I couldn't figure out how a website could conduct a random survey.

Here's the trick.  Way down at the very end, the WebMD entry says this (emphasis added): "WebMD's survey was completed by 2,960 random site visitors from Feb. 23 to 26, 2014. It has a margin of error of +/- 1.8%."

How cute.  There is no such thing as a "random" site visitor.  The people who visit a site are the ones who decide to click on it, and the people who decide to answer a poll on said site are the even narrower subset of those who want to be heard.

In other words, this "poll" has all the validity of a Glenn Beck site poll asking "random site visitors" whether Obama should be impeached.  Anyone wanna guess the answer?


Bill, the use of random site visitors was for the consumer part of the survey. The doctor part survey protocol is described this way: "Medscape’s survey was completed from Feb. 25 to March 3, 2014 by 1,544 doctors who are members of Medscape’s panel, representing more than 12 specialty areas."

I do not know how Medscape conducts its survey, and it also might depend on site visitors. But it seems possible that the Medscape approach/results are a bit more valid for doctors than for consumers.

One more point, Bill. I recall that you said that it should have been front-page news when NAAUSA polled its members (seemingly in an online survey comparable to this Medscape survey of doctors) and those whop opted to respond said they did not want to see all mandatory minimums repealed.

In part because I am still awaiting information about the NAAUSA survey process/results(and because the Medscape protocol is likewise opaque), I cannot speak directly to which online survey is more valid. But given that you thought the NAAUSA survey results were valid and huge news, I wonder if it is really fair to assail the seemingly similar Medscape survey and its results.


I agree that we need to know more about (1) how a doctor winds up on this panel, (2) how the responses were solicited or otherwise obtained, and (3) what the specific wording of the question was.

One thing that seems sure right now is that this was not just a random survey of the kind Gallup or Rasmussen would conduct, i.e., one taken by going to an inclusive almanac of all doctors and start dialing numbers that come up on a rotary wheel.

The information given thus far is insufficient to justify the entry's introduction, "New survey suggests that 'medical community supports the use of medical marijuana'." The sample size is much too small to draw any conclusions about the view of the "medical community." And, when six months ago, the much larger AMA said that its membership views pot as a "dangerous drug," I have to wonder whether the specifics we don't know about this survey -- the method of finding respondents, and the wording of the questions -- generated a skew.


First, the universe of AUSAs (about 5200) is much, much smaller than the "medical community." Second, the NAAUSA survey, I have found, asked specifically about the SSA, and not just about MM's generally. 76% were opposed to the SSA of the slightly more than 500 respondents. That is "hundreds," just as I originally said (and it's more likely thousands). Third, yes, when hundreds of non-political, career federal prosecutors publicly cross swords with the Attorney General about an issue of this importance, that is big news.

In order to understand that, just ask yourself this: If Alberto Gonzales had backed legislation for more stringent voter ID requirements, and even three -- much less hundreds -- of career attorneys in the Civil Rights Division had written to him saying they thought he was wrong and should reconsider, do you think that would have been news?

Let me rephrase that question. Do you think that would have been a Page One, above-the-fold story in the New York Times, backed up by the lead editorial decrying how dreadful it is that the Attorney General's political agenda trumps the seasoned judgment of career professionals?

Bill, I have the same question about the NAAUSA survey that you have about the WebMD survey. Specifically, I am still trying to learn (1) how many of the 5000+ AUSAs are members of NAAUSA, (2) how the NAAUSA responses were solicited or otherwise obtained (given that just 10% of the AUSA population responded), and (3) what the specific wording of the question was on the SSA.

Just as WebMD issued a (largely ignored) press release spinning what its opaque survey suggests about what docs think about medical marijuana, the NAAUSA wrote a (largely ignored) letter to AG Holder and the Senate spinning its opaque survey suggests about what AUSAs think about the SSA.

As I think we agree, the views of AUSAs about the SSA are much more important to an on-going federal sentencing reform debate than the views of some docs on medical marijuana. That is why I have been seeking more info about the NAAUSA survey since you first made much of it. And you have been kind enough to help me try to get this information. And yet, still no effort has been made to my knowledge to be less secretive about what you think should be a "Page One, above-the-fold story in the New York Times."

If/when I can get even a little more information about the NAAUSA survey, I will continue to give it lots more attention. But given the seeming similarity between the Medscape survey you assail and the NAAUSA survey you promote, I have to believe your substantive goals are driving your procedural lament.

Also, I hope you will fix your main post to more accurately represent the doctor part of this (admittedly not all that important) marijuana survey.


You're doing a first class job of trying to pivot the story away from the misleading headline of this doctors-love-pot "survey" to the NAAUSA survey -- a survey which is now old news, except it seems to you.

Let's cut to the chase. I think you'd agree that I know more about AUSA's than you do about doctors.

I will here and now offer you my affidavit that a majority of career federal prosecutors oppose the SSA.

Will you accept that as proof? Or is it your view that I would violate the oath?

The counterpart is that I'll ask you if you'll offer me your affidavit that a majority of doctors support the legalization of marijuana.

I doubt that you'll make me that offer, because (1) they don't, (2) you almost surely know they don't, and (3) I believe you are an honest man, no matter what you may make of me.


Always wanting the be helpful (ahem), I have been able to dig up some info for you.

1. There are about 1220 to 1230 members of NAAUSA. Almost every one is an active and current prosecutor.

2. The membership survey asked the following as its second question (i.e., it wasn't exactly hidden):

The Durbin/Lee bill (S 1410), will revise drug sentencing polices by giving federal judges more discretion in sentencing those convicted of non-violent offenses. The Durbin-Lee legislation also would reduce mandatory minimum sentences for most drug crimes by 50% and five year mandatory minimum sentences to two years. It would also allow previously convicted crack cocaine offenders to seek lighter sentences under the 2010 Fair Sentencing Act, even if their convictions pre-dated that statute, which reduced the disparity in criminal penalties between crack cocaine and powder cocaine offenses.

Should NAAUSA support or oppose the Durban/Lee Bill?

a. Support

b. Oppose

c. Take no position

3. The percentage saying "oppose" was 76%.


One other question if I might.

You have probably seen my entry today congratulating my friend and former colleague DEA Administrator Michele Leonhart for sticking up for the mandatory minimum sentences now under attack by the male power structure of Eric Holder, James Cole, Dick Durbin, Mike Lee and Patrick Leahy.

I know you are a strong backer of women taking leadership roles, and refusing to ride on the backseat of the good ole boys' bus.

Will you join me in applauding Michele's courageous stance against these insider/establishment men?

I do applaud the DEA chief, and I trust you saw my post praising her statement supporting sound pot research. Of course, Holder also supports MMs, he just thinks they can be overused. And I think it takes more courage for a government official to want to reduce his power than to claim he likes it and wants more.

I also thank you for more info about the NAAUSA survey, which interestingly does not ask for a personal view on the SSA but asks for whether NAAUSA should support or oppose the bill. And I do not doubt that you know well the position of many AUSAs, and I know the views of only a few doctors.

But i brought up the NAAUSA survey because it does not seem much more scientifically reliable than the Medscape survey. As I understand them, both were on line surveys of a group of members of an organization/panel that is itself a subgroup of the relevant population.


I will respond more specifically to your most recent comment, but first I want to go back to basics. There are two central questions.

First is whether most career federal prosecutors oppose the SSA. I say they do, and have offered my affidavit in support of that position.

Do you any longer doubt that the majority of career federal prosecutors oppose the SSA? If so, what is the documentation for your view?

Second is whether most doctors support the legalization of pot. You seem to have taken the position that they do support it, citing the WebMD survey of 1544 doctors on WebMD's "panel" (without either of us knowing how that panel is chosen). I think the WebMD survey is of far, far too small a group of doctors, HOWEVER THEY WERE SELECTED TO THE "PANEL," to say anything at all about how the majority of the medical profession feels about pot legalization. I have cited, in that regard, the recent AMA statement opposing legalization.

Do you still maintain that the WebMD survey shows that most doctors support pot legalization?

Covering your basics:

1. I do not doubt that many (and perhaps a majority) of career AUSAs do not want their power to threaten extra long drug MMs reduced via the SSA. What I do wonder about is how many AUSAs say they support the SSA, and also about whether some (many? most?) support FSA retroactivity that is in the FSA (and should be, I think, in a separate bill). I also wonder how many AUSAs might support cutting pot and coke sentence, but not heroin and others. I could on and on with my questions about that the NAAUSA survey might show and how your claim of there being an "open revolt" still lack any substance because I still can find very few AUSAs willing to talk openly about their opposition for the SSA.

2. I stand by the post title I used to introduce the WebMD story: "New survey suggests that 'medical community supports the use of medical marijuana'." Obviously, supporting the use of medical marijuana is MUCH different than "support[ing] pot legalization."

Notably, the AMA supports sensible research on medical marijuana and has for some time: http://medicalmarijuana.procon.org/sourcefiles/AMA09policy.pdf. And, as I think you know well, lots of medical and other research (which is badly needed) on marijuana and its extracts is really hard, especially for private drug companies, while MJ is a Schedule 1 drug. For that reason and others, I do suspect that the majority of doctors would support taking MJ off schedule 1.

Finally, and this is most critical, I think it takes much for courage for doctors to say that he favors reform of a federal law that would end up making make their jobs harder (as I think medical MJ would) than for AUSAs to say they oppose reform of a of a federal law that would end up making make their jobs harder (as I think the SSA would).

Last but not least, Bill, do you support all AMA policies on MJ reform? Do you think the feds, especially at this time of legal debate, should put the doctors in charge on this issue? I will readily say that I do not think career AUSAs ought to be leading voice as we consider federal sentencing reform, but I wonder if you think the AMA ought to be leading voice as we consider federal sentencing reform.

One more, Bill, as I found another (very imperfect) survey saying that "76% of American doctors favor of the use of medical marijuana [according to] an informal online poll from the New England Journal of Medicine.


Again, this is not proof of anything, but it does suggest that formal AMA policy might not accurately represent the views of doctors on medical marijuana (just like you know formal DOJ policy often does not accurately represent the views of AUSAs).

Doug --

"I do not doubt that many (and perhaps a majority) of career AUSAs do not want their power to threaten extra long drug MMs reduced via the SSA."

As Ronald Reagan said, there you go again.

I understand why the guttersnipe commenters on your blog so blithley impugn the motives of anyone in disagreement, but it's another matter entirely for you to join them.

Are you really so sure of your monopoly on virtue that you can't believe prosecutors oppose the SSA because THEY THINK IT'S BAD POLICY, rather than simply because it would reduce their power to "threaten" defendants?

Really -- are you?

You freely use this kind of language even though you can't help knowing that the actual threats (not to mention murder by the boatload) come from defendants, not prosecutors.

You also surely know that any defendant who thinks he's being "threatened" can tell the prosecutor to shove it and demand a trial. At that trial, the government alone bears the burden of proof beyond a reasonable doubt; the presumption of innocence alone is sufficient to acquit (as the jury is told); and the government has to get all 12 to obtain a conviction, while the defendant has to get but 1 to avoid it. So tell me once more who is holding the procedural cards?

It's also interesting that, while you attack the prosecutor as a sleaze interested only in making his job easier by being able to threaten, you blank on the mirror image you're so eagerly promoting: That the SSA, if passed, would make the DRUG DEALER'S job easier by reducing his risks and costs.

So, do tell: If this debate actually were only about whose job gets made easier, would you prefer that it be the AUSA's job, or the meth dealer's?

Doug -

"I could on and on with my questions about that the NAAUSA survey might show and how your claim of there being an 'open revolt' still lack any substance because I still can find very few AUSAs willing to talk openly about their opposition for the SSA."

Yup, that's because they'd prefer to keep their jobs rather than face retaliation from Eric Holder's political hatchet men. Do you blame them?

Beyond that, from the practicing AUSA's you have talked to, have you found a single one who supports the SSA?

Doug --

OK, let's look at the comparative reliability of the NAAUSA survey versus the WebMD poll.

To start: Neither was random, you're right about that, so that introduces at least a question of reliability as to each. On this we agree.

As you have noted, the WebMD survey was conducted among 1544 doctors who were members of an apparently pre-existing "panel." Neither of us knows how the panel was put together.

But here's the problem: The total number of American doctors dwarfs this figure. According to the Kaiser Family Foundation, there are 834,769 doctors practicing in the USA. (http://kff.org/other/state-indicator/total-active-physicians/). What this means is that the entire 1544 member WebMD panel comprises slightly less than two-tenths of one percent of American physicians. Put another way, the panel excluded 99.98 percent of the doctor population.

So, not to put too fine a point on it: Your WebMD panel -- which you grandly announce as "suggesting] that 'medical community supports the use of medical marijuana'" -- EXCLUDED VIRTUALLY THE ENTIRE MEDICAL COMMUNITY.

Far out!

By contrast, the NAAUSA survey covered its membership, which is roughly 1230 out of 5200 AUSA's, or about 24% of the total number of practicing AUSA'S. Twenty-four percent is, for sure, not a majority, but it is incomparably more likely to be representative than the two tenths of one percent you're touting from WebMD.

Given the gargantuan difference between the percentage of the relevant population each of these surveys measured, this conclusion is clear:

The WebMD poll tells you next to nothing about the overall medical profession's views either of medical OR of recreational pot. By contrast, the much more inclusive NAAUSA poll is, at the minimum, relatively quite instructive about what AUSA's think of the Smarter Sentencing Act. And they think precious little of it.

I am sure the AUSAs, Bill, think the SSA is bad policy precisely because it reduces their power to threaten extra long drug MMs. That is all the SSA would do. It does not reduce the ability of prosecutors to argue to judges for long drug sentences, nor lower the max judges can give to drug defendants who they think merit long sentences. It also does not get rid any enhancements for weapons or other aggravating drug dealing factors. What the SSA chiefly does is reduce MMs, which in turn reduces the power of prosecutors to threaten extra long drug MMs.

If you do not like my use of the term threat here, please suggest another term for what goes on when prosecutors say to a defendant that unless/until they plead guilty AND provide substantial assistance, they will be charged with a statutory MM of 5 or 10 or 20 years or more and a judge will be unable to sentence below that level no matter how strong the policy arguments might be to go lower (as in the Weldon Angelos and Chris Williams cases). I do not think "encourage" a plea is the right term for how prosecutors talk to defendants, but I would be eager to use another term other than threat that rightly describes the dynamics of negotiations that prosecutors have power within because of MMs and their severity. I do not mean threaten to be extra pejorative in this context, and I certain do not think AUSAs are "sleaze" (your word, not mine). Moreover, I am confident they want their jobs to be easier for noble (though self serving) reasons.

I believe you indicated that 26% of the AUSAs who responded to the NAAUSA survey suggested that did not want NAAUSA to oppose the SSA. Can I then reasonable conclude there are likely dozens of AUSAs who support the SSA? Sadly, not many AUSAs have reached out to speak with me even though I have sought to hear their voice and will always provide an anonymous outlet for their views. Indeed, one reason I am eager to learn more about the NAAUSA survey is so I can more fully understand the AUSA perspective on this front.

Finally, you last comment is comical. Most national polls survey only about 1000 adults out of roughly 200 million in the US. Do you say these polls "say next to nothing" because 99.999995% of adults are excluded?

What is key, of course, is how the poll is conducted. On both the WedMD and NAAUSA front, we just do not know (although there is circumstantial evidence to suggest the WedMD survey was MORE scientific than the NAAUSA poll).

Importantly, I think you previously indicated that only about 1/2 of the NAAUSA members responded to the survey. Thus, the NAAUSA results --- which I still have not seen in any form and which I still would like to see in any form --- represents only about 12% of the AUSA population.

The key point, Bill, is that the WedMD survey of doctors --- which you still fail to report accurately in this main post --- tells us something. Just like the NAAUSA survey tells us something. And I hope we will learn a lot more about what a lot of different doctors think about marijuana AND about what a lot of different AUSAs think about the SSA.

In part because you and other like-minded folks have done a strong job killing the SSA, I am not sure the NAAUSA debate matters much more. But what docs think about marijuana may be a very big deal in the months and years ahead.

Leave a comment

Monthly Archives