I was in my office this afternoon, reading the latest press-release-dressed-up-as-a-research-report from the Sentencing Project, in which Ryan S. King “proves” that methamphetamine isn’t really much of a problem. While I was wondering how long it would take Jack Shafer of Slate to jump on it, and betting myself 10-to-1 against Shafer, or any other reporter except Steve Suo, noticing how spectacularly the report abused the data — its primary data source, the National Survey on Drug Use and Health, notoriously misses about three-quarters of the nation’s population of serious drug abusers — there was a knock on my door.
My caller was a woman of about twenty, dressed, coiffed, and made up in the expensively casual fashion characteristic of Westside Los Angeles, though not in general of UCLA. She had in her hand the green form students use to drop a course; since this is Exam Week, it wasn’t hard to guess her mission. She was there to drop my drug policy class. As I signed the form, I asked her, as I always do, why she was dropping. She explained that she’d run into a drug problem, and had spent most of the quarter at a very expensive and well-known rehab center. Naturally, I asked what drug had been her problem. “Methamphetamine,” she said.
(No, I’m not making this up. That sort of coincidence is too weird for fiction, and only occurs in real life.)
Whenever a new or resurgent problem drug appears, or appears to appear, Congress raises the penalties for dealing that particular drug. No one expects the higher penalties to have any particular result: it’s just Washington’s version of a rain dance (or prayers for rain). That is, it expresses a communal desire that some result happen, without actually doing anything to bring that result about.
So it’s natural for those who think that making insanely long drug sentences even longer is a bad idea to want to deny the appearance of new or resurgent problem drugs. Since law enforcement, Congressional and journalistic accounts of drug problems tend to float free of mundane reality, it’s never hard to prove that the people most loudly trumpeting the new problem don’t know their posterior from second base. Then people like Jack Shafer can denounce the whole thing as a hoax, and we all can go back to sleep.
Shafer has been denouncing what he sees as the methamphetamine hoax for almost a year now. And of course he’s all over King’s report like a cheap suit, demanding that any reporter who covers the meth issue from now on “perform the simple due diligence of ringing up King at the Sentencing Project for a skeptical take before filing.” In the same essay, Shafer denounces reporters for accepting cops as authorities on drug abuse and dentists (as opposed to dental researchers) as authorities on the effects of meth on the teeth.
Presumably, then, Shafer regards Ryan S. King as an authority on drug problems. But he never says why. The report identifies King as a “Policy Analyst” for the Sentencing Project. King’s profile at Niemanwatch says of King, “His research specialization is the American correctional system.” But somehow it never mentions either where King got his research training or what scholarly articles he has published in peer-reviewed journals.
Instead we learn:
Mr. King has written for the New York Times and The Washington Post and has appeared on a broad range of local and national talk radio programs to discuss sentencing and corrections issues, including NPR’s The Tavis Smiley Show and Talk of the Nation. In addition, Mr. King is frequently invited to speak at conferences and other events to discuss the work of The Sentencing Project.
In other words, King has no more qualification for judging the extent of a drug abuse problem than the typical cop or dentist. King is a pseudo-scholar, working for a pseudo-think-tank. There’s nothing wrong with writing briefing papers against excessive sentencing, which is after all what the Sentencing Project is about, but it’s not the same as being an actual expert.
If reporters working on meth have an obligation to call King for his skeptical take on the question, presumably reporters working on the latest advance in evolutionary biology have an obligation to call the Discovery Institute for a “skeptical take” on Darwin, and reporters covering global warming need to talk to Sterling Burnett of the National Center for Policy Analysis for his — that is to say, Exxon Mobil’s — “skeptical take” on climate change.
(Shafer, with the parody of objectivity that helps make American journalism the wonderful thing that it is, also quotes Richard Rawson, one of the leading researchers on drug treatment, specializing in methamphetamine, who points out that the myth of the methamphetamine myth is believed only by residents of the Boston-Washington corridor, and that “west of Ohio and south of Chicago” knows full well that the problem is only too real. Rawson has a Ph.D., a faculty appointment at UCLA, and 125 journal papers to his credit. Shafer, generously says “I find wisdom in both King and Rawson,” and concludes that reporters, after calling King, “should talk to Rawson too,” as if the two sources had equivalent credibility.)
Now of course nothing says that uncredentialed researchers publishing advocacy documents rather than scholarly papers can’t get the right answers. But King spectacularly fails to do so; he clearly doesn’t understand his data sources, and is evidently more interested in “proving” a predetermined conclusion than carefully weighing the scattered bits of evidence.
King triumphantly reports that only 583,000 Americans are regular users of methamphetamine, putting it on a par with crack cocaine. Neither King nor Shafer seems to notice that crack is a rather substantial problem, and that, therefore, if a drug as dangerous as crack has become as widely used as crack, it’s more than a mythical problem. If heavy meth use is really as widespread as heavy crack use, it’s time to sit up and take notice. And neither King nor Shafer seems to notice that an estimate of the number of regular crack users that small is wildly implausible, and that there might therefore be something wrong with the data source.
In fact there is, and the problem is well known to those in the field. King is using the National Survey on Drug Use and Health (NS-DUH, pronounced by some of us “En-Ess-Duhhhhhhhhh”). NS-DUH is a survey of the household population; it excludes those in prisons, jails, residential treatment facilities, and dormitories, as well as the homeless. Even among its target population, NS-DUH has a sample non-response rate (respondent refused or couldn’t be located) of about 20%. It’s not hard to guess that heavy drug users might be less inclined than average to open the door when the nice man from the government knocks and says he’d like to ask a few questions.
It’s possible to estimate the number of regular crack users from other sources; the right number is something like 2 million, not 600,000. (As far as drug quantities go, an estimate of the total volume of crack and powder cocaine used in the U.S. based on NS-DUH responses gets about 10% of the actual number.) If there were really 2 million regular meth users, that would be a disaster. But the real number might be higher or lower, since there’s no reason to think that NS-DUH misses a constant fraction of heavy drug users.
King notes that NS-DUH has been criticized (citing a post on this weblog) and therefore turns his attention to ADAM, which used to measure drug use among big-city arrestees.
But ADAM wasn’t any better-placed to catch the meth phenomenon than the NS-DUH. The meth-using population tends to be exurban or rural, and is typically less criminally active before getting involved with the drug than, say, the crack-using population. So looking for meth use among big-city arrestees isn’t likely to catch anything like the total extent of the problem. (There’s a quite different meth problem around the dance scene, especially in the gay community, but so far that seems to be much smaller, and again among people not likely to be arrested.) Worse, ADAM stopped collecting data in 2003, so it tells us nothing about current events.
King does note that treatment admissions for meth have quintupled, as has the number of meth labs seized, while the self-reported number of new meth users each year has doubled over the past decade. But since those observations don’t fit his story, he explains them away.
Meth is a truly nasty drug: strongly reinforcing and highly toxic, with marked cognitive deficits among long-term users even after they stop. Its use has been growing, and spreading geographically. It hits a population not very likely to use other illicit drugs (other than cannabis): rural and small-town working-class and lower-middle-class whites. That means the meth problem is mostly happening where we don’t collect much data, where the social fabric is already badly frayed, and where neither enforcement nor treatment has much spare capacity. (King correctly notes that meth is treatable; he fails to note that the dropout rates for almost all forms of drug treatment other than opiate substitution therapy are high, and that treatment doesn’t work for people who don’t attend it.)
How big a problem methamphetamine is right now, and how big it’s likely to get, are matters mostly of guesswork. We don’t have the right data to make convincing current estimates or adequate models to make strong predictions. But in overall social impact, I’d rate it third among the illicit drugs, behind cocaine (including crack) and heroin, and way ahead of cannabis, “club drugs” including MDMA, and diverted licit pharmaceuticals including opioids such as Oxycontin and benzodiazepines such as Valium and Xanax. (Alcohol, of course, swamps them all.) And meth is still rising, while the heroin population continues to age and cocaine is on the way (slowly) down.
Offhand, I’m not sure what to do about meth. It’s a hard enforcement target because, unlike crack, it’s typically bought in large multiples of the dose size, so transactions are relatively infrequent. Making pseudoephedrine harder to acquire might have helped slow the geographic spread of the problem, since sudafed is the precursor used in the home “meth labs” that are often the first sources of the drug in a new community. Even today, denying sudafed to the home cookers would have some environmental benefits. But it’s hard to imagine that the Mexican drug gangs won’t figure out a way to fill any gap in supply created by putting the home labs out of business. Expanding treatment supply where it’s scarce compared to treatment demand is certainly worthwhile, but lots of people who need meth treatment aren’t actually demanding it. Once meth users hit the criminal justice system, we can try to coerce them into treatment, but in fact probation departments aren’t very effective in getting their clients to comply with treatment orders.
Judge Steven Alm in Honolulu, where meth has been problem for a long time, is apparently getting good results by requiring frequent tests and imposing quick and reliable sanctions every time a probationer tests positive. However, I’ve been pushing that sort of program (invented, independently as far as I can tell, by J.J. Gallegher of Project Sentry, by the late John Kaplan, and by Eric Wish) for twenty years now, and in most places the organizational barriers to making it work seem to be insuperable.
So it’s possible that the meth wave is less a problem we ought to be trying to fix than a situation we need to ty to adjust to as best we can. But that’s no excuse for pretending it’s not happening. Media criticism is good clean fun, but it’s no substitute for studying the actual phenomena.
"NS-DUH is a survey of the household population; it excludes those in prisons, jails, residential treatment facilities, and dormitories, as well as the homeless."
Actually, it covers dormitories and probably many of the homeless.
——
The civilian, noninstitutionalized population of the United States aged 12 and older, including residents of noninstitutional group quarters such as college dormitories, group homes, shelters, rooming houses, and civilians dwelling on military installations.
——
"It's possible to estimate the number of regular crack users from other sources; the right number is something like 2 million, not 600,000."
Which sources are these? (Do you know when the results of the Virginia river cocaine study are due?)
One more thing –
"it's more than a mythical problem."
I never got the impression that Shafer denied that meth is a problem. Instead, his gist seems to be that no *new* epidemic has sprung up in the past couple of years. If current prevalence of meth qualifies as an epidemic, then this epidemic has existed for atleast 10-15 years now. In that light, it is disingenuous for Newsweek, and MSM in general, to wake up one fine day in 2003/4/5 and shout wolf, when that wolf has been hanging around for atleast a decade. I suppose Shafer thinks that 'epidemic' is too strong a characterization because MSM hasn't termed it as such before the last couple of years.
Allow me to guess that if Shafer were not, like me, annoyed at having to buy sudafed behind the counter, he wouldn't be giving a rat's ass about whether the meth epidemic were a myth or not.
My state has behind the counter sudafed, and front shelf mixes of pseudoephedrine and antihistamines. I had to laugh. How long before the home labs are using simple chromatography to bypass even this measure?
"So it's possible that the meth wave is less a problem we ought to be trying to fix than a situation we need to ty to adjust to as best we can."
If cracking down on a bad drug can cause people to switch to a worse one, (Which is something we've seen over and over during this "war".) maybe easing up on a bad drug can cause people to switch away from a worse one.
I wonder how bad drugs would be, if they were designed by the drug companies to minimize the side effects, rather than being designed by garage chemists to minimize the use of whatever precursor the government has decided to make hard to obtain?
Very nice post here, I'll be quoting and linking over at my blog. I just wanted to hit again at one of the points you touch on, and that is the relative lack of epidemiological data on not only the extent of the problem, but the effects of the drug over time and over a large population. For some reason the gov. and groups like Partnership for a Drug Free America have no problem spending money on campaigns against drugs (which I'm not arguing against), but aren't thrilled at the prospect of actually spending money on research to better arm ourselves against the problem.
I would be more inclined to use the term 'epidemic' for a situation in which a disease, probably contagious, spreads rapidly through a population until, perhaps, it becomes 'pandemic'. In my life I've seen a lot of problems described as 'epidemics' that in fact appeared to be in some way self-limiting.
My general impression is that if society is working reasonably well, most people will not have a drug problem. Our society seems to resemble a man who inherited money and decided to buy a power tool for every task he faces. Deciding which part is working well, and which part isn't, can get pretty complex.
Until we come to drug prohibition. With so many people in jail, and so many people using, we can see it's not working.
But Wait!- There's More!- Doctors afraid to prescribe analgesics, patients self-medicating, drug companies pushing the latest artificial analgesic, the supporting cast of drug task forces and poseur politicians- and, of course, a vigorous black market that could easily turn to smuggling something we would like even less than cocaine.
So maybe, for our own peace of mind, we need to focus on the details, and not see what's happening to us. I'm just not so sure that will help anything.
I am curious. If the NS-DUH, MTF and ADAM are not valid studies in Mr. Kleiman's opinion; which ones should Mr. King have used? It is well known that all three studies are highly flawed, a fact that Mr. King acknowleges in his report by referencing none other than Mr. Kleiman (p.12). The reality is that any report on illicit and illegal drug activity (NS-DUH, MTF, ADAM, DAWN, YRBS, Harvard CAS, etc.)is likely to either under-estimate or over-extrapolate the numbers because actual hard numbers are impossible to obtain and verify.
"(Alcohol, of course, swamps them all.) "
So does nicotine. Cigarette smoking is in decline and is now just the number 1 cause of avoidable death in the USA. And believe me it is "strongly reinforcing" that is highly addictive(he types as he chews his nicorette).
I don't often scan all the articles on Slate but this morning I did. When I saw Shafer's piece and I almost sent you (Mark) a heads-up so you could get your retort ready. Jeepers, you didn't need me. You've got that much to say from just stream of consciousness? Impressive.
Of course, I agree with what you say about Shafer and King. Two quibbles – don't call PSE sudafed. That's Pfizer brand and BTW they have reformulated it w/o PSE. 2nd a success rate of less than 10% is usually not considered effective and that's what you get with treatment programs. Even if treatment did show a higher success rate, it is fiscally impossible for most jurisdictions with meth problems to undertake programs at the needed scale.
Pardon me Mark if you've heard this before but the only forward is to systematiclay attack the meth supply chain. That means cracking down on Chinese PSE mfgrs., breaking the nexus between illegal Mexican immigrants and meth imports, using tools like GPS trackers to complicate the relationship between dealers/cookers and users/clients and by removing the source of funds for most meth purchases (petty theft, grand theft auto, ID theft). Reduce supply from China and Mexico and the ability of addicts to pay for it.
No big drug busts in Modesto, no lab seizures in Paw Paw, no miracle cures from Hythiam, just plain old economics.
BTW why does Shafer say what he says? Just to be contrarian? What a wanker.
Remove the source of funds for meth buyers? Uh, that would be American Suburbia, where everyone owns a car, houses are isolated and police are thinspread, there is a constant coming and going at all hours of the day and night, and mailboxes are made of tin.
And it's not exactly a secret that recidivist drug users (other than pot smokers) are usually people with little education and less prospects of having a good job. Which leads to the strong suspicion that if you take away one intoxicant they'll find another.
Actually, estimating the prevalence of a socially stigmatized trait is a manageable problem. Statisticians have had the methodology available since the mid-1960s.
The methodology is a randomized-response survey. On sensitive items, you have alternative versions of the question. One version is innocuous (Are you awake now?, or Did the coin show heads?) and one is the item of interest (Did you use meth yesterday?) The keys to the technique are:
1. The respondent uses some sort of randomization method to decide which question to answer.
2. The randomization procedure has a known probability of selecting each version of the question.
3. The innocuous question has known probabilities for its possible responses. The easiest way is to make P{Yes} = 0 or 1.
4. The enumerator does not know which version of the question is being answered; the respondent understands that her answer is not tied to a particular version of the question; and hence, no one can figure out if she was telling the enumerator that she used meth yesterday.
Now, even though the status of any individual respondent is unknown, because the data are a mixture of two binomial distributions, we can mathematically disaggregate the data and figure out what proportion of the population is engaging in the stigmatized behavior.
Like everything in real life, this is a TANSTAAFL problem. To get better answers, you have to sacrifice precision. That is, it costs more to get the same precision with randomized response than with direct questioning. My experience is that government survey outfits are unwilling to pay the price, they'd rather have high precision and low accuracy than slightly lower precision and much higher accuracy.
BC
Bargain Countertenor – can you point to an illustration of this technique? Also, your technique can't take care of the non-response rate (typically 20-25%).
Randomized response methods are in most modern sampling textbooks. It's typically used in small scale surveys that are after sensitive data. The application that I know best was its use in a methadone maintenance program that was required to ask if the clients had used illegal opiates in the previous week.
In principle there is no reason that you couldn't use the method in a large scale survey that involved face-to-face interview. There are training problems for the enumerators, but they aren't horrible.
As far as non-response rates go, one of the reasons for going to randomized response methods is to reduce non-response. It won't eliminate it, just as it won't eliminate false responses to the sensitive item. But it does usually reduce both.
The other drawback of RRT items is that you can't do logistic regressions or discriminant analyses on them. The mixture distribution destroys the information you need. You also have to be very careful about slicing and dicing the results.
When I'm back at the office next week I'll see if I can find a couple of cites. I can't access my search tools from here.
BC
Actually, I checked my files at home and found these cites:
Tracy, Paul and Fox, James 1981 The Validity of Randomized Response for Sensitive Measurements. American Sociological Review 46:187 – 200.
(My summary)
Tracy and Fox used RRT and direct questioning to ask about self-reported arrests in a household survey. They then compared the response with police arrest records, and found that RRT outperformed direct questioning. The RRT estimate was found to have smaller mean squared error than the direct question estimate.
Goodstadt, Michael and Gruson, Valerie. 1975 The Randomized Response Technique: A Test on Drug Use. Journal of the American Statistical Association 70:814-
(My summary)
Goodstadt and Gruson used a bifurcated survey that used either RRT or direct questioning about drug abuse in a population of high school students. They found that RRT reduced the refusal rate and produced significantly higher estimates of drug-use rates. They conclude that direct questioning results in underestimated incidence of drug-use.
Shimizu, I and Bonham, G 1978 Randomized Response Techique in National Survey. JASA 73:35-39.
(Author's abstract)
The randomized response technique was used in the 1973 National Survey of Family Growth to produce estimates of the number of women having abortions during a 12-month period in the conterminous United States. The model applied used two unrelated questions in separate half-samples, with a coin as the randomizing device. While the technique resulted in a higher estimate for the number of women with abortions than has previously been obtained through direct questions or reporting systems, it also yielded divergent estimates of abortion from the two half-samples. Possible causes for this divergence are discussed.
BC,
Maybe I don't fully understand the procedure, but it looks like it still requires the respondent to trust the enumerator, and the entire system. If I were doing something that could get me sent to jail I might not be that trusting.
BC , thanks for the cites. I'll look RRT up. Still, I must ask: how will one verify that RRT is providing more accurate answers? Alcohol prevalence can be roughly compared with (legal) sales inventory. There's no such stable, reliable reference for illegal drugs.
Bernard Yomtov: the idea is that we can't figure out who, specifically, is confessing to drug use. A simplified version of the procedure would be this:
I give you a questionaire with ten questions. For each question, you flip a coin. If you get heads, you answer "yes" to the question. If you get tails, you answer the question honestly. You can answer the question "yes" safely, because I don't know who's used meth (or whatever), and who just got heads on the coinflip. But since we know the distribution of heads and tails, we can disaggregate the data and figure out what percentage of people said they've used meth.
The problem with this method is that you need more than double the sample size to get equally good results—double because only half your sample answers the questions, and less because there's additional imprecision from having the random coin-flip factor in there (you won't get exactly 50% heads). But it removes a major disincentive to answer honestly, and so generally improves truthfulness.
The simplified problem is that people who face the prospect of criminal sanctions if they are found out have no incentive to take the chance that there's some trick involved that they don't quite understand.
Jadagul,
Thanks. That's how I understood it. I think the objection I was raising is the same as Brett Bellmore's. Rational or not, suspicions will be there.
Another problem is this. BC writes that he used the procedure in a "methadone maintenance program that was required to ask if the clients had used illegal opiates in the previous week."
I take that to mean the same people were surveyed weekly. That makes it possible to calculate how likely it is that the client had in fact used opiates at some point.
I'm sure this problem, like false responses, is well recognized. I wonder how they are dealt with.
The RRT sounds like a clever test that pretty much makes it immaterial how worried the respondent is about answering the questions. If I'm reading this right, false negatives should be screened as well as false positives.
This is good for estimating rates of use, etc. However, in my rural county the ability to detect "drug use" by probationers is so good that some of them actually ask to be allowed to serve their time- being on probation year after year, when the original sentence would have been six monthes, makes no sense to them. Nor does it to me.
Daksya,
The short answer to your question (How can you know RRT is providing better answers) is you can't.
The only way you can tell is to have some alternate source of information. When the question of RRT's accuracy has been examined, it has usually been found to be better than direct questioning. It's not perfect, but it IS (usually) better.
BC