Bad Science, or, the Art of Audiophile Cabling

I woke up this morning, refreshed and pleased with the day. It was a Monday, and all things considered, I like Mondays.

I never said I don’t have serious problems.

Anyway, there I was, enjoying my second cup of extraordinarily brewed Sulawesi coffee, straight from my Moccamaster, when I realized, quite suddenly, that my blogging lacked a certain something. Some pizzazz. Some character. Something … Mejias. Yep, that’d do it, thought I, just a little something to punch things up a bit.

So, with that said, today I will blog as if I was Stereophile’s newest addition, Young Stephen Mejias.

I slugged back the rest of my morning infusion of life-giving blood and grabbed the phone. Time for some perspective, I thought. How very Anton Ego of me.

As I was mulling through my first magic cup, I’d realized that my wife was out. I was a bit puzzled by this, given the weekend we’d just had entertaining our out of town friends. The last couple of nights, she’d been up late singing with the impromptu band that had formed and re-formed each night after the wee-ones had been sent to bed. I figured she’d have been down for the count, but no. It wasn’t until I got her voice mail that I remember: kids, camp, starts today. Whoops.

I left a message, sketching my idea (she’d love it) when Joe called in.

Joe wanted to know all about the shootout I staged last week. Now, Joe is something of a scientist — okay, that’s not true, he’s a goddamn rocket scientist as he works at Goddard. Quite the audiophile, too, and has been doing the DIY audiophile thing for years. Honestly, he makes me a little nervous. I never know when he’s going to start spouting something technical at me. Last summer, he actually made me do math when I started mouthing off about audiophile cables. Which is just wrong on so many levels — I left the EE program at Maryland for the Philosophy department in order to avoid as much math as humanly possible, after all. I shudder, remembering.

“So, how’d it go?” Joe asked. “Don’t tell me — you guys were, and I quote, “blown away” by the “revelatory” experience of those $7k speaker cables, weren’t you! Ha!” I could hear him chortling away as he rattled pens or firing pins or whatever it was he had stacks of in his little Greenbelt office cube.

“Well, actually, I overcooked that one. Pity, too, because it had such promise.”

He was silent a moment. No clatter from his desk told me he was furiously trying to parse what I just said. After a moment, he said, “Say what?”

I’d told him a month or two ago that I’d purchased an Audiodharma Cable Cooker. Clearly a waste of money in Joe’s eyes, since cable “break in” is a crock and, in fact, audiophile cabling is a total scam. When coat hangers wouldn’t do, Romex would, and that was the end of the matter. Electrons is electrons, says he.

At this point I decided that eggs would be just thing. Organic, free range, & local — what could be better? Jarlsberg and some chives, thought I. To Joe, I said: “Yes, it’s true. I managed to turn a Stereolab Reference cable into something that out Cardas’d a Cardas! Can you imagine?” I continued blithely, “well, I’ve had it in the main system now for a couple-five days and it seems to have settled down again.”

I could hear his sputtering.

“What’s the matter, Joe?”

“You are so full of shit!” He was practically shouting. I love getting him riled up when he’s at work. Two eggs, over easy. Now, where was that hot sauce again? Yes, I know, it kills the flavor, but we all have our vices, don’t we? And drop two slices of home made whole wheat (and organic!) sourdough into the toaster … mmm mmm, good.

As I was seasoning the eggs with some sea salt and fresh cracked pepper, Joe found his centering point.

“Look. Tell me you at least DBT’d your shoot out. Right? I mean, you didn’t do any of these change ups sighted or anything, did you?”

“Of course we did!” said I. “DBTs are a total waste of time.”

At which point, Joe might have had a heart attack. I’m not sure. There was some incoherency from the other end of the phone that lasted long enough for me to spread some farm-fresh (man, I love living out in the country) butter on those beautiful hunks of bread. I was about to chew my arm off, I was so hungry.

“Dude. You must chill,” says I. “Here’s the thing. DBTs are fine for drug testing. In fact, I can’t think of another way to reliably tell you if a drug can be reliably and effectively used by a third party. DBTs are fantastic in those cases. They can remove user bias, eliminate the placebo results and generally provide high reliability when the sample pool is large enough. But here’s the thing,” I said, sliding my now perfectly fried eggs on top of the two slabs of buttered toast. “The question isn’t, and has never been, whether audiophile-grade cables are safe to use — nor whether or not they can be reliably counted on to make an audible impact.”

Silence, then: “No, actually, that’s exactly wrong. The question is whether or not audiophile cabling can possibly have a contribution to the overall sound quality of the system they’re used in!”

“Not so,” I countered. “Consider: in a DBT, the question you’re answering isn’t whether the drug being tested isn’t effective at all, it’s whether its safe and effective for the sample pool being tested. It’s perfectly possible that a single user of the drug is wildly and successfully treated by the drug. Perhaps they’re even cured of whatever it is that ails them. Now, in a drug trial, this single case is usually eliminated from the final pool as an outlier. And rightfully so! If the success can’t be reliably replicated across the sample pool, there’s really no way to know that the drug being tested is in fact the cause of the sudden cure. This would be a problem, right?”

He was cautious now, sensing a trap. “I have some issues with this, but go on.”

“Fine. When you translate this back to audio, what you’d get is an outlier that can reliably identify changes. You, as a tester, have no idea why this one person might be able to do this, but who really cares? That’s the wrong question. You’re looking at it as a question of knowledge, and from the perspective of the DBT, such results might not be statistically relevant. Which puts you in an odd position. The test may well show inconclusivity, but … that outlier has reliably shown that he, somehow, can ID the differences. Take this a step or two out — what if that outlier is a buyer, someone actually looking to get himself some new audio cables — what do we tell him? That since none of his compatriots could reproduce his result, he’s somehow deluded? That all cables sound the same — when he himself has clearly shown that they do not?”

He snorted. “You’ve really gone off into La-La Land, haven’t you? Please, let me help you. First, you’re confusing scientific methodology with statistical sampling. If the result of a DBT test is a null result, that’s not saying that there is no effect at all, all it says is that there is no reliably observable effect. That’s different. In the case of drug testing, I’m guessing it’s about appeal — they want a drug that has reliable effects for as wide a market as possible. But in the case of the ‘wild success’, believe me, the drug companies will go nuts trying to find out if that success story is in fact a category of success stories that they can market to. It is most definitely not an outlier in the sense that it’s just discarded. Likewise for those that are exactly opposite. Failures are extraordinarily useful datapoints — they now know what to market away from. It’s the ones in the middle that are the problematic ones. The ones with the 50%-50% results. That’s the null result. That’s the group that the drug — or the study — has failed. Too many of those and you’ve got problems. Got it?”

While he was lecturing, I had lightly dosed the eggs with some Tabasco (sometimes, the classics are the best) and smushed the two halves of bread into a rough sandwich. The first bite was more than a little unmanageable, so all I managed was “mmpfh mmmpphmm” in response.

He continued. “Okay, now as to methodology, DBT is valid exactly because of the strength of statistical sampling. If something can’t be reliably observed, it’s quite possible that the thing attempting to be observed actually isn’t. Not that it doesn’t exist, or that it can’t exist, but just that people might not be able to see, hear, touch, taste or feel it. You know what I mean? There are so many ways we can fool ourselves into believing something that simply isn’t true, and DBT is the way to pull all that out — when averaged across a large enough sample pool. Man, I need some coffee.”

Between bites, I said, “And what about the guy who got it right?”

I could hear the shrug over the phone, “Maybe he got lucky.”

“Or maybe he didn’t.”

“Maybe not. Maybe he’s magic.”

“Yes, perhaps he’s a Leprechaun. Or perhaps your methodology is flawed. What you’re arguing is that since the sample pool, as a whole, can’t reliably detect a given change — or respond to a given drug in some measurable way — that there is no change to detect. Or that the change is too subtle, or too muddled, to be reliable picked up, yes?”

“Yes, that’s right.”

“So, let me say it again — what do you say to the reliable detector? Dismissing him out of hand is tantamount of fitting facts to theory — what you have isn’t an outlier, its a positive result. It’s the positive result. Yes, you have a covering theory that allows you to dismiss it, but there is an alternative explanation that is compatible with the results. Perhaps more so.”

“No, hear me out,” I forestalled him, then continued as I munched up the last delectable bits. “The issue here isn’t, ‘can people, on average, hear differences in cables’, entertaining though that’s been. That’s what a DBT can tell you, and however that comes out, that’s great. Instead, the question has ever been ‘do cables make a difference’. This is important, because that question is very different from ‘how do we know cables make a difference’. That latter question? Again, that’s the one that DBTs answer. But what we needed all along was an answer to the former. The approach of DBT, then, a waste of time.”

I popped another bite in my mouth. Is there really anything better than a runny yolk on freshly toasted bread? I think not! “Honestly, you’ve been looking at this all wrong. The positive result isn’t just whether, statistically speaking, a random person might hear the difference. That’d be interesting, sure. But that’s a question about knowledge, not about reality. The answer we’re looking for is whether anyone hears a difference. Flip it around a second: if there is no difference, no one should do much better than guessing on any given DBT. Admittedly, some DBTs probably turn out just this way. But — and here’s the kicker — not all do. Quite the contrary, there are reliable reports that there are outliers in most audio DBTs. Which that shouldn’t happen if you’re right and ‘cables don’t matter’.”

I let that hang there for a minute before delivering the coup de grace.

“The problem is that we audiophiles are not just terrible scientists — hell, we’ll believe any damn thing — but it’s that we’re also terrible philosophers. More specifically, our analogies suck. This isn’t drug testing. We haven’t been trying to find out how effective a drug is. We’re trying to figure out if the drug does anything at all. That means that unless our tests produce crystal clear universal results — which would be awesome — outliers are what we were supposed to have been looking at all along.”

“No! Outliers are statistical anomalies. They’re not data. In fact, they’re most likely errors. What statistical sampling – and this isn’t DBT, by the way – what sampling does is remove the anomalies. This is Stat 101 stuff here!”

“Bah. Joe, let me put it another way. What you’re attempting to answer with your statistics is a question about reality. Philosophers call this a ‘category mistake’ – you don’t answer metaphysical questions with a debate about epistemology — ‘does this exist?’ isn’t answered by saying ‘most people can’t tell’. That’s putting the cart before the horse. That’s Philosophy … ah … 401. Sorry. Anyway, look. Assume that you’re right – most people won’t be able to tell most of the time. Now, what about the people who supposedly can? Get a bunch of them in a room. Train the hell out of them. Teach them exactly what they expect to hear with such changes and then drill them on those expectations. Then DBT them. If you’re right, and there is nothing there, none of it should matter a whit, right? And if it’s the case that none of them can reliably pick out changes, then you’re done. But if some can? You’re also done. But what you’re not looking for is uniformity – you’re looking for evidence of existence. Be the scientist, fine. But you need to ask the right questions. And when you trot out DBT, you’re not.”

Joe hung up shortly after that, mumbling something about slippery philosophers. “I’m sure you’ve completely screwed up your logic in there somewhere, smarty-pants.”

He’s probably right. But I didn’t much care, because shortly after he rang off, my wife rang in. Lunch out on the town was just what this doctor ordered, and I’d be lying if I said a glass of Chenin Blanc didn’t just go amazingly well with a lightly tossed heirloom tomato salad. Sure, I’d probably be passed out by mid-afternoon — siesta, baby – but I figured, what the hell. It was a Monday, and I like Mondays. And an excuse to curl up in the middle of the day with my wife? What could be better?