[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [ARSCLIST] ABX for Dummies
Hi, Bruce,
I concur with Richard, as usual. Allow me to add a few points from
someone who has never participated (much less designed or evaluated) an
ABX test (but has casually been part of many a blind test, not to
mention blind wine tastings ;-)). This is probably my last posting on
the subject, as I fear that some seem to hold rather religious beliefs
on either side of the issue.
1. All tests have limitations. I have no doubt that some use ABX testing
as "a convenient rhetorical device to dismiss an opinion with which one
disagrees", but it seems that it may have value for some of us. Even
within that framework, however, there are ABX tests better designed than
others. For example, the original ABX designers suggest showing a more
exaggerated effect (say, MP3-encode something five times serially) to
"train" those who are not "golden-eared". I guess I intrinsically trust
the idea, but I am sure it can be perfected. Is there a better test out
there? One that can reliably show that the average listener *can* tell
the difference between 16 bit and 24-bit encoded audio? There very well
may be.
2. Engineer members of ARSCList (as opposed to an audiophile list) are
more concerned with fidelity rather than the perception of sound. Thus
we are more worried about whether a process or a piece of equipment
changes the sound at all. To do this, we do A to B comparisons
constantly, but ideally we would like not to be influenced by other
factors. For example, I think most audio engineers have had the
following experience: you are equalising a channel, turn the knob(s),
think "ah... here we go", and then realise the EQ is not engaged. Can we
accept that the mind can play tricks on you? Then it seems a
well-designed blind test is helpful. But perhaps it is not for everyone:
As I said in an earlier post, seeing an expensive amplifier does change
the perception of its sound, much as placebos can cure people --actually
cure them; and people who hear such differences or are cured by placebos
are no more nor less honest, gullible, or malicious than those who do not.
3. I very much enjoyed your "shades of paint" scenario, but now allow me
to propose a different one. A salesman comes to my house with a bag of
exotic crystals that he swears will make a my recordings sound closer to
the original when placed in a corner. If I fail to hear the differenceof
crystals vs. no crystals, could I be blamed for not buying them if my
budget is limited? That is all some of us are trying to do: trying to
figure out what to spend our money and resources on --and it seems the
weight of the proof should be on the positive hypothesis. (Also, I would
submit that if I painted my mother's house with Pewter Fog instead of
Pearl Mist, she would not be able to tell the difference).
Cheers,
marcos
Go to a paint store. As we want to avoid “coloration”, go to the rack
of “Neutral” paint patch strips, and choose one where the differences
in tint, hue, and saturation are minimal. (Of course, to be truly
”blind”, you can have someone else do all this). While the clerk is
distracted, swipe 11 strips. After you get past mall security, pick
the two patches that are the closest. Let’s say they are called Pewter
Fog and Pearl Mist. Cut out the patches (eliminating the names) and
label the backs of Pewter Fog A and the Pearl Mists B. Cover 10 sets
of A and B labels with opaque but removable tape. Set aside the
remaining pair labeled A and B, and dump the rest into a bowl deep
enough that the contents cannot be seen or counted.
Place the two labeled A/B patches in front of the subject, label side
up. Place one empty deep bowl labeled A to the left, and another
labeled B to the right. Put the bowl of taped samples where they can
be reached but not seen.
The subject (it can be yourself) can turn over A and examine it but
must replace it face down to look at B, and vice-versa. Placing both
face down, a sample is drawn from the bowl, and the subject must
decide if it is identical to A or B, and place it in the appropriate
bowl. The process is continued until all the samples are sorted. You
can then peel back the tape and see how many were correctly ID’ed and
sorted. Any more than 10 correct and you are better than chance. Score
20 and you can be referred to on this list as “Golden-Eyed”.
While this seems a pleasant enough parlor game, it would not cut the
mustard in even an intro course in Experimental Psych. The result
above is just a single data point; one needs another variable for the
test to have meaning. Repeat the test several times to see if the
skill can be learned. Test 100 people and sort by age, gender,
religion, etc. Change the lighting from incandescent to fluorescent.
Compare putative experts (artists, house painters) to the overall
population. Choose different patches and correlate to the pigment
differences. Etc. And of course, you can have lots of fun twisting the
statistics, but we won’t go there right now.
If very few subjects can reliably tell the difference between Pearl
Mist and Pewter Fog, they are perceptually identical, according to the
statistics typically applied. If the new CEO of Benjamin Moore was
hired from Crayola, he may decide to "dumb down" the variety of colors
available and eliminate one (or both) colors from the line in order to
impress the stockholders with his bold vision. Of course, the test
actually tells us nothing about the qualities of paint that determine
consumer preference – like durability, fade resistance, hiding power,
etc. Or which shade your mother would choose for the dining room. And
the marketing dept would never let such decisions be made “blind”,
anyway. The use of evocative names acknowledges that some buyers would
prefer a rich, luminous shade like Pearl Mist rather than a cool
metallic one like Pewter Fog, even if the two shades can’t be
distinguished by ABX testing. And of course, the putative
“Golden-Eyes” like artists or housepainters would do no better than
average folk because their background involves no special training in
remembering subtleties of dried paint chips, even if they splash
around in wet paint every day.
I realize of course, that resorting to “analogies” might rile up some
of the more “scientific” types on the list, but perhaps actually
participating in a simple double blind test can illuminate some of the
grey (or is it gray?) areas in such discussions.
More to come.
Bruce