Re: [ARSCLIST] A/B testing: another approach

[Malcolm Davidson <malcolm_davidson@xxxxxxxxxxxxxxxxxx>]

For many people the brain's ability to perceive subtle difference is
severely limited when the samples are played sequentialy.  It's difficult to
remember the "reference" track and the one being listened to in the moment.
For large difference it is much easier so that a sequential test does have
some validity comparing say 96 KHz 24 bit, CD and MP3.

When we did all the SDMI (Secure Digital Music Initiative)  watermark
evaluations, not only were we evaluating the watermarking technologies, but
we were also (unintenionally) evaluating the abilities of the participants.
This varied widely amongst all the so called experts.  With a certain amount
of training people can improve their listening skills.  We observed
individuals who were perceived as "golden ears" who could not pick out a
watermark, whereas there were some participants who could pick out certain
watermarks with ease, yet did not have a reputation as an expert listener.

Any type of comparative listening test is highly subjective for the brain
can pick out subtelties that we are not skilled at measuring. For example
bit for bit identical streams have been consistantly thought of as different
by expert listeners,  (separate files on a hard disk)   due in part to the
buffering and speed matching of the data as it is read off the disk.  It
changes the small amount of jitter of the digital stream which subsequently
alters the noise floor of the D/A.  This has influences the spacial imaging
slightly of stereo.  However many people might describe it differently and
it might not necessariy be a bad thing.  With the complexity of the "supply
chain", to the end user,  how close does the final product replicate the
actual recording experience and do people care and are they willing to pay
for it?

Malcolm Davidson

----- Original Message ----- 
From: "Matthew Barton" <mbarton@xxxxxxx>
To: <ARSCLIST@xxxxxxxxxxxx>
Sent: Monday, January 28, 2008 11:30 AM
Subject: Re: [ARSCLIST] A/B testing: another approach


> It wasn't a scientific experiment--just an engineer having a bit of
> fun--though he seems to have had a point to make, if not a detailedl
> hypothesis to test. I just thought the structure was interesting, and
> worth considering if our goal is to develop a listening test or tests.
> Perhaps the thing that I find most interesting here is that it involved
> a real-time, uninterrupted  listening experience of A, B, C, and D.
> Perhaps the brain does respond differently to such a listening
> experience.
>
> Matthew Barton
> MBRS
> The Library of Congress
> 101 Independence Ave., SE
> Washington, DC 20540-4696
> 202-707-5508
> email: mbarton@xxxxxxx
>
> >>> Marcos Sueiro Bal <mls2137@xxxxxxxxxxxx> 1/28/2008 10:40:58 AM >>>
> Matt,
>
> This is an interesting link, but as a scientific experiment it does not
>
> seem very useful: What is the hypothesis? How are we quantifying it? A
>
> statement such as "my fellow listeners appeared to be equally
> uncomfortable" does not seem conducive to analysis.
>
> If the highs were perceived not to be "as silky smooth" --in other
> words, if the differentiating factor has been identified after just one
>
> listen of a short passage--, should not the same listener be able to
> correctly identify such a difference in a blind test? Logic seems to
> indicate that he should, but perhaps the brain works in mysterious
> ways.
>
> Incidentally, it seems that not all ABX tests have concluded that
> listeners are less sensitive than we thought. I was told in school that
>
> most average Joes can hear at most a difference of 1 dB, but a group of
>
> 5 listeners in an ABX test perceived differences of 0.4 dB 93% of the
> time (note: this is not a peer-reviewed paper, and this is from the ABX
>
> web page, so it is not conclusive evidence).
>
> http://www.provide.net/~djcarlst/abx_lvl.htm
>
> Cheers,
>
> marcos
>
> Matthew Barton wrote:
> > Here's a link to an article from the October issue of Stereophile,
> in
> > which an interesting approach to blindfold testing is described:
> >
> > http://www.stereophile.com/asweseeit/1007awsi/index.html
> >
> > This is not an analog vs. digital article, and I'm not endorsing the
> > test or or its results, or any conclusions in the article, but I
> think
> > the approach is interesting. Instead of an A:B comparision, in which
> > listeners first heard A, and then B, and were asked for opinions,
> this
> > engineer created a composite patchwork of different formats using a
> > repetitive passage from a recent recording of Handel's Messiah. He
> > didn't tell his audience that this is what they would be hearing:
> >
> > "It turned out that we'd been unwittingly involved in a blind
> listening
> > test. The DVD-A was a ringer. Philip had chosen a Handel chorus in
> which
> > the same music is heard four times. He had prepared four versions of
> the
> > chorus—the original 24-bit/88.2kHz data transcoded straight from
> the
> > DSD master; a version sample-rate–converted and decimated to
> 16/44.1
> > CD data; an MP3 version at 320kbps; and, finally, an MP3 version at
> > 192kbps—and spliced them together in that order. The last three
> > versions had been subsequently upsampled back to 24/88.2 so that the
> > DAC's performance would not be a variable. The peak and average
> levels
> > were the same for all four versions; the only difference we would
> hear
> > would be the reductions in bandwidth and resolution. "-- from
> "Watching
> > the Detectives," by John Atkinson, Stereophile, October, 2007.
> >
> > We can all argue about the specs here, but the most interesting
> thing
> > to me is that the changes in the audio unfolded over four iterations
> of
> > the same passage of music in the same recording. Listeners were not
> > asked to use their memory of recording A to appraise recording B,or
> vice
> > versa. They heard (or did not hear) the changes as part of
> continuous
> > listening experience.
> >
> >
> > Matthew Barton
> > MBRS
> > The Library of Congress
> > 101 Independence Ave., SE
> > Washington, DC 20540-4696
> > 202-707-5508
> > email: mbarton@xxxxxxx
> >
>
>