Hi-Res Music Challenge Again: Doing It Right!
It’s been over a month since my last post. Time flies when there’s travel (Charlie and I had a road trip to the mountains of Colorado for some late season skiing), a weekend in Big Sky with friends, and a long weekend in manning my table for the annual AXPONA show in Chicago (more about that in a future post). The big news is that I’ve been granted a sabbatical for the fall semester from CSU Dominguez Hills to conduct a research project into the audibility of high-resolution music.
Here’s a portion of what I submitted to the University research committee:
Statement of Purpose and Description of the Proposed Project
a. Goals of the project – The goal of the proposed research project is to test whether music consumers can reliably identify a Hi-Res Music selection versus a downconverted of the same file at CD or standard resolution using real high-resolution sources, rigorous testing methodologies, and statistical analysis. This issue is a major point of contention among audio consumers and despite previous studies remains unsettled.
Since Edison’s invention of audio recording in the late 19th century, it has generally been assumed that recording fidelity has steadily increased. Music consumers have been offered and purchased music on a wide variety of physical formats including crude, low fidelity cylinders, monophonic lacquer 78 rpm discs, stereo vinyl 33 1/3 rpm LPs, analog tapes, 8-track cartridges, compact cassettes, compact discs (CDs), and recently pure audio Blu-ray discs. The arrival of digital audio file formats ushered in the era of CDs, music “ripping”, and MP3 playback. Recently, Digital Music Retailers (DMR) like iTunes and Amazon Music that provided file downloads have given market share to music streaming services including Spotify, Apple Music, and Tidal.
One of the latest initiatives being promoted by NARAS (The Recording Academy®), the CTA (Consumer Technology Association), the DEG (Digital Entertainment Group), major consumer electronics companies, and the major music labels is called Hi-Res Audio/Music or High-Resolution Audio/Music. This new digital format is the latest new format to promise additional fidelity for consumers. But does it? Does merely doubling or quadrupling the sample rate of a digital sound file and adding 8-bits to each sample deliver perceptible improvements in fidelity?
b. Preliminary Work – I have been actively involved in the production, promotion, and discussion of high-resolution audio since its introduction in 2000. My label, AIX Records, has recorded and released almost 100 high-resolution, award-winning music albums in a variety of physical and downloadable formats. I am a recognized expert in the area of high-resolution recording. I have served on boards (CEA High End Audio) and technical working groups (AES), given papers and keynote address at prestigious conferences, (CES, Las Vegas, AES Latin America, Bogota, Columbia, AES London, InterBEE Tokyo, AXPONA Chicago), written over 1000 articles/blog posts on the topic (www.realHD-Audio.Com), and consulted on a previous study for the Consumer Technology Association.
On two separate occasions, I have collaborated with online publications in casual high-res audio studies (AVS Forum and the “HD Audio Challenge”). I have critiqued and reviewed previous studies in the area and found serious flaws in their methodology and results. It’s past time to do a proper study.
c. Specific Activities – There will be multiple phases required to complete the proposed study. They are: 1. Identify, select, evaluate, and prepare up to 20 music examples for use in the listening phase of the research, 2. Identify, qualify, and engage with up to 1000 participants across a wide demographic, 3. Distribute and collect the survey materials among participants, 4. Analyze results, and 5. Present statistical breakdown of findings. These activities can be accomplished within the 15 weeks of the leave requested.
d. Specific Benchmarks – The proposed research will contribute to — and possibly resolve — the ongoing debate regarding the value proposition of “hi-res audio”. The well-known Audio Engineering Society paper by E. Brad Meyer and David Moran of the Boston Audio Society entitled: Audibility of a CD-Standard A/DA/A Loop Inserted into High-Resolution Audio Playback conducted in 2007 was shown to be seriously flawed by myself and others because it didn’t use real high-resolution sources.
The proposed study will be submitted to for publication in the AES Journal, other publications (print and online), and otherwise widely disseminated among the professional audio community, audiophiles, and consumers.
Specific benchmarks will include: announcements of the study on major online audio sites, development of approved survey methodology and test materials, acceptance of test subjects, and presentation of outcomes.
e. Summary of Project – This proposal describes a study into the perceptibility of Hi-Res Audio recordings vs. downconverted versions at CD specifications. A number of music examples will be prepared in both Hi-Res and CD fidelity, distributed, and evaluated by a pool of qualified test subjects. Their feedback will be analyzed and published.
Initial Steps – Announce the Study and Seek Input
Unfortunately, there is no funding associated with the granting of a sabbatical. There will undoubtedly be costs associated with preparing, executing, and reporting this study. I am considering starting a crowdsourcing campaign to offset some of these costs. However, I don’t think it’s going to be expensive primarily because I am not planning on hosting playback sessions at my studio or other facility — although I am not opposed to doing so if the consensus is that will provide for a better result.
My intention is to improve and repeat the HD-Audio Challenge that I created on this site some many months ago. Over 4000 people read the blog, almost 250 downloaded the test files, and just over 100 responded with their submissions. I provided 6 different selections of music in both the original 96/24 PCM format and downconverted to 44.1/16 CD standard. Participants were encouraged to play the files on any system (speakers and/or headphones) and then let me know which one of the pair was high-resolution. The results seemed to indicate that listeners did no better than random guessing — although I did not do a rigorous statistical analysis. I will engage with statisticians for the proposed study (if you’re one of those experts, please feel free to contact me).
I’m also very open to fielding any suggestions or comments that could improve this project. There will be complaints and objections to whatever I choose to do but I’m open to modifying my plans if warranted and beneficial. All of the content will come from my own AIX Records catalog because I know the catalog exceeds the fidelity of standard CDs — at least by software analysis. There will be a wide variety of music genres, ensembles, and instrumental/vocal selections represented among the roughly 20 tracks. If you know of specific tracks from my catalog that you believe show perceptible improvement over standard CDs, again please let me know.
This should be fun. I hope that this study will contribute to our knowledge of hi-res music. I believe it can.
Dear Sir,
One suggestion regarding your new Hi-Res Challenge.
Downloads go to my computer. Then, getting that music to my Hi-Fi system goes through different pieces of hardware. Doing that, I always get the feeling that the audio quality in my listening room is not as good as it could be. However, when I get a Blu-ray disc from you, and I play it on my Oppo UDP-205 (which is connected into other high quality components), I believe I get the ultimate experience.
Therefore, would it be possible for you to burn a Blu-ray disc for your new challenge ?
It’s doubtful that I’ll be able to deliver Blu-rays to participants…although maybe that something I should consider funding via the crowd sourcing campaign if enough people think it would be valuable. You can use a USB stick as well.
Hi Mark,
Great initiative but I’m just a little worried that if you’re going to get unknown people do the testing the way your previous test was done, you could leave yourself open to fraud. If the high resolution crowd are happy to lie to consumers the way they have, what’s to stop them from setting up a bunch of email addresses, downloading your files, revealing the format with software and then telling you they ‘heard the difference’?
If you want to do this so nobody can dispute the outcome it has to be done in a more controlled environment. Otherwise you’ve already lost the war before your first battle.
Good luck!
Regards,
Pim
I can certainly appreciate the risk of having people do this on the honor system (I will require a promise from all participants that they didn’t cheat of falsify their responses) but I don’t see getting 1000 people to come to a specific location for a fixed length of time.
Hi Mark, this is a great idea and I have been waiting for ages for someone to do exactly what you are proposing. Archimago makes a good point, and I also think that doing this study with remote participants is an issue. You don’t need a 1000 participants to get statistically significant results, it would be better to get a 100 in a controlled environment. The other point I’d like to make is about the stimuli, which are always overlooked in studies I have seen, organised by audiophiles or engineers society. Using music that you’ve recorded is great of course, using short samples is important as well, to avoid the recency effect (I think it’s 90 seconds max, my knowledge of experiment design is limited) and using AXB system as well instead of ABX. If you could work with a music psychologist that would add a lot of weight to your proposal. Well, you probably knew all that, but all the best for this project, I really hope it materialises.
Thanks Pierre. The next phase of the study will definitely happen…it is part of the research proposal for which I received a sabbatical. I don’t agree that a controlled environment is a requirement unless one wishes to establish a highly artificial case. I’m interested in whether the promoted improvements of high-resolution audio are perceptible by the average or even high-end listener in their own systems. I like Archimago’s suggestion that I break apart the sample rate and word length parameters.
Hello Paul,
Brilliant, about bloody time!
My only comments to add to your proposal would be to include the (dreaded to you I know) DSD digital format in the proceedings and to also try to assess in some way the ‘long term’ effects of standard vs high definition music. From our own communications we’ve discussed this – even if there is no obvious immediate recognisable ability for people to tell the difference is there still any kind of preference toward higher definition audio?
Thanks John but DSD is a disaster format and won’t be part of the study for all of the reasons that I outlined in my book. Too much ultrasonic noise and other compromises. And I don’t have any DSD files.
Wouldn’t it be better to include some high quality made lossy/320 files?
This is a bit more real world than having DSD files.
Very glad to see you doing this Mark! It’s amazing that a proper study has not yet been done yet, and that so many have been flawed. In particular I have been dissatisfied with the studies that ask participants their “preference” (between resolutions or formats, PCM vs DSD, etc) without there being a definitive study establishing that people can reliably identify a difference between samples. My partner and I did your informal test several months ago and we both identified 5 out of 6 of the high res files. But I wonder if we could do it reliably 10 times.
If your study accommodates participants working remotely I would be happy to volunteer.
Thanks Marc…everyone is welcome.
I was going to say more or less the same thing. Too many such studies are framed in terms of preference or vague qualities like “clarity” when it is yet to be determined whether a difference is consistently detectable at all.
I can make a prediction for your project: if you, as usual, take all the precautions form the beginning (recording), editing, and mastering then the resulting CD quality tracks (16 bit stereo 44Khz sampling rate) will be totally indistinguisable from any “hi-res” format at higher bits per sample or sampling freq; I know you do it the other way: capturing, editing and mastering from, say 24 bits 96Khz pcm, but I’ve hear a lot of well-mastered SACDs to tell you that the original CD spec (back in the 80s) is more than capable of delivering “hi-res” sound to anyone interested in fidelity; of course, you then need the proper playback chain: DAC, amp and headphones/speakers but the prediction still stands: anything from your record label (or Chesky Records for example) will be “hi-res” even in CD format
Well, by definition anything on CDs is standard-resolution. The object in this study is to help determine if it matters to distributing audio at higher sample rates and longer words.
Hi Mark,
This looks very interesting. To me, the most challenging music to capture is a complex classical symphony. If there are advantages of hi-res recordings, symphonic music is where I would expect to most likely pick up the difference. The dynamic range contrasts would be easier to spot, as well as reproducing the various instrument subtilties.
Do you plan to make all the files at the same bit length/sampling rate, and let the testers identify which files were sourced from the CD or Hi Res?
One other thought: I was thinking of conducting the test normally through speakers, and then going through the test a second time with a dedicated headphone setup, and see what (if any differences) can be picked up via using headphones.
I think the USB approach for playback makes the most sense for those of us Oppo owners.
This great news!
My 2 cents and comments.
* if the sound was the same I would still want hi-res files as filters are easier to design in the DAC and if cost was close to red book.
* why not have AV shops host the test to get customers in their stores?
* maybe ask what sounds different and what sounds better? My theory is people may like”distortion” for lack of better term as some like grain in film and not digital cameras.
* maybe stream the files and not download them and have a companion web interface interact and keep score?
Good luck!
Hi Mark
This project will be more than interesting!
However I have a question.
Quote ” This proposal describes a study into the perceptibility of Hi-Res Audio recordings vs. downconverted versions at CD specifications”, which is your objective.
Would it not be as important to study the opposite perception…..
CD specification files upconverted to claimed “Hi-Res Audio” files. That’s what I usually download since I’m downloading older songs all recorded at CD specs, but upconverted to say 96 from sites such as HD Tracks And now Qobuz are offering a download package.
Wondered if that approach would also be part of your project?
Howard
Hi Mark,
As per the comment by Pim above, maintaining the blind condition with a 16/44.1 vs. 24/96 test is a major issue.
To be honest, this is why I’ve never tried it on my blog but did try a simpler test with 16 vs. 24-bit instead; purposely adding some data in the lowest bits so that typical software like Adobe Audition would not easily detect the 16-bit samples. I also purposely made it harder for test subjects to run null tests by adding some variation in fade in and such:
http://archimago.blogspot.com/2014/06/24-bit-vs-16-bit-audio-test-part-i.html
All-in-all, I think the blind was maintained based on monitoring the threads and forums back in 2014.
I might have missed it if brought up earlier, but also remember that there has been a meta-analysis published by Reiss on this back in 2016 – “A Meta-Analysis of High Resolution Audio Perceptual Evaluation”:
https://pdfs.semanticscholar.org/02ea/42eaeb7639c887f66222765e908116605a87.pdf
Lots of discussions there and I think the conclusions are less than conclusive :-). I particularly found the studies where “Interestingly, several studies reported results suggesting that for some trials, participants had an uncanny ability to discriminate far worse than guessing (Oohashi 1991, Meyer 2007, Woszcyk 2007,Pras 2010)” fascinating. Hmmmm… Does that mean that some people were somehow averse to the hi-res sound, and appeared to choose the non-hi-res to a significant degree!? We have to be open to that possibility…
All the best!
Mark
“All of the content will come from my own AIX Records catalog because I know the catalog exceeds the fidelity of standard CDs — at least by software analysis.”
Why can’t you also use a well known 24/192 recording from either Qobuz or HDTRACKS ? (Or even 2L)
I’m not getting a reason why a down sampling versions of such a recording won’t do ?
They might do, but by using his own catalogue Mark is 100 % sure that everything is okay, as he was the one doing the recordings 🙂
I would be happy to participate if I can, and I have a decent amount of experience with blind testing – many passed and many failed.
Here’s my suggestion for how the entire test should be done:
* Everyone participating should do an ABX test, preferably with 16 trials, or at the very least 12 trials.
* The people who can reliably tell the two formats apart, should be tested again, preferably a couple of times to play it safe.
This way we would eliminate flukes and such. The obsessive audiophiles also always claim that only certain people can hear the differences, so if certain people, like those particular audiophiles, can reliably tell hi-res apart from standard resolution, testing them again would either confirm their claims or disspell them (and weed out cheating).
I’m wondering if there is a way to cheat an ABX test, but this is something I will look into.
Mark,
This is fantastic that someone is finally doing this. One suggestion is to set up a proctored test that you could give to interested students, recording industry folks, musicians,and audiophiles. If you could do ten percent of the samples this way, you could assess the accuracy of the 90% of samples that are done on the honor basis. It would also be nice to keep track of the different groups I mentioned above to see if they perform differently on the test.