The Hi-Res Challenge – We Have A Winner!
I’m thrilled that so many audio enthusiasts have expressed an interest in taking the High-Resolution Challenge. At last count, I’ve supplied the FTP credentials to over 300 interested readers. And I’ve started receiving responses. I’m not going to reveal the correct answers or otherwise discuss the results but I thought I would talk about a specific case in some detail because this person managed to get all 6 correct! Brodie was the first person to identify the high-resolution files in every case. So I wrote back and asked that he please confirm that he didn’t analyze the files or peek at their spectra. He assured me that he hadn’t. But I found his descriptions interesting and thought I should share them here. I have been thinking about writing a quick guide on what to listen for when experiencing high-resolution audio. The files downloaded from the FTP site may be your first encounter with real high-resolution audio. Really! The stuff you get from HDtracks, HighResAudio, SuperHiRez, and other download sites typically lack sufficient fidelity to be considered hi-res music.
Here is Brodie’s email (I have removed elements that would identify which tracks are which in the challenge):
“Hi Mark,
Many thanks for the ftp access. I’m enjoying listening to some beautiful recordings tonight. So imagine the scene. A 63 year old man who has listened to far too much loud music in his teens and twenties sitting in a rather harsh listening space. In front the equipment is an ebay purchased $250 HP Elite Book pro taking the sound via HDMI into an Emotiva XMC-1 preamp that is performing the D to A conversion. The analog domain is a pair of Emotiva XPR-5s using two channels to bi-amp a pair of trusty 20 year old Tannoy System 12 DMT Mk II studio monitors some 15 ft away. These are augmented with a pair of Tannoy ST-50 super tweeters. The room has dry-wall ceiling, walls and behind me is a glass wall. Floor is carpet, on the walls at the first reflection points are two native American rugs that cover 6ftx4ftx2in rockwool batting.
Can I hear a difference? Well, everything at my age says that I shouldn’t – especially as I find it very difficult to follow a conversation in a noisy bar – so much so that the bar’s acoustics became more important that the craft beer being served, but I do. All the example tracks are just fabulous, but there’s something about one of each pair of them that makes them stand out to me, so are my pick as the Hi-Res version of each track. Here’s my “guesses” as to which track is the Hi-Res version and why I picked it as such.
Tune 1. the guitar strings have more attack as does the percussion and the cymbal has an edge to it that’s lacking in Version X.
Tune 2. less than 10 seconds into the track and brass section is just so much more alive. The sax has an edge to it that’s missing from X. The high-hat is crisper. Track X sounds muted by comparison.
Tune 3. again it is the grip and smack of the percussion that makes me think this is the Hi-Res version. The rim shots plus the guitar just come over more realistically.
Tune 4. Drums and cymbals just sounds more realistic, more detailed, plus I think there’s a bit more dynamic range in the percussion.
Tune 5. Now this was a tough one there’s almost nothing in the first 15 seconds to choose one over the other. Then there’s the flutist’s breath, the “shewesh” which stands out as different between the two versions. Then I go back to the presentation of the cymbals and the vocals at about 90 seconds into the track – so I’m picking X as the Hi-res version.
Tune 6. Ah the Brandenburg Concerto, a tune I’ve heard before. But who was clomping around the studio? I once heard an audio engineer/speaker designer say the phrase “Smack and Grip”. The system must be able to handle the transient sounds (smack) while at the same time controlling the drivers (grip) so as not to muddy the sound. This track is a real test of both smack and grip. Version X’s presentation of the harpsichord was, to me, a lot cleaner than version X. The initial attack of the violins was however much open and airy.
Mark, you’ll have to let me know how I did. Everything says that at my age there’s no way that I should be able to hear any difference between these tracks. But I did. I hope the descriptions above make some sense. I’ve had a thoroughly enjoyable evening listening to some great recordings. I love the way that AIX recordings are close mic-ed and get the best possible sound captured before the room starts to become too much of an instrument in its own right.
Earlier I used the adjective muted to describe what I was hearing. It’s a bit similar to the difference between wearing glasses or contact lenses. Both help correct my vision, but I see better with contacts rather than glasses. You will need to confirm this, but I think I hear better in Hi-Res too.
Redbook CD specification was and still is rather good especially at covering music recorded onto tape. Everything gets captured. So much music today with auto-tune and techno tracks rather than people who can sing in tune backed by bands that play instruments other than synthesizers frankly don’t need to be recorded at more than CD spec. But there is still an audience for great music played well. I’m still not into downloads – preferring to buy physical media and rip my self. Plus I spend a portion of the year in Wyoming where the internet is still very hit or miss. Our service craps out every day there. Nor am I a part of the Vinyl resurgence. Pops and clicks, static and they warp and wear out – which counts them out for me. Well recorded PCM makes me happy.
Once again, many thanks for the files. I shall audition them with friends and see if I can’t get some buddies hooked on AIX and proper Hi-Res audio too.
Regards,
Brodie.
PS. My only “snake oil” in use tonight was a) each amp is fed by it’s own dedicated 20 amp circuit from the breaker box, oh and the interconnects are Kimber Heros. The bi-amp set-up uses balanced for the high frequencies and regular RCA phonos for the lower frequencies. These feed the two cross-over inputs on the monitors. The HDMI cable is 4K compatible – but who needs 18 Gb/s to get the audio through from source to D-to-A converter.”
So there you have it. Brodie describes hearing distinct differences between the files. I may send him another set of 5-10 tracks to confirm his abilities…we’ll see.
When listening to the files, simply relax and let the music deliver its message. Let yourself become sensitive to increase dynamics — the louder transients will hit harder you harder in the high-resolution versions. If you have supertweeters or headphones that extend into the ultrasonic range, you might feel more clarity, sparkle, or additional crispness in the sound of cymbals and metal instruments. They sound less cluttered and more transparent.
As I was walking along the beach this morning, I thought about adding another variant to the mix. What you’re hearing with the existing files does not represent the realities of the commercial music business. The CD version has far more fidelity than virtually every CD you’ve ever hear. But what if I create a “commercially mastered” version of these tracks. I would do what I used to do for The Allman Brothers, Bad Company, Kiss, and others — smash the dynamic range, hype the high frequencies and boost the bass. Listening to typical CD fidelity vs. a real high-resolution track would be eye opening.
From the description of the differences heard by Brodie, they seem to have one thing in common, they all relate to high frequency content. Unless you believe you can hear beyond 20 khz, the difference could simply be attributable to the high frequency content within the audible passband. This might not be surprising dependent on the differences in the way the signal was processed. If you tried it again where the treble for the non HD recording was slightly boosted, would you get the same results? The possibility has to be eliminated to confirm a genuine qualitative audible difference.
In studying the Fletcher Munson curves there are many things we can learn besides the fact that bass boost is needed at low frequencies to sound balanced at lower listening levels than live. Where the curves are fully extended on the high end, the sensitivity to sound suddenly diminishes very rapidly as the curve for the threshold of hearing crosses the curve for the threshold of pain. Either you can’t hear above 20 khz except to experience it as pain or there is something wrong with this test that has given similar results for many decades.
I don’t think any modification to the files is necessary or desired. The question is simply can humans perceive a difference between an HD original file and a CD res version of the same original? Apparently, Brodie can. Additional study might be able to determine what aspects of the sound he identifies or maybe it’s just an internal feeling. At this stage, it’s really that simple.
The problem I have Mark is that as in so many experiments we can’t be sure that the only differences between the HD and downconverted files are simply a change to the file sizes. Isolating the one variable you want to test for is the difficulty.
I agree…but remember this is just a fun exercise not a rigorous attempt to settle the question.
Okay, I’ll accept that. Sounds reasonable in that context. I’m sure you’d agree that only a rigorous scientifically valid test that is well thought out would demonstrate true audible differences. I’m sure you would agree that setting up and conducting such a test to yield even tentatively conclusive results would require controlled conditions. One thing I think you would be allowed to do is pre-screen the test subjects to select only those with the most acute hearing. Those with lesser hearing capabilities would skew the results towards a false null data set that would be invalid.
Interesting to hear Brodies comments, I think most of us that guessed, because both samples were so good, might want to go back and try again. Taking a Q from Brodies insights. Maybe listen to more than 30 second segments, might help as well. I have a almost all streaming system, except when I listen to my AIX BR’s or other BR’s, so for me it was a stretch to run 3.5mm to RCA cables from my Media center computer to my DAC might have produced some issues. I was only using a DB meter on my iPhone set to averages, to make sure that both tracks were of equal volume, but Mark had already balanced them so well, so that that was pointless. Just like when I go to my good Bud’s house with a $30,000 5.1 B&W system, 90% of the music he listens to is only available in CD quality or flac 16/44 content. I am still trying to get my AIX discs back from him.
Tell him to give you your discs back and have him invest in his own AIX Records releases.
If 300 people took a wild guess at the identity of 6 files, statistically 5 of them would get all correct. Just something to bear in mind.
If Brodie or anyone else is just willy nilly guessing in the HD Challenge, they shouldn’t be participating. I recognize that this is not a rigorous study, but I was still impressed with his eloquent descriptions. I want to send him some further files to see if his percentage holds up.
I’d guess he did hear differences. The real question is exactly what was that attributable to.
I’m not accusing anyone of anything. However, if there actually is no discernible difference, then people must necessarily be guessing, whether they are aware of it or not. This possibility must be considered, or the test is meaningless.
There are also a lot of people that have reported they cannot perceive any difference.
If their high frequency hearing ability is limited to to age, illness, injury, and that is the real difference then you wouldn’t expect them to hear any difference. The question I have is if you match the spectra exactly in the audible range by equalization would those who can hear a difference still be able to hear it?
To attribute the difference to HD alone you’d have to eliminate all other possibilities. Then start with testees who heard the difference up to this point and use them as your filtered test sample. Sorry to be the skeptic but the results of a flawed experiment doesn’t really tell you much. I’m keeping an open mind should you decide to pursue the experiment further. Based on what I think I know is where the source of my skepticism comes from but if you prove me wrong I’ll accept your verdict. I trust your integrity and honesty, something I don’t confer on many people in this industry. I also respect your formal training and knowledge which is the only reason I’m here.
Identifying an individual or several that can reliably detect differences is at least a starting point. Obviously, further testing would have to be done. As to the question of spectra in the audible range…the file supplied do match precisely. The only change was the sample range, which slices off the highest octave in my examples. I’ve done spectral and amplitude analyses and confirmed that the differences are only those that would occur based on the decreased sample rate and word lengths. A next step that might be something to fund via Kickstarter would be to test brain activity using an MRI while listening. My suspicions lead me in that direction. We’ll see.
The odds of coin-tossing your way to 6/6 correct (without even turning on your hifi) are i in 64.
The odds of any one person cheating are roughly 1 in 2.
We have to be realistic. Getting excited about announcing winners in a cheatable test, is a wee bit naive.
Grant, it was meant to be a joke.
I think the concept of the point is the key feature to this exercise. As your data will demonstrate, only a few people can perceive a difference between a original 24/96 file and the down sampled to 16/44 file. If you participated in the test, it was as described a coin flipper, I think the key test is can individuals tell the difference from a lower rez music file, either early digital, or original analogue, up sampled to the so called Hi Rez format. True digital PCM 24/96 recorded music, in a good studio does make a difference. Over the net, that is a hard test to perform, and as Mark describes this is just a fun test and food for thought. The original recording environment does make a huge difference.
Lies, damn lies, and statistics! However, the journey is worth it, and the fact that there is some discussion is good. I really like your idea of the “remastering” of the files. Just a quick review of the files, downloaded earlier today: I did a quick listening review, with my wife interested in “playing”, and of note, her comment was “all of the tracks sound great”. With no more than 1:00 minute of listening per track, I have my guesses, but hers were mostly “B’s” (the last, or most recently, listened-to). I have my initial impressions that seem clear to me, but wish to listen a little more critically (no cheating!). But, to your idea of the “remastering for the masses”: following that quick session, I put on the Yes “Fragile” Blu-Ray from 2015, as an example for my wife. That disc contains the “original mix” as well as a “vinyl transfer” of the album…and this is where I disagree with the criticism of non-High Resolution music being made available in a “high resolution” format: if the “best available” source music (master tapes, or one-off’s when masters not available) has been revisited, and the music remastered for improved dynamic range or soundstage that is possible in hi-res “buckets”, I believe there are evident improvements over the (original) redbook or mastered-for-LP versions. Of course, these are specific cases (and too few!).
Going back to your beach insights, I believe it would be much more evident during this listening test to determine a hi-res/high dynamic range track from a “mastered for the masses” track compared to a well-mastered downsample to redbook, but as you realize, that is a very different comparison.
Thanks Thor!
Mark, did you eventually create a commercial-style mastering for comparison ? Thats a really good idea and IMHO would be most educational.
I have taken one of my best tracks and did a mastering comparison at 5 different levels of compression. It’s part of the files for the Music and Audio book.
Hi Thor,
Absolutely agree that older recordings can benefit and to be honest is mainly what interests me as most of my favourite music is old stuff. Good transfers of old analogue tape done recently in comparison to CD transfers done in the 80’s 90’s certainly seem to contain more frequency when analysed – I’ve been spending time looking at older CD files I have vs recent HD transfers and there is a fair amount of stuff knocked out of the CD’s that I’m far happier knowing is at least captured now in these newer HD transfers. Does it make any real difference? IMHO probably not very much but I enjoy the idea of owning and playing something as close as possible to the ‘original’ master tape.
It’s the labelling of these ‘products’ that causes Mark the consternation I think it’s safe to say, provenance is the keyword, if it’s an old recording transferred then call it a HD transfer, if it’s been recorded in HD (whatever the hell that means – above CD ‘quality’ – at least 48/24 I dunno….) then say so. Simple.
Provenance, Provenance, Provenance!!! (please)
John, you’re starting to sound like me.
Does that mean I qualify for a discount? 🙂
So, was Brodie the only person so far who got all six correct?
Yes I believe so, I have received about 50 responses so far and I haven’t done a strict tally. I casually take a look at the incoming submissions and report back on most. I’ll start tallying this weekend.
It would be a great idea to ask Brodie to listen to more files, but also to ask him to take an ABX test. I know I’m a stickler for that, but as Mans said, statistically we should expect a couple of people to guess it right. Nevertheless, let’s not jump to a conclusion in any way, and let’s see if he can guess more content and pass an ABX test – the ABX test is after all the gold standard. I’d be happy to help Brodie set up an ABX test, so you can give him my contact information if you (and he) like.
I’ll reach out to him about the ABX test. In thinking of next steps, it might be interesting to provide another 10 tracks to those individuals that scored better than 50% and have them use the ABX test option. This first round could be considered a qualifying phase.
Yes, good idea :-). I’m here to help with the ABX test if need be, although it’s pretty straightforward once you install Foobar and the separate ABX plugin.
Although I still think an ABX test would be the best option, then here’s another suggestion: Send him the same files again, but mix them up, so some will be the same order and some will be different. Then let’s see if he gets them all correct again.
I will reach out to Brodie and see what he’s up for.
Hi Mark,
I took the “test,” turned in my results, then read Brodie’s email you posted. I also took notes regarding my Tune choices.
I think that revealing Brodie’s notes to listeners who haven’t turned in their answers gives an unfair edge because now they have only to identify the specifics of those notes to help pick the correct answers. I encountered some of the same distinctions in some of the Tunes. If I had read his notes and compared them to mine before turning in my answers, that definitely would have influenced my answers.
I realize this is primarily a fun exercise; however, accurate results, I presume, is the intended outcome.
Picky, picky, picky?
I think Brodie’s notes are a guide to what to listen for — or at least what he found worked for him. Training is a strong component in this type of listening. I remember many years ago going through the Phillips “Golden Ear Training” program. It was hard even for a seasoned mastering engineer. I got to the point where I could tell which frequency was altered by as little as a half a dB.
This was very interesting. My partner and I (65 and 63 year old ears) sat together and listened. We each correctly identified 5 out of 6 hi res files. The two we had difficulty with were 4 and 5, and we were reversed in our incorrect choices. We read Brodie’s comments after we listened. FYI … Outlaw preamp and amplifier; Magneplanar 3.7 speakers.
P.S. I’m skeptical that anyone can in any way perceive frequencies above 20KHz but I do believe we’re hearing the difference between 16 and 24 bits in dynamic range manifesting itself mostly in the higher frequencies.
It’s not the “perception” of frequencies in the ultrasonic range that count. I believe there are technical benefits to extending the frequency range to 48 kHz.
I’ve always thought that this type of test should include one set of two identical files, both cd or hires. I’d love to hear the justifications.
Interestingly, there have been a significant number of participants the respond by saying they cannot tell them apart.
I came from computeraudiophile very excited upon hearing about a blind test, only to find out that the creator isnt going to discuss the results instead analizying a single result. Why? Why in the name of the Holy Sound?
The results of that charming old man may be interesting but what about the global stats? Should we assume, since the author doesn want to discuss them, that statistically differences between to highquality formats were non audible?
Manuel, it was meant as a joke. There are no “winners” in this debate and certainly no winners with regards to picking high-resolution audio in this casual survey. I have been responding to individual requests for the answers. However, there are still many individuals that want to take the challenge and I felt waiting to reveal and discuss the answers could wait. I look forward to discussing the results and have no expectations either way.
Interesting that a 63 yr old gent has augmented his speakers with supertweeters with a minimum 3rd order high pass at 14Khz http://cdn-docs.av-iq.com/other//ST%2050_User%20Manual.pdf
I wonder if this indicates that he at least believes he can hear >14k at 63?
Would Brodie be willing to take a proctored ABX with the same tracks?
I have been in touch with Brodie and asked him to participate in a follow up set of tests using the A|B|X methodology. Who knows why he managed to nail the test.
If Brody was 1 in about 50 people who got all 6 fragments right, he did not ‘nail’ the test, but simply got lucky.
Out of every 64 people who participate in the test, 1 is most likely to have all six answers ‘correct’ – even if all participants are totally deaf and respond by arbitrarily pushing ‘A’ or ‘B’.
If you have 500 totally deaf people participating in the test, the chance of ZERO participants having all of the answers ‘correct’ is close to ZERO.
You’re certainly correct that some number of people could randomly select all 6 high-res versions. Brody (and a few others) successfully submitted results that most others did not. I’ve never claimed that this test is authoritative but I do know that it uses bona fide high-resolution content.
Hi Mark! When will be the results and correct answers released?
Sorry, I’ll get them up asap.
Interesting test, thanks for sharing. A couple things jump out immediately as pretty big red flags though.
1) This phrase is a little suspect: “I wrote back and asked that he please confirm that he didn’t analyze the files or peek at their spectra. He assured me that he hadn’t.”
So, we have a blind listening test without proper controls to assure that it is actually blind? That seems like a problem, right? 🙂 So, not exactly a scientific test, or one with even basic controls in place to assure it was done fairly. So we can’t read too much into this. But that’s not all. There’s also:
2) Flipping heads on a coin 6 times in a row will happen 1.5% of the time, or 1 out of 64 times.
Apparently this test has been attempted over 300 times? So this should actually happen more often than that! 🙂
Please let me know if there’s anything I’m missing.
Hope that makes sense,
Justin
Thanks for the comments Justin. The HD-Audio Challenge is not intended to be a rigorous scientific research project. My interest was to find out if audiophiles with varying levels of experience, equipment, and age groups could reliably identify a bona fide high-resolution audio file over a downconversion of the same file at CD-resolution. It’s true that I have to trust that people will not cheat. I ask that everyone participating agree to the rules and I trust that virtually everyone did so. Out of 8000 responses, there were very clear trends…the odds of selecting the HD version was no better than random choice. The gentleman that helped with the statistical analysis saw some variation in the results for some of the tracks but agreed that the null hypothesis was confirmed.
Dear sir, I am late to the party as usual. Today I happened upon your website for the first time, however I’d like to give your curated files a try. I just recently upgraded my headphones and now have a proper DAC. Are the tracks still available to latecomers?