A codec listening test is a scientific study designed to compare two or more lossy audio codecs, usually with respect to perceived fidelity and/or compression efficiency.

Most tests take the form of a double-blind comparison. Commonly used methods are known as "ABX" or "ABC/HR" or "MUSHRA". There are various software packages available for individuals to perform this type of testing themselves with minimal assistance.

In an ABX test, the listener has to identify an unknown sample X as being A or B, with A (usually the original) and B (usually the encoded version) available for reference. The outcome of a test must be statistically significant. This setup ensures that the listener is not biased by his/her expectations, and that the outcome is not likely to be the result of chance. If sample X cannot be determined reliably with a low p-value in a predetermined number of trials, then the null hypothesis cannot be rejected and it cannot be proved that there is a perceptible difference between samples A and B. This usually indicates that the encoded version will actually be transparent to the listener.

In an ABC/HR test, C is the original which is always available for reference. A and B are the original and the encoded version in randomized order. The listener must first distinguish the encoded version from the original (which is the Hidden Reference that the "HR" in ABC/HR stands for), prior to assigning a score as a subjective judgment of the quality. Different encoded versions can be compared against each other using these scores.

Results

Many double-blind music listening tests have been carried out. The following table lists the results of several listening tests that have been published online. To obtain meaningful results, listening tests must compare codecs' performance at similar or identical bitrates, since the audio quality produced by any lossy encoder will be trivially improved by increasing the bitrate. If listeners cannot consistently distinguish a lossy encoder's output from the uncompressed original audio, then it may be concluded that the codec has achieved transparency (data compression).

Popular codecs compared in these tests include MP3, AAC (and extensions), Vorbis, Musepack, and WMA. The RealAudio Gecko, ATRAC3, QDesign, and MP3pro codecs appear in some tests, despite much lower adoption as of 2007. Many encoder and decoder implementations (both proprietary and open source) exist for some codecs, such as MP3, which is the oldest and best-known codec still in widespread use today.

The Musepack and Vorbis codecs are the product of open source projects, and were designed to avoid patented algorithms; both have competed well with patented, proprietary codecs such as MP3, AAC, and WMA according to many of these listening tests.

valign = "top"> valign = "top"> valign = "top"> valign = "top"> valign = "top"> valign = "top"> valign = "top">

Source	Dates	Codecs	Bitrate (kbit/s)	Implementations	Musical genres	Samples	Listeners	Winner
ff123	2001	multiple	~128	MP3: Lame 3.89beta --abr 134 -h --nspsytune --athtype 2 --lowpass 16 --ns-bass -8 MP3: Xing within AudioCatalyst 2.1 128 kbit/s, high frequency mode disabled, simple stereo disabled AAC: Liquifier Pro 5.0.0 Beta 2, Build 24 streaming 128, equalization disabled, dynamics disabled, dual mono encoding disabled, audio bandwidth overridden by the program, set at 17995 Hz MPC: mppenc.exe version 1.7.9c -radio -ltq_gain 10 -tmn 12 -nmt 4.8 WMA8: Windows Media Player 7.1 (version 7.01.00.3055); wmadmoe.dll version 8.0.0.0371 128 kbit/s Vorbis: Oggdrop RC2 for Windows 32 128 kbit/s		1	16	Musepack and AAC
ff123	October 2001-January 2002	multiple	~128	MP3: Lame 3.89beta --abr 134 -h --nspsytune --athtype 2 --ns-bass -8 MP3: Xing within AudioCatalyst 2.1 128 kbit/s, high frequency mode disabled, simple stereo disabled AAC: Liquifier Pro 5.0.0 Beta 2, Build 24 streaming 128, equalization disabled, dynamics disabled, dual mono encoding disabled, audio bandwidth overridden by the program, set at 17995 Hz MPC: *mppenc.exe version 1.7.9c -radio -ltq_gain 10 -tmn 12 -nmt 4.8 WMA8: Windows Media Player 7.1 (version 7.01.00.3055); wmadmoe.dll version 8.0.0.0371 128 kbit/s Vorbis: **Oggdrop pre-RC3 for Windows 32; from CVS (10/26/01) 128 kbit/s	Various	3	25-28	Musepack or Vorbis
ff123	July 2002	multiple	~64	Ogg Vorbis 1.0 -b 64 --managed Ogg Vorbis 1.0 -q 0 MMJB 7.2 mp3pro 64 WMA8 at 64 kbit/s (using WMP 7.1 to encode) Quicktime 6.0 MPEG-4 AAC Low complexity at 64 kbit/s	Various	12	24-41	MP3pro
Roberto Amorim	June 2003	AAC	128 CBR	Psytel AACenc 2.15 -br 128 Ahead/Nero 5.5.10.35 128 kbit/s CBR, high quality Sorenson Squeeze 3.5 (FhG Pro) 128 kbit/s Apple QuickTime 6.3 (Apple/Dolby) 128 kbit/s high quality FAAC 1.17b -a 64 (64 kbit/s/channel, ABR)	Various	10	11-18	QuickTime
Roberto Amorim	July 2003	multiple	~128	Apple QuickTime 6.3 MP4 encoder 128 kbit/s high quality LAME MP3 Encoder 3.90.3 --alt-preset 128 Musepack 1.14 --quality 4 --xlevel Ogg Vorbis post-1.0 CVS -q 4.25 Windows Media Audio v9 PRO bitrate-managed 2-pass VBR 128 kbit/s	Various	12	14-24	Musepack	WMA, and Vorbis tied for close second
Roberto Amorim	September 2003	multiple	~64	Ahead/Nero 6.0.0.15 HE AAC VBR profile Streaming :: Medium, high quality Ogg Vorbis post-1.0 CVS -q 0 MP3pro (from Adobe Audition 1.0) VBR quality 40, Current Codec, allow M/S and IS, allow narrowing, no CRC Real Audio Gecko (from Real Producer 9.0.1) 64 kbit/s Windows Media Audio v9 VBR quality 50 QuickTime 6.3 AAC LC 64 kbit/s, Best Quality	Various	12	30-43	Nero HE AAC	This test showed that listeners preferred 128 kbit/s MP3 audio encoded by LAME to all the tested codecs at 64 kbit/s, with greater than 99% confidence: "No codec delivers the marketing plot [sic] of same quality as MP3 at half the bitrates."
Roberto Amorim	January 2004	MP3	~128	LAME encoder 3.95 --preset 128 FhG MP3 encoder from Adobe Audition 1.0 VBR quality 40, "Current - Best" codec. Apple iTunes 4.2 MP3 112 kbit/s VBR, Highest quality, joint stereo, smart encoding Gogo-no-coda 3.12 -b 128 -a -q 0 Audioactive Encoder 2.04 128 kbit/s High Quality Xing MP3 Encoder 1.5 VBR quality normal	Various	12	11-22	LAME
Roberto Amorim	February 2004	AAC	~128	Ahead/Nero AACenc v 2.6.2.0 -internet profile, high quality, LC Apple iTunes 4.2 (Apple/QuickTime) 128 kbit/s Compaact! 1.2beta3 (zPlane/HHI) VBR 5, high quality, LC FAAC 1.23.5 -q 115 Real Producer 10 beta (CodingTechnologies) 128 kbit/s	Various	12	19-29	iTunes	FAAC codec improved greatly since previous test
Roberto Amorim	May 2004	multiple	~128	LAME encoder 3.96 -V5 --athaa-sensitivity 1 Apple iTunes 4.2 128 kbit/s AAC Ogg Vorbis aoTuV tuning b2 -q 4.35 Musepack 1.14b --quality 4.15 --xlevel Sony Atrac3 132 kbit/s Microsoft WMA9 Std Bitrate VBR 128 kbit/s	Various	18	12-27	aoTuV and Musepack
Roberto Amorim	June 2004	multiple	32 CBR	LAME encoder 3.96 -b 32 Nero Ahead HE AAC+PS 32 kbit/s CBR High Quality Ogg Vorbis post-1.0.1CVS --managed -b 32 resampled with SSRC Real Audio 32 kbit/s stereo music codec in Helix Producer 10 QDesign Music Codec 2 Pro 32 kbit/s at 32kHz, Quality mode Microsoft WMA9 Std 32 kbit/s at 32kHz MP3pro 32 kbit/s at 32kHz, in Adobe Audition 1.5	Various	18	47-77	Nero Digital
HydrogenAudio user "guruboolez"	July 2004	multiple	~175	MPC: musepack -standard MP3: LAME 3.97 alpha -V 3; -V 2 Vorbis: megamix -q 6,00; -q 6,99; -q 5,50	Classical	18	1	Musepack
HydrogenAudio user "guruboolez"	August 2005	multiple	~180	AAC: faac 1.24.1. Release date: end 2004 (?). Setting: -q175 AAC: Nero Digital aacenc32 v.3.2.0.15. Release date: June 2005. Setting: -streaming (high/default encoder). MP3: LAME 3.97 alpha 11. Release date: July 2005. Setting: -V2 --vbr-new MPC: mppenc 1.15v. Release date: march 2005. Setting: --quality 5 Vorbis: aoTuV beta 4 based on 1.1.1. Release date: July 2005. Setting: -q6,00	Classical	18	1	aoTuV	The author reflects on substantial improvements in Vorbis encoding since his previous test (above): "Vorbis is now –thanks to Aoyumi [creator of aoTuV]– an excellent audio format for 180 kbit/s encodings (and classical music)."
gURuBoOleZ​Z	August 2005	MP3	~96	AAC-LC: iTunes 4.9 / QuickTime 7.02 CBR 96 MP3: LAME 3.97 alpha 11 --abr 99 MPC: mppenc 1.14 --xlevel --quality 3 (or --thumb) Ogg Vorbis: aoTuV / LANCER beta 4 based on SVN 1.1.1 -q2,00 WMA Standard: WMA 9.1 CBR 96	Classic, various	150 classical, 35 various	1	aoTuV and AAC tied (classical), aoTuV (various)
Sebastian Mares	December 2005	multiple	~140 (nominal 128)	Nero AAC 3.1.0.2 VBR/Stereo - Streaming, 100-120 kbit/s [LC AAC] iTunes AAC 6.0.1.3 128 kbit/s, VBR LAME 3.97 Beta 2 -V5 --vbr-new Ogg Vorbis AoTuV 4.51 Beta -q 4.25 WMA Professional 9.1 Quality-Based VBR, Q50 Shine 0.1.4 (Low Anchor) -b 128	Various	18	18-30	4-way tie (all except Shine)	"I think this test shows that with the current encoders, the quality at 128 kbit/s is very good... It's time to move to bitrates like 96 kbit/s or even lower (64 kbit/s)."
Mp3-tech.org	March 2006	AAC	48	3gpp 6.3.0 48kbit/s CBR Coding Technologies - Winamp 5.2 beta 393 48kbit/s CBR HE-AAC Coding Technologies - Winamp 5.2 beta 393 48kbit/s CBR HEv2-AAC Nero Digital 4.9.9.95 48kbit/s ABR HE-AAC Nero Digital 4.9.9.96 48kbit/s ABR HEv2-AAC iTunes 6.0.2 (Low Anchor) 48kbit/s CBR LAME 3.97b2 (High Anchor) -V5	Various	18	10-20
Sebastian Mares	November 2006	multiple	~48	Ogg Vorbis AoTuV 5 Beta -q -1 WMA Professional 10 1-pass CBR, 48 kbit/s Nero HE-AAC May 26 2006 -q 0.2 WMA Standard 9.2 Quality-Based VBR, Q10 iTunes AAC 7.0.2.16 48 kbit/s, CBR	Various	20	22-34	Nero Digital	aoTuV tied for second
Sebastian Mares	July 2007	multiple	~64	Ogg Vorbis AoTuV 5 Beta -q 0 WMA Professional 10 1-pass CBR, 64 kbit/s Nero HE-AAC Jul 20 2007 -q 0.24	Various	18	21-33	Nero Digital and WMA Professional

See also

• Comparison of audio codecs
• Ascertaining high fidelity: double-blind tests
• Vorbis

External links

• Hydrogenaudio - Community audiophile site, host of most non-commercial ABX testing
• ff123's ABC/HR Audio Comparison Utility for Windows
• SoundExpert Continuous blind listening tests of codecs over the internet