CVE-2014-6440: Heap overflow in VLC transcode module

Wed 04 March 2015 by bblough

Executive Summary

VLC versions before 2.1.5 contain a vulnerability in the transcode module that may allow a corrupted stream to overflow buffers on the heap. With a non-malicious input, this could lead to heap corruption and a crash. However, under the right circumstances, a malicious attacker could potentially use this vulnerability to hijack program execution, and on some platforms, execute arbitrary code.

Methodology

This vulnerability was found by fuzzing using CERT's Basic Fuzzing Framework (BFF) and a sample file from mplayerhq. Testing was initially performed on Debian Sid (x86) using the VLC version 2.1.2-2+b3 from Debian, and later on Windows XP 32-bit using VLC version 2.1.3 from upstream.

After some time, BFF produced a fuzzed file that caused the following crash:

    *** Error in `/usr/bin/vlc': free(): corrupted unsorted chunks: 0xb384fd68 *** 
    ======= Backtrace: ========= 
    /lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x75e52)[0xb756be52]
    /lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x76b90)[0xb756cb90]
    /usr/lib/libvlccore.so.7(+0x7df4b)[0xb7481f4b] 
    /usr/lib/vlc/plugins/audio_filter/libmpgatofixed32_plugin.so(+0xb5a)[0xb3466b5a]
    /usr/lib/libvlccore.so.7(aout_FiltersPlay+0x131)[0xb7477021] 
    /usr/lib/vlc/plugins/stream_out/libstream_out_transcode_plugin.so(+0x579a)[0xb47a979a]
    /usr/lib/vlc/plugins/stream_out/libstream_out_transcode_plugin.so(+0x20f8)[0xb47a60f8] 
    /usr/lib/libvlccore.so.7(+0xa3ce1)[0xb74a7ce1]
    /usr/lib/libvlccore.so.7(+0x3a0bf)[0xb743e0bf]
    /lib/i386-linux-gnu/i686/cmov/libpthread.so.0(+0x6cf1)[0xb76b2cf1] 
    /lib/i386-linux-gnu/i686/cmov/libc.so.6(clone+0x5e)[0xb75e5c3e]

Testing under valgrind showed multiple invalid writes past the end of allocated heap chunks (heap overflow), and following the execution using gdb confirmed that this was not a false positive.

Root Cause

Further investigation narrowed the issue to two functions in modules/audio_filter/converter/mpgatofixed32.c.

In the function Convert, this code allocates heap space based on the number of samples, bits, and channels detected:

    size_t i_out_size = p_block->i_nb_samples *
    p_filter->fmt_out.audio.i_bitspersample *
    p_filter->fmt_out.audio.i_channels / 8;

    block_t *p_out = block_Alloc( i_out_size );

However, in the function DoWork, the samples and channels are detected using a different method:

    struct mad_pcm * p_pcm = &p_sys->mad_synth.pcm;
    unsigned int i_samples = p_pcm->length;
    mad_fixed_t const * p_left = p_pcm->samples[0];
    mad_fixed_t const * p_right = p_pcm->samples[1];
    float *p_samples = (float *)p_out_buf->p_buffer;

    if( i_samples != p_out_buf->i_nb_samples )
    {
        msg_Err( p_filter, "unexpected samples count (corrupt stream?): "
                 "%u / %u", i_samples, p_out_buf->i_nb_samples );
        p_sys->i_reject_count = 3;
        goto reject;
    }

    /* Interleave and keep buffers in mad_fixed_t format */
    if ( p_pcm->channels == 2 )
    {
        while ( i_samples-- )
        {
            //assert( *p_left < MAD_F_ONE );
            //assert( *p_left >= -MAD_F_ONE );
            //assert( *p_right < MAD_F_ONE );
            //assert( *p_right >= -MAD_F_ONE );
            *p_samples++ = (float)*p_left++ / (float)MAD_F_ONE;
            *p_samples++ = (float)*p_right++ / (float)MAD_F_ONE;
        }
    }
    else
    {
        assert( p_pcm->channels == 1 );
        while ( i_samples-- )
        {
            //assert( *p_left < MAD_F_ONE );
            //assert( *p_left >= -MAD_F_ONE );
            *p_samples++ = (float)*p_left++ / (float)MAD_F_ONE;
        }
    }

With a sufficiently corrupted input stream, this allows for a situation where the buffer is allocated to contain one channel of data, but then two channels of data are written to it, causing the write to overflow the buffer.

Remediation

Prior to being notified of this issue, the VLC team had already made changes to the 2.2 development branch here, here, and here that corrects this issue by reinitializing the filters when a format change is detected. However, the fixes had not yet been backported to the 2.1 maintenance branch.

Once notified, the VLC team quickly resolved the issue by backporting the relevant patches to the maintenance branch here, here, and here. They also added an additional check on both the development and maintenance branches for good measure.

CVE-2014-6440 was assigned to this issue.

Timeline

2014-04-18: VLC team notified of issue

2014-04-19: Fixed in VLC repository

2014-07-06: VLC 2.1.5 maintenance release