You probably want to take a look at
ffmpeg and
libavcodec. It is the backend used by many video players on Linux and can encode and decode many video formats. This will just give you a raw buffer of pixels when decoded, so you just have to figure out how to show them.