ffmpeg with NVENC/CUDA install on Linux

Need to compile, point libraries and build dirs to local user folder for build so that it is easier to remove and clean up if you mess up instead of putting directly into /usr/local, which is default.

Nvidia's guide, is well.. semi accurate, but not verbatim.

The Nvidia guide ¹⁾ does not work with the limited amount of packages required. I referred to the official ffmpeg compile guide ²⁾ which gives you the following:

sudo apt-get update -qq && sudo apt-get -y install autoconf automake build-essential cmake git-core libass-dev libfreetype6-dev libgnutls28-dev libmp3lame-dev libsdl2-dev libtool libva-dev libvdpau-dev libvorbis-dev libxcb1-dev libxcb-shm0-dev libxcb-xfixes0-dev meson ninja-build pkg-config texinfo wget yasm zlib1g-dev

Honestly, I think just installing pkg-config from the repo will suffice, but this just helps overall to ensure ffmpeg compiles without issues.

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Debian&target_version=11&target_type=deb_network

wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo add-apt-repository contrib
sudo apt-get update
sudo apt-get -y install cuda

Install linux headers

Per NVIDIA³⁾, they want you to install headers based on uname -r. Instead, install the package from the base arch level, not specific kernel version. This way when you update, the headers update too. In my case, running Debian, it would be linux-headers-amd64

Per NVIDIA,⁴⁾ they want you to download the Cuda Headers from VideoLan's Git page. You can't always trust it though, it may be a version off. In my last go-round, they were still on version 11, while my setup was looking for version 12. Either way, download the files from an alternative source (like FFMpeg's Git Repo https://github.com/FFmpeg/nv-codec-headers), and compile as normal.

First things first, do an ldconfig -v to see if there are any conflicts with the shared libs

Then run an ldd of the binary you want to check to show all of the dependent shared libs. If any are missing, you will need to check on what was missed in the pre-steps. The NPP libs are part of the cuda toolkit

ldd /usr/local/bin/ffmpeg ##Or wherever you compiled ffmpeg
	linux-vdso.so.1 (0x00007ffd5ed27000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f45a5902000)
	libnppig.so.11 => /lib/x86_64-linux-gnu/libnppig.so.11 (0x00007f45a3a35000)
	libnppicc.so.11 => /lib/x86_64-linux-gnu/libnppicc.so.11 (0x00007f45a3310000)
	libnppidei.so.11 => /lib/x86_64-linux-gnu/libnppidei.so.11 (0x00007f45a28b7000)
	libnppif.so.11 => /lib/x86_64-linux-gnu/libnppif.so.11 (0x00007f459ee2e000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f459ee28000)
	libvpx.so.6 => /lib/x86_64-linux-gnu/libvpx.so.6 (0x00007f459ec0b000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f459ebe9000)
	libdav1d.so.4 => /lib/x86_64-linux-gnu/libdav1d.so.4 (0x00007f459eab5000)
	libfdk-aac.so.2 => /lib/x86_64-linux-gnu/libfdk-aac.so.2 (0x00007f459e972000)
	libopus.so.0 => /lib/x86_64-linux-gnu/libopus.so.0 (0x00007f459e917000)
	libx264.so.160 => /lib/x86_64-linux-gnu/libx264.so.160 (0x00007f459e655000)
	libx265.so.192 => /lib/x86_64-linux-gnu/libx265.so.192 (0x00007f459d6d6000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f459d511000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f45a7823000)
	libnppc.so.11 => /lib/x86_64-linux-gnu/libnppc.so.11 (0x00007f459d280000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f459d275000)
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f459d0a8000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f459d08e000)
	libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f459d07e000)

I do prefer using the full guide, as there are additional steps you'll need to get everything set for ffmpeg

https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#pre-installation-actions

Grab ffmpeg from its source

git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg/

You'll want to check out the full docs ⁵⁾ for the env setups.

export LD_LIBRARY_PATH=/usr/local/cuda-12.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Don't think this is needed..

PATH="$HOME/bin:$PATH" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig"

Running the ffmpeg configure script

This is the official way of doing things:

./configure --enable-nonfree --enable-cuda-nvcc --enable-libnpp --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --disable-static --enable-shared

This is a WIP, hoping to cut this down to the minimal needed for doing any files thru nvenc only.

./configure --pkg-config-flags="--static" --enable-nonfree --enable-cuda-nvcc --enable-libnpp --extra-cflags="-I$HOME/ffmpeg_build/include" --extra-cflags=-I/usr/local/cuda/include --extra-ldflags="-L$HOME/ffmpeg_build/lib" --extra-ldflags=-L/usr/local/cuda/lib64 --extra-libs="-lpthread -lm" --ld="g++" --enable-gpl --enable-libfdk-aac --enable-libopus --enable-libdav1d --enable-libvpx --enable-libx264 --enable-libx265