Apologies for the extremely late reply. I actually ended up back here searching for something else.
I'm not adding a real camera feed to the NVR so I'm not sure my actual settings would help. I created a daily sales stats feed using FFMPEG and MediaMTX running on a Raspberry Pi 3b+. FFMPEG converts a text file into a 1280x720 feed then pipes that to MediaMTX to create a real rtsp feed. I couldn't get the NVR to read from FFMPEG alone.
On my NV5208E I have the option "Customized" as @xundre mentioned, so after I got my feed to play in VLC I just copied the url over for both the main and sub streams.
On my NV5216E-8P the "Customized" option was missing so initially I pointed it at the the NV5208E with ONVIF over our VPN bridge to pull the feed. This was a bit ridiculous because the raspi is at the same location as the NV5216 and the NV5208 is at a different location so the feed had to cross the bridge twice. I later wrote a very basic mock ONVIF server that runs on the same raspi as and points to MediaMTX to minimize the hops and failure points.