First of all ADV7181 DDR mode is really only intended to output 24 bit RGB. My recommendation is don't even attempt to use DDR mode than you don't have to worry about interfacing to it and you apparently only need YCbCr formats.
The NTSC/PAL cameras will need to go through the SDP and it output 8/16-bit YCbCr.
RS-170 is basically B&W NTSC can can go through the SDP. Check out this thread for mode discussion on this ADV7181C full scale luma
You only need to run RS-170 through the CP if you plan to output RGB in DDR mode.