HI Saurabh,
Unless I'm reading your schematic wrong it looks like the CVBS in is routing to two different devices and you've no stuffed the tie to ground of the CVBS connector. This makes me wonder if you are having signal integrity issues that are causing all the interupts and issues.
Is it possible for you to put a 0 ohm in R97 and to de-pop C106 to see if it alters the behavior?
Dave