Hi Meir,
That is not a standard format so standard PAL scripts might be ok but will probably confuse the display.
What are you using for a display? It's unlikely any HDMI display would display your non-standard format anyway. Have you looked at the voltage on the TMDS output or connected the output to an HDMI analyzer?
What I'm trying to figure out is if the part is actually putting out your format but the display doesn't know what to do with it. If you are seeing ~3V on all the TMDS lines then the ADV7511 is probably outputting your data but maybe the H/V/DE are off from what you expect. Since I doubt any HDMI display will show the format, you really need an HDMI analyzer to see what's actually coming out. We use an Astro VA-1838 here.
If the TMDS lines are 0V then nothing is coming out and you probably need to modify your script although we don't have anything for the format you are using.
As you are probably aware ADI is in the middle of a end of year shutdown so we probably won't be able to do much to help you figure a script out until after the shutdown on 1/6.
Dave