How are you combining the two tones before the ADL5380? Was the -20 dBm number what was set at the signal generator or at the input of the ADL5380?
Can you also proved the signal levels, as read at the spectrum analyzer, of the main tones and IMD tones in dBm. Also, what baseband frequency are you operating at?
Can you should me the your math on how you are calculating the IP3? Input or output will be OK.
There could be a couple of things going on here:
1) You could be over-driving the output of the ADRF6516; it is hard to say because I don't know the exact signal levels through the signal chain and/or,
2) You could be calculating IP3 incorrectly.
I'm going to attach a signal chain calculator, done in excel to show you what the signal levels ought to be.
Thanks,
Joel