I would recommend looking at the signal wideband (several GHz) to see the spurious present at the Tx output.
Also, Tx section may be being overdriven. The output spectrum in the frequency range you are showing typically is dominated by the image spur. Try to scale the digital data some to see if you get an improvement.