Sorry but you already got the answer -
All filters are programmed in accordance with the filter and calibration guide.
It doesn’t matter which driver you use Linux or NO-OS - they all do the same.
I understand why you’re asking – but the answer is the SW engineer implemented whatever is documented .
And like I said before the filter response is a composite of multiple none ideal filters.
There is no simple answer.
We’re trying to ease filter design with some Matlab tools documented here –
http://wiki.analog.com/resources/eval/user-guides/ad-fmcomms2-ebz/software/filters
Please be aware that this is work in progress and the final filter designer will be released soon.
In your post here:
http://ez.analog.com/message/132664#132664
T. Lili stated that “You can use the approach you are using, finding minimum filter bandwidth that works in your application.”
The Matlab tools were putting in place will help you to calculate FIR filter coefficients that will also compensate the analog filter roll offs.
So I think looking into the Matlab model is a good way to start.
For standards like WiMAX – I’m pretty sure that you can get some recommended settings on the AD9361 forum.
-Michael