In situations like these, you might try to use DMA to your advantage so that calculations are in internal RAM. You need to make sure that you know when DMAs are actually going to happen so that the access is still not too randomized. This is generally fairly straightforward if the SDRAM only has data and is not storing executable code.
This is also a place where the ADSP-21469 really works much better. It supports DDR2 which is much faster (but still has the intrinsic read speed issues). DDR2 is a bit tricky to layout out on a board so this needs to be done with clear understanding of the signal integrity and timing constraints.
Al Clark
www.danvillesignal.com