HI Alon,
If you are using BF548 processor, you can refer Chip Bus Hierarchy chapter in the processor's HRM. The On-Chip L2 Interface section provide details about access time & latencies involved in various L2 accesses.
Unfortuantely, I am not aware of any ways to reduce these latencies. Someone having understanding of it, may be able to offer some suggestions.