DMA taking longer to read 64 bytes of data in stop mode from SFLASH(QUAD-SPI) which is creating distortion in output audio

Using ADSP -21569 custom board for audio processing SPI2 DMA SFLASH communication , where I have to read 64 bytes of data per frame from SFLASH and copy that data to model buffer which will be passed to the speakers and we have maximum of 15 channels to be read in single frame so calculation (64 Bytes x 15 channels = 960 Bytes) am just lopping it for 15 channels, using Quad SPI as OSPI is not sup[ported by SFLASH

--> Problem - When I read data up to 8 channels am not seeing any distortion going beyond 8 channels causing a distorted audio output .

-> Observation - DMA taking 6.636 micro second to read 64 bytes of data for single channel , when I try to read data for 9 channel actually DMA will create a delay of 6.636us x 9 = 59.724us while copying the data to model buffer which is causing the distorted output.

Could you please suggest a way to reduce the DMA read time ? or any solution