You have to use another server to host your fast downloads.
That or you could setup bandwidth priorities to give the map downloads more bandwidth. but that would cause the game to lag.
Unless ofcourse you do have another server with the downloads, in which case the server is slow. >: