I heard that some people do get it done that way. As long as you have bandwith, it should not be a trouble.
But keep in mind that the IVR could be as simple as a P.C.with a TFTP software on it.
Keeping the IVR next to the gateway, on the same network,it will be a “cleaner” solution