ideally, you should do some traces on the Abis interface :
check how long it takes to establish a UL TBF,
how long it takes to send the RLC blocks,
how long it takes before a reply is received from network,
and how long it takes to establish a DL TBF.
If TBF establishment is quick enough, then investigate the network side. Ping the GGSN, then ping a server located on same network as GGSN, then your gateway to get out of your packet CN, and then to a server on internet.
See how the different servers generate different response times.
Also, regarding the TBF estab, try doing several consecutive pings : the 1st ping takes more time because TBF must be established. The next ping should be much faster because the TBF are already established and “re-used” for the second ping.