IMEI authentication may not be mandatory (in your country), so you could by-pass it. The only drawback is the lesser security when setting up calls, and the inability to know the handset model from a call trace.
Regarding security, there is probably a good reason to keep the IMEI check, but I’m not familiar with this aspect of things. If I were you, I would bring up the topic to security experts.
Regarding tracing, it depends if you’re using massive probing servers such as Astellia. Showing the IMEI information for each call is a great way to isolate problematic handset models.
Finally, if your government or a private company tests your network and finds out that it is less protected that your competitors, it may
(1) play against you when they have to choose which operator to subscribe to.
(2) may be bad publicity.
The IMEI phase is not taking up so much time I think… It’s a phase which is run in parallel to other phases, which means it doesn’t “block” the call setup process (afaik). I wonder how much time you’d gain by removing it.