there is a formula for that… but it’s totally empirical. Better than the formula is to follow the recommendations, based on experiments :
For GSM900 : at least 3m horizontal separation
For DCS1800 : at least 1.5m horizontal separation
Additional rule : if height of antenna > 10*3m (or 10*1.5m for DCS1800), then the separation should not be equal to the values above, but should be at least equal to Height/10.
If you’re using vertical separation, then the distances are shorter but it is slightly less effective.
Pix