- This topic has 22 replies, 1 voice, and was last updated 9 years, 10 months ago by Pix.
26th December 2009 at 23:16 #60491lilyGuest
Maybe somebody has already faced the problem we’re trying to solve now – please, help!!
We’ll be very grateful!
Some days ago we detected deteriorating TBF drop rate for several BSCs – both in DL and UL. The average daily value increased from ~ 2% to 28-30%!!!! It’s all BSS drops. Those BSCs belongs to different MFSs, one of MFSs is old – G2, all others – Evolution G2,5. This problem affects ALL cells in each BSC (not some ones). Two days after that problem appeared three more BSCs became ill. Corresponding reports in NPO are terrible 🙁 We changed the GPU board for one of BSCs connected to G2.5 MFS (to spare board in the rack). Nothing happened :(( All other KPI are within limits – no congestions, no problems with CPUs etc.
According to message flow during TBF release, BSS TBF drops have no specials triggers and counters, it’s only the difference between TBF normal release and all others nonacceptable releases.
I feel this is a clear matter to contact TAC but maybe there are some well-known Alcatel tricks here – like with Power Control in GPRS and deteriorating of TBF establishment success rate? 🙂
Many thanks in advance!29th December 2009 at 21:24 #60492PixGuest
People are still working on 26th December ???
I’ll check my database next week (back to work), but from memory I can’t remember anything that impact GPRS drops. The way you describe it, it sounds like a perfect task for TAC gurus ( 🙂 ).
is it the start of a “2010 bug” in GSM ?
-pix31st December 2009 at 20:24 #60493lilyGuest
Good evening Pix!
Thank you very much!
Oh, it was very hot time before New Year holidays – so yes, even 26 :))
We wrote to our TAC – not an official query at first, but a detailed letter. I hope they’ll be able to help.
But all business – in the nest year :)))
Happy New year!!!!
Here’s wishing you more happiness
Than all my words can tell,
Not just alone for New Years Eve
But for all the year as well.
Lily6th January 2010 at 14:19 #60494PixGuest
Sorry, I can’t find anything on UL & DL TBF drops. Most of the problems are either DL or UL, and are fixed with a patch.
Are you still in B9 ? Perhaps it is a sign that you should migrate to B10… The whole world is waiting for you… You are the last ones using RNO :)) (or almost…)10th January 2010 at 20:46 #60495lilyGuest
Dear Pix, good evening!
Many many thanks to you.
We’ll join the world very soon 🙂 And now we use both RNO and NPO, we aren’t so obsolete 🙂 Soon we’ll migrate to B10 and then …
About that problem with TBF drops: solution has just revealed itself. Two days ago we had problem with one of those BSCs – “BSC connection lost”, so we had to reset active omcp board. After such manipulations maintenance was restored of course, but moreover – problem with TBF drops was solved too :)) Is it Alcatel’s temper or Alcatel’s magic? We shared this info with our TAC – they agreed to repeat the experiment with three other BSCs. If it works, then – four nights and all 12 BSCs will be treated.
Oh, one question about NPO (sorry, I have little experience with this tool now): I couldn’t find sheet “Adjacency” – it should be (it was in RNO) on left panel, below the Cell/BSS/TRx/BTS etc sheets. But now it is absent. Whole our department was trying to find it but unsuccessfully.
Maybe we were searching in wrong place?
Once more – many thanks!10th January 2010 at 22:37 #60496PixGuest
One day with NPO, and already you are pointing out one shameful characteristic…
In order to display adjacencies and TRX, you must work in a Working Zone with less than 2000 cells.
If you are not happy with that, refer to your NPO Administrator Guide (you can see it from the icon Box) and look up “Working zones” part. The first chapter is describing the trick to increase the “2000” limit.
“enjoy” NPO… I guess it takes some time getting used to NPO, but you’ll see the tool gives much more freedom than RNO. Less “grace” but more power.
Thanks for sharing the OMCP “trick”. Let me know how it turns out on the other BSCs.11th January 2010 at 20:14 #60497lilyGuest
Dear Pix, good evening!
I wasn’t looking for flaws in NPO, really :)) I only needed some HO statistic for several neighbor cells, but I’m absolutely happy with possibility you described 🙂 2000 cells is enough for me now 🙂 Pity, I haven’t been in office today so I couldn’t try…
Oh, yes, generally my first impression about NPO is positive – fast, flexible, nice tool. It’s pleasant to work with
We are going to repeat the manipulation with other three BSCs tomorrow (or maybe the day after tomorrow), in the night. Of course, I’ll share the results. We’re worrying only about G2 BSCs – the similar procedure is reseting CPRC – but from our experience they don’t like such manipulations very very much. Consequences could be unexpected and unpleasant 🙂 But we’ll see 🙂
Good night!18th January 2010 at 07:41 #60498VinhGuest
My name Vinh,i come from HCM,Viet Nam.
Can u help me about software commissioning Acatel A9100?
My email: firstname.lastname@example.org
Thanks and best regards.
Vinh18th January 2010 at 22:15 #60499lilyGuest
Dear Pix, good evening!
About our experiment and results:
Three times we repeated that experiment (different MFSs, different types of BSCs, etc). But all the same – we had no success. Alcatel’s magic is much more complicated 🙂 So we’re waiting answer and help from our TAC.
While it is silence we also have been trying to analyze this problem by ourselves. I’ve noticed one thing – that abnormal TBF drop due to BSS correlates perfectly with counter P103 (“Number of GCH frames badly received by the MFS (due to a bad CRC)” Whenever the EGCH layer detects that a GCH frame has not been correctly decoded in the MFS (due a bad CRC) ).
The difference between values of this counter for “normal” BSCs and for those BSCs with high TBF drop is not times or hundreds, it’s thousands (pictures in NPO are astonishing). I don’t know now either it is a hint which could help to analyze this problem or it’s only an obvious consequence 🙂 I understand what CRC is and what is its purpose, but I can’t find any details about the process of its calculations, which could help us to understand how to solve our problem, especially – about it’s calculation for packet data…
Now we’re elaborating two propositions: one – rebooting MFSs and other – deactivating CRC check for one or two BSCs…..
Maybe we’ll be lucky :))))))
I’ll share the results.
Good night!!!20th January 2010 at 15:43 #60500PixGuest
Thanks Lily, I’m reading your story with impatience now. The suspens is killing me. Lily vs. TAC, who’s the best ? 🙂
Your findings about CRC are interesting, you could forward them to the TAC.
I didn’t even know the CRC could be deactivated in the BSC 🙂
pix23rd January 2010 at 23:54 #60501lilyGuest
Dear Pix, good evening!
Sorry for such a long delay with answer – we have awful weather here and one-day business trip turned out to became a five day isolation from civilization 🙂
Oh. no – it’s not me vs. TAC, it’s collective human mind vs. Alcatel’s temper 🙂 yesterday my boss said that he’s giving up because alcatel equipment has female character 🙂
We made the next step in our experiments and now I’ve just seen the results – I have no words, no words at all!
But all in order –
At first – you are absolutely right, it’s impossible – to deactivate CRC checking at BSC. All we could find and done (at that moment) was deactivation CRC between SGSN and MFS (on Aters) – although it wasn’t what we needed. My expectations were obvious: ether the problem will be solved (less probable outcome) or nothing will change (more probable end as for me). But Alcatel always can surprise you: even in the simplest situation, when you have only two hypotheses, it easily can show you the third variant. And this time wasn’t an exception 🙂 The P103 counter changed – it decreased twice!!! I checked several times – we deactivated CRC for both links from MFS to SGSN for this BSC.
After half an hour of emotions new questions came:
1) The full name of P103 (from catalog) is NB_ERROR_UL_GCH_FRAME. Only UL? Does it mean that we must consider only part from MS to MFS?
2) If – yes, then what is the second part, calculating CRC?
3) Can we deactivate CRC for that very part? Or what could be a problem with it calculation?
I’m looking for answers to these questions and many others now. I hope soon we’ll have new results.
P.S. Of course, I’ve shared all info with TAC, but they are very busy now – neighbor region of us is migrating now from B9 to B10 and the process isn’t such smooth, so guys from TAC almost every night spend near the equipment and we have authority to do what we consider to necessary to solve this problem
Good night!!26th January 2010 at 06:25 #60502PixGuest
(note to myself : don’t want to loose track of this topic… have to look for reasons for P103 in BSC and MFS)
thanks lily for all the long posts, very instructive and funny 🙂3rd March 2010 at 22:08 #60503lilyGuest
Dear Pix, good evening!
At last I can continue and finish this story which dragged on too long.
So where should I start?
Oh, we had been having a long frozen zone (because of preparation for migration to B10), so we could only watch our poor BSCs and did nothing. But one accident put the lid on that precise correlation between TBF drops due to BSS failure and P103 counter. One morning after unsuccessful attempt to move one of BSCs from old OMC-R to new one (it meant of course moving from one MFS to another) I noticed on that BSC the problem with abnormal TBF drops due to BSS appeared!!! But in this case it wasn’t accompanied by the P103 counter – it remained close to zero. And – no other changes, no other hints – just enormous values of TBF drops due to BSS. Still we could do nothing – frozen zone + a fault report on that BSC. But some time later we needed very badly one more BSC for our highly loaded region, we had one MX in test mode which could be easily used. We started moving BTSs into it, and again – I noticed that we had a problem – abnormal P103 (>200000 per our) this time, and no TBF drops due to BSS at all. Our theory failed so we started to solve these problems separately and first was that one with P103 counter (bad CRC). It was peculiar situation – we’re absolutely sure about our links (either Aters or Gb), BER was better than 10^-8, but the checksums that had been calculated from the same data were different for two ends. Obviously, the problem was in calculation process on those two ends. As we suggested it could be something like different type of CRC, only one-end calculation or some problem with CRCRR (CRC result register – one of them for example) where the result of checksum calculation was put. But we use only CRC4 and as for other hypothesis – we didn’t see the way to check them directly, so we started to check all standard “alcatel solution”: from resetting and changing GPU board to changing settings on SGSN. It didn’t help.
But the thing that really helped and solved that problem was the favorite procedure of deactivation -> activation the feature back again. This time the feature was CRC checking which can be enabled or disabled only from local terminal. So one day we (after a long pondering) decided to do it – deactivated and than activated back CRC checking on Aters. To be true – it was terrible few hours – we thought we lost our BSC – the process was so long. But fortunately it didn’t happen, although we had to reset lots of HW, including cprc board after all. That was the solution for high P103 (without any other problems) 🙂
Than we started to work at TBF drops due to BSS problem – another long chain of hypotheses and experiments – at last after rebooting whole MFS it vanished.
For those BSCs where we saw both problems we started from disabling/enabling CRC but after all we had to reset almost everything – and after such actions all problems have gone.
I was watching for all those treated BSCs for some hours but then we lost NPO (due to migration), so I wasn’t absolutely sure about the results. Today we’ve been given this tool back – and now I can see that since that day everything is excellent. Graphs from NPO are really astonishing 🙂
… and they lived happily ever after …
Best regards, Lily4th March 2010 at 19:50 #60504PixGuest
Well, it looks like I’m going to see this astonishing network by myself next week !
Thanks for the feedback, but in the end I learnt nothing new. The ALU law still works…
“if it fails, reboot it”
pix4th March 2010 at 21:18 #60505lilyGuest
Bonsoir, dear Pix!
I’ve just replied to your letter 🙁
One note – in Kiev we used Siemens BSS and NSS, so you’ll see, of course, an astonishing network, but much more astonishing it is in southern regions where we use Alcatel :)))))) So – welcome to Odessa!!!
About Alcatel law – I think it would be excellent to write a large book about what you should restart, reset and disable-enable in every situation, because sometimes it takes too much time to find that blessed entity 🙂 Such a book could became a bestseller 🙂 and we would bequeath it to our descendants 🙂
oh, please, don’t mind I’m just kidding – trying to cheer myself after all