I can try. Here [1] is some of the terminology. We were assisted by Orange (from the UK) to set up the first GSM network in the U.S. using primarily Ericsson equipment. We soft launched in San Diego at the RNC, then re-launched officially state wide in California and Nevada using Ericsson AXE-10 switches (mainframes). We had several MSC's. These were the switches that routed calls to the SS7 network. Those had BSC's which were the switches that routed calls from within a region to the MSC's. The cell sites attach to the BSC's. The cell sites were connected to the telco network with channelized T1's. (24 channels) Each channel was divided up into multiple voice and stand alone dedicated control channels. The phones registered to an HLR (Home Location Register). The HLR, ALR and EIR used for hardware and user mapping / registration. That data fed into billing systems. Access to those systems was performed using the x.25vbis protocol. We also had value added systems that provided support for SMS (text messaging), message waiting indicators (for voicemail) and other various services. It turned out that text messaging was super popular, so it went from being a value added service to a primary service. We did not charge per message, despite our sister company in the UK doing so. In the early days, you could tail all the text messages and almost keep up. All of our vendor documentation was in massive binders. None of the documentation was digital. Any changes to code would take many months to get revised documentation if it was revised. I think I may have been one of the few people to ever use the EIR to block a stolen phone. One of our cell techs left their Nokia-9000 (fancy early prototype keyboard flip phone, promoted in the move The Saint) and they were very expensive. I spoke with the phone thief, offered them a reward, but they declined so I turned the phone into a paperweight. That feature was never used by customer support as far as I know, for fear of bricking the wrong phones. One of the cool features of GSM is call prioritization. If a phone is flagged as a first responder, they can kick a person off a congested cell site to free up a voice channel.
One service we never implemented was Wildfire. (no documents to cite that I know of). She used the same tech the NSA used to monitor voice calls. She was created in response to the hands-free laws that were about to be passed. You could say in the middle of a call, "Wildfire!" "Yes" "Call mom" or "bridge on mom". She had some funny Easter-eggs as well, at least in demo mode. "Wildfire!" "Yes" "What sound does a cow make?" "mooo..." I guess she could be considered the first iteration of something like Siri, but on the server side. The coolest thing about Wildfire was that she could understand every language and every dialect with zero training. [EDIT: I stand corrected, see threads below, apparently she did require training, but came to use fully populated / trained. ] She ran on SCO Unix. Thankfully our lobbyists were able to kick that can far down the road and cell phones evolved to a point where hands-free was possible on the devices vs. being required on our network. No idea what the official name of that code was. This was in the late 90's and cellphones were a bit more primitive than they are today.
I was responsible for doing switch upgrades in the off peak hours (generally start around 2am). In most cases this was not service interrupting. We injected code written in PLEX [2] live into the switches. To test call routing I would dial 911 to verify it worked, then call coworkers. One time I forgot to apply a U.S. specific code patch that muted the operator override tone to tell you the operator is on your call. That made call quality testing awkward. "Who is that? Is someone on our call?". They would put me up in really shady cheap hotels. One of the hotels in El Cajon had walls so thin you could hear people 4 rooms away. I woke up every day to cops yelling at someone to get on the ground any time I had to go there.
One time a switch upgrade went sideways while I was at the kids basketball game. They escalated to me to fix it. Problem was that the only fix was a full rollback. If I recall correctly, the B-Tree tables were corrupted and most calls were not routing properly to landline. Cell to cell was still functional. I had to reboot an MSC (full reload) from my Nokia-9000. I had to telnet into the gateway then connected via x.25vbis and ran a full reload. Hopefully they have at least moved to ssh by now. So if you were curious, rebooting those mainframes from disk (MFM disks IIRC) took 40 minutes.
SYREI:RANK=RELOAD,REASON="Resume updated"
That took Northern California off the network for 40 minutes. The moment I executed the command, everyone around me stopped ignoring their kids and put their phone away. The problem was resolved when everything finished loading and initializing.
I was also also had the privilege of monitoring one of the first mass-spammers of cell phones in the U.S. I was going to have their SS7 link cut, but my management said "They are paying their bills, keep your nose out of it". Most of the folks loaned out from Orange were our management and leadership team. I did not get along with some of them as some of them openly hated Americans. One of the times they messed with me was paging me to go watch a modem light all night, repeatedly, because. Between the many layers of bureaucracy of telco and the wildly toxic management, I left wireless telco and never looked back. The mostly hired people from the military, probably assuming they would just say "yes sir" to anything. Clearly they did not talk to my former commanders.
I also had to debug cell site issues. Nine times out of ten we would have to either reboot a sector or re-initialize it. In the mean time, bugs would be submitted to Ericsson developers. We had many developers from Sweden on site. I enjoyed working with the folks from Sweden. They had a good sense of humor and knew how to enjoy life. This was impressive considering the vast knowledge of both mainframes and cellsites they were required to have low-level knowledge of.
I know I am leaving out a lot of things, but most of the other experiences you could have at any big company. This is from a long time ago, so apologies in advance if I get a few details wrong. I already corrected one of them thanks to added details from leon1das. Nice to see there are other GSM folks here!
Hi! From ScyllaDB here. There are a few things that help us really get the most out of hardware and network IO.
1. Async everywhere - We use AIO and io_uring to make sure that your inter-core communications are non-blocking.
2. Shard-per-core - It also helps if specific data is pinned to a specific CPU, so we partition on a per-core basis. Avoids cross-CPU traffic and, again, less blocking.
3. Schedulers - Yes, we have our own IO scheduler and CPU scheduler. We try to get every cycle out of a CPU. Java is very "slushy" and though you can tune a JVM it is never going to be as "tight" performance-wise.
4. Direct-attached NVMe > networked-attached block storage. I mean... yeah.
We're making Scylla even faster now, so you might want to check out our blogs on Project Circe:
One service we never implemented was Wildfire. (no documents to cite that I know of). She used the same tech the NSA used to monitor voice calls. She was created in response to the hands-free laws that were about to be passed. You could say in the middle of a call, "Wildfire!" "Yes" "Call mom" or "bridge on mom". She had some funny Easter-eggs as well, at least in demo mode. "Wildfire!" "Yes" "What sound does a cow make?" "mooo..." I guess she could be considered the first iteration of something like Siri, but on the server side. The coolest thing about Wildfire was that she could understand every language and every dialect with zero training. [EDIT: I stand corrected, see threads below, apparently she did require training, but came to use fully populated / trained. ] She ran on SCO Unix. Thankfully our lobbyists were able to kick that can far down the road and cell phones evolved to a point where hands-free was possible on the devices vs. being required on our network. No idea what the official name of that code was. This was in the late 90's and cellphones were a bit more primitive than they are today.
I was responsible for doing switch upgrades in the off peak hours (generally start around 2am). In most cases this was not service interrupting. We injected code written in PLEX [2] live into the switches. To test call routing I would dial 911 to verify it worked, then call coworkers. One time I forgot to apply a U.S. specific code patch that muted the operator override tone to tell you the operator is on your call. That made call quality testing awkward. "Who is that? Is someone on our call?". They would put me up in really shady cheap hotels. One of the hotels in El Cajon had walls so thin you could hear people 4 rooms away. I woke up every day to cops yelling at someone to get on the ground any time I had to go there.
One time a switch upgrade went sideways while I was at the kids basketball game. They escalated to me to fix it. Problem was that the only fix was a full rollback. If I recall correctly, the B-Tree tables were corrupted and most calls were not routing properly to landline. Cell to cell was still functional. I had to reboot an MSC (full reload) from my Nokia-9000. I had to telnet into the gateway then connected via x.25vbis and ran a full reload. Hopefully they have at least moved to ssh by now. So if you were curious, rebooting those mainframes from disk (MFM disks IIRC) took 40 minutes.
That took Northern California off the network for 40 minutes. The moment I executed the command, everyone around me stopped ignoring their kids and put their phone away. The problem was resolved when everything finished loading and initializing.I was also also had the privilege of monitoring one of the first mass-spammers of cell phones in the U.S. I was going to have their SS7 link cut, but my management said "They are paying their bills, keep your nose out of it". Most of the folks loaned out from Orange were our management and leadership team. I did not get along with some of them as some of them openly hated Americans. One of the times they messed with me was paging me to go watch a modem light all night, repeatedly, because. Between the many layers of bureaucracy of telco and the wildly toxic management, I left wireless telco and never looked back. The mostly hired people from the military, probably assuming they would just say "yes sir" to anything. Clearly they did not talk to my former commanders.
I also had to debug cell site issues. Nine times out of ten we would have to either reboot a sector or re-initialize it. In the mean time, bugs would be submitted to Ericsson developers. We had many developers from Sweden on site. I enjoyed working with the folks from Sweden. They had a good sense of humor and knew how to enjoy life. This was impressive considering the vast knowledge of both mainframes and cellsites they were required to have low-level knowledge of.
I know I am leaving out a lot of things, but most of the other experiences you could have at any big company. This is from a long time ago, so apologies in advance if I get a few details wrong. I already corrected one of them thanks to added details from leon1das. Nice to see there are other GSM folks here!
[1] - https://en.wikipedia.org/wiki/Network_switching_subsystem
[2] - https://en.wikipedia.org/wiki/AXE_telephone_exchange