9/11 Pager Messages
2001-09-11
05:36:40
Skytel
[005051613]
D
ALPHA
Case 064408 MA1BAM Tucson, AZ Alltel Unplanned Outage 09/11 04:35 Note#0 P1: CDMA-MM MM-8 went out of service after a node swap. Bridge number 847-435-4201 [Cachun_Combs]
2001-09-11
05:54:10
Skytel
[005051613]
D
ALPHA
(2 of 2) ng from the backup node. nioc2 is up and nioc3 is coming up. [Monica_Jaramillo]
2001-09-11
05:55:40
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064408 MA1BAM Tucson, AZ Alltel Unplanned Outage 09/11 04:52 Note#1 P1: CDMA-MM There is not access to the omc at this point. performing swap on mm8a that was scratch yesterday the cdfs were missi
2001-09-11
06:04:40
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064408 MA1BAM Tucson, AZ Alltel 100% Out, Cum=00:32 Unplanned Outage 09/11 05:03 Note#2 P1: CDMA-MM There are processing calls on mm8b ( ( primary node). performing a scsiim-init mm8a level=2 acti
2001-09-11
06:05:40
Skytel
[005051613]
D
ALPHA
(2 of 2) on=full (backup node) [Monica_Jaramillo]
2001-09-11
06:26:40
Skytel
[005051613]
D
ALPHA
(2 of 2) is fully restored. [Monica_Jaramillo]
2001-09-11
06:28:40
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064408 MA1BAM Tucson, AZ Alltel 0% Out, Cum=00:54 Fully Restored 09/11 05:24 Note#3 P1: CDMA-MM The mm8b (Primary) it is up and and there are call processing, backup node is up too. the system
2001-09-11
09:00:40
Skytel
[005051613]
D
ALPHA
NACNRC2@motorola.com||P1 Winter
2001-09-11
09:00:42
Skytel
[005051613]
D
ALPHA
Haven System Case 060445 Update: Per NTL request, on yesterdays conf call, The EBTS - GR Patch 03.09.05.08 was been mapped in the Florida urban (orlsys, whsys, miasys , fl4sys ) last night. Market will schedu
2001-09-11
09:57:41
Skytel
[005051613]
D
ALPHA
Case 064419 RD1CTL Raleigh, NC Alltel No Outage 09/11 08:54 Note#0 P1: EMX2500 NO IPRs on the switch. Both Call Ext are down. Using 847-435-4202. CP 100% effected analog and digital. [Ahmed_Qureshi]
2001-09-11
10:00:41
Skytel
[005051613]
D
ALPHA
Case 064419 RD1CTL Raleigh, NC Alltel Unplanned Outage 09/11 08:58 Note#1 P1: EMX2500 NO I
2001-09-11
10:00:42
Skytel
[005051613]
D
ALPHA
PRs on the switch. Both Call Ext are down. Using 847-435-4202. CP 100% effected analog and digital. [Ahmed_Qureshi]
2001-09-11
10:13:40
Skytel
[005051613]
D
ALPHA
NACNRC2@motorola.com||CRITICAL Hanover DAP Case 064420 Escalating to Critical per market request
2001-09-11
10:14:55
Skytel
[005051613]
D
ALPHA
(2 of 2) traffic volume, Its a capacity iss
2001-09-11
10:14:57
Skytel
[005051613]
D
ALPHA
ue. Currently customer is taking analog cell sites out of service and spans. Call Ext crashed earlier. Call EXT are in a process of coming up. [Ahmed_Qureshi]
2001-09-11
10:15:56
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064419 RD1CTL Raleigh, NC Alltel 100% Out, Cum=00:23 Unplanned Outage 09/11 09:12 Note#2 P1: EMX2500 EMX outage recovery, CNRC CSE, Alcatel and customer on technical bridge. Customer have
2001-09-11
10:15:57
Skytel
[005051613]
D
ALPHA
very high
2001-09-11
10:30:40
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064419 RD1CTL Raleigh, NC Alltel 100% Out, Cum=00:38 Unplanned Outage 09/11 09:28 Note#3 P1: EMX2500 System crashed due to a system overload... The customer is going to remove half of the incomming
2001-09-11
10:32:40
Skytel
[005051613]
D
ALPHA
(2 of 2) lands spans, and also half of the modems at the channel back, thus attempting to prevent another crash once restoring the call ext procs. [Robert_Kosyluk]
2001-09-11
10:57:55
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064419 RD1CTL Raleigh, NC Alltel 100% Out, Cum=01:05 Unplanned Outage 09/11 09:55 Note#4 P1: EMX2500 System still down, all analog cell sites and digital as well. We are currently attempting
2001-09-11
10:57:57
Skytel
[005051613]
D
ALPHA
to get
2001-09-11
10:59:55
Skytel
[005051613]
D
ALPHA
(2 of 2) Dave Oaks access to the system. [Robert_Kos
2001-09-11
10:59:57
Skytel
[005051613]
D
ALPHA
yluk]
2001-09-11
11:09:41
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064421 AT3AIR Atlanta, GA Verizon Wireles No Outage 09/11 10:06 Note#0 P1: EMX2500 100% Outage d
2001-09-11
11:09:42
Skytel
[005051613]
D
ALPHA
ue to call overload. Dual processor outage. Outage recovery is currently working on off-loading the swi
2001-09-11
11:10:55
Skytel
[005051613]
D
ALPHA
(2 of 2) tch. [Rosabel_Jumilla]
2001-09-11
11:12:42
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064421 AT3AIR Atlanta, GA Verizon Wireles No Outage 09/11 10:07 Note#1 P1: EMX2500 100% Outage in AT3AIR due to call overload. Dual
2001-09-11
11:12:44
Skytel
[005051613]
D
ALPHA
processor outage. Outage recovery is currently working on off-loadi
2001-09-11
11:13:55
Skytel
[005051613]
D
ALPHA
(2 of 2) ng the switch. [Rosabel_Jumilla]
2001-09-11
11:16:42
Skytel
[005051613]
D
ALPHA
Case 064423 CH1ATM Charlotte, NC Alltel No Outage 09/11 10:12 Note#0 P1: EMX2500 Customer is getting paging overload and remote 96 IPRs. CP is not effected at current time. [Ahmed_Qureshi]
2001-09-11
11:35:56
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064421 AT3AIR Atlanta, GA Verizon Wireles 100% Out, Cum=01:23 Unplanned Outage
2001-09-11
11:35:57
Skytel
[005051613]
D
ALPHA
09/11 10:33 Note#2 P1: EMX2500 Bridge# 847-435-4201 is opened for this outage. CBSC outage recovery is working on throttling th
2001-09-11
11:37:41
Skytel
[005051613]
D
ALPHA
(2 of 2) e calls from the CBSC. [Rosabel_Jumilla]
2001-09-11
11:51:56
Skytel
[005051613]
D
ALPHA
(2 of 2) rottling is still in progress at the CBSC. [Rosabel_Jumilla]
2001-09-11
11:53:41
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064421 AT3AIR Atlanta, GA Verizon Wireles 100% Out, Cum=01:38 Unplanned Outage 09/11 10:49 Note#3 P1: EMX2500 One Call Ext procecessor i
2001-09-11
11:53:42
Skytel
[005051613]
D
ALPHA
s now back in service. Site confirmed call processing is back. Call th
2001-09-11
12:01:41
Skytel
[005051613]
D
ALPHA
Case 064421 AT3AIR Atlanta, GA Verizon Wireles 50% Out, Cum=01:48 Unplanned Outage 09/11 10:59 Note#4 P1: EMX2500 CBSC has been throttled to 50%. BHCA is stable at 125K [Rosabel_Jumilla]
2001-09-11
12:05:56
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064423 CH1ATM Charlotte, NC Alltel No Outage 09/11 11:04 Note#1 P1: EMX2500 This Switch has been Throttled back... Transit tr
2001-09-11
12:05:57
Skytel
[005051613]
D
ALPHA
unks to 75% incoming TG remover 25% in MOB DGLOBL repage has been d
2001-09-11
12:07:42
Skytel
[005051613]
D
ALPHA
(2 of 2) isabled. DAS and DAC Data have bee aborted. The site is @ aprx 170,000 BHCA [Brian_McDonough]
2001-09-11
12:29:42
Skytel
[005051613]
D
ALPHA
NACNRC2@motorola.com||P1 Elmsford System Case 064414 Update: SMS, DAP, MSC, BSC, & System team members online working with MSO. Bridge established @ 847-576-7977. More updates to follow.
2001-09-11
12:40:41
Skytel
[005051613]
D
ALPHA
Case 064419 RD1CTL Raleigh, NC Alltel 100% Out, Cum=02:49 Unplanned Outage 09/11 11:38 Note#5 P1: EMX2500 CBSC-1 is enabled and is comming back in service, also analog service has bee
2001-09-11
12:40:42
Skytel
[005051613]
D
ALPHA
n restored. [Robert_Kosyluk]
2001-09-11
12:41:56
Skytel
[005051613]
D
ALPHA
Case 064419 RD1CTL Raleigh, NC Alltel 60% Out, Cum=02:50 Partially Restored 09/11 11:40 Note#6 P1: EMX2500 Digital test call are being completed on cbsc-1. [Rob
2001-09-11
12:41:57
Skytel
[005051613]
D
ALPHA
ert_Kosyluk]
2001-09-11
12:55:42
Skytel
[005051613]
D
ALPHA
NACNRC2@motorola.com||P1 Hanover DAP Case 064420 Update: CPU has stabilized, PSE will continue to montior the situation, bridge continues.
2001-09-11
13:04:41
Skytel
[005051613]
D
ALPHA
Case 064419 RD1CTL Raleigh, NC Alltel 60% Out, Cum=03:13 Partially Restored 09/11 12:02 Note#7 P1: EMX2500 CBSCs [Robe
2001-09-11
13:04:42
Skytel
[005051613]
D
ALPHA
rt_Kosyluk]
2001-09-11
13:25:11
Skytel
[005051613]
D
ALPHA
(2 of 2) g call processing (20% of the system) [Rosabel_Jumilla]
2001-09-11
13:26:56
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064421 AT3AIR Atlanta, GA Verizon Wireles 50% Out, Cum=03:12 Unplanned Outage 09/11 12:23
2001-09-11
13:26:57
Skytel
[005051613]
D
ALPHA
Note#5 P1: EMX2500 CBSC 3 & 4 is at 50% throttling CBSC 5 is at 25% throttling CBSC 7 is fully operational No analo
2001-09-11
13:36:41
Skytel
[005051613]
D
ALPHA
Case 064419 RD1CTL Raleigh, NC Alltel 30% Out, Cum=03:43 Partially Restored 09/11 12:33 Note#8 P1: EMX2500 CBSC 2 has been enabled and is processing calls. [Robert_Kosyluk]
2001-09-11
14:07:12
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064419 RD1CTL Raleigh, NC Alltel 10% Out, Cum=04:15 Partially Restored 09/11 13:05 Note#9 P1:
2001-09-11
14:07:14
Skytel
[005051613]
D
ALPHA
EMX2500 CBSC is now enabled. There is still a group of sites outs of service, and some land trunking tha
2001-09-11
14:08:41
Skytel
[005051613]
D
ALPHA
(2 of 2) t is oos. [Robert_Kosyluk]
2001-09-11
14:19:56
Skytel
[005051613]
D
ALPHA
Case 064419 RD1CTL Raleigh, NC Alltel 0% Out, Cum=04:28 Fully Restored 09/11 13:18 Note#10 P1: EMX2500 All CBSCs and EMX are 100% restored. [Robert_Kosyluk]
2001-09-11
14:24:41
Skytel
[005051613]
D
ALPHA
(2 of 2) BHCA is stable at 120K [Rosabel_Jumilla]
2001-09-11
14:25:56
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064421 AT3AIR Atlanta, GA Verizon Wireles 30% Out, Cum=04:11 Unplanned Outage 09/11 13:22 Note#6 P1: EMX2500 CBSC 3,4 & 5 is at 25% throttling CBSC 7 is fully operational Analog Service is
2001-09-11
14:25:57
Skytel
[005051613]
D
ALPHA
available at 10%
2001-09-11
15:29:56
Skytel
[005051613]
D
ALPHA
NACNRC2@motorola.com||CRITICAL Elmsford System Case 064414 High CPU Utilization on TDAP's 1 thru 9. SMS, DAP, MSC, BSC, & System team members online working with MSO. Bridge established @ 847-576-7977. Case has been allocated to the System
2001-09-11
15:32:41
Skytel
[005051613]
D
ALPHA
NACNRC2@motorola.com||P1 Hanover DAP Case 064420 Update: System is stable, market wants the case to remain CRITICAL, bridge is ending.
2001-09-11
15:38:41
Skytel
[005051613]
D
ALPHA
Case 064421 AT3AIR Atlanta, GA Verizon Wireles UNKNOWN, Cum=05:25 Unplanned Outage 09/11 14:36 Note#7 P1: EMX2500 Digital is 100% operational Analog is 25% in service BHCA is 150K [Rosabel_Jumilla]
2001-09-11
16:06:41
Skytel
[005051613]
D
ALPHA
Case 064445 LI5ATM Little Rock, AR Alltel No Outage 09/11 15:04 Note#0 P1: EMX2500 Licensed for 175000 CDRs, Currently at 253,000 CDRs Customer is requesting ticket to be opened as P1. [Pamela_Buckl
2001-09-11
16:06:43
Skytel
[005051613]
D
ALPHA
ey]
2001-09-11
16:19:11
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064445 LI5ATM Little Rock, AR Alltel No Outage 09/11 15:16 Note#1 P1: EMX2500 Site is seeng upwards
2001-09-11
16:19:12
Skytel
[005051613]
D
ALPHA
of 250,000 busy hour call attempts.....site staff want to begin throttling procedures.....note
2001-09-11
16:19:56
Skytel
[005051613]
D
ALPHA
(2 of 2) this is different from original title. [Morris_Jones]
2001-09-11
16:56:41
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064445 LI5ATM Little Rock, AR Alltel 0% Out, Cum=00:56 No Outage 09/11 15:54 Note#4 P1: EMX2500
2001-09-11
16:56:43
Skytel
[005051613]
D
ALPHA
Switch currently running at 220,000 Busy Hour Call Attempts (dropping) and is stable.... CPU Usa
2001-09-11
16:57:41
Skytel
[005051613]
D
ALPHA
(2
2001-09-11
16:57:43
Skytel
[005051613]
D
ALPHA
of 2) ge is currently at 83% in the TRANS, CALEXT Site has reduces the number of SIGS on the analog sites... [Morris_Jones]
2001-09-11
17:16:41
Skytel
[005051613]
D
ALPHA
Case 064449 EV3GTE Evansville, IN Verizon Wireles No Outage 09/11 16:14 Note#0 P1: EMX2500 EMX2500 is sheding pages on cellsites. [Harold_Rabin]
2001-09-11
17:23:41
Skytel
[005051613]
D
ALPHA
Case 064421 AT3AIR Atlanta, GA Verizon Wireles 0% Out, Cum=07:11 Fully Restored 09/11 16:21 Note#8 P1: EMX2500 System is fully restored. CPU Usage and BHCA are within tolerance. [Rosabel_Jumilla]
2001-09-11
17:31:41
Skytel
[005051613]
D
ALPHA
(1 of 2) Case 064445 LI5ATM Little Rock, AR Alltel
2001-09-11
17:31:43
Skytel
[005051613]
D
ALPHA
0% Out, Cum=01:32 No Outage 09/11 16:30 Note#5 P1: EMX2500 Busy Hour Call Attempts have dropped down to 194,000......customer has started to bring in SIGS.
2001-09-11
17:33:41
Skytel
[005051613]
D
ALPHA
(2 of 2) .... Customer has declared he is comfortable with the state of his system...bridge has been dropped [Morris_Jones]
2001-09-11
21:15:41
Skytel
[005051613]
D
ALPHA
NACNRC2@motorola.com||P1 Elmsford S
2001-09-11
21:15:43
Skytel
[005051613]
D
ALPHA
ystem Case 064414 Update: BSC/PSE is mornitoring GPROC utilization every 20 minutes. Bridge 847-538-1299 remain open for monitoring.
2001-09-11
22:34:41
Skytel
[005051613]
D
ALPHA
NACNRC2@motorola.com||P1 Elmsford System Case 064414 Update: nysys: all GPROCs' utilization are under 42%; nysys4: all GPROCs' utilization are under 47%; ffsys: all GPROCs' utilization are under 47%; gcsys: GPROCs' utilization are und