9/11 Pager Messages

2001-09-11
06:24:38
Arch
[1050350]
D
ALPHA
|NETWORK-P1 NOTICE #4243506 |Mexicocity-gw is down impacting mexico city feild sales office call managers. Minimum business impact due to time of day. Support is investigating. OUTAGE START: 11 Sep 03:14 PT NO ETU. Ops 408-527-0007 |232
2001-09-11
06:53:21
Arch
[1050350]
D
ALPHA
|NETWORK-P1 UPDATE #4243506 |Mexicocity-gw is down impacting FSO call managers. No impact due to time of day. Support investigating power outage. Next update at 05:30 PT. OUTAGE START: 11 Sep 03:14 PT NO ETU. Ops 408-527-0007 |226
2001-09-11
07:29:07
Arch
[1050350]
D
ALPHA
|NETWORK-P1 CLOSE PENDING #4243506 |Mexicocity-gw and FSO call managers are up. No impact due to time of day. Service was restored once power came up. OUTAGE START: 11 Sep 03:14 PT END: 11 Sep 04:25 PT Ops 408-527-0007 |219
2001-09-11
09:33:14
Arch
[1050350]
D
ALPHA
|CCO-P1 NOTICE #4243775 |sendmail down on cco-sj-1. Minimal impact due to load balancing on cco-sj-2 and -3. Support investigating. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187
2001-09-11
09:41:04
Arch
[1050350]
D
ALPHA
|ERP-P1 NOTICE #4243807 |SJPROD down. ERP production db unable to accept new connections and impacting access to Status Agent. Support escalation in progress. OUTAGE START: 11 Sep 6:33 PT
2001-09-11
09:41:06
Arch
[1050350]
D
ALPHA
NO ETU. Ops 408.527.0007 |213
2001-09-11
09:41:53
Arch
[1050350]
D
ALPHA
rloura|If you haven't seen it yet, two planes crashed into the two WTC towers at 9am this morning.
2001-09-11
09:42:18
Arch
[1050350]
D
ALPHA
rloura|If you haven't seen it yet, two planes crashed into the two WTC towers at 9am this morning.
2001-09-11
10:08:40
Arch
[1050350]
D
ALPHA
|ERP-P1 UPDATE/CLOSE PENDING #4243807 |SJPROD db instance on host high was hung and is now up. ERP production db is now able to accept new connections. Status Agent is also available. ERP_dba's restarted instance
2001-09-11
10:08:42
Arch
[1050350]
D
ALPHA
to restore service. -1/2
2001-09-11
10:08:49
Arch
[1050350]
D
ALPHA
OUTAGE START: 11 Sep 6:33 PT END: 11 Sep 7:00 PT Ops 408.527.0
2001-09-11
10:08:51
Arch
[1050350]
D
ALPHA
007 |299 -2/2
2001-09-11
10:08:53
Arch
[1050350]
D
ALPHA
|CCO-P1
2001-09-11
10:08:55
Arch
[1050350]
D
ALPHA
UPDATE #4243775 |cco-sj-1 not responding. Support removing host from service to allow apps availibility on cco-sj-2 and -3. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187
2001-09-11
11:03:42
Arch
[1050350]
D
ALPHA
amitra|John Bruno's
2001-09-11
11:03:43
Arch
[1050350]
D
ALPHA
Direct Reports: Per Lance Perry, please call bridge 408.527.2663 id id700071 immediately. Thank you, Al/Ops
2001-09-11
11:29:36
Arch
[1050350]
D
ALPHA
|CCO-P1 CLOSE PENDING #4243775 |cco-sj-1
2001-09-11
11:29:38
Arch
[1050350]
D
ALPHA
has been brought back in to service. Support continues to investigate cause of outage. OUTAGE START: 11 Sep 06:19 PT END: 11 Sep 08:00 PT Ops 408.527.0007 |196
2001-09-11
11:46:19
Arch
[1050350]
D
ALPHA
|VOICE-P1 NOTICE #42
2001-09-11
11:46:21
Arch
[1050350]
D
ALPHA
44044 |No calls able to go through to EMEA due to high load on national and international circuits. Voiceservices investigating. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 4085270007 |202
2001-09-11
11:47:09
Arch
[1050350]
D
ALPHA
mharmon|For Pat, she's going to office to pick up computer & then going back home. RTP is on voluntary evacuation.
2001-09-11
11:59:00
Arch
[1050350]
D
ALPHA
"Keith Koo"|The Sr Staff mtg for today has been cancelled |Team, Due to today's events. Lanc
2001-09-11
11:59:02
Arch
[1050350]
D
ALPHA
e's Sr Staff meeting has been cancelled.|132
2001-09-11
12:06:30
Arch
[1050350]
D
ALPHA
|POWER STATUS : No stage alerts at this time ||46
2001-09-11
13:27:00
Arch
[1050350]
D
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls to EMEA intermittent due to high load on national and international circuits. Voiceservices continuing to investigate. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 4085270007 |205
2001-09-11
14:20:33
Arch
[1050350]
D
ALPHA
|NETWORK-P1 NOTICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network availability. SJ LAN support investigating issues with Code Red filters being removed. OUTAGE START: 11 Sep -1/2
2001-09-11
14:20:35
Arch
[1050350]
D
ALPHA
11:05 PT NO ETU. Ops 408.527.0007 |268 -2/2
2001-09-11
14:23:52
Arch
[1050350]
D
ALPHA
|VOICE-P1 NOTICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network and IP phone availability. SJ LAN support investigating issues with Code Red filters. OUTAGE START: 11 Sep -1/2
2001-09-11
14:23:54
Arch
[1050350]
D
ALPHA
11:05 PT NO ETU. Ops 408.527.0007 |265 -2/2
2001-09-11
14:31:07
Arch
[1050350]
D
ALPHA
|P1 NOTICE #4244617 |CCEWP unavailable on ladybird impacting content delopyment to CCO. Support is investigating. OUTAGE START: 11 Sep 11:21 PT NO ETU. Ops 408.527.0007 |169
2001-09-11
14:32:03
Arch
[1050350]
D
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls between US and EMEA intermittent due to high load on national and international circuits. Voice Services investigating. Next update at 16:00 PT. OUTAGE STA
2001-09-11
14:32:05
Arch
[1050350]
D
ALPHA
RT: 11 Sep 08:31 PT NO ETU. Ops 408.527.0007 |233
2001-09-11
14:40:05
Arch
[1050350]
D
ALPHA
|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |217
2001-09-11
14:40:23
Arch
[1050350]
D
ALPHA
|VOICE-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007
2001-09-11
14:40:25
Arch
[1050350]
D
ALPHA
|215
2001-09-11
15:03:48
Arch
[1050350]
D
ALPHA
cchurchi|I have sent email and vmail to you with additional netapp info. Chuck
2001-09-11
15:09:34
Arch
[1050350]
D
ALPHA
lperry|JB, things are under control here, most employees offsite. I may leave shortly myself. Call if you need anything. Lance
2001-09-11
15:11:08
Arch
[1050350]
D
ALPHA
|P1 CLOSE PENDING #4244617 |CCEWP available on ladybird. Clients can deploy content to CCO. Support restarted applic to restore service. OUTAGE START: 11 Sep 11:21 PT END: 11 Sep 12:05 PT Ops 408.527.0007 |205
2001-09-11
15:16:48
Arch
[1050350]
D
ALPHA
|VOICE-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |22
2001-09-11
15:16:53
Arch
[1050350]
D
ALPHA
4
2001-09-11
15:16:59
Arch
[1050350]
D
ALPHA
|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:
2001-09-11
15:17:01
Arch
[1050350]
D
ALPHA
30 PT Ops 408.527.0007 |226
2001-09-11
15:30:47
Arch
[1050350]
D
ALPHA
pburleig|I am leaving office, but within 1 hour will be fully available via pager, e-mail, or phone mail. thx. Pat
2001-09-11
15:34:44
Arch
[1050350]
D
ALPHA
|NETWORK-P1 CLOSE PENDING #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |233
2001-09-11
15:34:59
Arch
[1050350]
D
ALPHA
|VOICE-P1 CLOSE PENDING #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |231
2001-09-11
15:41:53
Arch
[1050350]
D
ALPHA
|P1 NOTICE #42
2001-09-11
15:41:55
Arch
[1050350]
D
ALPHA
44953 |sjc-i2prod has system resource issues impacting access to Demand Planning tool to forecast product demand. Support rebooting host. OUTAGE START: 11 Sep 12:32 PT ETU: 11 Sep 14:00 PT Ops 408.527.0007 |220
2001-09-11
15:44:19
Arch
[1050350]
D
ALPHA
|NETWORK-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO
2001-09-11
15:44:21
Arch
[1050350]
D
ALPHA
ETU. Ops 408.527.0007 |207
2001-09-11
15:45:04
Arch
[1050350]
D
ALPHA
|VOICE-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408.527.0007 |205
2001-09-11
15:58:16
Arch
[1050350]
D
ALPHA
|VOICE-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degrad
2001-09-11
15:58:17
Arch
[1050350]
D
ALPHA
ation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2
2001-09-11
15:58:19
Arch
[1050350]
D
ALPHA
527.0007 |242 -2/2
2001-09-11
16:01:23
Arch
[1050350]
D
ALPHA
|NETWORK-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2
2001-09-11
16:01:25
Arch
[1050350]
D
ALPHA
527.0007 |244 -2/2
2001-09-11
16:15:51
Arch
[1050350]
D
ALPHA
|NETWORK-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice servic
2001-09-11
16:15:53
Arch
[1050350]
D
ALPHA
es stable and restored. Support removed all Code Red filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2
2001-09-11
16:16:08
Arch
[1050350]
D
ALPHA
13:10 PT Ops 408.527.0007 |260 -2/2
2001-09-11
16:17:08
Arch
[1050350]
D
ALPHA
|VOICE-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all Code Red filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2
2001-09-11
16:17:13
Arch
[1050350]
D
ALPHA
13:10 PT Ops 408.527.0007 |258 -2/2
2001-09-11
17:12:53
Arch
[1050350]
D
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod still unavailable impacting access to Demand Planning too
2001-09-11
17:12:54
Arch
[1050350]
D
ALPHA
l to forecast product demand. Support continues to investigate. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |208
2001-09-11
18:42:01
Arch
[1050350]
D
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls between US and EMEA intermittent due to high load on
2001-09-11
18:42:03
Arch
[1050350]
D
ALPHA
national and international circuits. MCI and Sprint expect phone lines to become stable by tomorrow afternoon. Voice Services support monitoring -1/2
2001-09-11
18:42:05
Arch
[1050350]
D
ALPHA
OUTAGE START: 11 Sep 08:31 PT ETU: 12 Sep 12:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
18:47:39
Arch
[1050350]
D
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unava
2001-09-11
18:47:41
Arch
[1050350]
D
ALPHA
ilable. Support working to fix hive corruption to restore services to host and application. OUTAGE START: 11 Sep 12:32 PT NO ETU. djaramil 408.527.0007 |220
2001-09-11
20:20:12
Arch
[1050350]
D
ALPHA
|P1 NOTICE #4245646 |outback-ams and outback-ams1 down impacting availability of NATKIT - NSA's customer network troubleshooting application - in EMEA. Support investigating. OUTAGE START: 11 Sep 17:09 PT NO ETU. Ops 408-527-0007 |230
2001-09-11
21:34:38
Arch
[1050350]
D
ALPHA
|P1 UPDATE/CLOSE PENDING #4245646 |NATKIT, NSA's customer network troubleshooting application in EMEA is available. Support rebooted servers to re
2001-09-11
21:34:40
Arch
[1050350]
D
ALPHA
store service. OUTAGE START: 11 Sep 17:09 PT END: 11 Sep 18:28 PT Ops 408-527-0007 |229
2001-09-11
22:19:26
Arch
[1050350]
D
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support attempting to bring dev server into production to replace corrupted server. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
23:33:11
Arch
[1050350]
D
ALPHA
|P1 CLOSE PENDING #4244953 |sjc-i2prod and Demand Planning tool are up. Support replaced corrupted server with development server to restore service. OU
2001-09-11
23:33:13
Arch
[1050350]
D
ALPHA
TAGE START: 11 Sep 12:32 PT END: 11 Sep 20:31 PT Ops 408.527.0007 |218