9/11 Pager Messages
2001-09-11
05:32:19
Arch
[0805210]
C
ALPHA
|P2 NOTICE #4243326 |Etisalat link Dubai to Cairo failed impacting users WAN connectivity. Etisalat contacted Cairo PTT. OUTAGE START: 11 Sep 10:21 CET NO ETU. walho x4112 |172
2001-09-11
05:38:38
Arch
[0805210]
C
ALPHA
|P2 NOTICE #4243071 |Concert link from Madrid to Lisbon flapping. No user impact due to ISDN backup. EMEA WAN opened ticket with Concert. OUTAGE START: 11 Sep 7:30 CET NO ETU. walho x4112 |188
2001-09-11
06:25:09
Arch
[0805210]
C
ALPHA
|NETWORK-P1 NOTICE #4243506 |Mexicocity-gw is down impacting mexico city feild sales office call managers. Minimum business impact due to time of day. Support is investigating. OUTAGE START: 11 Sep 03:14 PT NO ETU. Ops 408-527-0007 |232
2001-09-11
06:54:30
Arch
[0805210]
C
ALPHA
|NETWORK-P1 UPDATE #4243506 |Mexicocity-gw is down impacting FSO call managers. No impact due to time of day. Support investigating power outage. Next update at 05:30 PT. OUTAGE START: 11 Sep 03:14 PT NO ETU. Ops 408-527-0007 |226
2001-09-11
07:29:07
Arch
[0805210]
C
ALPHA
|NETWORK-P1 CL
2001-09-11
07:29:09
Arch
[0805210]
C
ALPHA
OSE PENDING #4243506 |Mexicocity-gw and FSO call managers are up. No impact due to time of day. Service was restored once power came up. OUTAGE START: 11 Sep 03:14 PT END: 11 Sep 04:25 PT Ops 408-527-0007 |219
2001-09-11
07:33:30
Arch
[0805210]
C
ALPHA
|P2 CLOSE #4228756 |Sita link Amsterdam to Prague no longer dropping packets. EMEA WAN restricted bandwith. OUTAGE START: 6 Sep 8:47 CET END: 7 Sep 12:53 CET walho x4112 |170
2001-09-11
07:55:00
Arch
[0805210]
C
ALPHA
|P2 CLOSE #4239098 | Http wwwin-emea-prod1:80 has been restore. Support investigating. OUTAGE START: 10 Sep 14:05 CET END: 11 Sep 11:00 CET josvenss x4112 |155
2001-09-11
08:16:07
Arch
[0805210]
C
ALPHA
|P2 NOTICE #4243602 |CAP/JRun on wwwin-apps-1 and 2 unavailable, impacting servlet code execution for the CAP application. Support investigating. OUTAGE START: 10 Sep 23:01 PT END: Ops 408-527-0007 |198
2001-09-11
08:22:54
Arch
[0805210]
C
ALPHA
|P2 NOTICE #4243626 |Lima, Peru circuit down, impacting network connectivity for Lima Sales office. WAN support investigating with MCI. OUTAGE START: 11 Sep 00:46 PT NO ETU. Ops 408-527-0007 |191
2001-09-11
08:37:14
Arch
[0805210]
C
ALPHA
|P2 CLOSE #4243626 |Lima, Peru circuit back up. Network connectivity available to Lima Sales office. Circuit restored by MCI. OUTAGE START: 11 Sep 00:46 PT END: 11 Sep 05:34 PT Ops 408-527-0007 |194
2001-09-11
09:06:01
Arch
[0805210]
C
ALPHA
|P2 CLOSE #4243602 |CAP/JRun on wwwin-apps-1&2 available
2001-09-11
09:06:03
Arch
[0805210]
C
ALPHA
. Servlet code execution available for CAP application. Support restarted application to restore service. OUTAGE START: 10 Sep 23:01 PT END: 11 Sep 05:55 PT Ops 408-527-0007 |230
2001-09-11
09:26:55
Arch
[0805210]
C
ALPHA
|P2 CLOSE #4224767 |BVRUMXPD BVABTPD WEBIPD BVCECPD BVWPRPD BVPOSPD BVSSRPPD databases on Minerva back up. EMEA DBAdmins restored the services after cold backup. OUTAGE START: 5 Sep 11:45 CET
2001-09-11
09:26:57
Arch
[0805210]
C
ALPHA
END: 6 Sep 2:45 CET walho x4112 |224
2001-09-11
09:33:55
Arch
[0805210]
C
ALPHA
|CCO-P1 NOTICE #4243775 |sendmail down on cco-sj-1. Minimal impact due to load b
2001-09-11
09:33:57
Arch
[0805210]
C
ALPHA
alancing on cco-sj-2 and -3. Support investigating. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187
2001-09-11
09:37:08
Arch
[0805210]
C
ALPHA
|P2 NOTICE #4243796 |sendmail down on sj-msg-core-4. Minimal impact due to redundancy. Contacting support. OUTAGE START: 11 Sep 05:47 PT NO ETU. Ops 408.527.0007 |162
2001-09-11
09:37:50
Arch
[0805210]
C
ALPHA
amitra|bridge-erp: Please call bridge 408.527.2663 id700071 re: SJPROD db going down. Thanks, Al/Ops
2001-09-11
09:42:18
Arch
[0805210]
C
ALPHA
|ERP-P1 NOTICE #4243807 |SJPROD down. ERP production db unable to accept new connections and impacting access to Status Agent. Support escalation in progress.
2001-09-11
09:42:19
Arch
[0805210]
C
ALPHA
OUTAGE START: 11 Sep 6:33 PT NO ETU. Ops 408.527.0007 |213
2001-09-11
09:54:55
Arch
[0805210]
C
ALPHA
dcorso|Need ya on Bridge for ERP outage. SJPROD Down. x72663 id 700071
2001-09-11
10:09:36
Arch
[0805210]
C
ALPHA
|ERP-P1 UPDATE/CLOSE PENDING #4243807 |SJPROD db instance on host high was hung and is now up. ERP production db is now able to accept new connections. Status Agent is also available. ERP_dba's restarted instance to restore service. -1/2
2001-09-11
10:09:38
Arch
[0805210]
C
ALPHA
OUTAGE START: 11 Sep 6:33 PT END: 11 Sep 7:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
10:10:04
Arch
[0805210]
C
ALPHA
|CCO-P1 UPDATE #4243775 |cco-sj-1 not responding. Support removing host from service to allow apps availibility on cco-sj-2 and -3. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 40
2001-09-11
10:10:06
Arch
[0805210]
C
ALPHA
8.527.0007 |187
2001-09-11
11:30:13
Arch
[0805210]
C
ALPHA
|CCO-P1 CLOSE PENDING #4243775 |cco-sj-1 has been brought back in to service. Support continues to investigate cause of outage. OUTAGE START: 11 Sep 06:19 PT
2001-09-11
11:30:15
Arch
[0805210]
C
ALPHA
END: 11 Sep 08:00 PT Ops 408.527.0007 |196
2001-09-11
11:46:58
Arch
[0805210]
C
ALPHA
|P2 NOTICE/CLOSE #4244027 |Epage services were unavailable due to high load on all epage servers. OUT
2001-09-11
11:47:00
Arch
[0805210]
C
ALPHA
AGE START: 11 Sep 8:10 PT END: 11 Sep 8:32 PT Ops 408.527.0007 |164
2001-09-11
11:47:07
Arch
[0805210]
C
ALPHA
|VOICE-P1 NOTICE #4244044 |No calls able to go through to EMEA due to high load on national and international circuits. Voiceservices investigating. OUTAGE START: 11 Sep 08:31 PT NO ETU. O
2001-09-11
11:47:09
Arch
[0805210]
C
ALPHA
ps 4085270007 |202
2001-09-11
12:06:28
Arch
[0805210]
C
ALPHA
|POWER S
2001-09-11
12:06:30
Arch
[0805210]
C
ALPHA
TATUS : No stage alerts at this time ||46
2001-09-11
12:47:35
Arch
[0805210]
C
ALPHA
|P2 NOTICE #4244245 |Network link to Bogota, Colombia down impacting access to local FSO. Contacting support. OUTAGE START: 11 Sep 9:32 PT NO ETU. Ops 408.527.0007 |164
2001-09-11
13:18:59
Arch
[0805210]
C
ALPHA
50-THE WORKSHOP TODAY HAS BEEN CANCELED. WE WILL HAVE TOMORROW SESSION. PLS CONTACT ME W/ ANY...
2001-09-11
13:19:01
Arch
[0805210]
C
ALPHA
2...QUESTIONS. SHERRY 79442
2001-09-11
13:27:43
Arch
[0805210]
C
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls to EMEA intermittent due to high load on national and international circuits. Voiceservices continuing to investigate. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops
2001-09-11
13:27:44
Arch
[0805210]
C
ALPHA
4085270007 |205
2001-09-11
14:20:59
Arch
[0805210]
C
ALPHA
|NETWORK-P1 NOTICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermit
2001-09-11
14:21:00
Arch
[0805210]
C
ALPHA
tent network availability. SJ LAN support investigating issues with Code Red filters being removed. OUTAGE START: 11 Sep -1/2
2001-09-11
14:21:01
Arch
[0805210]
C
ALPHA
11:05 PT NO ETU. Ops 408.527.0007 |268 -2/2
2001-09-11
14:24:37
Arch
[0805210]
C
ALPHA
|VOICE-P1 NOTICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network and IP phone availability. SJ LAN support investigating issues with Code Red
2001-09-11
14:24:38
Arch
[0805210]
C
ALPHA
filters. OUTAGE START: 11 Sep -1/2
2001-09-11
14:24:40
Arch
[0805210]
C
ALPHA
11:05 PT NO ETU. Ops 408.527.0007 |265 -2/2
2001-09-11
14:31:12
Arch
[0805210]
C
ALPHA
|P1 NOTICE #4244617 |CCEWP unavailable on ladybird impacting content delopyment to CCO. Support is investigating. OUTAGE START: 11 Sep 11:21 PT NO ETU. Ops 408.527.0007 |169
2001-09-11
14:32:44
Arch
[0805210]
C
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls between US and EMEA intermittent due to high load on national and international circuits. Voice Services investigating. Next update at 16:00 PT. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 408.527.0007 |233
2001-09-11
14:37:46
Arch
[0805210]
C
ALPHA
mschore|All is okay here except for the obvious anguish and anger over events. Will stay online.
2001-09-11
14:40:46
Arch
[0805210]
C
ALPHA
|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |217
2001-09-11
14:41:05
Arch
[0805210]
C
ALPHA
|VOICE-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
14:43:40
Arch
[0805210]
C
ALPHA
usdesai|Hi John: I am working from home. Spoke to Darlene and we decided to stay home. My family is fine. My husband is working in New Jersey, I spoke to him and he is ok. Thank you,
2001-09-11
14:43:42
Arch
[0805210]
C
ALPHA
Usha
2001-09-11
15:11:19
Arch
[0805210]
C
ALPHA
|P1 CLOSE PENDING #4244617 |CCEWP available on ladybird. Clients can deploy content to CCO. Sup
2001-09-11
15:11:21
Arch
[0805210]
C
ALPHA
port restarted applic to restore service. OUTAGE START: 11 Sep 11:21 PT END: 11 Sep 12:05 PT Ops 408.527.0007 |205
2001-09-11
15:17:14
Arch
[0805210]
C
ALPHA
|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services
2001-09-11
15:17:16
Arch
[0805210]
C
ALPHA
stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |226
2001-09-11
15:18:01
Arch
[0805210]
C
ALPHA
|VOICE-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |224
2001-09-11
15:30:47
Arch
[0805210]
C
ALPHA
pburleig|I am leaving office, but within 1 hour will be fully available via pager, e-mail, or phon
2001-09-11
15:30:49
Arch
[0805210]
C
ALPHA
e mail. thx. Pat
2001-09-11
15:34:53
Arch
[0805210]
C
ALPHA
|NETWORK-P1 CLOSE PENDING #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.
2001-09-11
15:34:55
Arch
[0805210]
C
ALPHA
527.0007 |233
2001-09-11
15:35:49
Arch
[0805210]
C
ALPHA
|VOICE-P1 CLOSE PENDING #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded c
2001-09-11
15:35:51
Arch
[0805210]
C
ALPHA
ode on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |231
2001-09-11
15:42:04
Arch
[0805210]
C
ALPHA
|P1 NOTICE #4244953 |sjc-i2prod has system resource issues impacting access to Demand Planning tool to forecast product demand. Support rebooting host. OUTAGE START: 11 Sep 12:32 PT ETU: 11 Sep 14:00 PT Ops 408.527.0007 |220
2001-09-11
15:45:04
Arch
[0805210]
C
ALPHA
|NETWORK-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408.527.0007 |207
2001-09-11
15:45:47
Arch
[0805210]
C
ALPHA
|VOICE-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408.527.0007 |205
2001-09-11
15:59:02
Arch
[0805210]
C
ALPHA
|VOICE-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. O
2001-09-11
15:59:04
Arch
[0805210]
C
ALPHA
UTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2
2001-09-11
15:59:06
Arch
[0805210]
C
ALPHA
527.0007 |242 -2/2
2001-09-11
16:02:36
Arch
[0805210]
C
ALPHA
|NETWORK-P1 UPDATE #4245013 |SJ Site 4 Ph
2001-09-11
16:02:38
Arch
[0805210]
C
ALPHA
ase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2
2001-09-11
16:02:49
Arch
[0805210]
C
ALPHA
527.0007 |244 -2/2
2001-09-11
16:16:47
Arch
[0805210]
C
ALPHA
|NETWORK-P1 CLOSE PENDING
2001-09-11
16:16:49
Arch
[0805210]
C
ALPHA
#4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all Code Red filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2
2001-09-11
16:16:51
Arch
[0805210]
C
ALPHA
13:10 PT Ops 408.527.0007 |260 -2/2
2001-09-11
16:18:06
Arch
[0805210]
C
ALPHA
|VOICE-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all Code Red
2001-09-11
16:18:08
Arch
[0805210]
C
ALPHA
filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2
2001-09-11
16:18:10
Arch
[0805210]
C
ALPHA
13:10 PT Ops 408.527.0007 |258 -2/2
2001-09-11
17:13:15
Arch
[0805210]
C
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod still unavailable impacting access to Demand Planning tool to forecast product demand. Support continues to investigate. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |208
2001-09-11
18:20:35
Arch
[0805210]
C
ALPHA
amitra|John, please call Ops 408.527.0007 re: ODSPROD. Thanks, Al/Ops
2001-09-11
18:43:22
Arch
[0805210]
C
ALPHA
es support monitoring -1/2
2001-09-11
18:43:24
Arch
[0805210]
C
ALPHA
OUTAGE START: 11 Sep 08:31 PT ETU: 12 Sep 12:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
18:47:59
Arch
[0805210]
C
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support working to fix hive corruption to restore services to host and application. OUTAGE START: 11 Sep 12:32 PT NO ETU. djaramil 408.527.0007 |220
2001-09-11
20:20:29
Arch
[0805210]
C
ALPHA
|P1 NOTICE #4245646 |outback-ams and outback-ams1 down impacting availability of NATKIT - NSA's customer network troubleshooting application - in EMEA. Support investigating. OUTAGE START: 11 Sep 17:09 PT NO ETU. Ops 408-527-0007 |230
2001-09-11
20:50:16
Arch
[0805210]
C
ALPHA
Christopher Cherwin|Emergency Change Request update 62866 |John - FYI - deployed another ECR for Joe Mastropolo. |113
2001-09-11
21:34:42
Arch
[0805210]
C
ALPHA
|P1 UPDATE/CLOSE PENDING #4245646 |NATKIT, NSA's customer network troubleshooting application in EMEA is available. Support rebooted servers to restore service. OUTAGE START: 11 Sep 17:09 PT END: 11 Sep 18:28 PT Ops 408-527-0007 |229
2001-09-11
21:39:15
Arch
[0805210]
C
ALPHA
51-HEY HON, GIVE ME A CALL AT HOME WHEN YOU GET A CHANCE.
2001-09-11
22:19:26
Arch
[0805210]
C
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support attempting to bring dev server into production to rep
2001-09-11
22:19:28
Arch
[0805210]
C
ALPHA
lace corrupted server. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
23:33:45
Arch
[0805210]
C
ALPHA
|P1 CLOSE PENDING #4244953 |sjc-i2prod and Demand Planning tool are up. Support replaced corrupted server with development server to restore service. OUTAGE START: 11 Sep 12:32 PT END: 11 Sep 20:31 PT Ops 408.527.0007 |218
2001-09-11
23:45:05
Arch
[0805210]
C
ALPHA
52-GOOD NIGHT, SWEETHEART. SLEEP WELL. XOXO