9/11 Pager Messages

2001-09-11
05:32:19
Arch
[1749363]
A
ALPHA
|P2 NOTICE #4243326 |Etisalat link Dubai to Cairo failed impacting users WAN connectivity. Etisalat contacted Cairo PTT. OUTAGE START: 11 Sep 10:21 CET NO ETU. walho x4112 |172
2001-09-11
05:38:38
Arch
[1749363]
A
ALPHA
|P2 NOTICE #4243071 |Concert link from Madrid to Lisbon flapping. No user impact due to ISDN backup. EMEA WAN opened ticket with Concert. OUTAGE START: 11 Sep 7:30 CET NO ETU. walho x4112 |188
2001-09-11
06:24:37
Arch
[1749363]
A
ALPHA
|NETWORK-P1 NOTICE #4243506 |Mexicocity-gw is down impacting mexico city feild sales office call managers. Minimum business impact due to time of day. Support is investigating. OUTAGE START: 11 Sep 03:14 PT NO ETU. Ops 408-527-0007 |232
2001-09-11
06:53:20
Arch
[1749363]
A
ALPHA
|NETWORK-P1 UPDATE #4243506 |Mexicocity-gw is down impacting FSO call managers. No impact due to time of day. Support investigating power outage. Next update at 05:30 PT. OUTAGE START: 11 Sep 03:14 PT NO ETU. Ops 408-527-0007 |226
2001-09-11
07:29:07
Arch
[1749363]
A
ALPHA
|NETWORK-P1 CLOSE PENDING #4243506 |Mexicocity-gw and FSO call managers are up. No impact due to time of day. Service was restored once power came up. OUTAGE START: 11 Sep 03:14 PT END: 11 Sep 04:25 PT Ops 408-527-0007 |219
2001-09-11
07:33:30
Arch
[1749363]
A
ALPHA
|P2 CLOSE #4228756 |Sita link Amsterdam to Prague no longer dropping packets. EMEA WAN restricted bandwith. OUTAGE START: 6 Sep 8:47 CET END: 7 Sep 12:53 CET walho x4112 |170
2001-09-11
07:54:59
Arch
[1749363]
A
ALPHA
|P2 CLOSE #4239098 | Http wwwin-emea-prod1:80 has been restore. Support investigating. OUTAGE START: 10 Sep 14:05 CET END: 11 Sep 11:00 CET josvenss x4112 |155
2001-09-11
08:16:07
Arch
[1749363]
A
ALPHA
|P2 NOTICE #4243602 |CAP/JRun on wwwin-apps-1 and 2 unavailable, impacting servlet code execution for the CAP application. Support investigating. OUTAGE START: 10 Sep 23:01 PT END: Ops 408-527-0007 |198
2001-09-11
08:22:54
Arch
[1749363]
A
ALPHA
|P2 NOTICE #4243626 |Lima, Peru circuit down, impacting network connectivity for Lima Sales office. WAN support investigating with MCI. OUTAGE START: 11 Sep 00:46 PT NO ETU. Ops 408-527-0007 |191
2001-09-11
08:37:14
Arch
[1749363]
A
ALPHA
|P2 CLOSE #4243626 |Lima, Peru circuit back up. Network connectivity available to Lima Sales office. Circuit restored by MCI. OUTAGE START: 11 Sep 00:46 PT END: 11 Sep 05:34 PT Ops 408-527-0007 |194
2001-09-11
09:06:01
Arch
[1749363]
A
ALPHA
|P2 CLOSE #4243602 |CAP/JRun on wwwin-apps-1&2 available. Servlet code execut
2001-09-11
09:06:03
Arch
[1749363]
A
ALPHA
ion available for CAP application. Support restarted application to restore service. OUTAGE START: 10 Sep 23:01 PT END: 11 Sep 05:55 PT Ops 408-527-0007 |230
2001-09-11
09:26:55
Arch
[1749363]
A
ALPHA
|P2 CLOSE #4224767 |BVRUMXPD BVABTPD WEBIPD BVCECPD BVWPRPD BVPOSPD BVSSRPPD databases on Minerva back up. EMEA DBAdmins restored the services after cold backup. OUTAGE START: 5 Sep 11:45 CET END: 6 Sep 2:45 CET
2001-09-11
09:26:57
Arch
[1749363]
A
ALPHA
walho x4112 |224
2001-09-11
09:33:05
Arch
[1749363]
A
ALPHA
|CCO-P1 NOTICE #4243775 |sendmail down on cco-sj-1. Minimal impact due to load balancing on cco-sj-2 and -3. Support investigating. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187
2001-09-11
09:37:08
Arch
[1749363]
A
ALPHA
|P2 NOTICE #4243796 |sendmail down on sj-msg-core-4. Minimal impact due to redundancy. Contacting support. OUTAGE START: 11 Sep 05:47 PT NO ETU. Ops 408.527.0007 |162
2001-09-11
09:40:23
Arch
[1749363]
A
ALPHA
|ERP-P1 NOTICE #4243807 |SJPROD down. ERP production db unable to accept new connections and impacting access to Status Agent. Support escalation in progre
2001-09-11
09:40:25
Arch
[1749363]
A
ALPHA
ss. OUTAGE START: 11 Sep 6:33 PT NO ETU. Ops 408.527.0007 |213
2001-09-11
10:08:14
Arch
[1749363]
A
ALPHA
|ERP-P1 UPDATE/CLOSE PENDING #4243807 |SJPROD db instance on host high was hung and is no
2001-09-11
10:08:16
Arch
[1749363]
A
ALPHA
w up. ERP production db is now able to accept new connections. Status Agent is also available. ERP_dba's restarted instance to restore service. -1/2
2001-09-11
10:08:31
Arch
[1749363]
A
ALPHA
OUTAGE START: 11 Sep 6:33 PT END: 11 Sep 7:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
10:08:38
Arch
[1749363]
A
ALPHA
|CCO-P1 UPDATE #4243775 |cco-sj-1 not responding. Support removing host from service to allow apps availibility on cco-sj-2 and -3. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.
2001-09-11
10:08:40
Arch
[1749363]
A
ALPHA
0007 |187
2001-09-11
11:29:08
Arch
[1749363]
A
ALPHA
|CCO-P1 CLOSE PENDING #4243775 |cco-sj-1 has been brought back in to service. Support continues to investigate cause of outage. OUTAGE START: 11 Sep 06:19 PT END: 11 Sep 08:00 PT Ops 408.527.0007 |196
2001-09-11
11:46:38
Arch
[1749363]
A
ALPHA
|P2 NOTICE/CLOSE #4244027 |Epage services were unavailable due to high load on all epa
2001-09-11
11:46:39
Arch
[1749363]
A
ALPHA
ge servers. OUTAGE START: 11 Sep 8:10 PT END: 11 Sep 8:32 PT Ops 408.527.0007 |164
2001-09-11
11:58:43
Arch
[1749363]
A
ALPHA
"Keith Koo"|The Sr Staff mtg for today has been cancelled |Team, Due to today's events. Lance's Sr Staff meeting has been cancelled.|132
2001-09-11
12:06:04
Arch
[1749363]
A
ALPHA
|POWE
2001-09-11
12:06:06
Arch
[1749363]
A
ALPHA
R STATUS : No stage alerts at this time ||46
2001-09-11
12:47:18
Arch
[1749363]
A
ALPHA
|P2 NOTICE #4244245 |Network link to Bogota, Colombia down impacting access to local FSO. Contacting support. OUTAGE START: 11 Sep 9:32 PT NO ETU. Ops 408.527.0007 |164
2001-09-11
14:19:54
Arch
[1749363]
A
ALPHA
|NETWORK-P1 NOTICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network availability. SJ LAN support investigating issues with Code Red filters being removed. OUTAGE START: 11 Sep -1/2
2001-09-11
14:19:55
Arch
[1749363]
A
ALPHA
11:05 PT NO ETU. Ops 408.527.0007 |268 -2/2
2001-09-11
14:30:20
Arch
[1749363]
A
ALPHA
|P1 NOTICE #4244617 |CCEWP unavailable on ladybird impacting content delopyment to CCO. Support is investigating. OUTAGE START: 11 Sep 11:21 PT NO ETU. Ops 408.527.0007 |169
2001-09-11
14:39:22
Arch
[1749363]
A
ALPHA
|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilizing. SJ LAN supp
2001-09-11
14:39:23
Arch
[1749363]
A
ALPHA
ort working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |217
2001-09-11
15:10:14
Arch
[1749363]
A
ALPHA
|P1 CLOSE PENDING #4244617 |CCEWP available on ladybird. Clients can deploy content to CCO. Support restarted applic to restore service. OUTAGE
2001-09-11
15:10:16
Arch
[1749363]
A
ALPHA
START: 11 Sep 11:21 PT END: 11 Sep 12:05 PT Ops 408.527.0007 |205
2001-09-11
15:16:29
Arch
[1749363]
A
ALPHA
|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |226
2001-09-11
15:30:29
Arch
[1749363]
A
ALPHA
pburleig|I am leaving office, but within 1 hour will be fully available via pager, e-mail, or phone mail. thx. Pat
2001-09-11
15:33:57
Arch
[1749363]
A
ALPHA
|NETWORK-P1 CLOSE PENDING #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore
2001-09-11
15:33:59
Arch
[1749363]
A
ALPHA
redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |233
2001-09-11
15:40:53
Arch
[1749363]
A
ALPHA
|P1 NOTICE #4244953 |sjc-i2prod has system resource issues impacting access to Demand Planning tool to forecast product demand. Support rebooting host. OUTAGE START: 11 Sep 12:32 PT ETU: 11 Sep 14:00 PT Ops 40
2001-09-11
15:40:55
Arch
[1749363]
A
ALPHA
8.527.0007 |220
2001-09-11
15:43:57
Arch
[1749363]
A
ALPHA
|NETWORK-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408.527.0007 |207
2001-09-11
16:00:00
Arch
[1749363]
A
ALPHA
|NETWORK-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2
2001-09-11
16:00:02
Arch
[1749363]
A
ALPHA
527.0007 |244 -2/2
2001-09-11
16:15:30
Arch
[1749363]
A
ALPHA
|NETWORK-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all Code Red filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11
2001-09-11
16:15:32
Arch
[1749363]
A
ALPHA
Sep 12:30 PT END: 11 Sep -1/2
2001-09-11
16:15:34
Arch
[1749363]
A
ALPHA
13:10 PT Ops 408.527.0007 |260 -2/2
2001-09-11
17:12:04
Arch
[1749363]
A
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod still unavailable impacting access to Demand Planning tool to forecast product deman
2001-09-11
17:12:06
Arch
[1749363]
A
ALPHA
d. Support continues to investigate. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |208
2001-09-11
18:46:37
Arch
[1749363]
A
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support working to fix hive corruption to restore services to host and application. OUTAGE START: 11 Sep 12:32 PT NO ETU. djaramil 408.527.0007 |220
2001-09-11
20:19:29
Arch
[1749363]
A
ALPHA
|P1 NOTICE #4245646 |outback-ams and outback-ams1 down impacting availability of NATKIT - NSA's customer network troubleshooting application - in EMEA. Support investigating. OUTAGE START: 11 Sep 17:09 PT NO ETU. Ops 4
2001-09-11
20:19:31
Arch
[1749363]
A
ALPHA
08-527-0007 |230
2001-09-11
21:33:51
Arch
[1749363]
A
ALPHA
|P1 UPDATE/CLOSE PENDING #4245646 |NATKIT, NSA's customer network troubleshooting application in EMEA is available. Support rebooted servers to restore service. OUTAGE START: 11 Sep 17:09 PT END: 11 Sep 18:28 PT Ops 408-527-0007 |229
2001-09-11
22:18:06
Arch
[1749363]
A
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support attempting to bring dev server into production to rep
2001-09-11
22:18:08
Arch
[1749363]
A
ALPHA
lace corrupted server. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
23:32:22
Arch
[1749363]
A
ALPHA
|P1 CLOSE PENDING #4244953 |sjc-i2prod and Demand Planning tool are up. Support replaced corrupted server with development server to restore service. OUTAGE START: 11 Sep 12:32 PT END: 11 Sep 20:31 PT Ops 408.527.00
2001-09-11
23:32:24
Arch
[1749363]
A
ALPHA
07 |218