9/11 Pager Messages
2001-09-11
05:32:19
Arch
[0918725]
B
ALPHA
|P2 NOTICE #4243326 |Etisalat link Dubai to Cairo failed impacting users WAN connectivity. Etisalat contacted Cairo PTT. OUTAGE START: 11 Sep 10:21 CET NO ETU. walho x4112 |172
2001-09-11
05:38:38
Arch
[0918725]
B
ALPHA
|P2 NOTICE #4243071 |Concert link from Madrid to Lisbon flapping. No user impact due to ISDN backup. EMEA WAN opened ticket with Concert. OUTAGE START: 11 Sep 7:30 CET NO ETU. walho x4112 |188
2001-09-11
07:33:30
Arch
[0918725]
B
ALPHA
|P2 CLOSE #4228756 |Sita link Amsterdam to Prague no longer dropping packets. EMEA WAN restricted bandwith. OUTAGE START: 6 Sep 8:47 CET END: 7 Sep 12:53 CET walho x4112 |170
2001-09-11
07:55:00
Arch
[0918725]
B
ALPHA
|P2 CLOSE #4239098 | Http wwwin-emea-prod1:80 has been restore. Support investigating. OUTAGE START: 10 Sep 14:05 CET END: 11 Sep 11:00 CET josvenss x4112 |155
2001-09-11
08:16:07
Arch
[0918725]
B
ALPHA
|P2 NOTICE #4243602 |CAP/JRun on wwwin-apps-1 and 2 unavailable, impacting servlet code execution for the CAP application. Support investigating. OUTAGE START: 10 Sep 23:01 PT END: Ops 408-527-0007 |198
2001-09-11
08:22:54
Arch
[0918725]
B
ALPHA
|P2 NOTICE #4243626 |Lima, Peru circuit down, impacting network connectivity for Lima Sales office. WAN support investigating wi
2001-09-11
08:22:56
Arch
[0918725]
B
ALPHA
th MCI. OUTAGE START: 11 Sep 00:46 PT NO ETU. Ops 408-527-0007 |191
2001-09-11
08:37:14
Arch
[0918725]
B
ALPHA
|P2 CLOSE #4243626 |Lima, Peru circuit back up. Network connectivity available to Lima Sales office. Circuit restored by MCI. OUTAGE START: 11 Sep 00:46 PT END: 11 Sep 05:34 PT Ops 408-527-0007 |194
2001-09-11
09:06:01
Arch
[0918725]
B
ALPHA
|P2 CLOS
2001-09-11
09:06:03
Arch
[0918725]
B
ALPHA
E #4243602 |CAP/JRun on wwwin-apps-1&2 available. Servlet code execution available for CAP application. Support restarted application to restore service. OUTAGE START: 10 Sep 23:01 PT END: 11 Sep 05:55 PT Ops 408-527-0007 |230
2001-09-11
09:26:57
Arch
[0918725]
B
ALPHA
|P2 CLOSE #4224767 |BVRUMXPD BVABTPD WEBIPD BVCECPD BVWPRPD BVPOSPD BVSSRPPD databases on Minerva back up. EMEA DBAdmins restored the services after cold backup. OUTAGE START: 5 Sep 11:45 CET END: 6 Sep 2:45 CET walho x4112 |224
2001-09-11
09:33:14
Arch
[0918725]
B
ALPHA
|CCO-P1 NOTICE #4243775 |sendmail down on cco-sj-1. Minimal impact due to load balancing on cco-sj-2 and -3. Support investigating. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187
2001-09-11
09:37:08
Arch
[0918725]
B
ALPHA
|P2 NOTICE #4243796 |sendmail down on sj-msg-core-4. Minimal impact due to redundancy. Contacting support. OUTAGE START: 11 Sep 05:47 PT NO ETU. Ops 408.527.0007 |162
2001-09-11
09:41:05
Arch
[0918725]
B
ALPHA
|ERP-P1 NOTICE #4243807 |SJPROD down. ERP production db unable to accept new connections and impacting access to Status Agent. Support escalation in progress. OUTAGE START: 11 Sep 6:33 PT NO ETU. Ops 408.527.0007 |213
2001-09-11
10:08:38
Arch
[0918725]
B
ALPHA
|ERP-P1 UPDATE/CLOSE PENDING #42
2001-09-11
10:08:40
Arch
[0918725]
B
ALPHA
43807 |SJPROD db instance on host high was hung and is now up. ERP production db is now able to accept new connections. Status Agent is also available. ERP_dba's restarted instance to restore service. -1/2
2001-09-11
10:08:44
Arch
[0918725]
B
ALPHA
OUTAGE START: 11 Sep 6:33 PT END: 11 Sep 7:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
10:08:53
Arch
[0918725]
B
ALPHA
|CCO-P1 UPDATE #4243775 |cco-sj-1 not responding. Support removing host from service to allow apps availibility on cco-sj-2 and -3. OUTAGE START: 11 Sep 06:19 PT NO
2001-09-11
10:08:55
Arch
[0918725]
B
ALPHA
ETU. Ops 408.527.0007 |187
2001-09-11
11:03:34
Arch
[0918725]
B
ALPHA
amitra|John Bruno's Direct Reports: Per Lance Perry, please call bridge 408.527.2663 id id700071 immediately. Tha
2001-09-11
11:03:36
Arch
[0918725]
B
ALPHA
nk you, Al/Ops
2001-09-11
11:29:36
Arch
[0918725]
B
ALPHA
|CCO-P1 CLOSE PENDING #4243775 |cco-sj-1 has been brought back in to service.
2001-09-11
11:29:38
Arch
[0918725]
B
ALPHA
Support continues to investigate cause of outage. OUTAGE START: 11 Sep 06:19 PT END: 11 Sep 08:00 PT Ops 408.527.0007 |196
2001-09-11
11:46:58
Arch
[0918725]
B
ALPHA
|P2 NOTICE/CLOSE #4244027 |Epage services were unavailab
2001-09-11
11:47:00
Arch
[0918725]
B
ALPHA
le due to high load on all epage servers. OUTAGE START: 11 Sep 8:10 PT END: 11 Sep 8:32 PT Ops 408.527.0007 |164
2001-09-11
12:14:06
Arch
[0918725]
B
ALPHA
dleitzel|Will you still be able to meet today at 11:00 (PDT) to discuss GSD SOW? Please page or call. I am checking with IBM also. David Leitzell 525-8835
2001-09-11
12:47:35
Arch
[0918725]
B
ALPHA
|P2 NOTICE #4244245 |Network link to Bogota, Colombia down impacting access to local FSO. Contacting support. OUTAGE START: 11 Sep 9:32 PT NO ETU. Ops 408.527.0007 |164
2001-09-11
14:31:07
Arch
[0918725]
B
ALPHA
|P1 NOTICE #4244617 |CCEWP unavailable on ladybird impacting content delopyment to CCO. Support is investigating. OUTAGE START: 11 Sep 11:21 PT NO ETU. Ops 408.527.0007 |169
2001-09-11
15:11:08
Arch
[0918725]
B
ALPHA
|P1 CLOSE PENDING #4244617 |CCEWP available on ladybird. Clients can deploy content to CCO. Support restarted applic to restore service. OUTAGE START: 11 Sep 11:21 PT END: 11 Sep 12:05 PT Ops 408.527.0007 |205
2001-09-11
15:41:53
Arch
[0918725]
B
ALPHA
|P1 NOTICE #4244953 |sj
2001-09-11
15:41:55
Arch
[0918725]
B
ALPHA
c-i2prod has system resource issues impacting access to Demand Planning tool to forecast product demand. Support rebooting host. OUTAGE START: 11 Sep 12:32 PT ETU: 11 Sep 14:00 PT Ops 408.527.0007 |220
2001-09-11
17:12:53
Arch
[0918725]
B
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod still unavailable impacting access to Demand Planning tool to forecast product demand. Support continues to investigate. OUTAGE START: 11 Sep 12:32 PT NO ET
2001-09-11
17:12:54
Arch
[0918725]
B
ALPHA
U. Ops 408.527.0007 |208
2001-09-11
18:47:39
Arch
[0918725]
B
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support working to fix hive corruption to restore services to host and application. OUTAGE START: 11 Sep 12:32 PT NO ETU. djaramil 408.527.0007 |220
2001-09-11
20:20:10
Arch
[0918725]
B
ALPHA
|P1 NOTICE #4245646 |outback-ams and outback-ams1 down impacting availability of NATKIT - NSA's customer network troubleshooting application - in EMEA. Support investigating. OUTAGE START: 11 Sep 17:09 PT NO ETU. Ops 408-527-0007 |230
2001-09-11
21:34:38
Arch
[0918725]
B
ALPHA
|P1 UPDATE/CLOSE PENDING #4245646 |NATKIT, NSA's customer network troubleshooting application in EMEA is available. Support rebooted servers to restore service. OUTAGE START: 11 Sep 17:09 PT END: 11 Sep 18:28 PT Ops 408-527-0007 |229
2001-09-11
22:19:08
Arch
[0918725]
B
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support attempting to bring dev server into production to replace corrupted server. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
23:33:11
Arch
[0918725]
B
ALPHA
|P1 CLOSE PENDING #4244953 |sjc-i2prod and Demand Planning tool are up. Support replaced corrupted server with development server to restore service. OU
2001-09-11
23:33:13
Arch
[0918725]
B
ALPHA
TAGE START: 11 Sep 12:32 PT END: 11 Sep 20:31 PT Ops 408.527.0007 |218