9/11 Pager Messages
2001-09-11
09:33:55
Arch
[0925625]
C
ALPHA
|CCO-
2001-09-11
09:33:57
Arch
[0925625]
C
ALPHA
P1 NOTICE #4243775 |sendmail down on cco-sj-1. Minimal impact due to load balancing on cco-sj-2 and -3. Support investigating. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187
2001-09-11
09:42:27
Arch
[0925625]
C
ALPHA
|ERP-P1 NOTICE #4243807 |SJPROD down. ERP production db unable to accept new connections and impacting access to Status Agent. Support escalation in progress. OUTAGE START: 11 Sep 6:33 PT NO ETU. Ops 408.527.0007 |213
2001-09-11
10:09:17
Arch
[0925625]
C
ALPHA
|ERP-P1 UPDATE/CLOSE PENDING #4243807 |SJPROD db instance on host high was hung and is now up. ERP production db is now able to accept new connections. Status Agent is also available. ERP_dba's restarted instance to restore service. -1/2
2001-09-11
10:09:19
Arch
[0925625]
C
ALPHA
OUTAGE START: 11 Sep 6:33 PT END: 11 Sep 7:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
10:10:04
Arch
[0925625]
C
ALPHA
|CCO-P1 UPDATE #4243775 |cco-sj-1 not responding. Support removing host from service to allow apps availibility on cco-sj-2 and -3. OUTAG
2001-09-11
10:10:06
Arch
[0925625]
C
ALPHA
E START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187
2001-09-11
11:30:41
Arch
[0925625]
C
ALPHA
|CCO-P1 CLOSE PENDING #4243775 |cco-sj-1 has been brought back in to service. Support continues to investigate cause of outage. OUTAGE START: 11 Sep 06:19 PT END: 11 Sep 08:00 PT Ops 408.527.0007 |196
2001-09-11
11:47:08
Arch
[0925625]
C
ALPHA
|VOICE-P1 NOTICE #4244044 |No calls able to go through to EMEA due to high load on national and international circuits. Voiceservices investigating. OUTAGE
2001-09-11
11:47:09
Arch
[0925625]
C
ALPHA
START: 11 Sep 08:31 PT NO ETU. Ops 4085270007 |202
2001-09-11
12:09:36
Arch
[0925625]
C
ALPHA
"Williams, William"|guess I missed brad |i've been working on scanner problems on the SJPROD bridge. maybe next time. Cheers! -b|128
2001-09-11
13:27:43
Arch
[0925625]
C
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls to EMEA intermittent due to high load on national and international circuits. Voiceservices continuing to investigate. OUTAGE START: 11 Sep 08:31 PT NO ETU. O
2001-09-11
13:27:44
Arch
[0925625]
C
ALPHA
ps 4085270007 |205
2001-09-11
14:24:37
Arch
[0925625]
C
ALPHA
|VOICE-P1 NOTICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network and IP phone availability. SJ LAN support investigating
2001-09-11
14:24:38
Arch
[0925625]
C
ALPHA
issues with Code Red filters. OUTAGE START: 11 Sep -1/2
2001-09-11
14:24:40
Arch
[0925625]
C
ALPHA
11:05 PT NO ETU. Ops 408.527.0007 |265 -2/2
2001-09-11
14:31:12
Arch
[0925625]
C
ALPHA
|P1 NOTICE #4244617 |CCEWP unavailable on ladybird impacting content delopyment to CCO. Support is investigating. OUTAGE START: 11 Sep 11:21 PT NO ETU. Ops 408.527.0007 |169
2001-09-11
14:32:44
Arch
[0925625]
C
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls between US and EMEA intermittent due to high load on national and international circuits. Voice Services investigating. Next update at 16:00 PT. OUTAGE START: 11 Sep 0
2001-09-11
14:32:46
Arch
[0925625]
C
ALPHA
8:31 PT NO ETU. Ops 408.527.0007 |233
2001-09-11
14:41:05
Arch
[0925625]
C
ALPHA
|VOICE-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
15:11:19
Arch
[0925625]
C
ALPHA
|P1 CLOSE PENDING #4244617 |CCEWP available on ladybird. Clients can
2001-09-11
15:11:21
Arch
[0925625]
C
ALPHA
deploy content to CCO. Support restarted applic to restore service. OUTAGE START: 11 Sep 11:21 PT END: 11 Sep 12:05 PT Ops 408.527.0007 |205
2001-09-11
15:18:01
Arch
[0925625]
C
ALPHA
|VOICE-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |224
2001-09-11
15:35:59
Arch
[0925625]
C
ALPHA
|VOICE-P1 CLOSE PENDING #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |231
2001-09-11
15:42:12
Arch
[0925625]
C
ALPHA
|P1 NOTICE #4244953 |sjc-i2prod has system resource issues impacting access to Demand Planning tool to forecast product demand. Support rebooting host. OUTAGE START: 11 Sep 12:32 PT ETU: 11 Sep 14:00 PT Ops 408.527.0007 |220
2001-09-11
15:45:47
Arch
[0925625]
C
ALPHA
|VOICE-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11
2001-09-11
15:45:49
Arch
[0925625]
C
ALPHA
Sep 12:30 PT NO ETU. Ops 408.527.0007 |205
2001-09-11
15:59:29
Arch
[0925625]
C
ALPHA
|VOICE-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2
2001-09-11
15:59:30
Arch
[0925625]
C
ALPHA
527.0007 |242 -2/2
2001-09-11
16:18:06
Arch
[0925625]
C
ALPHA
|VOICE-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all
2001-09-11
16:18:08
Arch
[0925625]
C
ALPHA
Code Red filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2
2001-09-11
16:18:36
Arch
[0925625]
C
ALPHA
13:10 PT Ops 408.527.0007 |258 -2/2
2001-09-11
17:13:15
Arch
[0925625]
C
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod still unavailable impacting access to Demand Planning tool to forecast product demand. Support continues to investigate. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |208
2001-09-11
18:43:20
Arch
[0925625]
C
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls
2001-09-11
18:43:22
Arch
[0925625]
C
ALPHA
between US and EMEA intermittent due to high load on national and international circuits. MCI and Sprint expect phone lines to become stable by tomorrow afternoon. Voice Services support monitoring -1/2
2001-09-11
18:43:24
Arch
[0925625]
C
ALPHA
OUTAGE START: 11 Sep 08:31 PT ETU: 12 Sep 12:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
18:47:59
Arch
[0925625]
C
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support working to fix hive corruption to restore services to host and application. OUTAGE START: 11 Sep 12:32 PT NO ETU. djaramil
2001-09-11
18:48:01
Arch
[0925625]
C
ALPHA
408.527.0007 |220
2001-09-11
20:20:29
Arch
[0925625]
C
ALPHA
|P1 NOTICE #4245646 |outback-ams and outback-ams1 down impacting availability of NATKIT - NSA's customer network troubleshooting application - in EMEA. Support investigating. OUTAGE START: 11 Sep 17:09 PT NO ETU. Ops 408-527-0007 |230
2001-09-11
21:35:15
Arch
[0925625]
C
ALPHA
|P1 UPDATE/CLOSE PENDING #4245646 |NATKIT, NSA's customer network troubleshooting application in EMEA is available. Support rebooted servers to restore service. OUTAGE START: 11 Sep 17:09 PT END: 11 Sep 18:28 PT Ops 408-527-0007 |229
2001-09-11
22:19:38
Arch
[0925625]
C
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support attempting to bring dev server into production to replace corrupted server. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
23:33:45
Arch
[0925625]
C
ALPHA
|P1 CLOSE PENDING #4244953 |sjc-i2prod and Demand Planning tool are up. Support replaced corrupted server with development server to restore service. OUTAGE START: 11 Sep 12:32 PT END: 11 Sep 20:31 PT Ops 408.527
2001-09-11
23:33:47
Arch
[0925625]
C
ALPHA
.0007 |218