9/11 Pager Messages

|Turtle do_cycle is complete. 20010911 00_13

|Snipe do_cycle is complete. 20010911 00_19

|OPC: cciweb-sj-1: /dev/vx/dsk/rest, mounted at /rest, is at 100

|OPC: lada: System load has exceeded operations threshold. Load is at 0.52

|OPC: franklin: /dev/vg04/lvol1, mounted at /mail, is at 95

[IS-HQ DOWN] ftp-poc-2

[IS-HQ DOWN] ftp-poc-2 FTP

[IS-HQ UP] ftp-poc-2 FTP

[IS-HQ UP] ftp-poc-2

[IS-HQ DOWN] ftz-mac (unresolvable)

[IS-HQ DOWN] ftp-poc-1 FTP

[IS-HQ DOWN] ftp-poc-1

[IS-HQ UP] ftp-poc-1 FTP

[IS-HQ UP] ftp-poc-1

|P2 NOTICE #4243326 |Etisalat link Dubai to Cairo failed impacting users WAN connectivity. Etisalat contacted Cairo PTT. OUTAGE START: 11 Sep 10:21 CET NO ETU. walho x4112 |172

|P2 NOTICE #4243071 |Concert link from Madrid to Lisbon flapping. No user impact due to ISDN backup. EMEA WAN opened ticket with Concert. OUTAGE START: 11 Sep 7:30 CET NO ETU. walho x4112 |188

drip root|ALERT !!!! /users is 94 % full on drip |ALERT !!!! /users is 94 % full on drip |89

|OPC: caswell: /dev/vg01/lvol1, mounted at /apps, is at 97

[IS-HQ DOWN] [mexclstr1a, mexclstr1b]

|NETWORK-P1 NOTICE #4243506 |Mexicocity-gw is down impacting mexico city feild sales office call managers. Minimum business impact due to time of day. Support is investigating. OUTAGE START: 11 Sep 03:14 PT NO ETU. Ops 408-527-0007 |232

|OPC: franklin: /dev/vg04/lvol1, mounted at /mail, is at 96

|EIS * CISCO * web reports are done. | EIS CISCO web reports have been successfully generated and available to the users on production. Thanks EIS System.|154

|OPC: kodak: System kodak seems to be down.

[IS-HQ UP] [mexclstr1a, mexclstr1b]

|NETWORK-P1 CLOSE PENDING #4243506 |Mexicocity-gw and FSO call managers are up. No impact due to time of day. Service was restored once power came up. OUTAGE START: 11 Sep 03:14 PT END: 11 Sep 04:25 PT Ops 408-527-0007 |219

|P2 CLOSE #4228756 |Sita link Amsterdam to Prague no longer dropping packets. EMEA WAN restricted bandwith. OUTAGE START: 6 Sep 8:47 CET END: 7 Sep 12:53 CET walho x4112 |170

|P2 CLOSE #4239098 | Http wwwin-emea-prod1:80 has been restore. Support investigating. OUTAGE START: 10 Sep 14:05 CET END: 11 Sep 11:00 CET josvenss x4112 |155

|Start 5 am DB recoveries

|OPC: mrpowers: /dev/vx/dsk/dgmrpowers/vrec5, mounted at /rec5, is at 95

|P2 NOTICE #4243602 |CAP/JRun on wwwin-apps-1 and 2 unavailable, impacting servlet code execution for the CAP application. Support investigating. OUTAGE START: 10 Sep 23:01 PT END: Ops 408-527-0007 |198

[IS-HQ DOWN] ftp-poc-3 FTP

[IS-HQ DOWN] ftp-poc-3

|P2 NOTICE #4243626 |Lima, Peru circuit down, impacting network connectivity for Lima Sales office. WAN support investigating with MCI. OUTAGE START: 11 Sep 00:46 PT NO ETU. Ops 408-527-0007 |191

[IS-HQ UP] ftp-poc-3 FTP

[IS-HQ UP] ftp-poc-3

|P2 CLOSE #4243626 |Lima, Peru circuit back up. Network connectivity available to Lima Sales office. Circuit restored by MCI. OUTAGE START: 11 Sep 00:46 PT END: 11 Sep 05:34 PT Ops 408-527-0007 |194

|OPC: kodak: System kodak seems to be down.

|P2 CLOSE #4243602 |CAP/JRun on wwwin-apps-1&2 available. Servlet code execution available for CAP application. Support resta

rted application to restore service. OUTAGE START: 10 Sep 23:01 PT END: 11 Sep 05:55 PT Ops 408-527-0007 |230

|OPC: lada: System load has exceeded operations threshold. Load is at 0.64

|OPC: minerva: /dev/vx/dsk/vroot, mounted at /, is at 97

[IS-HQ DOWN] cco-sj-1 sendmail

|OPC: kodak: System kodak seems to be down.

|P2 CLOSE #4224767 |BVRUMXPD BVABTPD WEBIPD BVCECPD BVWPRPD BVPOSPD BVSSRPPD databases on Minerva back up. EMEA DBAdmins restored the services after cold backup. OUTAGE START: 5 Sep 11:45 CET END: 6 Sep 2:45 CET walho x4112 |224

[IS-HQ DOWN] ftp-poc-1 FTP

[IS-HQ DOWN] ftp-poc-1

[IS-HQ UP] ftp-po

c-1 FTP

[IS-HQ UP] ftp-poc-1

|CCO-P1 NOTICE #4243775 |sendmail down on cco-sj-1. Minimal impact due to load balancing on cco-sj-2 and -3. Support investigating. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187

[IS-HQ DOWN] cco-sj-1 Status Agent

[IS-HQ DOWN] cco-sj-2 Status Agent

[IS-HQ DOWN] cco-sj-3 Status Agent

[IS-HQ UP] cco-sj-3 Status Agent

|P2 NOTICE #4243796 |sendmail down on sj-msg-core-4. Minimal impact due to redundancy. Contacting support. OUTAGE START: 11 Sep 05:47 PT NO ETU. Ops 408.527.0007 |162

amitra|bridge-erp: Please call bridge 408.527.2663 id700071 re: SJPROD db going down. Thanks, Al/Ops

[IS-HQ DOWN] mfg-erp Oracle Database (SJPROD)

[IS-HQ DOWN] cco-sj-3 Status Agent

|ERP-P1 NOTICE #4243807 |SJPROD down. ERP production db unable to accept new connections and impacting access to Status Agent. Support escalation in progress. OUTAGE START: 11 Sep 6:33 PT NO ETU. Ops 408.527.0007 |213

[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CELC (prod), CTMP20, Case Open (caseopen), DPRGAppMonitor (DPRG-prod), Marketplace (CM), NPM, Pricing Agent (pricingagent)]

[IS-HQ DOWN] cco-sj-1 [IPC Extract (ipcextract), Lead Time Agent, SCC]

[IS-HQ UP] cco-sj-1 [Auto Registration (autoreg), IPC Extract (ipcextract), Marketplace (CM), Pricing Agent (pricingagent), SCC]

[IS-HQ DOWN] cco-sj-1 STA

[IS-HQ UP] cco-sj-1 STA

[IS-HQ DOWN] cco-sj-1 Case Update (caseupdate)

[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CTMP, Config-lim (config_lim), IPC Extract (ipcextract), M

arketplace (CM), Pricing Agent (pricingagent), SVO Submit II]

[IS-HQ UP] cco-sj-1 [Auto Registration (autoreg), CTMP, CTMP20, Case Open (caseopen), Case Update (caseupdate), Config-lim (config_lim), DPRGAppMonitor (DPRG-prod), IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), NPM]

[IS-HQ UP] cco-sj-1 [Pricing Agent (pricingagent)]

[IS-HQ UP] cco-sj-1 CELC (prod)

[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CELC (prod), CTMP20, DPRGAppMonitor (DPRG-prod)]

[IS-HQ UP] cco-sj-1 Auto Registration (autoreg)

[IS-HQ UP] cco-sj-1 [CTMP20, DPRGAppMonitor (DPRG-prod), SVO Submit II]

[IS-HQ DOWN] cco-sj-1 NPM

[IS-HQ UP] cco-sj-1 [CELC (prod), NPM]

[IS-HQ DOWN] cco-sj-1 [CTMP20, DPRGAppMonitor (DPRG-prod)]

[IS-HQ DOWN] cco-sj-1 [CELC (prod), COLT (Student), View Return (viewreturn)]

[IS-HQ UP]

cco-sj-1 COLT (Student)

|OPC: topic-sj: System load has exceeded threshold. Load is at 20.89

[IS-HQ DOWN] cco-sj-1 [Case Open (caseopen), Lead Time Agent, NPM, STA]

[IS-HQ DOWN] cco-sj-1 COLT (Student)

[IS-HQ UP] cco-sj-1 Lead Time Agent

[IS-HQ DOWN] cco-sj-1 SCC OLD

[IS-HQ UP] mfg-erp Oracle Database (SJPROD)

|OpC: MAXRUNALARM - Job has exceeded Maximum Run Time. Hostname is uranium.cisco.com page sent to ops-duty

[IS-HQ UP] cco-sj-1 [SCC OLD, STA]

[IS-HQ DOWN] cco-sj-1 [Config-lim (config_lim), ICS, SVO Status]

[IS-HQ UP] cco-sj-1 COLT (Student)

[IS-HQ UP] cco-sj-1 [Config-lim (config_lim), ICS, SVO Status]

[IS-HQ DOWN] cco-sj-1 CTMP

[IS-HQ UP] cco-sj-1 [Case Open (caseopen), View Return (

viewreturn)]

[IS-HQ DOWN] cco-sj-1 Case Update (caseupdate)

[IS-HQ UP] cco-sj-1 Case Update (caseupdate)

[IS-HQ DOWN] cco-sj-1 Service Order Status

[IS-HQ UP] cco-sj-1 [Config-lim (config_lim), Service Order Status, Status Agent]

[IS-HQ DOWN] cco-sj-1 Config-lim (config_lim)

[IS-HQ UP] cco-sj-1 CTMP

[IS-HQ UP] cco-sj-2 Status Agent

[IS-HQ DOWN] cco-sj-1 [Case Open (caseopen), Case Update (caseupdate)]

[IS-HQ UP] cco-sj-3 Status Agent

[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CTMP, Config-lim (config_lim), IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), Service Order Status, Status Agent]

|ERP-P1 UPDATE/CLOSE PENDING #4243807 |SJPROD db instance on host high was hung and is now up. ERP production db is now able

to accept new connections. Status Agent is also available. ERP_dba's restarted instance to restore service. -1/2

OUTAG

E START: 11 Sep 6:33 PT END: 11 Sep 7:00 PT Ops 408.527.0007 |299 -2/2

|CCO-P1 UPDATE #4243775 |cco-sj-1 not responding. Support removing host from service to allow apps availibility on cco-sj-2 and -3. OUTAGE START: 11 Sep 06:19 PT NO ETU.

Ops 408.527.0007 |187

[IS-HQ DOWN] cco-sj-1 [ICS, SCC OLD]

[IS-HQ UP] cco-sj-1 [Auto Registration (autoreg), CTMP, Case Update (caseupdate), Config-lim (config_lim), ICS, IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), SCC OLD, Service Order Status, Status Agent]

[IS-HQ UP] cco-sj-1 [CELC (prod), Case Open (caseopen)]

[IS-HQ UP] cco-sj-1 [CTMP20, DPRGAppMonitor (DPRG-prod), NPM]

[IS-HQ DOWN] cco-sj-1 [CELC (prod), NPM]

[IS-HQ UP] cco-sj-1 NPM

[IS-HQ UP] cco-sj-1 CELC (prod)

[IS-HQ DOWN] cco-sj-1 NPM

[IS-HQ DOWN] cco-sj-1 DPRGAppMonitor (DPRG-prod)

[IS-HQ UP] cco-sj-1 [DPRGAppMonitor (DPRG-prod), NPM]

[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CELC (prod), COLT (Student), CTMP, CTMP20, Case Open (caseopen), Case Query (casequery), Case Update (caseupdate), Config-lim (config_lim), DPRGAppMonitor (DPRG-prod), H

TTP (cco-admin)]

[IS-HQ DOWN] cco-sj-1 [HTTP (cco-sj-1-admin), HTTP (www.cisco.com), HTTPS (Secure Server), ICS, IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), NPM, Pricing Agent (pricingagent), SCC, SCC OLD, STA, Service Order Status]

[IS-HQ DOWN] cco-sj-1 [Status Agent, View Return (viewreturn)]

[IS-HQ DOWN] cco-sj-1 [Invoice agent (invoiceagent), SVO Status, SVO Submit II]

[IS-HQ DOWN] cco-sj-1 [Aged Accounts Summary Tool (Age, DistiCredit, Submit Return (submitreturn), Upgrade Agent]

[IS-HQ UP] cco-sj-1 sendmail

[IS-HQ UP] cco-sj-1 [Auto Registration (autoreg), CEL

C (prod), COLT (Student), CTMP, CTMP20, Case Open (caseopen), Case Query (casequery), Case Update (caseupdate), Config-lim (config_lim), DPRGAppMonitor (DPRG-prod), HTTP (cco-admin)]

[IS-HQ UP] cco-sj-1 [HTTP (cco-sj-1-admin), HTTP (www.cisco.com), HTTPS (Secure Server), ICS, IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), NPM, Pricing Agent (pricingagent), SCC, SCC OLD, STA, Service Order Status]

[IS-HQ UP] cco-sj-1 [Status Agent, View Return (viewreturn)]

[IS-HQ UP] cco-sj-1 [Aged Accounts Summary Tool (Age, Disti

Credit, Invoice agent (invoiceagent), SVO Status, SVO Submit II, Submit Return (submitreturn), Upgrade Agent]

[IS-HQ DOWN] ftp-poc-3

[IS-HQ DOWN] ftp-poc-3 FTP

[IS-HQ UP] ftp-poc-3 FTP

[IS-HQ UP] ftp-poc-3

|CCO-P1 CLOSE PENDING #4243775 |cco-sj-1 has been brought back in to service. Support continues to investigate cause of outage. OUTAGE START: 11 Sep 06:19 PT END: 11 Sep 08:00 PT Ops 408.527.0007 |196

[IS-HQ DOWN] ftp-poc-2 FTP

[IS-HQ DOWN] ftp-poc-2

[IS-HQ UP] ftp-poc-2 FTP

[IS-HQ UP] ftp-poc-2

|P2 NOTICE/CLOSE #4244027 |Epage

services were unavailable due to high load on all epage servers. OUTAGE START: 11 Sep 8:10 PT END: 11 Sep 8:32 PT Ops 408.527.0007 |164

|OPC: uranium: /dev/vg01/lvol1, mounted at /apps, is at 97

|V

OICE-P1 NOTICE #4244044 |No calls able to go through to EMEA due to high load on national and international circuits. Voiceservices investigating. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 4085270007 |202

[IS-HQ DOWN] cco-sj-1 Status Agent

| BV_PB:Backup:info:18:00:02:RDCP:RDCP BVOLS to NVOLS

[IS-HQ UP] cco-sj-1 Status Agent

|POWER STATUS : No stage alerts at this time ||46

|OPC: bus: Test, disregard

|OPC: uranium: /dev/vg01/lvol1, mounted at /apps, is at 98

|P2 NOTICE #4244245 |Network link to Bogota, Colombia down impacting access to local FSO. Contacting support. OUTAGE START: 11 Sep 9:32 PT NO ETU. Ops 408.527.0007 |164

| BV_PB:Backup:info:18:56:50:

PRESYNC:Break away backup plexes

| BV_PB:Backup:info:18:57:22:SYNC:active resilvering

|OPC: uranium: /dev/vg01/lvol1, mounted at /apps, is at 99

|VOICE-P1 UPDATE #4244044 |Calls to EMEA intermittent due to high load on national and international circuits. Voiceservices continuing to investigate. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 4085270007 |205

|OPC: mrpowers: /dev/vx/dsk/dgmrpowers/vrec5, mounted at /rec5, is at 95

|snipe 5AM recovery complete

|OPC: lada: System load has exceeded operations threshold. Load is at 0.28

|EIS * ASIA * web reports are done. | EIS ASIA web reports have been successfully generated and available to the users on production. Thanks EIS System.|152

|NETWORK-P1 NO

TICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network availability. SJ LAN support investigating issues with Code Red filters being removed. OUTAGE START: 11 Sep -1/2

11:05 PT NO ETU. Ops 408.527.0007 |268 -2/2

||SUCCESS:

snipe RECOVER_STANDBY for PROD2 .|44

[IS-HQ DOWN] ladybird CCEWP

|VOICE-P1 NOTICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network and IP phone availability. SJ LAN support investigating issues with Code Red fi

lters. OUTAGE START: 11 Sep -1/2

11:05 PT NO ETU. Ops 408.527.0007 |265 -2/2

|P1 NOTICE #4244617 |CCEWP unavailable on ladybird impacting content delopyment to CCO. Support is investigating. OUTAGE START:

11 Sep 11:21 PT NO ETU. Ops 408.527.0007 |169

|OPC: radon: /dev/vg00/lvol6, mounted at /opt, is at 98

|VOICE-P1 UPDATE #4244044 |Calls between US and EMEA intermittent due to high load on national and international circuits. Voice Services investigating. Next update at 16:00 PT. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 408.527.0007 |233

|:BV_PB:info:20:33:12:Backup:BREAK: disassociate plexes

|:BV_PB:info:20:33:27:Backup:DBONLINE: end hot backups/start dbs

| BV_PB:Backup:info:20:32:20:SYNC:RESILVERING COMPLETE

|:BV_PB:info:20:33:39:Backup:BMERGE: Attaching plexes to backup vols

|:BV_PB:info:20:33:52:Backup:ALEX: execute alex rpf stores

|P1::BV_PB Alexandria op-cards have been started, please monitor till success

|:BV_PB:info:20:34:09:Backup:COMPLETE:ERP backup script complete

|AUTOSYS(PRD): job=qtc_XHOLDS starting_state=1440 seconds. Please modify job to failure, then contact job_owner to verify job restart status . ||144

|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |217

Oracle on Xenon|AUTOSYS(PRD): job=qtc_XHOLDS starting_state=1447 seconds. Please modify job to failure, then contact job_owner to verify job restart status . ||159

|VOICE-P1 UPDATE #4244564 |Site 4 network service restored and IP ph

one services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |215

|AUTOSYS(PRD): job=qtc_XHOLDS starting_state=2040 seconds. Please modify job to failure, then contact job_owner to verify job restart status . ||144

Oracle on Xenon|AUTOSYS(PRD): job=qtc_XHOLDS starting_state=2048 seconds. Please modify job to failure, then contact jo

b_owner to verify job restart status . ||159

[IS-HQ UP] ladybird CCEWP

|P1 CLOSE PENDING #4244617 |C

CEWP available on ladybird. Clients can deploy content to CCO. Support restarted applic to restore service. OUTAGE START: 11 Sep 11:21 PT END: 11 Sep 12:05 PT Ops 408.527.0007 |205

|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |226

|VOICE-P1 UPDATE #4244564 |Site 4 network service restor

ed and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |224

|OPC: uranium: /dev/vg01/lvol1, mounted at /apps, is at 100

[IS-HQ DOWN] ftp-poc-1 FTP

[IS-HQ DOWN] ftp-poc-1

[IS-HQ UP] ftp-poc-1

[IS-HQ UP] ftp-poc-1 FTP

|NETWORK-P1 CLOSE PENDING #42

44564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |233

[IS-HQ DOWN] sjc-i2prod

|VOICE-P1 CLOSE PENDING #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |231

|OPC: tahoe: /dev/vg02/vusers, mounted at /users, is at 97

|OPC: tahoe: /dev/vg02/vusers, mounted at /users, is at 98

|P1 NOTICE #4244953 |sjc-i2prod has system resource issues impacting access to Demand Planning tool to forecast product demand. Support rebooting host. OUTAGE START: 11 Sep 12:32 PT ETU: 11 Sep 14:00 PT Ops 40

8.527.0007 |220

|NETWORK-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408.527.0007 |207

|VOICE-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408.527.0007 |205

|OPC: eman-dev: /dev/vx/dsk/dgemandev/vusers, mounted at /users, is at 96

|OPC: morris: /dev/vx/dsk/ROOTVOL, mounted at /, is at 100

|VOICE-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2

527.0007 |242 -2/2

|NETWORK-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2

527.0007 |244 -2/2

[IS-HQ DOWN] ftp-poc-3 FTP

[IS-HQ DOWN] ftp-poc-3

[IS-HQ UP] ftp-poc-3 FTP

[IS-HQ UP] ftp-poc-3

|NETWORK-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all Code Red filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2

13:10 PT Ops 408.527.0007 |260 -2/2

|VOICE-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all Code Red

filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2

13:10 PT Ops 408.527.0007 |258 -2/2

|OPC: peerless: /dev/vx/dsk/dgpe

erless/vusers, mounted at /users, is at 100

[IS-HQ DOWN] ftp-poc-2

[IS-HQ DOWN] ftp-poc-2 FTP

[IS-HQ UP] [ftp-poc-2, sjc-i2prod]

[IS-HQ UP] ftp-poc-2 FTP

|OPC: escort: /dev/vx/dsk/dgescort/users, mounted at /users, is at 95

|OPC: lbj: /dev/vx/dsk/documdg/vtddoc07, mounted at /dm_vault/cisco_prodt/cisco_td_doc_07, is at 95

|OPC: caswell: /dev/vg01/lvol1, mounted at /apps, is at 98

|P

1 UPDATE #4244953 |sjc-i2prod still unavailable impacting access to Demand Planning tool to forecast product demand. Support continues to investigate. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |208

[IS-HQ DOWN] ladybird CCEWP

[IS-HQ UP] ladybird CCEWP

|VOICE-P1 UPDATE #4244044 |Calls between US and EMEA intermittent due to high load on national and international circuits. MCI and Sprint expect phone lines to become stable by tomorrow afternoon. Voice Services support monitoring -1/2

OUTAGE START: 11 Sep 08:31 PT ETU: 12 Sep 12:00 PT Ops 408.527.0007 |299 -2/2

|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support working to fix hive corruption to restore services to host and application. OUTAGE START: 11 Sep 12:32 PT NO ETU. djaramil 408.527.0007 |220

|OPC: marvel: /dev/vgarch/v20000-01, mounted at /archive1, is at 92

|OPC: mcoapp-sj-2: /dev/vx/dsk/rootdg/log, mounted at /fs/log, is at 99

|OPC: mcoapp-sj-2: /dev/vx/dsk/rootdg/log, mounted at /fs/log, is at 100

|OPC: lada: System load has exceeded operations threshold. Load is at 0.73

[IS-HQ DOWN] sjc-i2prod

[IS-HQ DOWN] ftp-poc-3 FTP

[IS-HQ DOWN] ftp-poc-3

[IS-HQ UP] ftp-poc-3 FTP

[IS-HQ UP] ftp-poc-3

|OpC PSoft Alert CSPROD00 SQR Program NOTE: csprod00 is starting. (TEST, PLEASE DISREGARD)

|Start 5 pm DB recoveries

[IS-HQ DOWN] outback-ams1

|P1 NOTICE #4245646 |outback-ams and outback-ams1 down impacting availability of NATKIT - NSA's customer network troubleshooting application - in EMEA. Support investigating

. OUTAGE START: 11 Sep 17:09 PT NO ETU. Ops 408-527-0007 |230

|OPC: solstice: System solstice seems to be down.

[IS-HQ DOWN] brussels sendmail

[IS-HQ DOWN] glasgow sendmail

[IS-HQ DOWN] johannesburg sendmail

[IS-HQ DOWN] london sendmail

[IS-HQ DOWN] madrid sendmail

[IS-HQ DOWN] nordic sendmail

[IS-HQ UP] brussels sendmail

[IS-HQ UP] glasgow sendmail

[IS-HQ UP] johannesburg sendmail

[IS-HQ UP] london sendmail

[IS-HQ UP] madrid sendmail

[IS-HQ UP] nordic sendmail

|OPC: blender-sj1: /dev/vx/dsk/usr, mounted at /usr, is at 96

[IS-HQ UP] outback-ams1

|OpC PSoft Alert CSPROD97 SQR Program NOTE: csprod97 is Ending. (TEST, PLEASE DISREGARD)

|OPC: prowler: System prowler seems to be down.

|OPC: harrier: System harrier seems to be down.

[IS-HQ DOWN] [harrier, outback-ams1, prowler]

[IS-HQ UP]

[harrier, prowler]

[IS-HQ UP] outback-ams1

|P1 UPDATE/CLOSE PENDING #4245646 |NATKIT, NSA's customer network troubleshooting application in EMEA is available. Support rebooted servers to restore service. OUTAGE START: 11 Sep 17:09 PT END: 11 Sep 18:28 PT Ops 408-527-0007 |229

|OpC PSoft Alert CSPROD99 SQR Program NOTE: csprod99 is Ending. (TEST, PLEASE DISREGARD)

|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support attempting to bring dev server into production to replace corrupted server. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |215

|OpC PSoft Alert CSREPT00 SQR Program Note: csrept00 Starting. (TEST, PLEASE DISREGARD)

[IS-HQ UP] sjc-i2prod

|EIS * EUROPE * web reports are done. | EIS EUROPE web reports have been successfully generated and available to the users on production. Thanks EIS System.|156

|unable to umount /db506B on elroy to begin backup urgent attention needed

[IS-HQ DOWN] sjc-i2prod

[IS-HQ UP] sjc-i2prod

[IS-HQ DOWN] sjc-i2prod

[IS-HQ UP] sjc-i2prod

|OpC: MAXRUNALARM - Job has exceeded Maximum Run Time. Hostname is hurley.cisco.com page sent to ops-duty

|P1 CLOSE PENDING #4244953 |sjc-i2prod and Demand Planning tool are up. Support replaced corrupted server with development server to restore service. OUTAGE START: 11 Sep 12:32 PT END: 11 Sep 20:31 PT Ops 408.527.0007 |218

|OPC: wwwin-emea-dev: /dev/vx/dsk/wdevdg/vwdev, mounted at /fs/wdev, is at 95

|OPC: wwwin-emea-dev: /dev/vx/dsk/wdevdg/vwdev, mounted at /fs/wdev, is at 96

| BV_PB:Backup:info:06:01:03:RDCP:RDCP BVOLS to NVOLS

|OPC: lada: System load has exceeded operations threshold. Load is at 0.69

[IS-HQ DOWN] ftp-poc-2 FTP

[IS-HQ DOWN] ftp-poc-2

[IS-HQ UP] ftp-poc-2 FTP

[IS-HQ UP] ftp-poc-2

| BV_PB:Backup:info:06:57:24:PRESYNC:Break away backup plexes

| BV_PB:Backup:info:06:57:59:SYNC:active resilvering

|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 95

|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 96

|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 97

|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 98

|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 99

|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 100

||SUCCESS: snipe RECOVER_STANDBY for PROD2 .|44

| 5PM recovery complete

|Start do_cycle.

||SUCCESS: snipe RECOVER_STANDBY for PROD .|43

|OPC: rex: /dev/vx/dsk/sundg/v122sun-12, mounted at /beyond/122/sun12, is at 95

| BV_PB:Backup:info:08:35:21:SYNC:RESILVERING COMPLETE

|:BV_PB:info:08:36:12:Backup:BREAK: disassociate plexes

|:BV_PB:info:08:36:28:Backup:DBONLINE: end hot backups/start dbs

|:BV_PB:info:08:36:40:Backup:BMERGE: Attaching plexes to backup vols

|:BV_PB:info:08:36:53:Backup:COMPLETE:ERP backup script complete