9/11 Pager Messages

2001-09-11
03:13:10
Arch
[0931795]
A
ALPHA
|Turtle do_cycle is complete. 20010911 00_13
2001-09-11
03:19:38
Arch
[0931795]
A
ALPHA
|Snipe do_cycle is complete. 20010911 00_19
2001-09-11
04:06:31
Arch
[0931795]
A
ALPHA
|OPC: cciweb-sj-1: /dev/vx/dsk/rest, mounted at /rest, is at 100
2001-09-11
04:10:01
Arch
[0931795]
A
ALPHA
|OPC: lada: System load has exceeded operations threshold. Load is at 0.52
2001-09-11
04:22:19
Arch
[0931795]
A
ALPHA
|OPC: franklin: /dev/vg04/lvol1, mounted at /mail, is at 95
2001-09-11
04:35:49
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-2
2001-09-11
04:35:51
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-2 FTP
2001-09-11
04:36:29
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-2 FTP
2001-09-11
04:36:31
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-2
2001-09-11
05:00:40
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftz-mac (unresolvable)
2001-09-11
05:19:28
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-1 FTP
2001-09-11
05:19:30
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-1
2001-09-11
05:19:32
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-1 FTP
2001-09-11
05:19:34
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-1
2001-09-11
05:32:04
Arch
[0931795]
A
ALPHA
|P2 NOTICE #4243326 |Etisalat link Dubai to Cairo failed impacting users WAN connectivity. Etisalat contacted Cairo PTT. OUTAGE START: 11 Sep 10:21 CET NO ETU. walho x4112 |172
2001-09-11
05:38:49
Arch
[0931795]
A
ALPHA
|P2 NOTICE #4243071 |Concert link from Madrid to Lisbon flapping. No user impact due to ISDN backup. EMEA WAN opened ticket with Concert. OUTAGE START: 11 Sep 7:30 CET NO ETU. walho x4112 |188
2001-09-11
05:40:10
Arch
[0931795]
A
ALPHA
drip root|ALERT !!!! /users is 94 % full on drip |ALERT !!!! /users is 94 % full on drip |89
2001-09-11
06:01:58
Arch
[0931795]
A
ALPHA
|OPC: caswell: /dev/vg01/lvol1, mounted at /apps, is at 97
2001-09-11
06:16:28
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] [mexclstr1a, mexclstr1b]
2001-09-11
06:24:49
Arch
[0931795]
A
ALPHA
|NETWORK-P1 NOTICE #4243506 |Mexicocity-gw is down impacting mexico city feild sales office call managers. Minimum business impact due to time of day. Support is investigating. OUTAGE START: 11 Sep 03:14 PT NO ETU. Ops 408-527-0007 |232
2001-09-11
06:28:28
Arch
[0931795]
A
ALPHA
|OPC: franklin: /dev/vg04/lvol1, mounted at /mail, is at 96
2001-09-11
06:38:19
Arch
[0931795]
A
ALPHA
|EIS * CISCO * web reports are done. | EIS CISCO web reports have been successfully generated and available to the users on production. Thanks EIS System.|154
2001-09-11
07:01:49
Arch
[0931795]
A
ALPHA
|OPC: kodak: System kodak seems to be down.
2001-09-11
07:23:58
Arch
[0931795]
A
ALPHA
[IS-HQ UP] [mexclstr1a, mexclstr1b]
2001-09-11
07:29:09
Arch
[0931795]
A
ALPHA
|NETWORK-P1 CLOSE PENDING #4243506 |Mexicocity-gw and FSO call managers are up. No impact due to time of day. Service was restored once power came up. OUTAGE START: 11 Sep 03:14 PT END: 11 Sep 04:25 PT Ops 408-527-0007 |219
2001-09-11
07:33:39
Arch
[0931795]
A
ALPHA
|P2 CLOSE #4228756 |Sita link Amsterdam to Prague no longer dropping packets. EMEA WAN restricted bandwith. OUTAGE START: 6 Sep 8:47 CET END: 7 Sep 12:53 CET walho x4112 |170
2001-09-11
07:55:09
Arch
[0931795]
A
ALPHA
|P2 CLOSE #4239098 | Http wwwin-emea-prod1:80 has been restore. Support investigating. OUTAGE START: 10 Sep 14:05 CET END: 11 Sep 11:00 CET josvenss x4112 |155
2001-09-11
08:02:18
Arch
[0931795]
A
ALPHA
|Start 5 am DB recoveries
2001-09-11
08:10:59
Arch
[0931795]
A
ALPHA
|OPC: mrpowers: /dev/vx/dsk/dgmrpowers/vrec5, mounted at /rec5, is at 95
2001-09-11
08:15:50
Arch
[0931795]
A
ALPHA
|P2 NOTICE #4243602 |CAP/JRun on wwwin-apps-1 and 2 unavailable, impacting servlet code execution for the CAP application. Support investigating. OUTAGE START: 10 Sep 23:01 PT END: Ops 408-527-0007 |198
2001-09-11
08:22:27
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-3 FTP
2001-09-11
08:22:29
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-3
2001-09-11
08:23:09
Arch
[0931795]
A
ALPHA
|P2 NOTICE #4243626 |Lima, Peru circuit down, impacting network connectivity for Lima Sales office. WAN support investigating with MCI. OUTAGE START: 11 Sep 00:46 PT NO ETU. Ops 408-527-0007 |191
2001-09-11
08:23:11
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-3 FTP
2001-09-11
08:23:48
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-3
2001-09-11
08:37:18
Arch
[0931795]
A
ALPHA
|P2 CLOSE #4243626 |Lima, Peru circuit back up. Network connectivity available to Lima Sales office. Circuit restored by MCI. OUTAGE START: 11 Sep 00:46 PT END: 11 Sep 05:34 PT Ops 408-527-0007 |194
2001-09-11
08:47:29
Arch
[0931795]
A
ALPHA
|OPC: kodak: System kodak seems to be down.
2001-09-11
09:06:08
Arch
[0931795]
A
ALPHA
|P2 CLOSE #4243602 |CAP/JRun on wwwin-apps-1&2 available. Servlet code execution available for CAP application. Support resta
2001-09-11
09:06:10
Arch
[0931795]
A
ALPHA
rted application to restore service. OUTAGE START: 10 Sep 23:01 PT END: 11 Sep 05:55 PT Ops 408-527-0007 |230
2001-09-11
09:09:59
Arch
[0931795]
A
ALPHA
|OPC: lada: System load has exceeded operations threshold. Load is at 0.64
2001-09-11
09:13:10
Arch
[0931795]
A
ALPHA
|OPC: minerva: /dev/vx/dsk/vroot, mounted at /, is at 97
2001-09-11
09:21:20
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 sendmail
2001-09-11
09:26:08
Arch
[0931795]
A
ALPHA
|OPC: kodak: System kodak seems to be down.
2001-09-11
09:27:03
Arch
[0931795]
A
ALPHA
|P2 CLOSE #4224767 |BVRUMXPD BVABTPD WEBIPD BVCECPD BVWPRPD BVPOSPD BVSSRPPD databases on Minerva back up. EMEA DBAdmins restored the services after cold backup. OUTAGE START: 5 Sep 11:45 CET END: 6 Sep 2:45 CET walho x4112 |224
2001-09-11
09:31:29
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-1 FTP
2001-09-11
09:31:31
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-1
2001-09-11
09:32:18
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-po
2001-09-11
09:32:19
Arch
[0931795]
A
ALPHA
c-1 FTP
2001-09-11
09:32:21
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-1
2001-09-11
09:33:05
Arch
[0931795]
A
ALPHA
|CCO-P1 NOTICE #4243775 |sendmail down on cco-sj-1. Minimal impact due to load balancing on cco-sj-2 and -3. Support investigating. OUTAGE START: 11 Sep 06:19 PT NO ETU. Ops 408.527.0007 |187
2001-09-11
09:35:29
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 Status Agent
2001-09-11
09:35:31
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-2 Status Agent
2001-09-11
09:35:33
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-3 Status Agent
2001-09-11
09:36:38
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-3 Status Agent
2001-09-11
09:36:40
Arch
[0931795]
A
ALPHA
|P2 NOTICE #4243796 |sendmail down on sj-msg-core-4. Minimal impact due to redundancy. Contacting support. OUTAGE START: 11 Sep 05:47 PT NO ETU. Ops 408.527.0007 |162
2001-09-11
09:37:20
Arch
[0931795]
A
ALPHA
amitra|bridge-erp: Please call bridge 408.527.2663 id700071 re: SJPROD db going down. Thanks, Al/Ops
2001-09-11
09:37:21
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] mfg-erp Oracle Database (SJPROD)
2001-09-11
09:39:08
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-3 Status Agent
2001-09-11
09:40:59
Arch
[0931795]
A
ALPHA
|ERP-P1 NOTICE #4243807 |SJPROD down. ERP production db unable to accept new connections and impacting access to Status Agent. Support escalation in progress. OUTAGE START: 11 Sep 6:33 PT NO ETU. Ops 408.527.0007 |213
2001-09-11
09:45:29
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CELC (prod), CTMP20, Case Open (caseopen), DPRGAppMonitor (DPRG-prod), Marketplace (CM), NPM, Pricing Agent (pricingagent)]
2001-09-11
09:46:18
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [IPC Extract (ipcextract), Lead Time Agent, SCC]
2001-09-11
09:46:59
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Auto Registration (autoreg), IPC Extract (ipcextract), Marketplace (CM), Pricing Agent (pricingagent), SCC]
2001-09-11
09:47:01
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 STA
2001-09-11
09:47:38
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 STA
2001-09-11
09:47:40
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 Case Update (caseupdate)
2001-09-11
09:48:38
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CTMP, Config-lim (config_lim), IPC Extract (ipcextract), M
2001-09-11
09:48:40
Arch
[0931795]
A
ALPHA
arketplace (CM), Pricing Agent (pricingagent), SVO Submit II]
2001-09-11
09:49:19
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Auto Registration (autoreg), CTMP, CTMP20, Case Open (caseopen), Case Update (caseupdate), Config-lim (config_lim), DPRGAppMonitor (DPRG-prod), IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), NPM]
2001-09-11
09:49:21
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Pricing Agent (pricingagent)]
2001-09-11
09:50:18
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 CELC (prod)
2001-09-11
09:51:38
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CELC (prod), CTMP20, DPRGAppMonitor (DPRG-prod)]
2001-09-11
09:52:31
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 Auto Registration (autoreg)
2001-09-11
09:53:49
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [CTMP20, DPRGAppMonitor (DPRG-prod), SVO Submit II]
2001-09-11
09:53:51
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 NPM
2001-09-11
09:54:31
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [CELC (prod), NPM]
2001-09-11
09:56:38
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [CTMP20, DPRGAppMonitor (DPRG-prod)]
2001-09-11
09:57:19
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [CELC (prod), COLT (Student), View Return (viewreturn)]
2001-09-11
09:57:21
Arch
[0931795]
A
ALPHA
[IS-HQ UP]
2001-09-11
09:57:23
Arch
[0931795]
A
ALPHA
cco-sj-1 COLT (Student)
2001-09-11
09:58:08
Arch
[0931795]
A
ALPHA
|OPC: topic-sj: System load has exceeded threshold. Load is at 20.89
2001-09-11
09:58:49
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Case Open (caseopen), Lead Time Agent, NPM, STA]
2001-09-11
09:59:49
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 COLT (Student)
2001-09-11
09:59:51
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 Lead Time Agent
2001-09-11
10:01:08
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 SCC OLD
2001-09-11
10:02:10
Arch
[0931795]
A
ALPHA
[IS-HQ UP] mfg-erp Oracle Database (SJPROD)
2001-09-11
10:02:12
Arch
[0931795]
A
ALPHA
|OpC: MAXRUNALARM - Job has exceeded Maximum Run Time. Hostname is uranium.cisco.com page sent to ops-duty
2001-09-11
10:02:14
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [SCC OLD, STA]
2001-09-11
10:02:29
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Config-lim (config_lim), ICS, SVO Status]
2001-09-11
10:03:08
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 COLT (Student)
2001-09-11
10:03:10
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Config-lim (config_lim), ICS, SVO Status]
2001-09-11
10:03:12
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 CTMP
2001-09-11
10:04:08
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Case Open (caseopen), View Return (
2001-09-11
10:04:10
Arch
[0931795]
A
ALPHA
viewreturn)]
2001-09-11
10:04:12
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 Case Update (caseupdate)
2001-09-11
10:05:08
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 Case Update (caseupdate)
2001-09-11
10:05:49
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 Service Order Status
2001-09-11
10:05:53
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Config-lim (config_lim), Service Order Status, Status Agent]
2001-09-11
10:05:57
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 Config-lim (config_lim)
2001-09-11
10:06:44
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 CTMP
2001-09-11
10:06:59
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-2 Status Agent
2001-09-11
10:07:49
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Case Open (caseopen), Case Update (caseupdate)]
2001-09-11
10:08:17
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-3 Status Agent
2001-09-11
10:08:31
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CTMP, Config-lim (config_lim), IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), Service Order Status, Status Agent]
2001-09-11
10:08:38
Arch
[0931795]
A
ALPHA
|ERP-P1 UPDATE/CLOSE PENDING #4243807 |SJPROD db instance on host high was hung and is now up. ERP production db is now able
2001-09-11
10:08:40
Arch
[0931795]
A
ALPHA
to accept new connections. Status Agent is also available. ERP_dba's restarted instance to restore service. -1/2
2001-09-11
10:08:42
Arch
[0931795]
A
ALPHA
OUTAG
2001-09-11
10:08:44
Arch
[0931795]
A
ALPHA
E START: 11 Sep 6:33 PT END: 11 Sep 7:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
10:08:46
Arch
[0931795]
A
ALPHA
|CCO-P1 UPDATE #4243775 |cco-sj-1 not responding. Support removing host from service to allow apps availibility on cco-sj-2 and -3. OUTAGE START: 11 Sep 06:19 PT NO ETU.
2001-09-11
10:08:47
Arch
[0931795]
A
ALPHA
Ops 408.527.0007 |187
2001-09-11
10:08:49
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [ICS, SCC OLD]
2001-09-11
10:08:51
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Auto Registration (autoreg), CTMP, Case Update (caseupdate), Config-lim (config_lim), ICS, IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), SCC OLD, Service Order Status, Status Agent]
2001-09-11
10:09:10
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [CELC (prod), Case Open (caseopen)]
2001-09-11
10:10:19
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [CTMP20, DPRGAppMonitor (DPRG-prod), NPM]
2001-09-11
10:12:49
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [CELC (prod), NPM]
2001-09-11
10:12:51
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 NPM
2001-09-11
10:15:38
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 CELC (prod)
2001-09-11
10:15:40
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 NPM
2001-09-11
10:16:59
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 DPRGAppMonitor (DPRG-prod)
2001-09-11
10:17:00
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [DPRGAppMonitor (DPRG-prod), NPM]
2001-09-11
10:21:55
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Auto Registration (autoreg), CELC (prod), COLT (Student), CTMP, CTMP20, Case Open (caseopen), Case Query (casequery), Case Update (caseupdate), Config-lim (config_lim), DPRGAppMonitor (DPRG-prod), H
2001-09-11
10:21:57
Arch
[0931795]
A
ALPHA
TTP (cco-admin)]
2001-09-11
10:21:59
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [HTTP (cco-sj-1-admin), HTTP (www.cisco.com), HTTPS (Secure Server), ICS, IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), NPM, Pricing Agent (pricingagent), SCC, SCC OLD, STA, Service Order Status]
2001-09-11
10:22:17
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Status Agent, View Return (viewreturn)]
2001-09-11
10:24:29
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Invoice agent (invoiceagent), SVO Status, SVO Submit II]
2001-09-11
10:25:29
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 [Aged Accounts Summary Tool (Age, DistiCredit, Submit Return (submitreturn), Upgrade Agent]
2001-09-11
10:35:27
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 sendmail
2001-09-11
10:50:42
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Auto Registration (autoreg), CEL
2001-09-11
10:50:43
Arch
[0931795]
A
ALPHA
C (prod), COLT (Student), CTMP, CTMP20, Case Open (caseopen), Case Query (casequery), Case Update (caseupdate), Config-lim (config_lim), DPRGAppMonitor (DPRG-prod), HTTP (cco-admin)]
2001-09-11
10:50:45
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [HTTP (cco-sj-1-admin), HTTP (www.cisco.com), HTTPS (Secure Server), ICS, IPC Extract (ipcextract), Lead Time Agent, Marketplace (CM), NPM, Pricing Agent (pricingagent), SCC, SCC OLD, STA, Service Order Status]
2001-09-11
10:51:28
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Status Agent, View Return (viewreturn)]
2001-09-11
10:51:55
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 [Aged Accounts Summary Tool (Age, Disti
2001-09-11
10:51:57
Arch
[0931795]
A
ALPHA
Credit, Invoice agent (invoiceagent), SVO Status, SVO Submit II, Submit Return (submitreturn), Upgrade Agent]
2001-09-11
10:58:58
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-3
2001-09-11
10:59:00
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-3 FTP
2001-09-11
10:59:51
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-3 FTP
2001-09-11
10:59:53
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-3
2001-09-11
11:29:19
Arch
[0931795]
A
ALPHA
|CCO-P1 CLOSE PENDING #4243775 |cco-sj-1 has been brought back in to service. Support continues to investigate cause of outage. OUTAGE START: 11 Sep 06:19 PT END: 11 Sep 08:00 PT Ops 408.527.0007 |196
2001-09-11
11:45:41
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-2 FTP
2001-09-11
11:45:43
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-2
2001-09-11
11:45:45
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-2 FTP
2001-09-11
11:45:49
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-2
2001-09-11
11:46:30
Arch
[0931795]
A
ALPHA
|P2 NOTICE/CLOSE #4244027 |Epage
2001-09-11
11:46:32
Arch
[0931795]
A
ALPHA
services were unavailable due to high load on all epage servers. OUTAGE START: 11 Sep 8:10 PT END: 11 Sep 8:32 PT Ops 408.527.0007 |164
2001-09-11
11:46:58
Arch
[0931795]
A
ALPHA
|OPC: uranium: /dev/vg01/lvol1, mounted at /apps, is at 97
2001-09-11
11:47:00
Arch
[0931795]
A
ALPHA
|V
2001-09-11
11:47:02
Arch
[0931795]
A
ALPHA
OICE-P1 NOTICE #4244044 |No calls able to go through to EMEA due to high load on national and international circuits. Voiceservices investigating. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 4085270007 |202
2001-09-11
11:58:41
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] cco-sj-1 Status Agent
2001-09-11
12:00:30
Arch
[0931795]
A
ALPHA
| BV_PB:Backup:info:18:00:02:RDCP:RDCP BVOLS to NVOLS
2001-09-11
12:00:32
Arch
[0931795]
A
ALPHA
[IS-HQ UP] cco-sj-1 Status Agent
2001-09-11
12:06:58
Arch
[0931795]
A
ALPHA
|POWER STATUS : No stage alerts at this time ||46
2001-09-11
12:10:51
Arch
[0931795]
A
ALPHA
|OPC: bus: Test, disregard
2001-09-11
12:38:17
Arch
[0931795]
A
ALPHA
|OPC: uranium: /dev/vg01/lvol1, mounted at /apps, is at 98
2001-09-11
12:47:17
Arch
[0931795]
A
ALPHA
|P2 NOTICE #4244245 |Network link to Bogota, Colombia down impacting access to local FSO. Contacting support. OUTAGE START: 11 Sep 9:32 PT NO ETU. Ops 408.527.0007 |164
2001-09-11
12:56:48
Arch
[0931795]
A
ALPHA
| BV_PB:Backup:info:18:56:50:
2001-09-11
12:56:50
Arch
[0931795]
A
ALPHA
PRESYNC:Break away backup plexes
2001-09-11
12:57:28
Arch
[0931795]
A
ALPHA
| BV_PB:Backup:info:18:57:22:SYNC:active resilvering
2001-09-11
13:16:03
Arch
[0931795]
A
ALPHA
|OPC: uranium: /dev/vg01/lvol1, mounted at /apps, is at 99
2001-09-11
13:27:50
Arch
[0931795]
A
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls to EMEA intermittent due to high load on national and international circuits. Voiceservices continuing to investigate. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 4085270007 |205
2001-09-11
13:41:18
Arch
[0931795]
A
ALPHA
|OPC: mrpowers: /dev/vx/dsk/dgmrpowers/vrec5, mounted at /rec5, is at 95
2001-09-11
14:04:48
Arch
[0931795]
A
ALPHA
|snipe 5AM recovery complete
2001-09-11
14:14:31
Arch
[0931795]
A
ALPHA
|OPC: lada: System load has exceeded operations threshold. Load is at 0.28
2001-09-11
14:20:07
Arch
[0931795]
A
ALPHA
|EIS * ASIA * web reports are done. | EIS ASIA web reports have been successfully generated and available to the users on production. Thanks EIS System.|152
2001-09-11
14:20:38
Arch
[0931795]
A
ALPHA
|NETWORK-P1 NO
2001-09-11
14:20:41
Arch
[0931795]
A
ALPHA
TICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network availability. SJ LAN support investigating issues with Code Red filters being removed. OUTAGE START: 11 Sep -1/2
2001-09-11
14:20:42
Arch
[0931795]
A
ALPHA
11:05 PT NO ETU. Ops 408.527.0007 |268 -2/2
2001-09-11
14:21:40
Arch
[0931795]
A
ALPHA
||SUCCESS:
2001-09-11
14:21:42
Arch
[0931795]
A
ALPHA
snipe RECOVER_STANDBY for PROD2 .|44
2001-09-11
14:24:20
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ladybird CCEWP
2001-09-11
14:24:25
Arch
[0931795]
A
ALPHA
|VOICE-P1 NOTICE #4244564 |Multiple lan switches bouncing in various Site 4 Bldgs. Site 4 experiencing intermittent network and IP phone availability. SJ LAN support investigating issues with Code Red fi
2001-09-11
14:24:27
Arch
[0931795]
A
ALPHA
lters. OUTAGE START: 11 Sep -1/2
2001-09-11
14:24:44
Arch
[0931795]
A
ALPHA
11:05 PT NO ETU. Ops 408.527.0007 |265 -2/2
2001-09-11
14:30:38
Arch
[0931795]
A
ALPHA
|P1 NOTICE #4244617 |CCEWP unavailable on ladybird impacting content delopyment to CCO. Support is investigating. OUTAGE START:
2001-09-11
14:30:40
Arch
[0931795]
A
ALPHA
11 Sep 11:21 PT NO ETU. Ops 408.527.0007 |169
2001-09-11
14:32:05
Arch
[0931795]
A
ALPHA
|OPC: radon: /dev/vg00/lvol6, mounted at /opt, is at 98
2001-09-11
14:32:40
Arch
[0931795]
A
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls between US and EMEA intermittent due to high load on national and international circuits. Voice Services investigating. Next update at 16:00 PT. OUTAGE START: 11 Sep 08:31 PT NO ETU. Ops 408.527.0007 |233
2001-09-11
14:32:42
Arch
[0931795]
A
ALPHA
|:BV_PB:info:20:33:12:Backup:BREAK: disassociate plexes
2001-09-11
14:33:05
Arch
[0931795]
A
ALPHA
|:BV_PB:info:20:33:27:Backup:DBONLINE: end hot backups/start dbs
2001-09-11
14:33:07
Arch
[0931795]
A
ALPHA
| BV_PB:Backup:info:20:32:20:SYNC:RESILVERING COMPLETE
2001-09-11
14:33:36
Arch
[0931795]
A
ALPHA
|:BV_PB:info:20:33:39:Backup:BMERGE: Attaching plexes to backup vols
2001-09-11
14:33:38
Arch
[0931795]
A
ALPHA
|:BV_PB:info:20:33:52:Backup:ALEX: execute alex rpf stores
2001-09-11
14:33:55
Arch
[0931795]
A
ALPHA
|P1::BV_PB Alexandria op-cards have been started, please monitor till success
2001-09-11
14:33:57
Arch
[0931795]
A
ALPHA
|:BV_PB:info:20:34:09:Backup:COMPLETE:ERP backup script complete
2001-09-11
14:39:48
Arch
[0931795]
A
ALPHA
|AUTOSYS(PRD): job=qtc_XHOLDS starting_state=1440 seconds. Please modify job to failure, then contact job_owner to verify job restart status . ||144
2001-09-11
14:40:23
Arch
[0931795]
A
ALPHA
|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |217
2001-09-11
14:40:50
Arch
[0931795]
A
ALPHA
Oracle on Xenon|AUTOSYS(PRD): job=qtc_XHOLDS starting_state=1447 seconds. Please modify job to failure, then contact job_owner to verify job restart status . ||159
2001-09-11
14:40:53
Arch
[0931795]
A
ALPHA
|VOICE-P1 UPDATE #4244564 |Site 4 network service restored and IP ph
2001-09-11
14:40:55
Arch
[0931795]
A
ALPHA
one services stabilizing. SJ LAN support working on desktop clusters to restore redundancy. OUTAGE START: 11 Sep 11:05 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
14:49:48
Arch
[0931795]
A
ALPHA
|AUTOSYS(PRD): job=qtc_XHOLDS starting_state=2040 seconds. Please modify job to failure, then contact job_owner to verify job restart status . ||144
2001-09-11
14:51:06
Arch
[0931795]
A
ALPHA
Oracle on Xenon|AUTOSYS(PRD): job=qtc_XHOLDS starting_state=2048 seconds. Please modify job to failure, then contact jo
2001-09-11
14:51:08
Arch
[0931795]
A
ALPHA
b_owner to verify job restart status . ||159
2001-09-11
15:03:27
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ladybird CCEWP
2001-09-11
15:10:27
Arch
[0931795]
A
ALPHA
|P1 CLOSE PENDING #4244617 |C
2001-09-11
15:10:29
Arch
[0931795]
A
ALPHA
CEWP available on ladybird. Clients can deploy content to CCO. Support restarted applic to restore service. OUTAGE START: 11 Sep 11:21 PT END: 11 Sep 12:05 PT Ops 408.527.0007 |205
2001-09-11
15:17:19
Arch
[0931795]
A
ALPHA
|NETWORK-P1 UPDATE #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |226
2001-09-11
15:18:01
Arch
[0931795]
A
ALPHA
|VOICE-P1 UPDATE #4244564 |Site 4 network service restor
2001-09-11
15:18:03
Arch
[0931795]
A
ALPHA
ed and IP phone services stabilized. SJ LAN support working on upgrading code on redundant gateway. OUTAGE START: 11 Sep 11:05 PT ETU: 11 Sep 13:30 PT Ops 408.527.0007 |224
2001-09-11
15:28:06
Arch
[0931795]
A
ALPHA
|OPC: uranium: /dev/vg01/lvol1, mounted at /apps, is at 100
2001-09-11
15:33:57
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-1 FTP
2001-09-11
15:33:59
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-1
2001-09-11
15:34:31
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-1
2001-09-11
15:34:32
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-1 FTP
2001-09-11
15:34:34
Arch
[0931795]
A
ALPHA
|NETWORK-P1 CLOSE PENDING #42
2001-09-11
15:34:36
Arch
[0931795]
A
ALPHA
44564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |233
2001-09-11
15:34:53
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] sjc-i2prod
2001-09-11
15:36:01
Arch
[0931795]
A
ALPHA
|VOICE-P1 CLOSE PENDING #4244564 |Site 4 network service restored and IP phone services stabilized. SJ LAN support upgraded code on gateway to restore redundancy. OUTAGE START: 11 Sep 11:05 PT END: 11 Sep 12:30 PT Ops 408.527.0007 |231
2001-09-11
15:37:38
Arch
[0931795]
A
ALPHA
|OPC: tahoe: /dev/vg02/vusers, mounted at /users, is at 97
2001-09-11
15:40:36
Arch
[0931795]
A
ALPHA
|OPC: tahoe: /dev/vg02/vusers, mounted at /users, is at 98
2001-09-11
15:41:16
Arch
[0931795]
A
ALPHA
|P1 NOTICE #4244953 |sjc-i2prod has system resource issues impacting access to Demand Planning tool to forecast product demand. Support rebooting host. OUTAGE START: 11 Sep 12:32 PT ETU: 11 Sep 14:00 PT Ops 40
2001-09-11
15:41:17
Arch
[0931795]
A
ALPHA
8.527.0007 |220
2001-09-11
15:45:02
Arch
[0931795]
A
ALPHA
|NETWORK-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408.527.0007 |207
2001-09-11
15:45:42
Arch
[0931795]
A
ALPHA
|VOICE-P1 NOTICE #4245013 |SJ Site 4 Phase B experiencing network and voice service degradation. SJ network and voice services support investigating. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408.527.0007 |205
2001-09-11
15:53:27
Arch
[0931795]
A
ALPHA
|OPC: eman-dev: /dev/vx/dsk/dgemandev/vusers, mounted at /users, is at 96
2001-09-11
15:55:08
Arch
[0931795]
A
ALPHA
|OPC: morris: /dev/vx/dsk/ROOTVOL, mounted at /, is at 100
2001-09-11
15:59:00
Arch
[0931795]
A
ALPHA
|VOICE-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2
2001-09-11
15:59:02
Arch
[0931795]
A
ALPHA
527.0007 |242 -2/2
2001-09-11
16:02:27
Arch
[0931795]
A
ALPHA
|NETWORK-P1 UPDATE #4245013 |SJ Site 4 Phase B experiencing desktop network and voice service degradation. Support working to remove all Code Red filters from Site 1, 2, 3, 4, 5 in SJ LAN. OUTAGE START: 11 Sep 12:30 PT NO ETU. Ops 408. -1/2
2001-09-11
16:02:28
Arch
[0931795]
A
ALPHA
527.0007 |244 -2/2
2001-09-11
16:13:27
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-3 FTP
2001-09-11
16:13:28
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-3
2001-09-11
16:14:08
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-3 FTP
2001-09-11
16:14:10
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-3
2001-09-11
16:16:19
Arch
[0931795]
A
ALPHA
|NETWORK-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all Code Red filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2
2001-09-11
16:16:21
Arch
[0931795]
A
ALPHA
13:10 PT Ops 408.527.0007 |260 -2/2
2001-09-11
16:18:00
Arch
[0931795]
A
ALPHA
|VOICE-P1 CLOSE PENDING #4245013 |SJ Site 4 Phase B desktop network and voice services stable and restored. Support removed all Code Red
2001-09-11
16:18:02
Arch
[0931795]
A
ALPHA
filters from Site 4 in SJ LAN to stabilize services. OUTAGE START: 11 Sep 12:30 PT END: 11 Sep -1/2
2001-09-11
16:18:04
Arch
[0931795]
A
ALPHA
13:10 PT Ops 408.527.0007 |258 -2/2
2001-09-11
16:21:06
Arch
[0931795]
A
ALPHA
|OPC: peerless: /dev/vx/dsk/dgpe
2001-09-11
16:21:08
Arch
[0931795]
A
ALPHA
erless/vusers, mounted at /users, is at 100
2001-09-11
16:36:26
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-2
2001-09-11
16:36:28
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-2 FTP
2001-09-11
16:37:08
Arch
[0931795]
A
ALPHA
[IS-HQ UP] [ftp-poc-2, sjc-i2prod]
2001-09-11
16:37:10
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-2 FTP
2001-09-11
16:40:36
Arch
[0931795]
A
ALPHA
|OPC: escort: /dev/vx/dsk/dgescort/users, mounted at /users, is at 95
2001-09-11
16:46:34
Arch
[0931795]
A
ALPHA
|OPC: lbj: /dev/vx/dsk/documdg/vtddoc07, mounted at /dm_vault/cisco_prodt/cisco_td_doc_07, is at 95
2001-09-11
17:04:54
Arch
[0931795]
A
ALPHA
|OPC: caswell: /dev/vg01/lvol1, mounted at /apps, is at 98
2001-09-11
17:12:37
Arch
[0931795]
A
ALPHA
|P
2001-09-11
17:12:39
Arch
[0931795]
A
ALPHA
1 UPDATE #4244953 |sjc-i2prod still unavailable impacting access to Demand Planning tool to forecast product demand. Support continues to investigate. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |208
2001-09-11
17:16:47
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ladybird CCEWP
2001-09-11
17:20:37
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ladybird CCEWP
2001-09-11
18:42:54
Arch
[0931795]
A
ALPHA
|VOICE-P1 UPDATE #4244044 |Calls between US and EMEA intermittent due to high load on national and international circuits. MCI and Sprint expect phone lines to become stable by tomorrow afternoon. Voice Services support monitoring -1/2
2001-09-11
18:42:56
Arch
[0931795]
A
ALPHA
OUTAGE START: 11 Sep 08:31 PT ETU: 12 Sep 12:00 PT Ops 408.527.0007 |299 -2/2
2001-09-11
18:47:07
Arch
[0931795]
A
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support working to fix hive corruption to restore services to host and application. OUTAGE START: 11 Sep 12:32 PT NO ETU. djaramil 408.527.0007 |220
2001-09-11
19:09:46
Arch
[0931795]
A
ALPHA
|OPC: marvel: /dev/vgarch/v20000-01, mounted at /archive1, is at 92
2001-09-11
19:15:16
Arch
[0931795]
A
ALPHA
|OPC: mcoapp-sj-2: /dev/vx/dsk/rootdg/log, mounted at /fs/log, is at 99
2001-09-11
19:18:37
Arch
[0931795]
A
ALPHA
|OPC: mcoapp-sj-2: /dev/vx/dsk/rootdg/log, mounted at /fs/log, is at 100
2001-09-11
19:19:37
Arch
[0931795]
A
ALPHA
|OPC: lada: System load has exceeded operations threshold. Load is at 0.73
2001-09-11
19:43:27
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] sjc-i2prod
2001-09-11
19:54:07
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-3 FTP
2001-09-11
19:54:08
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-3
2001-09-11
19:54:57
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-3 FTP
2001-09-11
19:55:37
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-3
2001-09-11
20:02:16
Arch
[0931795]
A
ALPHA
|OpC PSoft Alert CSPROD00 SQR Program NOTE: csprod00 is starting. (TEST, PLEASE DISREGARD)
2001-09-11
20:02:18
Arch
[0931795]
A
ALPHA
|Start 5 pm DB recoveries
2001-09-11
20:12:36
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] outback-ams1
2001-09-11
20:19:27
Arch
[0931795]
A
ALPHA
|P1 NOTICE #4245646 |outback-ams and outback-ams1 down impacting availability of NATKIT - NSA's customer network troubleshooting application - in EMEA. Support investigating
2001-09-11
20:19:29
Arch
[0931795]
A
ALPHA
. OUTAGE START: 11 Sep 17:09 PT NO ETU. Ops 408-527-0007 |230
2001-09-11
20:20:01
Arch
[0931795]
A
ALPHA
|OPC: solstice: System solstice seems to be down.
2001-09-11
20:28:59
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] brussels sendmail
2001-09-11
20:29:01
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] glasgow sendmail
2001-09-11
20:29:03
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] johannesburg sendmail
2001-09-11
20:29:04
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] london sendmail
2001-09-11
20:29:06
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] madrid sendmail
2001-09-11
20:29:08
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] nordic sendmail
2001-09-11
20:29:10
Arch
[0931795]
A
ALPHA
[IS-HQ UP] brussels sendmail
2001-09-11
20:29:36
Arch
[0931795]
A
ALPHA
[IS-HQ UP] glasgow sendmail
2001-09-11
20:29:38
Arch
[0931795]
A
ALPHA
[IS-HQ UP] johannesburg sendmail
2001-09-11
20:29:40
Arch
[0931795]
A
ALPHA
[IS-HQ UP] london sendmail
2001-09-11
20:29:42
Arch
[0931795]
A
ALPHA
[IS-HQ UP] madrid sendmail
2001-09-11
20:29:44
Arch
[0931795]
A
ALPHA
[IS-HQ UP] nordic sendmail
2001-09-11
20:37:16
Arch
[0931795]
A
ALPHA
|OPC: blender-sj1: /dev/vx/dsk/usr, mounted at /usr, is at 96
2001-09-11
20:49:46
Arch
[0931795]
A
ALPHA
[IS-HQ UP] outback-ams1
2001-09-11
21:04:45
Arch
[0931795]
A
ALPHA
|OpC PSoft Alert CSPROD97 SQR Program NOTE: csprod97 is Ending. (TEST, PLEASE DISREGARD)
2001-09-11
21:08:45
Arch
[0931795]
A
ALPHA
|OPC: prowler: System prowler seems to be down.
2001-09-11
21:09:25
Arch
[0931795]
A
ALPHA
|OPC: harrier: System harrier seems to be down.
2001-09-11
21:10:25
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] [harrier, outback-ams1, prowler]
2001-09-11
21:14:55
Arch
[0931795]
A
ALPHA
[IS-HQ UP]
2001-09-11
21:14:57
Arch
[0931795]
A
ALPHA
[harrier, prowler]
2001-09-11
21:20:27
Arch
[0931795]
A
ALPHA
[IS-HQ UP] outback-ams1
2001-09-11
21:34:17
Arch
[0931795]
A
ALPHA
|P1 UPDATE/CLOSE PENDING #4245646 |NATKIT, NSA's customer network troubleshooting application in EMEA is available. Support rebooted servers to restore service. OUTAGE START: 11 Sep 17:09 PT END: 11 Sep 18:28 PT Ops 408-527-0007 |229
2001-09-11
21:35:45
Arch
[0931795]
A
ALPHA
|OpC PSoft Alert CSPROD99 SQR Program NOTE: csprod99 is Ending. (TEST, PLEASE DISREGARD)
2001-09-11
22:18:32
Arch
[0931795]
A
ALPHA
|P1 UPDATE #4244953 |sjc-i2prod and Demand Planning tool still unavailable. Support attempting to bring dev server into production to replace corrupted server. OUTAGE START: 11 Sep 12:32 PT NO ETU. Ops 408.527.0007 |215
2001-09-11
22:25:36
Arch
[0931795]
A
ALPHA
|OpC PSoft Alert CSREPT00 SQR Program Note: csrept00 Starting. (TEST, PLEASE DISREGARD)
2001-09-11
22:52:36
Arch
[0931795]
A
ALPHA
[IS-HQ UP] sjc-i2prod
2001-09-11
22:54:56
Arch
[0931795]
A
ALPHA
|EIS * EUROPE * web reports are done. | EIS EUROPE web reports have been successfully generated and available to the users on production. Thanks EIS System.|156
2001-09-11
23:00:15
Arch
[0931795]
A
ALPHA
|unable to umount /db506B on elroy to begin backup urgent attention needed
2001-09-11
23:19:15
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] sjc-i2prod
2001-09-11
23:21:15
Arch
[0931795]
A
ALPHA
[IS-HQ UP] sjc-i2prod
2001-09-11
23:25:35
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] sjc-i2prod
2001-09-11
23:31:56
Arch
[0931795]
A
ALPHA
[IS-HQ UP] sjc-i2prod
2001-09-11
23:31:58
Arch
[0931795]
A
ALPHA
|OpC: MAXRUNALARM - Job has exceeded Maximum Run Time. Hostname is hurley.cisco.com page sent to ops-duty
2001-09-11
23:32:56
Arch
[0931795]
A
ALPHA
|P1 CLOSE PENDING #4244953 |sjc-i2prod and Demand Planning tool are up. Support replaced corrupted server with development server to restore service. OUTAGE START: 11 Sep 12:32 PT END: 11 Sep 20:31 PT Ops 408.527.0007 |218
2001-09-11
23:36:05
Arch
[0931795]
A
ALPHA
|OPC: wwwin-emea-dev: /dev/vx/dsk/wdevdg/vwdev, mounted at /fs/wdev, is at 95
2001-09-11
23:42:15
Arch
[0931795]
A
ALPHA
|OPC: wwwin-emea-dev: /dev/vx/dsk/wdevdg/vwdev, mounted at /fs/wdev, is at 96
2001-09-12
00:01:56
Arch
[0931795]
A
ALPHA
| BV_PB:Backup:info:06:01:03:RDCP:RDCP BVOLS to NVOLS
2001-09-12
00:24:31
Arch
[0931795]
A
ALPHA
|OPC: lada: System load has exceeded operations threshold. Load is at 0.69
2001-09-12
00:47:16
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-2 FTP
2001-09-12
00:47:18
Arch
[0931795]
A
ALPHA
[IS-HQ DOWN] ftp-poc-2
2001-09-12
00:47:20
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-2 FTP
2001-09-12
00:47:56
Arch
[0931795]
A
ALPHA
[IS-HQ UP] ftp-poc-2
2001-09-12
00:57:05
Arch
[0931795]
A
ALPHA
| BV_PB:Backup:info:06:57:24:PRESYNC:Break away backup plexes
2001-09-12
00:57:54
Arch
[0931795]
A
ALPHA
| BV_PB:Backup:info:06:57:59:SYNC:active resilvering
2001-09-12
01:35:05
Arch
[0931795]
A
ALPHA
|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 95
2001-09-12
01:35:44
Arch
[0931795]
A
ALPHA
|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 96
2001-09-12
01:37:14
Arch
[0931795]
A
ALPHA
|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 97
2001-09-12
01:38:25
Arch
[0931795]
A
ALPHA
|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 98
2001-09-12
01:40:25
Arch
[0931795]
A
ALPHA
|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 99
2001-09-12
01:41:05
Arch
[0931795]
A
ALPHA
|OPC: mcoolap-sj-1: /dev/vx/dsk/mcoolapdg/misc, mounted at /fs/misc, is at 100
2001-09-12
01:47:25
Arch
[0931795]
A
ALPHA
||SUCCESS: snipe RECOVER_STANDBY for PROD2 .|44
2001-09-12
01:48:44
Arch
[0931795]
A
ALPHA
| 5PM recovery complete
2001-09-12
02:16:25
Arch
[0931795]
A
ALPHA
|Start do_cycle.
2001-09-12
02:17:04
Arch
[0931795]
A
ALPHA
||SUCCESS: snipe RECOVER_STANDBY for PROD .|43
2001-09-12
02:32:55
Arch
[0931795]
A
ALPHA
|OPC: rex: /dev/vx/dsk/sundg/v122sun-12, mounted at /beyond/122/sun12, is at 95
2001-09-12
02:35:25
Arch
[0931795]
A
ALPHA
| BV_PB:Backup:info:08:35:21:SYNC:RESILVERING COMPLETE
2001-09-12
02:36:04
Arch
[0931795]
A
ALPHA
|:BV_PB:info:08:36:12:Backup:BREAK: disassociate plexes
2001-09-12
02:36:06
Arch
[0931795]
A
ALPHA
|:BV_PB:info:08:36:28:Backup:DBONLINE: end hot backups/start dbs
2001-09-12
02:36:08
Arch
[0931795]
A
ALPHA
|:BV_PB:info:08:36:40:Backup:BMERGE: Attaching plexes to backup vols
2001-09-12
02:36:55
Arch
[0931795]
A
ALPHA
|:BV_PB:info:08:36:53:Backup:COMPLETE:ERP backup script complete