环境:
(1),Oracle 10g RAC + Oracle 10g RAC Dataguard最大性能模式配置
(2),rac1,rac2 Primary Database
(3),vmrac1,vmrac2 Physical Standby Database
(4),vmrac1 运行MRP
现状:
rac1,rac2因故障崩溃,已无法使用,现将备库进行启动,实现切换.
主库归档日志都已经完全传输到备库,备库上无archive gap.
备库vmrac1,vmrac2为mount状态,MRP运行在vmrac1上,vmrac1和vmrac2的alter日志的最后状态如下.
Fri Aug 10 15:13:49 2012
alter database recover managed standby database disconnect from session
Fri Aug 10 15:13:49 2012
Attempt to start background Managed Standby Recovery process (RDPSDB1)
MRP0 started with pid=36, OS id=9735
Fri Aug 10 15:13:49 2012
MRP0: Background Managed Standby Recovery process started (RDPSDB1)
Fri Aug 10 15:13:54 2012
Managed Standby Recovery not using Real Time Apply
Fri Aug 10 15:13:55 2012
Completed: alter database recover managed standby database disconnect from session
Fri Aug 10 15:13:56 2012
Media Recovery Waiting for thread 2 sequence 44
Fri Aug 10 15:13:54 2012
Managed Standby Recovery not using Real Time Apply
分析:
主库由于故障crash掉,因为使用最大性能模式,配置使用归档日志进行实现,所以当主库crash时,肯定有redo或归档日志无法全部传输到备库,意味着数据丢失.需要强行进行切换.
Failover步骤
1,查看当前归档应用现状
select thread#,sequence#,name,first_change# FSCN,first_time,next_change# NSCN,next_time,applied from v$archived_log order by 1,3
THREAD# SEQUENCE# NAME FSCN FIRST_TIM NSCN NEXT_TIME APP
---------- ---------- -------------------------------------------------- ---------- --------- ---------- --------- ---
1 22 +RECOVERY/rdpsdbst/1_22_774628717.dbf 1676860 17-FEB-12 1863631 19-FEB-12 YES
1 23 +RECOVERY/rdpsdbst/1_23_774628717.dbf 1863631 19-FEB-12 2004434 20-FEB-12 YES
1 24 +RECOVERY/rdpsdbst/1_24_774628717.dbf 2004434 20-FEB-12 2102888 20-FEB-12 YES
1 25 +RECOVERY/rdpsdbst/1_25_774628717.dbf 2102888 20-FEB-12 2230411 21-FEB-12 YES
1 26 +RECOVERY/rdpsdbst/1_26_774628717.dbf 2230411 21-FEB-12 2381707 22-FEB-12 YES
1 27 +RECOVERY/rdpsdbst/1_27_774628717.dbf 2381707 22-FEB-12 2418101 22-FEB-12 YES
1 28 +RECOVERY/rdpsdbst/1_28_774628717.dbf 2418101 22-FEB-12 2542375 23-FEB-12 YES
1 29 +RECOVERY/rdpsdbst/1_29_774628717.dbf 2542375 23-FEB-12 2691527 24-FEB-12 YES
1 30 +RECOVERY/rdpsdbst/1_30_774628717.dbf 2691527 24-FEB-12 2820123 24-FEB-12 YES
1 31 +RECOVERY/rdpsdbst/1_31_774628717.dbf 2820123 24-FEB-12 2890467 25-FEB-12 YES
1 32 +RECOVERY/rdpsdbst/1_32_774628717.dbf 2890467 25-FEB-12 2918970 25-FEB-12 YES
THREAD# SEQUENCE# NAME FSCN FIRST_TIM NSCN NEXT_TIME APP
---------- ---------- -------------------------------------------------- ---------- --------- ---------- --------- ---
1 33 +RECOVERY/rdpsdbst/1_33_774628717.dbf 2918970 25-FEB-12 2951731 26-FEB-12 YES
1 34 +RECOVERY/rdpsdbst/1_34_774628717.dbf 2951731 26-FEB-12 2983302 26-FEB-12 YES
1 35 +RECOVERY/rdpsdbst/1_35_774628717.dbf 2983302 26-FEB-12 3029193 26-FEB-12 YES
1 36 +RECOVERY/rdpsdbst/1_36_774628717.dbf 3029193 26-FEB-12 3056600 27-FEB-12 YES
1 37 +RECOVERY/rdpsdbst/1_37_774628717.dbf 3056600 27-FEB-12 3095754 27-FEB-12 YES
1 38 +RECOVERY/rdpsdbst/1_38_774628717.dbf 3095754 27-FEB-12 3118564 28-FEB-12 YES
1 39 +RECOVERY/rdpsdbst/1_39_774628717.dbf 3118564 28-FEB-12 3144789 28-FEB-12 YES
1 40 +RECOVERY/rdpsdbst/1_40_774628717.dbf 3144789 28-FEB-12 3173740 28-FEB-12 YES
1 41 +RECOVERY/rdpsdbst/1_41_774628717.dbf 3173740 28-FEB-12 3199907 28-FEB-12 YES
1 42 +RECOVERY/rdpsdbst/1_42_774628717.dbf 3199907 28-FEB-12 3248780 29-FEB-12 YES
1 43 +RECOVERY/rdpsdbst/1_43_774628717.dbf 3248780 29-FEB-12 3288280 29-FEB-12 YES
THREAD# SEQUENCE# NAME FSCN FIRST_TIM NSCN NEXT_TIME APP
---------- ---------- -------------------------------------------------- ---------- --------- ---------- --------- ---
1 44 +RECOVERY/rdpsdbst/1_44_774628717.dbf 3288280 29-FEB-12 3302788 29-FEB-12 YES
1 45 +RECOVERY/rdpsdbst/1_45_774628717.dbf 3302788 29-FEB-12 3349925 01-MAR-12 YES
1 46 +RECOVERY/rdpsdbst/1_46_774628717.dbf 3349925 01-MAR-12 3405795 02-MAR-12 YES
1 47 +RECOVERY/rdpsdbst/1_47_774628717.dbf 3405795 02-MAR-12 3435621 02-MAR-12 YES
1 48 +RECOVERY/rdpsdbst/1_48_774628717.dbf 3435621 02-MAR-12 3478738 02-MAR-12 YES
1 49 +RECOVERY/rdpsdbst/1_49_774628717.dbf 3478738 02-MAR-12 3554415 02-MAR-12 YES
1 50 +RECOVERY/rdpsdbst/1_50_774628717.dbf 3554415 02-MAR-12 3649783 03-MAR-12 YES
1 51 +RECOVERY/rdpsdbst/1_51_774628717.dbf 3649783 03-MAR-12 3672859 03-MAR-12 YES
1 52 +RECOVERY/rdpsdbst/1_52_774628717.dbf 3672859 03-MAR-12 3777134 06-MAR-12 YES
1 53 +RECOVERY/rdpsdbst/1_53_774628717.dbf 3777134 06-MAR-12 3807330 06-MAR-12 NO
1 54 +RECOVERY/rdpsdbst/1_54_774628717.dbf 3807330 06-MAR-12 3831734 06-MAR-12 NO
THREAD# SEQUENCE# NAME FSCN FIRST_TIM NSCN NEXT_TIME APP
---------- ---------- -------------------------------------------------- ---------- --------- ---------- --------- ---
2 21 +RECOVERY/rdpsdbst/2_21_774628717.dbf 1676856 17-FEB-12 1706294 18-FEB-12 YES
2 22 +RECOVERY/rdpsdbst/2_22_774628717.dbf 1706294 18-FEB-12 1784495 18-FEB-12 YES
2 23 +RECOVERY/rdpsdbst/2_23_774628717.dbf 1784495 18-FEB-12 1878778 19-FEB-12 YES
2 24 +RECOVERY/rdpsdbst/2_24_774628717.dbf 1878778 19-FEB-12 2015866 20-FEB-12 YES
2 25 +RECOVERY/rdpsdbst/2_25_774628717.dbf 2015866 20-FEB-12 2145526 20-FEB-12 YES
2 26 +RECOVERY/rdpsdbst/2_26_774628717.dbf 2145526 20-FEB-12 2221645 21-FEB-12 YES
2 27 +RECOVERY/rdpsdbst/2_27_774628717.dbf 2221645 21-FEB-12 2337535 21-FEB-12 YES
2 28 +RECOVERY/rdpsdbst/2_28_774628717.dbf 2337535 21-FEB-12 2377288 21-FEB-12 YES
2 29 +RECOVERY/rdpsdbst/2_29_774628717.dbf 2377288 21-FEB-12 2504231 23-FEB-12 YES
2 30 +RECOVERY/rdpsdbst/2_30_774628717.dbf 2504231 23-FEB-12 2610098 23-FEB-12 YES
2 31 +RECOVERY/rdpsdbst/2_31_774628717.dbf 2610098 23-FEB-12 2679030 23-FEB-12 YES
THREAD# SEQUENCE# NAME FSCN FIRST_TIM NSCN NEXT_TIME APP
---------- ---------- -------------------------------------------------- ---------- --------- ---------- --------- ---
2 32 +RECOVERY/rdpsdbst/2_32_774628717.dbf 2679030 23-FEB-12 2785848 24-FEB-12 YES
2 33 +RECOVERY/rdpsdbst/2_33_774628717.dbf 2785848 24-FEB-12 2890466 25-FEB-12 YES
2 34 +RECOVERY/rdpsdbst/2_34_774628717.dbf 2890466 25-FEB-12 2952029 26-FEB-12 YES
2 35 +RECOVERY/rdpsdbst/2_35_774628717.dbf 2952029 26-FEB-12 3057053 27-FEB-12 YES
2 36 +RECOVERY/rdpsdbst/2_36_774628717.dbf 3057053 27-FEB-12 3145356 28-FEB-12 YES
2 37 +RECOVERY/rdpsdbst/2_37_774628717.dbf 3145356 28-FEB-12 3248781 29-FEB-12 YES
2 38 +RECOVERY/rdpsdbst/2_38_774628717.dbf 3248781 29-FEB-12 3349927 01-MAR-12 YES
2 39 +RECOVERY/rdpsdbst/2_39_774628717.dbf 3349927 01-MAR-12 3478739 02-MAR-12 YES
2 40 +RECOVERY/rdpsdbst/2_40_774628717.dbf 3478739 02-MAR-12 3516045 02-MAR-12 YES
2 41 +RECOVERY/rdpsdbst/2_41_774628717.dbf 3516045 02-MAR-12 3554302 02-MAR-12 YES
2 42 +RECOVERY/rdpsdbst/2_42_774628717.dbf 3554302 02-MAR-12 3588411 03-MAR-12 YES
THREAD# SEQUENCE# NAME FSCN FIRST_TIM NSCN NEXT_TIME APP
---------- ---------- -------------------------------------------------- ---------- --------- ---------- --------- ---
2 43 +RECOVERY/rdpsdbst/2_43_774628717.dbf 3588411 03-MAR-12 3777249 06-MAR-12 YES
56 rows selected.
2,关闭出运行MRP外的其他节点,这里关闭vmrac2数据库实例.
[oracle@vmrac1 ~]$ srvctl stop instance -d RDPSDBST -i "RDPSDB2" -o immediate
3,在MRP节点上关闭apply.在vmrac1上,注意alert日志.
SQL>alter database recover managed standby database cancel;
Fri Aug 10 16:11:32 2012
alter database recover managed standby database cancel
Fri Aug 10 16:11:34 2012
MRP0: Background Media Recovery cancelled with status 16037
Fri Aug 10 16:11:34 2012
Errors in file /u01/app/oracle/admin/RDPSDB/bdump/rdpsdb1_mrp0_9735.trc:
ORA-16037: user requested cancel of managed recovery operation
Recovery interrupted!
Fri Aug 10 16:11:35 2012
Waiting for MRP0 pid 9735 to terminate
Waiting for MRP0 pid 9735 to terminate
Waiting for MRP0 pid 9735 to terminate
Waiting for MRP0 pid 9735 to terminate
Fri Aug 10 16:11:38 2012
Errors in file /u01/app/oracle/admin/RDPSDB/bdump/rdpsdb1_mrp0_9735.trc:
ORA-16037: user requested cancel of managed recovery operation
Fri Aug 10 16:11:38 2012
MRP0: Background Media Recovery process shutdown (RDPSDB1)
Fri Aug 10 16:11:39 2012
Managed Standby Recovery Canceled (RDPSDB1)
Completed: alter database recover managed standby database cancel
4,将standby database强制转换成primary database
SQL>alter database recover managed standby database finish;
Fri Aug 10 16:17:59 2012
alter database recover managed standby database finish
Fri Aug 10 16:17:59 2012
SKIP STANDBY LOGFILE option no longer needed for RECOVERFINISH. Option ignored
Fri Aug 10 16:17:59 2012
Attempt to do a Terminal Incomplete Recovery (RDPSDB1)
Fri Aug 10 16:17:59 2012
Media Recovery Start: Managed Standby Recovery (RDPSDB1)
Fri Aug 10 16:17:59 2012
Managed Standby Recovery not using Real Time Apply
Fri Aug 10 16:18:02 2012
Media Recovery Waiting for thread 2 sequence 44
Resetting archiving SCN from 0:3831734 to 0:3777249
Terminal Recovery timestamp is '08/10/2012 16:18:02'
Terminal Recovery: applying standby redo logs.
Terminal Recovery: thread 2 seq# 44 redo required
Terminal Recovery: End-Of-Redo log allocation
MRP: Validating standby redo logfile 8
Fri Aug 10 16:18:02 2012
Media Recovery Log +DATA/rdpsdbst/onlinelog/group_8.267.774894163
Terminal Recovery: log 8 reserved for thread 2 sequence 44
Identified End-Of-Redo for thread 2 sequence 44
Terminal Recovery: thread 1 seq# 53 redo required
Terminal Recovery: +RECOVERY/rdpsdbst/1_53_774628717.dbf
Fri Aug 10 16:18:07 2012
Incomplete recovery applied all redo ever generated.
Recovery completed through change 3777250
Fri Aug 10 16:18:08 2012
Media Recovery Complete (RDPSDB1)
MRP: Validating standby redo logfile 5
Fri Aug 10 16:18:08 2012
RFS LogMiner: Client disabled from further notification
Fri Aug 10 16:18:09 2012
Terminal Recovery: log 5 reserved for thread 1 sequence 54
Terminal recovery recovered to last consistent point in redo stream
But it did not apply all redo; some data may be lost
Terminal Recovery: successful completion
Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
Resetting standby activation ID 1598029795 (0x5f3fffe3)
Completed: alter database recover managed standby database finish
这个指令告诉standby database的MRP,不要再等待redo了,尽可能多的应用现有的redo,并把End-Of-Redo(EOR)记录在日志头.
一旦standby应用了这个EOR记录之后,MRP进程就会结束退出,剩下的就是和正常的switchover相同的操作了.
5,现在standby database处于EOR记录状态,正常switchover.
SQL>alter database commit to switchover to primary with session shutdown;
Fri Aug 10 16:32:03 2012
alter database commit to switchover to primary with session shutdown
Fri Aug 10 16:32:03 2012
ALTER DATABASE SWITCHOVER TO PRIMARY (RDPSDB1)
Fri Aug 10 16:32:03 2012
If media recovery active, switchover will wait 900 seconds
Standby terminal recovery start SCN: 3777249
SwitchOver after complete recovery through change 3777250
Online log +DATA/rdpsdbst/onlinelog/group_1.270.774897787: Thread 1 Group 1 was previously cleared
Online log +RECOVERY/rdpsdbst/onlinelog/group_1.263.774897801: Thread 1 Group 1 was previously cleared
Online log +DATA/rdpsdbst/onlinelog/group_2.271.774897811: Thread 1 Group 2 was previously cleared
Online log +RECOVERY/rdpsdbst/onlinelog/group_2.264.774897825: Thread 1 Group 2 was previously cleared
Online log +DATA/rdpsdbst/onlinelog/group_3.272.774897835: Thread 2 Group 3 was previously cleared
Online log +RECOVERY/rdpsdbst/onlinelog/group_3.265.774897843: Thread 2 Group 3 was previously cleared
Online log +DATA/rdpsdbst/onlinelog/group_4.273.774897855: Thread 2 Group 4 was previously cleared
Online log +RECOVERY/rdpsdbst/onlinelog/group_4.266.774897865: Thread 2 Group 4 was previously cleared
Standby became primary SCN: 3777248
Converting standby mount to primary mount.
Fri Aug 10 16:32:04 2012
Switchover: Complete - Database mounted as primary (RDPSDB1)
Completed: alter database commit to switchover to primary with session shutdown
Fri Aug 10 16:32:04 2012
ARC5: STARTING ARCH PROCESSES
ARC8: Archival started
ARC5: STARTING ARCH PROCESSES COMPLETE
ARC5: Becoming the 'no SRL' ARCH
ARC8 started with pid=24, OS id=5599
6,关闭数据库,启动数据库.
[oracle@vmrac1 ~]$ srvctl config database -d RDPSDBST -a
vmrac1 RDPSDB1 /u01/app/oracle/product/10.2.0/db_1
vmrac2 RDPSDB2 /u01/app/oracle/product/10.2.0/db_1
DB_NAME: RDPSDB
ORACLE_HOME: /u01/app/oracle/product/10.2.0/db_1
SPFILE: +DATA/RDPSDBST/spfileRDPSDBST.ora
DOMAIN: null
DB_ROLE: null
START_OPTIONS: mount
POLICY: AUTOMATIC
ENABLE FLAG: DB ENABLED
修改数据库为
START_OPTIONS: open
[oracle@vmrac1 ~]$ srvctl modify database -d RDPSDBST -s NULL
关闭数据库
[oracle@vmrac1 ~]$ srvctl stop database -d RDPSDBST -o immediate
打开数据库,观察两个节点的启动alter日志
SQL>startup
Fri Aug 10 16:52:29 2012
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 192.168.25.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
processes = 150
__shared_pool_size = 83886080
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size = 0
spfile = +DATA/rdpsdbst/spfilerdpsdbst.ora
sga_target = 285212672
control_files = +DATA/rdpsdbst/controlfile/current.257.774892559, +RECOVERY/rdpsdbst/controlfile/backup.256.774892565
db_file_name_convert = +DATA/RDPSDB, +DATA/RDPSDBST
log_file_name_convert = +DATA/RDPSDB, +DATA/RDPSDBST
db_block_size = 8192
__db_cache_size = 188743680
compatible = 10.2.0.1.0
log_archive_config = dg_config=(RDPSDB,RDPSDBST)
log_archive_dest_1 = location=+RECOVERY/RDPSDBST
log_archive_dest_2 = service=RDPSDBST valid_for=(online_logfiles,primary_role) db_unique_name=RDPSDB
log_archive_max_processes= 8
fal_client = RDPSDB
fal_server = RDPSDBST
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +DATA
db_recovery_file_dest = +RECOVERY
db_recovery_file_dest_size= 4294967296
standby_file_management = AUTO
thread = 1
instance_number = 1
undo_management = AUTO
undo_tablespace = UNDOTBS1
remote_login_passwordfile= EXCLUSIVE
db_domain =
service_names = RDPSDB
dispatchers = (PROTOCOL=TCP) (SERVICE=RDPSDBXDB)
remote_listener = LISTENERS_RDPSDB
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/admin/RDPSDB/bdump
user_dump_dest = /u01/app/oracle/admin/RDPSDB/udump
core_dump_dest = /u01/app/oracle/admin/RDPSDB/cdump
audit_file_dest = /u01/app/oracle/admin/RDPSDB/adump
db_name = RDPSDB
db_unique_name = RDPSDBST
open_cursors = 300
pga_aggregate_target = 94371840
Cluster communication is configured to use the following interface(s) for this instance
10.10.10.11
Fri Aug 10 16:52:30 2012
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=6538
DIAG started with pid=3, OS id=6540
PSP0 started with pid=4, OS id=6542
LMON started with pid=5, OS id=6544
LMD0 started with pid=6, OS id=6546
LMS0 started with pid=7, OS id=6548
MMAN started with pid=8, OS id=6558
DBW0 started with pid=9, OS id=6560
LGWR started with pid=10, OS id=6562
CKPT started with pid=11, OS id=6565
SMON started with pid=12, OS id=6567
RECO started with pid=13, OS id=6569
CJQ0 started with pid=14, OS id=6571
MMON started with pid=15, OS id=6573
Fri Aug 10 16:52:30 2012
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=16, OS id=6575
Fri Aug 10 16:52:30 2012
starting up 1 shared server(s) ...
Fri Aug 10 16:52:30 2012
lmon registered with NM - instance id 1 (internal mem no 0)
Fri Aug 10 16:52:31 2012
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
0
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Fri Aug 10 16:52:31 2012
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Post SMON to start 1st pass IR
Fri Aug 10 16:52:31 2012
LMS 0: 0 GCS shadows traversed, 0 replayed
Fri Aug 10 16:52:31 2012
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=19, OS id=6610
Fri Aug 10 16:52:31 2012
ALTER DATABASE MOUNT
Fri Aug 10 16:52:31 2012
This instance was first to mount
Fri Aug 10 16:52:31 2012
Starting background process ASMB
ASMB started with pid=21, OS id=6623
Starting background process RBAL
RBAL started with pid=22, OS id=6627
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Fri Aug 10 16:52:38 2012
SUCCESS: diskgroup DATA was mounted
SUCCESS: diskgroup RECOVERY was mounted
Fri Aug 10 16:52:42 2012
Setting recovery target incarnation to 2
Fri Aug 10 16:52:42 2012
Successful mount of redo thread 1, with mount id 1614415311
Fri Aug 10 16:52:42 2012
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Fri Aug 10 16:52:42 2012
ALTER DATABASE OPEN
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Fri Aug 10 16:52:42 2012
Assigning activation ID 1614415311 (0x603a05cf)
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=28, OS id=6939
ARC1 started with pid=29, OS id=6941
ARC2 started with pid=30, OS id=6961
ARC3 started with pid=31, OS id=6963
ARC4 started with pid=32, OS id=6967
ARC5 started with pid=33, OS id=6969
Fri Aug 10 16:52:43 2012
ARC0: Archival started
ARC1: Archival started
ARC2: Archival started
ARC3: Archival started
ARC4: Archival started
ARC5: Archival started
ARC6: Archival started
ARC7: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC7 started with pid=35, OS id=6974
ARC6 started with pid=34, OS id=6971
Fri Aug 10 16:52:43 2012
Thread 1 opened at log sequence 55
Current log# 2 seq# 55 mem# 0: +DATA/rdpsdbst/onlinelog/group_2.271.774897811
Current log# 2 seq# 55 mem# 1: +RECOVERY/rdpsdbst/onlinelog/group_2.264.774897825
Successful open of redo thread 1
Fri Aug 10 16:52:43 2012
ARC0: STARTING ARCH PROCESSES
Fri Aug 10 16:52:43 2012
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Fri Aug 10 16:52:43 2012
ARC7: Becoming the heartbeat ARCH
Fri Aug 10 16:52:43 2012
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Aug 10 16:52:43 2012
SMON: enabling cache recovery
Fri Aug 10 16:52:43 2012
ARC8: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC8 started with pid=36, OS id=7004
Fri Aug 10 16:52:45 2012
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
Fri Aug 10 16:52:47 2012
Successfully onlined Undo Tablespace 1.
Dictionary check beginning
Fri Aug 10 16:52:47 2012
Errors in file /u01/app/oracle/admin/RDPSDB/bdump/rdpsdb1_dbw0_6560.trc:
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '+DATA'
ORA-17503: ksfdopn:2 Failed to open file +DATA
ORA-15045: ASM file name '+DATA' is not in reference form
Fri Aug 10 16:52:47 2012
Errors in file /u01/app/oracle/admin/RDPSDB/bdump/rdpsdb1_dbw0_6560.trc:
ORA-01186: file 201 failed verification tests
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '+DATA'
Fri Aug 10 16:52:47 2012
File 201 not verified due to error ORA-01157
Fri Aug 10 16:52:47 2012
Dictionary check complete
Fri Aug 10 16:52:47 2012
SMON: enabling tx recovery
Fri Aug 10 16:52:49 2012
Re-creating tempfile +DATA as +DATA/rdpsdbst/tempfile/temp.294.790966369
Database Characterset is UTF8
Threshold validation cannot be done before catproc is loaded.
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=37, OS id=7498
Fri Aug 10 16:53:00 2012
LOGSTDBY: Validating controlfile with logical metadata
Fri Aug 10 16:53:00 2012
LOGSTDBY: Validation complete
Completed: ALTER DATABASE OPEN
Fri Aug 10 16:53:03 2012
db_recovery_file_dest_size of 4096 MB is 12.60% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Fri Aug 10 16:53:45 2012
Reconfiguration started (old inc 2, new inc 4)
List of nodes:
0 1
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Fri Aug 10 16:53:47 2012
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Fri Aug 10 16:53:47 2012
LMS 0: 4631 GCS shadows traversed, 2336 replayed
Fri Aug 10 16:53:47 2012
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
Fri Aug 10 16:53:50 2012
Error 12560 received logging on to the standby
Fri Aug 10 16:53:50 2012
Errors in file /u01/app/oracle/admin/RDPSDB/bdump/rdpsdb1_arc7_6974.trc:
ORA-12560: TNS:protocol adapter error
PING[ARC7]: Heartbeat failed to connect to standby 'RDPSDBST'. Error is 12560.
Fri Aug 10 16:55:16 2012
Thread 1 advanced to log sequence 56
Current log# 1 seq# 56 mem# 0: +DATA/rdpsdbst/onlinelog/group_1.270.774897787
Current log# 1 seq# 56 mem# 1: +RECOVERY/rdpsdbst/onlinelog/group_1.263.774897801
Fri Aug 10 16:53:40 2012
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 10.10.10.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 192.168.25.0 configured from OCR for use as a public interface
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
processes = 150
__shared_pool_size = 83886080
__large_pool_size = 4194304
__java_pool_size = 4194304
__streams_pool_size = 0
spfile = +DATA/rdpsdbst/spfilerdpsdbst.ora
sga_target = 285212672
control_files = +DATA/rdpsdbst/controlfile/current.257.774892559, +RECOVERY/rdpsdbst/controlfile/backup.256.774892565
db_file_name_convert = +DATA/RDPSDB, +DATA/RDPSDBST
log_file_name_convert = +DATA/RDPSDB, +DATA/RDPSDBST
db_block_size = 8192
__db_cache_size = 188743680
compatible = 10.2.0.1.0
log_archive_config = dg_config=(RDPSDB,RDPSDBST)
log_archive_dest_1 = location=+RECOVERY/RDPSDBST
log_archive_dest_2 = service=RDPSDBST valid_for=(online_logfiles,primary_role) db_unique_name=RDPSDB
log_archive_max_processes= 8
fal_client = RDPSDB
fal_server = RDPSDBST
db_file_multiblock_read_count= 16
cluster_database = TRUE
cluster_database_instances= 2
db_create_file_dest = +DATA
db_recovery_file_dest = +RECOVERY
db_recovery_file_dest_size= 4294967296
standby_file_management = AUTO
thread = 2
instance_number = 2
undo_management = AUTO
undo_tablespace = UNDOTBS2
remote_login_passwordfile= EXCLUSIVE
db_domain =
service_names = RDPSDB
dispatchers = (PROTOCOL=TCP) (SERVICE=RDPSDBXDB)
remote_listener = LISTENERS_RDPSDB
job_queue_processes = 10
background_dump_dest = /u01/app/oracle/admin/RDPSDB/bdump
user_dump_dest = /u01/app/oracle/admin/RDPSDB/udump
core_dump_dest = /u01/app/oracle/admin/RDPSDB/cdump
audit_file_dest = /u01/app/oracle/admin/RDPSDB/adump
db_name = RDPSDB
db_unique_name = RDPSDBST
open_cursors = 300
pga_aggregate_target = 94371840
Cluster communication is configured to use the following interface(s) for this instance
10.10.10.12
Fri Aug 10 16:53:41 2012
cluster interconnect IPC version:Oracle UDP/IP
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=12597
DIAG started with pid=3, OS id=12599
PSP0 started with pid=4, OS id=12601
LMON started with pid=5, OS id=12603
LMD0 started with pid=6, OS id=12605
MMAN started with pid=8, OS id=12617
DBW0 started with pid=9, OS id=12619
LMS0 started with pid=7, OS id=12607
LGWR started with pid=10, OS id=12621
CKPT started with pid=11, OS id=12623
SMON started with pid=12, OS id=12625
RECO started with pid=13, OS id=12627
CJQ0 started with pid=14, OS id=12629
MMON started with pid=15, OS id=12631
Fri Aug 10 16:53:42 2012
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=16, OS id=12633
Fri Aug 10 16:53:42 2012
starting up 1 shared server(s) ...
Fri Aug 10 16:53:42 2012
lmon registered with NM - instance id 2 (internal mem no 1)
Fri Aug 10 16:53:45 2012
Reconfiguration started (old inc 0, new inc 4)
List of nodes:
0 1
Global Resource Directory frozen
* allocate domain 0, invalid = TRUE
Communication channels reestablished
* domain 0 valid = 1 according to instance 0
Fri Aug 10 16:53:47 2012
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Fri Aug 10 16:53:47 2012
LMS 0: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Fri Aug 10 16:53:47 2012
LMS 0: 0 GCS shadows traversed, 0 replayed
Fri Aug 10 16:53:47 2012
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=19, OS id=12765
Fri Aug 10 16:53:48 2012
ALTER DATABASE MOUNT
Fri Aug 10 16:53:48 2012
Starting background process ASMB
ASMB started with pid=21, OS id=12796
Starting background process RBAL
RBAL started with pid=22, OS id=12800
Loaded ASM Library - Generic Linux, version 2.0.2 (KABI_V2) library for asmlib interface
Fri Aug 10 16:53:51 2012
SUCCESS: diskgroup DATA was mounted
SUCCESS: diskgroup RECOVERY was mounted
Fri Aug 10 16:53:57 2012
Setting recovery target incarnation to 2
Fri Aug 10 16:53:59 2012
Successful mount of redo thread 2, with mount id 1614415311
Fri Aug 10 16:53:59 2012
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE MOUNT
Fri Aug 10 16:54:04 2012
ALTER DATABASE OPEN
Picked broadcast on commit scheme to generate SCNs
Fri Aug 10 16:54:11 2012
LGWR: STARTING ARCH PROCESSES
ARC1 started with pid=30, OS id=13402
ARC0 started with pid=29, OS id=13400
ARC2 started with pid=31, OS id=13408
ARC3 started with pid=32, OS id=13410
ARC4 started with pid=33, OS id=13416
ARC5 started with pid=34, OS id=13418
ARC6 started with pid=35, OS id=13420
Fri Aug 10 16:54:17 2012
ARC0: Archival started
ARC1: Archival started
ARC2: Archival started
ARC3: Archival started
ARC4: Archival started
ARC5: Archival started
ARC6: Archival started
ARC7: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC7 started with pid=36, OS id=13430
Fri Aug 10 16:54:36 2012
Thread 2 opened at log sequence 45
Current log# 4 seq# 45 mem# 0: +DATA/rdpsdbst/onlinelog/group_4.273.774897855
Current log# 4 seq# 45 mem# 1: +RECOVERY/rdpsdbst/onlinelog/group_4.266.774897865
Fri Aug 10 16:54:36 2012
ARC5: STARTING ARCH PROCESSES
Fri Aug 10 16:54:36 2012
ARC7: Becoming the 'no FAL' ARCH
ARC7: Becoming the 'no SRL' ARCH
Fri Aug 10 16:54:36 2012
ARC1: Becoming the heartbeat ARCH
Fri Aug 10 16:54:36 2012
Successful open of redo thread 2
Fri Aug 10 16:54:36 2012
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Aug 10 16:54:36 2012
ARC8: Archival started
ARC5: STARTING ARCH PROCESSES COMPLETE
ARC8 started with pid=37, OS id=13862
Fri Aug 10 16:54:43 2012
SMON: enabling cache recovery
Fri Aug 10 16:55:08 2012
Successfully onlined Undo Tablespace 5.
Fri Aug 10 16:55:08 2012
SMON: enabling tx recovery
Fri Aug 10 16:55:08 2012
Database Characterset is UTF8
Fri Aug 10 16:55:23 2012
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=38, OS id=15202
Fri Aug 10 16:55:44 2012
Completed: ALTER DATABASE OPEN
Fri Aug 10 16:55:49 2012
Error 12560 received logging on to the standby
Fri Aug 10 16:55:49 2012
Errors in file /u01/app/oracle/admin/RDPSDB/bdump/rdpsdb2_arc1_13402.trc:
ORA-12560: TNS:protocol adapter error
PING[ARC1]: Heartbeat failed to connect to standby 'RDPSDBST'. Error is 12560.
7,出现的情况
在节点1和节点2出现下面的情况,这里新的primary database会尝试去连接standby database,因此打开过程会挂起一会,当尝试几次后会,最终会打开数据库,因为spfile配置里还有这样2个设置,它会导致远程归档去寻找RDPSDBST,即原来的主库.
log_archive_config string dg_config=(RDPSDB,RDPSDBST)
log_archive_dest_2 string service=RDPSDBST valid_for=(online_logfiles,primary_role) db_unique_name=RDPSDB
解决方法:
SQL> alter system reset log_archive_config scope=spfile sid='*';
SQL> alter system reset log_archive_dest_2 scope=spfile sid='*';
然后再重启数据库.
Fri Aug 10 16:59:12 2012
Error 12560 received logging on to the standby
Fri Aug 10 16:59:12 2012
Errors in file /u01/app/oracle/admin/RDPSDB/bdump/rdpsdb1_arc7_6974.trc:
ORA-12560: TNS:protocol adapter error
PING[ARC7]: Heartbeat failed to connect to standby 'RDPSDBST'. Error is 12560.
Fri Aug 10 17:00:12 2012
Shutting down archive processes
Fri Aug 10 17:00:17 2012
ARCH shutting down
ARC8: Archival stopped
Fri Aug 10 16:59:37 2012
Shutting down archive processes
Fri Aug 10 16:59:39 2012
Shutting down archive processes
Fri Aug 10 16:59:42 2012
ARCH shutting down
ARC8: Archival stopped
Fri Aug 10 16:59:43 2012
Archive process shutdown avoided: 8 active
Fri Aug 10 17:01:14 2012
Error 12560 received logging on to the standby
Fri Aug 10 17:01:14 2012
Errors in file /u01/app/oracle/admin/RDPSDB/bdump/rdpsdb2_arc1_13402.trc:
ORA-12560: TNS:protocol adapter error
PING[ARC1]: Heartbeat failed to connect to standby 'RDPSDBST'. Error is 12560.
7,验证faileover切换结果
SQL>select protection_mode,protection_level from v$database;
PROTECTION_MODE PROTECTION_LEVEL
-------------------- --------------------
MAXIMUM PERFORMANCE MAXIMUM PERFORMANCE
SQL>select database_role from v$database;
DATABASE_ROLE
----------------
PRIMARY
SQL>select OPEN_MODE from v$database;
OPEN_MODE
----------
READ WRITE
完结
