Step by Step Oracle 11gR2 RAC Standby Database Manual Failover

Oracle 11gR2 RAC to RAC Standby Database Manual Failover Steps

 

Description:-

In this article we are going to performing the Oracle 11gR2 two node RAC to RAC manual failover steps to a physical standby database.

Let us consider, we have primary site with two node RAC and standby site also configured with two node RAC in 11gR2. You can refer here, for two node RAC to RAC dataguard setup in 11gR2 and also we are using the same configuration from the referred one.

Here, we can see the primary and standby configuration:-

Let us verify the status and configuration of the RAC database:-

$ srvctl config database -d ORCL_PRODRAC1 -a
Database unique name: ORCL_PRODRAC1
Database name: ORCL
Oracle home: /oradb/app/oracle/product/11.2.0.4/db_1
Oracle user: oracle
Spfile: +DBWR_DATA/ORCL_PRODRAC1/spfileORCL_PRODRAC1.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: ORCL_PRODRAC1
Database instances: ORCL1,ORCL2
Disk Groups: DBWR_DATA,DBWR_FRA
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed
$ srvctl config database -d ORCL_PRODRAC2 -a
Database unique name: ORCL_PRODRAC2
Database name: ORCL
Oracle home: /oradb/app/oracle/product/11.2.0.4/db_1
Oracle user: oracle
Spfile: +DBWR_DATA/ORCL_PRODRAC2/spfileORCL_PRODRAC2.ora
Domain:
Start options: read only
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: ORCL_PRODRAC2
Database instances: ORCL1,ORCL2
Disk Groups: DBWR_DATA,DBWR_FRA
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed

Let us assume, the primary and standby sites are geographical segmented and due to a natural disaster or some power outage issue both the nodes of the primary site become unavailable for the application and we may not be able to restore/recover the primary site within a reasonable period of time,. In this situation, we will be pushed to make the available standby database which is with most recent recovery to be converted into primary and reduce the downtime of the database and make the database available for continuation of the business.

So, let’s follow the below steps for the above scenario.

A) Pre-Checks for Failover
1) Ensure primary is using Flashback for reinstate the failed primary
2) Flush any unsent redo in primary
3) Verify the standby database has the most recent archived redo log from each primary redo thread
4) Identify and resolve any archived redo log gaps
5) Repeat Step 4 until all gaps are resolved for each thread

B) Failover
1) Stop Redo Apply
2) Finish applying all received redo data
3) Verify that the target standby database is ready to become a primary database
4) Switch the physical standby database to the primary role
5) Open the new primary database

Let us verify the status and configuration of the RAC database:-

$ srvctl config database -d ORCL_PRODRAC1 -a
Database unique name: ORCL_PRODRAC1
Database name: ORCL
Oracle home: /oradb/app/oracle/product/11.2.0.4/db_1
Oracle user: oracle
Spfile: +DBWR_DATA/ORCL_PRODRAC1/spfileORCL_PRODRAC1.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: ORCL_PRODRAC1
Database instances: ORCL1,ORCL2
Disk Groups: DBWR_DATA,DBWR_FRA
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed
$ srvctl config database -d ORCL_PRODRAC2 -a
Database unique name: ORCL_PRODRAC2
Database name: ORCL
Oracle home: /oradb/app/oracle/product/11.2.0.4/db_1
Oracle user: oracle
Spfile: +DBWR_DATA/ORCL_PRODRAC2/spfileORCL_PRODRAC2.ora
Domain:
Start options: read only
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: ORCL_PRODRAC2
Database instances: ORCL1,ORCL2
Disk Groups: DBWR_DATA,DBWR_FRA
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed

A) Pre-Failover Checks
Step 1 – Ensure primary is using Flashback for reinstate the failed primary:-  (Primary Side)

In order to reinstate the failed primary database, the flashback option must be enabled in the primary

SQL> select flashback_on from v$database;

FLASHBACK_ON
------------------
NO

If the flashback is disabled, follow the below step to enable flashback in the primary. This should be done before the failover.

I) Ensure db_recovery_file_dest is set

SQL> show parameter db_recovery_file_dest

NAME                          TYPE        VALUE
--------------------------    ----------- ------------------
db_recovery_file_dest         string      +DBWR_FRA
db_recovery_file_dest_size    big integer 12000M

If the above command returns any value, go for step III. Else set a proper directory or diskgroups.

SQL> alter system set db_recovery_file_dest='+DBWR_FRA' SCOPE=spfile sid='*';

System altered.

II) Ensure db_recovery_file_dest_size is set

SQL> alter system set db_recovery_file_dest_size=20G SCOPE=spfile sid='*';

System altered.

III) Stop and start the database

$ srvctl stop database -d ORCL_PRODRAC1
$ srvctl start database -d ORCL_PRODRAC1 -o mount
SQL> alter database flashback on;

Database altered.
SQL> alter database open; <--- Both Nodes

Database altered.
SQL> alter system set db_flashback_retention_target=2880 sid='*';

System altered.

Let our scenario Primary Database Crush Due to several reason like Hardware failure.

B) Failover
Step 1 – Stop Redo Apply:- (Standby Side) 

Issue the following SQL statement on the target standby database.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

Database altered.

Step 2 – Finish applying all received redo data:-

Issue the following SQL statement on the target standby database

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH;

Database altered.

Step 3 – Verify that the target standby database is ready to become a primary database:-

Query the SWITCHOVER_STATUS column of the V$DATABASE view on the target standby database

SQL> SELECT SWITCHOVER_STATUS FROM V$DATABASE;

SWITCHOVER_STATUS
--------------------
TO PRIMARY

A value of either TO PRIMARY or SESSIONS ACTIVE indicates that the standby database is ready to be switched to the primary role. If neither of these values is returned, verify that Redo Apply is active and continue to query this view until either TO PRIMARY or SESSIONS ACTIVE is returned.

Step 4 – Switch the physical standby database to the primary role:-

Issue the following SQL statement on the target standby database

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WITH SESSION SHUTDOWN;

Database altered.

Note:-
The WITH SESSION SHUTDOWN clause can be omitted from the switchover statement if the query of the SWITCHOVER_STATUS column performed in the previous step returned a value of TO PRIMARY.

Step 5 – Open the new primary database:-

SQL> ALTER DATABASE OPEN;  <-- Both Nodes

Database altered.
$ srvctl status database -d ORCL_PRODRAC2 -v
Instance ORCL1 is running on node prodrac201. Instance status: Open.
Instance ORCL2 is running on node prodrac202. Instance status: Open.

Do not use srvctl command to start the secondary instance in the new primary database, since we configured the “Start options as read only” and Oracle Cluster Register do not recognize the switchover/failover automatically. So, if we use srvctl the secondary instance will shutdown as below

$ tail -f alert_ORCL2.log
Lost write protection disabled
Completed: ALTER DATABASE MOUNT /* db agent *//* {1:13107:16648} */
ALTER DATABASE OPEN READ ONLY /* db agent *//* {1:13107:16648} */
AUDIT_TRAIL initialization parameter is changed to OS, as DB is NOT compatible for database opened with read-only access
Errors in file /oradb/app/oracle/diag/rdbms/orcl_prodrac2/ORCL2/trace/ORCL2_ora_5999.trc:
ORA-16002: database already open for read/write access by another instance
ORA-16002 signalled during: ALTER DATABASE OPEN READ ONLY /* db agent *//* {1:13107:16648} */...
Sun Aug 19 16:15:39 2018
Shutting down instance (abort)
License high water mark = 1
USER (ospid: 6022): terminating the instance
Instance terminated by USER, pid = 6022
Sun Aug 19 16:15:39 2018
Instance shutdown complete

Note:-
We can modify the database configuration details as below and use srvctl if we need.

$ srvctl modify database -d ORCL_PRODRAC2 -r PRIMARY -s OPEN
$ srvctl config database -d ORCL_PRODRAC2 -a
Database unique name: ORCL_PRODRAC2
Database name: ORCL
Oracle home: /oradb/app/oracle/product/11.2.0.4/db_1
Oracle user: oracle
Spfile: +DBWR_DATA/ORCL_PRODRAC2/spfileORCL_PRODRAC2.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: ORCL_PRODRAC2
Database instances: ORCL1,ORCL2
Disk Groups: DBWR_DATA,DBWR_FRA
Mount point paths:
Services:
Type: RAC
Database is enabled
Database is administrator managed
  1. After this we will start the new primary Database using SRVCTL
  2. After failover take fresh rman backup of New primary database

Reference: 

https://docs.oracle.com/cd/E11882_01/server.112/e41134/role_management.htm#SBYDB00625

Catch Me On:- Hariprasath Rajaram

Telegram:https://t.me/joinchat/I_f4DhGF_Zifr9YZvvMkRg
LinkedIn:https://www.linkedin.com/in/hari-prasath-aa65bb19/
Facebook:https://www.facebook.com/HariPrasathdba
FB Group:https://www.facebook.com/groups/894402327369506/
FB Page: https://www.facebook.com/dbahariprasath/?
Twitter: https://twitter.com/hariprasathdba

Upgrade Oracle database from 11g to 12c using the RMAN Backup

Description:-

In this article we are going to see upgrade Oracle database from the version 11g to the version 12c using the RMAN

Environment Detail’s

High Level Steps:-

  1. Oracle Database 11.2.0.3 Rman Full Backup
  2. Oracle Database 12c Restore
  3. Alter database open
  4. Upgrade Database

Let’s start the Demo:-

  1. Oracle Database 11.2.0.3 Rman Full Backup

Step1:- Verify the Database name and archive log is enabled on database 11g.

[oracle@gg-11 ~]$ export ORACLE_SID=oradbwr
[oracle@gg-11 ~]$ sqlplus ‘/as sysdba’
SQL*Plus: Release 11.2.0.3.0 Production on Fri Aug 3 01:20:57 2018
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select name,log_mode from v$database;
NAME LOG_MODE
——— ————
ORADBWR ARCHIVELOG

Step2:-Testing After Upgrade to 12c

SQL> create table chennai (name varchar(10));
Table created.

SQL> insert into chennai values(‘HARI’);
1 row created.

SQL> commit;
Commit complete.

Step3:-Going to take 11g database Full backup using rman

[oracle@gg-11 ~]$ export ORACLE_SID=oradbwr
[oracle@gg-11 ~]$ rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Fri Aug 3 01:26:53 2018
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
connected to target database: ORADBWR (DBID=1410039869)

RMAN> backup database plus archivelog;

Starting backup at 03-AUG-18
current log archived
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=41 device type=DISK
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=2 RECID=1 STAMP=983150751
input archived log thread=1 sequence=3 RECID=2 STAMP=983150751
input archived log thread=1 sequence=4 RECID=3 STAMP=983150755
input archived log thread=1 sequence=5 RECID=4 STAMP=983150755
input archived log thread=1 sequence=6 RECID=5 STAMP=983150756
input archived log thread=1 sequence=7 RECID=6 STAMP=983150757
input archived log thread=1 sequence=8 RECID=7 STAMP=983150758
input archived log thread=1 sequence=9 RECID=8 STAMP=983150824
channel ORA_DISK_1: starting piece 1 at 03-AUG-18
channel ORA_DISK_1: finished piece 1 at 03-AUG-18
piece handle=/u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/o1_mf_annnn_TAG20180803T012704_fp6rjjmz_.bkp tag=TAG20180803T012704 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 03-AUG-18
Starting backup at 03-AUG-18
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/u01/app/oracle/oradata/oradbwr/system01.dbf
input datafile file number=00002 name=/u01/app/oracle/oradata/oradbwr/sysaux01.dbf
input datafile file number=00003 name=/u01/app/oracle/oradata/oradbwr/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/oradbwr/users01.dbf
channel ORA_DISK_1: starting piece 1 at 03-AUG-18
channel ORA_DISK_1: finished piece 1 at 03-AUG-18
piece handle=/u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/o1_mf_nnndf_TAG20180803T012705_fp6rjkt8_.bkp tag=TAG20180803T012705 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:45
Finished backup at 03-AUG-18
Starting backup at 03-AUG-18
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=10 RECID=9 STAMP=983150870
channel ORA_DISK_1: starting piece 1 at 03-AUG-18
channel ORA_DISK_1: finished piece 1 at 03-AUG-18
piece handle=/u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/o1_mf_annnn_TAG20180803T012751_fp6rkz5m_.bkp tag=TAG20180803T012751 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 03-AUG-18
Starting Control File and SPFILE Autobackup at 03-AUG-18
piece handle=/u01/app/oracle/fast_recovery_area/ORADBWR/autobackup/2018_08_03/o1_mf_s_983150872_fp6rl0lw_.bkp comment=NONE
Finished Control File and SPFILE Autobackup at 03-AUG-18

Step4:- Copy the rman backup pieces from Server 11g to Server 12c

Datafile and archivelog

[oracle@gg-11 ~]$ cd /u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/
[oracle@gg-11 2018_08_03]$ scp o1_mf_nnndf_TAG20180803T012705_fp6rjkt8_.bkp o1_mf_annnn_TAG20180803T012751_fp6rkz5m_.bkp oracle@192.168.2.157:/u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/
oracle@192.168.2.157's password: 
o1_mf_nnndf_TAG20180803T012705_fp6rjkt8_.bkp 100% 993MB 26.8MB/s 00:37 
o1_mf_annnn_TAG20180803T012751_fp6rkz5m_.bkp 100% 3072 3.0KB/s 00:00 

Controlfile autobackup location

[oracle@gg-11 2018_08_03]$ cd /u01/app/oracle/fast_recovery_area/ORADBWR/autobackup/
[oracle@gg-11 autobackup]$ scp 2018_08_03/o1_mf_s_983150872_fp6rl0lw_.bkp oracle@192.168.2.157:/u01/app/oracle/fast_recovery_area/ORADBWR/autobackup/
oracle@192.168.2.157's password: 
o1_mf_s_983150872_fp6rl0lw_.bkp

Step5:- Create and copy the pfile $ passwordfile from source 11g to destination 12c

[oracle@gg-11 dbs]$ scp orapworadbwr initoradbwr.ora oracle@192.168.2.157:/u01/app/oracle/product/12.2.0/dbhome_2/dbs
oracle@192.168.2.157’s password:
orapworadbwr 100% 1536 1.5KB/s 00:00
initoradbwr.ora 100% 1046 1.0KB/s 00:00

2.Oracle Database 12c Restore

Step6:-Change the pfile parameters for 12c database

compatible=’12.2.0.1.0′

Step7:- Create directory for 12c database

[oracle@gg-12 dbs]$ mkdir -p /u01/app/oracle/oradata/oradbwr/
[oracle@gg-12 ]$ mkdir -p /u01/app/oracle/admin/oradbwr/adump

Step8:- Startup the 12c database on the 12c Server

[oracle@gg-12 dbs]$ export ORACLE_SID=oradbwr
[oracle@gg-12 dbs]$ sqlplus '/as sysdba'
SQL*Plus: Release 12.2.0.1.0 Production on Fri Aug 3 02:21:04 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.
Total System Global Area 931135488 bytes
Fixed Size 8627296 bytes
Variable Size 301992864 bytes
Database Buffers 616562688 bytes
Redo Buffers 3952640 bytes

SQL> create spfile from pfile;
File created.

Step9:- Restore the Control file on the 12c Server.

[oracle@gg-12 ~]$ export ORACLE_SID=oradbwr
[oracle@gg-12 ~]$ rman target /
Recovery Manager: Release 12.2.0.1.0 - Production on Fri Aug 3 02:26:50 2018
Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.
connected to target database: ORADBWR (not mounted)

RMAN> restore controlfile from autobackup;

Starting restore at 03-AUG-18
using channel ORA_DISK_1
recovery area destination: /u01/app/oracle/fast_recovery_area
database name (or database unique name) used for search: ORADBWR
channel ORA_DISK_1: AUTOBACKUP /u01/app/oracle/fast_recovery_area/ORADBWR/autobackup/o1_mf_s_983150872_fp6rl0lw_.bkp found in the recovery area
AUTOBACKUP search with format "%F" not attempted because DBID was not set
channel ORA_DISK_1: restoring control file from AUTOBACKUP /u01/app/oracle/fast_recovery_area/ORADBWR/autobackup/o1_mf_s_983150872_fp6rl0lw_.bkp
channel ORA_DISK_1: control file restore from AUTOBACKUP complete
output file name=/u01/app/oracle/oradata/oradbwr/control01.ctl
output file name=/u01/app/oracle/fast_recovery_area/oradbwr/control02.ctl
Finished restore at 03-AUG-18

Step10:-Mount the Database

RMAN> alter database mount;
Statement processed
released channel: ORA_DISK_1

Step11:-Catalog the 11g backup within the controlfile

RMAN> catalog start with '/u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/';

Starting implicit crosscheck backup at 03-AUG-18
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=42 device type=DISK
Crosschecked 3 objects
Finished implicit crosscheck backup at 03-AUG-18
Starting implicit crosscheck copy at 03-AUG-18
using channel ORA_DISK_1
Finished implicit crosscheck copy at 03-AUG-18
searching for all files in the recovery area
cataloging files...
cataloging done
List of Cataloged Files
=======================
File Name: /u01/app/oracle/fast_recovery_area/ORADBWR/autobackup/o1_mf_s_983150872_fp6rl0lw_.bkp
searching for all files that match the pattern /u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/
no files found to be unknown to the database

Step12:-Restore Database

RMAN> restore database;

Starting restore at 03-AUG-18
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=8 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/oradbwr/system01.dbf
channel ORA_DISK_1: restoring datafile 00002 to /u01/app/oracle/oradata/oradbwr/sysaux01.dbf
channel ORA_DISK_1: restoring datafile 00003 to /u01/app/oracle/oradata/oradbwr/undotbs01.dbf
channel ORA_DISK_1: restoring datafile 00004 to /u01/app/oracle/oradata/oradbwr/users01.dbf
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/o1_mf_nnndf_TAG20180803T012705_fp6rjkt8_.bkp
channel ORA_DISK_1: piece handle=/u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/o1_mf_nnndf_TAG20180803T012705_fp6rjkt8_.bkp tag=TAG20180803T012705
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:35
Finished restore at 03-AUG-18

Step13:-Recover database upto last sequence

RMAN> run {
set until sequence = 11;  (this sequence number i get from 11g DB)
recover database;
}

executing command: SET until clause
Starting recover at 03-AUG-18
using channel ORA_DISK_1
starting media recovery
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=10
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/o1_mf_annnn_TAG20180803T012751_fp6rkz5m_.bkp
channel ORA_DISK_1: piece handle=/u01/app/oracle/fast_recovery_area/ORADBWR/backupset/2018_08_03/o1_mf_annnn_TAG20180803T012751_fp6rkz5m_.bkp tag=TAG20180803T012751
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=/u01/app/oracle/fast_recovery_area/ORADBWR/archivelog/2018_08_03/o1_mf_1_10_fp6wvskv_.arc thread=1 sequence=10
channel default: deleting archived log(s)
archived log file name=/u01/app/oracle/fast_recovery_area/ORADBWR/archivelog/2018_08_03/o1_mf_1_10_fp6wvskv_.arc RECID=10 STAMP=983155281
media recovery complete, elapsed time: 00:00:00
Finished recover at 03-AUG-18

3.Alter database open

Step14:-Connect sqlplus and open the database

[oracle@gg-12 dbs]$ export ORACLE_SID=oradbwr
[oracle@gg-12 dbs]$ sqlplus ‘/as sysdba’
SQL*Plus: Release 12.2.0.1.0 Production on Fri Aug 3 02:44:51 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 – 64bit Production

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure   (we can’t open in normal mode)
ORA-00604: error occurred at recursive SQL level 1
ORA-00904: “ACDRROWTSINTCOL#”: invalid identifier
Process ID: 4004
Session ID: 1 Serial number: 22560

4.Upgrade Database

Step15:- Startup upgrade and run the catctl.pl  script .

[oracle@gg-12 dbs]$ sqlplus ‘/as sysdba’
SQL*Plus: Release 12.2.0.1.0 Production on Fri Aug 3 03:11:46 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to an idle instance.

SQL> startup upgrade

ORACLE instance started.
Total System Global Area 931135488 bytes
Fixed Size 8627296 bytes
Variable Size 301992864 bytes
Database Buffers 616562688 bytes
Redo Buffers 3952640 bytes
Database mounted.
Database opened.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

[oracle@gg-12 dbs]$ cd $ORACLE_HOME/rdbms/admin
[oracle@gg-12 admin]$ $ORACLE_HOME/perl/bin/perl catctl.pl -n 4 catupgrd.sql

Argument list for [catctl.pl]
Run in c = 0
Do not run in C = 0
Input Directory d = 0
Echo OFF e = 1
Simulate E = 0
Forced cleanup F = 0
Log Id i = 0
Child Process I = 0
Log Dir l = 0
Priority List Name L = 0
Upgrade Mode active M = 0
SQL Process Count n = 4
SQL PDB Process Count N = 0
Open Mode Normal o = 0
Start Phase p = 0
End Phase P = 0
Reverse Order r = 0
AutoUpgrade Resume R = 0
Script s = 0
Serial Run S = 0
RO User Tablespaces T = 0
Display Phases y = 0
Debug catcon.pm z = 0
Debug catctl.pl Z = 0
catctl.pl VERSION: [12.2.0.1.0]
STATUS: [production]
BUILD: [RDBMS_12.2.0.1.0_LINUX.X64_170125]
/u01/app/oracle/product/12.2.0/dbhome_2/rdbms/admin/orahome = [/u01/app/oracle/product/12.2.0/dbhome_2]
/u01/app/oracle/product/12.2.0/dbhome_2/bin/orabasehome = [/u01/app/oracle/product/12.2.0/dbhome_2]
catctlGetOrabase = [/u01/app/oracle/product/12.2.0/dbhome_2]
Analyzing file /u01/app/oracle/product/12.2.0/dbhome_2/rdbms/admin/catupgrd.sql
Log file directory = [/tmp/cfgtoollogs/upgrade20180803031607]
catcon: ALL catcon-related output will be written to [/tmp/cfgtoollogs/upgrade20180803031607/catupgrd_catcon_4242.lst]
catcon: See [/tmp/cfgtoollogs/upgrade20180803031607/catupgrd*.log] files for output generated by scripts
catcon: See [/tmp/cfgtoollogs/upgrade20180803031607/catupgrd_*.lst] files for spool files, if any
Number of Cpus = 1
Database Name = oradbwr
DataBase Version = 11.2.0.3.0
catcon: ALL catcon-related output will be written to [/u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/oradbwr/upgrade20180803031609/catupgrd_catcon_4242.lst]
catcon: See [/u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/oradbwr/upgrade20180803031609/catupgrd*.log] files for output generated by scripts
catcon: See [/u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/oradbwr/upgrade20180803031609/catupgrd_*.lst] files for spool files, if any
Log file directory = [/u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/oradbwr/upgrade20180803031609]
Parallel SQL Process Count = 4
Components in [oradbwr]
Installed [APEX APS CATALOG CATJAVA CATPROC CONTEXT EM JAVAVM ORDIM OWM SDO XDB XML XOQ]
Not Installed [DV MGW ODM OLS RAC WK]
------------------------------------------------------
Phases [0-115] Start Time:[2018_08_03 03:16:10]
------------------------------------------------------
*********** Executing Change Scripts ***********
Serial Phase #:0 [oradbwr] Files:1 Time: 128s
*************** Catalog Core SQL ***************
Serial Phase #:1 [oradbwr] Files:5 Time: 67s
Restart Phase #:2 [oradbwr] Files:1 Time: 0s
Serial Phase #:109 [oradbwr] Files:1 Time: 60s
Serial Phase #:110 [oradbwr] Files:1 Time: 0s
Serial Phase #:111 [oradbwr] Files:1 Time: 107s
***************** Post Upgrade *****************
Serial Phase #:112 [oradbwr] Files:1 Time: 425s
**************** Summary report ****************
Serial Phase #:113 [oradbwr] Files:1 Time: 2s
Serial Phase #:114 [oradbwr] Files:1 Time: 0s
Serial Phase #:115 [oradbwr] Files:1 Time: 40s
------------------------------------------------------
Phases [0-115] End Time:[2018_08_03 05:15:24]
------------------------------------------------------
Grand Total Time: 7161s 
LOG FILES: (/u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/oradbwr/upgrade20180803031609/catupgrd*.log)
Upgrade Summary Report Located in:
/u01/app/oracle/product/12.2.0/dbhome_2/cfgtoollogs/oradbwr/upgrade20180803031609/upg_summary.log
Grand Total Upgrade Time: [0d:1h:30m:21s]

Step16:- Recompile the Invalid objects using utlrp.sql

SQL> @?/rdbms/admin/utlrp.sql

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_BGN 2018-08-03 05:24:35
DOC> The following PL/SQL block invokes UTL_RECOMP to recompile invalid
DOC> objects in the database. Recompilation time is proportional to the
DOC> number of invalid objects in the database, so this command may take
DOC> a long time to execute on a database with a large number of invalid
DOC> objects.
DOC>
DOC> Use the following queries to track recompilation progress:
DOC>
DOC> 1. Query returning the number of invalid objects remaining. This
DOC> number should decrease with time.
DOC> SELECT COUNT(*) FROM obj$ WHERE status IN (4, 5, 6);
DOC>
DOC> 2. Query returning the number of objects compiled so far. This number
DOC> should increase with time.
DOC> SELECT COUNT(*) FROM UTL_RECOMP_COMPILED;
DOC>
DOC> This script automatically chooses serial or parallel recompilation
DOC> based on the number of CPUs available (parameter cpu_count) multiplied
DOC> by the number of threads per CPU (parameter parallel_threads_per_cpu).
DOC> On RAC, this number is added across all RAC nodes.
DOC>
DOC> UTL_RECOMP uses DBMS_SCHEDULER to create jobs for parallel
DOC> recompilation. Jobs are created without instance affinity so that they
DOC> can migrate across RAC nodes. Use the following queries to verify
DOC> whether UTL_RECOMP jobs are being created and run correctly:
DOC>
DOC> 1. Query showing jobs created by UTL_RECOMP
DOC> SELECT job_name FROM dba_scheduler_jobs
DOC> WHERE job_name like 'UTL_RECOMP_SLAVE_%';
DOC>
DOC> 2. Query showing UTL_RECOMP jobs that are running
DOC> SELECT job_name FROM dba_scheduler_running_jobs
DOC> WHERE job_name like 'UTL_RECOMP_SLAVE_%';
DOC>#
PL/SQL procedure successfully completed.

TIMESTAMP
--------------------------------------------------------------------------------
COMP_TIMESTAMP UTLRP_END 2018-08-03 05:33:52
DOC> The following query reports the number of invalid objects.
DOC>
DOC> If the number is higher than expected, please examine the error
DOC> messages reported with each object (using SHOW ERRORS) to see if they
DOC> point to system misconfiguration or resource constraints that must be
DOC> fixed before attempting to recompile these objects.
DOC>#
OBJECTS WITH ERRORS
-------------------
0

DOC> The following query reports the number of exceptions caught during
DOC> recompilation. If this number is non-zero, please query the error
DOC> messages in the table UTL_RECOMP_ERRORS to see if any of these errors
DOC> are due to misconfiguration or resource constraints that must be
DOC> fixed before objects can compile successfully.
DOC> Note: Typical compilation errors (due to coding errors) are not
DOC> logged into this table: they go into DBA_ERRORS instead.
DOC>#

ERRORS DURING RECOMPILATION
---------------------------
0

Function created.
PL/SQL procedure successfully completed.

Function dropped.
...Starting validation 05:34:07
...Database user "SYS", database schema "APEX_050000", user# "121" 05:34:07
...grant execute on "SYS"."DBMS_CRYPTO_INTERNAL" to APEX_050000 05:34:10
...272 packages
...265 package bodies
...465 tables
...8 functions
...16 procedures
...4 sequences
...497 triggers
...1582 indexes
...255 views
...0 libraries
...14 types
...5 type bodies
...0 operators
...0 index types
...Begin key object existence check 05:34:17
...Completed key object existence check 05:34:17
...Setting DBMS Registry 05:34:17
...Setting DBMS Registry Complete 05:34:17
...Exiting validate 05:34:17
PL/SQL procedure successfully completed


Step17:-check the database status 

SQL> select comp_name,version from dba_registry where status = 'VALID';

COMP_NAME                                  VERSION
----------------                ------------------------------
Oracle Database Catalog Views            12.2.0.1.0
Oracle Database Packages and Types       12.2.0.1.0
JServer JAVA Virtual Machine             12.2.0.1.0
Oracle XDK                               12.2.0.1.0
Oracle Database Java Packages            12.2.0.1.0
OLAP Analytic Workspace                  12.2.0.1.0
Oracle Workspace Manager                 12.2.0.1.0
Oracle Text                              12.2.0.1.0
Oracle XML Database                      12.2.0.1.0
Oracle Multimedia                        12.2.0.1.0
Spatial                                  12.2.0.1.0
Oracle OLAP API                          12.2.0.1.0
Oracle Application Express               5.0.4.00.12

13 rows selected.

Successfully upgrade to 12c database

Catch Me On:- Hariprasath Rajaram

Telegram:https://t.me/joinchat/I_f4DhGF_Zifr9YZvvMkRg
LinkedIn:https://www.linkedin.com/in/hari-prasath-aa65bb19/
Facebook:https://www.facebook.com/HariPrasathdba
FB Group:https://www.facebook.com/groups/894402327369506/
FB Page: https://www.facebook.com/dbahariprasath/?
Twitter: https://twitter.com/hariprasathdba