Magic Upgrade to 21c with Replay Upgrade Feature

Last week I did some upgrade tests from 19c to 21c. The plan was to copy a 19c PDB to a CDB 21c using Online Clone and then upgrade it to 21c. I have done this many times for 19c as well and just wanted to verify that it still worked with 21c.

Clone PDB to 21c CDB
Since it is a LAB, I gave the user SYSTEM in the CDB19c the appropriate duplicate permissions and then I created from the CDB 21c the appropriate DB link for the clone

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Sep 17 13:17:06 2021
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
SQL> grant sysoper, create pluggable database to system container=all;
Grant succeeded.
-- create db link for clone
oracle@ora21:/home/oracle/ [cdb01] sqh
SQL*Plus: Release 21.0.0.0.0 - Production on Fri Sep 17 13:18:31 2021
Version 21.3.0.0.0
Copyright (c) 1982, 2021, Oracle.  All rights reserved.
Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.3.0.0.0
SQL> create database link cdb19c2 connect to system identified by manager using 'cdb19c2';
Database link created.

Now I clone the 19c pdb into the 21c CDB over the dblink

SQL> create pluggable database pdb19c_copy from pdb19c@cdb19c2;
Pluggable database created.

Upgrade PDB to 21c
So the next step is to open the PDB and afterwards execute the upgrade script for upgrading the catalog of the pdb to 21c.

SQL> alter pluggable database pdb19c_copy open;

And now with Oracle 21c the magic part happens, or i would say, that a autonoumous Feature found the way to the on prem word.
After a few seconds the opening of the PDB was still not finished, but no error was displayed on the prompt, I looked into the alert log of the database to see what exactly the problem was. When I looked through the entries, I was quite surprised to see that my PDB was in the middle of an upgrade to 21c without Oracle giving me a choice.
A look on the pdb state confirmed what oracle tells me in the alert.log

SQL> show pdbs

CON_ID     CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
2          PDB$SEED                       READ ONLY NO
3          PDB19c_COPY                    MIGRATE   YES

Here you can see a very interesting snipped of the alert.log

PDB19C_COPY(6):Starting Upgrade on PDB Open
2021-09-17T13:21:09.305411+02:00
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture
PDB19C_COPY(6):ORA-65229 signalled during: alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture...
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG end install '19.0.0.0.0'
PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG end install '19.0.0.0.0'
PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade
'19.0.0.0.0' to '21.0.0.0.0.partial' on error capture
PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade
'19.0.0.0.0' to '21.0.0.0.0.partial' on error capture

On the first line you can see, that oracle has started to upgrade my pdb immediate after i issued the open command.
The next lines are very interesting. If you look at the statements beginning with “alter pluggable database application … ” you can see, that oracle uses Multitenant application container functionality to upgrade the pdb to the new version. It seams, that they manage different versions of the cdb catalog as applications.

With the next command oracle tries to upgrade the application APP$CDB$CATALOG to 19.0.0.0.0 what’s in fact the data dictionary version. This gives an error. I can’t confirm it, but in context of application containers the begin install command always install the newest application version in the pdb. In this context i would expect, that oracle tries to apply the newest 19c patches to my pdb but this failed for some reason.

DB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture 
PDB19C_COPY(6):ORA-65229 signalled during: alter pluggable database application APP$CDB$CATALOG begin install '19.0.0.0.0' on error capture...

Then the next stage is upgrade of the catalog from version 19.0.0.0.0 to 21.0.0.0.0.partial. The syntax is exactly the same if you upgrade an application pdb to a new version.

PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade 
'19.0.0.0.0' to '21.0.0.0.0.partial' on error capture 
PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade '19.0.0.0.0' to '21.0.0.0.0.partial' on error capture

And in the last step oracle performs the upgrade from the version 21.0.0.0.0.partial to 21.0.0.0.0. I have no idea why this additional step to 21.0.0.0.0.partial is necessary.

PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG begin upgrade '21.0.0.0.0.partial' to '21.0.0.0.0' on error capture
PDB19C_COPY(6):Completed: alter pluggable database application APP$CDB$CATALOG begin upgrade '21.0.0.0.0.partial' to '21.0.0.0.0' on error capture

Finally the Upgrade of the Application APP$CDB$CATALOG is done and oracle open the pdb with the new version 21c

PDB19C_COPY(6):alter pluggable database application APP$CDB$CATALOG end upgrade
Pluggable database PDB19C_COPY opened read write

Reverse Engineering
To see if my assumptions are correct, we can query the most important app container views in the CDB and the PDB and see what is in there.
Let’s check the applications that are registered in the cdb$root

SQL> select app_name, app_id, app_version, app_capture_module from dba_applications;

APP_NAME                           APP_ID APP_VERSION                    APP_CAPTURE_MODULE
------------------------------ ---------- ------------------------------ ----------------------------------------------------------------
APP$CDB$SYSTEM                          2 1.0                            catcon(pid=26657)
APP$CDB$PDBONLY$NCDBTOPDB              21 1.0                            catcon(pid=26657)
APP$CDB$CATALOG                4294967293 21.0.0.0.0                     catcon(pid=24698)
APP$CDB$CATALOG                4294967292 21.0.0.0.0                     catcon(pid=8832)
APP$CDB$CATALOG                4294967291 21.0.0.0.0                     catcon(pid=1120)

We can see that we have an application APP$CDB$CATALOG that holds the upgrade scripts to upgrade a pdb to 21c and also we have a application APP$CDB$PDBONLY$NCDBTOPDB which is responsible auto upgrading a non cdb to 21c and additionally including the scripts for the conversion from noncdb to pdb
And if we have a look at the app versions, we can see, that oracle stores the catalog upgrade scripts from 12.2 onwards. This allows to auto upgrade the pdb to 21c for all versions beginning with 12.2

SQL> select app_name, app_version from dba_app_versions;

APP_NAME                                 APP_VERSION
---------------------------------------- ------------------------------
APP$CDB$SYSTEM                           1.0
APP$CDB$PDBONLY$NCDBTOPDB                1.0.upgmode
APP$CDB$PDBONLY$NCDBTOPDB                1.0
APP$CDB$CATALOG                          12.2.0.1.0
APP$CDB$CATALOG                          21.0.0.0.0.partial
APP$CDB$CATALOG                          21.0.0.0.0
APP$CDB$CATALOG                          18.0.0.0.0
APP$CDB$CATALOG                          21.0.0.0.0.partial
APP$CDB$CATALOG                          21.0.0.0.0
APP$CDB$CATALOG                          19.0.0.0.0
APP$CDB$CATALOG                          21.0.0.0.0.partial
APP$CDB$CATALOG                          21.0.0.0.0

Replay Upgrade Feature
To learn a bit more about what exactly Oracle is doing here, I talked to Mike Dietrich. He confirmed my experience and told me that the feature implemented with 21c is called “Replay Upgrade”. The process to upgrade a PDB is stored in the CDB from 21c on and as soon as a PDB with a lower version is opened, this recorded process is automatically replayed to upgrade the PDB. This new feature is even rudimentarily documented here.
A big disadvantage with this automatic upgrade process is that no single upgrade log is created for it. In my lab, which consists of an empty PDB, it worked without problems, but for real databases, this is not recommended, since no analysis can be done in case of an error. Also, the replay upgrade does not update the Timezone file to the latest version. This must be done manually afterwards. This was also confirmed by Mike Dietrich and therefore for an upgrade you should always work with the “autoupgrade tool”, which is officially recommended by Oracle.
Based on the documentation, the replay feature in the CDB can also be disabled. I would recommend this, because you have back the full control over upgrades afterwards.

ALTER DATABASE UPGRADE SYNC OFF

To be complete, the feature can be reactivated with the following command.

ALTER DATABASE UPGRADE SYNC ON

Container Compatibility
The Replay Upgrade feature only applies if exactly the same components are installed in the Source CDB and the Target CDB. If this is not the case, the PDB clone cannot be created at all and terminates with an error.

SQL> create pluggable database pdb19_copy from pdb19@cdb19c;
create pluggable database pdb19_copy from pdb19@cdb19c
*
ERROR at line 1:
ORA-65346: The PDB version is lower and components (APS, CONTEXT, DV, OLS,
ORDIM, SDO, XOQ) are missing in CDB.