By Mouhamadou Diaw
With Oracle 21c (In fact Oracle 20c), we have a nice new feature with the broker which is the Fast-start Failover Callout Configuration scripts.
I talked about this feature in some presentations I did about the new features with Oracle 21c for the broker
This feature will allow to automate tasks that must be performed before and after a fast-start failover operation.
Yes this means that with Oracle 21c, we can configure some tasks to be executed before and after doing a fast_start failover. Sure that this feature will make life easier.
The use of this feature is very simple
We have to configure the DG_ADMIN variable which will be the default location of Client-side Broker Files. See my previous blog
The callout scripts should be located in the $DG_ADMIN/config_ConfigurationSimpleName/callout.
The configuration file must be named fsfocallout.ora, and it’s mandatory.
This fsfocallout.ora file will contain the names of the pre-callout configuration script and post-callout configuration script.
The pre_callout script must create a .suc file and a .err file. Before doing a FSFO, then the observer will check for .suc file or a for a .err file. If a .suc file exists then the observer will initiate a FSFO.
The .suc file and the .err file must be at the same location that the callout scripts.
Below the configuration I am using
oraadserver : 192.168.2.21 for the primary database
oraadserver2 : 192.168.2.22 for the standby database
oraadserver3 : 192.168.2.23 for the observer
The FSFO is already configured
1
2
3
4
5
6
7
8
9
10
11
12
13
|
DGMGRL> show configuration Configuration - db21 Protection Mode: MaxPerformance Members: DB21_SITE1 - Primary database DB21_SITE2 - (*) Physical standby database Fast-Start Failover: Enabled in Potential Data Loss Mode Configuration Status: SUCCESS (status updated 5 seconds ago) |
And the observer is running
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
DGMGRL> show observer Configuration - db21 Fast-Start Failover: ENABLED Primary : DB21_SITE1 Active Target: DB21_SITE2 Observer "oraadserver3" - Master Host Name : oraadserver3 Last Ping to Primary : 2 seconds ago Last Ping to Target: 0 seconds ago Log File: /u01/app/oracle/admin/prod20/broker_files/config_db21/log/observer_oraadserver3.log State File: /u01/app/oracle/admin/prod20/broker_files/config_db21/dat/fsfo.dat DGMGRL> |
On my observer server, the $DG_ADMIN variable is configured (directories of the DG_ADMIN must have some prerequisites (see previous blog or the documentation )
1
2
3
|
[oracle@oraadserver3 callout]$ echo $DG_ADMIN/ /u01/app/oracle/admin/prod20/broker_files/ [oracle@oraadserver3 callout]$ |
And in the $DG_ADMIN
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
|
[oracle@oraadserver3 callout]$ ls -lR $DG_ADMIN/ /u01/app/oracle/admin/prod20/broker_files/ : total 0 drwx------. 2 oracle oinstall 6 Feb 3 2021 admin drwx------. 5 oracle oinstall 40 Feb 25 11:35 config_db21 /u01/app/oracle/admin/prod20/broker_files/admin : total 0 /u01/app/oracle/admin/prod20/broker_files/config_db21 : total 4 drwx------. 2 oracle oinstall 99 Mar 4 16:36 callout drwx------. 2 oracle oinstall 21 Nov 29 13:50 dat drwx------. 2 oracle oinstall 4096 Feb 25 13:35 log /u01/app/oracle/admin/prod20/broker_files/config_db21/callout : total 12 -rwx------. 1 oracle oinstall 979 Mar 4 16:24 fsfocallout.ora -rwx------. 1 oracle oinstall 57 Feb 25 11:36 fsfo_postcallout -rwx------. 1 oracle oinstall 351 Mar 4 16:35 fsfo_precallout -rw-r--r--. 1 oracle oinstall 0 Mar 4 16:36 fsfo_precallout.suc /u01/app/oracle/admin/prod20/broker_files/config_db21/dat : total 12 -rw-r--r--. 1 oracle oinstall 8344 Mar 4 16:48 fsfo.dat /u01/app/oracle/admin/prod20/broker_files/config_db21/log : total 84 -rw-r-----. 1 oracle oinstall 66093 Mar 4 16:48 observer_oraadserver3.log [oracle@oraadserver3 callout]$ |
In the callout directory, we have the FSFO callout scripts
1
2
3
4
5
6
7
8
|
oracle@oraadserver: /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] pwd /u01/app/oracle/admin/prod20/broker_files/config_db21/callout oracle@oraadserver: /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] ls -l total 12 -rwx------. 1 oracle oinstall 882 Feb 1 16:11 fsfocallout.ora -rwx------. 1 oracle oinstall 37 Feb 1 16:48 fsfo_postcallout -rwx------. 1 oracle oinstall 38 Feb 1 16:47 fsfo_precallout oracle@oraadserver: /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/ [DB21 (CDB$ROOT)] |
Let’s see the contents of each file
fsfocallout.ora : fsfo callout configuration file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
|
[oracle@oraadserver3 callout]$ cat fsfocallout.ora # This is a fast-start failover configuration file. # The pre-callout script that is run before fast-start failover is enabled. FastStartFailoverPreCallout=fsfo_precallout # The timeout value (in seconds) for pre-callout script #FastStartFailoverPreCalloutTimeout=1200 FastStartFailoverPreCalloutTimeout=25 # The name of the suc file created by the pre-callout script. FastStartFailoverPreCalloutSucFileName=fsfo_precallout.suc # The name of the error file that the pre-callout script creates FastStartFailoverPreCalloutErrorFileName=fsfo_precallout.err # Action taken by observer if the suc file does not exist after FastStartFailoverPreCalloutTimeout seconds # or if an error file is detected before FastStartFailoverPreCalloutTimeout seconds passed FastStartFailoverActionOnPreCalloutFailure=STOP #FastStartFailoverActionOnPreCalloutFailure=CONTINUE # The post-callout script that is run after fast-start failover succeeds FastStartFailoverPostCallout=fsfo_postcallout [oracle@oraadserver3 callout]$ |
fsfo_precallout : to be executed before the FSFO
1
2
3
4
5
6
7
8
9
10
11
|
[oracle@oraadserver3 callout]$ cat fsfo_precallout #! /bin/bash if [ 1 -lt 100 ] then touch /tmp/test echo "starting fun observer" > /tmp/test touch /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout .suc else touch /u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout .err fi [oracle@oraadserver3 callout]$ |
fsfo_postcallout : to be executed after the FSFO
1
2
3
4
5
|
[oracle@oraadserver3 callout]$ cat fsfo_postcallout #! /bin/bash touch /tmp/test_2 echo "end fun observer" >> /tmp/test_2 [oracle@oraadserver3 callout]$ |
Now let’s simulate a FSFO and let’s see if the pre-callout and post-callout tasks are executed
For this we will execute a shut abort of the primary database
1
2
3
4
5
6
7
8
9
|
SQL> select db_unique_name from v$ database ; DB_UNIQUE_NAME ------------------------------ db21_site1 SQL> shut abort ORACLE instance shut down. SQL> |
In the observer logfile, we can see following
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
|
[oracle@oraadserver3 callout]$ cat /u01/app/oracle/admin/prod20/broker_files/config_db21/log/observer_oraadserver3 .log 1 [W000 2022-03-04T17:34:19.533+01:00] Primary database cannot be reached. [W000 2022-03-04T17:34:19.534+01:00] Fast-Start Failover threshold has not exceeded. Retry for the next 15 seconds [W000 2022-03-04T17:34:20.534+01:00] Try to connect to the primary. [W000 2022-03-04T17:34:33.053+01:00] Primary database cannot be reached. [W000 2022-03-04T17:34:33.053+01:00] Fast-Start Failover threshold has not exceeded. Retry for the next 1 second [W000 2022-03-04T17:34:34.064+01:00] Try to connect to the primary. [W000 2022-03-04T17:34:36.084+01:00] Primary database cannot be reached. [W000 2022-03-04T17:34:36.084+01:00] Fast-Start Failover threshold has expired. [W000 2022-03-04T17:34:36.085+01:00] Succeeded to parse FSFO callout config file '/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfocallout.ora' [W000 2022-03-04T17:34:36.085+01:00] Try to connect to the standby. [W000 2022-03-04T17:34:36.085+01:00] Check if the standby is ready for failover. [W000 2022-03-04T17:34:36.106+01:00] Doing pre-FSFO callout. [W000 2022-03-04T17:34:37.107+01:00] Found the pre-FSFO callout suc file '/u01/app/oracle/admin/prod20/broker_files/config_db21/callout/fsfo_precallout.suc' [W000 2022-03-04T17:34:37.107+01:00] Will continue Fast-Start Failover. [S025 2022-03-04T17:34:37.107+01:00] Fast-Start Failover started... 2022-03-04T17:34:37.107+01:00 Initiating Fast-Start Failover to database "DB21_SITE2" ... [S025 2022-03-04T17:34:37.108+01:00] Initiating Fast-start Failover. 2022-03-04T17:34:37.124+01:00 Performing failover NOW, please wait... 2022-03-04T17:34:57.927+01:00 Failover succeeded, new primary is "DB21_SITE2" . 2022-03-04T17:34:57.927+01:00 Failover processing complete, broker ready. 2022-03-04T17:34:57.927+01:00 [S025 2022-03-04T17:34:57.927+01:00] Fast-Start Failover finished... [W000 2022-03-04T17:34:57.927+01:00] Failover succeeded. Restart pinging. [W000 2022-03-04T17:34:57.937+01:00] Primary database has changed to DB21_SITE2. [W000 2022-03-04T17:34:57.940+01:00] Try to connect to the primary. … … [W000 2022-03-04T17:35:08.317+01:00] Failed to ping the new standby. [W000 2022-03-04T17:35:08.317+01:00] Try to connect to the new standby DB21_SITE1. [W000 2022-03-04T17:35:11.317+01:00] Connection to the new standby restored! [W000 2022-03-04T17:35:11.331+01:00] Failed to ping the new standby. [W000 2022-03-04T17:35:12.345+01:00] Try to connect to the new standby DB21_SITE1. [W000 2022-03-04T17:35:15.331+01:00] Connection to the new standby restored! [W000 2022-03-04T17:35:17.322+01:00] Failed to ping the new standby. [W000 2022-03-04T17:35:17.322+01:00] Try to connect to the new standby DB21_SITE1. [W000 2022-03-04T17:35:23.322+01:00] Connection to the new standby restored! [W000 2022-03-04T17:35:39.344+01:00] Try to connect to the primary DB21_SITE2. [W000 2022-03-04T17:35:41.324+01:00] Connection to the primary restored! [W000 2022-03-04T17:35:41.324+01:00] Wait for new primary to be ready to reinstate. [W000 2022-03-04T17:35:42.342+01:00] New primary is now ready to reinstate. [W000 2022-03-04T17:35:43.342+01:00] Issuing REINSTATE command . [oracle@oraadserver3 callout]$ |
In the callout directory we have the .suc file
1
2
3
|
[oracle@oraadserver3 callout]$ ls -ltra *.suc -rw-r--r--. 1 oracle oinstall 0 Mar 4 17:34 fsfo_precallout.suc [oracle@oraadserver3 callout]$ |
And in /tmp directory
1
2
3
4
|
[oracle@oraadserver3 callout]$ ls -l /tmp/test * -rw-r--r--. 1 oracle oinstall 22 Mar 4 17:34 /tmp/test -rw-r--r--. 1 oracle oinstall 17 Mar 4 17:34 /tmp/test_2 [oracle@oraadserver3 callout]$ |
1
2
3
4
5
|
[oracle@oraadserver3 callout]$ cat /tmp/test starting fun observer [oracle@oraadserver3 callout]$ cat /tmp/test_2 end fun observer [oracle@oraadserver3 callout]$ |
Conclusion
Sure, that this feature will help (configuring clients before FSFO, stopping application ….)