Infrastructure at your Service

Before Oracle 12.1 the methods used to configure ASM were
• udev
• asmlib
Oracle 12.1 comes with a new method called Oracle ASM Filter Driver (Oracle ASMFD).
In Oracle documentation we can find following:
Oracle ASM Filter Driver (Oracle ASMFD) is a kernel module that resides in the I/O path of the Oracle ASM disks. Oracle ASM uses the filter driver to validate write I/O requests to Oracle ASM disks.
The Oracle ASMFD simplifies the configuration and management of disk devices by eliminating the need to rebind disk devices used with Oracle ASM each time the system is restarted.
The Oracle ASM Filter Driver rejects any I/O requests that are invalid. This action eliminates accidental overwrites of Oracle ASM disks that would cause corruption in the disks and files within the disk group. For example, the Oracle ASM Filter Driver filters out all non-Oracle I/Os which could cause accidental overwrites.

In the following blog I am going to migrate from asmlib to asmfd. I am using a cluster 12.1 with 2 nodes.

Below we present our actual configuration.

[[email protected] ~]# crsctl check cluster -all
**************************************************************
rac12a:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac12b:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[[email protected] ~]#


[[email protected] ~]# crsctl get cluster mode status
Cluster is running in "flex" mode
[[email protected] ~]#

[[email protected] ~]# ps -ef | grep pmon
grid      7217     1  0 11:20 ?        00:00:00 asm_pmon_+ASM1
grid      8070     1  0 11:21 ?        00:00:00 apx_pmon_+APX1
oracle    8721     1  0 11:22 ?        00:00:00 ora_pmon_mydb_1
root     14395  2404  0 11:32 pts/0    00:00:00 grep --color=auto pmon
[[email protected] ~]#

First let’s get information about our ASM disks. We will use these outputs later to migrate the disks to ASMFD disks

[[email protected] ~]# oracleasm listdisks | xargs oracleasm querydisk -p             
Disk "ASM_DATA" is a valid ASM disk
/dev/sdc1: LABEL="ASM_DATA" TYPE="oracleasm"
Disk "ASM_DIVERS" is a valid ASM disk
/dev/sdd1: LABEL="ASM_DIVERS" TYPE="oracleasm"
Disk "ASM_OCR1" is a valid ASM disk
/dev/sdg1: LABEL="ASM_OCR1" TYPE="oracleasm"
Disk "ASM_OCR2" is a valid ASM disk
/dev/sdi1: LABEL="ASM_OCR2" TYPE="oracleasm"
Disk "ASM_VOT1" is a valid ASM disk
/dev/sde1: LABEL="ASM_VOT1" TYPE="oracleasm"
Disk "ASM_VOT2" is a valid ASM disk
/dev/sdh1: LABEL="ASM_VOT2" TYPE="oracleasm"
Disk "ASM_VOT3" is a valid ASM disk
/dev/sdf1: LABEL="ASM_VOT3" TYPE="oracleasm"
[[email protected] ~]#

To migrate to ASMFD, we first have to change the value of the parameter diskstring for the ASM instance. The actual value can be get by using

[[email protected] trace]$ asmcmd dsget
parameter:ORCL:*
profile:ORCL:*
[[email protected] trace]$

Let’s set the new value on both nodes

[email protected] trace]$ asmcmd dsset 'ORCL:*','AFD:*'

We can then verify

[[email protected] trace]$ asmcmd dsget
parameter:ORCL:*, AFD:*
profile:ORCL:*,AFD:*
[[email protected] trace]$

Once the new value of the diskstring set, let stop the cluster on both nodes

[[email protected] ~]# crsctl stop cluster
[[email protected] ~]# crsctl stop cluster

Once the cluster is stopped we have to disable and stop asmlib on both nodes

[[email protected] ~]# systemctl disable oracleasm
Removed symlink /etc/systemd/system/multi-user.target.wants/oracleasm.service.

[[email protected] ~]# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

[[email protected] ~]# oracleasm exit
Unmounting ASMlib driver filesystem: /dev/oracleasm
Unloading module "oracleasm": oracleasm
[[email protected] ~]#

[[email protected] ~]# ls -ltr /dev/oracleasm/
total 0
[[email protected] ~]#

Now let’s remove all packages relative to ASMLIB on both nodes

[[email protected] oracle]# rpm -e oracleasm-support-2.1.11-2.el7.x86_64 oracleasmlib-2.0.12-1.el7.x86_64
warning: /etc/sysconfig/oracleasm saved as /etc/sysconfig/oracleasm.rpmsave
[[email protected] oracle]#

The next step is to stop acfsload on both nodes

[[email protected] ~]# lsmod | grep acfs
oracleacfs           3343483  0
oracleoks             500109  2 oracleacfs,oracleadvm
[[email protected] ~]#

[[email protected] ~]# acfsload stop
[[email protected] ~]# lsmod | grep acfs
[[email protected] ~]#

As root, we can now configure Oracle ASMFD to filter at the node level. In my case steps were done on both nodes

[[email protected] oracle]# asmcmd afd_configure
Connected to an idle instance.
AFD-627: AFD distribution files found.
AFD-636: Installing requested AFD software.
AFD-637: Loading installed AFD drivers.
AFD-9321: Creating udev for AFD.
AFD-9323: Creating module dependencies - this may take some time.
AFD-9154: Loading 'oracleafd.ko' driver.
AFD-649: Verifying AFD devices.
AFD-9156: Detecting control device '/dev/oracleafd/admin'.
AFD-638: AFD installation correctness verified.
[[email protected] oracle]#

Once the configuration done, we can check AFD state on all nodes

[[email protected] oracle]# asmcmd afd_state
Connected to an idle instance.
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'DISABLED' on host 'rac12a.localdomain'
[[email protected] oracle]#

We can see that afd module is loaded but the filtering is disabled. We then have to edit the oracleafd.conf to enable the filtering

[[email protected] etc]# cat oracleafd.conf
afd_diskstring='/dev/sd*1'

And then we have to run on both nodes

[[email protected] etc]# asmcmd afd_filter -e
Connected to an idle instance.
[[email protected] etc]#

[[email protected] ~]#  asmcmd afd_filter -e
Connected to an idle instance.
[[email protected] ~]#

Running again the afd_state command, we can confirm that the filtering is now enabled.

[[email protected] etc]# asmcmd afd_state
Connected to an idle instance.
ASMCMD-9526: The AFD state is 'LOADED' and filtering is 'ENABLED' on host 'rac12a.localdomain'
[[email protected] etc]#

Now we can migrate all asm disks.

[[email protected] etc]# asmcmd afd_label ASM_DATA /dev/sdc1 --migrate
Connected to an idle instance.
[[email protected] etc]# asmcmd afd_label ASM_DIVERS /dev/sdd1 --migrate
Connected to an idle instance.
[[email protected] etc]# asmcmd afd_label ASM_OCR1 /dev/sdg1 --migrate
Connected to an idle instance.
[[email protected] etc]# asmcmd afd_label ASM_OCR2 /dev/sdi1 --migrate
Connected to an idle instance.
[[email protected] etc]# asmcmd afd_label ASM_VOT1 /dev/sde1 --migrate
Connected to an idle instance.
[[email protected] etc]# asmcmd afd_label ASM_VOT2 /dev/sdh1 --migrate
Connected to an idle instance.
[[email protected] etc]# asmcmd afd_label ASM_VOT3 /dev/sdf1 --migrate
Connected to an idle instance.
[[email protected] etc]#

We can verify the ASMFD disks using the command

[[email protected] ~]# asmcmd afd_lsdsk
Connected to an idle instance.
--------------------------------------------------------------------------------
Label                     Filtering   Path
================================================================================
ASM_DATA                    ENABLED   /dev/sdc1
ASM_DIVERS                  ENABLED   /dev/sdd1
ASM_OCR1                    ENABLED   /dev/sdg1
ASM_OCR2                    ENABLED   /dev/sdi1
ASM_VOT1                    ENABLED   /dev/sde1
ASM_VOT2                    ENABLED   /dev/sdh1
ASM_VOT3                    ENABLED   /dev/sdf1
[[email protected] ~]#

Let’s update the afd.conf so that ASMFD can mount ASMFD disks.

[[email protected] etc]# cat afd.conf
afd_diskstring='/dev/sd*'
afd_filtering=enable

When the ASMFD disks are visible on both nodes, we can start acsfload on both nodes

[[email protected] etc]# acfsload start
ACFS-9391: Checking for existing ADVM/ACFS installation.
ACFS-9392: Validating ADVM/ACFS installation files for operating system.
ACFS-9393: Verifying ASM Administrator setup.
ACFS-9308: Loading installed ADVM/ACFS drivers.
ACFS-9154: Loading 'oracleoks.ko' driver.
ACFS-9154: Loading 'oracleadvm.ko' driver.
ACFS-9154: Loading 'oracleacfs.ko' driver.
ACFS-9327: Verifying ADVM/ACFS devices.
ACFS-9156: Detecting control device '/dev/asm/.asm_ctl_spec'.
ACFS-9156: Detecting control device '/dev/ofsctl'.
ACFS-9322: completed
[[email protected] etc]#

Now the conversion is done and we can start crs on both nodes

[[email protected] ~]# crsctl start crs

[[email protected] ~]# crsctl start crs

We can remove all asmlib references in the parameter diskstring

[[email protected] trace]$ asmcmd dsget
parameter:ORCL:*, AFD:*
profile:ORCL:*,AFD:*

[[email protected] trace]$ asmcmd dsset 'AFD:*'

[[email protected] trace]$ asmcmd dsget
parameter:AFD:*
profile:AFD:*
[[email protected] trace]$

Once the cluster started, we can verify the disk names

[[email protected] trace]$ asmcmd lsdsk
Path
AFD:ASM_DATA
AFD:ASM_DIVERS
AFD:ASM_OCR1
AFD:ASM_OCR2
AFD:ASM_VOT1
AFD:ASM_VOT2
AFD:ASM_VOT3
[[email protected] trace]$

We can also use following command to confirm that now ASMFD is being used

set linesize 300
col PATH for a20
set pages 20
col LIBRARY for a45
col NAME for a15
select inst_id,group_number grp_num,name,state,header_status header,mount_status mount,path, library
from gv$asm_disk order by inst_id,group_number,name;


   INST_ID    GRP_NUM NAME            STATE    HEADER       MOUNT   PATH                 LIBRARY
---------- ---------- --------------- -------- ------------ ------- -------------------- ---------------------------------------------
         1          1 ASM_DIVERS      NORMAL   MEMBER       CACHED  AFD:ASM_DIVERS       AFD Library - Generic , version 3 (KABI_V3)
         1          2 ASM_OCR1        NORMAL   MEMBER       CACHED  AFD:ASM_OCR1         AFD Library - Generic , version 3 (KABI_V3)
         1          2 ASM_OCR2        NORMAL   MEMBER       CACHED  AFD:ASM_OCR2         AFD Library - Generic , version 3 (KABI_V3)
         1          3 ASM_DATA        NORMAL   MEMBER       CACHED  AFD:ASM_DATA         AFD Library - Generic , version 3 (KABI_V3)
         1          4 ASM_VOT1        NORMAL   MEMBER       CACHED  AFD:ASM_VOT1         AFD Library - Generic , version 3 (KABI_V3)
         1          4 ASM_VOT2        NORMAL   MEMBER       CACHED  AFD:ASM_VOT2         AFD Library - Generic , version 3 (KABI_V3)
         1          4 ASM_VOT3        NORMAL   MEMBER       CACHED  AFD:ASM_VOT3         AFD Library - Generic , version 3 (KABI_V3)
         2          1 ASM_DIVERS      NORMAL   MEMBER       CACHED  AFD:ASM_DIVERS       AFD Library - Generic , version 3 (KABI_V3)
         2          2 ASM_OCR1        NORMAL   MEMBER       CACHED  AFD:ASM_OCR1         AFD Library - Generic , version 3 (KABI_V3)
         2          2 ASM_OCR2        NORMAL   MEMBER       CACHED  AFD:ASM_OCR2         AFD Library - Generic , version 3 (KABI_V3)
         2          3 ASM_DATA        NORMAL   MEMBER       CACHED  AFD:ASM_DATA         AFD Library - Generic , version 3 (KABI_V3)
         2          4 ASM_VOT1        NORMAL   MEMBER       CACHED  AFD:ASM_VOT1         AFD Library - Generic , version 3 (KABI_V3)
         2          4 ASM_VOT2        NORMAL   MEMBER       CACHED  AFD:ASM_VOT2         AFD Library - Generic , version 3 (KABI_V3)
         2          4 ASM_VOT3        NORMAL   MEMBER       CACHED  AFD:ASM_VOT3         AFD Library - Generic , version 3 (KABI_V3)

14 rows selected.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Mouhamadou Diaw
Mouhamadou Diaw

Consultant