Configure and test fencing for Openstack

Hits: 887

Troubleshoot Fencing on Redhat or Centos Linux

We are setting up fencing for Enterprise Customers for Openstack. In Linux fencing is called for whenever you have a cluster. Using PCS and fencing the other servers can turn off a damaged server until it is fixed.

Pay attention that mac address is the provision nic

However, the instack doesn’t require the mac

the script to create fencing.yaml fails because it requires the “optional’ field of MAC address. It seems that the MAC address is not really required for the fencing since IPMI uses the IP address , Show output of -v command

the link testing fencing says to try out different options to see which can control the IPMI. We are using Dell hardware with IDRAC

 

How to test fencing

from 6.1 of https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/director_installation_and_usage/index

mac

(Optional) A list of MAC addresses for the network interfaces on the node. Use only the MAC address for the Provisioning NIC of each system.

 

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/sect-fencing_the_controller_nodes

 

https://access.redhat.com/solutions/18803

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/sect-fencing_the_controller_nodes#test_fencing

 

bug for ipmilan https://access.redhat.com/solutions/1410613

 

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/ch-fencing-haar

 

 

https://access.redhat.com/solutions/15575 – fencing explained

 

ipmitool channel getciphers ipmi 1 -H 10.70.130.220 -U admin -P changeme

 

this command got erros

 

Get Session Challenge command failed

Error: Unable to establish LAN session

Error: Unable to establish IPMI v1.5 / RMCP session

 

 

from 21.2 https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/sect-fencing_the_controller_nodes

(undercloud) [stack@vmundercloudrvt overcloud-templates]$ openstack overcloud generate fencing –ipmi-lanplus –ipmi-level administrator ../instackenv.json

Action tripleo.parameters.generate_fencing execution failed: Failed to run action [action_ex_id=None, action_cls='<class ‘mistral.actions.action_factory.GenerateFencingParametersAction’>’, attributes='{}’, params='{u’ipmi_level’: u’administrator’, u’ipmi_cipher’: None, u’ipmi_lanplus’: True, u’delay’: None, u’os_auth’: None, u’nodes_json’: [{u’pm_password’: u’changeme’, u’name’: u’rvtcst01′, u’pm_type’: u’pxe_ipmitool’, u’pm_addr’: u’10.70.131.120′, u’capabilities’: u’node:rvtcst-0,boot_option:local’, u’memory’: u’131072′, u’disk’: u’558′, u’arch’: u’x86_64′, u’cpu’: u’2′, u’pm_user’: u’admin’}, {u’pm_password’: u’changeme’, u’name’: u’rvtcst03′, u’pm_type’: u’pxe_ipmitool’, u’pm_addr’: u’10.70.131.122′, u’capabilities’: u’node:rvtcst-2,boot_option:local’, u’memory’: u’131072′, u’disk’: u’558′, u’arch’: u’x86_64′, u’cpu’: u’2′, u’pm_user’: u’admin’}]}’]

local variable ‘mac_addr’ referenced before assignment

(undercloud) [stack@vmundercloudrvt overcloud-templates]$

 

 

 

 

 

fence_ilo -a 10.70.130.220 -l admin -p changeme  -o status

 

 

 

[root@rhcco-220 ~]# fence_idrac -a 10.70.130.220 -l admin -p changeme  -o status

2019-04-23 15:15:02,493 ERROR: Failed: Unable to obtain correct plug status or plug is not available

 

[root@rhcco-220 ~]# fence_ipmilan -a 10.70.130.220 -l admin -p changeme  -o status -v

2019-04-23 15:29:26,066 INFO: Executing: /usr/bin/ipmitool -I lan -H 10.70.130.220 -p 623 -U admin -P [set] -L ADMINISTRATOR chassis power status

 

2019-04-23 15:29:34,151 DEBUG: 1  Get Session Challenge command failed

Error: Unable to establish LAN session

Error: Unable to establish IPMI v1.5 / RMCP session

 

 

2019-04-23 15:29:34,153 ERROR: Failed: Unable to obtain correct plug status or plug is not available

 

 

[root@rhcco-220 ~]# which ipmitool

/bin/ipmitool

[root@rhcco-220 ~]# fence_ilo5  -a 10.70.130.220 -l admin -p changeme  -o status -v

2019-04-23 15:30:06,835 INFO: Executing: /usr/bin/ipmitool -I lanplus -H 10.70.130.220 -p 623 -U admin -P [set] -L ADMINISTRATOR chassis power status

 

2019-04-23 15:30:06,871 DEBUG: 0 Chassis Power is on

 

 

[root@rhcco-220 ~]# fence_idrac  -a 10.70.130.220 -l admin -p changeme  -o status -v

2019-04-23 15:32:58,987 INFO: Executing: /usr/bin/ipmitool -I lan -H 10.70.130.220 -p 623 -U admin -P [set] -L ADMINISTRATOR chassis power status

 

2019-04-23 15:33:07,026 DEBUG: 1  Get Session Challenge command failed

Error: Unable to establish LAN session

Error: Unable to establish IPMI v1.5 / RMCP session

 

 

2019-04-23 15:33:07,027 ERROR: Failed: Unable to obtain correct plug status or plug is not available

 

 

[root@rhcco-220 ~]#

 

 

[root@rhcco-220 ~]# pcs stonith describe fence_ipmilan

fence_ipmilan – Fence agent for IPMI

 

fence_ipmilan is an I/O Fencing agentwhich can be used with machines controlled by IPMI.This agent calls support software ipmitool (http://ipmitool.sf.net/). WARNING! This fence agent might report success before the node is powered off. You should use -m/method onoff if your fence device works correctly with that option.

 

Stonith options:

auth: IPMI Lan Auth type.

cipher: Ciphersuite to use (same as ipmitool -C parameter)

hexadecimal_kg: Hexadecimal-encoded Kg key for IPMIv2 authentication

inet4_only: Forces agent to use IPv4 addresses only

inet6_only: Forces agent to use IPv6 addresses only

ipaddr: IP address or hostname of fencing device

ipport: TCP/UDP port to use for connection with device

lanplus: Use Lanplus to improve security of connection

login: Login name

method: Method to fence

passwd: Login password or passphrase

passwd_script: Script to run to retrieve password

port: IP address or hostname of fencing device (together with –port-as-ip)

privlvl: Privilege level on IPMI device

target: Bridge IPMI requests to the remote target address

quiet: Disable logging to stderr. Does not affect –verbose or –debug-file or logging to syslog.

verbose: Verbose mode

debug: Write debug information to given file

delay: Wait X seconds before fencing is started

ipmitool_path: Path to ipmitool binary

login_timeout: Wait X seconds for cmd prompt after login

port_as_ip: Make “port/plug” to be an alias to IP address

power_timeout: Test X seconds for status change after ON/OFF

power_wait: Wait X seconds after issuing ON/OFF

shell_timeout: Wait X seconds for cmd prompt after issuing command

retry_on: Count of attempts to retry power on

sudo: Use sudo (without password) when calling 3rd party software

sudo_path: Path to sudo binary

priority: The priority of the stonith resource. Devices are tried in order of highest priority to lowest.

pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names. Eg. node1:1;node2:2,3 would tell the cluster to use port 1 for node1 and ports 2 and 3 for node2

pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).

pcmk_host_check: How to determine which machines are controlled by the device. Allowed values: dynamic-list (query the device), static-list (check the pcmk_host_list attribute), none (assume every device can fence every machine)

pcmk_delay_max: Enable a random delay for stonith actions and specify the maximum of random delay. This prevents double fencing when using slow devices such as sbd. Use this to enable a random delay for stonith actions. The overall

delay is derived from this random delay value adding a static delay so that the sum is kept below the maximum delay.

pcmk_delay_base: Enable a base delay for stonith actions and specify base delay value. This prevents double fencing when different delays are configured on the nodes. Use this to enable a static delay for stonith actions. The overall

delay is derived from a random delay value adding this static delay so that the sum is kept below the maximum delay.

pcmk_action_limit: The maximum number of actions can be performed in parallel on this device Pengine property concurrent-fencing=true needs to be configured first. Then use this to specify the maximum number of actions can be performed

in parallel on this device. -1 is unlimited.

 

Default operations:

monitor: interval=60s

[root@rhcco-220 ~]#

 

 

[root@rhcco-220 ~]# pcs stonith show stonith-fence_ipmilan-f8f21e4a4600

Resource: stonith-fence_ipmilan-f8f21e4a4600 (class=stonith type=fence_ipmilan)

Attributes: ipaddr=10.70.130.221 lanplus=true login=admin passwd=changeme pcmk_host_list=rhcco-1 privlvl=administrator power_timeout=21

Operations: monitor interval=60s (stonith-fence_ipmilan-f8f21e4a4600-monitor-interval-60s)

[root@rhcco-220 ~]#

 

Steve Bar Yakov Gindi

OpenStack and DevOps Expert