Views: 3557
Troubleshoot Fencing on Redhat or Centos Linux
We are setting up fencing for Enterprise Customers for Openstack. In Linux fencing is called for whenever you have a cluster. Using PCS and fencing the other servers can turn off a damaged server until it is fixed.
Pay attention that mac address is the provision nic
However, the instack doesn’t require the mac
the script to create fencing.yaml fails because it requires the “optional’ field of MAC address. It seems that the MAC address is not really required for the fencing since IPMI uses the IP address , Show output of -v command
the link testing fencing says to try out different options to see which can control the IPMI. We are using Dell hardware with IDRAC
How to test fencing
from 6.1 of https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/director_installation_and_usage/index
mac
(Optional) A list of MAC addresses for the network interfaces on the node. Use only the MAC address for the Provisioning NIC of each system.
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/sect-fencing_the_controller_nodes
https://access.redhat.com/solutions/18803
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/sect-fencing_the_controller_nodes#test_fencing
bug for ipmilan https://access.redhat.com/solutions/1410613
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/ch-fencing-haar
https://access.redhat.com/solutions/15575 – fencing explained
ipmitool channel getciphers ipmi 1 -H 10.70.130.220 -U admin -P changeme
this command got erros
Get Session Challenge command failed
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session
from 21.2 https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/sect-fencing_the_controller_nodes
(undercloud) [stack@vmundercloudrvt overcloud-templates]$ openstack overcloud generate fencing –ipmi-lanplus –ipmi-level administrator ../instackenv.json
Action tripleo.parameters.generate_fencing execution failed: Failed to run action [action_ex_id=None, action_cls='<class ‘mistral.actions.action_factory.GenerateFencingParametersAction’>’, attributes='{}’, params='{u’ipmi_level’: u’administrator’, u’ipmi_cipher’: None, u’ipmi_lanplus’: True, u’delay’: None, u’os_auth’: None, u’nodes_json’: [{u’pm_password’: u’changeme’, u’name’: u’rvtcst01′, u’pm_type’: u’pxe_ipmitool’, u’pm_addr’: u’10.70.131.120′, u’capabilities’: u’node:rvtcst-0,boot_option:local’, u’memory’: u’131072′, u’disk’: u’558′, u’arch’: u’x86_64′, u’cpu’: u’2′, u’pm_user’: u’admin’}, {u’pm_password’: u’changeme’, u’name’: u’rvtcst03′, u’pm_type’: u’pxe_ipmitool’, u’pm_addr’: u’10.70.131.122′, u’capabilities’: u’node:rvtcst-2,boot_option:local’, u’memory’: u’131072′, u’disk’: u’558′, u’arch’: u’x86_64′, u’cpu’: u’2′, u’pm_user’: u’admin’}]}’]
local variable ‘mac_addr’ referenced before assignment
(undercloud) [stack@vmundercloudrvt overcloud-templates]$
fence_ilo -a 10.70.130.220 -l admin -p changeme -o status
[root@rhcco-220 ~]# fence_idrac -a 10.70.130.220 -l admin -p changeme -o status
2019-04-23 15:15:02,493 ERROR: Failed: Unable to obtain correct plug status or plug is not available
[root@rhcco-220 ~]# fence_ipmilan -a 10.70.130.220 -l admin -p changeme -o status -v
2019-04-23 15:29:26,066 INFO: Executing: /usr/bin/ipmitool -I lan -H 10.70.130.220 -p 623 -U admin -P [set] -L ADMINISTRATOR chassis power status
2019-04-23 15:29:34,151 DEBUG: 1 Get Session Challenge command failed
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session
2019-04-23 15:29:34,153 ERROR: Failed: Unable to obtain correct plug status or plug is not available
[root@rhcco-220 ~]# which ipmitool
/bin/ipmitool
[root@rhcco-220 ~]# fence_ilo5 -a 10.70.130.220 -l admin -p changeme -o status -v
2019-04-23 15:30:06,835 INFO: Executing: /usr/bin/ipmitool -I lanplus -H 10.70.130.220 -p 623 -U admin -P [set] -L ADMINISTRATOR chassis power status
2019-04-23 15:30:06,871 DEBUG: 0 Chassis Power is on
[root@rhcco-220 ~]# fence_idrac -a 10.70.130.220 -l admin -p changeme -o status -v
2019-04-23 15:32:58,987 INFO: Executing: /usr/bin/ipmitool -I lan -H 10.70.130.220 -p 623 -U admin -P [set] -L ADMINISTRATOR chassis power status
2019-04-23 15:33:07,026 DEBUG: 1 Get Session Challenge command failed
Error: Unable to establish LAN session
Error: Unable to establish IPMI v1.5 / RMCP session
2019-04-23 15:33:07,027 ERROR: Failed: Unable to obtain correct plug status or plug is not available
[root@rhcco-220 ~]#
[root@rhcco-220 ~]# pcs stonith describe fence_ipmilan
fence_ipmilan – Fence agent for IPMI
fence_ipmilan is an I/O Fencing agentwhich can be used with machines controlled by IPMI.This agent calls support software ipmitool (http://ipmitool.sf.net/). WARNING! This fence agent might report success before the node is powered off. You should use -m/method onoff if your fence device works correctly with that option.
Stonith options:
auth: IPMI Lan Auth type.
cipher: Ciphersuite to use (same as ipmitool -C parameter)
hexadecimal_kg: Hexadecimal-encoded Kg key for IPMIv2 authentication
inet4_only: Forces agent to use IPv4 addresses only
inet6_only: Forces agent to use IPv6 addresses only
ipaddr: IP address or hostname of fencing device
ipport: TCP/UDP port to use for connection with device
lanplus: Use Lanplus to improve security of connection
login: Login name
method: Method to fence
passwd: Login password or passphrase
passwd_script: Script to run to retrieve password
port: IP address or hostname of fencing device (together with –port-as-ip)
privlvl: Privilege level on IPMI device
target: Bridge IPMI requests to the remote target address
quiet: Disable logging to stderr. Does not affect –verbose or –debug-file or logging to syslog.
verbose: Verbose mode
debug: Write debug information to given file
delay: Wait X seconds before fencing is started
ipmitool_path: Path to ipmitool binary
login_timeout: Wait X seconds for cmd prompt after login
port_as_ip: Make “port/plug” to be an alias to IP address
power_timeout: Test X seconds for status change after ON/OFF
power_wait: Wait X seconds after issuing ON/OFF
shell_timeout: Wait X seconds for cmd prompt after issuing command
retry_on: Count of attempts to retry power on
sudo: Use sudo (without password) when calling 3rd party software
sudo_path: Path to sudo binary
priority: The priority of the stonith resource. Devices are tried in order of highest priority to lowest.
pcmk_host_map: A mapping of host names to ports numbers for devices that do not support host names. Eg. node1:1;node2:2,3 would tell the cluster to use port 1 for node1 and ports 2 and 3 for node2
pcmk_host_list: A list of machines controlled by this device (Optional unless pcmk_host_check=static-list).
pcmk_host_check: How to determine which machines are controlled by the device. Allowed values: dynamic-list (query the device), static-list (check the pcmk_host_list attribute), none (assume every device can fence every machine)
pcmk_delay_max: Enable a random delay for stonith actions and specify the maximum of random delay. This prevents double fencing when using slow devices such as sbd. Use this to enable a random delay for stonith actions. The overall
delay is derived from this random delay value adding a static delay so that the sum is kept below the maximum delay.
pcmk_delay_base: Enable a base delay for stonith actions and specify base delay value. This prevents double fencing when different delays are configured on the nodes. Use this to enable a static delay for stonith actions. The overall
delay is derived from a random delay value adding this static delay so that the sum is kept below the maximum delay.
pcmk_action_limit: The maximum number of actions can be performed in parallel on this device Pengine property concurrent-fencing=true needs to be configured first. Then use this to specify the maximum number of actions can be performed
in parallel on this device. -1 is unlimited.
Default operations:
monitor: interval=60s
[root@rhcco-220 ~]#
[root@rhcco-220 ~]# pcs stonith show stonith-fence_ipmilan-f8f21e4a4600
Resource: stonith-fence_ipmilan-f8f21e4a4600 (class=stonith type=fence_ipmilan)
Attributes: ipaddr=10.70.130.221 lanplus=true login=admin passwd=changeme pcmk_host_list=rhcco-1 privlvl=administrator power_timeout=21
Operations: monitor interval=60s (stonith-fence_ipmilan-f8f21e4a4600-monitor-interval-60s)
[root@rhcco-220 ~]#
Steve Bar Yakov Gindi
OpenStack and DevOps Expert