Details
-
Bug
-
Status: Closed (View Workflow)
-
Major
-
Resolution: Fixed
-
None
-
None
-
2025-11
Description
cmapi systemd service depends on network-online target, so by the time loadbrm is called, all the network interfaces must be ready and have IP addresses.
But we've seen in the logs that loadbrm wasn't able to extract ip addresses (and we know that OS is configured and later obtains IP addresses).
Nov 25 15:25:19 prod-cs-02.insight-health.de systemd[1]: Starting loadbrm... |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Detecting CMAPI is up and running on 127.0.0.1. |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: INFO: CMAPI seems to be unavailable. |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: INFO: Trying to detect primary without cmapi running. |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Current DBRM_Controller/IPAddr is prod-cs-01 |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "ens3f0" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "eno1" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "ens3f1" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "ens4f0" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "eno2" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "ens4f1" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "eno145" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "eno146" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "ens2f0" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Exception while getting IP address of an "ens2f1" interface |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: Traceback (most recent call last): |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: File "/usr/bin/mcs-loadbrm.py", line 111, in get_ip_address_by_nic |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: struct.pack('256s', bytes(ifname[:15], 'utf-8')) |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: OSError: [Errno 99] Cannot assign requested address |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Found hostnames {'localhost4.localdomain4', 'localhost4', 'localhost.localdomain', 'localhost'}. |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: DEBUG: Detecting CMAPI is up and running on prod-cs-01. |
Nov 25 15:25:19 prod-cs-02.insight-health.de mcs-loadbrm.py[3146]: INFO: CMAPI seems to be unavailable. |
Only localhosts were found (loopback interfaces are ready).
I suspect the problem was in the bond interfaces that can presumably cause network-online to signal readiness before its slave interfaces get the IPs. So we can add retries if hostnames extraction returns only localhosts.
We don't really know how to reproduce it, but adding this check and retries will not be harmful and may help us in the future.