Issue:

IPA wait_for_open_ports: localhost [8080, 8443] timeout 300

Today we were updating the server with yum update on our IPA server.

The yum update finished successfully, but the IPA was not starting after the upgrade. the ipactl restart was also throwing an error. The web interface came up after the server restart but it was throwing “unknown error” while trying to log in via the web interface.

Below is the error from /var/log/ipaupgrade.log

2017-09-13T08:38:11Z DEBUG stderr=
2017-09-13T08:38:11Z DEBUG wait_for_open_ports: localhost [8080, 8443] timeout 300
2017-09-13T08:43:11Z ERROR IPA server upgrade failed: Inspect /var/log/ipaupgrade.log and run command ipa-server-upgrade manually.
2017-09-13T08:43:11Z DEBUG File "/usr/lib/python2.7/site-packages/ipapython/admintool.py", line 172, in execute
return_value = self.run()
File "/usr/lib/python2.7/site-packages/ipaserver/install/ipa_server_upgrade.py", line 46, in run
server.upgrade()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1913, in upgrade
upgrade_configuration()
File "/usr/lib/python2.7/site-packages/ipaserver/install/server/upgrade.py", line 1646, in upgrade_configuration
upgrade_pki(ca, fstore)
File "/usr/lib64/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/usr/lib/python2.7/site-packages/ipaserver/install/installutils.py", line 1134, in stopped_service
service_obj.start(instance_name)
File "/usr/lib/python2.7/site-packages/ipaplatform/redhat/services.py", line 211, in start
instance_name, capture_output=capture_output, wait=wait)
File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 300, in start
self.wait_for_open_ports(self.service_instance(instance_name))
File "/usr/lib/python2.7/site-packages/ipaplatform/base/services.py", line 270, in wait_for_open_ports
self.api.env.startup_timeout)
File "/usr/lib/python2.7/site-packages/ipapython/ipautil.py", line 1227, in wait_for_open_ports
raise socket.timeout("Timeout exceeded")

2017-09-13T08:43:11Z DEBUG The ipa-server-upgrade command failed, exception: timeout: Timeout exceeded
2017-09-13T08:43:11Z ERROR Timeout exceeded
2017-09-13T08:43:11Z ERROR The ipa-server-upgrade command failed. See /var/log/ipaupgrade.log for more information

As recommended on the error log, we tried running ipa-server-upgrade command but it was throwing the same error as above.

Debugging

We have verified that the telnet on localhost port 8080 and 8443 is working fine.
Tried disabling SELinux and firewalld and rebooted the server, The issue was not solved.

Solution

As we checked in detail, it turned out that the client had disabled IPv6 on the kernel as a part of PCI DSS Benchmarking. But the /etc/hosts entry for ipv6 localhost was still on the hosts file.