Troubleshooting Endpoint Health Checks

This is a Private Preview feature. Cohesity recommends you use this feature only in non-production environments. Contact your Cohesity account team to enable the feature.

To verify whether endpoints are successfully connected to the SaaS Connectors, Cohesity performs a series of health checks on each endpoint listed on the View SaaS Connector Details.

  • If all checks pass, the health status is displayed as Passed in the Endpoint Details table.

  • If any check fails, the health status is displayed as Failed in the Endpoint Details table.

This topic provides information on the health checks performed on each endpoint and the steps to resolve any health check failures. If you are unable to troubleshoot the endpoint health check errors, then Cohesity Support.

Health Checks for Gateway Endpoint

The SaaS Connector performs gateway reachability check on the specified gateway endpoint to ensure connectivity.

Reachability: Gateway Connectivity Check

The SaaS Connector verifies the reachability of the gateway endpoint by pinging the configured gateway IP address from the SaaS Connector using the following command: ping <gateway_IP>

If the health check fails, perform the following steps to troubleshoot:

  1. Verify Gateway IP Configuration:

    1. From the SaaS Connector UI, confirm that the correct gateway IP is being used.

    2. If the gateway IP is incorrect, obtain the correct IP address from your network admin or cloud provider and update it in the SaaS Connector UI.

  2. Validate network range compatibility:

    1. Run one of the following commands on the SaaS Connector to check the SaaS Connector’s IP:

      Ifconfig

      Or

      ip addr show

    2. Ensure the SaaS Connector's IP is in the same subnet as the configured gateway.

      Example, SaaS Connector IP: 192.168.1.50/24

      • If Gateway IP: 192.168.1.1 -The SaaS Connector’s IP is in the same subnet.

      • If Gateway IP: 192.168.2.1 -The SaaS Connector’s IP is in a different subnet.

    3. Restart networking services: sudo systemctl restart networking

Health Check for DNS Endpoint

The SaaS Connector performs DNS resolution check on the specified DNS endpoint to ensure connectivity.

Reachability: DNS Connectivity Check

The SaaS Connector attempts to connect to <DNS_IP>:53 for all DNS servers listed in the SaaS Connector configuration.

If the health check fails, perform the following steps to troubleshoot:

  1. Try resolving a domain (for example, google.com) using the dig command: dig google.com

    The DNS resolution may fail if the following message is displayed:

    server can't be reached
    no servers could be reached

    This error can be fixed by performing the following steps:

    • Ensure port 53 is not blocked by Local firewall, antivirus, software, router, or enterprise network controls.

    • Check the DNS settings: /etc/resolv.conf: cat /etc/resolv.conf.

    • If it doesn't have valid DNS entries, then perform one of the following steps:

      • Run the following command in the Cohesity CLI to change the entry: iris_cli cluster update-dns-servers dns-server-ips=<dns-ip>

      • Update the DNS entries from the SaaS Connector UI.

Some VPNs or proxies may enforce DNS redirection. If these DNS servers are down or blocked, resolution will fail.

Health Check for NTP Endpoint

The SaaS Connector performs DNS resolution and reachability verification on the NTP endpoint specified in the SaaS Connector UI.

Resolution: DNS Resolution Check

The SaaS Connector checks the DNS resolution of the NTP endpoint name.

If the DNS resolution health check fails, refer to the DNS Reachability troubleshooting steps.

Reachability: Network Reachability Check

The SaaS Connector checks network reachability using the command: ntpdate -q <NTP endpoint_name>

If the network reachability health check fails, perform the following steps to troubleshoot:

  1. Confirm NTP Traffic is allowed by ensuring that UDP port 123 is open.

  2. Run the following command to ping the NTP server’s IP address:

    ping <ip address of NTP server>

    You can resolve the NTP server IP address using the following command: nslookup <NTP endpoint_name>

    If the ping fails, it's likely a DNS issue. Refer to the DNS Reachability troubleshooting steps.

  3. Run a Traceroute to identify where the connection fails: traceroute <NTP endpoint_name>

    If the output contains “* * *” at a certain hop, it means that the router is not responding — possibly due to filtering or firewall rules.

    If the traceroute fails to reach the destination, it indicates one of the following:

    • Firewall blocking

    • Routing misconfigurations

    • DNS resolution errors

    In such cases, revisit your network firewall, router settings, and follow the Troubleshoot Reachability checks.

Health Check for Reverse Tunnel (SSH/HTTPS) Endpoint

The SaaS Connector performs TCP connectivity check on the Reverse Tunnel (RT) endpoint to ensure connectivity.

Reachability: TCP Connectivity Check

The SaaS Connector verifies connectivity to the RT endpoint by performing a TCP dial on: <RT endpoint_name>:<port>

If the health check fails, refer to the Troubleshoot Reachability section for detailed steps to diagnose and resolve the issue.

Health Check for Control Plane Endpoint

The SaaS Connector performs DNS resolution check, TCP connectivity check, and API check on the control plane endpoint to ensure connectivity.

Resolution: DNS Resolution Check

The SaaS Connector attempts to resolve the control plane endpoint name.

If the DNS Resolution health check fails, refer to the DNS Reachability troubleshooting steps.

Reachability: TCP Connectivity Check

The SaaS Connector attempts a TCP dial on port 443 of the control plane endpoint: <control_plane endpoint name>:443

If the TCP connectivity check fails, perform the following steps to troubleshoot:

  1. Follow the steps documented in the Troubleshoot TCP Reachability checks section.

  2. Check if the control plane endpoint is resolvable on the SaaS Connector: nslookup <control_plane endpoint_name>

    If the control plane endpoint is not resolvable, then it is likely a DNS issue. Follow the steps in DNS Reachability.

  3. Check if the control plane endpoint is accepting TCP connections: sudo nping --tcp -p 443 <control_plane endpoint_name>

    The check may fail if:

    • Wrong endpoint name is used for network unreachable errors.

    • No packets are received. Check for firewall settings.

    • RST responses are received. Check if 443 is open.

  4. Check for certificates: openssl s_client -connect <control_plane endpoint_name>:443.

    This should return:

    • CONNECTED(00000003)

    • TLS version and cipher suite

    • A full certificate chain

    The check may fail if:

    • Command hangs as server expects input (for example, HTTP request headers)

    • TLS handshake fails due to one of the following:

      • Untrusted certificate authority

      • Certificate hostname mismatch

      • Expired certificate

      • Cipher suite mismatch

      • Missing fields in the certificate (for example, SAN)

      • Bugs in certificate generation logic (on client or server).

API Check

The SaaS Connector performs an internal API call to check if the control plane is reachable.

If the API check fails, perform the following steps to troubleshoot:

  1. Check if the control plane endpoint is resolvable on the SaaS Connector: nslookup <control_plane endpoint_name>

    If the control plane endpoint is not resolvable, then it is likely a DNS issue. Follow the steps in DNS Reachability.

  2. Check if the control plane endpoint is accepting TCP connections: sudo nping --tcp -p 443 <control_plane endpoint_name>

    The check may fail if:

    • Wrong endpoint name is used for network unreachable errors.

    • No packets are received. Check for firewall settings.

    • RST responses are received. Check if 443 is open.

Health Check for Data Plane Endpoint

The SaaS Connector performs DNS resolution check, TCP connectivity check, and API check on the data plane endpoint to ensure connectivity.

Resolution: DNS Resolution Check

The SaaS Connector attempts to resolve the data plane endpoint name.

If the DNS resolution health check fails, refer to the DNS Reachability troubleshooting steps.

Reachability: TCP Connectivity Check

The SaaS Connector attempts a TCP dial on port 443 of the data plane endpoint: <data_plane endpoint_name>:443

If the TCP connectivity check fails, perform the following steps to troubleshoot:

  1. Follow the steps documented in the Troubleshoot TCP Reachability checks section.

  2. Check if the data plane endpoint is resolvable on the SaaS Connector: nslookup <data_plane endpoint_name>

  3. If the data plane endpoint is not resolvable, then it is likely a DNS issue. Follow the steps in DNS Reachability.

  4. Check if the data plane endpoint is accepting TCP connections: sudo nping --tcp -p 443 <data_plane endpoint_name>

    The check may fail if:

    • Wrong endpoint name is used for network unreachable errors.

    • No packets are received. Check for firewall settings.

    • RST responses are received. Check if 443 is open.

  5. Check for certificates: openssl s_client -connect <data_plane endpoint_name>:443.

    This should return:

    • CONNECTED(00000003)

    • TLS version and cipher suite

    • A full certificate chain

    The check may fail if:

    • Command hangs as server expects input (for example, HTTP request headers)

    • TLS handshake fails due to one of the following:

    • Untrusted certificate authority

    • Certificate hostname mismatch

    • Expired certificate

    • Cipher suite mismatch

    • Missing fields in the certificate (for example, SAN)

    • Bugs in certificate generation logic (on client or server).

API Check

The SaaS Connector performs an internal API call to check if the data plane is reachable.

If the API check fails, perform the following steps to troubleshoot:

  1. Check if the data plane endpoint is resolvable on the SaaS Connector: nslookup <data_plane endpoint_name>

    If the data plane endpoint is not resolvable, then it is likely a DNS issue. Follow the steps in DNS Reachability.

  2. Check if the Data plane endpoint is accepting TCP connections: sudo nping --tcp -p 443 <data_plane endpoint_name>

    The check may fail if:

    • Wrong endpoint name is used for network unreachable errors.

    • No packets are received. Check for firewall settings.

    • RST responses are received. Check if 443 is open.

Health Check for Cloud Object Store Endpoint

The SaaS Connector performs both DNS resolution and TCP connectivity checks to validate access to the cloud object store endpoint, which is essential for uploading and retrieving data.

Resolution: DNS Resolution Check

The SaaS Connector attempts to resolve the cloud object store endpoint name

If the DNS resolution health check fails, refer to the DNS Reachability troubleshooting steps.

Reachability: TCP Connectivity Check

The SaaS Connector attempts a TCP dial on port 443 of the cloud object store endpoint: <cloud_object_store endpoint_name>:443

If the TCP connectivity check fails, perform the following steps to troubleshoot:

  1. Follow the steps documented in the Troubleshoot TCP Reachability checks section.

  2. Check if the cloud object store endpoint is resolvable on the SaaS Connector: nslookup <cloud_object_store endpoint_name>

    If the cloud object store endpoint is not resolvable, then it is likely a DNS issue. Follow the steps in DNS Reachability.

  3. Check if the cloud object store endpoint is accepting TCP connections: sudo nping --tcp -p 443 <cloud_object_store endpoint_name>

    The check may fail if:

    • Wrong endpoint name is used for network unreachable errors.

    • No packets are received. Check for firewall settings.

    • RST responses are received. Check if 443 is open.

API Check

The SaaS Connector performs an internal API call to check if the cloud object store endpoint is reachable.

If the API check fails, perform the following steps to troubleshoot:

  1. Check if the cloud object store endpoint is resolvable on the SaaS Connector: nslookup <cloud_object_store endpoint_name>

    If the cloud object store endpoint is not resolvable, then it is likely a DNS issue. Follow the steps in DNS Reachability.

  2. Check if the cloud object store endpoint is accepting TCP connections: sudo nping --tcp -p 443 <cloud_object_store endpoint_name>

    The check may fail if:

    • Wrong endpoint name is used for network unreachable errors.

    • No packets are received. Check for firewall settings.

    • RST responses are received. Check if 443 is open.

Health Check for SaaS Connector Upgrade Package Endpoint

The SaaS Connector performs both DNS resolution and TCP connectivity checks to validate access to the upgrade package endpoint, which is essential for keeping the connectors on the latest version.

Resolution: DNS Resolution Check

The SaaS Connector attempts to resolve the upgrade package endpoint name

If the DNS resolution health check fails, refer to the DNS Reachability troubleshooting steps.

Reachability: TCP Connectivity Check

The SaaS Connector attempts a TCP dial on port 443 of the upgrade package endpoint: <upgrade_package endpoint name>:443

If the TCP connectivity check fails, perform the following steps:

  1. Follow the steps documented in the Troubleshoot TCP Reachability checks section.

  2. Check if the upgrade package endpoint is resolvable on the SaaS Connector: nslookup <upgrade_package endpoint_name>

  3. Check if the upgrade package endpoint is accepting TCP connections: sudo nping --tcp -p 443 <upgrade_package endpoint_name>

    The check may fail if:

    • Wrong endpoint name is used for network unreachable errors.

    • No packets are received. Check for firewall settings.

    • RST responses are received. Check if 443 is open.

Troubleshoot TCP Reachability checks

Perform the following steps to troubleshoot connectivity issues between SaaS Connectors.

  1. Open the /etc/hosts file and look for static mappings that may interfere with DNS resolution.

    If there is an entry for the hostname that is incorrect, then:

    • Fix it by updating the entry with the correct hostname, or

    • Remove it to allow the SaaS Connector to query the DNS server for the correct entry.

  2. Run the following command to check the TCP connectivity: nc -vz <hostname> <port>

  3. Check for the following errors:

    • Validate if the hostname provided is correct.

    • Lookup failed-Indicates DNS resolution error. Refer to the DNS Reachability troubleshooting steps.

    • No route to host-Indicates firewall issue. Refer to the Firewall settings troubleshooting section.

    • Network is unreachable-Indicates the network interface might be down. Check interface status with the following command: ip link show

    • If the interface shows DOWN or NO-CARRIER, it's inactive.

    Resolution:

    • Bring up the interface:

      sudo ip link set <interface-name> up

    • Restart networking services:

      sudo systemctl restart networking

  4. Run the following command: telnet <host> <port>

    If this fails:

    • Ensure the target port is open.

    • Verify firewall rules on both the source and destination SaaS Connectors.

Troubleshoot Firewall Setting Error

Perform the following steps to verify and troubleshoot the firewall settings error:

  1. Run the following command to check if the firewall is active: sudo firewall-cmd --state

    If the output is running, the firewall is enabled and active.

    If the firewall is active, view the current rules: sudo firewall-cmd --list-all

  2. This will display the list of all firewall settings, services, and port/protocol combinations currently allowed. Check if the port/protocol combinations in question are allowed by these settings. If it is not listed, inbound traffic on that port is likely being blocked.

  3. Perform a quick test to check if ports are allowed: sudo firewall-cmd --add-port=443/tcp

  4. After adding the port, test if endpoint connectivity is restored.