Replacing a node in a cluster
You can replace a cluster node if the node is in an unhealthy state because of a hardware failure that cannot be repaired, such as a boot disk or a power supply failure.
When you replace the node, data from the faulted node is rebuilt on the new node. During this time, the backup and recovery operations continue to run on the remaining healthy nodes in the cluster.
The network settings and the node name of the faulted node are assigned to the new node. After the NetBackup services fail over to the new node, it can run backup and recovery jobs.
Note:
After a node is deleted from the cluster, its private IP addresses are kept reserved in the cluster and are not used when a new node is added in the cluster.
Review the following guidelines for replacing a node:
You can replace only a single node at a time.
The new node must have the same hardware configuration as the existing cluster nodes.
You cannot perform any other node operations such as adding a new node when the node is being replaced. A node cannot be replaced if any of the previous cluster reconfiguration tasks are in progress or have failed.
All the private and public NICs of the node where the management server is running must be up. If any of the NICs are down, the replace node operation fails.
You cannot replace a node if there is more than one faulty node in a cluster of up to five nodes. A 5-node cluster provides resiliency of one node and a disk. A cluster with six or more nodes provides resiliency of two nodes, or two SSDs, or four HDDs. For a cluster with six or more nodes, if two or more nodes are faulty, contact Veritas Support. You must contact Veritas Support if failures exceed the supported fault tolerance.
If you repair the faulty node and plan to use the same node as a replacement node, Veritas recommends that you reinstall the NetBackup Flex Scale ISO image on the node and perform a factory reset to reset the node to its default factory settings.
If site-based disaster recovery is configured, wait at least two hours after a node failure before attempting to replace the node.
Before you replace the node, ensure that the following requirements are met.
Power on the new node and install the NetBackup Flex Scale ISO image on the node that you want to add to the cluster. Ensure that EEB 4067542 is installed on the node. For more details, see the NetBackup Flex Scale Installation and Configuration Guide.
You must format the disks and ensure that no data is present on the disks; else the replace node operation will fail.
The new node that you want to use to replace the existing node must be connected to the same private and public network as that of the existing cluster. For details about the cabling information, See NetBackup Flex Scale network cabling.
Ensure that the system clock of the new node is in sync with the cluster nodes. If the system clock is not synchronized, the replace node operation fails.
To replace a faulted node from the cluster, complete the following steps:
- Use any one of the following options to log in using the user account that you created when you configured the cluster:
Use a user account with both Appliance Administrator and NetBackup Administrator role, or a user account with only an Appliance administrator role to log in to the NetBackup Flex Scale web interface
https://ManagementServerIPorFQDN/webuiwhere ManagementServerIPorFQDN is the public IP address or the FQDN that you specified for the NetBackup Flex Scale management server and API gateway during the cluster configuration, and then in the left pane click Cluster Monitor > Infrastructure.Use a user account with an Appliance Administrator role to log in to the NetBackup Flex Scale infrastructure management console
https://ManagementServerIPorFQDN:14161where ManagementServerIPorFQDN is the public IP address or the FQDN that you specified for the NetBackup Flex Scale management server and API gateway during the cluster configuration, and then in the left pane click Monitor > Infrastructure.
- Click the Nodes tab.
The list of cluster nodes is displayed with the unhealthy node marked in faulted state.
- If the node that you want to replace is up but in an unhealthy state because of a hardware failure, power off the node. To power off the node, click the Actions menu (vertical ellipsis) from the right side of the row in the UI, and click Shutdown node.
If you are in the NetBackup Flex Scale UI, when prompted, click Open cluster console to open the NetBackup Flex Scale infrastructure management console in a new browser tab. In the NetBackup Flex Scale infrastructure management console, when prompted to confirm, click Shut down node.
- Click Scan for nodes to discover the nodes that are connected to the private network and can be used to replace the unhealthy node.
The nodes in the private network are discovered and displayed under the Discovered nodes section with details such as the node name, serial number, and status.
- For the unhealthy node, click the Actions menu (vertical ellipsis) from the right side of the row in the UI, and click Replace node.
If you are in the NetBackup Flex Scale UI, when prompted, click Open cluster console to open the NetBackup Flex Scale infrastructure management console in a new browser tab. In the NetBackup Flex Scale infrastructure management console, when prompted to confirm, click Replace node.
- In the Replace node dialog box, complete the following steps:
Set the priority to backup and recovery jobs or data rebalancing. Click Overall system performance to set the priority to NetBackup backup and recovery jobs while data rebalancing runs in the background at a lower priority. Click Faster reconfiguration to set the priority to rebalancing data across the cluster. This option increases the speed of node replacement operations so that the new node can begin participating in backup and recovery operations faster but can affect the backup and recovery jobs. The time it takes to rebalance is dependent upon the amount of data being moved between nodes and the priority being set. By default the priority is set to Overall system performance.
Select the node that you want to use to replace the unhealthy node.
Click Replace node.
- To monitor the status of each of the tasks and the progress of the replace node operation click View details on the pop-up window that is displayed on the Infrastructure page. The ongoing and completed tasks for the replace node operation are also displayed in Recent activity or you can go to the Dashboard to check the health of all the nodes.
Before adding the selected node to the cluster, the replace node operation synchronizes the patches or add-ons that are installed on the cluster with the node. However if the version on the selected node is later than the cluster nodes, the replace node operation cannot proceed. If AutoSupport and Call Home settings are configured for the cluster, these settings are synchronized with the node.
After the unhealthy node is replaced, you can view the new node and its details on the Nodes tab. The STIG status of cluster is synchronized with this new node. If the STIG option is enabled for the cluster, the STIG option is also enabled for the node and vice versa. The FIPS option is enabled for the node as this option is enabled with the default factory settings for the Veritas Operating System (VxOS) and for NetBackup MSDP when you create a NetBackup Flex Scale cluster.