To view the status of a node, click its IP address on the left side of the Admin Console. If the cluster is configured to use subclusters, expand a subcluster node name to display IP addresses and then click an IP address to display the node information.To search for a particular node, see.
The top row of the Node Status page provides summary information about the node and its associated volumes, such as uptime and storage usage statistics.
- Streams: Counts the total number of managed data components (such as replicas and segments), not logical objects (such as video files).
- Trapped: Calculates the space that is pending reclamation by the Netmail Store defragmentation process. This process is controlled by several Netmail Store parameters
The node status page automatically refreshes every 30 seconds.
Shutting Down or Restarting a Node
To shut down or restart a node, click Shutdown Node or Restart Node in the Admin Console.
A node that is shutdown or rebooted by an Administrator will appear with a
Maintenance state on other nodes in the cluster.
Note: If you set up a multi-server configuration as described in Appendix A - Implementing a Multi-Server Chassis, shutdown or restart impacts all nodes in the multi-server chassis because all nodes in a multi-server chassis share system resources.
Identifying a Volume
You can identify one or all volumes on a node using the links on the right side of the Admin Console under Restart Node.
The Identify function allows you to select a particular volume and enable the corresponding LED drive light, which can be helpful in identifying a failed or failing drive. Simply select the targeted volume and the amount of time that the light will be enabled.
On the Node Status page, an Identify light displays next to the targeted volume for easy identification.
See Appendix E - Drive Identification API for how to create a plug-in API to enable the drive light. If a hardware-specific API is not used, Netmail Store will revert to a default process to flash the light.
Retiring a Volume
You can retire one or all volumes on a node using the links on the right side of the Netmail Store Admin Console under Restart Node.
On occasion, you may need to replace Netmail Store volumes for regular maintenance or to upgrade the cluster nodes with higher capacity disk drives. If multiple volumes need to be replaced across multiple Netmail Store nodes, the volumes should be retired one at a time. When you initiate a retire, you choose if you would like a minimally disruptive retire that is limited to just the volume(s) being retired, or an accelerated retire that uses all nodes in the cluster to replicate objects on the retiring volume(s) as quickly as possible. Note that the cluster-wide retire may impact performance as it does put additional load on the cluster.
Clicking Retire Node retires all volumes on the node, at the same time. Clicking Retire next to a volume retires only the selected volume. A volume is also retired automatically if a configurable number of errors occur. For more information, see Retiring Volumes.
Before you retire a node or volume, make sure that the cluster:
- Has enough capacity for the objects on the retiring node to replicate elsewhere.
- Has enough unique nodes to replicate the objects with only one replica on any given node.
Note: Retire succeeds only if objects can be replicated elsewhere in the cluster. As a result, the Retire action will not trim a stream until it can guarantee that at least two replicas exist in the cluster or the existing number of replicas matches the
minReplicas parameter value
A retiring node or volume accepts no new or updated objects. Retiring a node or volume means all of its objects, including replicas, are copied to other nodes in the cluster.
On the Admin Console's Node Status page, the Node Operations section includes a Retire Rate that tracks the number of objects per hour that were removed from a retiring volume. The SNMP MIB includes this same value in the
retireRatePerHour MIB entry. If no volumes on the node are retiring, the value is
0. After all objects are copied, the node or volume's state changes to Retired and Netmail Store no longer uses the node or volume. At this point, you should remove and repair the volume or discard it.
Errors and Announcements
The last 10 errors and announcements appear on the Node Status page. If there are no errors or announcements, the page is blank. The error count in the node summary grid corresponds to the list of errors in the error section.
Messages now display in the node status area if you remove or insert a disk drive into a running node. This feature, sometimes referred to as hot plugging (adding a new drive) or hot swapping (replacing a failed drive), enables you to remove failed disk drives for analysis or to add storage capacity to a node at any time.
For example, when you add a volume, the following message appears:
mounted /dev/sdb, volumeID is 561479FB832DCC526B1D7EDCD06B83E1
When you remove a volume, the following message appears:
removed /dev/sdb, volumeID was 561479FB832DCC526B1D7EDCD06B83E1
Note: These messages appear at the announcement level. Additional debug level messages appear in the syslog.
Node Status Reporting
You can troubleshoot node errors and announcements by viewing the following sections on the Node Status page:
- Node Info
- Node Configuration
- Node Operations
- Hardware Status
- Test Network button
You can access these sections at the bottom of the Node Status page. The information in each section can be helpful when working with Support to resolve an issue.
Node Info Section
The Node Info status section contains general information about the hardware installed on the node, as well as time server information and current uptime. Use this status information to determine if your node requires additional hardware resources.
For example, if the Index Utilization and Buffer Utilization values rise to 80% or more, the Netmail Store Admin Console generates an alert that indicates the node may require additional RAM to maintain cluster performance. Additionally, if the Time value does not match the same value in the remaining cluster nodes, the node may not be communicating properly with an NTP server.
Additional Node Info reports
Scroll to the bottom of the Node Info section to access these links to additional reports:
- SNMP Repository (the SNMP repository dump)
- Object Counts (the Python classes in use)
- Uncollectable Garbage
- HTML Templates
- Loggers... (the settings window for changing the logging levels)
- Dmesg dump (the last 1000 messages logged by the Linux kernel reading buffer, for diagnosing a Netmail Store issue when a system panic or error occurs)
- Hwinfo dump (the Linux hardware detection tool output)
Node Configuration Section
The Node Configuration status section contains the cluster and network configuration settings assigned to the node. Use this status information to quickly verify your system configuration without using SNMP commands.
Node Operations Section
The Node Operations status section describes the state of the node. If you encounter a problem in your storage cluster, a Netmail Support representative can use the information in this page to help you determine if the node is communicating effectively with other nodes and resources in the cluster.
For example, some cluster features (such as the Capacity column value in the Netmail Store Admin Console) will not update until the HP cycles are completed separately on each node. The HP Cycle time parameter increases exponentially as the number of streams increase on the node. Additionally, if the SCSP Last read bid and SCSP Last write bid parameters are high, the node may not be servicing new requests.
Hardware Status Section
The Hardware Status section contains status and operational reporting (if available) for various hardware components installed on the node. Use this status information to retrieve node system data, such as the serial number and BIOS version.
Hardware status reporting is dependent on hardware that supports and populates IPMI sensors, SMART status, and, in some cases, manufacturer-specific components such as SAS. Depending on your hardware, not all status fields are populated. The hardware status values are independently scanned and populated for each node, allowing variations in supported utilities on a node-by-node basis.
Additional Hardware Status Reports
Scroll to the bottom of the Hardware Status section to access these links to additional reports:
- Test Network option pings all nodes in the cluster to ensure that all nodes can communicate with each other using TCP/IP and UDP. If one or more nodes do not appear in the display, you may have a network issue in the cluster.
- Test Volumes option pings the volumes on your local hard drives and provides a response time (in milliseconds).
- Dmesg Dump option displays the last 1000 messages logged by the Linux kernel reading buffer. These messages can help you troubleshoot and diagnose a Netmail Store issue when a system panic or error occurs.
- Hwinfo Dump (the Linux hardware detection tool output).
- Send Health Report (script that sends the hardware health report to the configured destination).
Test Network performs two sets of tests:
- First, it sends 100 UDP multicasts to the cluster and computes the results:
- Which nodes responded
- How many responses returned
- How long the responses took, on average
- Next, it fetches the status page (port 80) via TCP for all of those responding nodes (only once for each node). It tracks the total time for each of those round trips.
The data in the Network Test Results window lets you compare the responding nodes with the list of nodes that you expected to see in the cluster. You can also evaluate UDP packet loss and TCP connectivity within the cluster.
Important: If one or more nodes do not appear in the display, you may have a network issue in the cluster.