Skip to end of metadata
Go to start of metadata

In addition to basic network configurations, there are several more complex network configurations that can be used with Netmail Store.

On this page:

Sample Networks

Read through the following sample Netmail Store networks to get a sense of the various ways in which Netmail Store can be configured.

Layer 3 Switching

Layer 3 switching for IP, also known as routing, refers to the Network Layer (layer 3) of the OSI Reference Model. The Layer 3 (L3) switch acts as a gateway that routes packets between subnets. The L3 switch only introduces packets into a subnet to which the packets are addressed, which serves to segregate network communications.

In setting up a Netmail Store cluster, an L3 switch is an effective tool to isolate the intercommunications of the Netmail Store nodes from the rest of the network. This isolation is important so that Netmail Store nodes have predicable network bandwidth for their use and so that their multicast/unicast traffic doesn’t interfere with computers on the rest of the network.

Switching Hardware

When selecting Ethernet switching hardware, consider that many client workstations still come with 100Mbps network interfaces. It may not be cost effective to deploy an excess of Gigabit ports to nodes that cannot make use of them. Additionally, many workstation-class operating systems and applications may be unable to effectively utilize bandwidth beyond that which is provided by a 100Mbps port.

With more sophisticated switching hardware, the network segments can be isolated as different VLANs on the same device. Additionally, some enterprise-class switching hardware includes the functionality of an L3 switch.

In addition to client isolation, an administrator may wish to design the Netmail Store subnet to use redundant switches in order to provide fault-tolerance in the event of a switch failure. Netmail Store nodes support dual network configurations where the node is plugged into one or two switches.

When deploying Netmail Store with multiple switches, the inter-connection between the switches requires planning. In order to allow for full speed communications between active ports on different switches, the network connection between switches must be faster than the individual ports. The can be achieved with a proprietary technology available from the switch vendor or through a mechanism such as link aggregation.

Internet Deployments

Network security is one of the top considerations during the deployment of any service on the Internet or within an extensive enterprise WAN. In these types of deployments, a firewall or filtering router should be placed in front of Netmail Store in order to control the kind of traffic and requests that are allowed to reach the cluster nodes.

By default, the SCSP port is TCP/80, but it should be changed to match the scspport value set in the node or cluster configuration file for the Netmail Store cluster if it is something other than 80.

If the firewall is sophisticated enough to examine Layer 7 (Application Layer), or the contents of the HTTP requests, further restrictions should be made to allow only GET, HEAD, POST, DELETE requests. If a cluster is exposed read-only to these external clients, the POST and DELETE requests can be blocked to prevent updates to the cluster. In order to prevent client access to the node status page, the firewall should deny “GET /” requests to the cluster nodes.

Administrators should block Internet access to the web console port (default TCP/90) and to the SNMP port (UDP/161). In wide-area networks, further restrictions may be desirable to restrict access to these services to specific administrative networks and/or workstations.

Anytime critical devices such as firewalls are introduced into a network architecture, they should be deployed in redundant pairs in order to minimize the chance of failures that cut-off all client access.

Network Booting

Netmail Store supports network booting of the storage cluster nodes through Intel’s Preboot Execution Environment (PXE) specification. This is commonly called “network booting” and is supported by most modern network adapters. Netmail Store also supports the centralizing of the node configuration files on an HTTP or FTP server in order to simplify the management of large clusters. PXE booting requires an administrator to configure a DHCP server and a TFTP server in order to support network booting. An optional HTTP or FTP server is required in order to allow for a centralized location for the Netmail Store node configurations.

Warning: Netmail Store may erase all non-Netmail Store data on hosts that are accidentally booted from the network. Extra care must be taken to ensure that the DHCP server only provides Netmail Store booting instructions to the correct network hosts.

PXE Boot Setup

The following steps are necessary to setup network booting.

To setup network booting:

  1. Configure DHCP server with next-server and filename parameters.

  2. Configure TFTP server with PXE bootstrap, configuration, and Netmail Store files.

  3. Set up the nodes’ BIOS configurations for network booting.

The following example shows the configuration lines from the ISC DHCP server that is commonly available on Unix systems. It shows the use of the next-server parameter to define the IP address of the TFTP server and the filename parameter to define the bootstrap loader program that will be downloaded.

 group {
  next-server 172.16.1.10;
  filename "/pxelinux.0";
  # Hosts allowed to network boot into Netmail Store
  host clusternode1 { hardware ethernet 00:90:cb:bf:45:26; }
  host clusternode2 { hardware ethernet 00:90:b2:92:09:e4; }
  host clusternode3 { hardware ethernet 00:90:0d:46:7a:b4; }
 }

In the previous DHCP configuration file example, the Netmail Store nodes are explicitly defined by the network MAC address in order to prevent the unintended booting of Netmail Store by other servers or workstations.

In addition to the DHCP, an administrator will need to configure a TFTP server from which the Netmail Store software will be loaded.

To set up a TFTP boot server:

  1. Install and configure TFTP server software on the boot server.

  2. Create the /tftpboot directory hierarchy.

  3. Copy kernel and fsimage files to /tftpboot/profiles/storagecluster directory.

There are many free and commercially available packages that provide a TFTP server and Unix distributions commonly include this in their standard setup. The tftp-hpa package for Unix is known to work with Netmail Store. It is available in source code from http://www.kernel.org/pub/software/network/tftp/ and as a binary package in many Linux distributions.

After installing the TFTP server, it is necessary to configure it to serve the directory where the network boot files are located. This directory is traditionally called /tftpboot because TFTP is almost exclusively used for booting network devices. A sample template of this directory is included in the Netmail Store software distribution in the samples/Network-Boot directory.

The Netmail Store software distribution media contains two files named kernel and fsimage that must be copied into the tftpboot/profiles/castor directory on the TFTP server. These files contain the Netmail Store embedded operating system and will be loaded at boot time by every Netmail Store storage node.

After copying the directory template and the Netmail Store software files, the tftpboot directory on the TFTP server should contain the following files.

File name

Description

tftpboot/pxelinux.0

Boot loader program

tftpboot/profiles/castor/fsimage

CAStor software

tftpboot/profiles/castor/kernel

CAStor OS kernel

tftpboot/pxelinux.cfg/default

PXELINUX configuration file

For additional information regarding the use of the PXELINUX boot loader, please refer to the included documentation and ZIP file within the samples/Network Boot directory on the Netmail Store distribution media.

DHCP and Boot Server Redundancy

If you want to ensure you do not have a single point of failure when your DHCP server goes offline, you will need to configure both primary and secondary DHCP servers. Details on the necessary steps to set up the ISC DHCP daemon for redundancy can be found in the following article: http://www.madboa.com/geek/dhcp-failover/.

In order to provide redundancy at the network booting layer, your primary and secondary DHCP servers can be used as TFTP servers as well. In their setups, make the next-server parameter point to their own IP address. This way, whichever DHCP server answers the DHCP query will also be the server that handles the PXE boot. To prevent any network interruptions, the TFTP boot server(s) should be on the same switch domain as the Netmail Store nodes.

Configuration File Server

Netmail Store supports the centralizing of the node configuration files on an HTTP or FTP server and may be used with network booting or with the standard USB flash drive booting. For large Netmail Store clusters, the centralized configuration server can simplify the management of configuration files by allowing configuration file updates by remote administrators and by providing a method to group configurations of similar nodes.

When using a configuration file server, the Netmail Store node is booted with a kernel configuration parameter that contains a URL that points to a configuration list file. The list file is a plain-text file that contains a list of URLs from which to retrieve all of the configuration files to be used by the Netmail Store node.

Network Boot Loader Configuration

This is an example PXELINUX configuration file from the tftpboot/pxelinux.cfg directory on the TFTP boot server.

 default profiles/castor/kernel
  append initrd=profiles/castor/fsimage
   ramdisk_size=128000 root=/dev/ram0
   castor_cfg=http://172.16.1.200/CAStor/cfg\-list.txt

Note: The append command must be entered on a single line. The line has been split for clarity in this example.

USB Boot Loader Configuration

This is an example section of the syslinux.cfg file that is contained on the USB flash drive.

 label normal
  kernel kernel
  append initrd=fsimage ramdisk_size=128000 root=/dev/ram0
   castor_cfg=http://172.16.1.200/CAStor/cfg\-list.txt

Configuration List File

The configuration list file is pointed to by the kernel castor_cfg parameter and is a list URLs for all the configuration files that are to be loaded by a Netmail Store node. The Netmail Store configuration files are evaluated top-down and in the order that they are listed in the configuration list file. Netmail Store configuration parameters may be defined multiple times, but the last definition is the only one that will be used.

By allowing redefinitions of parameters, an administrator can layer configuration files so that they contain generally applicable values for a cluster, a group of similar nodes, and values specific to one node.

This sample configuration list file shows this how this layer is done.

 http://172.16.1.200/CAStor/cluster.cfg
 http://172.16.1.200/CAStor/subcluster.cfg
 http://172.16.1.200/CAStor/testnode.cfg

Each of the configuration files listed in the list file uses the same format as the Netmail Store node.cfg file. For an explanation of this format and the parameter, see Node Configuration.

Network Devices and Priority

In Netmail Store’s normal operating mode, all Ethernet network devices are encapsulated into a redundant bond interface where one device is active and the others are backups. In this default mode, the first network device is the preferred one. An administrator may wish to override this behavior in the following situations:

  • Network adapter, such as an IPMI card, must be excluded from use
  • Preferred network adapter change for network load management
  • Bonding of multiple adapters for increased throughput

In order to override Netmail Store’s default handling of the network devices, an administrator edits the boot configuration file for a node. If the node is booting from the USB flash drive, this is the syslinux.cfg file. If the node is booting from the network, this is the pxelinux.cfg file. Prior to editing the Netmail Store configuration, an administrator will need to configure the ports on the switch for the appropriate mode; these are normally link-aggregation or 802.3ad. Netmail Store supports modes of active-backup, balance-rr or 802.3ad. Netmail Store supports all Linux bonding driver supported modes.

In the configuration file, a new kernel parameter called castor_net is included in the append clause. The previous castor_nics parameter has been deprecated with the 2010.1 release and replaced by castor_net, which utilizes a deterministic list of adapter names. The castor_net parameter allows both specification of the bonding mode for the adapters as well as an ordered list of the network devices that Netmail Store may use, where the first device is the preferred interface that will be used whenever it is online. The list of network devices is comma separated and must utilize the adapter names assigned by Netmail Store.

Netmail Store assigns device names to adapters based on a sorted list of MAC addresses. Device names begin with eth followed by a number suffix starting at 0 and incrementing. Adding network hardware can change the assignment order. The hardware report (reached by holding the Shift key during the Netmail Store boot sequence and then typing “utility” at the boot prompt) provides a list of the Netmail Store network device names and their corresponding MAC addresses for reference.

These are usage examples. The other portions of the append clause have been abbreviated for clarity. Note the trailing colon after the bonding mode:

 append initrd=... castor_net=active-backup:eth1,eth0
 append initrd=... castor_net=balance-rr:eth0,eth1
 append initrd=... castor_net=802.3ad:eth1,eth0
 append initrd=... castor_net=eth1,eth2
 append initrd=... castor_net=802.3ad:

Proxying the Admin Console

Administrators running Netmail Store on a private, protected network may wish to allow clients on the external network to view the Netmail Store console without also providing them access to the nodes themselves on the private network. This can by accomplished by proxying the console from a privileged server that straddles the internal and external networks. For instance, from a server running Apache, the following rewrite rule for URLs could be applied via the mod_rewrite module:

 <Location /storage/>
 RewriteEngine On
 RewriteRule
 ^.*/storage/([^/]+)/(.*)$
 http://$1:90/$2 [P,L]
 </Location>

Please reference the Apache documentation for mod_rewrite for full details.

IGMP Snooping Support

Managed switches commonly make use of IGMP snooping in order to direct multicast traffic to their ports. By default, Netmail Store uses IGMPv2 responses to host membership queries. The igmpVersion parameter may be used to force the use of version 1, 2, or 3.

  • No labels