PIX Failover Feature

The firewall’s critical role in the network security design makes device failure of any kind a serious consideration. The failover feature allows an identical PIX firewall unit to provide redundancy if the primary unit fails. One unit is considered the “active” or “primary” unit, while the other is considered the “standby” or “secondary” unit. The active unit performs its normal network functions, while the standby unit only monitors the other unit, ready to take control if the active unit fails.

Since PIX OS v5.1, PIX models support stateful failover, allowing the system to maintain connection state information for the TCP connection during the failover from the primary unit to the standby unit. If failover occurs, the secondary unit assumes the IP and MAC addresses of the primary unit and begins accepting traffic. Because the other network devices don’t see any change in these addresses, no ARP entries change or timeouts occur anywhere in the network.

Understanding Failover

Traditionally, the two PIX Firewall units are connected by a special high-speed serial cable when using cable-based failover, although a faster solution involves a dedicated Ethernet connection to a dedicated switch/hub (or VLAN) for LAN-based failover. When using stateful failover, a separate, dedicated 100 Mbps or Gigabit Ethernet connection is required for cable-based failover and is recommended for LAN-based failover.

Once the primary unit is configured and the necessary cabling attached, the primary unit automatically copies the configuration to the standby unit when it’s powered up.

If the failover feature is enabled, the ACT indicator light on the front of the PIX 515e, PIX 525, and PIX 535 is lighted when the unit is the active unit and it’s off when the device is the standby unit.

Figure 22-17 shows a simple failover system without protected DMZ(s). Each firewall connects to an inside and an outside switch, while the failover serial cable connects the two directly together. If a protected DMZ existed, each firewall’s perimeter interface would have to connect to a switch in the DMZ.

Figure 22-17: Two PIX Firewall units forming a simple serial failover pair

Identical Units

The two PIX units hardware must be configured exactly the same to appear as a single unit to the network. Failover requires two units that are identical in the following respects:

Platform (PIX 515 and PIX 515e won’t work together)
Interfaces
Software version
Amount of RAM
Flash memory
Activation key type (DES or 3DES)

Software licensing is an issue when choosing units to create a failover pair. At least one of the failover pair must have an Unrestricted (UR) license. The second unit can have either a Failover (FO) or a UR license. Restricted (R) units can’t be used as any part of a failover pair and two FO licensed units can’t be used to create a failover pair. PIX 501/506/506E units don’t support the failover features.

Note?

Cisco’s pricing strategy for failover units means a substantial financial savings exists when an unrestricted/failover pair is used compared to two unrestricted units. The failover unit can cost one third as much as an unrestricted unit.

Communicating a Failover

The two PIX failover devices can maintain communication and facilitate rapid failover transitions using either of the following:

Special modified RS-232 serial failover cable with transfer rates of 115K
Faster dedicated LAN connection from 10MB half-duplex to Gigabit full-duplex

This connection allows the units to exchange unit identification, and to monitor the power status of the other unit and other failover related communications. Power or cable failure is detected within 15 seconds and triggers a failover switch.

The failover pair uses the failover connection and all network interfaces to exchange special failover “hello” packets every 15 seconds. If two consecutive hello packet cycles are missed, the failover process starts testing the interfaces to determine which unit failed and transfers active control to the secondary unit, if appropriate.

The default 15-second hello cycle can be modified with the failover poll seconds command. The minimum value is 3 seconds and the maximum is 15 seconds. A shorter poll time can allow the PIX Firewall to detect a failure faster and trigger the handoff, but it could be fooled by temporary network congestion.

Failover Serial Cable

The special serial failover cable ends are labeled, and they define the primary and secondary units. If the failover cable connection is presently identifying the unit as primary, the unit becomes the active unit and the configuration is copied to the standby unit. If a PIX unit comes up without a failover cable, then it automatically becomes the active unit.

The serial failover cable allows each unit to detect if the cable is connected at both ends, connected locally but disconnected at the other end, or disconnected locally and the other end is unknown. In addition, the cable can tell if the power is interrupted at the other end. A failure of any of these parameters on the active unit causes the standby unit to trigger a failover.

Because both units will have identical IP and MAC addresses, if both units are powered down, it’s critical that the failover cable be in place when power is restored. If not, both units will come up active and create duplicate address problems.

Configuration Replication

The two PIX Firewall units should be exactly the same and running the same software release. Unless stateful failover is configured, only the primary unit is configured by the administrator. That configuration is replicated over the failover cable from the active unit to the standby unit in three ways:

After the standby unit boots up, the active unit replicates its configuration via the failover cable to the standby unit.
Commands entered on the active unit are automatically replicated via the failover cable to the standby unit.
The write standby command on the active unit sends the entire configuration in memory via the failover cable to the standby unit.

Configuration replication only occurs from Flash memory to Flash memory so, after making configuration changes, use the write memory command to write the configuration into Flash memory. Because the failover cable is a serial link, the replication can take a while with a large configuration.

When the Primary Fails

If a primary unit failure occurs, Syslog messages are sent indicating the cause of the failure, and then the switchover occurs. The standby unit assumes the IP and MAC addresses of its immediate predecessor and starts accepting traffic. After the primary unit is fixed and placed back online, it can’t automatically resume as the active unit because of the duplicate addresses, so it comes up as the standby unit.

A switchover can be manually initiated from either unit. The failover active command on the primary unit or the no failover active command on the secondary unit triggers the change. When a failover occurs and both devices are operational, each will assume the IP address and MAC address of its immediate predecessor. The new active unit will start accepting traffic.

Stateful Failover

Since PIX OS v5.1, stateful failover allows per-connection state table information to be continuously sent to the standby unit. If a failover occurs, both devices have same connection state information allowing end user sessions to be transferred without interruption. With systems not using stateful failover links, the standby unit does not have the state information requiring all active connections to be dropped until they can be reestablished.

Stateful failover can be triggered by any of the following situations:

The active PIX Firewall loses power or is turned off
The stateful failover dedicated link goes down for two “hello” cycles as defined by the failover poll command (default 30 seconds)
The failover active command is used on the standby unit
The no failover active command is used on the active unit
The active PIX Firewall is rebooted, including a reload command
Block memory exhaustion for 15 consecutive seconds or more on the active unit

After a stateful failover, the standby unit will assume the active unit configuration, TCP connection table, including the timeout information of each connection, xlate table, and system up time. What won’t be assumed by the new active unit are the user authentication (uauth) table, the ISAKMP and IPSec SA table, the ARP table, and the routing information.

Stateful Failover Hardware Requirements

Stateful failover requires a dedicated 100 Mbps or Gigabit Ethernet link between the units with a MTU set to 1500 to be used exclusively for passing state information between the two PIX Firewall units. No hosts or routers should be connected to this link. The interface implementations that can be used for this dedicated stateful failover link include the following:

Cat 5 crossover cable directly connecting the two units
100BaseTX half-duplex switch using straight Cat 5 cables
Full-duplex 100BaseTX using a dedicated switch or a dedicated VLAN on a switch
Full-duplex 1000BaseTX using a dedicated switch or a dedicated VLAN on a switch

The failover serial cable must be installed and working properly.

Figure 22-18 shows the same simple design as earlier with a stateful failover link installed using crossover cable.

Figure 22-18: Failover system with stateful failover cable installed

Stateful failover is a new feature, so requirements and configuration commands are in transition. Be sure to check the correct documentation for the Firewall OS version. Early versions didn’t support the crossover cable and v6.1 didn’t support half-duplex failover links.

Failover Configuration with Failover Cable

Before configuring, make certain the two PIX Firewall units are identical, as discussed earlier, and the standby unit is powered off. The steps to configure failover with a failover cable are as follows:

Set the clock on the active PIX Firewall unit using the clock set time command or using the Network Time Protocol (NTP) commands introduced in Chapter 18 for version 6.2 and newer.
Connect the failover serial cable to the units. Make sure the end labeled “Primary” attaches to the primary unit and the end labeled “Secondary” connects to the secondary unit. Don’t power up the secondary unit.
If stateful failover is planned, attach a crossover cable between the primary and secondary units for the network interfaces.
Go to Configuration mode with the configure terminal command.
Always specify the speed for the interface, such as 10baset for 10 Mbps or 100basetx for 100 Mbps. Don’t use the auto or the 1000auto option on any interface. Verify that the interface speed and duplex settings match any connected devices. Use the write terminal command to confirm the settings. For stateful failover, set the dedicated interface speed, using either the 100full or the 1000sxfull command. Set the link maximum transfer unit by using the mtu interface_name 1500 command. For PIX Firewall version 6.2, the MTU size must be at least 1,500 for the stateful failover link and at least 576 for the LAN-based failover link.
Use the clear xlate command after changing the interface command.
Use the ip address command to assign IP addresses to each interface. The following output shows examples of the commands so far.
Use the failover command statement to enable the failover feature. The no failover command will disable the failover feature.

The related failover active command on the standby unit triggers a failover switch, causing that unit to become the active unit. The no failover active command from the active unit triggers a failover switch to make the standby unit become the active unit. This command is used to force an active unit offline for maintenance and to return a updated unit to service.
```
Pix# clock set 14:27:0 jun 1 2004
Pix# config t
Pix(config)# nameif ethernet2 intf2 sec50
Pix(config)# ip address outside 10.1.1.1 255.255.255.0
Pix(config)# ip address inside 192.168.1.1 255.255.255.0
Pix(config)# ip address intf2 192.168.2.1 255.255.255.252
Pix(config)# interface e0 100full
Pix(config)# interface e1 100full
Pix(config)# interface e2 100full
Pix(config)# mtu intf2 1500
Pix(config)# clear xlate
Pix(config)# failover
```

Use the show ip address command to see the addresses. The Current IP Addresses is the same as the System IP Addresses on the failover active unit.

Pix(config)# show ip address
System IP Addresses:
 ????ip address outside 10.1.1.1 255.255.255.0
 ????ip address inside 192.168.1.1 255.255.255.0
 ????ip address intf2 192.168.2.1 255.255.255.252
Current IP Addresses:
 ????ip address outside 10.1.1.1 255.255.255.0
 ????ip address inside 192.168.1.1 255.255.255.0
 ????ip address intf2 192.168.2.1 255.255.255.252

Use the show failover command to verify the failover feature by looking for the This host: primary - Active statement. You can see failover is on and the other unit isn’t powered up.

Pix(config)# show failover
Failover On
Cable status: Other side powered off
Reconnect timeout 0:00:00
Poll frequency 15 seconds
 ???This host: primary - Active
 ???????????????Active time: 330 (sec)
 ???????????????Interface intf2 (192.168.2.1): Normal (Waiting)
 ???????????????Interface outside (10.1.1.1): Normal (Waiting)
 ???????????????Interface inside (192.168.1.1): Normal (Waiting)
 ???Other host: secondary - Standby
 ???????????????Active time: 0 (sec)
 ???????????????Interface intf2 (0.0.0.0): Unknown (Waiting)
 ???????????????Interface outside (0.0.0.0): Unknown (Waiting)
 ???????????????Interface inside (0.0.0.0): Unknown (Waiting)

Interface flag	Indicates
Failed	Interface has failed
Link Down	Interface line protocol is down
Normal	Interface is working correctly
Shut Down	Interface has been administratively shut down
Unknown	IP address isn’t configured for the interface, so it can’t determine the status
Waiting	Monitoring the other unit’s network interface hasn’t started yet

Use the failover IP address int_name ip_addr command to define the standby unit’s interface addresses. The IP addresses for the standby unit are different from the active unit’s addresses, but in the same subnet for each interface. The standby unit needn’t be powered up for this command to work correctly.

Without setting, the failover IP addresses failover won’t work, the show failover command will display 0.0.0.0 for the IP address, and monitoring of the interfaces will remain in the “waiting” state.

Pix(config)# failover ip address inside 192.168.1.2
Pix(config)# failover ip address outside 10.1.1.2
Pix(config)# failover ip address intf2 192.168.2.2

Pix(config)# show failover
Failover On
Cable status: Other side powered off
Reconnect timeout 0:00:00
Poll frequency 15 seconds
 ???????This host: primary - Active
 ???????????????Active time: 740 (sec)
 ???????????????Interface intf2 (192.168.2.1): Normal (Waiting)
 ???????????????Interface outside (10.1.1.1): Normal (Waiting)
 ???????????????Interface inside (192.168.1.1): Normal (Waiting)
 ???Other host: secondary - Standby
 ???????????????Active time: 0 (sec)
 ???????????????Interface intf2 (192.168.2.2): Unknown (Waiting)
 ???????????????Interface outside (10.1.1.2): Unknown (Waiting)
 ???????????????Interface inside (192.168.1.2): Unknown (Waiting)

Use the failover link [stateful_if_name] command to enable stateful failover. Use the no failover link command to disable the feature.

Pix(config)# failover link intf2
Pix(config)# show failover
Failover On
Cable status: Other side powered off
Reconnect timeout 0:00:00
Poll frequency 15 seconds
 ???????This host: primary - Active
 ???????????????Active time: 740 (sec)
 ???????????????Interface intf2 (192.168.2.1): Normal (Waiting)
 ???????????????Interface outside (10.1.1.1): Normal (Waiting)
 ???????????????Interface inside (192.168.1.1): Normal (Waiting)
 ???Other host: secondary - Standby
 ???????????????Active time: 0 (sec)
 ???????????????Interface intf2 (192.168.2.2): Unknown (Waiting)
 ???????????????Interface outside (10.1.1.2): Unknown (Waiting)
 ???????????????Interface inside (192.168.1.2): Unknown (Waiting)

13. If necessary, use the failover poll seconds command to set a hello interval shorter than 15 seconds (range 3 to 15).
Power up the secondary unit. The primary unit will detect it and start synchronizing the configurations. The messages “Sync Started” and “Sync Completed” will appear.
If any other changes are made to the active unit configuration, use the write memory command to save the configuration and to synchronize the standby unit.

LAN-Based Failover Configuration

PIX Firewall version 6.2 introduces support for LAN-based failover, eliminating the need for the Failover serial cable to connect the primary and secondary units. LAN-based failover overcomes the six-foot distance limitations of the Failover cable.

A dedicated LAN interface and a dedicated switch/hub (or VLAN) is required to implement LAN-based failover. An Ethernet crossover cable can’t be used to connect the two PIX Firewalls.

Because failover messages might be transmitted over Ethernet connections that are relatively less secure than the dedicated Failover serial cable, PIX Firewall version 6.2 provides message encryption and authentication using a manual preshared key.

The four command mode failover lan commands include the following syntax. Each uses the no form of the command to remove the feature.

Pix(config)# failover lan unit {primary | secondary}
Pix(config)# failover lan interface if_name Pix(config)# failover lan key key_secret
Pix(config)# failover lan enable

enable	Enables LAN-based failover; otherwise, serial cable failover is used
key	Enables encryption and authentication of LAN-based failover messages
key_secret	The shared secret key for encryption
primary \| secondary	Specifies the unit to be primary or secondary PIX Firewall to use for LAN-based failover. Equivalent of serial cable labels

The basic configuration of the active firewall doesn’t change and won’t be restated here. The LAN-based failover does require some configuration on the standby unit and those commands are addressed here. If properly configured, the LAN-based failover configurations for the two units will be different, reflecting which is primary and which is secondary. To configure LAN-based failover, follow these steps:

Don’t connect the failover LAN interfaces until told to do so.
Configure the primary PIX Firewall unit as previously discussed.
Still on the primary unit, connect the LAN failover interface to the network and add the following lines to configure the LAN-based failover. Lanlink is the interface (Ethernet4) used for the failover connection, while 1234567 is the key used for encrypting traffic over the LAN failover link.
```
Pix(config)# no failover 
Pix(config)# failover lan unit primary 
Pix(config)# failover lan interface lanlink
Pix(config)# failover lan key 1234567 
Pix(config)# failover lan enable 
Pix(config)# failover 
```
Use the write memory command to save the primary unit configuration to Flash.

Power on the secondary unit without the LAN-based failover interface connected. The following commands are necessary for the secondary unit to connect to the primary unit using the LAN-based failover interface. Once this connection is established, the rest of the primary unit configuration is replicated over the failover connection.

Pix2(config)# nameif ethernet4 lanlink security20 
Pix2(config)# interface ethernet4 100full 
Pix2(config)# ip address lanlink 192.168.3.1 255.255.255.0 
Pix2(config)# failover ip address lanlink 192.168.3.2 
Pix2(config)# failover lan unit secondary ???????????????????(optional) 
Pix2(config)# failover lan interface lanlink
Pix2(config)# failover lan key 1234567 
Pix2(config)# failover lan enable 
Pix2(config)# failover

Use the write memory command to save the secondary unit configuration to Flash.
Reboot both units and connect the LAN-based failover interfaces to the designated failover switch, hub, or VLAN.
If any of the failover lan command needs to be changed, you need to disconnect the LAN-based failover interface and repeat the preceding steps.

Verifying Failover Configuration

Use the following commands to verify that failover configuration is correct.

show failover Use the show failover command to verify the status of the connection and to determine which unit is active. You saw sample output earlier in this section.

show failover [lan [detail]] The show failover lan command displays the LAN-based failover information useful for debugging purposes.

pix(config)# show failover lan 
Lan Based Failover is Active
 ???????interface lanlink (192.168.3.1): Normal, peer (192.168.3.2): Normal

The show failover lan detail command displays the connection details, as well as traffic summary information.

pix(config)# show failover lan detail
Lan Failover is Active
This Pix is Primary
Command Interface is lanlink
Peer Command Interface IP is 192.168.3.2
My interface status is 0x1
Peer interface status is 0x1
Peer interface downtime is 0x0
Total msg send: 103093, rcvd: 103031, droped: 0, retrans: 13, send_err: 0
Total/Cur/Max of 51486:0:5 msgs on retransQ
msgs on retransQ if any
LAN FO cmd queue, count: 0, head: 0x0, tail: 0x0
Failover config state is 0x5c
Failover config poll cnt is 0
Failover pending tx msg cnt is 0
Failover Fmsg cnt is 0