System troubleshooting Riverstone RS routers
From ipinfinity.com
In checking the system health of a Riverstone Networks RS router, there are several categories of statistics that are very useful. In addition there are common symptoms that can be observed to determine the changing state of the RS system. This document will begin by discussing the symptoms of a changing systems health. During this discussion various statistics will be reviewed to determine the cause of the symptoms. Some of the symptoms that can be isolated and monitored on the RS switch-routers are symptoms that affect the basic resources of the RS. This section only describes some of these resources, but also how to monitor them via CLI commands. Please, keep in mind that resource utilization is dependent on the features enabled on the RS, the traffic patterns into the RS, and other environmental influences. Some of these resources, such as CPU, memory, L2 and L3 tables, are covered in detail below.
Contents |
Syslog Services
The RS platform supports a full implementation of a Syslog client for the centralized reporting of system events in four levels of severity. Additionally the RS platform supports eleven facility instances.
At least one Syslog server should be used per administrative domain (in this instance we are defining an administrative domain as a single OSPF area). A preferable configuration and one that Riverstone recommends is the use of redundant Syslog servers. This recommended configuration offers a primary and redundant Syslog server per administrative domain or a redundant Syslog server configured within a bordering administrative domain.
A logical aggregation scheme is recommended for Syslog messaging. This will ensure that Syslog files are presented in a manner that makes sense and aids the troubleshooting process.
Basic Syslog Configuration
The configuration shown below illustrates a rudimentary Syslog configuration. This configuration provides for a Syslog server located at 10.100.100.100 that it will communicate to at local7. The internal Syslog will have a buffer size of 100 messages kept in local memory as well as keeping a local copy of Syslog messages within a circular buffer on the internal flash card.
RS# system set syslog server 10.100.100.100 local-copy RS# system set syslog facility local7 RS# system set syslog buffer-size 100
Core Files
When the RS platform experiences a crash the core files are sent to two places – if the console port is connected to a terminal then the crash will be output on this screen – the core file is also written to the PCMCIA card installed in the active control module.
Deriving Additional Information From Core Events
In RoS versions 9.n and later an extended core file format was introduced. This was included to derive additional information in the event that a crash condition occurred and engineering required more information than that otherwise included in the standard crash file. This of course increases the time to recover should a crash be experienced since additional registers are polled and statistics taken during the assert process.
In other circumstances it may be desirable to have a smaller core file that contains less information and provides for quicker recovery times (This can be useful in circumstances where a known issue is causing a CM crash and fail over). In this instance the following configuration command should be used:
R1(config)# system set extended-debug enable-min-crash-dump
Retrieving a Core File
On the RS platform core files are held in the flash. The following illustrates the copying of a core file from the RS platform to a TFTP server:
RS31-CC-LAB# copy slot0:core to tftp-server TFTP server? 172.21.1.57 Destination filename? core040605.txt ############################################################################################## %TFTP-I-XFERRATE, Sent 67956 bytes in 1.0 seconds
CPU Utilization
The CPU is the “heart” of the switch-router. Different from software-based routers, most of the CPU work is to process and forward the initial packet between two parties and install its flow directly in hardware for subsequent traffic that matches the initial pattern. As was mentioned above, the CPU utilization will, most likely, depend on the features enabled on the switch-router and the traffic patterns on the network. Therefore the baseline for the CPU utilization will vary for different network environments. The best way to determine what is the normal (baseline) CPU utilization for any RS, is to monitor its utilization on a periodic basis. Depending on the features enabled, changing the default values, in some of the features, may be necessary to better accommodate the specific mix of functionality and allow the RS to perform its functions without compromising performance or stability. In general the RS CPU utilization should not consistently run above 65%. This will allow the necessary resources to react to short term changes on the network. The command used to check the CPU utilization, is the enable mode command:
rs# system show cpu-utilization CPU Utilization (5 seconds): 3% (60 seconds): 3%
If it is noticed the RSs running above its normal CPU utilization, the following command will indicate the tasks requiring the CPU cycles.
rs# statistics show top
Some of the tasks that might be seen as the top users of the CPU can provide a hint to the source of the activity requiring the excessive use of the CPU. The information provided by the “statistics show top” command is relative to the last time this command was executed.
Each control module has a certain amount of DRAM memory. It could vary from 32MB in the early days, up to 512MB. Some of this memory is used to store the operating system and it’s associated data structures, vlan-port mapping, routing tables and RMON data. The following enable mode command can be used to determine the memory configuration and its utilization, not only for the CPU but also the amount of memory for each linecard:
rs# system show capacity memory Capacity MIB Storage Information: Type Description Total Bytes Bytes Free Bytes Used % Uti Remov Fail ------- -------------- ----------- ---------- ---------- ----- ----- ---- CPU Internal CPU 201326592 144490496 56836096 28.23 False 0 FLASH Internal Flash 387072 188416 198656 51.32 False 0 L2HW port et.2.1 376832 376704 128 0.03 False 0 L2HW port et.2.2 376832 376704 128 0.03 False 0 L2HW port et.2.3 376832 376768 64 0.02 False 0 … L2HW port t1.5.1 901120 901120 0 0.00 False 0 L2HW port t1.5.2 901120 901120 0 0.00 False 0 L2HW port NP.5.3 901120 901120 0 0.00 False 0 L2HW port NP.5.4 901120 901120 0 0.00 False 0 L3HW module 2 15531776 15531776 0 0.00 False 0 L3HW module 5 15531776 15531776 0 0.00 False 0
In order to determine how much memory is needed on each system, it is necessary to review the configuration of the RS to determine what features have been enabled on the RS and what role the RS will playing on the network. A simple example would be having an RS with one EBGP peering session, receiving the whole Internet routing table. Assuming the routing table is 110 thousand routes, the user would need at least 192 MB of memory, since the system and other tasks require memory. Since, the number of routes advertised over the Internet tends to grow, it is necessary to have enough memory to scale, also keeping in mind the user might eventually want to bring up new peering sessions.
There are also another command available to check the amount of memory in the CPU of the RS:
rs# system show hardware
The same command with a few more options can display the information per linecard.
rs# system show hardware slot 2 verbose
Keep in mind that depending on the situation, the user might want to disable some feature in order to free-up some memory for another task. (i.e. disable RMON in order to continue to receive the entire Internet routing table), until physical memory upgrade is performed.
Troubleshooting High CPU Utilization
In this section is detailed some of the causes of high CPU utilization along with some of their potential solutions.
L3 Bucket overflow
This is one of the symptoms that might cause the NI H task to consume more CPU cycles than normal. This usually happens when the RS’s default configurations for layer-3 operation is not the best for the role this RS is playing in the network, causing the RS to attempt to populate a small portion of the L3 table entries on the specified line cards. This information can be checked using the command:
rs# system show capacity memory
There are two areas of the configuration that will help to tune the RS, in order to resolve this type of problem. The first one is the L3 hashing-variant the RS uses this algorithm for more efficient L3-packet look-up.
Because the L3-lookup table is organized as a hash indexed table, changing the hashing variant (lookup algorithm) can possibly change the way the RS programs the table entries to accommodate the packets in a way they don’t clump together in the table.
Remember, this will always depend on the traffic pattern and other environmental influences. There is a configuration command to change this hashing-variant on a per line card basis:
rs(config)# ip l3-hash module <num>|all variant <num>
Please note that setting the variant to 8 (auto-hash), will cause the RS to automatically choose between the 8 different hashing-variants (0-7, 0 being the default). The second operation that could cause high CPU utilization for the NI_H task, is the forwarding-mode the RS is configured to use. The default forwarding-mode the RS uses is to extract the whole L3 and L4 information for each packet, this mode is called Application mode.
Depending on the traffic pattern and location of this RS in the network, there will be a point where the CPU has to force a layer 3 flow to age out, in order to accommodate a brand new flow in hardware. An example of when this happens would be: behind the switch-router sits a very busy web-server, and on the other side is the Internet. This way, the RS will create a different flow for every station on the Internet that connects to this server. Supposing this server happens to be a FTP server as well, maybe for the same station the router will be creating two or three different flows, since it creates a different flow based on the whole L3 and L4 information, as mentioned above. There is a configuration command that allows the user to change this default behavior:
rs(config)# ip set port <port-list> forwarding-mode destination based| host-flow-based
In addition to the default full-flow mode, this operation can be changed to work in destination-based mode (look at the destination ip address when making a forward decision) or host-flow-based mode (look at the source and destination ip addresses only when making a forwarding decision). Taking into consideration the example described earlier, if changing the flow mode to destination-flow-based, the router would create only one flow for all the stations trying to talk to that server.
There is another feature that works almost the same way of changing the forwarding-mode, this feature is called custom-forwarding mode. The biggest difference from the “regular” forwarding-mode is that the user can customize the forwarding mode.
NOTE: The use of ACLs is limited when changing the forwarding-mode of the router.
The easiest way to check if the router is experiencing L3 bucket overflows, is to use the following enable mode command:
rs# system show capacity memory Capacity MIB Storage Information: Type Description Total Bytes Bytes Free Bytes Used % Uti Remov Fail ------- -------------- ----------- ---------- ---------- ----- ----- ---- CPU Internal CPU 201326592 144490496 56836096 28.23 False 0 FLASH Internal Flash 387072 188416 198656 51.32 False 0 L2HW port et.2.1 376832 376704 128 0.03 False 0 L2HW port et.2.2 376832 376704 128 0.03 False 0 L2HW port et.2.3 376832 376768 64 0.02 False 0 … L2HW port t1.5.1 901120 901120 0 0.00 False 0 L2HW port t1.5.2 901120 901120 0 0.00 False 0 L2HW port NP.5.3 901120 901120 0 0.00 False 0 L2HW port NP.5.4 901120 901120 0 0.00 False 0 L3HW module 2 15531776 15531776 0 0.00 False 49682 L3HW module 5 15531776 15531776 0 0.00 False 0
On the above command output, check for the L3HW information. Execute this command a couple of times. If the values in the “Fail” column is increasing, that means L3 Bucket Overflows are being experienced on the module.
L2 Bucket Overflow
The definition for this behavior is pretty close to the L3 bucket overflows. One of the differences is that the hashing collisions occur in layer 2 instead of layer 3 and there are not as many options to change for L2 hash collisions as there are for layer 3. The task that consumes CPU cycles in this case is the L2_LRN_ task. There are two things that can be changed if L2 Bucket Overflows are being experienced: hashing-mode and bridging-mode.
Hashing-mode
There are 4 different L2 hashing modes and also an automatic one (m-auto). To check the hash-mode in use for a specific port, use the following enable command:
rs# port show hash-mode <port-list> | all-ports
If there is a need to change the L2 hash mode, use the following configuration command:
rs(config)# port set hash-mode <mode> <port-list> | all-ports
Bridging-mode
There are two possible configuration options for the bridging-mode; the default is the address-based bridging which will store the VLAN ID and the source MAC in the table. Ports can also be changed to use flow-bridging, which will store the VLAN ID, source and destination MAC addresses in the table.
The enable mode command to check the bridging mode for a specific port is:
rsr# port show bridging-status <port-list> | all-ports
The configuration command to change the bridging-mode is:
rs(config)# port flow-bridging <port-list> | all-ports
Please note that usually, only the hashing mode will be changed in order to resolve an L2 Bucket Overflow. In addition to this, most likely L2 hashing collisions will occur on test beds and not with production network MAC addresses. In order to verify if the router is experiencing L2 bucket overflows, use the following enable mode command:
rs# system show capacity memory Capacity MIB Storage Information: Type Description Total Bytes Bytes Free Bytes Used % Uti Remov Fail ------- -------------- ----------- ---------- ---------- ----- ----- ---- CPU Internal CPU 201326592 144490496 56836096 28.23 False 0 FLASH Internal Flash 387072 188416 198656 51.32 False 0 L2HW port et.2.1 376832 376704 128 0.03 False 9302 L2HW port et.2.2 376832 376704 128 0.03 False 0 L2HW port et.2.3 376832 376768 64 0.02 False 0 … L2HW port t1.5.1 901120 901120 0 0.00 False 0 L2HW port t1.5.2 901120 901120 0 0.00 False 0 L2HW port NP.5.3 901120 901120 0 0.00 False 0 L2HW port NP.5.4 901120 901120 0 0.00 False 0 L3HW module 2 15531776 15531776 0 0.00 False 0 L3HW module 5 15531776 15531776 0 0.00 False 0
On the above command output, check for the L2HW information. Execute this command a couple of times If the values in the “Fail” column is increasing, that means L2 Bucket Overflows are being experienced on that port.
Environmental health status
There is one enable mode command that allows the user to check some of the environmental related hardware information:
rs# system show environmental-info Environmental Status Temperature: normal Fan Tray: normal Power Supply 1: normal 2: failed
Note that on the RS 38000, the same command displays more complete information when the command above is executed:
rs# system show environmental-info Environmental Status Slot Module Name Status Temp Hi-Trig Lo-Trig Overall System Normal 1 24-10/100-TX normal 25.5'C 80.0'C 5.0'C 14 4-CT3 normal 15.5'C 80.0'C 5.0'C Clock card normal 20.5'C 80.0'C 5.0'C Control Module-0 normal 18.5'C 80.0'C 5.0'C Fabric Card-2 normal 10.5'C Power Supply PS1 normal(on) Power Supply PS2 normal(off) Power Supply PS3 (empty) Power Supply PS4 (empty) Fan Tray normal
Keep in mind that most of this type of information can also be gathered by a Network Management Station via SNMP polling and/or SNMP traps sent by the RS. In addition to that, it is recommended that SNMP should be also used to constantly monitor some of the system resources, for example: memory and CPU utilization.
