Saturday, February 25, 2006
Effective use of IOS show command output.
One thing that I have seen mentioned frequently in reading blogs and in talking to people good at debugging network problems is that it is usually a mistake while debugging to keep issuing show running to use the running configuration to try and debug the problem. As I get a little better at troubleshooting with practice, I am beginning to understand better that point of view.
For one thing, the running configuration is essentially static data and getting into the habit of needing to see it again and again is a little pointless. I find myself sometimes issuing show run repeatedly thinking I must have missed something the last time I looked at it (ten seconds earlier!) Also, with configurations of any complexity or with routers that have a large number of interfaces, the configurations are so big that you can't keep all that information in your head as you scroll through page after page of configuration data. It sometimes does help to use grep-like functionality in IOS (with begin and include) to see relevant config quickly but that does not mean one needs to see it multiple times just to be sure!
What is better is to become adept at using output from show commands for debugging. Judicious use of debug commands is also helpful but it is important to be very careful because output from debug commands can itself become the problem as data overruns the console and one can't even type the command to turn off the debug through that noise. Many is the time when I have had to reboot routers just to recover from a debug command turned on without too much thought. The show commands, on the other hand, are always safe. Also, the key to the superiority of show commands to other techniques is the fact that they usually reflect real time status of whatever you are monitoring to debug your problem. What is important is to be able to use that output effectively to spot causes of problems etc. Data explicitly displayed in show command output can be used to deduce information about the system that perhaps even the Cisco engineer who implemented the command did not think of.
As an example, let's say you are scanning output from show ip ospf neighbor on router R1 and see the following line
What is obvious is that R1 has a neighbor with router-id 2.2.2.2 and link address 20.20.20.2 which is a DROTHER. However, you can also immediately deduce that R1 must be a DR or a BDR because it is in FULL state with neighbor 2.2.2.2. If it were not the DR or BDR, it would only be in 2WAY state with 2.2.2.2.
Another example is with the output of show ip bgp neighbor where you can see the local and remote ports to see which is 179. That information can be used to deduce which end initiated the BGP neighbor connection over TCP. Given the right problem, this data could be a clue to some underlying issue.
Another example with OSPF might be if you see the following in the output of show ip ospf database router
along with the information that there is a loopback interface with the address 80.80.80.80. The above output can then be used to deduce that the loopback interface must have ip ospf network point-to-point configured under it because the OSPF database is indicating a mask other than 255.255.255.255 for this stub link which is the default for loopback interfaces.
Such facility with seeing beyond the literal output from show commands and reading between the lines seems to be a useful skill to develop to get better at debugging.
For one thing, the running configuration is essentially static data and getting into the habit of needing to see it again and again is a little pointless. I find myself sometimes issuing show run repeatedly thinking I must have missed something the last time I looked at it (ten seconds earlier!) Also, with configurations of any complexity or with routers that have a large number of interfaces, the configurations are so big that you can't keep all that information in your head as you scroll through page after page of configuration data. It sometimes does help to use grep-like functionality in IOS (with begin and include) to see relevant config quickly but that does not mean one needs to see it multiple times just to be sure!
What is better is to become adept at using output from show commands for debugging. Judicious use of debug commands is also helpful but it is important to be very careful because output from debug commands can itself become the problem as data overruns the console and one can't even type the command to turn off the debug through that noise. Many is the time when I have had to reboot routers just to recover from a debug command turned on without too much thought. The show commands, on the other hand, are always safe. Also, the key to the superiority of show commands to other techniques is the fact that they usually reflect real time status of whatever you are monitoring to debug your problem. What is important is to be able to use that output effectively to spot causes of problems etc. Data explicitly displayed in show command output can be used to deduce information about the system that perhaps even the Cisco engineer who implemented the command did not think of.
As an example, let's say you are scanning output from show ip ospf neighbor on router R1 and see the following line
2.2.2.2 1 FULL/DROTHER 00:00:32 20.20.20.2 Ethernet0
What is obvious is that R1 has a neighbor with router-id 2.2.2.2 and link address 20.20.20.2 which is a DROTHER. However, you can also immediately deduce that R1 must be a DR or a BDR because it is in FULL state with neighbor 2.2.2.2. If it were not the DR or BDR, it would only be in 2WAY state with 2.2.2.2.
Another example is with the output of show ip bgp neighbor where you can see the local and remote ports to see which is 179. That information can be used to deduce which end initiated the BGP neighbor connection over TCP. Given the right problem, this data could be a clue to some underlying issue.
Another example with OSPF might be if you see the following in the output of show ip ospf database router
Link connected to: a Stub Network
(Link ID) Network/subnet number: 80.80.80.0
(Link Data) Network Mask: 255.255.255.0
Number of TOS metrics: 0
TOS 0 Metrics: 1
along with the information that there is a loopback interface with the address 80.80.80.80. The above output can then be used to deduce that the loopback interface must have ip ospf network point-to-point configured under it because the OSPF database is indicating a mask other than 255.255.255.255 for this stub link which is the default for loopback interfaces.
Such facility with seeing beyond the literal output from show commands and reading between the lines seems to be a useful skill to develop to get better at debugging.
