5. Performance Tuning and Troubleshooting

5.1.1. TCP Receive Window

For many of us, a default Linux installation is going to give something close to optimum performance. Windows 9x users often get a big boost by increasing their TCP Receive Window (RWIN). But this is because it is too small to start with. This is just not the case with Linux where the default value is 32KB.

The exception here is if you have to routinely deal with a high latency connection. For instance, if your provider has a satellite uplink that is consistently adding unusual latency (250ms or greater?). Then a larger TCP Window will likely help. For more on TCP Receive Window and related issues, look at http://www.psc.edu/networking/perf_tune.html.

The Receive Window is a buffer that helps control the flow of data. If set too low, it can be a bottleneck and restrict throughput. The optimum value for this depends completely on your bandwidth and latency. Latency being what you would find as average roundtrip time (RTT) based on your typical destinations and conditions. This can be determined with ping. For example, the Linux default of 32KB is acceptable up to speeds of 2 Mbps and a typical latency of 125ms or so, or 1.0 Mbps and latency of 250ms. Setting this value too high can also adversely effect throughput, so don't over do it.

An example courtesy of Juha Saarinen of New Zealand:

The commonly used formula for working out the tcp buffer is the "bandwidth delay product" one:

      Buffer size = Bandwidth (bits/s) * RTT (seconds)

In my case, I have roughly 8Mbps downstream, but the ATM network can only support ~3.5Mbps sustained. I'm far away from the rest of the world, so to squeeze in a sufficient amount of 1,500 byte packets, with average RTTs of 250ms, I should probably have a buffer of (3,500,000/8)*.25 = 106KB. (I've got 128KB at the moment, which works fine.)

The Receive Window can be dynamically set in the /proc filesystem. This requires entering a value that is twice the desired buffer size:

 #echo 262144 > /proc/sys/net/core/rmem_default 
 #echo 262144 > /proc/sys/net/core/rmem_max

 

The above example actually sets the value to 128K. The Send Window can also be set, but is not as likely to be a limiting factor on DSL connections as the Receive Window:

 
 #echo 262144 > /proc/sys/net/core/wmem_default 
 #echo 262144 > /proc/sys/net/core/wmem_max

 

These values can also be set using the sysctl command. See the man page.

Other suggested kernel options for those who want to squeeze every last bit out of that copper (selected entries only):

 # sysctl -a 
 net.ipv4.tcp_rfc1337 = 1
 net.ipv4.ip_no_pmtu_disc = 0
 net.ipv4.tcp_sack = 1
 net.ipv4.tcp_fack = 1
 net.ipv4.tcp_window_scaling = 1
 net.ipv4.tcp_timestamps = 1
 net.ipv4.tcp_ecn = 0

 

A brief description of these, and other, options may be found in /usr/src/linux/Documentation/networking/ip-sysctl.txt, in the kernel source directory.

5.1.2. Interleaving

"Interleaving" is an error control mechanism of ADSL with DMT line encoding. DMT is now the standard for ADSL, and is by far and away the most prevalent form of ADSL. Interleaving buffers the raw data and corrects errors on the fly at the DSLAM. This can significantly help marginal loops that may be prone to line errors. The downside is that this buffering also adds significant latency to the connection. So for those with reasonable quality lines, interleaving is of no real benefit, and may actually add unnecessary latency.

Interleaving is an adjustable parameter and can be turned on or off by the telco. Many telcos seem to like to have this on by default, since it probably reduces tech support calls in those cases where it does help stabilize a line. But everyone else pays a price.

How to know if your line is interleaved or not, and how to change it? Good question. Generally speaking, if your first hop or two on a traceroute is less than 25ms or so, you can pretty much figure that interleaving is off. But there may be other factors such as how far away those hops actually are. Unless your modem accurately reports this, the only other real way to know is to talk to someone at the telco. This may prove easier said than done.

"FastPath" DMT is synonymous with "interleaving off". Again, this only applies to ADSL/DMT.

5.1.3. TCP Bottlenecks

DSL connections may suffer performance degradations under certain circumstances. Thankfully, Linux has very robust and flexible networking tools to help us deal with these.

One such common situation is where traffic bottlenecks are created whenever data from a fast network segment hits a slower one. Such as ethernet hitting a DSL modem/router. This can cause short term traffic backlogs, known as "queues" in the device. Queuing can result in degraded performance, particularly for interactive protocols (like telnet or ssh) and streaming protocols (like RealAudio), and increased latency for ICMP and other network protocols. This is most evident when the upstream link is saturated (since downstream data is queued at the ISP's end and we can't do as much about that). The queued traffic is processed such that lower volume traffic protocols (like ssh) often get drowned out so to speak, by the higher volume, bulk traffic (like http or ftp), as there isn't any special prioritizing in default usage.

And if the upstream queuing, or other factors, causes enough of a delay, it can even decrease downstream bandwidth utilization by slowing the ACKnowledgements (which are heading upstream), that are required to keep a download moving at optimal rates. So it is possible that an upload can hurt a simultaneous download.

Such effects can be largely mitigated with Linux's built-in traffic shaping abilities. The user space tool for manipulating the kernel's advanced traffic routing features is iproute, sometimes packaged as iproute2. This includes various tools that can classify and prioritize traffic with a considerable degree of flexibility. It also requires various kernel config options to be turned on. And is also fairly close to Black Magic ;-) The definitive document on this is the Advanced Routing and Traffic Control HOWTO (http://tldp.org/HOWTO/Adv-Routing-HOWTO.html). Pay particular attention to the "Cookbook" Section #15, and in particular #15.8, "The Ultimate Traffic Conditioner: Low Latency, Fast Up & Downloads". A great read!

5.1.4. Dropped PPP Connections

PPPoE and PPPoA both rely on the venerable PPP protocol. This protocol incorporates the Link Control Protocol (LCP), which is used to maintain the viability of the connection. Each end can send LCP echoes to other end, and if there is no response in the alloted time frame, the session is presumed dead, and is torn down. Again, either end can initiate this process. The client should then negotiate a new connection. But, this normally means a new IP address is assigned along with the new session.

Perhaps this is undesirable? While you certainly can't control what happens on the remote end in this regard, you can adjust PPP's very flexible way of dealing with LCP echoes on your end, to increase the number of echoes, extend the interval and timeout period on your end. This might help prolong the life of an unstable connection in situations with marginal line conditions, or a buggy peer on the other end. Read your client's documentation. YMMV.

Some providers may deliberately enforce some time limit. There is not much you can do about this.

Also, frequently dropped connections are often an indication of a line problem of some kind. This is something the telco should investigate.

5.2.1. No sync

The modem sync LED has never been green.

• If doing a self-install, the DSL jack may be wired wrong, or the splitter may be wired wrong. Also, the modem may be wired differently than standard telco devices. See above.

• Is the modem Linux compatible? If ethernet interfaced, this should not be a problem. But PCI or USB modems may require drivers just to achieve sync. This could be a show stopper since many PCI and USB modems are not Linux compatible.

• Call your provider and make sure the line was provisioned. It is always possible someone dropped the ball. They may even be able to run a remote test on your line just to verify. There is a also remote possibility that the DSLAM is down. They should know this as well.

• Disconnect the modem power cord and disconnect the DSL cord from the wall jack. Plug it into the test jack inside the NID (outside phone box), and run an extension cord if necessary for power. Temporarily disconnect the wiring to the inside phone circuit. This should effectively bypass any inside wiring and environmental issues. The ethernet cable to the NIC does not need to be connected to run this test (true only for ethernet modems). The modem will sync fine without it. (Easier said than done, I know.) But if possible, move enough of your system where you can view the modem's diagnostics (if available) and get the sync rate. If this works, there is probably something wired incorrectly inside, or a short in a connection somewhere, or there is severe electrical interference on the DSL line. Double check the splitter and wall jack connections. If a splitterless installation, look for bad wiring, bad (e.g. corroded) connections on all jacks, bad splices, or defective microfilters!

  If no sync on the above test, either the line was not readied, the modem is defective, or the DSLAM is down. Note that PCI and USB modems will need to load drivers before syncing, and thus make this test a little more complicated.

• If you installed microfilters, remove these temporarily and unplug all telco devices, such as fax machines, etc. Possibly a mircofilter is defective and shorting out the line.

5.2.2. Network Card (NIC) Problems

Symptoms here are: NIC is not recognized, modules won't load, or ifconfig shows the interface is not up, or is generating lots of errors, etc.

• Turn off Plug 'n Pray in BIOS. This may be labeled as "non-Microsoft OS" or similar. A sometimes symptom here is that the NIC is assigned IRQ 0. Or there may be an error message like "resource temporarily unavailable".

• Check for IRQ conflicts with cat /proc/interrupts. If the NIC is sharing an IRQ, try moving cards around in slots, or tinker with BIOS IRQ settings. If an ISA card, you may need to get the setup utility from the manufacturer and use it to set IRQ, etc. This may require booting to DOS. Modern systems should theoretically be able to handle IRQ sharing, so it is not necessarily a problem in and of itself. Only if something is misbehaving.

• Possibly the wrong module is being loaded. Look through the kernel source documentation in /usr/src/linux/Documentation/* for your card or chipset. Also, for comments and update information in /usr/src/linux/drivers/net/*.c for your respective chipset. It is worth noting that there is more than one module for some card types. This seems to be true of tulip and 3Com cards. Check boot messages or use lspci -v to see how the kernel is identifying your card. You can use insmod, rmmod, and modprobe to test different modules. See the respective man pages for more information.

• Check the manufacturer's web site for Linux documentation. Or look at Donald Becker's informative site at http://www.scyld.com/network/.

• Some Linux NIC drivers reportedly work better as non-modular. In other words, compile them into the kernel instead of as a module.

• It is also possible that the card is bad, or the drivers just aren't up to snuff. Try another card. And you don't need an expensive, high quality card necessarily either.

5.2.3. IP Connection Problems

Read this section if you are sure the modem is syncing, the NIC is recognized and seems to be working properly, client software is installed and running without error, but the connection to the ISP fails. Verify the modem is indeed syncing by the LED(s). An IP connection failure may be evidenced by ifconfig not showing an active eth0 interface (or ppp0 for PPPoX), or pinging gateway and other destinations generates 'network unreachable' or similar errors.

• Make sure you know which protocol your ISP is using. Are they using DHCP? PPPoX? It is critical that you have this right. You may have to ask tech support.

• If you are using DHCP, does the ISP require MAC address authentication, and if so, do they have the right address? Did they or you typo it? If the ISP requires hostname authentication, is your DHCP client passing the required hostname? This is done with the "- h" command line option.

• Look at /var/log/messages and see if any useful clues are there. Also, run tcpdump while trying to initiate the connection. tcpdump output is fairly cryptic, but you should be able to determine if there is any response at all.

• If PPPoX, is the ISP using username as an id, or username@isp.com?

• CHAP, PAP, or other? I would set up both CHAP and PAP (see man pppd) just to be safe.

• Try pinging the default gateway's address. Get this with 'route -n'. If you can ping by IP address (i.e. 111.222.333.444), but not by hostname, then likely nameservers are not correctly setup in /etc/resolv.conf. This is configurable as to whether your connection protocol (e.g. PPPoE) does this automatically or not. And different distributions may have their own way of setting this up, so check their documentation first. In a pinch, just add them manually to /etc/resolv.conf. pppd also has the "usepeerdns" option that can be enabled.

• For rp-pppoe, let the PPPoE client bring up the ethernet interface. Do not have it come up on boot. Make sure there is no existing default route before starting PPPoE. For rp-pppoe, David Skoll recommends that /etc/ppp/options be left empty.

• If running a firewall (e.g. with ipchains), try temporarily taking it down. Possibly this is misconfigured, and not allowing packets through.

• Roaring Penguin has a very nice debug output with all kinds of system info, and even tips for correcting problems. See the docs for turning this well-done feature on.

• If the modem was purchased from a source other than your ISP, it may the wrong kind of modem. SDSL needs an SDSL modem, for instance. Also, for ADSL there are CAP and DMT encodings, and these are incompatible with each other.

  The modem may need to be configured for your ISP's service. All modems have configurations for VCI, VPI, encapsulation, etc. Call tech support for this information. Modem configuration is usually done by either telnetting or web browsing to the modem's IP address.

5.4.1. Miscellaneous Network Problems

Some odds and ends:

• Some Web pages won't load. For PPPoX users, the MTU value could be too high. This will cause packet fragmentation, and likely will cause misbehaving routers to fail to route your requests per Path MTU Discovery specs.The correct ppp0 device setting should be a maximum of 1492, but actually it needs to be 8 bytes less than any router you pass through on the way to the site. If a router somewhere is misconfigured, you could have problems. Try experimenting with lower MTU values. Any LAN hosts behind the connection, may even need to be even lower -- 1452 or maybe even 1412. If ECN is enabled, it might also cause this problem. Cured with "echo 0 > cat /proc/sys/net/ipv4/tcp_ecn".

• Ping by IP address works, but not hostname. The nameservers are not being setup correctly in /etc/resolv.conf. Check your client's (DHCP, PPPoX) documentation or enter these manually with a text editor. Get the correct DNS server addresses from your ISP.

• PPPoX disconnects. Unfortunately, PPPoX is more likely to drop connections than routed or bridged networks. PPP can be sensitive to any line condition which results in a temporary interruption of the connection. This may not be completely solvable, depending on what and where the problem is. Check your client's docs for "LCP Keepalive" features. There generally is a timeout on each end of the connection if the other end does not respond. If worse comes to worse, set up a cron job to watch the connection, and re-establish if necessary.

  Some providers may also be enforcing idle timeout disconnects. This is a different issue altogether, since it is deliberate. The solution here is to switch providers if you can.

• Interface or route goes down for no reason. If ifconfig and/or route show the interface and/or route has automagically disappeared, it may be due to a buggy NIC driver.

• Sub-par performance, or errors with the interface (e.g. eth0), may possibly be caused by a duplex mismatch. This would be most apparent when maxing out the connection. Most DSL modems and routers typically are set to half duplex, and your NIC that interfaces with the modem should be set likewise.