OS X 10.9 brings fast but choppy Thunderbolt networking

If you open your network settings in the System Preferences after upgrading to OS X 10.9 "Mavericks", you'll be informed that a new "Thunderbolt Bridge" network interface was added to the system. So it's now possible to network two Macs over Thunderbolt. Let's take our new network for a spin.

Obviously you need a Thunderbolt cable to connect two Thunderbolt-equipped Macs. I got the 0.5 meter one from Apple, which is quite short, and connected my mid-2011 MacBook Air to a brand new MacBook Pro ("with Retina Display", but that almost goes without saying now). The MacBook Pro has 20 Gbps Thunderbolt 2 while the Air has to make do with the regular 10 Gbps Thunderbolt—but on both the Network Utility reports a link speed of 10 Gbps.

On the Air, the network interface now came up, but not on the Pro. The reason for that is that the Thunderbolt Bridge interface is actually a virtual bridge interface that allows network packets to be passed from one physical network interface to another. This also means that packets toward those interfaces should now go through the bridge interface. The Air's lone Thunderbolt port was part of the Thunderbolt Bridge out of the box, but on the Pro only one of the ports was part of the bridge—and of course I had connected my cable to the other port. This was easily fixed.

With the Thunderbolt Bridge interface up and running on both computers, I turned off Wi-Fi on the Pro to make sure that I wouldn't accidentally connect wirelessly. I then connected from the Air to the Pro through the Finder, mounting the Pro's SSD as a network share. Now I was in the position to copy files back and forth and time how long that took.

The result: much longer than expected, in the Pro-to-Air direction, at least. However, by monitoring network throughput in the Activity Monitor, it turned out that when copying from the MacBook Pro to the MacBook Air, the transfers could be extremely fast, sometimes showing transfer speeds of a gigabyte per second. But at random intervals, the transfer would stall for a number of seconds, and then continue at either very low or very high speed.

In the other direction, from the older, slower MacBook Air to the brand new, much faster MacBook Pro, there were no such issues. Here, Activity Monitor fairly consistently showed around 500MB per second worth of network traffic. However, that number doesn't correspond to the transfer times and file sizes. Based on these, the transfer speed was 200MB/sec. Some of the discrepancy is explained by network overhead and TCP taking some time to ramp up to multi-Gbps speeds, but I'm pretty sure what's really going on here is that the system counts the network activity of both the physical Thunderbolt interface and the virtual bridge interface, thus reporting twice as much network activity as what really took place.

It remains unclear why the transfers from the Pro to the Air were so inconsistent. An obvious reason would be that the Pro sends packets faster than the Air can handle, but looking at the interface statistics (netstat -i en1 -ss), it turns out that the difference between packets sent and packets received was only 0.02 percent—not really enough to cause the observed behavior. Getting out tcpdump and inspecting the data traffic also didn't show any reason for the slowdown: I couldn't find a single instance of SACK blocks, which would indicate lost packets in the middle of a transfer. The only indication that something is going on is that once in a while, there are simply no packets for 10 milliseconds or so, for no reason that I could find.

Maybe there is a software issue, or maybe there is a bottleneck in one of the Thunderbolt implementations—the mid-2011 MacBook Air was the first model to gain Thunderbolt. With Mavericks, Apple switched to SMB as the default protocol to transfer files locally, but using AFP didn't make a difference.

Although the interfaces report using the standard Ethernet packet size or MTU (maximum transfer unit) of 1500 bytes, in reality the SMB transfers typically happened as four 65212-byte packets and then a 1556-byte packet—which makes for a 256kB payload. However, sometimes the packet sizes would be different.

The Thunderbolt network interface also indicates that it supports TCP segmentation offloading for both IPv4 and IPv6 (TSO4 and TSO6), but presumably, there's no actual network hardware in the Thunderbolt interface that could perform this function. The idea behind TSO is that the network software creates one large packet or segment, and the networking hardware splits that packet into pieces that conform to the MTU limit. This allows gigabit-scale networks to operate without using excessive amounts of CPU time. What seems to be happening here is that the system maintains an outward appearance of using the standard MTU size so nothing unexpected happens, but then simply transmits the large TCP segment over Thunderbolt without bothering with the promised segmentation.

As Apple actually implemented Thunderbolt networking as Ethernet over Thunderbolt rather than IP over Thunderbolt, it's possible to add other Ethernet and Ethernet-like interfaces (such as Wi-Fi) to the Thunderbolt Bridge: use "manage virtual interfaces" in the little gear menu under the list of interfaces in the network settings.

I had the Pro bridge its Wi-Fi interface to the Air over Thunderbolt, and from the Air's perspective it was just like it was connected to the Wi-Fi network: other computers showed up in the Finder and everything. However, the Pro had a kernel panic and the Air had a hard time copying files or loading web pages. Could be a side effect of the creative take on TSO, or maybe something else is going on.

Assuming stability and consistency will be improved, is Thunderbolt networking useful in the first place?

Obviously, outfitting Macs with Thunderbolt 10 Gigabit Ethernet adapters and hooking them up to a 10GE switch is the superior solution. With an equally superior price tag. In situations where a fast network is only needed to copy files between two computers, using a $30 or $40 Thunderbolt cable is much more convenient at a fraction of the price. I can also imagine a Mac Pro being outfitted with a 10GE adapter, and then one or two other Macs connecting to the 10GE network through a Thunderbolt connection to that Mac Pro. And if Thunderbolt networking catches on, we may even see Thunderbolt ports on NAS devices.

Say what you will about Apple, but they're not afraid to try something new. As this new feature shows, it really pays off to have a generic high speed port in every Mac. I can't wait to see what else Thunderbolt has in store for us.

IP over Thunderbolt looks like it has lots of potential, but on the other hand, IP over Firewire was pretty cool a decade ago, too. Windows actually handled IP over Firewire in much the same way as OS 10.9 handles Thunderbolt networking, allowing you to bridge between Firewire and Ethernet ports. But then it was dropped in Vista. So enjoy your fast network while it lasts.

Update: by popular demand, I also tested with iPerf, which tests raw TCP performance, without reading or writing to disk. iPerf transfers data without the overhead incurred by actual file-sharing protocols, and these numbers won't be representative of actual performance in most cases. Sending from the MacBook Pro to the MacBook Air happened at 5.3Gbps, in the other direction slightly faster at 5.7Gbps.