Using Maxwell To Test TCP Backoff and Recovery
Creative
Commons Attribution-NonCommercial-NoDerivs 3.0 Unported
License.
http://creativecommons.org/licenses/by-nc-nd/3.0/
Creative
Commons Attribution-NonCommercial-NoDerivs 3.0 Unported
License.
http://creativecommons.org/licenses/by-nc-nd/3.0/
This script and video are a companion to one (yet to be written) that shows how to use Mini Maxwell to achieve similar ends.
This video is in lecture format – The speaker, Tech, is an expert on the subject of internet protocols.
Tech is speaking with Curious – who is also well versed in internet protocols and who is responsible for a TCP stack. Curious is concerned whether his TCP stack adequately performs congestion avoidance.
Curious is our target audience. We do not want to condescend to their intelligence.
However, we do want to establish a baseline at the start by giving some light-weight background materials that may be beneath the knowledge level we presume of the audience – these are Chris' famous “sync bits”.
The setting is a laboratory table. Tech and Curious are standing behind the table chatting.
On the table is a Maxwell and a pair of small computers wired to the Maxwell.
The video consists of three parts:
The laboratory table – Tech and Curious are both on screen. The purpose of this section is to give the user the introductory “sync bits” and also a visual foundation to give him/her context for what we will be doing in the next part.
Tech voice-overs of screen captures.
Return to the laboratory table with both Tech and Curious for a summary and conclusion.
Present day
Tech – A university level level expert in TCP/IP networking. Costuming could be tweed coat (with leather elbow patches?) or a lab coat over button-down shirt with tie and khakis.
Curious – A senior programming person. Costuming could be jeans and a T-shirt.
A table on which we have a Maxwell box and two computers, each wired to the Maxwell via its own ethernet switch.
Table
Clipboard or book
A Maxwell box with keyboard, video, mouse (no cables needed)
Qty 2: Ethernet switches
Qty 2: Small computers to play devices under test.
Qty 4: Short ethernet cables to wire the above together.
Qty 1: Ethernet cable leading off-screen to user's network.
Lavaliere microphone to left channel – Tech.
Lavaliere microphone to right channel – Curious.
Warm up lights (at least 2 minutes).
Check eyeglass reflections
Turn off air conditioning fan
Record 30 second sample of baseline background noise through each microphone.
We will need to make screen video captures of one setting up Maxwell filters, delay, and delay automation that are paced in conjunction with the voice-over dialog.
We will be using the following overlay graphics:
We will use a four camera setup.
There will also be cutaways to computer screen shots.
However, because we have only two cameras we will shoot twice, first with Camera 1 and 4, and then with Cameras 2 and 3.
Camera Angles:
Camera 1: Full scene of the two characters in full figure. Long shot. Eye level. Used to establish scene and for things like the overlay graphic.
Camera 2: Medium close (head and shoulders) of Tech. Camera position is roughly 30 degrees off center from Camera 1. Camera is slightly below eye level in order to give Tech an authoritative status with respect to the viewer, i.e. Tech is looking very slightly downward toward the viewer.
Camera 3: Medium close (head and shoulders) of Curious. Camera position is roughly 30 degrees off center from Camera 1 on the opposite site from Camera 2. Camera is slightly above eye level in order to put Curious into an inferior position vis-a-vis Tech.
Camera 4: This is the same as Camera 1 but zoomed in to be a medium shot. It will be used to allow the graphic overlay to be seen reasonably large.
Tech is standing behind a table on which are placed two computers, a big Maxwell box, two mini-maxwells, and appropriate network wiring. A small video screen, keyboard, and mouse may be attached to the Maxwell if it can be done without making things too cluttered.
As the scene opens, Tech is engaged reading something (a clipboard would be appropriate if Tech is in a lab coat, a book if dressed in tweeds.)
Camera 1: Establishing shot
(Tech, noticing the arrival of Curious from Stage Right, looks up from the book and turns to face the incoming Curious.)
Tech:
Come on in; I've been expecting you.
(Curious enters from Stage Right, moves to center-right. Tech and Curious turn as necessary to face one another as they begin to chat.)
I hear you're trying to test TCP congestion avoidance code.
I found a tool that will help.
(Tech turns right and Curious turns left by about 30 degrees so that they are both looking at the Maxwell box in front of them.)
This Maxwell box from InterWorking Labs can exercise your TCP code..
..so you can evaluate your stack under controlled and reproducible conditions.
Curious:
That sounds useful!
Camera 2
Tech:
As you know the internet is a shared resource..
..so TCP code that does not properly react to congestion harms everyone.
TCP is supposed to be a good network citizen.
(Inset traffic seen from Metropolis)
Networks, like highways, can have traffic jams.
This usually happens at junctions..
..places like intersections on highways..
..or routers in networks.
And sending more traffic into a congested area just makes the problem worse.
(Inset freeway ramp metering light image.)
A good TCP implementation backs off when it sees signs of network congestion..
..and then it slowly resumes its transmission rate.
That sounds easy – but the devil is in the details.
(Remove insets)
Camera 3
Curious:
What about explicit congestion notification in IP and TCP?
Camera 2
Tech:
Unfortunately ECN is not widely deployed..
..so a TCP stack can not depend on it.
That means a TCP stack has to play detective.
Camera 3
Curious:
I wanna play Sherlock Holmes!
Let's see..
(Thinking)
Congestion causes queues in internet routers to grow..
(Ah ha! moment)
..so increasing packet round trip time would be one clue!
(Thinking)
And routers limit queue growth by discarding packets..
..so packet loss would be another clue.
(Thinking)
A TCP stack can infer packet delay and loss by examining TCP acknowledgments.
(Ah ha!)
So duplicate ACKs would be a really good clue!
How'd I do?
Camera 1
Tech:
Very well!
But unfortunately there's no perfect algorithm..
..much less any perfect implementation.
A lot of TCP code works fine in routine situations.
But that code might misbehave under different..
..even only slightly different..
..conditions.
Curious:
The network certainly changes from second to second.
Tech:
That means it is very important to test every TCP stack across a broad range of possible network conditions.
But not everyone does enough testing.
Camera 3
Curious:
(Amused agreement)
Yeah, I've had to reboot too many wedged boxes.
Camera 2
Tech:
Code that has not been broadly tested is like an airplane that was tested only in calm weather..
..and loses its wings when it encounters a storm.
Let me show you how to use Maxwell to drive your TCP code through a sequence of congestion cycles.
We will use Maxwell to mimic congestion-induced buildup of queues in internet routers.
(Speak next line while turning to face directly forward)
Here's a diagram:
Camera 4
(Overlay sawtooth diagram of the delay cycles so that it appears to be an image floating between the viewer and Tech and Curious. The chart is correctly seen by the viewer; Tech and Curious are looking at it from behind but are ignoring the fact that they are seeing a reversed image.)
(The diagram should be imagined to be about the size of one of our carpet rectangles held at arms length in front of Tech and Curious.)
(Facing directly forward Tech points to the initial baseline of the chart, half-extended arm gestures are appropriate.)
We will begin with a baseline of 20 milliseconds of delay.
(Swing arm/finger along the baseline towards the ramp-up.)
We will hold that baseline for 60 seconds.
(Point to the first ramp-up)
Then, over the next 60 seconds, we will increase the delay..
(Point to the flat piece at the top of the first ramp-up)
At 120 seconds into the cycle, we will reach 820 milliseconds of delay
and then..
(Point to the sudden drop on the chart)
we will suddenly drop the delay back to our original value of 20.
Then we'll start all over again.
(Fade out the overlay chart.)
Camera 3
(Tech and Curious both face one another at roughly 30 degree angle, )
Curious:
That ought to give my congestion avoidance code a workout.
Could you show me how to set this up?
Camera 1
Tech:
No problem.
(Points to the two computers)
On this table I have two computers that are talking to one another using the open-source "iperf" client-and-server programs.
(Points to the Maxwell in the middle)
In the middle I have Maxwell.
The Maxwell acts as a layer 2 bridge..
..like a smart bump on the ethernet wire.
The Maxwell will impose the cycles of packet delay.
Let's take a look at the Maxwell screen.
(Cut to screen videos that track along with the dialog)
(Voice-over)
(Show Flows screen.)
First we will isolate the TCP connection we want to exercise.
On this screen I have selected TCP and have entered the IP addresses and port numbers for our target connection.
Curious:
What about the return packets?
Tech:
Maxwell can give different treatment to each direction.
(Cut to delay screen)
This screen controls how much we are going to delay the packets in the chosen TCP stream.
Because I want to have a changing rate of delay I will click this button..
(Points to the Automation button and clicks, thus bringing up the Automation page. Then Tech points to the various parts of the pulse-model values.)
I've already entered the parameters that describe the sawtooth diagram we looked at a moment ago.
Let's go back to our main delay screen to get things going.
(Cut back to the delay panel.)
(Point to delay diagram box)
Notice the green box that shows a diagram of one cycle.
We'll click the green running man icon to put things into motion.
(Clicks green running man and then moves mouse to the green diagram box in which the crosshair lines should be moving, albeit too slowly to be easily seen on the screen.)
The cycles are now running..
..and the diagram is showing the progress of each cycle.
(End voice-over)
Camera 4
(Facing directly forward)
I've graphed the iperf TCP data rate so that we can see how it changes as Maxwell ramps-up the network latency.
(Overlay sawtooth diagram of the delay cycles with the TCP data rate so that it appears to be a room-sized image between the viewer and Tech and Curious.)
This graph shows us how the TCP stacks respond to the changing delay conditions created by Maxwell.
Notice how the TCP data rate reacts dramatically to the increased delay..
..even though the underlying bandwidth remains the same.
It looks to me as if the TCP stacks did not negotiate a sufficiently large window scaling factor to handle this amount of latency.
And see how the data rate nicely recovers when the delay is removed..
..not all TCP stacks recover so gracefully.
Camera 3
(Tech and Curious resume the 30 degree angle as before.)
Curious:
It would be interesting to see what happens if we tell Maxwell to discard a few packets when the latency is at its peak.
Or could we have Maxwell shuffle the TCP acknowledgments?
Camera 2
Tech:
Yes..
(Joking pedantic tone)
..But I'll leave that as an exercise for the reader.
(Serious again)
Here we used Maxwell to generate delay variations designed to induce TCP congestion avoidance.
You could use Maxwell to check how your TCP stack handles patterns of packet loss..
or you could combine packet loss with delay or with duplication.
Maxwell goes much further than these simple impairments.
For example, Maxwell can also modify the packets in flight.
Have you ever wondered how well your code can handle unusual..
..but perfectly legal..
..IP packet fragments?
Camera 3
Curious:
We've never tested that!
Camera 1
Tech:
You should..
..the IP reassembly code in a lot of stacks is quite buggy.
Curious:
It seems that Maxwell is a tool that no network lab should be without!
Tech:
We hear that a lot.
Maxwell can create real-world network conditions in your lab..
..giving you an opportunity to really understand how your code responds.
With Maxwell's help you can make your products more robust and more bug free.
Curious:
Wow, this has been helpful!
Where can I learn more?
Camera 2
Tech:
Come visit the website at IWL dot com.
Page