You are here: Home / Past Courses / Fall 2015 - ECPE 177 / Projects / Project 4 - Network Tester (Part 2)

Project 4 - Network Tester (Part 2)

Project Objectives

In this project, you will complete your networking testing tool to measure network bandwidth and latency. In doing so, you will gain:

  • Hands-on experience with performance testing
  • Hands-on experience with UDP sockets

 

Requirements

The high-level goal of Project 4 is as follows:

Measure network bandwidth, round-trip latency, and packet loss using your own custom client/server program. These measurements should be provided for both TCP and UDP packets. This program models the basic functionality of your instructor's favorite tool: Netperf

You must use the Python 3 programming language, specifically version 3.4.x or newer, which should be widely available.
(Note: On most systems, the binary will be called python3.  To be safe, run python --version and python3 --version at the command line to see what you have.) 

You must develop your test program on a Linux operating system.  Either a virtual machine or dual-boot arrangement is acceptable. (Your assignments will be graded on a Ubuntu 14.04 LTS machine, if you are interested in developing in exactly the same environment.)

Your test program should be runnable from a file called tester.py.  You can import additional Python files (so that your entire project is not in a single file), but the user should not invoke these helper files directly.

The following command-line arguments should be supported: 

  • --help   :  This argument will print out a helpful message describing what arguments the program takes
  • --version   :  This argument will print out the version number of the program (e.g. tester.py 2.0)
  • --test  : The argument can select the various test modes:  TCP_STREAM, TCP_RR, UDP_STREAM, UDP_RR. If not set, the default should be TCP_STREAM. This argument is only used by the client.
  • --target  :  This argument specifies the IP or hostname the client contacts to run the test.  If not set, the default should be localhost. This argument is only used by the client.
  • --time  :   This argument specifies the duration of the test in seconds.  If not set, the default should be 30 seconds. This argument is only used by the client.
  • --client  : This argument configures the tester to act as a client  (this option is mutually exclusive with --server)
  • --server  : This argument configures the tester to act as a server  (this option is mutually exclusive with --client)

You should use the argparse Python library instead of parsing the arguments yourself. Argparse will provide the --help and --version arguments for "free".

Some arguments (like --test and --time) are only relevant if client mode is activated, and should have no effect on the server.

Example invocations:

$ ./tester.py --help
usage: tester.py [-h] [--version] [--test TEST] [--target TARGET]
                 [--client | --server]

Network tester for COMP/ECPE 177

optional arguments:
  -h, --help       show this help message and exit
  --version        show program's version number and exit
  --test TEST      Set mode: TCP_STREAM, TCP_RR, UDP_STREAM, UDP_RR
  --target TARGET  Target IP of client
  --time TIME      Test time in seconds
  --client         Client mode
  --server         Server mode

$ ./tester.py --server
(The program runs in server mode, quietly listening for clients to connect...)
$ ./tester.py --client --target=10.15.20.30 --test=TCP_RR
(The program runs in client mode, connects to 10.15.20.30, and begins a TCP_RR latency test) 

 

To recap, your single program (tester.py) can operate in either a client mode or server mode, depending on how it is invoked.  Thus, you are submitting only one program for this project, not two. 

 

Tests

Your network test program must support the following four tests. Note that, in these measurements, STREAM is a unidirectional flow of data from client to server, while RR (round-robin) is a ping-pong of a single packet back and forth between client and server.

  • TCP_STREAM - A streaming bandwidth test over TCP
    • Goal: saturate the network link with a continuous stream of maximum-sized TCP packets from client to server
    • Test results: Average bandwidth from client to server, reported in Mbps (Megabits per second)
  • TCP_RR - A round-robin (RR) latency test over TCP
    • Goal: exchange a minimum-sized (1 byte) payload between client and server over TCP
    • Test results:  Average round-trip latency between client and server, reported in roundtrip packets/second
  • UDP_STREAM - A streaming bandwidth test over UDP
    • Goal: saturate the network link with a continuous stream of maximum-sized UDP packets from client to server
    • Test results: 
      • Average bandwidth from client to server, reported in Mbps (Megabits per second)
      • Percentage of total packets sent by client that were dropped in-flight (never received by server)
  • UDP_RR - A round-robin (RR) latency test over UDP
    • Goal: exchange a minimum-sized (1 byte) payload between client and server over UDP
    • Test results: 
      • Average round-trip latency between client and server, reported in roundtrip packets/second
      • Percentage of total packets sent by client that were dropped in-flight (never received by server)

 

Note 1: For the TCP_RR test (and only that test), your program should disable Nagle's algorithm for the data socket (not control socket), on both the client and server side.  This is because Nagle's algorithm waits for additional data before sending out a message, and thus will distort our latency test and make the network appear overly slow. To accomplish this in Python, use this line on the socket after it is created:  
my_socket.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

Note 2: For the UDP_STREAM test, what do you think should happen if there is packet loss?  (For reference, you can compare against what the real Netperf does in that situation)

Note 3: For the UDP_RR test, what do you think should happen if there is packet loss?  (For reference, you can compare against what the real Netperf does in that situation)

Note 4: For maximum bandwidth in the STREAM tests, when picking the size to transmit, use the following values:

  • TCP_STREAM test:  64kB (64*1024) is a reasonable size.  This will allow the operating system and NIC to segment the data as desired. 
  • UDP_STREAM test: Max length of Ethernet data portion (1500 bytes) minus Size of IP header (20 bytes, no options) minus Size of UDP header (8 bytes with no options) = 1472 bytes for UDP data
    • Any larger than this, and you will get fragmented IP packets!  (bad for performance)
    • Protocol References:  Ethernet (IEEE 802.3), IP, TCP, UDP

 

Threads

Your client and server will use two sockets at the same time for communication:  a data socket, used for the test data, and a control socket, used to negotiate the parameters of the test with the server and return test results when finished.  Both the data socket and control socket must be active at the same time. To support this, your project must implement thread-based parallelism by providing two threads:

  1. Main thread - This thread will manage the control socket for the client or server.  This thread should always be active as long as your program is running.
  2. Data thread - This thread will manage the data socket for the client or server. The data thread should be created with the user starts a test, and destroyed when the test is finished. 

 

Why do we need both threads active at the same time?  For example, in the UDP_STREAM test, the only way for the server to know that the data test is concluded is to receive the TEST_DONE message over the control socket.  If your server is stuck in a while-loop trying to receive messages from the data socket, then it will never notice the new control message, and thus will never exit..  The easiest way to accomplish this is to use a parallel programming method like threads.

 

Data Socket

The data socket only carries raw test data between client and server. No control messages are to be sent over this socket.  The client will connect to the server to open the data phase of the test, and the client will close the socket with the server to conclude the data phase of the test. The server should listen on an ephemeral port for this data socket, and not be hardwired to listen on any particular port.  The listening port number of the data socket is communicated from the server to the client over the separate control socket.

 

Control Socket

Before exchanging test data, the client and server should communicate over a separate "control" TCP socket to negotiate the parameters of the test.  This control socket will also be used to return experiment data (such as packet loss) from the server to the client.  The communication over the control socket is done in the form of four messages:  Request, Reply, Done, and Results.  Note that these messages are composed of bytes packed into a fixed-sized structure, not the variable length ASCII strings seen in the earlier HTTP project.  

Tip: Although you could build such a structure manually, using the built-in Python struct module is much easier!  You will want to use the "!" symbol when defining your structure in order to specify that the structure is to be built without padding ("standard" data sizes), and that the data format is big endian. 

 

Communication Protocol

Below is an example conversation between client and server, including message format:

< Client opens connection to server on TCP port 5678 >

Test Request (3-byte message sent from client to server to initiate test)

  • Protocol Version (1 byte) - This assignment implements version 1 of the protocol. The field value should be 0x01
  • Message Type (1 byte) - TEST_REQUEST (0x01)
  • Test Type (1 byte) - The following values are permitted
    • TCP_STREAM (0x01)
    • TCP_RR (0x02)
    • UDP_STREAM (0x03)
    • UDP_RR (0x04)
< Server launches data thread. The data socket begins listening on an ephemeral port for test data. >

Test Reply (4-byte message sent from server to client to approve test)

  • Protocol Version (1 byte) - This assignment implements version 1 of the protocol. The field value should be 0x01
  • Message Type (1 byte) - TEST_REPLY (0x02)
  • Test Port (2 bytes) - The TCP or UDP port the client should use to exchange performance data with. (This is the ephemeral port number the server is listening on)
    • Note: This multi-byte integer must be in ***BIG ENDIAN*** format!

 

< Client launches data thread>

< Client opens a second TCP or UDP socket with the server on the port specified in the Test Reply message >

< Client sends test data over data socket >

< Client closes data socket when finished with test >

<Client closes data thread > 

Test Done (2-byte message sent from client to server to denote test has concluded)

  • Protocol Version (1 byte) - This assignment implements version 1 of the protocol. The field value should be 0x01
  • Message Type (1 byte) - TEST_DONE (0x03)

 

Test Results (10-byte message sent from server to client)

  • Protocol Version (1 byte) - This assignment implements version 1 of the protocol. The field value should be 0x01
  • Message Type (1 byte) - TEST_RESULTS (0x04)
  • UDP Packets Received (8 bytes) - Total packets received in this test. Only used in UDP_STREAM and UDP_RR tests; set to 0 otherwise
    • Note: This multi-byte integer must be in ***BIG ENDIAN*** format!

< After a Test Results message is sent from the server to the client, the server will close the control socket. >

< The server should stay running after a test completes, and wait for a new test >

 

Robust Design

Your server should set a timeout on the data socket.  If no data communication is received from the client in 45 seconds, consider the test failed. (Perhaps the client crashed, or the network is down...)  Terminate the data thread, and reset the control thread back to its initial condition: waiting for a new client to initiate a test.

Your client should set a timeout on the data socket, but only in the case of a TCP_RR test.   If no data communication is received from the server in 45 seconds, consider the test failed. (Perhaps the server crashed, or the network is down...)  Terminate the data thread, and exit the client with a clean error message.

Your client should set a timeout on the control socket.  If no data communication is received from the server in "--time + 10" seconds (the length of the test, plus an extra 10 seconds), consider the test failed. (Perhaps the server crashed, or the network is down...)  Terminate the data thread (if not already closed), and exit the client with a clean error message.

Your client and server should both capture attempts to exit the program via CTRL-C, and exit gracefully.  The specific way you exit is up to you, but your program should not print out any ugly Python exception stack traces and error messages.  Instead, you should print out a message saying "Exiting Client" or "Exiting Server". 

When performing a test measurement, both the client and server should be silent during normal operation.  Printing messages to the screen takes time and resources that should be spent running the test!  It is OK to print messages before a test and after a test.

As a general rule, a buggy client should not be able to crash the server! If something goes wrong, your server should abort the test, close the socket(s) with the client, and resume waiting for a new client.

You should be able to run 60 or 120 second long experiments without any of the previously-mentioned timeouts occurring.

 

Testing

In addition to whatever testing you deem necessary, the following additional tests are required:

  1. You must test your server with a client written by three other students in the class (and solve/fix any bugs found so that you implement the standard protocol!)
  2. You must test your client with a server written by three other student in the class (and solve/fix any bugs!)

 

These tests must be done over a real network, not just localhost within your virtual machine.   The networking lab will be open for this performance testing (see Lab Schedule)

When submitting your project, describe your testing results in the file testing.pdf.  In it, name the students you tested your project with, describe any bugs found/fixed, and provide a table with the performance results obtained from your program.
Name TCP_STREAM
(Mbits/s)		 TCP_RR
(Packets/s)		 UDP_STREAM
(Mbits/s, loss %)		 UDP_RR
(Packets/s)
Student 1 
(my client)
Student 1
(my server)
Student 2
(my client)
Student 2
(my server)
Student 3
(my client)
Student 3
(my server)

 

Resources

See the main resource page for links that helped me when developing my solution.

 

Checkpoints

This project has one weekly checkpoint due, in addition to the final project deadline. Checkpoints give the instructor an opportunity to review your in-progress work, and, if problems are found, provide helpful feedback in advance of the project deadline. Checkpoints are graded as full credit, half credit, or no credit.

  • Checkpoint 1: 
    • TCP_STREAM, TCP_RR, UDP_STREAM, and UDP_RR tests are implemented. (Polish, like timeouts and performance tuning socket options, can be omitted here in favor of basic functionality).

 

Submission

There are slight differences between Python 3.x versions (3.2, 3.3, and 3.4) To ensure I use the same version of Python while grading that you did during development, include the following version-checking code during your program's initialization.
Note: Replace "3,4" with the version number of Python that you used.

import sys
if not sys.version_info[:2] == (3,4):
 print("Error: need Python 3.4 to run program")
 sys.exit(1)
else:
 print("Using Python 3.4 to run program")

Your submitted project must include:

  • Python program code
  • A testing.pdf file that describes your testing method, described above

If your Python program is just a single .py file, simply upload it to Canvas directly, along with the other requested files.

Otherwise, if your program contains multiple source code files, create a .tar.gz compressed archive and upload that.  To create the archive, assuming your files are in the folder "project4", run:

$ tar -cvzf project4.tar.gz project4

Once created, upload this archive file to the corresponding Canvas assignment and submit.  To extract your archive, I will run:

$ tar -xvf project4.tar.gz