Project 5 - HTTP Load Balancer

Project Objectives

In this project, you will be implementing a HTTP load balancer providing Direct Server Return in the Python3 programming language. In doing so, you will gain: 

• Hands-on experience with low-level Ethernet, ARP, and IP packet headers
• Hands-on experience with Python programming, including details such as file I/O, string parsing, and command-line argument parsing.

To support these objectives, you are not allowed to use any pre-built HTTP, URL-parsing, or socket management libraries.

Load Balancing Background

An HTTP load balancer is a device that distributes incoming HTTP requests to a collection of HTTP servers (aka a server farm), presenting the illusion of a single, more capable, more reliable server to the user.

A traditional load balancer implements the following system. A firewall is employed, so that only the load balancer is accessible from the public Internet, and the real web servers are hidden.

1. An HTTP request arrives from a client on the Internet destined for the load balancer server, which is the only device that is publicly accessible.
2. The load balancer speaks HTTP. It decodes the request, decides which web server to forward the request to, and generates a new request destined to the actual web server.
3. The web server generates the response and returns the requested information to the load balancer
4. The load balancer forwards the requested information to the client 

Load Balancer - Traditional

In this project, you are going to implement an HTTP load balancer that provides one key performance-oriented feature: Direct Server Return (DSR). Here, the load balancer is bypassed on the return path, improving system performance.

In this system, both the load balancer and web servers have a secondary "Virtual IP" (VIP) address on top of their primary (real) IP address. Clients on the Internet only know about the VIP address, and send requests to that. A firewall is employed, so that only the load balancer is accessible from the public Internet, and the real web servers are hidden. 

1. An HTTP request arrives from a client on the Internet destined for the load balancer's VIP address
2. The load balancer does not speak HTTP. Instead, the load balancer replaces the destination MAC address of the arriving Ethernet->IP->TCP frame with the MAC address of one of the real web servers, and forwards the frame back to the Ethernet switch. Aside from this modification, the HTTP request is the original request from the client, including the client's original IP address and MAC address.  The web server must be on the same subnet as the load balancer.
3. The web server receives the modified HTTP request, generates the response and returns the requested information directly to the client on the Internet. In a Direct Server Return system, the load balancer is not involved in the return message from the web server to the client.

Load Balancer - Direct Server Return

 Background reading (from Barracuda Networks, a manufacturer of load balancers, firewalls, and all manner of network devices):

• Direct Server Return Deployment [barracuda.com]
• Deployment of DSR in a Linux Environment [barracuda.com]

Important note: To proceed, you will need two computers or two Linux virtual machines: One to serve as the load balancer, and one to server as the web server.  In addition, you need a third machine (or your native OS) to run a web browser and act as the client.

Operation

In operation, your load balancer program must perform the following tasks.

At program startup:

• Check if the virtual NIC (virtual0) is available by obtaining a list of all interfaces  (see instructions later for creating the virtual interface)
• Programmatically obtain the MAC address for your specified virtual NIC - this should NOT be hardwired in your program or specified on the command line
• Parse command line arguments or use default values
• Print running configuration
• Create a raw socket and bind it to the virtual NIC interface - This will ensure that your program captures all frames that are received.
• This example code demonstrates this mode of socket operation.  You should add error testing/catching:
  s = socket.socket( socket.AF_PACKET , socket.SOCK_RAW , socket.ntohs(0x0003))
  s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
  s.bind(("virtual0",0))

At program runtime:

• Capture every Ethernet frame that arrives at the virtual NIC
• Decode raw Ethernet frames (if the socket is configured as specified as above, your buffer will contain the Ethernet header first, followed by all other packet headers)
• If the Ethernet frame is an ARP request: 
• Check if the ARP request is sent to the Virtual IP address
• If yes, generate a raw Ethernet ARP reply and transmit it back to the requester using the fake MAC address (00:11:11:....) owned by the load balancer. The requester will learn this Virtual IP <-> Virtual MAC address pairing.
• If the Ethernet frame is an IP packet (destined to Virtual IP address) + TCP packet (destined to Virtual port number)
• Decide which web server to forward this message to.  You need a simple "load balancing" algorithm that ensures that repeat requests from the same client (by IP address) are always forwarded to the same target web server.  You do not need to worry if the web server is up or down (assuming it is up), nor do you need to try to balance traffic evenly across all servers. A real load balancer would use a more sophisticated algorithm, of course...
• Change the destination MAC address of this packet to the target web server and re-transmit the packet
• All other Ethernet frames should be ignored
• Go back and capture another Ethernet frame, running forever until the user enters CTRL-C.

Note that, in order for your load balancing system to function, your web server also needs to be configured to bind specifically to the VIP address (and not the default real IP address for the interface). After completing this, your web server will accept HTTP requests sent to the VIP address and will send HTTP replies using the same VIP address.

Load Balancer Configuration

The load balancer computer must be configured to support the Virtual IP address in addition to the normal IP address on its primary network interface.  This is easy to do in a single command, but wait, there's a nasty catch!  We do NOT want the Linux network stack to be active on the VIP address at all.  This address will be used to accept incoming HTTP requests and forward them to the real web server.  There is no web server running on the load balancer.  Thus, if the Linux network stack is active on the VIP address, it will do nasty things like close all incoming requests, since it (correctly) knows that no HTTP server is running.

What's the solution?  We need a raw Ethernet device (essentially a second NIC, but a virtual NIC, not a physical one).  The following commands will create a virtual interface (e.g. a virtual NIC) called virtual0 that is connected to the same physical network as eth0. This virtual NIC has the MAC address 00:11:11:11:11:11.  Then, the virtual NIC is activated. Finally, the real NIC is set to operate in promiscuous mode, so it captures packets to any MAC address (i.e. including the fake MAC address), instead of just the NIC's real address.

sudo ip link add link eth0 address 00:11:11:11:11:11 virtual0 type macvlan
sudo ifconfig virtual0 up 
sudo ifconfig eth0 promisc 

Confirm that the virtual Ethernet NIC virtual0 exists:

ifconfig

Expected results - note that eth0 is in PROMISC mode now, and virtual0 exists, is up/running, and has a HWAddr of 00:11:11:....

eth0      Link encap:Ethernet  HWaddr 00:0c:29:2c:37:24  
          inet addr:172.16.196.153  Bcast:172.16.196.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fe2c:3724/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:4670 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2976 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2518958 (2.5 MB)  TX bytes:292713 (292.7 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:788 errors:0 dropped:0 overruns:0 frame:0
          TX packets:788 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:74128 (74.1 KB)  TX bytes:74128 (74.1 KB)

virtual0  Link encap:Ethernet  HWaddr 00:11:11:11:11:11  
          inet6 addr: fe80::211:11ff:fe11:1111/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1303 errors:0 dropped:0 overruns:0 frame:0
          TX packets:213 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:143354 (143.3 KB)  TX bytes:22612 (22.6 KB)

At this point, in Python we can send and receive packets on this virtual interface without any interference from the operating system at all. 

Note:  These commands must be repeated each time the computer or virtual machine is restarted. (You can script them at startup, but it's safer this way, so these commands aren't running even after ECPE 177 finishes)

Web Server Configuration

The web server computer(s) must be configured to answer requests destined to the VIP address instead of their normal IP address. And, more significantly, the web servers must be configured to keep their existence secret.  ARP, the Address Resolution Protocol, is used to discover the MAC address for a given IP address. In this distributed web server design, only the load balancer is allowed to respond to ARP requests for the **VIP address**. Otherwise, clients will learn the secret identities of the real web server(s), and try to contact them directly. 

First, configure Linux to only respond to ARP requests for the primary network interface IP(s). 

sudo sysctl -w net.ipv4.conf.lo.arp_ignore=1
sudo sysctl -w net.ipv4.conf.lo.arp_announce=2
sudo sysctl -w net.ipv4.conf.all.arp_ignore=1
sudo sysctl -w net.ipv4.conf.all.arp_announce=2

As documented at https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt, these options mean:

• arp_ignore: Reply to ARP requests only if the target IP address is the same address that is configured for the incoming interface (instead of any interface on the computer)
• arp_announce: When sending ARP requests, always use the best local address.  In this case, that means the real IP address on the physical network interface, thereby keeping secret the virtual IP address that is only attached as an alias on the loopback interface.

Second, configure Linux to use the Virtual IP address as a virtual "loopback" address.

sudo ifconfig lo:1 <<virtual-ip-address>> netmask 255.255.255.255 -arp up

Confirm that the loopback alias is active:

ifconfig

Expected results:

eth0      Link encap:Ethernet  HWaddr 00:0c:29:98:ee:a9  
          inet addr:<<real-ip-address>>  Bcast:xxx.xxx.xxx.xxx  Mask:xxx.xxx.xxx.xxx
          inet6 addr: fe80::20c:29ff:fe98:eea9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1970 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1040 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2355634 (2.3 MB)  TX bytes:110164 (110.1 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:280 errors:0 dropped:0 overruns:0 frame:0
          TX packets:280 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:35479 (35.4 KB)  TX bytes:35479 (35.4 KB)

lo:1      Link encap:Local Loopback  
          inet addr:<<virtual-ip-address>>  Mask:xxx.xxx.xxx.xxx
          UP LOOPBACK RUNNING  MTU:65536  Metric:1

Note:  These commands must be repeated each time the computer or virtual machine is restarted. (You can script them at startup, but it's safer this way, so these commands aren't running even after ECPE 177 finishes)

Requirements

You must use the Python 3 programming language, specifically version 3.4.x or newer, which should be widely available.

(Note: On most systems, the binary will be called python3.  To be safe, run python --version and python3 --version at the command line to see what you have.) 

You must develop your web server on a Linux operating system.  Either a virtual machine or dual-boot arrangement is acceptable. (Your assignments will be graded on a Ubuntu 14.04 LTS machine, if you are interested in developing in exactly the same environment.)

Your load balancer should be runnable from a file called balance.py.  You can import additional Python files (so that your entire project is not in a single file), but the user should not invoke these helper files directly.

 The following command-line arguments should be supported:

You should use the argparse Python library instead of parsing the arguments yourself. Argparse will provide the --help and --version arguments for "free".

Example invocations:

$ ./balance.py --help
usage: balance.py [-h] [--version] [--intf VIP_INTERFACE] [--port VIP_PORT]
                  [--ip VIP_IP] [--targets TARGETS]

HTTP Load Balancer for COMP/ECPE 177

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --intf VIP_INTERFACE  VIP interface (e.g. eth0)
  --port VIP_PORT       VIP port number (e.g. 80)
  --ip VIP_IP           VIP IP address (e.g. 192.168.0.100)
  --targets TARGETS     Comma separated list of HTTP servers for proxy to
                        target (hostname or IP)

$ sudo ./balance.py --ip=172.16.196.200 --targets=172.16.196.151
Using Python 3.4 to run program
Running load balancer server
======= CONFIGURATION =======
Load Balancer interfaces:  [(1, 'lo'), (2, 'eth0'), (3, 'virtual0')]
VIP interface: virtual0
VIP IP: 172.16.196.200
VIP Port: 80
VIP MAC Address: 00:11:11:11:11:11
Target IP list: ['172.16.196.151']
Target MAC list: ['00:0C:29:98:EE:A9']
=============================
Starting Balancer server
<<Load balancer runs and silently inspects/forwards packets>>

Restrictions

You cannot use the following Python built-in modules in this course. Zero points will be awarded for an assignment that uses: 

• http.client - FORBIDDEN!
• http.server - FORBIDDEN!
• socketserver - FORBIDDEN!
• urllib - FORBIDDEN!

In addition to the list above, you cannot use pre-written HTTP server, HTTP load balancer, URL parsing, or socket management libraries obtained from other online sources.

This assignment is to completed individually. You can discuss problems and potential solutions with other students, but you cannot share completed programs or significant pieces of completed code.

See the honor code in the syllabus for specific rules on re-using code found online or in other references. (Specifically, the amount of code reused, and the policy for documenting the reuse)

Tips

• If you're using two virtual machines:  When running Wireshark on the web server for testing, uncheck the option "Use Promiscuous mode on all interfaces" (under the advanced settings) before starting packet capture. Otherwise, the nature of the software network underlying the virtual machines means that Wireshark packet capture will show all of the packets seen by both the load balancer and web server. For example, you would see both the original TCP SYN packet sent from the client to the load balancer, and then the nearly-identical-but-with-modified-destination-MAC packet forwarded from the load balancer to the web server.  This makes the Wireshark log harder to understand.

Resources

See the main resource page for links that helped me when developing my solution.

See also: example Python client and example Python server

Checkpoints

This project has one weekly checkpoint due, in addition to the final project deadline. Checkpoints give the instructor an opportunity to review your in-progress work, and, if problems are found, provide helpful feedback in advance of the project deadline. Checkpoints are graded as full credit, half credit, or no credit.

• Checkpoint 1: 
• Prepare your second Linux virtual machine (or physical machine).  Suggested test strategy:
• Machine 1 (native PC) - Web browser
• Machine 2 (VM #1) - Load balancer
• Machine 3 (VM #2) - Web server #1
• All actions described in the "At Program Startup" section above
• Check if the virtual NIC (virtual0) is available
• Programmatically obtain the MAC address for your specified virtual NIC
• Parse command line arguments
• Print running configuration
• Create a raw socket and bind it to the virtual NIC interface
• Partial implementation of the "At Program Runtime" section above
• Capture every Ethernet frame that arrives at the virtual NIC
• Decode raw Ethernet frames
• If the Ethernet frame is an ARP request, decode it, and print the decoding to the screen (Who sent the ARP? What IP address was it asking about?)
• Go back and capture another Ethernet frame, running forever until the user enters CTRL-C.

Submission

There are slight differences between Python 3.x versions (3.2, 3.3, and 3.4) To ensure I use the same version of Python while grading that you did during development, include the following version-checking code during your program's initialization.
Note: Replace "3,4" with the version number of Python that you used.

import sys
if not sys.version_info[:2] == (3,4):
 print("Error: need Python 3.4 to run program")
 sys.exit(1)
else:
 print("Using Python 3.4 to run program")

If your Python program is just a single .py file, simply upload it to Canvas directly.

Otherwise, if your program contains multiple source code files, create a .tar.gz compressed archive and upload that.  To create the archive, assuming your files are in the folder "project5", run:

$ tar -cvzf project5.tar.gz project5

Once created, upload this archive file to the corresponding Canvas assignment and submit.  To extract your archive, I will run:

$ tar -xvf project5.tar.gz