Ethernet Hub Tutorial - Implementation

Module Organization

Before beginning the tutorial, recall from the class lecture that the NetFPGA platform is constructed as a sequence of independent modules, and data flows through these modules as a pipeline.

There are two data paths where packet data enters and leaves your module: Input (from the preceding module) and Output (to the next module). Each path is composed of the following signals:

• Data (64 bits)
• Control (8 bits)
• Write
• Ready

Each module tells the preceding module when it is ready to accept data, and that preceding module pushes data forward and uses the write signal to denote when the data and control buses are currently valid. Note that modules are *not* required to push their data forward in one continuous packet-sized burst. They can start and stop at any time, even in the middle of a packet.

Modules communicate by appending headers to the beginning of an arriving packet. They can also communicate when no packet is available by just sending a header. An example packet with header is shown below:

Packet with Control Headers

Note 1: The first byte of the packet is stored in the most significant byte position (byte 7, i.e. bits 63-56), and so on until the data is finished.

Note 2: The last word of packet data can always be detected by testing for a non-zero control value.

Note 3: To determine where the last byte of packet data is, examine the control value closely. For example, a control value of x40 (0b01000000) at the end of a packet indicates that there are two bytes in the last word, and the last data byte is stored in bits 55:47. Remember, bytes are stored in the most significant byte position first. Similarly, a control value of 0x04 (0b00000100) indicates that there are six bytes in the final word, and the last byte is stored in bits 23:16.

The four MAC RX Queues (one for each Ethernet port) defined in $NF_ROOT/lib/verilog/io_queues/ethernet_mac/src/rx_queue.v each can optionally add a control header at the beginning of every packet received. This behavior is enabled by default from nf2_core.v, which also gives each MAC port a unique queue number. The format of the header is placed at the beginning of every received packet is:

IOQ Packet Header (added by RX Queue)

In the NetFPGA design, there are 8 possible input (RX) ports and 8 possible output (TX) ports. For example, the 8 input ports are:

• Ethernet MAC ports - 4
• CPU (DMA) ports corresponding to the IO sources/sinks - 4

These ports have a specific numbering scheme:

• Even-numbered ports are Ethernet MAC ports - Numbered 0, 2, 4, 6, ...
• Odd numbered ports CPU (DMA) ports - Numbered 1, 3, 5, 7, ...

Initial Setup

To begin, start your X server, enable X11 tunneling, and SSH to ecs-network.serv.pacific.edu.

To create the Ethernet Hub, first start by copying the full NetFPGA directory into your own home directory:

cp -R /exports/provided/netfpga/ ~/. 

Then, enter your new projects directory and copy the reference_nic project, which will be the foundation of your new design. Call your new project ethernet_hub.

cd netfpga/projects
cp -R reference_nic ethernet_hub

Tip: Normally, this would be a good time to check this project into Subversion, but this is not necessary for this tutorial. More details on using Subversion will be provided in a future tutorial.

Change the environment variable in your .bashrc file to point to your new ethernet_hub project instead of the old reference_nic project.

emacs ~/.bashrc &

Change this line:

export NF_DESIGN_DIR=${NF_ROOT}/projects/reference_nic

to:

export NF_DESIGN_DIR=${NF_ROOT}/projects/ethernet_hub

Log off of SSH and log in again to re-load your .bashrc file.

Let's print out the two key environment variables and inspect them. The first should be the location of the base library, and the second should be the location of your current project directory.

echo $NF_ROOT
echo $NF_DESIGN_DIR

Enter your new project directory again.

cd $NF_DESIGN_DIR

As discussed in the preceding class, the critical module to edit is the Output Port Lookup module, which selects the output port for all incoming packets. Because we want to change the existing behavior of this module, let's copy it into our new project directory first. Make a new directory for this module (so we stay organized) inside src/ (which holds all the Verilog code).  Then, copy the existing source file to the new directory.

mkdir -p $NF_DESIGN_DIR/src/output_port_lookup
cp $NF_ROOT/lib/verilog/core/output_port_lookup/nic/src/output_port_lookup.v \
   $NF_DESIGN_DIR/src/output_port_lookup/

Now that we have our own custom module, we need to make sure that the existing library module is not included. Edit the /include/lib_modules.txt file:

cd $NF_DESIGN_DIR/include
emacs lib_modules.txt &

Delete the line that tells the compiler to include the NIC-specific version of Output Port Lookup:

output_port_lookup/nic     <--Delete this line!

Finally, modify the XML file that is used for control register generation.

cd $NF_DESIGN_DIR/include
emacs project.xml

Edit the "name" and "description" variables to change the project from "Reference NIC" to something like "Ethernet Hub".

In the same XML file, delete the line that tells the compiler to look for registers in the NIC-specific version of Output Port Lookup:

core/output_port_lookup/nic    <- Delete this line!

Output Port Lookup in NIC

Next, we examine the Verilog for output_port_lookup.v and see how it functions as a NIC.

cd $NF_DESIGN_DIR/src/output_port_lookup 
emacs output_port_lookup.v &

The goal of the Output Port Lookup module is to tag packets with their correct output ports. There is only one of these modules in the system, as the preceeding input arbiter module has demultiplexed the input queues, and the subsequent output abriter module will look at the tag and place the packet in the correct output queue.

The Output Port Lookup module functions by examining the input data interface and decoding the control signals. One task this module must do is figure out if the incoming data is from a CPU port or a MAC port. If the data is from a CPU port, it needs to go out the corresponding MAC port, and vice-versa. This code determines if it is from a CPU port:

/* pkt is from the cpu if it comes in on an odd numbered port */
assign pkt_is_from_cpu = in_data[`IOQ_SRC_PORT_POS];

Note 1: `IOQ_SRC_PORT_POS is a Macro that decodes to the value 16.
Note 2: This signal is only valid when the packet is currently at the header stage (control value 0xFF).
Note 3: The Source Port vector is binary-encoded, while the Destination Port vector is one-hot encoded.

How does this work? As mentioned previously, CPU ports have odd ID numbers. Thus, by examining the least significant bit of the source port in the packet header (which is binary encoded), we can immediately distinguish between packets coming from the CPU and MAC.

Next, the source port is decoded (i.e. converted from binary to one-hot encoding) with the following Verilog:

/* Decode the source port */
always @(*) begin
   decoded_src = 0;
   decoded_src[in_data[`IOQ_SRC_PORT_POS+15:`IOQ_SRC_PORT_POS]] = 1'b1;
end

How does this work? The 16 bits of the in_data bus where the source port is stored is taken as a number. (Let's say it was MAC port #0). Then, this corresponding bit (position #0) is set to 1 in the output one-hot vector decoded_src.

Finally, the output port is determined with the following Verilog:

if(in_wr && in_ctrl==IO_QUEUE_STAGE_NUM) begin
   if(pkt_is_from_cpu) begin
       in_data_modded[`IOQ_DST_PORT_POS+15:`IOQ_DST_PORT_POS] = {1'b0, decoded_src[15:1]};
   end
   else begin
      in_data_modded[`IOQ_DST_PORT_POS+15:`IOQ_DST_PORT_POS] = {decoded_src[14:0], 1'b0};
   end
end

Note 1: `IOQ_DST_PORT_POS is a Macro that decodes to the value 48.

How does this work? This code fragment only takes effect when the control header is being presented on the bus. Then, in the reference NIC design, this file routes packets from the odd-numbered CPU port to the corresponding even-numbered MAC port and vice versa. So, if the packet is from the CPU, it shifts the one-hot vector *right* and saves it in the destination location, and if the incoming packet is from the NIC, it shifts *left* and saves.

The shifted one-hot bit vector that represents the desired output queue is saved to the empty bits in the original packet header (bits 63:48). The newly updated control header is:

Packet Headers (as modified by Output Port Lookup)

The rest of the output_port_lookup.v file consists mainly of a state machine that monitors the input database and toggles between 2 states:

• Receiving control (module) headers *or* waiting for the next header
• Receiving packet data

Questions

• A small FIFO is provided in the output_port_lookup.v file. What does it do (if anything) in the reference NIC design? What might it do in an Ethernet switch or router design?
• Why is the output port *not* binary encoded like the input port was?

Now that you understand the NIC better, it's time to change it into a Hub!

Tutorial Assignment - Output Port Lookup for Hub

An Ethernet hub (repeater) is a very simple device. Upon receiving an Ethernet frame from one of its ports, it simply copies that frame out all other ports except for the port it was received on. Essentially, an Ethernet hub broadcasts all packets out on all ports. In this tutorial, you will modify your output_port_lookup.v file to act like an Ethernet Hub with the following characteristics:

• If a packet is received on an (even-numbered) MAC port, it will be broadcast out all MAC ports except for the input port.
• If a packet is received on an (odd-numbered) CPU port, it will be broadcast out all MAC ports. (Of course, a real Ethernet hub does not normally have a host system like NetFPGA does. This capability allows the host system to inject packets into the network.)
• You hub will *not* send packets to any CPU port that leads to the host system.

Homework Assignment - Compile Hub

Once you have completed the hub implementation, you will need to compile your verilog to produce a new hardware bitfile. Although you would normally quickly simulate the design first, because hardware compilation takes so long (45-60 minutes if you need to build the Core Generator netlists from scratch), we need to do it between tutorial sessions. Thus, before the next hardware tutorial, please SSH into ecs-network.serv.pacific.edu from home and follow the provided instructions to build your new project and produce a hardware bitfile.

• Hardware Common Tasks - Building a Project