You are here: Home / Past Courses / Spring 2011 - ECPE 293B / Tutorials / Ethernet Hub Tutorial - Simulation

Ethernet Hub Tutorial - Simulation

In this course, we will use the ModelSim HDL simulator to verify our design before testing on real hardware. A substantial Perl library is provided that can generate packets and feed them into the simulated system for analysis. The simulator can be run in GUI mode to allow you to visually manipulate signals and examine hardware state. In addition, it can also be run in a command-line mode that will automatically simulate the hardware, feed in input packets, save output packets, and detect any errors between the actual and expected outputs.

We highly recommended that you devote time to building high-quality automated tests, especially for the Router project when your design becomes complex. Then, every time you build a new hardware version, you can run the automated tests to ensure that no functionality has been broken, instead of manually viewing the ModelSim waveforms and attempting to spot errors visually.

Setup

To begin, first start your local X11 server, enable X forwarding (tunneling), and SSH to ecs-network.serv.pacific.edu. The simulator is run on the main class server (where you also compile your projects).

First, set your design directory to the reference nic project by editing the ~/.bashrc file in your home directory.

emacs ~/.bashrc

Change the line "export NF_DESIGN_DIR" to point to the "reference_nic" project, instead of "ethernet_hub"

export NF_DESIGN_DIR=/path/to/the/project/reference_nic

Log out and log back in to make sure this change takes effect

exit

Once you log back in, run the simulator

nf_run_test.pl --major nic --minor short

We will come back to this later in the tutorial. We're doing this step now so that it can run in the background while you proceed through the rest of the tutorial.

 

Simulation Background

The following components of the simulator will be the most useful to you throughout your project:

  • $NF_ROOT/bin/nf_run_test.pl
    • Main simulator initialization script (run this!)
  • project/<XYZ>/verif
    • In your home directory, project "XYZ" has a verif/ subdirectory that contains multiple tests with packet generation and analysis scripts
  • /tmp/$USER/verif/<PROJECT-NAME>/<SPECIFIC-TEST>/
    • Location of temporary files used during simulator execution. (This is a good place to see the test packets that have been generated)

The other major components of the simulator:

  • $NF_ROOT/bin/nf_compare.pl
    • Script that compares the actual output of the simulator with the expected values.
  • $NF_ROOT/lib/Makefiles/sim_makefile
    • Simulator makefile that will compile hardware into products for ModelSim
  • $NF_ROOT/lib/Perl5/NF/Base.pm
    • Perl module with basic NetFPGA functions (checks environment variables)
  • $NF_ROOT/lib/Perl5/Test/PacketLib.pm
    • Perl module used to create and manipulate packets (that will be converted for use in the simulator by other modules)
  • $NF_ROOT/lib/Perl5/NF/PacketGen.pm
    • Perl module used to generate files with packet data formatted for use in ModelSim
  • $NF_ROOT/lib/Perl5/NF/PacketCmp.pm
    • Perl module used to compared packets output from the simulator with the expected packets.

In the original "reference_nic" project that was provided, let's look at the simulation directory ("verif").

cd projects/reference_nic/verif
ls -ls
>> Output:    test_nic_long/       test_nic_short/        test_phy_reg/

In this directory, there are three different simulation tests available to run. Each test is in a folder with a standard name convention: test_<major>_<minor>. In the case of this project, the major category can either test the PHY or NIC, and the minor category is either a long or short NIC test. When running the simulator, these category names are used to decide what test(s) to run. For example, you could run all of the NIC tests (of any length), or all of the short tests (of any component). You will find it useful to generate different tests to exercise different features of your projects.

Let's look inside one of the test folders:

cd test_nic_short/
ls -ls
>> Output:    config.txt       make_pkts.pl
  • The make_pkts.pl file is the heart of of this test. It uses a Perl library to generate test packets entering from the MAC or CPU ports at specific times, and saves these test packets in a format readable by the simulator. Furthermore, it generates the expected output packets so that it can compare those to the actual packets output after a run of the simulator.
  • The config.txt file saves test-specific configuration options. For example, in the normal NIC test, it saves the simulation end time that is used by the run script.

As mentioned previously, the run script in an individual test directory can not be run directly. Instead, it needs to be invoked by the simulation initialization script, nf_run_test.pl.

The initialization script is normally run with the following arguments, where X and Y are major and minor categories, respectively:

nf_run_test.pl --major <X> --minor <Y>

Common options:

  • --major <string>
    • Specify the string to match on the first part of the test directory name. This is a perl regular expression. Default is to match all major tests.
  • --minor <string>
    • Specify the string to match on the last part of the test directory name. This is a perl regular expression. Default is to match all minor tests
  • --gui
    • Run the simulator interactively with a GUI. (Without this command, the simulator can still be used to create text output files and validate your design.)
  • --help
    • Display full help for the simulation initialization script

 

This top-level script will perform the following functions:

  • Run the simulator makefile to build the hardware (this is not a full compile to a bitfile)
  • Go to your project design directory (as specified in $NF_DESIGN_DIR)
  • Search the /verif directory for any/all tests matching the major/minor search strings
  • For each test matching the search, it will:
    • Copy the entire test folder to /tmp/<USERNAME>/verif/<PROJECT-NAME>/<SPECIFIC-TEST>
      • Note that this will overwrite any existing test of the same name that might be stored there!
    • Execute the script called run inside the test directory. The run script will:
      • Generate the test packets by calling make_pkts.pl and save the raw packet files in /tmp/<USERNAME>/verif/<PROJECT-NAME>/<SPECIFIC-TEST>/packet_data/
      • Call the ModelSim simulator
      • Validate the simulator output (if in non-GUI mode) to determine if there were any errors
  • Record which tests passed and which tests failed
  • Print a summary output


To read the documentation on the packet-handling Perl modules, enter

perldoc $NF_ROOT/lib/Perl5/NF/PacketCmp.pm
perldoc $NF_ROOT/lib/Perl5/NF/PacketGen.pm
perldoc $NF_ROOT/lib/Perl5/Test/PacketLib.pm
etc...

To view the source code of the Perl modules directly (and see more functions than are listed in the documentation), enter

less  $NF_ROOT/lib/Perl5/NF/PacketGen.pm

 

Creating a Simulation Test

 

Walkthrough of test_nic_short

Let's examine the mechanics of the short nic test in order to adapt it to our Ethernet hub. Open up the make_pkts.pl file in $NF_DESIGN_DIR/verif/test_nic_short/ using your favorite editor.

The simulation scripts are written in Perl, which can be a tricky language to understand if you haven't worked with it before (or even if you have...). Some tutorials are available online at Perl.com or Perlmonks.org

The first section of make_pkts.pl sets up the environment for the NetFPGA. It loads the necessary libraries and sends initial configuration messages to put the hardware in a known initial state. After this, several variables are declared.

Next, the for loop generates twelve packets. Three packets are injected per input port. nf_packet_in is the function that actually injects the packet, and cpu_rxfifo_rd_pkt tells the simulator that it should expect a packet to be received by the CPU. PCI_create_and_send_pkt simulates a packet being transmitted by the CPU through the NetFPGA to an output port. Examine all the comments and make sure you understand what this simulation script is doing.

IMPORTANT: Unfortunately, the simulator's infrastructure expects ports to be numbered in the range 1-4; while the hardware numbers the ports from 0-3. Keep this in mind as you write simulator tests and build new hardware!

At the very beginning of the tutorial, you started running the simulation with the following command:

nf_run_test.pl --major nic --minor short

The first time that you run the simulator, it will take awhile because it needs to build cores using Xilinx's coregen tool. The simulator can run very quickly after this first run is done. By now the simulation should be finished running, so let us examine the results. This test will finish with output like the following:

.
.
.
# Timecheck: 153645.00ns
#               160100 Simulation has reached finish time - ending.
# ** Note: $finish    : /exports/provided/NF2/lib/verilog/testbench/target32.v(616)
#    Time: 160100 ns  Iteration: 0  Instance: /testbench/target32
--- Simulation is complete. Validating the output. 
	Comparing simulation output for port 1 ...
	Port 1 matches [3 packets]
	Comparing simulation output for port 2 ...
	Port 2 matches [3 packets]
	Comparing simulation output for port 3 ...
	Port 3 matches [3 packets]
	Comparing simulation output for port 4 ...
	Port 4 matches [3 packets]
	Comparing simulation output for DMA queue 1 ...
	DMA queue 1 matches [3 packets]
	Comparing simulation output for DMA queue 2 ...
	DMA queue 2 matches [3 packets]
	Comparing simulation output for DMA queue 3 ...
	DMA queue 3 matches [3 packets]
	Comparing simulation output for DMA queue 4 ...
	DMA queue 4 matches [3 packets]
--- Test  PASSED
Test test_nic_short passed!
------------SUMMARY---------------
PASSING TESTS:  
		test_nic_short
FAILING TESTS: 
TOTAL: 1 PASS: 1  FAIL: 0

When the simulation has finished, the final thing it does is to make sure that the packets output by the simulator match the expected output that was specified in make_pkts.pl. According to the above, the four physical ports each output 3 packets, and all of them matched the expected values. These are the packets that were transmitted by the CPU. Furthermore, 3 packets were DMA'd to the CPU for each port, and all of them match. These are the packets that were injected into the input ports and sent up to the CPU.

It is worth examining the packets that were injected into the NIC. When writing simulation tests in the future, this will help you make sure that the packets being injected are constructed in the way you envisioned.

cd $NF_WORK_DIR/verif/reference_nic/test_nic_short/packet_data

You should see packet dumps with the following names.

  • ingress_port_X
    • Incoming packets to NetFPGA via MAC Queue X
  • ingress_dma
    • Incoming packets to NetFPGA via all CPU Queues
  • egress_port_X
    • Outgoing packets from NetFPGA via MAC Queue X (actual output from simulator)
  • egress_dma_X
    • Outgoing packets from NetFPGA via CPU Queue X (actual output from simulator)
  • expected_port_X
    • Expected outgoing packets from NetFPGA via MAC Queue X (will be compared against actual output from simulator)
  • expected_dma_X
    • Expected outgoing packets from NetFPGA via CPU Queue X (will be compared against actual output from simulator)
  • pci_sim_data
    • Data for PCI accesses that occur during the test
  • ingress_hardware
    • A tcpdump/wireshark formatted file of all the packets that should be input to NetFPGA. (Suitable for transmitting across the network).

Understanding these files can be an invaluable debugging tool when you are building your router.

Note: Eventually, you will be interested in simulating accesses to registers. The test_phy_reg test provides a good example of how you can read and write registers in the simulator. This will allow you to examine the timing diagram of a working set of registers.

 

Create an Ethernet hub test

You know enough about the simulation infrastructure now to create your own test of your Ethernet hub. You will create a test that injects one packet into the first port and verifies that the same packet comes out on each of the other ports. Begin by editing the .bashrc file in your home directory, and set the NF_DESIGN_DIR back to your Ethernet hub project. Log out and log in, and then use test_nic_short as a baseline.

cp -r $NF_DESIGN_DIR/verif/test_nic_short $NF_DESIGN_DIR/verif/test_hub_port

Your task is to modify test_hub_port/make_pkts.pl such that it injects one packet into port 1 and expects this packet to be seen on ports 2, 3, and 4. Make sure that the ingress_port and expected_port packets (in $NF_WORK_DIR/verif/test_hub_port/packet_data) are what you expect. Then verify that your Ethernet hub passes the test that you just created.

Hint: You will need to use the nf_expected_packet function from $NF_ROOT/lib/Perl5/NF/PacketGen.pm . Other functions in this Perl library may be useful to you throughout the project. perldoc <name of .pm file> can be used to read the in-line documentation.

Hint: Modify the require line near the top.  Currently it says require reg_defines_reference_nic; but it should be changed to reference the name of your current project.  (This register definition file is automatically generated by a script that parses the hardware design files)

After completion, you can run your new simulation test at the command line:

$NF_ROOT/bin/nf_run_test.pl --major hub --minor port

 

Ethernet Hub GUI Simulation

Now, let's simulate your new Ethernet hub with the GUI.

Run the main simulator script: (Make sure you have your X server running!)

$NF_ROOT/bin/nf_run_test.pl --major hub --minor port --gui

This command uses the following options:

  • Major test category is hub
  • Minor test category is port (thus, it is running the test stored in the verif/test_hub_port directory of your project.)
  • GUI - ModelSim will appear on the screen (via X tunneling) and allow you to interact with the simulator.

The project should compile, the simulator script should run, and ModelSim should appear on your screen. Currently, ModelSim is paused and is waiting for you to select the signals to examine. Let's find some signals that we're interested in, and simulate them. In the Workspace pane (left column), browse the hierarchy of Verilog modules to find the module you edited in the previous tutorial:

  • testbench->u_board->nf2_top->nf2_core->user_data_path->output_port_lookup

If you click on output_port_lookup, a list of all of its signals will appear in the Objects pane in the middle of the screen. Select all of the objects, and drag them into the Wave pane on the right column. Now, the simulator will record their values. To un-pause the simulator, go to the menu and choose Simulate->Run->Run-all. This will run the simulator until it reaches the breakpoint specified in the config.txt file, which is plenty of virtual time. (You can pick a shorter increment of time in ModelSim, of course). Choose "No" when it asks if you are ready to finish, and switch back to the Waveform viewer via the tab near the bottom.

Now, you can examine the results of your simulation. Because there is only one packet in this simulation, most of the time the system is idle. Zoom *way* out, scroll back to the beginning of time, and slowly zoom in. You should see packets moving through the output_port_lookup module.

After you finish looking at the waveforms, close ModelSim, and let's look at the files used for the simulation we just completed. Browse the temporary directory created for your test, and look at some of the input and output files:

cd /tmp/$USER/verif/ethernet_hub/test_hub_port/
ls -ls

You should see your make_pkts.pl and run scripts copied here, along with packets in the packet_data subfolder. Remember, this is just a copy, and the main version always lives in your home directory. Anything in /tmp can be deleted at any time. When you run the simulator in GUI mode, the ingress packets and expected packets will be filled in correctly, but there will be no egress packets here. The idea is that you verified the egress packets by hand if you used the GUI.