Locating Files / Searching Files

To find a specific file somewhere on the class server (such as rx_queue.v), use the following command

locate rx_queue.v

To search the text of all NetFPGA library files to find out where a specific variable is used or defined at, (such as the ENABLE_HEADER parameter in rx_queue.v), use the following command. Note that it is case sensitive.

grep -R "ENABLE_HEADER" $NF_ROOT

Administrative Tools

On your NetFPGA system, there is a special program called op (short for "Operator") that allows you to perform a limited number of functions that require administrative (root) access.  You can access the current list of commands by entering op -l. Some key commands are shown below:

Restart machine

op reboot

Re-initialize NetFPGA PCI controller

op cpci_reprogram.pl <<optional args>>

Run Wireshark

op wireshark &

Run tcpdump

op tcpdump <<args>>
op tcpdump -i ethXX

Run bitfile download utility

op nf_download /path/to/bitfile.bit

Run dumnet packet generator

op dumbnet <<args>>

Run arp configuration tool

op arp <<optional args>>

Run ping (need to be root to do a ping flood stress test on your router)

op ping <<optional args>>

Run software router (scone-base). Note that root access is only needed for the hardware/software integration project, where the software uses raw sockets to communicate with hardware.

op sr <<args>>

Creating a Project

We recommend copying the reference_nic project when creating a new project, because it automatically includes all essential NetFPGA library files. This project can be found at $NF_ROOT/projects/reference_nic. Your new project should be saved at $NF_ROOT/projects/<your_project_name>.

When creating a new project, be sure to update your profile's .bashrc variable to point to your new project. For example, change

export NF_DESIGN_DIR=${NF_ROOT}/projects/reference_nic  

to

export NF_DESIGN_DIR=${NF_ROOT}/projects/<your_project_name>

Tip 1: During the course projects, you may find that you do not want to "start over" at the Ethernet NIC project again, such as when you go from the Ethernet Switch to Ethernet Router projects. You can always copy and rename any project. However, after copying, you may want to do a make clean and make really_clean in the synth/ directory of the new project to remove all compiler-generated temporary files from the previous project. Otherwise, your new project will begin its life in a cluttered state.

Tip 2: - Be *careful* when copying a project that is checked into Subversion! You do *not* want to copy the hidden .svn folders that store repository data. See the Subversion Information page for more details on how to safely copy items in the repository.

Tip 3: Update the include/project.xml file to change the project name to something other than "Reference NIC"

Using Library Modules

You can choose to use some, all, or even none of the original library modules. The $NF_ROOT/projects/<your_project_name>/include/lib_modules.txt file specifies a list of modules to use from the library. The modules are specified relative to the $NF_ROOT/lib/verilog/ directory. You can choose to use different modules simply by changing the module path in the lib_modules.txt file.

Tip: By copying the lib_modules.txt file including in the reference_nic project, you will include all of the modules of the basic NIC design.

Adding Custom Code

To add your own code, write the verilog files and put them in the $NF_ROOT/projects/<your_project_name>/src directory.

Tip: You can choose to partition your verilog files into separate directories under src. Note that only one level of hierarchy is usable.

Overriding Library Code

You might decide that you only need to change one file of a library module. You don't need to copy all the sources and remove the library module from lib_modules.txt. Simply copy the file you wish to modify to your project's src directory and modify it there.

If you wish to completely re-write an existing library module, you can copy the library code and modify it in your project directory. Since you are not using the original library code, be sure to remove that module name from the lib_modules.txt file.

Using Xilinx CoreGen Modules

If you need to use IP cores generated with Xilinx's Coregen, simply use the graphical CoreGen utility to generate your core.

coregen &

You'll need to create a new temporary project somewhere in your home directory, and set it to use the same FPGA type/size/speed/package as the real NetFPGA board. The file $NF_DESIGN_DIR/synth/nf2_top.map should provide the necessary specifications (specifically, xc2vp50 FPGA, ff1152 package, and -7 speed grade). Further, set this project to general wrapper files and example code in Verilog, not VHDL. (Project->Project Options->Generation->Design Entry: Verilog). This temporary project does not need to be part of Subversion, since you won't use it after the core is generated.

After creating your new temporary project, you'll be able to browse the list of available cores for your FPGA family and instantiate them. Once you have instantiated your core(s), copy the .xco file that was generated to the $NF_DESIGN_DIR/synth directory (i.e. the synth directory inside your project). You don't need any of the other files, as the Make process will automatically create the netlist from the .xco file.  In addition, you may want to copy some of the instantiation code out of the .v file that was generated, and paste it into your own custom modules.

Tip: If using a memory, you might want to create a .coe file (coefficient file) that will set initial values. In this case, you'll need to copy the .coe file to the /synth directory as well, and update the .xco file to point to it. Due to the way the Makefile changes its place in the directory structure, you might need an unusual path in the .xco file, such as

CSET coefficient_file=../my_coefficient_file.coe

Tip: If you are unable to open the PDF documents for each core (by right-clicking on the core name and choosing "View Data Sheet"), you need to specify the PDF program used to view the file. Go to File->Preferences, uncheck the box labeled "Use default PDF viewer", and enter the path to your PDF viewer.  (Options include /usr/bin/acroread ,  /usr/bin/xpdf , or /usr/bin/evince)

Using Xilinx Timing Analyzer

The Xilinx Timing Analyzer tool can be used to view all the clock domains in your design and see what signals do not meet the timing requirements placed on them. To run the timing analyzer, enter

timingan_old &

To analyze your design, go to the File->Open Design menu, and enter the paths to your design. For example:

Design File:

projects/ethernet_switch/synth/nf2_top.ncd

Physical Constraints File:

projects/ethernet_switch/synth/nf2_top.pcf

Click OK to open your design. After processing, choose Analyze->Against Timing Constraints. The following options are important:

• "Timing Constraints" tab:
  • You can choose what timing constraints you wish to view. The "TS_core_clk_int" is the main 125MHz domain, but you probably want to include all of the constraints to make sure there aren't any other errors.
  • You can view all signals (pass & fail) for a domain, or just failing signals. Select "Report failing paths against timing constraints", because that is normally what we are interested in.
• "Options" tab:
  • You can choose how many failing signals you want to print (under "Limit Report To:"). The default is 3, but we recommend viewing at least 50 to get a better idea of the scope of the problem.

Click OK to analyze your design based on the options you selected. This should take a minute or two to process before displaying the results. The report will show your clock domains and selected signals from those domains. All signals in RED TEXT have failed their timing constraint.

Using Xilinx Floorplanner

The Xilinx Floorplanner tool can be used to view the design implementation on the FPGA and see where each module is physically placed. You can use this information to gain an understanding of which areas of the FPGA are congested and where optimization effort should be applied. To run the floorplanner, enter:

floorplanner &

Choose File->Open, select your design (e.g. $NF_DESIGN_DIR/synth/nf2_top.ngd), and select OK twice. It will take a minute or two to process your design before displaying it.

In the Floorplanner, you can select modules in the Design Hierarchy window, and they will be visually highlighted in the FPGA placement window. If you zoom in far enough, you can select individual FPGA blocks, and by clicking/hovering your mouse, learn what specific part of your design it corresponds to. This can be useful in optimizing your projects or identifying why signals are slow. If you see that your module is widely dispersed across the FPGA (perhaps because it produces or consumes data from modules on the other side of the chip), it might be having problems meeting timing constraints. Thus, you'll need to add additional registers to buffer the signal.

Tip: By default, the Floorplanner placement window highlights both the FPGA elements *and* draws wires connecting them together. This is a poor choice as a program default since the wires make it very difficult to view the underlying design. To turn off these wires, click the toolbar button with the text label "Toggle Ratsnest" (it's the 8th button from the right with the pink wire icon)

Using Xilinx Output Files

The $NF_DESIGN_DIR/synth directory of your project is filled with compiler-generated text files containing important project information.

• nf2_top.map and nf2_top.mrp - Output files for the Xilinx Map tool (which converts your logic into actual FPGA devices such as LUTs). These reports summarize your design and shows you how many FPGA resources (like Block RAMs, slices, etc.) are occupied and how many are still available for your design.
• nf2_top.srp - Output file from the HDL synthesis tool. This usually has "helpful" descriptions of any syntax errors in your HDL, as well as warnings for any latches that you might have accidentally created.
• nf2_top_par.twr - Output file from the timing analysis program, showing all your clock domains and counting the number of signals in each domain that failed timing. The ideal phrase to have in this file is the magic line "All constraints were met."
  • Note: You should ignore the Timing Summary section at the bottom of this file. The "Maximum frequency" statistic is not useful for this design because it has multiple clock domains. You should only be concerned with the number of timing errors found for each individual timing constraint.
• nf2_top_par.par - Outfile file from the place&route program, showing how long it took for the route to be calculated, and how many iterations it took to meet timing. The more iterations, the harder it is working to fit your design.
• synth.txt - Outfile file containing the entire command-line output of the compilation process (in case you missed it as it was scrolling past)

Reading / Writing Hardware Registers

Two utilities are provided to allow you to read and write to registers from the command line: regread and regwrite. To use these tools, you need to provide one or more register addresses, and one or more values to write.

regread [addr1] [addr2] ...
regwrite [addr1] [val1] [addr2] [val2] ...

For example:

regread 0x0600004
regwrite 0x0600004 0x101

To compile these tools for your own use, you need to copy them from the provided library into your own directory. The sw/ subdirectory of your project folder would be a good place. For example, if your project is ethernet_switch, copy both the reg_access, reg_lib and common directories to your own project's sw/ directory.

cp -R $NF_ROOT/lib/C/common $NF_DESIGN_DIR/sw/
cp -R $NF_ROOT/lib/C/reg_lib $NF_DESGIN_DIR/sw/
cp -R $NF_ROOT/lib/C/reg_access $NF_DESIGN_DIR/sw/

Then, enter the reg_access directory and compile the utilities:

cd $NF_DESIGN_DIR/sw/reg_access
make

Note: These tools must be compiled on your group's netfpga system (a 32-bit machine) instead of on the ecs-network server (a 64-bit machine). If you accidentally compile the utilities on the wrong system, simply do a make clean in both the reg_access and common directories and recompile it again.

Building / Cleaning a Project

• To build your hardware project, go into the $NF_DESIGN_DIR/synth directory and run the Makefile.

make

The Makefile will invoke the Xilinx ISE compiler, build your project, and produce an output bitfile named nf2_top_par.bit in the /synth directory. In addition, it will rename that bitfile to the name of your project and copy it to $NF_ROOT/bitfiles.

Tip: Xilinx Coregen needs X to run, even from the command line as part of the Makefile. Make sure that you have an X server and have enabled X11 forwarding if IP cores are being built (for example, the first time you are simulating or implementing a design.)

Tip: To check the timing performance of your design, look at the nf2_top_par.twr file.

• To clean your hardware project (i.e. delete all compiler-produced files), go into the $NF_DESIGN_DIR/synth directory and issue the following command:

make clean

• To really clean your hardware project (including deleting all of the hardware cores generated by CoreGen such as FIFOs, the Ethernet MAC, and PCI core), go into the $NF_DESIGN_DIR/synth directory and issue the command:

make really_clean

Simulating your Design

The simulation environment and capabilities are described in detail on the Simulation Tutorial page. Each project can have multiple tests that can be run individually or in sequence.

You do not need to build the hardware before simulating it. The simulator will produce all the files it needs, which is only a small subset of the work required to build the hardware.

The simulator initialization script is normally run with the following arguments, where X and Y are major and minor categories, respectively:

nf_run_test.pl --major <X> --minor <Y> 

Common options:

• --major <string>
  • Specify the string to match on the first part of the test directory name. This is a perl regular expression. Default is to match all major tests.
• --minor <string>
  • Specify the string to match on the last part of the test directory name. This is a perl regular expression. Default is to match all minor tests
• --gui
  • Run the simulator interactively with a GUI. (Without this command, the simulator can still be used to create text output files and validate your design.)

Tip: Xilinx Coregen needs X to run, even from the command line as part of the Simulator Makefile. Make sure that you have an X server and have enabled X11 forwarding if IP cores are being built (for example, the first time you are simulating or implementing a design.)

Resetting your FPGA

A CPCI utility is provided that reloads the PCI controller bitstream on the Spartan FPGA. (You are not responsible for the PCI controller or this FPGA). As a side effect of this process, the main Virtex FPGA is also completely reset. Thus, if you need to completely clear the Virtex FPGA for some reason, you can run the command:

op cpci_reprogram.pl

Downloading your Design

To download your compiled bitfile to the NetFPGA board, first SSH to your NetFPGA machine. Then enter:

op nf_download /path/to/bitfile.bit

What if your design doesn't work in hardware, but worked in the simulator? There are two common reasons for this:

1. Timing errors - Your design is trying to accomplish too much in a single 125MHz clock period. Look at the clock report for the place and route (PAR) tool.  Although all clock domains are important, only the core_clk_int domain should be impacted by your design. Make sure this domain has 0 timing errors and a timing score of 0.  For good measure, scroll to the bottom of the report and make sure you see the line "All constrains were met."
2. Latches -  A latch is a storage element with an input, output, and enable pin. When enable is active, the output transparently follows the input (with some small delay). When the enable becomes inactive, the output freezes. Latches are discouraged for use in FPGAs because they make timing constraints and analysis more difficult. Unfortunately, you can un-intentionally create a latch in Verilog if you write a combinational process or always block where an output is not assigned under all possible input conditions. In other words, it is possible for one of the inputs to change without affecting the output. In synthesis jargon, this is known as incomplete assignment.  The Xilinx Synthesis Tool (XST) will automatically infer latches and only gives you a warning that it has done so. Search the synthesis report file for warnings and eliminate them by explicitly setting the output for all states!   The warning to look for should look like this:

WARNING:Xst:737 - Found 1-bit latch for signal <signal_name>. 
Latches may be generated from incomplete case or if statements.
We do not recommend the use of latches in FPGA/CPLD designs,
as they may lead to timing problems.

Verilog to HTML Converter

v2html is a free Perl script that converts verilog designs into webpages, and allows you to visually navigate and search your source code. It can be a useful tool in helping you understand the larger NetFPGA project.

To use this utility, make a new directory, download the tool, and unpack it.

mkdir v2html
cd v2html
wget http://www.burbleland.com/v2html/v2html.tar.gz
tar -xzf v2html.tar.gz

Make the following control script in the directory. Call it build.sh:

emacs build.sh &

Be sure to update SRC_DIR_2 to point to the project you wish to analyze. (You might set it to $NF2_DESIGN_DIR)

#!/bin/bash

# Specify:
# (1) The read-only NetFPGA library
# (2) Your project source directory
# (3) The destination directory (in /tmp)

SRC_DIR_1=${NF_ROOT}
SRC_DIR_2=${NF_DESIGN_DIR}
DST_DIR=/tmp/${USER}/v2html_src

## Check if X11 forwarding is enabled on client.
if [ "${DISPLAY}" = "" ];
then
    echo "Error: X11 tunneling must be enabled before running this command"
    exit 1
fi

# See http://technofreakatchennai.wordpress.com/2007/10/04/bash-copy-with-find/
# for BASH example


# Search for all .v files in source directories, and copy to /tmp
# It is important to copy the USER PROJECT files SECOND, because they
# overwrite the standard library files of the same name that were copied first.
rm -rf ${DST_DIR}
mkdir ${DST_DIR}
mkdir ${DST_DIR}/output
find ${SRC_DIR_1}/ -name \*.v -exec cp {} ${DST_DIR}/ \;
find ${SRC_DIR_2}/ -name \*.v -exec cp {} ${DST_DIR}/ \;

# Had problems displaying the register subsystem
# (crashed the converter), so omitting
rm $DST_DIR/oq_regs*

# Convert to HTML, and open with Firefox
./v2html -F $DST_DIR/*.v -o $DST_DIR/output
firefox $DST_DIR/output/frame.html &

chmod +x build.sh

Run the script, which will parse the verilog files and open a Firefox browser

./build.sh

In the Firefox window that appears, you can navigate your project. The top frame is the hierarchy, the middle frame is the source viewer, and the bottom frame shows the definition. Not all of the files included have a direct place in the hierarchy. (For example, the PCI core running the Spartan FPGA won't be part of the main hierarchy on the Virtex FPGA) To find, for example, the output_port_lookup module, scroll down in the top frame to u_board, and click the folder icon to expand the following path:

u_board->nf2_top->nf2_core->user_data_path->output_port_lookup

Note: If a module is instantiated inside a Verilog generate loop, then v2html does not correctly place the module in the hierarchy, and instead displays at at the top level. For example, the nf2_mac_grp module is shown at the top level, but 4 copies of it are actually generated at at u_board->nf2_top->nf2_core. Alas, this free script is not quite as perfect as a real Verilog compiler!

Transmitting Raw Packets with Dumbnet

Dumbnet (dnet) is a utility to create and send raw packets across the network. You can write simple scripts that construct and send pre-determined packets to your router, which might allow you to construct reproducible tests.

The following BASH script uses dumbnet to create and send some packets. You can run this script and monitor the connection in wireshark to see that these ARP and ECHO packets get a correct response from the network. Be sure the update the destination IP and MAC address in the examples to reflect your group's network, and make sure that the script is executable (chmod +x <script name>.sh).

#!/bin/bash

# Dumbnet / dnet documentation:
#  http://libdnet.sourceforge.net/dnet.8.txt
#  http://www.networksorcery.com/

# For debugging:
# set -x

#######################################
# Environment setup
#######################################

# Network device to transmit packets on 
# (For example, this is the local loopback NIC on the NetFPGA machine
# that connects into the hub/switch/router)
ETH_DEV="eth1"

# MAC address of this network device
ETH_SRC=`ifconfig $ETH_DEV | awk '/eth/ {print $5}'`

# IP address of this network device
IP_SRC=`ifconfig $ETH_DEV | grep "inet addr" | awk -F : ' {print $2}' | awk ' {print $1}'`

# MAC Address of broadcast "device
ETH_BCAST="FF:FF:FF:FF:FF:FF"

#echo $ETH_DEV
#echo $ETH_SRC
#echo $IP_SRC

#######################################
# Examples
#######################################
#
# Notes:
#  sha = Source Hardware (MAC) Address
#  spa = Source Protocol (IP) Address
#  tha = Target Hardware (MAC) Address
#  tpa = Target Protocol (IP) Address

# Notes:
#  * Only the final command (dumbnet send) should be called with op
#  * You can use Wireshark to monitor packets you transmit. 
#      (It will warn you about "Malformed Packets" if there is an error)


# Example 1:
#  Send an ARP request from this machine asking about 192.168.1.66
#  First we create the ARP section, and wrap it in an Ethernet frame
dumbnet arp op req sha $ETH_SRC spa $IP_SRC tpa 192.168.1.66 | \
dumbnet eth type arp src $ETH_SRC dst $ETH_BCAST | \
op dumbnet send $ETH_DEV


# Example 2:
#  Send an ICMP echo request to 192.168.1.66
#  First we create the ICMP request (with a made-up string comprising 
#  the identifier/sequence#/data portion of the ICMP request), then we
#  wrap it in an IP header and an Ethernet header
#  Note that we had to provide the MAC address corresponding to the destination IP.
echo "echo-packet-data-filling-space" | \
dumbnet icmp type 8 code 0 | \
dumbnet ip proto icmp src $IP_SRC dst 192.168.1.66 | \
dumbnet eth type ip src $ETH_SRC dst 00:25:90:30:11:65 | \
op dumbnet send $ETH_DEV

Adding Static ARP Entries

The hardware router will be unable to respond to or initiate ARP requests before it is integrated with the software. Thus, to test the router with applications such as ping, you will need to add static ARP entries to ensure that ARP packets are not sent. Refer to the Network Topology page to determine the exact entries needed for your group. Each group will need to perform the following tasks:

• On netfpga01 machine: Add a static ARP entry defining IP address of (router port 3) = MAC address (of router port 3)
• On server3 machine: Add a static ARP entry defining IP address (of router port 2) = MAC address (of router port 2)
• On server2 machine: Add a static ARP entry defining IP address (of router port 1) = MAC address (of router port 1)
• On server1 machine: Add a static ARP entry defining IP address of (router port 0) = MAC address (of router port 0)
• On your hardware router: Initialize the hardware ARP table with the IP and MAC addresses of netfpga0x, server1, server2, and server3. (Type ifconfig on those machines to find the MAC address of the desired interfaces.)

The following are examples of the ARP commands that should be used:

Show current entries in ARP cache

arp

Remove ARP entry

op arp -d 10.143.201.1

Add a new (static) ARP entry

op arp -s 10.143.201.1 00:00:00:00:06:0x

Note: You must choose MAC addresses that do not conflict with other groups! We recommend the following scheme:

00:00:00:0g:0i where g=Group number and i=interface number.  So, for group 1:
00:00:00:01:00 (for router eth0)
00:00:00:01:01 (for router eth1)
00:00:00:01:02 (for router eth2)
00:00:00:01:03 (for router eth3)

Note: You can add or remove entries for other groups with these commands. Please be careful to only edit MAC and IP addresses assigned to your group!

Compile Specified Verilog Files

Go to the synth directory and open nf2_top.prj. Remove all the entries listed and add "verilog work <filepath>" for each file you want to try compiling. For example:

verilog work ../src/output_port_lookup/<source_file1>.v
verilog work ../src/output_port_lookup/<source_file2>.v

Next type the following in the synth directory:

xst -intstyle ise -ifn nf2_top.scr

The synthesis tool will complain that no top module was found, but you can ignore this since we're just testing compilation.