Raw Packet Sockets

The Hardware/Software Interface

The NetFPGA device driver presents the cpu queues as normal network interfaces, allowing applications to access NetFPGA. Packets are transferred between NetFPGA and the software using this network interface abstraction. In Linux, a packet socket allows the transfer of a complete packet (including headers) between the application and the network interface. This is a convenient mechanism for transferring complete packets between NetFPGA and the software.

Creating a packet socket

A packet socket can be created in the following way:

int s;
s = socket(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL));

The unix socket command creates an endpoint for a communication channel. In this case, we are creating a raw (SOCK_RAW), packet (PF_PACKET) channel. As previously stated, a packet socket allows the transfer of an entire packet, including headers, directly from the application. This is in contrast to "normal" sockets in which the application only sends or receives payload data and the operating system adds or strips the headers. There are two types of packet sockets, raw and dgram. A raw packet socket will include Ethernet headers in all packets. The operating system will add and remove Ethernet headers in a dgram packet socket. The final argument to socket indicates the protocols that should be used on the socket. Here, (ETH_P_ALL) indicates that all packets (regardless of protocol) should be used. See man packet for further details.

Once created, the socket must actually be associated with the appropriate "network interface". In this case, you will want to associate the socket with one of the NetFPGA cpu queues. These queues are mapped to the network interface names "nf2c0", "nf2c1", "nf2c2", and "nf2c3". This must be done with a Linux ioctl call to associate the socket. Note that this is more complicated than creating a normal network socket. This is because normally, you just want to connect to another computer on the network. In that case, you do not care which network interface you use. Here we want to make sure the socket uses a particular network interface.

The Linux socket ioctl calls use a ifreq structure to indicate the socket you are referring to (see man netdevice):

struct ifreq {
  char ifr_name[IFNAMSIZ]; /* Interface name */
  union {
    struct sockaddr ifr_addr;
    struct sockaddr ifr_dstaddr;
    struct sockaddr ifr_broadaddr;
    struct sockaddr ifr_netmask;
    struct sockaddr ifr_hwaddr;
    short           ifr_flags;
    int             ifr_ifindex;
    int             ifr_metric;
    int             ifr_mtu;
    struct ifmap    ifr_map;
    char            ifr_slave[IFNAMSIZ];
    char            ifr_newname[IFNAMSIZ];
    char *          ifr_data;
  };
};

All we care about is ifr_name and ifr_ifindex. To find the ifr_ifindex of a particular network interface (in this case "nf2c0"), you can do the following:

struct ifreq ifr;
bzero(&ifr, sizeof(struct ifreq));
strncpy(ifr.ifr_name, "nf2c0", IFNAMSIZ);

if (ioctl(s, SIOCGIFINDEX, &ifr) < 0) {
  perror("ioctl SIOCGIFINDEX");
  exit(1);
}

If the ioctl is successfully, ifr.ifr_ifindex will then be the index of the network interface "nf2c0". We then need to bind the socket to this interface:

struct sockaddr_ll saddr;
bzero(&saddr, sizeof(struct sockaddr_ll));
saddr.sll_family = AF_PACKET;
saddr.sll_protocol = htons(ETH_P_ALL);
saddr.sll_ifindex = ifr.ifr_ifindex;

if (bind(s, (struct sockaddr *)(&saddr), sizeof(saddr)) < 0) {
  perror("bind error");
  exit(1);
}

If bind is successful, the socket s will then be ready to send and receive raw packets to/from NetFGPA.

Receiving packets

Once the packet socket is properly created and bound to the appropriate network interface, you can read from it like any other socket. In order for your router to function correctly, you must continually read from the four packet sockets corresponding to the four cpu queues on NetFPGA. This effectively replaces VNS in your router. This is the job of sr_cpu_input, which will get called by the main thread when using NetFPGA instead of VNS.

In order to read a packet from a packet socket, you simply read from it like any other socket:

bytes_read = read(s, buf, len);

In the above call to read, s must be a properly created and bound socket, buf must be a buffer (an array of unsigned char) that is large enough to hold the largest packet you might receive, and len is the length of buf. The call to read will return the number of bytes that were actually read from the socket. As long as your buffer is large enough, you may assume that each call to read returns one complete packet. Once you have received a packet, you should log it (sr_log_packet) and send it to your software (sr_integ_input).

Note that the read call is a blocking system call. This means that it will not return until data is available. Your sr_cpu_input function, however, must be able to read from all four sockets. You do not know which socket will receive a packet next, so you must concurrently handle all four sockets. One way to do this is by using select.

Sending packets

Once the packet socket is properly created and bound to the appropriate network interface, you can also write to it like any other socket. When using NetFPGA instead of VNS, sr_integ_low_level_output will callsr_cpu_output instead of a function to inject the packet into VNS. Once you have properly created and bound the packet sockets to the NetFPGA interfaces, your sr_cpu_output function can send packets directly to NetFPGA.

In order to write a packet to a packet socket, you simply write to it like any other socket:

bytes_written = write(s, buf, len);

In the above call to write, s must be a properly created and bound socket, buf must be a buffer (an array of unsigned char) that holds a properly formatted packet (including all headers), and len is the length of the packet. Note that write may return before the entire packet is written. You must check the return value from the call to write (bytes_written in the code above) and compare it to len to determine if the entire packet was sent. If not, you must adjust buf and len appropriately and call write again. Waiting for the call to write to complete will not affect the functionality of your router (think about why), so you do not need to worry about using select or other means for concurrently handling sockets when sending packets.

You should log all packets (sr_log_packet) that you send to NetFPGA.