P4 Developer Day 2018 Spring (#159)

* Repository reorganization for 2018 Spring P4 Developer Day.

* Port tutorial exercises to P4Runtime with static controller (#156)

* Switch VM to a minimal Ubuntu 16.04 desktop image

* Add commands to install Protobuf Python bindings to user_bootstrap.sh

* Implement P4Runtime static controller for use in exercises

From the exercise perspective, the main difference is that control plane
rules are now specified using JSON files instead of CLI commands. Such
JSON files define rules that use the same name for tables, keys, etc. as
in the P4Info file.

All P4Runtime requests generated as part of the make run process are
logged in the exercise's “logs” directory, making it easier for students
to see the actual P4Runtime messages sent to the switch.

Only the "basic" exercise has been ported to use P4Runtime.
The "p4runtime" exercise has been updated to work with P4Runtime
protocol changes.

Known issues:
- make run hangs in case of errors when running the P4Runtime controller
    (probably due to gRPC stream channel threads not terminated properly)
- missing support for inserting table entries with default action
    (can specify in P4 program as a workaround)

* Force install protobuf python module

* Fixing Ctrl-C hang by shutdown switches

* Moving gRPC error print to function for readability

Unforuntately, if this gets moved out of the file, the process hangs.
We'll need to figure out how why later.

* Renaming ShutdownAllSwitches -> ShutdownAllSwitchConnections

* Reverting counter index change

* Porting the ECN exercise to use P4 Runtime Static Controller

* updating the README in the ecn exercise to reflect the change in rule files

* Allow set table default action in P4Runtime static controller

* Fixed undefined match string when printing P4Runtime table entry

* Updated basic_tunnel exercise to use P4Runtime controller.

* Changed default action in the basic exercise's ipv4_lpm table to drop

* Porting the MRI exercise to use P4runtime with static controller

* Updating readme to reflect the change of controller for mri

* Update calc exercise for P4Runtime static controller

* Port source_routing to P4 Runtime static controller (#157)

* Port Load Balance to P4 Runtime Static Controller (#158)

exercises/source_routing/Makefile

@@ -0,0 +1,5 @@
+BMV2_SWITCH_EXE = simple_switch_grpc
+NO_P4 = true
+P4C_ARGS = --p4runtime-file $(basename $@).p4info --p4runtime-format text
+
+include ../../utils/Makefile

exercises/source_routing/README.md

@@ -0,0 +1,147 @@
+# Implementing Source Routing
+
+## Introduction
+
+The objective of this exercise is to implement source routing.  With
+source routing, the source host guides each switch in the network to
+send the packet to a specific port. The host puts a stack of output
+ports in the packet. In this example, we just put the stack after
+Ethernet header and select a special etherType to indicate that.  Each
+switch pops an item from the stack and forwards the packet according
+to the specified port number.
+
+Your switch must parse the source routing stack. Each item has a bos
+(bottom of stack) bit and a port number. The bos bit is 1 only for the
+last entry of stack.  Then at ingress, it should pop an entry from the
+stack and set the egress port accordingly.  Note that the last hop can
+also revert back the etherType to `TYPE_IPV4`.
+
+> **Spoiler alert:** There is a reference solution in the `solution`
+> sub-directory. Feel free to compare your implementation to the
+> reference.
+
+## Step 1: Run the (incomplete) starter code
+
+The directory with this README also contains a skeleton P4 program,
+`source_routing.p4`, which initially drops all packets. Your job (in
+the next step) will be to extend it to properly to route packets.
+
+Before that, let's compile the incomplete `source_routing.p4` and
+bring up a network in Mininet to test its behavior.
+
+1. In your shell, run:
+   ```bash
+   make
+   ```
+   This will:
+   * compile `source_routing.p4`, and
+   * start a Mininet instance with three switches (`s1`, `s2`, `s3`) configured
+     in a triangle, each connected to one host (`h1`, `h2`, `h3`).
+     Check the network topology using the `net` command in mininet.
+     You can also change the topology in topology.json
+   * The hosts are assigned IPs of `10.0.1.1`, `10.0.2.2`, etc
+     (`10.0.<Switchid>.<hostID>`).
+
+2. You should now see a Mininet command prompt. Open two terminals for
+   `h1` and `h2`, respectively:
+   ```bash
+   mininet> xterm h1 h2
+   ```
+3. Each host includes a small Python-based messaging client and
+   server. In `h2`'s xterm, start the server:
+   ```bash
+   ./receive.py
+   ```
+4. In `h1`'s xterm, send a message from the client:
+   ```bash
+   ./send.py 10.0.2.2
+   ```
+
+5. Type a list of port numbers. say `2 3 2 2 1`.  This should send the
+   packet through `h1`, `s1`, `s2`, `s3`, `s1`, `s2`, and
+   `h2`. However, `h2` will not receive the message.
+   
+6. Type `q` to exit send.py and type `exit` to leave each xterm and
+   the Mininet command line.
+
+The message was not received because each switch is programmed with
+`source_routing.p4`, which drops all packets on arrival.  You can
+verify this by looking at `/tmp/p4s.s1.log`.  Your job is to extend
+the P4 code so packets are delivered to their destination.
+
+## Step 2: Implement source routing
+
+The `source_routing.p4` file contains a skeleton P4 program with key
+pieces of logic replaced by `TODO` comments.  These should guide your
+implementation---replace each `TODO` with logic implementing the
+missing piece.
+
+A complete `source_routing.p4` will contain the following components:
+
+1. Header type definitions for Ethernet (`ethernet_t`) and IPv4
+   (`ipv4_t`) and Source Route (`srcRoute_t`).
+2. **TODO:** Parsers for Ethernet and Source Route that populate
+   `ethernet` and `srcRoutes` fields.
+3. An action to drop a packet, using `mark_to_drop()`.
+4. **TODO:** An action (called `srcRoute_nhop`), which will:
+	1. Set the egress port for the next hop. 
+	2. remove the first entry of srcRoutes
+5. A control with an `apply` block that:
+    1. checks the existence of source routes.
+    2. **TODO:** if statement to change etherent.etherType if it is the last hop
+    3. **TODO:** call srcRoute_nhop action
+6. A deparser that selects the order in which fields inserted into the outgoing
+   packet.
+7. A `package` instantiation supplied with the parser, control, and deparser.
+    > In general, a package also requires instances of checksum verification
+    > and recomputation controls.  These are not necessary for this tutorial
+    > and are replaced with instantiations of empty controls.
+
+## Step 3: Run your solution
+
+Follow the instructions from Step 1. This time, your message from `h1`
+should be delivered to `h2`.
+
+Check the `ttl` of the IP header. Each hop decrements `ttl`.  The port
+sequence `2 3 2 2 1`, forces the packet to have a loop, so the `ttl`
+should be 59 at `h2`.  Can you find the port sequence for the shortest
+path?
+
+### Food for thought
+* Can we change the program to handle both IPv4 forwarding and source
+routing at the same time?
+* How would you enhance your program to let the first switch add the
+path, so that source routing would be transparent to end-hosts?
+
+### Troubleshooting
+
+There are several ways that problems might manifest:
+
+1. `source_routing.p4` fails to compile. In this case, `make` will
+   report the error emitted from the compiler and stop.
+2. `source_routing.p4` compiles but switches or mininet do not start.
+   Do you have another instance of mininet running? Did the previous
+   run of mininet crash?  if yes, check "Cleaning up Mininet" bellow.
+3. `source_routing.p4` compiles but the switch does not process
+   packets in the desired way. The `/tmp/p4s.<switch-name>.log`
+   files contain trace messages describing how each switch processes
+   each packet. The output is detailed and can help pinpoint logic
+   errors in your implementation.  The
+   `<switch-name>-<interface-name>_<direction>.pcap` files contain pcap captures
+   of all packets sent and received on each interface. Use `tcpdump -r <filename> -xxx` to
+   print the hexdump of the packets.
+
+#### Cleaning up Mininet
+
+In the cases above, `make` may leave a Mininet instance running in
+the background.  Use the following command to clean up these
+instances:
+
+```bash
+mn -c
+```
+
+## Next Steps
+
+Congratulations, your implementation works!  Move on to
+[Calculator](../calc).

exercises/source_routing/receive.py

@@ -0,0 +1,59 @@
+#!/usr/bin/env python
+import sys
+import struct
+
+from scapy.all import sniff, sendp, hexdump, get_if_list, get_if_hwaddr, bind_layers
+from scapy.all import Packet, IPOption
+from scapy.all import IP, UDP, Raw, Ether
+from scapy.layers.inet import _IPOption_HDR
+from scapy.fields import *
+
+def get_if():
+    ifs=get_if_list()
+    iface=None
+    for i in get_if_list():
+        if "eth0" in i:
+            iface=i
+            break;
+    if not iface:
+        print "Cannot find eth0 interface"
+        exit(1)
+    return iface
+
+class IPOption_MRI(IPOption):
+    name = "MRI"
+    option = 31
+    fields_desc = [ _IPOption_HDR,
+                    FieldLenField("length", None, fmt="B",
+                                  length_of="swids",
+                                  adjust=lambda pkt,l:l+4),
+                    ShortField("count", 0),
+                    FieldListField("swids",
+                                   [],
+                                   IntField("", 0),
+                                   length_from=lambda pkt:pkt.count*4) ]
+def handle_pkt(pkt):
+    print "got a packet"
+    pkt.show2()
+#    hexdump(pkt)
+    sys.stdout.flush()
+
+class SourceRoute(Packet):
+   fields_desc = [ BitField("bos", 0, 1),
+                   BitField("port", 0, 15)]
+class SourceRoutingTail(Packet):
+   fields_desc = [ XShortField("etherType", 0x800)]
+
+bind_layers(Ether, SourceRoute, type=0x1234)
+bind_layers(SourceRoute, SourceRoute, bos=0)
+bind_layers(SourceRoute, SourceRoutingTail, bos=1)
+
+def main():
+    iface = 'h2-eth0'
+    print "sniffing on %s" % iface
+    sys.stdout.flush()
+    sniff(filter="udp and port 4321", iface = iface,
+          prn = lambda x: handle_pkt(x))
+
+if __name__ == '__main__':
+    main()

exercises/source_routing/s1-runtime.json

@@ -0,0 +1,6 @@
+{
+  "target": "bmv2",
+  "p4info": "build/source_routing.p4info",
+  "bmv2_json": "build/source_routing.json",
+  "table_entries": [ ]
+} 

exercises/source_routing/s2-runtime.json

@@ -0,0 +1,6 @@
+{
+  "target": "bmv2",
+  "p4info": "build/source_routing.p4info",
+  "bmv2_json": "build/source_routing.json",
+  "table_entries": [ ]
+} 

exercises/source_routing/s3-runtime.json

@@ -0,0 +1,6 @@
+{
+  "target": "bmv2",
+  "p4info": "build/source_routing.p4info",
+  "bmv2_json": "build/source_routing.json",
+  "table_entries": [ ]
+} 

exercises/source_routing/send.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python
+import argparse
+import sys
+import socket
+import random
+import struct
+
+from scapy.all import sendp, send, get_if_list, get_if_hwaddr, bind_layers
+from scapy.all import Packet
+from scapy.all import Ether, IP, UDP
+from scapy.fields import *
+import readline
+
+def get_if():
+    ifs=get_if_list()
+    iface=None # "h1-eth0"
+    for i in get_if_list():
+        if "eth0" in i:
+            iface=i
+            break;
+    if not iface:
+        print "Cannot find eth0 interface"
+        exit(1)
+    return iface
+
+class SourceRoute(Packet):
+   fields_desc = [ BitField("bos", 0, 1),
+                   BitField("port", 0, 15)]
+
+bind_layers(Ether, SourceRoute, type=0x1234)
+bind_layers(SourceRoute, SourceRoute, bos=0)
+bind_layers(SourceRoute, IP, bos=1)
+
+def main():
+
+    if len(sys.argv)<2:
+        print 'pass 2 arguments: <destination>'
+        exit(1)
+
+    addr = socket.gethostbyname(sys.argv[1])
+    iface = get_if()
+    print "sending on interface %s to %s" % (iface, str(addr))
+
+    while True:
+        print
+        s = str(raw_input('Type space separated port nums '
+                          '(example: "2 3 2 2 1") or "q" to quit: '))
+        if s == "q":
+            break;
+        print
+
+        i = 0
+        pkt =  Ether(src=get_if_hwaddr(iface), dst='ff:ff:ff:ff:ff:ff');
+        for p in s.split(" "):
+            try:
+                pkt = pkt / SourceRoute(bos=0, port=int(p))
+                i = i+1
+            except ValueError:
+                pass
+        if pkt.haslayer(SourceRoute):
+            pkt.getlayer(SourceRoute, i).bos = 1
+
+        pkt = pkt / IP(dst=addr) / UDP(dport=4321, sport=1234)
+        pkt.show2()
+        sendp(pkt, iface=iface, verbose=False)
+
+    #pkt = pkt / SourceRoute(bos=0, port=2) / SourceRoute(bos=0, port=3);
+    #pkt = pkt / SourceRoute(bos=0, port=2) / SourceRoute(bos=0, port=2);
+    #pkt = pkt / SourceRoute(bos=1, port=1)
+
+
+if __name__ == '__main__':
+    main()

exercises/source_routing/solution/source_routing.p4

@@ -0,0 +1,181 @@
+/* -*- P4_16 -*- */
+#include <core.p4>
+#include <v1model.p4>
+
+const bit<16> TYPE_IPV4 = 0x800;
+const bit<16> TYPE_SRCROUTING = 0x1234;
+
+#define MAX_HOPS 9
+
+/*************************************************************************
+*********************** H E A D E R S  ***********************************
+*************************************************************************/
+
+typedef bit<9>  egressSpec_t;
+typedef bit<48> macAddr_t;
+typedef bit<32> ip4Addr_t;
+
+header ethernet_t {
+    macAddr_t dstAddr;
+    macAddr_t srcAddr;
+    bit<16>   etherType;
+}
+
+header srcRoute_t {
+    bit<1>    bos;
+    bit<15>   port;
+}
+
+header ipv4_t {
+    bit<4>    version;
+    bit<4>    ihl;
+    bit<8>    diffserv;
+    bit<16>   totalLen;
+    bit<16>   identification;
+    bit<3>    flags;
+    bit<13>   fragOffset;
+    bit<8>    ttl;
+    bit<8>    protocol;
+    bit<16>   hdrChecksum;
+    ip4Addr_t srcAddr;
+    ip4Addr_t dstAddr;
+}
+
+struct metadata {
+    /* empty */
+}
+
+struct headers {
+    ethernet_t              ethernet;
+    srcRoute_t[MAX_HOPS]    srcRoutes;
+    ipv4_t                  ipv4;
+}
+
+/*************************************************************************
+*********************** P A R S E R  ***********************************
+*************************************************************************/
+
+parser MyParser(packet_in packet,
+                out headers hdr,
+                inout metadata meta,
+                inout standard_metadata_t standard_metadata) {
+    
+    state start {
+        transition parse_ethernet;
+    }
+
+    state parse_ethernet {
+        packet.extract(hdr.ethernet);
+        transition select(hdr.ethernet.etherType) {
+            TYPE_SRCROUTING: parse_srcRouting;
+            default: accept;
+        }
+    }
+
+    state parse_srcRouting {
+        packet.extract(hdr.srcRoutes.next);
+        transition select(hdr.srcRoutes.last.bos) {
+            1: parse_ipv4;
+            default: parse_srcRouting;
+        }
+    }
+
+    state parse_ipv4 {
+        packet.extract(hdr.ipv4);
+        transition accept;
+    }
+
+}
+
+
+/*************************************************************************
+************   C H E C K S U M    V E R I F I C A T I O N   *************
+*************************************************************************/
+
+control MyVerifyChecksum(inout headers hdr, inout metadata meta) {   
+    apply {  }
+}
+
+
+/*************************************************************************
+**************  I N G R E S S   P R O C E S S I N G   *******************
+*************************************************************************/
+
+control MyIngress(inout headers hdr,
+                  inout metadata meta,
+                  inout standard_metadata_t standard_metadata) {
+
+    action drop() {
+        mark_to_drop();
+    }
+    
+    action srcRoute_nhop() {
+        standard_metadata.egress_spec = (bit<9>)hdr.srcRoutes[0].port;
+        hdr.srcRoutes.pop_front(1);
+    }
+
+    action srcRoute_finish() {
+        hdr.ethernet.etherType = TYPE_IPV4;
+    }
+
+    action update_ttl(){
+        hdr.ipv4.ttl = hdr.ipv4.ttl - 1;
+    }
+    
+    apply {
+        if (hdr.srcRoutes[0].isValid()){
+            if (hdr.srcRoutes[0].bos == 1){
+                srcRoute_finish();
+            }
+            srcRoute_nhop();
+            if (hdr.ipv4.isValid()){
+                update_ttl();
+            }
+        }else{
+            drop();
+        } 
+    }
+}
+
+/*************************************************************************
+****************  E G R E S S   P R O C E S S I N G   *******************
+*************************************************************************/
+
+control MyEgress(inout headers hdr,
+                 inout metadata meta,
+                 inout standard_metadata_t standard_metadata) {
+    apply {  }
+}
+
+/*************************************************************************
+*************   C H E C K S U M    C O M P U T A T I O N   **************
+*************************************************************************/
+
+control MyComputeChecksum(inout headers  hdr, inout metadata meta) {
+    apply {  }
+}
+
+/*************************************************************************
+***********************  D E P A R S E R  *******************************
+*************************************************************************/
+
+control MyDeparser(packet_out packet, in headers hdr) {
+    apply {
+        packet.emit(hdr.ethernet);
+        packet.emit(hdr.srcRoutes);
+        packet.emit(hdr.ipv4);
+    }
+}
+
+/*************************************************************************
+***********************  S W I T C H  *******************************
+*************************************************************************/
+
+V1Switch(
+MyParser(),
+MyVerifyChecksum(),
+MyIngress(),
+MyEgress(),
+MyComputeChecksum(),
+MyDeparser()
+) main;

exercises/source_routing/source_routing.p4

@@ -0,0 +1,191 @@
+/* -*- P4_16 -*- */
+#include <core.p4>
+#include <v1model.p4>
+
+const bit<16> TYPE_IPV4 = 0x800;
+const bit<16> TYPE_SRCROUTING = 0x1234;
+
+#define MAX_HOPS 9
+
+/*************************************************************************
+*********************** H E A D E R S  ***********************************
+*************************************************************************/
+
+typedef bit<9>  egressSpec_t;
+typedef bit<48> macAddr_t;
+typedef bit<32> ip4Addr_t;
+
+header ethernet_t {
+    macAddr_t dstAddr;
+    macAddr_t srcAddr;
+    bit<16>   etherType;
+}
+
+header srcRoute_t {
+    bit<1>    bos;
+    bit<15>   port;
+}
+
+header ipv4_t {
+    bit<4>    version;
+    bit<4>    ihl;
+    bit<8>    diffserv;
+    bit<16>   totalLen;
+    bit<16>   identification;
+    bit<3>    flags;
+    bit<13>   fragOffset;
+    bit<8>    ttl;
+    bit<8>    protocol;
+    bit<16>   hdrChecksum;
+    ip4Addr_t srcAddr;
+    ip4Addr_t dstAddr;
+}
+
+struct metadata {
+    /* empty */
+}
+
+struct headers {
+    ethernet_t              ethernet;
+    srcRoute_t[MAX_HOPS]    srcRoutes;
+    ipv4_t                  ipv4;
+}
+
+/*************************************************************************
+*********************** P A R S E R  ***********************************
+*************************************************************************/
+
+parser MyParser(packet_in packet,
+                out headers hdr,
+                inout metadata meta,
+                inout standard_metadata_t standard_metadata) {
+
+    
+    state start {
+        transition parse_ethernet;
+    }
+
+    state parse_ethernet {
+        packet.extract(hdr.ethernet);
+        /*
+         * TODO: Modify the next line to select on hdr.ethernet.etherType
+         * If the value is TYPE_SRCROUTING transition to parse_srcRouting
+         * otherwise transition to accept.
+         */
+        transition accept;
+    }
+
+    state parse_srcRouting {
+        /*
+         * TODO: extract the next entry of hdr.srcRoutes
+         * while hdr.srcRoutes.last.bos is 0 transition to this state
+         * otherwise parse ipv4
+         */
+        transition accept;
+    }
+
+    state parse_ipv4 {
+        packet.extract(hdr.ipv4);
+        transition accept;
+    }
+
+}
+
+
+/*************************************************************************
+************   C H E C K S U M    V E R I F I C A T I O N   *************
+*************************************************************************/
+
+control MyVerifyChecksum(inout headers hdr, inout metadata meta) {   
+    apply {  }
+}
+
+
+/*************************************************************************
+**************  I N G R E S S   P R O C E S S I N G   *******************
+*************************************************************************/
+
+control MyIngress(inout headers hdr,
+                  inout metadata meta,
+                  inout standard_metadata_t standard_metadata) {
+
+    action drop() {
+        mark_to_drop();
+    }
+    
+    action srcRoute_nhop() {
+        /*
+         * TODO: set standard_metadata.egress_spec 
+         * to the port in hdr.srcRoutes[0] and
+         * pop an entry from hdr.srcRoutes
+         */
+    }
+
+    action srcRoute_finish() {
+        hdr.ethernet.etherType = TYPE_IPV4;
+    }
+
+    action update_ttl(){
+        hdr.ipv4.ttl = hdr.ipv4.ttl - 1;
+    }
+    
+    apply {
+        if (hdr.srcRoutes[0].isValid()){
+            /*
+             * TODO: add logic to:
+             * - If final srcRoutes (top of stack has bos==1):
+             *   - change etherType to IP
+             * - choose next hop and remove top of srcRoutes stack
+             */
+
+            if (hdr.ipv4.isValid()){
+                update_ttl();
+            }
+        }else{
+            drop();
+        } 
+    }
+}
+
+/*************************************************************************
+****************  E G R E S S   P R O C E S S I N G   *******************
+*************************************************************************/
+
+control MyEgress(inout headers hdr,
+                 inout metadata meta,
+                 inout standard_metadata_t standard_metadata) {
+    apply {  }
+}
+
+/*************************************************************************
+*************   C H E C K S U M    C O M P U T A T I O N   **************
+*************************************************************************/
+
+control MyComputeChecksum(inout headers hdr, inout metadata meta) {
+    apply {  }
+}
+
+/*************************************************************************
+***********************  D E P A R S E R  *******************************
+*************************************************************************/
+
+control MyDeparser(packet_out packet, in headers hdr) {
+    apply {
+        packet.emit(hdr.ethernet);
+        packet.emit(hdr.srcRoutes);
+        packet.emit(hdr.ipv4);
+    }
+}
+
+/*************************************************************************
+***********************  S W I T C H  *******************************
+*************************************************************************/
+
+V1Switch(
+MyParser(),
+MyVerifyChecksum(),
+MyIngress(),
+MyEgress(),
+MyComputeChecksum(),
+MyDeparser()
+) main;

exercises/source_routing/topology.json

@@ -0,0 +1,16 @@
+{
+    "hosts": [
+        "h1",
+        "h2",
+        "h3"
+    ],
+    "switches": {
+        "s1": { "runtime_json" : "s1-runtime.json" },
+        "s2": { "runtime_json" : "s2-runtime.json" },
+        "s3": { "runtime_json" : "s3-runtime.json" }
+    },
+    "links": [
+        ["h1", "s1"], ["s1", "s2"], ["s1", "s3"], 
+        ["s3", "s2"], ["s2", "h2"], ["s3", "h3"]
+    ]
+}