Monday 1 August 2016

SDNCon 2017

SDNCon was a hands-on, network engineering meeting took place 7th and 8th July at Victoria University of Wellington (New Zealand). The organizers would like to thank Allied Telesis for their help with sponsoring the costs of the hackathon, NoviFlow for organising an evening networking event and REANNZ for help with organisation and publicity. We also thank Josh Bailey and Dean Pemberton for contributions in the form of advice on running the event and updates to the Faucet code base to allow us to make the most of it.

Where and when.

We used two University seminar rooms. The rooms were open from 9am until whenever teams wanted to finish on the first day and from 9am until 2pm on the second day. Demonstrations and judging ran from 2-5pm followed by pizza onsite.

Recruitment.

We advertised the event to previous attenders of SDNCon 2014 and other SDN workshops organized by VUW as well as through InternetNZ, REANNZ and the Wellington SDN Meetup Group (52 members). The event was attended by 20 people from Victoria University, Waikato University, University of New South Wales, Whitireia Polytechnic, Christchurch Polytechnic Institute of Technology (CPIT) and REANNZ as well as individuals who are members of the meetup group.

Some participants had never actually programmed using Faucet or Ryu whereas others had helped develop Faucet or came with a lot of prior experience with SDN/OpenFlow.

Approximately 10% of the participants were female.

Hackathon brief.

We asked that the projects be primarily related to ONF’s Faucet Controller application but could include the use of other controllers. We also provided a list of potential projects related to either using Faucet to solve a real world problem or to add new features to Faucet. Recognising different levels of preparation we also said that relative newcomers would be welcome but support that we could offer would be limited.

Preparation.

The entry bar for attendance was that all participants had worked through a set of exercises to get them up to speed with using Faucet on mininet (https://github.com/gwacter/sdn-workshop).

Testbed.

At the event we provided a testbed built out of Raspberry PIs and a Zodiac FX switch. The PIs had traffic generation tools installed on the end hosts and Faucet installed on host acting as a controller. We also had an AT X510 switch available and in addition REANNZ brought along an Acton switch and two AT X510 switches. We also provided laptops as well as the PIs for teams to use for their development and demonstrations.

We had Internet connectivity via the University wireless guest network and the Victoria Engineering Club also provided us with Internet access via their own switch that could support both wired and wireless access.

What each team did.

There were four teams:

1) Active network defence using Faucet and ONOS (UNSW).

The UNSW team (Himal, Jason, Vijay and Winston) wanted to implement a proof-of-concept active network defence scenario involving multiple controllers and a scenario with a transit provider (ONOS controller) and enterprise network (Faucet controller at the gateway). Faucet provided flow statistics, these were used to construct a statistical machine learning model capable of classifying flows as benign or malicious. Faucet was extended to send a control signal upstream to the ONOS controller to trigger the dropping of malicious traffic targeting the enterprise network. Traffic generators were used to create benign traffic and a denial-of-service tool used to launch the attack. The was successfully demonstrated using a testbed made up of two AT X510-28GTX switches and laptops

2) Multi switch support, topology discovery and broadcast suppression in Faucet (VUW).

The VUW team (Callum, David, Jarrod, Liang, Radek and Trung) wanted to address issue #112 - implementing support for configuring multi-datapaths. This had also been identified as a real-world feature that Whitireia would like to see implemented so Faucet could manage multiple switches in their labs.

The team successfully implemented multi-datapath support and topology discovery for all links within a given VLAN. This involved changes to Faucet configuration management and internal data structures as well as changes so it could work with the Ryu topology viewer application.

They demonstrated their project working on a testbed built from Raspberry PIs and Zodaic FX switches. They were able to show connectivity and that the topology viewer responded to switches being removed from the network.

A problem with multiple switches is broadcast storms. They also partially implemented a separate Ryu application with the aim of suppressing these by computing a spanning tree centrally for the network managed by Faucet.

Post-hackathon, the team is going working on the completion of broadcast suppression, complete unit tests for the changes and to agree with other Faucet contributors on the a backwards compatible format for the Faucet configuration file.

3) Implementing Faucet using OFDPA (CyberNet: Waikato WAND group, REANNZ and Michael F).

The Cyber Networking team (Brad, Chris, Michael and Richard) took on the project this year of writing an OFDPA pipeline for Faucet to see if it could be easily done. The idea was that if they could use the bridging table for Faucet that we should be able to learn many more hosts than if just ACL table was used. Using OFDPA is hard so this was a challenging project!

They made some minor tweaks to the Faucet OFDPA pipeline to properly support how OFDPA MAC learning works, but it worked well. At the end of the hackathon they got working:

- Tagged, Untagged VLAN separation
- MAC learning (with aging)

For the demo, the team connected their Accton AS5710-54X running Open Networking Linux with OFDPA to an Allied Telesis x510 running regular Faucet with a VLAN trunk between the two switches. We had half our machines on one switch, half on the other switch to show that worked. Iperfs between both halves of the network show we can hit line rate.

There are a few issues when doing MAC relearning, but we suspect that's fixable. They intend to implement some of the more advanced features of Faucet such as ACLs at a later stage.

4) Enterprise network (Whitireia, Christchurch Polytechnic Institute of Technology).

This team focused on how Faucet and more generally SDN/OpenFlow could be used in either teaching or in their own networks. Dave from CPIT successfully got a Ryu-based implementation of NAT and L2switching working on the Zodiac FX testbed. The team from Whitireia (Lisa, Ryan, Scott, Sue and Steve) implement scripts to build a virtual copy of their network in mininet and also used the Zodiac FX testbed to explore Faucet features that they would be using in an actual deployment.

Hackathon outputs.

Code produced by the teams:

1) Active network defence using Faucet and ONOS (UNSW).

Currently being cleaned up and we will update this report with a link once available.

2) Multi switch support, topology discovery and broadcast suppression in Faucet (VUW).

Code from the hackathon is available from https://github.com/gwacter/faucet under the subbranches multiple-dps (enhancements to Faucet), untag-llpd (Ryu’s topology discovery was being broken by Faucet mistakenly untag LLDP packets) and faucet-stp (spanning tree computation and enforcement).

3) Implementing Faucet using OFDPA (CyberNet: Waikato WAND group, REANNZ and Michael F).

Code from the hackathon is available here: https://github.com/wandsdn/faucet/tree/ofdpa

Hackathon resources and documentation:

1) The World's Smallest OpenFlow Testbed using the World's Smallest OpenFlow Switch (a recipe for building our testbed)

2) Resources for SDNCon 2016 https://github.com/gwacter/sdncon2016

3) SDNcon.net site provided by REANNZ  http://sdncon.net

Prizes and Judging.

Randy Fitton acted as the judge for the competition with support from the VUW organisers. In the judging notes, Randy said that it was actually hard to distinguish the work done because each tackled quite a different problem with diverse challenges.

UNSW was awarded first prize (real world scenario, great demonstration), VUW second prize (implemented features that take Faucet forward), Cyber Networking came third (fantastic technical work) and Whitireia/CPIT came fourth (met their aim of increasing understanding how Faucet works and how it and Ryu might be useful to them) .

Prizes included mugs, t-shirts, chocolate and soap. We didn’t want high value prizes because the focus of the event was building code and learning together rather than competition.

Feedback included “Great networking. Encouraging to share with a variety of people, with a common interest. Learnt some good things too! learnt a lot from the networking that took place between teams”.

Wednesday 27 July 2016

The World's Smallest OpenFlow Testbed using the World's Smallest OpenFlow Switch

The Zodiac FX is an OpenFlow enabled switch small enough to fit in the palm of your hand. Its small size allows OpenFlow-enabled applications to be tested on devices on your desk; there is no need for a rack full of switches!

In this post we cover the configuration of a testbed that was used at SDNcon 2016, hosted at Victoria University of Wellington. The test beds that were provided comprised of Zodiac FX switches and Raspberry Pi 2's. The diagram below illustrates the topology that was used. The grey box represents a Zodiac FX and the green boxes represent Raspberry Pi 2's.


The configuration of the switches were not changed, however firmware was updated when required. Information on updating the Zodiac FX firmware can be through the User Guide link at http://forums.northboundnetworks.com/index.php?topic=52.0. What we describe below is the configuration of the controller and the network hosts.

Pre-build Information

Use the following checklist before jumping into the build of the testbed.

Hardware

  • Raspberry Pis for the controller and hosts. Raspberry Pi 2's were used with Raspbian May 2016 (kernel version 4.4) as the OS.
  • Zodiac FX with firmware version 0.63 (if you would like to run Faucet) or higher. 0.64 was used for this testbed.
  • A USB-to-Ethernet adapter.
  • A USB WiFi dongle if the controller device is a Raspberry Pi 2 or earlier.

Skills

  • Competent use of the Linux command line (using bash) including a terminal-based text-editor.
  • Knowledge of configuring Linux-based hosts for network connectivity via the command line.
  • Ability to update the firmware of a Zodiac FX switch and access it via a console connection for debugging purposes.

Controller Configuration

This part of the post describes how to configure the controller. Details on the setup of Raspbian are provided, along with network configuration details. Following this, we go through the installation of the Ryu controller and the Faucet application.

Raspbian Configuration

  1. Write the Raspbian image to the micro SD card. Insert the card into the Raspberry Pi along with a mouse, keyboard and cable to a monitor. Turn on the device.
  2. After the Raspberry Pi has booted, a desktop should be presented with the Raspberry Pi logo. Open a terminal and run sudo raspi-config.
  3. The Raspberry Pi Software Con figuration Tool should be displayed in the terminal, this is the main menu. Select option 1 Expand Filesystem. Select OK in the prompt that follows.
  4. From the main menu select option 3 Boot Options to change the environment the Raspberry Pi boots into. Select option B1 Console.
  5. From the main menu select option 5 Internationalisation Options and change the following (note that you will need to navigate back to this menu after each change):
    1. Set the locale under the option I1 Change Locale. Use the space-bar to add a '*' next to en_NZ.UTF-8 UTF-8. In the Con figuring locales prompt use the up and down arrow keys to highlight en_NZ.UTF-8 and select using enter.
    2. Set the timezone under the option I2 Change Timezone. Using the up and down arrow keys to navigate and enter to select, select the following: Pacific Ocean, Auckland.
    3. Set the keyboard layout under the option I3 Change Keyboard Layout. Using the up and down arrow keys to navigate and enter to select, select the following:
      1. Keyboard model: Generic 105-key (Intl) PC
      2. Keyboard layout: Other
      3. Country of origin: English (US)
      4. Keyboard layout: English (US)
      5. Key to function as AltGr: The default for the keyboard layout
      6. Compose key: No compose key
      7. X server termination with Control+Alt+Backspace: <Yes>.
    4. Set the Wi-Fi country under the option I4 Change Wi- Country. Using the up and down arrow keys to navigate and enter to select, select NZ New Zealand then <OK> from the following prompt.
  6. From the main menu select option 9 Advanced Options, then option A2 Hostname. Acknowledge the character restrictions for hostnames by selecting <OK>. In the prompt, change the hostname to controller. Once done, press the down arrow key to select <OK>.
  7. From the main menu use the left and right arrow keys to select <Finish>, press enter. When presented with a prompt asking if you would like to reboot now select <Yes>. Keep in mind that the hostname can be assigned arbitrarily, the aforementioned hostname was used to match the diagram above.
  8. Boot the Raspberry Pi and login. Update the packages on the device by running sudo apt-get update followed by sudo apt-get upgrade. Following this, update the firmware by running sudo rpi-update.

Installation of Useful Packages

The lists below contains other useful packages that can be downloaded via apt-get.

General

  • git
  • python-dev
  • python-pip
  • tcpdump
  • vim

Serial Connections

  • cu
  • minicom
  • screen

Terminal Multiplexers

  • screen
  • tmux

Configuring the Network Interfaces

Open the file /etc/network/interfaces and edit the entries for eth0, eth1 and wlan0 to match the following.

# OpenFlow control-plane
auto eth0
iface eth0 inet static
address 10.0.1.8
netmask 255.255.255.0

# Gateway for the data-plane
allow-hotplug eth1
iface eth1 inet static
address 172.16.0.254
netmask 255.255.255.0

allow-hotplug wlan0
iface wlan0 inet manual
    wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

Open the file /etc/wpa_supplicant/wpa_supplicant.conf  to configure the WiFi interface to connect to your network. Below are the necessary details for connecting to an WiFi network using WiFi Protected Access (WPA).

network={
    ssid="My network
    key_mgmt=WPA-PSK
    psk="My password"
}

NAT Configuration

Add the following lines to /etc/rc.local to configure the controller for NAT on boot. Ensure that the last line in the file is exit 0. Note that the controller will need to be restarted for the changes to take effect.

printf "Configuring NAT between interfaces eth1 and wlan0.\n"
echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
/sbin/iptables -A FORWARD -i wlan0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i eth1 -o wlan0 -j ACCEPT
printf "NAT configuration complete.\n"

Installing Ryu and Faucet

The below instructions have been adapted from a Blogspot entry made by Josh Bailey (available as of 15/06/2016 from https://faucet-sdn.blogspot.co.nz/2016/06/raucet-raspberry-pi-faucet-controlling.html). The adaptation assumes that the network interfaces were configured as shown in the Configuring the Network Interfaces section of this post.

  1. Install Faucet's dependencies on the controller from a terminal.

  2. pi@controller ~ $ sudo apt-get install python-pip git libpython-dev python-yaml
    pi@controller ~ $ sudo pip install ipaddr
    pi@controller ~ $ sudo pip install --upgrade six
    pi@controller ~ $ sudo pip install ryu


  3. Install Faucet by cloning the source code from GitHub into the controller's home directory.

  4. pi@controller ~ $ cd
    pi@controller ~ $ git clone https://github.com/REANNZ/faucet


  5. Configure Faucet to use the Zodiac FX switch. Create a directory to store the configuration information in and create a file called faucet.yaml. vi is used as a terminal text-editor in the example below.

    pi@controller ~ $ mkdir ~/zodiacfx
    pi@controller ~ $ vi ~/zodiacfx/faucet.yaml

    Add the following configuration information and pay close attention to the indentation. Note that the value associated with the dp_id item should match the MAC address sticker on the bottom of your Zodiac FX. This is the MAC address associated with the controller switch port. This can also be obtained by connecting to the switch via a console connection and viewing the switch configuration if the sticker is not present. The X's present the hexadecimal digits that most probably need to be changed.

  6. dp_id: 0x70b3d56cXXXX
    name: "zodiac-fx-1"
    hardware: "ZodiacFX"
    interfaces:
      1:
        native_vlan: 100
      2:
        native_vlan: 100
      3:
        native_vlan: 100
    vlans:
      100:
        name: "100"


  7. Create a script to start Faucet. vi is used as a terminal text-editor in the example below.

  8. pi@controller ~ $ cd
    pi@controller ~ $ vi start_faucet.sh

    Enter the following into the script.
    export FAUCET_CONFIG=~/zodiacfx/faucet.yaml
    export FAUCET_LOG=~/zodiacfx/faucet.log
    export FAUCET_EXCEPTION_LOG=~/zodiacfx/faucet_exception.log
    /usr/local/bin/ryu-manager --config-file=~/faucet/src/ryu_faucet/org/onfsdn/faucet/ryu-faucet.conf --ofp-listen-host=10.0.1.8 --ofp-tcp-listen-port=6633 ~/faucet/src/ryu_faucet/org/onfsdn/faucet/faucet.py --verbose 

Host Configuration

This part of the post describes how to configure the host devices. Details on the setup of Raspbian are provided, along with network configuration details.

Raspbian Configuration

  1. Write the Raspbian image to the micro SD card. Insert the card into the Raspberry Pi along with a mouse, keyboard and cable to a monitor. Turn on the device.
  2. After the Raspberry Pi has booted, a desktop should be presented with the Raspberry Pi logo. Open a terminal and run sudo raspi-config.
  3. The Raspberry Pi Software Con figuration Tool should be displayed in the terminal, this is the main menu. Select option 1 Expand Filesystem. Select OK in the prompt that follows.
  4. From the main menu select option 3 Boot Options to change the environment the Raspberry Pi boots into. Select option B1 Console.
  5. From the main menu select option 5 Internationalisation Options and change the following (note that you will need to navigate back to this menu after each change):
    1. Set the locale under the option I1 Change Locale. Use the space-bar to add a '*' next to en_NZ.UTF-8 UTF-8. In the Con figuring locales prompt use the up and down arrow keys to highlight en_NZ.UTF-8 and select using enter.
    2. Set the timezone under the option I2 Change Timezone. Using the up and down arrow keys to navigate and enter to select, select the following: Pacific Ocean, Auckland.
    3. Set the keyboard layout under the option I3 Change Keyboard Layout. Using the up and down arrow keys to navigate and enter to select, select the following:
      1. Keyboard model: Generic 105-key (Intl) PC
      2. Keyboard layout: Other
      3. Country of origin: English (US)
      4. Keyboard layout: English (US)
      5. Key to function as AltGr: The default for the keyboard layout
      6. Compose key: No compose key
      7. X server termination with Control+Alt+Backspace: <Yes>.
    4. Set the Wi-Fi country under the option I4 Change Wi- Country. Using the up and down arrow keys to navigate and enter to select, select NZ New Zealand then <OK> from the following prompt.
  6. From the main menu select option 9 Advanced Options, then option A2 Hostname. Acknowledge the character restrictions for hostnames by selecting <OK>. In the prompt, change the hostname to hosta for the first host and hostb for the second host. Once done, press the down arrow key to select <OK>. Keep in mind that the hostnames can be assigned arbitrarily, the aforementioned hostnames are used to match the diagram above.
  7. From the main menu use the left and right arrow keys to select <Finish>, press enter. When presented with a prompt asking if you would like to reboot now select <Yes>.
  8. Boot the Raspberry Pi and login. Update the packages on the device by running sudo apt-get update followed by sudo apt-get upgrade. Following this, update the firmware by running sudo rpi-update.

Installation of Useful Packages

The lists below contains other useful packages that can be downloaded via apt-get.

General

  • git
  • libpcap-dev
  • python-dev
  • python-pip
  • tcpdump
  • vim

Terminal Multiplexers

  • screen
  • tmux

Traffic Generators and Manipulators

  • d-itg
  • iperf
  • nmap (nping is included)
  • python-scapy

Configuring the Network Interfaces

Open the file /etc/network/interfaces and edit the entry for eth0 to match the following. Note that the X should be set to 101 for hosta and 102 for hostb.

# Host data-plane
auto eth0
iface eth0 inet static
address 172.16.0.X
netmask 255.255.255.0
gateway 172.16.0.254
dns-nameservers 8.8.8.8


Bonus: Password-less SSH Connections

As a bonus, here is some information on configuring password-less SSH connections between the hosts. Be sure to note the data-plane IP addresses of each Raspberry Pi and their respective hostnames.

  1. On each host run ssh-keygen -t rsa -b 4096 to generate a private/public key pair for SSH.
  2. Add the public key of each host (found at ~/.ssh/id_rsa.pub) to the ~/.ssh/authorized_keys file on the other hosts. Each key should be on its own line.
  3. On each host, add the other hosts to the file ~/.ssh/config in the form bellow.

  4. Host <hostname>
      User pi
      HostName <Host IP address>


With the SSH config set, you can use ssh <hostname> to SSH into each host without a password and the need to enter an IP address.

Sunday 10 July 2016

BGP and Faucet

I have integrated BGP into Faucet by using Ryu BGP speaker library. A BGP app runs along side with Faucet and speaks BGP with external peers. Update messages received by BGP are forwarded to Faucet. Faucet updates its routing table and converts the updates to flowmods accordingly.

This approach is quite simple and you could replace the BGP app with a routing engine such as Quagga or Bird.

To investigate how well this works and to perform a simple "smoke test" on a virtual network (Mininet/single PC). I implemented a BGP update generator that generated 100 unique updates at the rate of 1000 update/second. Each update contains an announcement of 1, 2 and 5 prefixes. Curious about performance, I measured the time delay between the BGP update and resulting flowmod.

The delay decreases from the first update to the last. It appears the messages are being buffered before being sent. It is unclear where although this might be an artefact of our virtual environment where we have shared resources even though Mininet makes it appear that we don't.

As part of my PhD work, we will evaluate my code changes using a hardware platform. However, this small piece of works shows that it is relatively easy to integrate BGP into Faucet.

Back to the proposal. I'm now writing the literature review, part about multipath routing.

Thursday 30 June 2016

How to Perform Centralised Computation of Routes without Minimising the Revelation of Private Routing Information

One of the issues I am considering as part of my PhD is how to encourage ISPs and other ASes to share routing information despite privacy concerns, worry about loss of autonomy and economic incentives not to share this with other ASes.

The benefit of ASes sharing this information would be the ability to compute routing outcomes centrally leading to faster convergence compared to having each domain do it's own computation with partial knowledge and more efficient routers because computation could be offloaded to a "routing service".

However, a possible approach that allows computation to take place without revealing the inputs is described in A New Approach to Interdomain Routing Based on Secure Multi-Party Computation, Gupta et. al, 2012 published at HOTNETS 2012.

This is based on applying secure multi-party computation (SMPC) to interdomain routing. SPMC algorithms are based upon the intuition in Shamir's paper How to Share a Secret from 1979 and work on homomorphic encryption.

Their solution is partially centralised, in that k compute nodes are used to perform the computation together. It still more centralised than the current model leading to the benefits of improved convergence and offloading of work from routers. To provide a level of security, these servers would be owned by different providers and we assume that they will not collude to reveal any secrets entrusted to them.

Each AS splits their routing policy is split into k shares and send each share to the group of k compute nodes. A SPMC algorithm is used to compute the BGP-routing outcome across the shares received from the each AS. The result of this algorithm is only a partial outcome representing a share of the outcome (a route to a destination). Each compute node returns this partial outcome to each AS and the AS combines the k shares received to reveal the final result.

This scheme provides our privacy guarantees because no one node knows enough to reconstruct the routing policy nor the outputs of the computation. The scheme is also resistant to a degree of collusion, privacy is maintained as as long as fewer than k/2 nodes decide to pool their results.

Routing computation is simplified to make implementing the scheme feasible by assuming only the use of next-hop policies and computation is done on a per-domain basis rather than a prefix basis. A routing policy consists of two components: a ranking of all possible routes to a destination and a specification of which route to a destination will be exported to which peer. Network topology is known to all parties and denoted as [n] = {1...n} set of domains. Each domain u has two inputs: (1) ranking of its neighbors; and (2) for every two neighbor i and j, the willingness to export routes that have i as next-hop to j.

The scheme was implemented using the SEPIA Java SPMC library and the authors reported it takes 0.13 seconds compute the route to destination for a single domain with 19 neighbours and 10 iterations (the number of iterations is 2D+1 where D is the depth of the customer-provider hierarchy, see the definition in Gao and Rexford's paper). This experiment was carried out on a single compute node (a PC 2,7GHz quad core machine).

Is this a workable solution? No but it is a proof-of-concept. The code was not optimised and a full performance analysis was not done. Furthermore, the measurement was for the computation of only one route to destination and a full implementation would calculate many for each AS. This is obviously another scalability problem but the authors believe that this could be addressed if each compute node performs these calculation in parallel. This is feasible since there is no dependencies between computation but its practicality was not explored here.

Evaluation. This scheme provides good privacy guarantees and an implementation is described.

There are several open problems:

a) How scalable is this approach? Investigate its scalability with respect to neighbours, depth of hierarchy and number of computations required for a full BGP routing table. Also what is the effect on end-to-end performance and reliability of the Internet?

b) How do we hide network information such as who is the neighbour of a given AS? It is assumed that the inter domain network topology is known, that is every AS is willing to reveal its neighbours. The authors suggest that because this assumption may not hold, they intend to investigate also hiding network information using the SPMC approach could be explored. It doesn't appear to date that they did do this so this remains an open problem.

c) Can this scheme be generalised to prefix-based and QoS based routing? This would allow traffic engineering without revealing sensitive information.

d) Can this scheme be extended to address BGP instability due to failure?





Thursday 24 March 2016

Improving inter domain routing using software defined networking (SDN)

My PhD topic is "Improving Interdomain routing using software defined networking SDN". In particular, I am interested in how the Border Gateway Protocol could be replaced while maintaining compatability with autonomous systems (ASes) that still wish to use BGP.

My starting point is to consider if BGP reliability and scalability can be improved by treating the Internet as a single AS. The challenge is how to do this in a practical way that can be deployed while maintaining each member ASes autonomy. We believe that SDN can be used as a way to innovate.



Friday 12 February 2016

eResearch 2016

The VUW SDN group presented work on blackbox testing of SDN applications and ran a full day workshop on mininet and the faucet switch at eResearch NZ.