Compiling the mOS net library ============================= This chapter describes how to compile and build mOS net library from the mOS source release package: mos_release_${ver} (e.g., mos_release_0.3). We first explain the components of this distribution, and then describe the required steps for compiling it with each I/O library (DPDK or PCAP). Our I/O library of choice is DPDK. However, one can use PCAP I/O for testing/debugging purposes. Components of mOS distribution ----------------------------------------- The mOS networking stack release package contains the following subfolders: .. code-block:: console $ git clone https://github.com/ndsl-kaist/mOS-networking-stack.git $ ls mos_release_0.3/ core/ docs/ drivers/ LICENSE README.md samples/ scripts/ setup.sh util/ The main components of the distribution are: * ``core/`` contains the core source code of the mOS networking stack. * ``docs/`` contains user guides for developers (Programmer's Guide and mOS API Specification). * ``drivers/`` contains packet I/O libraries and drivers used by the stack. * ``LICENSE`` shows the license of our release. * ``samples/`` contains sample mOS applications (midstat, simple_firewall, nat, epserver, and epwget). Please refer to :doc:`../../samples/index` for further explanation. * ``scripts/`` contains the scripts for automating the installation process. * ``setup.sh`` is a user script for setting up mOS applications that will be discussed in later subsections. * ``util/`` contains utility libraries used by the core networking library and the sample applications. Compile and build mOS net library with DPDK -------------------------------------------- We first describe how to compile and build mOS net library with DPDK. Please make sure that your setup satisfies the conditions specified in :doc:`02_requirement`. Clone the mOS release to local directory ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: console $ git clone https://github.com/ndsl-kaist/mOS-networking-stack.git Compile and build mOS net library with DPDK ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When you run ``setup.sh`` with ``--compile-dpdk`` argument, the script first compiles Intel DPDK library, and then builds the mOS networking stack. .. code-block:: console $ cd mos_release_0.3/ $ ./setup.sh --compile-dpdk Once DPDK toolkit is successfully compiled, the script prints out the following message and prompts the user to comple the core stack. .. code-block:: console Build complete [x86_64-native-linuxapp-gcc] ================== Installing ./ Installation in ./ complete Done with DPDK setup. Press enter to start MOS setup ... On pressing enter, the second half of the compilation process begins. A successful compilation run ends with the following message: .. code-block:: console ---------------------------------------------------------- Done with MOS library setup ---------------------------------------------------------- .. _kernel_module: Configure runtime environment for mOS applications with DPDK ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ After your mOS net library compilation and build is complete, you need to set up the environment for running mOS applications with DPDK. When you execute the ``setup.sh`` script with ``--run-dpdk`` argument, a menu for setting up the run-time environment is displayed. .. code-block:: console $ ./setup.sh --run-dpdk ---------------------------------------------------------- Full setup (from start) ---------------------------------------------------------- [0] Full setup for running mOS with DPDK ---------------------------------------------------------- Step-by-step setup for running mOS with DPDK ---------------------------------------------------------- [1] Setup hugepage mappings [2] Load and bind Ethernet devices to IGB_UIO module [3] Bring the interfaces up (DPDK devices) [4] Create new MOS configuration files for sample apps [5] Unbind dpdk-registered NICs [6] Exit script Option: Steps [1] ~ [4] are mandatory steps for configuring your network interface with DPDK driver. Alternatively, you can just enter [0] that automatically performs all the steps from [1] ~ [4]. In this section, we explain what each step does in more detail. A portion of the text below is borrowed from http://dpdk.readthedocs.org/. **Step 1. Setup hugepage mappings** The following selection demonstrates the creation of hugepages in a NUMA system. 1024 2 MByte pages are assigned to each node. .. code-block:: console Option: 1 Removing currently reserved hugepages mounting /mnt/huge and removing directory Input the number of 2MB pages for each node Example: to have 128MB of hugepages available per node, enter '64' to reserve 64 * 2MB pages on each node Number of pages for node0: 1024 Number of pages for node1: 1024 Reserving hugepages Creating /mnt/huge and mounting as hugetlbfs **Step 2. Load and bind Ethernet devices to IGB_UIO module** The following selection first unloads any existing IGB_UIO module and then reloads and binds Ethernet devices for DPDK (& mOS) use. When an interface is bound to the IGB_UIO module, it also unbinds from its associated kernel driver module. .. code-block:: console Option: 2 Unloading any existing DPDK UIO module Loading DPDK UIO module Network devices using DPDK-compatible driver ============================================ Network devices using kernel driver =================================== 0000:03:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em1 drv=tg3 unused=igb_uio *Active* 0000:03:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em2 drv=tg3 unused=igb_uio Other network devices ===================== 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=igb_uio 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=igb_uio Enter PCI address of device(s) to bind to IGB UIO driver (e.g., "04:00.0 04:00.1"). > 04:00.0 OK Press enter to continue ... Please note that you can unbind the interface from DPDK driver and go back to using kernel driver by selecting [5]. This option is useful when you want to test mOS applications with PCAP module. .. code-block:: console Option: 5 em1 lo Network devices using DPDK-compatible driver ============================================ 0000:04:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused= Network devices using kernel driver =================================== 0000:03:00.0 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em1 drv=tg3 unused=igb_uio *Active* 0000:03:00.1 'NetXtreme BCM5720 Gigabit Ethernet PCIe' if=em2 drv=tg3 unused=igb_uio Other network devices ===================== 0000:04:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' unused=igb_uio Enter PCI address of device to unbind: 04:00.0 Enter name of kernel driver to bind the device to: ixgbe OK Press enter to continue ... .. warning:: As of release DPDK 1.4, DPDK applications no longer automatically unbind all supported network ports from the kernel driver in use. Therefore, all ports that are to be used by a mOS application with DPDK must be bound to the ``igb_uio`` module before the application is run. Any network ports under Linux control will be ignored by the DPDK poll-mode drivers and cannot be used by any Linux application (including system stats like ``sysstat``). .. hint:: You cannot unbind those interfaces that are currently online in your system. In order to bind such interfaces, one first needs to bring it offline (`e.g.` ifconfig down)). **Step 3. Bring the interfaces up (DPDK devices)** The following selection brings the DPDK network interfaces up, and assign any network address (IP address) to each port. .. code-block:: console Option: 3 igb_uio is loaded! [dpdk0] enter IP address[/mask] (e.g., 10.0.0.9[/24]) > 10.0.0.7/24 sudo /sbin/ifconfig dpdk0 10.0.0.7/24 up Press enter to continue ... .. note:: For :doc:`../config/01_inline`, you can leave the IP address unassigned. .. code-block:: console Option: 3 igb_uio is loaded! [dpdk0] enter IP address[/mask] (e.g., 10.0.0.9[/24]) > 0.0.0.0 sudo /sbin/ifconfig dpdk0 0.0.0.0 up Press enter to continue ... **Step 4. Create new MOS configuration files for sample apps** This selection creates new mOS startup configuration files (mos.conf) for each application. It automatically reads the machine specs (e.g., number of total CPU cores, available DPDK network interfaces) and records the current configuration. The details on the configuration parameters are explained in detail in :doc:`05_configuration` section. .. code-block:: console Option: 4 ... ------------------------------------------------ Done with configuration file setup. Use the arp command to add static ARP entries ------------------------------------------------ Press enter to continue ... .. hint:: You can subsequently run any of [1] ~ [5] afterwards to update the runtime environment. .. caution:: The setup.sh script is prepared only for Linux ``x86_64`` distributions. Please note that the user may have to manually compile DPDK library for other OS environments. **Step 5. Test that the installation is successful** You can test whether all 4 steps are successfully executed with four basic checks. **1.** DPDK-compatible interfaces are up. .. code-block:: console $ ifconfig dpdk0 Link encap:Ethernet HWaddr 90:e2:ba:7a:b0:00 inet addr:10.0.0.7 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::92e2:baff:fe7a:b000/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0 B) TX bytes:0 (0 B) dpdk1 Link encap:Ethernet HWaddr 90:e2:ba:7a:b0:01 inet addr:10.0.1.7 Bcast:10.0.1.255 Mask:255.255.255.0 inet6 addr: fe80::92e2:baff:fe7a:b001/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0 B) TX bytes:0 (0 B) **2.** Makefile has been created in each ``sample`` application directory. .. code-block:: console $ find samples/ -name 'Makefile' samples/epserver/Makefile samples/lighttpd-1.4.32/Makefile samples/simple_firewall/Makefile samples/epwget/Makefile samples/nat/Makefile samples/midstat/Makefile **3.** mOS networking stack startup configuration files have been placed in all ``sample`` application directories. .. code-block:: console $ find samples/ -name 'mos.conf' samples/epserver/config/mos.conf samples/lighttpd-1.4.32/config/mos.conf samples/simple_firewall/config/mos.conf samples/netstat/config/mos.conf samples/epwget/config/mos.conf samples/common/config/mos.conf samples/nat/config/mos.conf samples/midstat/config/mos.conf **4.** mOS networking library is successfully created. .. code-block:: console $ ls -lh core/lib/libmtcp.a -rw-rw-r-- 1 wheel wheel 3.9M May 11 21:29 core/lib/libmtcp.a Compile and build mOS net library with PCAP -------------------------------------------- We first describe how to compile and build mOS net library with PCAP. The first few steps are quite similar to how we prepare the DPDK version of the library. Once again we suggest you to make sure that your setup envrionment satisfies the conditions specified in :doc:`02_requirement`. Clone the mOS release to local directory ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Repeat this step if you have not already cloned the mOS net package on your machine. .. code-block:: console $ git clone https://github.com/ndsl-kaist/mOS-networking-stack.git .. note:: Please remember to delete outdated mOS configuration files (``mos.conf``) in case you previously set up the mOS networking stack with DPDK settings. The most convenient way to wipe out the files is by executing the following command: .. code-block:: console $ ./setup.sh --cleanup Compile and build mOS net library with PCAP ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ When you run the ``setup.sh`` script with ``--compile-pcap`` argument, it directly compiles the mOS net library. .. code-block:: console $ cd mos_release_0.3/ $ ./setup.sh --compile-pcap Configure runtime environment for mOS applications with PCAP ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Once the mOS net library compiles, you can set up the runtime environment for executing mOS applications with PCAP. .. code-block:: console $ sudo ./setup.sh --run-pcap During the execution of the script, the ``setup.sh`` script scans all Ethernet ports and prompts the user to choose the network interface he/she wants to register with the mOS net library. .. warning:: * PCAP version of monitoring applications should always be executed with a single core. PCAP I/O does not provide a built-in support for splitting traffic flows *symmetrically* across cores. * Please remember to bring the interfaces up (``ifconfig up``). .. note:: In case you want to detach dpdk-registered NICs for later PCAP use, we suggest using ``./setup.sh --run-dpdk``. Select option 5 to unbind NICs from DPDK's ``igb_uio`` driver. Also, make sure that you have removed old ``mos.conf`` files from monitoring applications' directories.