3. Running mTCP applications with mOS

This section shows one can use mOS library to run mTCP networking applications. We describe how one can set up sample mOS web server and client, and configure them to talk to each other.

Note

We have ported a couple of mTCP programs in mOS. These applications are functionally equivalent to those that are currently available in the mTCP release.

3.1. Environment

This section assumes an environment configured as described below.

../_images/endtcp_config.png
  • Data Network: Any Ethernet network that can deliver traffic between two TCP endpoints.
  • Host 1: This host runs a Web Client Application (epwget) instance, that runs on mOS stack. It has one NIC named dpdk0, which is connected to the Data Network.
  • Host 2: This host runs a Web Server Application (epserver) instance, that runs on mOS stack. It has one NIC named dpdk0, which is connected to the Data Network.

3.2. Configuration Steps

3.2.1. Compile and build mOS library and applications

Follow Compiling the mOS net library to set up the mOS net library and the environment.

  • In Step 3 of Section 3.2.3, please note that you need to assign a valid IP address to network interface to allow dpdk0 in each host to communicate with each other over the Data Network.

Refer to Compile and run sample applications, and make sure that you successfully compile the mOS applications on both endpoints.

  • In Host 1, you should compile epwget.
  • In Host 2, you should compile epserver.

3.2.2. Setup mOS application configurations

Before running the epwget and epserver applications, you need to configure the configuration files in each host.

  • In Host 1, please configure epwget.conf as below.
url = (IP address of Host 2)/(file name)
dest_port = 80
total_flows = (number of total flows to generate)
total_concurrency = (number of concurrent flows to run)
core_limit = (number of CPU cores to run)
  • In Host 1, please configure mos.conf as below. We assume that Host 1 uses CPU core 0 to 7 for running mOS epwget.
############### MOS configuration file ###############

#######################
# APPLICATION OPTIONS #
#######################

# application to run (epserver)
application {
             type = end
             run = epwget
             core_mask = 0x00FF      # 0-7
}

#######################
## mos configuration ##
#######################
mos {
             #######################
             ### TCP/IP OPTIONS ####
             #######################
             # maximum concurrency per core
             # (mos-specific parameter for preallocation)
             max_concurrency = 100000

             # maximum number of socket buffers per core
             max_num_buffers = 100000

             # disable the ring buffer
             no_ring_buffers = 0

             # receive buffer size of sockets
             rmem_size = 8192

             # send buffer size of sockets
             wmem_size = 8192

             # tcp timewait seconds
             tcp_tw_interval = 0

             # tcp timeout seconds
             # (tcp_timeout = -1 can disable the timeout check)
             tcp_timeout = 30

             #######################
             ##### I/O OPTIONS #####
             #######################
             # number of memory channels per socket ( dpdk - only )
             nb_mem_channels = 4

             # interfaces the mOS have (with core mapping)
             netdev {
                     dpdk0 0x00FF     # 0-7
             }

             #######################
             ### LOGGING OPTIONS ###
             #######################
             # A directory contains log files
             mos_log = logs/

             # NICs to print network statistics per second
             # if enabled, mTCP will print xx Gbps and xx pps for RX and TX
             stat_print = dpdk0
}
  • In Host 2, please configure epserver.conf as below.
www_main = (a folder path which contains the file to serve)
core_limit = (number of CPU core to run)
  • In Host 2, please configure mos.conf in the same manner that was mentioned for Host 1 with the exception of the following lines:
# application to run (epserver)
application {
             type = end
             run = epserver
             core_mask = 0x00FF      # 0-7
}

We also assume that Host 2 uses CPU core 0 to 7 for running the mOS epserver.

Important

  1. mOS ignores the forward option for mTCP applications.
  2. Please avoid using PCAP I/O module for mTCP applications.
  3. For optimal performance, please adjust the configuration parameters such as total_concurrency (in epwget.conf), max_concurrency, max_num_buffers, rmem_size, or wmem_size (in mos.conf) by referring to Configuration Parameter Tuning.

3.3. Run epserver and epwget

In Host 2, run epserver first with sudo privileges.

$ sudo ./epserver

In Host 1, when you run epwget with sudo privileges, you will see that epwget of Host 1 downloads the files from epserver running on Host 2.

$ sudo ./epwget

3.4. Troubleshooting

If your mOS program crashes, search through the printed log messages to check if you missed any of the following requirements:

  • You should load only the dpdk-registered interfaces (after loading the igb_uio driver) and configure hugepages as decribed in Compiling the mOS net library.
  • You should create a log folder specified in mos_log parameter in mos.conf (logs/ in our example) within the same directory that contains the mOS application binary.
  • Please check if your system is running out of memory by looking at the memory footprint (OOM errors). If you have memory deficiency issues, please reduce the number of socket buffers in mos.conf.

If the HTTP connections between Host 1 and 2 are lost, check the following items:

  • Make sure that Host 1 and Host 2 have full network connectivity over the interface that runs on default Linux Ethernet driver. Please note that mTCP networking stack uses the ARP and route tables from the kernel.
  • In other words, you can check the connectivity by going back to kernel network module (see the option 5 in Configure runtime environment for mOS applications with DPDK) and running default web server and client (e.g., nginx and wget) programs.

Warning

Please note that the mOS stack currently can only run on a CPU processor with 16 CPU cores or less. This restriction will be removed in future versions of the stack. For now, we suggest that the user disables all CPU core IDs > 16 using the sysfs interface.

$ cd /sys/devices/system/cpu
$ for i in {16..$MAX_CPU}
 do
 cd $i
 echo 0 > online
 cd ..
 done