Middlebox Stats (midstat)
================================

Introduction
------------
``netstat`` (network statistics) is a command-line tool in UNIX-like operating
systems which displays the network connection status for the Transmission Control
Protocol. It is often used for debugging the network and to determine the
number of ongoing TCP connections.

The original ``netstat`` focuses on the end-host networking stack;
we develop a middlebox version of ``netstat`` on top of mOS which we
call ``midstat``. midstat shows the live statistics of both sides
(client and server side) network connection. Network statistics
include the IP addresses, port numbers, and the running TCP states
of each side of the ongoing connection.


Code Walkthrough
-----------------
The following sections provide an explanation of the main components of the
mOS ``midstat`` code.
All mOS library functions used in the sample code are prefixed with ``mtcp_``
and are explained in detail in the `Programmer's Guide - mOS Programming API`_.
Note that we omit the error handling logic in the example code for brevity.

(1) The main() Function
~~~~~~~~~~~~~~~~~~~~~~~
The ``main()`` function performs the initialization and calls the execution
threads for each CPU core.

The first task is to initialize mOS thread based on the mOS configuration file.
``fname`` is the file path to the ``mos.conf`` file which will be
provided to ``mtcp_init()``.

.. code-block:: c

	/* parse mos configuration file */
	ret = mtcp_init(fname);

In case the mOS configuration needs an update `after` ``mtcp_init()`` call (e.g., 
for changing the number of cores required for running the mOS application), we 
can use ``mtcp_getconf()`` to retrieve the current config from the mOS core. 
``mtcp_setconf()`` function can then be used to re-set some of the selected
parameters (as shown in the example below).

.. code-block:: c

	/* set the core limit */
	mtcp_getconf(&mcfg);
	mcfg.num_cores = g_max_cores;
	mtcp_setconf(&mcfg);
				
Afterwards, the main() function will create mtcp context for each core and launch
the per-core thread starting from the function ``InitMonitor()``.
``InitMonitor()`` function will be discussed in the following section.

.. code-block:: c

	for (i = 0; i < g_max_cores; i++) {
		/* Run mOS for each CPU core */
		if (!(g_mctx[i] = mtcp_create_context(i))) {
			fprintf(stderr, "Failed to craete mtcp context.\n");
			return -1;
		}

		/* init monitor */
	    InitMonitor(g_mctx[i], ev_new_syn);
	}

(2) Per-thread Initialization Function
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``InitMonitor()`` function is the core functional part of the mOS thread
initialization. In order to allow ``midstat`` to track all the ongoing
connections, it first initializes the memory for the global connection 
queue per each core.

.. code-block:: c

	/* Initialize internal memory structures */
	TAILQ_INIT(&g_sockq[mctx->cpu]);

Afterwards, this application creates a ``MOS_SOCK_MONITOR_STREAM`` socket
to monitor TCP-related events of ongoing TCP flows.

.. code-block:: c

	/* create socket */
	sock = mtcp_socket(ctx->mctx, AF_INET, MOS_SOCK_MONITOR_STREAM, 0);


Since this program does not perform any payload monitoring, it disables
the socket buffer of each side as follows:

.. code-block:: c
				
	mtcp_setsockopt(mctx, sock, SOL_MONSOCKET, MOS_CLIBUF, &optval, sizeof(optval));
	mtcp_setsockopt(mctx, sock, SOL_MONSOCKET, MOS_SVRBUF, &optval, sizeof(optval));

The last step of the per-thread initialization is to register callback functions
for the TCP events that this program interested in. The ``RegisterCallbacks``
function will be described further in the later sections.

.. code-block:: c

	RegisterCallbacks(mctx, sock, ev_new_syn);

				  
(3) The ``RegisterCallbacks()`` Function 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The ``RegisterCallbacks()`` function is used for registering callback handler
functions for the TCP events in order to monitor the TCP-level behavior.

First, ``cb_creation()`` function is called whenever a new connection is created.

.. code-block:: c

	mtcp_register_callback(mctx, sock, MOS_ON_CONN_START, MOS_HK_SND, cb_creation);


The role of the ``cb_creation()`` function is twofold: (1) to retrieve the address
of each side, and (2) to insert the connection metadata into the global connection
queue.

.. code-block:: c
	
	static void
	cb_creation(mctx_t mctx, int sock, int side, event_t events, filter_arg_t *arg)
	{
		...
		mtcp_getpeername(mctx, c->sock, (void *)c->addrs, &addrslen, MOS_SIDE_CLI);

		/* Insert the structure to the queue */
		TAILQ_INSERT_TAIL(&g_sockq[mctx->cpu], c, link);
	}

Second, ``cb_st_chg()`` function is called whenever there is any TCP state change.
We note that ``cb_st_chg()`` function is registered at both sender- and receiver-side
``MOS_ON_TCP_STATE_CHANGE`` events, since the TCP states of each side can be changed
independently.

.. code-block:: c

	mtcp_register_callback(mctx, sock, MOS_ON_TCP_STATE_CHANGE, MOS_HK_SND, cb_st_chg);
	mtcp_register_callback(mctx, sock, MOS_ON_TCP_STATE_CHANGE, MOS_HK_RCV, cb_st_chg);

The role of the ``cb_st_chg()`` function is to retrieve the changed TCP state using
``mtcp_getsockopt()`` function. The ``side`` parameter given to the callback function
indicates the side whose TCP state changed.

.. code-block:: c

	static void
	cb_st_chg(mctx_t mctx, int sock, int side, event_t events, filter_arg_t *arg)
	{
		if (side == MOS_SIDE_CLI) {
			mtcp_getsockopt(mctx, c->sock, SOL_MONSOCKET, MOS_TCP_STATE_CLI,
			(void *)&c->cli_state, &intlen);
		}
		else {
			mtcp_getsockopt(mctx, c->sock, SOL_MONSOCKET, MOS_TCP_STATE_SVR,
			(void *)&c->svr_state, &intlen);
		}
	}

Third, ``cb_destroy()`` function is called whenever there is any connection which is about
to be closed.

.. code-block:: c

	mtcp_register_callback(mctx, sock, MOS_ON_CONN_END, MOS_HK_SND, cb_destroy);

The role of the ``cb_destroy()`` function is to find and delete the connection from
the global connection queue, and release the allocated memory.

.. code-block:: c

	/* Destroy connection structure */
	static void
	cb_destroy(mctx_t mctx, int sock, int side, event_t events, filter_arg_t *arg)
	{
		struct connection *c;
		if (!(c = find_connection(mctx->cpu, sock)))
			return;	
		TAILQ_REMOVE(&g_sockq[mctx->cpu], c, link);
		free(c);
	}


The last step of the ``RegisterCallbacks()`` function is to register timer callback for
printing the network statistics every second. Note that this function should be
registered only for CPU core id 0, to prevent duplicate printing.

.. code-block:: c

	/* CPU 0 is in charge of printing stats */
	if (mctx->cpu == 0 && mtcp_settimer(mctx, sock, &tv_1sec, cb_printstat)
	...


.. _`Programmer's Guide - mOS Programming API`: ../programmer/04_mos_api.html