
\chapter{Evaluation}\label{ch:evaluation}

This chapter evaluates the developed approach in terms of both its practical usefulness and its performance.
Section~\ref{sec:usefulness} examines how well the library and tools support automated testing in the context of the Operating Systems course.
Section~\ref{sec:performance} analyzes the performance overhead introduced by intercepting function calls.


\section{Usefulness for the Operating Systems Course}\label{sec:usefulness}

Up until recently the Operating Systems course (mentioned in Section~\ref{sec:motivation-and-goal}) was split into three exercise blocks:
Files, Shared Memory, Semaphores; Related Processes and Inter-Process Communication via Unnamed Pipes; and Sockets.
Table~\ref{tab:functions} lists all functions presented in the course and their implementation status in \texttt{intercept.so}.
As one may see, simple file stream functions are not currently implemented in \texttt{intercept.so}.
This is because of time restrictions on this work, and the fact that simple file operations may be tested easily in the conventional way of checking the resulting output.
Note that the future implementation of single functions is not very complex.
All other functions have at least interception and mocking (returning, failing) implemented.
For some functions the modification of function arguments has been implemented.

\begin{table}[h!]
  \centering
  \begin{threeparttable}
    \begin{tabular}{ c|l|c }
      Ex. Block & Functions                                           & Implementation \\
      \hline
      - & malloc, calloc, realloc, reallocarray, free                 & \texttt{icmrf} \\
      - & getopt                                                      & \texttt{ic-rf} \\
      - & sigaction                                                   & \texttt{ic-{}-f} \\
      - & mmap                                                        & \texttt{ic-rf} \\
      - & munmap                                                      & \texttt{icmrf} \\
      - & close                                                       & \texttt{icmrf} \\
      \hline
      1 & open, fopen, fdopen                                         & \texttt{-{}-{}-{}-{}-} \\
      1 & read, pread, write, pwrite                                  & \texttt{ic-rf} \\
      1 & fread, fgets, fgetc, fwrite, fputs, fputc, fprintf, fseek   & \texttt{-{}-{}-{}-{}-} \\
      1 & ferror, feof, clearerr, fileno, fflush, fclose              & \texttt{-{}-{}-{}-{}-} \\
      1 & getline, getdelim                                           & \texttt{ic-rf} \\
      1 & shm\_open, ftruncate, shm\_unlink                           & \texttt{icmrf} \\
      1 & sem\_open, shm\_close, sem\_unlink                          & \texttt{icmrf} \\
      1 & sem\_wait, sem\_post                                        & \texttt{icmrf} \\
      1 & sem\_trywait, sem\_timedwait                                & \texttt{icmrf} \\
      1 & sem\_getvalue, sem\_destroy                                 & \texttt{icmrf} \\
      \hline
      2 & fork, wait, waitpid                                         & \texttt{icmrf} \\
      2 & exec*, fexecve                                              & \texttt{ic-rf} \\
      2 & exit                                                        & \texttt{icm-{}-} \\
      2 & pipe                                                        & \texttt{ic-rf} \\
      2 & dup, dup2, dup3                                             & \texttt{icmrf} \\
      \hline
      3 & socket, bind, accept, connect                               & \texttt{ic-rf} \\
      3 & listen                                                      & \texttt{icmrf} \\
      3 & getaddrinfo                                                 & \texttt{ic-rf} \\
      3 & freeaddrinfo                                                & \texttt{icmrf} \\
      3 & send, recv                                                  & \texttt{ic-rf} \\
      3 & sendto, sendmsg, recvfrom, recvmsg                          & \texttt{ic-rf} \\
      3 & setsockopt                                                  & \texttt{-{}-{}-{}-{}-} \\
    \end{tabular}
    \begin{tablenotes}
      \item[\texttt{i}] Function may be intercepted.
      \item[\texttt{c}] Complex function arguments or return value(s) are recorded fully.
      \item[\texttt{m}] Function arguments may be modified.
      \item[\texttt{r}] Function may be mocked using a specified return value.
      \item[\texttt{f}] Function may be mocked using a specified error value.
    \end{tablenotes}
    \caption{List of relevant functions and their implementation status.}
    \label{tab:functions}
  \end{threeparttable}
\end{table}


\section{Performance}\label{sec:performance}

Although high performance was not a primary goal of this work, the performance degradation caused by interception and manipulation should not be excessive.
The following two subsections test and discuss the performance degradation of \texttt{intercept.so} compared to running a program without any interception or hooking.


\subsection{Performance when Intercepting}\label{subsec:performance-intercepting}

To test the performance of \texttt{intercept.so}, the following measurement environment was set up.
On an x86-64 machine with an AMD Ryzen 7 7700X 8-Core processor, a simple program was called with an increasing number of iterations it had to perform.
The program simply called the \texttt{pipe} function and then closed the created pipes in a for loop.
At first execution time of the program was measured without using \texttt{intercept.so} (``Baseline'').
Then \texttt{intercept.so} was preloaded, but without any action to perform when intercepting (``Intercepting'').
After that, logging to \texttt{stderr} was enabled (``Logging to stderr'').
Finally, the logs were written to a file (``Logging to file'').
For each of the four variants, the program was called with an iteration count beginning at 100 and increasing in steps of 100 up to 5000.
Each measurement was taken 30 times, with one second between program executions to rule out statistical outliers.
Figure~\ref{fig:intercept-performance} illustrates the results.

It is clearly visible that the initialization step of \texttt{intercept.so} always takes around 10 ms.
In comparison to that, the performance degradation of the intercepting procedure alone is negligible, only around +13\% compared to the baseline execution (see ``Intercept -- 10 ms'' and ``Baseline'').
Most of the delay is caused by logging the recorded function calls.

\begin{figure}
  \centering
  \begin{tikzpicture}
    \begin{axis}[
        xmin=100, xmax=5000,
        ymin=0, ymax=150,
        scaled y ticks=false,
        ymajorgrids=true,
        grid style=dashed,
        legend pos=north west,
        legend cell align={left},
        width=\textwidth,
        xlabel=Iterations,
        ylabel=Execution Time,
        x unit=\#,
        y unit=\si{\milli\second},
      ]

      \addplot[color=blue] table [x=Cycles, y expr=\thisrow{Baseline Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Baseline}
      \addplot[name path=a_top,color=blue!50,forget plot] table [x=Cycles, y expr=(\thisrow{Baseline Avg} + \thisrow{Baseline SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=a_down,color=blue!50,forget plot] table [x=Cycles, y expr=(\thisrow{Baseline Avg} - \thisrow{Baseline SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[blue!20,fill opacity=0.2] fill between[of=a_top and a_down];
      \addlegendentry{Baseline ±SD}

      \addplot[color=green] table [x=Cycles, y expr=\thisrow{Intercept Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Intercepting}
      \addplot[name path=b_top,color=green!50,forget plot] table [x=Cycles, y expr=(\thisrow{Intercept Avg} + \thisrow{Intercept SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=b_down,color=green!50,forget plot] table [x=Cycles, y expr=(\thisrow{Intercept Avg} - \thisrow{Intercept SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[green!20,fill opacity=0.2] fill between[of=b_top and b_down];
      \addlegendentry{Intercepting ±SD}

      \addplot[color=teal] table [x=Cycles, y expr=(\thisrow{Intercept Avg} - 10000000) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Intercepting -- 10 ms}

      \addplot[color=red] table [x=Cycles, y expr=\thisrow{Stderr Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Logging to stderr}
      \addplot[name path=c_top,color=red!50,forget plot] table [x=Cycles, y expr=(\thisrow{Stderr Avg} + \thisrow{Stderr SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=c_down,color=red!50,forget plot] table [x=Cycles, y expr=(\thisrow{Stderr Avg} - \thisrow{Stderr SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[red!20,fill opacity=0.2] fill between[of=c_top and c_down];
      \addlegendentry{Logging to stderr ±SD}

      \addplot[color=orange] table [x=Cycles, y expr=\thisrow{File Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Logging to File}
      \addplot[name path=d_top,color=orange!50,forget plot] table [x=Cycles, y expr=(\thisrow{File Avg} + \thisrow{File SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=d_down,color=orange!50,forget plot] table [x=Cycles, y expr=(\thisrow{File Avg} - \thisrow{File SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[orange!20,fill opacity=0.2] fill between[of=d_top and d_down];
      \addlegendentry{Logging to File ±SD}
    \end{axis}
  \end{tikzpicture}
  \caption{Execution times of a simple program using \texttt{intercept.so} with different output modes.}
  \label{fig:intercept-performance}
\end{figure}


\subsection{Performance when Manipulating}\label{subsec:performance-manipulating}

Meaningful performance evaluation of function call manipulation requires a specific server implementation, since the response speed of the server dominates overall performance.
As seen in Subsection~\ref{subsec:performance-intercepting}, most delay comes not from intercepting itself, but from the further processing.
This also applies to function call manipulation.
The performance degradation heavily depends on the response speed of the used socket.
Despite this, a simple performance test has been conducted.

The setup was the same as in Subsection~\ref{subsec:performance-intercepting}.
But this time \texttt{intercept.so} was configured to connect to a Unix domain socket.
At first, a simple C program was used to respond to the messages on the socket, only using \texttt{getline} and \texttt{fprintf}.
For the first test run the program always responded with the \texttt{"ok"} command (``Manipulate (simple ok)''), for the second with the \texttt{"return 0"} command (``Manipulate (simple return)'').
After that, the default Python implementation developed in this work, which parses the incoming messages automatically, was tested.
Again, at first always responding with the \texttt{"ok"} command (``Manipulate (Python ok)''), and then with the \texttt{"return 0"} command (``Manipulate (Python return)'').
Figure~\ref{fig:manipulate-performance} illustrates the results and some previous measurements for context.

Results of the simple C program show, that the communication over the socket alone has only minimal overhead compared with ``Logging to stderr''.
Due to the parsing of messages the Python program has slightly worse performance degradation.
The ``return'' test is slightly faster compared to the ``ok'' test, because the \texttt{pipe} function normally responds with \texttt{return 0; errno 0; fildes=[7,8]}, but when using \texttt{"return 0"} it responds with \texttt{return 0; errno 0}, which is less data to parse.

\begin{figure}
  \centering
  \begin{tikzpicture}
    \begin{axis}[
      xmin=100, xmax=5000,
      ymin=0, ymax=400,
      scaled y ticks=false,
      ymajorgrids=true,
      grid style=dashed,
      legend pos=north west,
      legend cell align={left},
      width=\textwidth,
      xlabel=Iterations,
      ylabel=Execution Time,
      x unit=\#,
      y unit=\si{\milli\second},
    ]

      \addplot[color=blue] table [x=Cycles, y expr=\thisrow{Baseline Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Baseline}
      \addplot[name path=a_top,color=blue!50,forget plot] table [x=Cycles, y expr=(\thisrow{Baseline Avg} + \thisrow{Baseline SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=a_down,color=blue!50,forget plot] table [x=Cycles, y expr=(\thisrow{Baseline Avg} - \thisrow{Baseline SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[blue!20,fill opacity=0.2] fill between[of=a_top and a_down];
      \addlegendentry{Baseline ±SD}

      \addplot[color=green] table [x=Cycles, y expr=\thisrow{Intercept Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Intercepting}
      \addplot[name path=b_top,color=green!50,forget plot] table [x=Cycles, y expr=(\thisrow{Intercept Avg} + \thisrow{Intercept SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=b_down,color=green!50,forget plot] table [x=Cycles, y expr=(\thisrow{Intercept Avg} - \thisrow{Intercept SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[green!20,fill opacity=0.2] fill between[of=b_top and b_down];
      \addlegendentry{Intercepting ±SD}

      \addplot[color=red] table [x=Cycles, y expr=\thisrow{Ok Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Manipulate (simple ok)}
      \addplot[name path=d_top,color=red!50,forget plot] table [x=Cycles, y expr=(\thisrow{Ok Avg} + \thisrow{Ok SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=d_down,color=red!50,forget plot] table [x=Cycles, y expr=(\thisrow{Ok Avg} - \thisrow{Ok SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[red!20,fill opacity=0.2] fill between[of=d_top and d_down];
      \addlegendentry{Manipulate (simple ok) ±SD}

      \addplot[color=orange] table [x=Cycles, y expr=\thisrow{Return Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Manipulate (simple return)}
      \addplot[name path=d_top,color=orange!50,forget plot] table [x=Cycles, y expr=(\thisrow{Return Avg} + \thisrow{Return SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=d_down,color=orange!50,forget plot] table [x=Cycles, y expr=(\thisrow{Return Avg} - \thisrow{Return SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[orange!20,fill opacity=0.2] fill between[of=d_top and d_down];
      \addlegendentry{Manipulate (simple return) ±SD}

      \addplot[color=cyan] table [x=Cycles, y expr=\thisrow{PyOk Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Manipulate (Python ok)}
      \addplot[name path=d_top,color=cyan!50,forget plot] table [x=Cycles, y expr=(\thisrow{PyOk Avg} + \thisrow{PyOk SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=d_down,color=cyan!50,forget plot] table [x=Cycles, y expr=(\thisrow{PyOk Avg} - \thisrow{PyOk SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[cyan!20,fill opacity=0.2] fill between[of=d_top and d_down];
      \addlegendentry{Manipulate (Python ok) ±SD}

      \addplot[color=magenta] table [x=Cycles, y expr=\thisrow{PyReturn Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Manipulate (Python return)}
      \addplot[name path=d_top,color=magenta!50,forget plot] table [x=Cycles, y expr=(\thisrow{PyReturn Avg} + \thisrow{PyReturn SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=d_down,color=magenta!50,forget plot] table [x=Cycles, y expr=(\thisrow{PyReturn Avg} - \thisrow{PyReturn SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[magenta!20,fill opacity=0.2] fill between[of=d_top and d_down];
      \addlegendentry{Manipulate (Python return) ±SD}
    \end{axis}
  \end{tikzpicture}
  \caption{Execution times of a simple program using \texttt{intercept.so} with different manipulation modes.}
  \label{fig:manipulate-performance}
\end{figure}
