
\chapter{Evaluation}\label{ch:evaluation}

Lorem Ipsum.


\section{Usefulness for the Operating Systems Course}\label{sec:usefulness}

Up until recently the Operating Systems Course (mentioned in Section~\ref{sec:motivation-and-goal}) was split into three exercise blocks:
Files, Shared Memory, Semaphores; Related Processes and Inter-Process Communication via Unnamed Pipes; and Sockets.
Table~\ref{tab:functions} lists all functions presented in the course and their implementation status in \texttt{intercept.so}.
As one may see, simple file stream functions are not currently implemented in \texttt{intercept.so}.
This is because of time restrictions on this work and the fact, that simple file operations may be tested easily in the conventional way of checking the resulting output.
Note, that the future implementation of single functions is not very complex.
All other functions have at least interception and mocking (returning, failing) implemented.
For some functions the modification of function arguments has been implemented.

\begin{table}[h!]
  \centering
  \begin{threeparttable}
    \begin{tabular}{ c|l|c }
      Ex. Block & Functions                                           & Implementation \\
      \hline
      - & malloc, calloc, realloc, reallocarray, free                 & \texttt{icmrf} \\
      - & getopt                                                      & \texttt{ic-rf} \\
      - & sigaction                                                   & \texttt{ic-{}-f} \\
      - & mmap                                                        & \texttt{ic-rf} \\
      - & munmap                                                      & \texttt{icmrf} \\
      - & close                                                       & \texttt{icmrf} \\
      \hline
      1 & open, fopen, fdopen                                         & \texttt{-{}-{}-{}-{}-} \\
      1 & read, pread, write, pwrite                                  & \texttt{ic-rf} \\
      1 & fread, fgets, fgetc, fwrite, fputs, fputc, fprintf, fseek   & \texttt{-{}-{}-{}-{}-} \\
      1 & ferror, feof, clearerr, fileno, fflush, fclose              & \texttt{-{}-{}-{}-{}-} \\
      1 & getline, getdelim                                           & \texttt{ic-rf} \\
      1 & shm\_open, ftruncate, shm\_unlink                           & \texttt{icmrf} \\
      1 & sem\_open, shm\_close, sem\_unlink                          & \texttt{icmrf} \\
      1 & sem\_wait, sem\_post                                        & \texttt{icmrf} \\
      1 & sem\_trywait, sem\_timedwait                                & \texttt{icmrf} \\
      1 & sem\_getvalue, sem\_destroy                                 & \texttt{icmrf} \\
      \hline
      2 & fork, wait, waitpid                                         & \texttt{icmrf} \\
      2 & exec*, fexecve                                              & \texttt{ic-rf} \\
      2 & exit                                                        & \texttt{icm-{}-} \\
      2 & pipe                                                        & \texttt{ic-rf} \\
      2 & dup, dup2, dup3                                             & \texttt{icmrf} \\
      \hline
      3 & socket, bind, accept, connect                               & \texttt{ic-rf} \\
      3 & listen                                                      & \texttt{icmrf} \\
      3 & getaddrinfo                                                 & \texttt{ic-rf} \\
      3 & freeaddrinfo                                                & \texttt{icmrf} \\
      3 & send, recv                                                  & \texttt{ic-rf} \\
      3 & sendto, sendmsg, recvfrom, recvmsg                          & \texttt{ic-rf} \\
      3 & setsockopt                                                  & \texttt{-{}-{}-{}-{}-} \\
    \end{tabular}
    \begin{tablenotes}
      \item[\texttt{i}] Function may be intercepted.
      \item[\texttt{c}] Complex function arguments or return value(s) are recorded fully.
      \item[\texttt{m}] Function arguments may be modified.
      \item[\texttt{r}] Function may be mocked using a specified return value.
      \item[\texttt{f}] Function may be mocked using a specified error value.
    \end{tablenotes}
    \caption{List of relevant functions and their implementation status.}
    \label{tab:functions}
  \end{threeparttable}
\end{table}


\section{Performance}\label{sec:performance}

Although high performance was not a primary goal of this work, the resulting performance degradation by intercepting and manipulation should not be too excessive.
The following two subsections test the performance degradation of \texttt{intercept.so} compared to running a program without any intercepting or hooking.


\subsection{Performance when Intercepting}\label{subsec:performance-intercepting}

Lorem Ipsum.

\begin{figure}
  \centering
  \begin{tikzpicture}
    \begin{axis}[
        xmin=0, xmax=5000,
        ymin=0, ymax=150,
        scaled y ticks=false,
        ymajorgrids=true,
        grid style=dashed,
        legend pos=north west,
        legend cell align={left},
        width=\textwidth,
        xlabel=Iterations,
        ylabel=Execution Time,
        x unit=\#,
        y unit=\si{\milli\second},
      ]

      \addplot[color=blue] table [x=Cycles, y expr=\thisrow{Baseline Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Baseline}
      \addplot[name path=a_top,color=blue!50,forget plot] table [x=Cycles, y expr=(\thisrow{Baseline Avg} + \thisrow{Baseline SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=a_down,color=blue!50,forget plot] table [x=Cycles, y expr=(\thisrow{Baseline Avg} - \thisrow{Baseline SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[blue!20,fill opacity=0.2] fill between[of=a_top and a_down];
      \addlegendentry{Baseline ±SD}

      \addplot[color=green] table [x=Cycles, y expr=\thisrow{Intercept Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Intercepting}
      \addplot[name path=b_top,color=green!50,forget plot] table [x=Cycles, y expr=(\thisrow{Intercept Avg} + \thisrow{Intercept SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=b_down,color=green!50,forget plot] table [x=Cycles, y expr=(\thisrow{Intercept Avg} - \thisrow{Intercept SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[green!20,fill opacity=0.2] fill between[of=b_top and b_down];
      \addlegendentry{Intercepting ±SD}

      \addplot[color=teal] table [x=Cycles, y expr=(\thisrow{Intercept Avg} - 10000000) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Intercepting -- 10 ms}

      \addplot[color=red] table [x=Cycles, y expr=\thisrow{Stderr Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Logging to stderr}
      \addplot[name path=c_top,color=red!50,forget plot] table [x=Cycles, y expr=(\thisrow{Stderr Avg} + \thisrow{Stderr SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=c_down,color=red!50,forget plot] table [x=Cycles, y expr=(\thisrow{Stderr Avg} - \thisrow{Stderr SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[red!20,fill opacity=0.2] fill between[of=c_top and c_down];
      \addlegendentry{Logging to stderr ±SD}

      \addplot[color=orange] table [x=Cycles, y expr=\thisrow{File Avg}/1000000, col sep=semicolon] {src/performance/stat.csv};
      \addlegendentry{Logging to File}
      \addplot[name path=d_top,color=orange!50,forget plot] table [x=Cycles, y expr=(\thisrow{File Avg} + \thisrow{File SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[name path=d_down,color=orange!50,forget plot] table [x=Cycles, y expr=(\thisrow{File Avg} - \thisrow{File SD}) / 1000000, col sep=semicolon] {src/performance/stat.csv};
      \addplot[orange!20,fill opacity=0.2] fill between[of=d_top and d_down];
      \addlegendentry{Logging to File ±SD}
    \end{axis}
  \end{tikzpicture}
  \caption{Execution times of a simple program using \texttt{intercept.so} with different output modes.}
  \label{fig:performance}
\end{figure}


\subsection{Performance when Manipulating}\label{subsec:performance-manipulating}

Lorem Ipsum.
