
\chapter{Related Work}\label{ch:related-work}

This chapter gives a rough overview on techniques and methods to intercept or hook system calls and function calls.
See also Section~\ref{sec:methods-for-intercepting}.
Some relevant methods will be discussed there in more detail.


\section{System and Function Call Hooking in Literature}\label{sec:call-hooking-in-literature}

The following subsections explore some applications of system and function call hooking.
There are possibly many other use-cases, but the following were deemed most important.


\subsection{Classification of Hooking Techniques}\label{subsec:classification}

Lopez et al. \cite{lopez2017} classify subroutine hooking techniques as follows:

\begin{itemize}
  \setlength\itemsep{0em}
  \item Subroutine Type: Function / System call.
  \item Hook Insertion: Static (before execution) / Dynamic (during execution).
  \item Instrumentation Type: Active (``manipulation'') / Passive (``interception'').
  \item Hooking Location: On-device / Off-device (most used for mobile devices).
  \item Hooking Scope: Inner Functions / Exported Functions (e.g.,\ libraries).
  \item OS Modification: Required / Not Required.
  \item Availability of Source Code: Open-source / Closed-source.
  \item Pricing Model: Free / Paid.
\end{itemize}

The technique developed in this work would be classified as follows:
Function, Static, Active+Passive, On-device, Exported Functions, OS Modification Not Required, target program may be Closed-source, Free.


\subsection{Windows Systems}\label{subsec:windows-systems}

Hunt and Brubacher \cite{detours} developed Detours, a library for instrumenting arbitrary Win32 functions on x86 machines.
Detours intercepts Win32 functions by re-writing target function images.


\subsection{Security Applications}\label{subsec:security-applications}

Fraser et al. \cite{fraser2000} introduced a general mechanism for securing unmodified commercial software by wrapping system calls at the library interface.
They hook system calls by replacing the standard library entry points with wrapper functions, similar to \texttt{LD\_PRELOAD}.
The wrapper functions are able to perform security checks or other security measures.

Garfinkel et al. \cite{ostia} developed Ostia, a sandboxing system, which uses system call hooking to secure applications.
They implemented their own ELF binary loader to load their emulation library into memory before the sandboxed program starts.
To communicate between this library and their \textit{agent}, they use Unix domain sockets.
The \textit{agent} then responds, according to its policies.
This is a similar approach to the one of this work (see Chapter~\ref{ch:manipulating-function-calls}).

Sze and Sekar \cite{spif} introduced Spif, an approach that defends against malware by tracking code and data origin on Windows systems.
They use Detours \cite{detours} to intercept low-level Windows API calls.

Kern \cite{kern2023} discusses the use of \texttt{LD\_PRELOAD} in cloud environments for HTTP deception.
This is done to analyze malware or other adversaries in real environments without their knowledge and without any risk of danger.
Examples are to override the \texttt{send} and \texttt{recv} functions of libc.

\subsection{Software Distribution}\label{subsec:software-distribution}

Guo and Engler \cite{guo2011cde} use system call hooking for creating portable software.
They developed CDE, which logs all files a program accesses during execution, including shared libraries.
All accessed files and the environment are bundled together and may now be executed on any other system having a compatible kernel without having to install any dependencies.


\subsection{Rapid Prototyping}\label{subsec:rapid-prototyping}

Spillane et al. \cite{spillane2007} use \texttt{ptrace} to hook system calls of another process to simulate these calls using a user-space program.
This is useful for rapid prototyping (e.g., file systems) because they may be developed as user-space program first, and only later on inside the kernel.


\section{Function Call Hooking}\label{sec:function-call-hooking}

All underlying techniques for function call interception on Linux systems are mentioned in Section~\ref{sec:methods-for-intercepting}.
See \texttt{ltrace} (Subsection~\ref{subsec:ltrace}), wrapper functions (Subsection~\ref{subsec:wrapper-functions}), and \texttt{LD\_PRELOAD} (Subsection~\ref{subsec:preloading}).


\section{System Call Hooking}\label{sec:system-call-hooking}

This section discusses further techniques regarding system call interception.
This excludes techniques discussed in Section~\ref{sec:methods-for-intercepting},
like \texttt{ptrace} (Subsection~\ref{subsec:ptrace}), and \texttt{strace} (Subsection~\ref{subsec:strace}).
Almost all following methods use binary rewriting to replace system calls with other instructions (except SUD, Subsection~\ref{subsec:syscall-user-dispatch}).
This is one of the reasons why they are not mentioned in Section~\ref{sec:methods-for-intercepting}.
Another reason is that this work focuses on function call interception rather than system call interception.


\subsection{\texttt{int3} Signaling}\label{subsec:int3-signaling}

\texttt{int3} is a one-byte instruction (\texttt{0xcc}) that invokes a software interrupt.
On Linux, the kernel handles it and raises \texttt{SIGTRAP} to the user-space process that executed \texttt{int3}.
The \texttt{int3} signaling technique exploits this behavior to hook system calls; it replaces \texttt{syscall}/\texttt{sysenter} with \texttt{int3} and employs the signal handler for \texttt{SIGTRAP} as the hook function.
Since \texttt{int3} is one byte, it can replace an arbitrary instruction without breaking neighboring instructions.
This technique is traditionally used in debuggers to implement breakpoints.
However, signal handling incurs a large overhead because it involves context manipulation by the kernel.
\cite{zpoline}


\subsection{Syscall User Dispatch (SUD)}\label{subsec:syscall-user-dispatch}

Syscall User Dispatch (SUD)~\cite{sud} was added in Linux 5.11, and it offers a way to redirect system calls to arbitrary user-space code.
For the SUD feature, the kernel implements a hook point at the entry point of system calls.
A user-space process can activate SUD via the \texttt{prctl} interface.
When SUD is activated, the hook point raises \texttt{SIGSYS} to the user-space process.
This mechanism allows a user-space program to leverage the \texttt{SIGSYS} signal handler as the system call hook.
However, similarly to the \texttt{int3} signaling technique, SUD imposes a significant performance penalty on the user-space program due to the overhead of the signal handling.
\cite{zpoline}


\subsection{zpoline}\label{subsec:zpoline}

zpoline is a system call hook mechanism for x86-64 CPUs.
Binary rewriting is used to replace (two-byte) \texttt{syscall}/\texttt{sysenter} instructions with a (two-byte) \texttt{callq *\%rax} instruction.
Because this instruction jumps to \texttt{rax}, where also the syscall number is stored, the trampoline code has to be initialized beginning at virtual address 0.
zpoline is exhaustive and achieves very low performance reduction (28--761 times less overhead compared to other exhaustive system call hooking techniques).
\cite{zpoline}


\subsection{DataHook}\label{subsec:datahook}

DataHook is a system call hooking technique for 32-bit programs based on glibc running on x86 or x86-64 machines.
It relies on glibc's way of performing system calls, namely a \texttt{call *\%gs:0x10} instruction to call the \texttt{\_\_kernel\_vsyscall} function.
The content of \texttt{gs:0x10} is backed up and modified to jump to a given hook function.
DataHook is only exhaustive when used on glibc-based programs.
It achieves a very low performance reduction (5--1429 times less overhead compared to existing hooking techniques).
\cite{datahook}
