A tool agnostic wrapper around standard Linux tools that deal with the HTTP(S) protocol: curl, wget, wfuzz. It can also create raw HTTP(S) requests. The lower level bash commands are derived from structured yet compact XML language definitions.
-
Install docker:
sudo apt install docker -
Clone this repository:
git clone [email protected]:1LAB9fJYvmL9FUQREiUc2Rz7weVCm42qBs/metahttp.git -
With
dockerdrunning, build the docker image from the Dockerfile:
cd metahttp; docker build -t "ubuntu1604/socat/metahttp" . -
Run a container based on the image:
docker run --name=ubuntu1604-socat-metahttp --detach -it -p 127.0.0.1:50774:50774/tcp -p 127.0.0.1:50774:50774/udp ubuntu1604/socat/metahttp -
As a sanity check, see if your localhost now exposes ports 50774/tcp and 50774/udp:
netstat -antup | grep 50774tcp 0 0 127.0.0.1:50774 0.0.0.0:* LISTEN 588215/docker-proxy
udp 0 0 127.0.0.1:50774 0.0.0.0:* 588238/docker-proxy
The metahttp.xml files are the basis of an HTTP session. Requests are derived from the contained metadata and integrated into bash scripts. The following examples assume that you have wget or curl installed.
As a first simple example we create a GET request against http://www.eff.org
meta/eff.org.metahttp.xml:
<session newcookies="true" baseurl="https://www.eff.org" stdout="-" xmlns="urn:1LAB9fJYvmL9FUQREiUc2Rz7weVCm42qBs">
<req tool="wget" verbose="false">
<header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0"/>
<header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/>
<form method="GET" action="/" enctype="application/x-www-form-urlencoded"/>
</req>
</session>
Now, in order to compile this meta data to bash instructions, run the following command (requires nc or ncat or socat on your system, telnet will do too):
cat meta/eff.org.metahttp.xml | ./metahttp.sh
which will generate the following output:
#!/bin/bash
rm -f cookies.txt; touch cookies.txt
echo ------------------------------------------------------------ wget GET 'https://www.eff.org/' :
wget \
--server-response \
--output-document - 2>&1 \
--header 'User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0' \
--header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
--load-cookies cookies.txt \
--save-cookies cookies.txt --keep-session-cookies \
--no-check-certificate \
--header 'Host: www.eff.org' \
--header 'Content-Type: application/x-www-form-urlencoded' \
$'https://www.eff.org/' \
For now, we just copy and paste the wget command and its arguments (the 11 lines following wget (including the last empty line)) in order to kick off the HTTP request:
wget \
--server-response \
--output-document - 2>&1 \
--header 'User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0' \
--header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
--load-cookies cookies.txt \
--save-cookies cookies.txt --keep-session-cookies \
--no-check-certificate \
--header 'Host: www.eff.org' \
--header 'Content-Type: application/x-www-form-urlencoded' \
$'https://www.eff.org/' \
... and we receive the HTTP response, similar to the following:
--2020-05-22 15:13:33-- https://www.eff.org/
Resolving www.eff.org (www.eff.org)... 151.101.112.201, 2a04:4e42:1b::201
Connecting to www.eff.org (www.eff.org)|151.101.112.201|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 55046
Server: nginx
Content-Type: text/html; charset=utf-8
(... the whole raw HTTP response following here...)
As a first example for a POST request we chose the search platform duckduckgo.com, as it allows for a straightforward search without a lot of background noise:
meta/duckduckgo.metahttp.xml:
<session newcookies="true" baseurl="https://duckduckgo.com" stdout="-" xmlns="urn:1LAB9fJYvmL9FUQREiUc2Rz7weVCm42qBs">
<req tool="curl" protocol="http/1.1" verbose="false">
<header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0"/>
<header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/>
<header name="Accept-Language" value="en-US,en;q=0.5"/>
<header name="Referer" value="https://duckduckgo.com/"/>
<header name="Connection" value="close"/>
<header name="Pragma" value="no-cache"/>
<header name="Cache-Control" value="no-cache"/>
<form method="POST" action="/html" enctype="application/x-www-form-urlencoded">
<input name="q" value="wfuzz session" urlencode="false"/>
</form>
</req>
</session>
Again, we compile this metadata, this time redirecting the output to a bash script:
cat meta/duckduckgo.metahttp.xml | ./metahttp.sh >duckduckgo.sh
This will result in the following file which you can make executable by issuing chmod +x duckduckgo.sh:
#!/bin/bash
rm -f cookies.txt; touch cookies.txt
echo ------------------------------------------------------------ curl POST 'https://duckduckgo.com/html' :
curl \
--include \
--request POST \
--http1.1 \
--header 'User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0' \
--header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
--header 'Accept-Language: en-US,en;q=0.5' \
--header 'Referer: https://duckduckgo.com/' \
--header 'Connection: close' \
--header 'Pragma: no-cache' \
--header 'Cache-Control: no-cache' \
--cookie cookies.txt \
--cookie-jar cookies.txt \
--header 'Host: duckduckgo.com' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-binary $'q=wfuzz session' \
$'https://duckduckgo.com/html' \
Again, when you run the bash script, you will see the complete HTTP response in your terminal.
As pen testers, we like to use a proxy in order to position ourselves as MITM (man in the middle). This helps us analyze/repeat/modify requests in detail.
Our next request (duckduckgo.proxy.metahttp.xml) will utilize a Burpsuite proxy on localhost port 8080:
<session newcookies="true" baseurl="https://duckduckgo.com" proxy="http://127.0.0.1:8080" stdout="-" xmlns="urn:1LAB9fJYvmL9FUQREiUc2Rz7weVCm42qBs">
<req tool="curl" insecure="true" protocol="http/1.1" verbose="false" useproxy="true">
<header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0"/>
<header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/>
<header name="Accept-Language" value="en-US,en;q=0.5"/>
<header name="Accept-Encoding" value="gzip, deflate"/>
<header name="Referer" value="https://duckduckgo.com/"/>
<header name="Connection" value="close"/>
<header name="Pragma" value="no-cache"/>
<header name="Cache-Control" value="no-cache"/>
<form method="POST" action="/html" enctype="application/x-www-form-urlencoded">
<input name="q" value="wfuzz session"/>
</form>
</req>
</session>
We transform the XML and save the output to file duckduckgo.proxy.sh
cat meta/duckduckgo.proxy.metahttp.xml | ./metahttp.sh >duckduckgo.proxy.sh
The contents of duckduckgo.proxy.sh are:
#!/bin/bash
rm -f cookies.txt; touch cookies.txt
echo ------------------------------------------------------------ curl POST 'https://duckduckgo.com/html' :
curl \
--proxy http://127.0.0.1:8080 \
--include \
--request POST \
--http1.1 \
--header 'User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0' \
--header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
--header 'Accept-Language: en-US,en;q=0.5' \
--header 'Accept-Encoding: gzip, deflate' \
--header 'Referer: https://duckduckgo.com/' \
--header 'Connection: close' \
--header 'Pragma: no-cache' \
--header 'Cache-Control: no-cache' \
--cookie cookies.txt \
--cookie-jar cookies.txt \
--ssl-no-revoke --insecure \
--header 'Host: duckduckgo.com' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-binary $'q=wfuzz%20session' \
$'https://duckduckgo.com/html' \
Note: Why do we explicitly tell this request to be insecure?
Under normal circumstances, we'd always want a TLS request to be as secure as possible. However, we're in a completely different context here: We've put a proxy between us and the HTTPS server, in order to act as MITM. That means that we explicitly chose an insecure setup, with the proxy exchanging the actual target site certificate with its own (Portswigger) one.
We make the file executable and run it: chmod +x duckduckgo.proxy.sh && ./duckduckgo.proxy.sh
... which results in the output of the HTTP response - not only in our terminal, but also in Burpsuite proxy.
We can even conveniently view the rendered web page on the proxy's response tab:
Once you have made yourself familiar with metahttp, please keep reading ADVANCED.md