Welcome to NBSoftSolutions, home of the software development company and writings of its main developer: Nick Babcock. If you would like to contact NBSoftSolutions, please see the Contact section of the about page.

Viewing WireGuard Traffic with Tcpdump

Part of the Wireguard series:

On the article, WireGuard VPN Walkthrough, Glen posted the tantalizing question:

How would you verify/confirm that the link is definitely encrypted? If you use OpenVPN and use Wireshark to sniff the packets, you see the OPENVPN protocol listed in the captured dump. Is there an equivalent for Wireguard?

For testing, here are my assumptions:

  • External Wireguard server is hosted at IP address 100.100.100.100
  • Local Wireguard interface is called wg1 at 10.192.122.2. We won’t be using wg-quick (see solution #2 if you want to setup the interface and follow along)
  • curl --interface eth0 http://httpbin.org/ip gives your external ip address (90.90.90.90)
  • curl --interface wg1 http://httpbin.org/ip gives vpn ip address (100.100.100.100)
  • If we can observe unencrypted data while listening to eth0 then the connection is not secure.

Plain HTTP is not secure so let’s watch for our request to httpbin. We’ll be able to snoop using

tcpdump -n -v -i eth0 port 80

Exectuing the eth0 curl statement, we’ll be able to clearly see our HTTP request and response:

02:58:45.452812 IP (tos 0x0, ttl 64, id 31717, offset 0, flags [DF], proto TCP (6), length 129)
    192.168.1.6.40318 > 54.235.130.91.80: Flags [P.], cksum 0x7b68 (incorrect -> 0x26e5), seq 283436622:283436699, ack 1472925892, win 229, options [nop,nop,TS val 323460534 ecr 112785182], length 77: HTTP, length: 77
        GET /ip HTTP/1.1
        Host: httpbin.org
        User-Agent: curl/7.47.0
        Accept: */*

02:58:45.504943 IP (tos 0x20, ttl 232, id 59362, offset 0, flags [DF], proto TCP (6), length 372)
    54.235.130.91.80 > 192.168.1.6.40318: Flags [P.], cksum 0x675f (correct), seq 1:321, ack 77, win 105, options [nop,nop,TS val 112785194 ecr 323460534], length 320: HTTP, length: 320
        HTTP/1.1 200 OK
        Connection: keep-alive
        Server: gunicorn/19.7.1
        Date: Thu, 26 Apr 2018 00:15:35 GMT
        Content-Type: application/json
        Access-Control-Allow-Origin: *
        Access-Control-Allow-Credentials: true
        X-Powered-By: Flask
        X-Processed-Time: 0
        Content-Length: 33
        Via: 1.1 vegur

        { 
          "origin": "90.90.90.90"
        }
  • We see our eth0 address (192.168.1.6) talking to httpbin’s server at 54.235.230.91
  • The response contains our expected external ip address (90.90.90.90)
  • And oh no, we’re snooping!

Let’s look at what executing the wg1 curl will look like with:

tcpdump -n -v -i wg1 port 80
tcpdump: listening on wg1, link-type RAW (Raw IP), capture size 262144 bytes
03:10:13.382048 IP (tos 0x0, ttl 64, id 26588, offset 0, flags [DF], proto TCP (6), length 129)
    10.192.122.2.42904 > 54.225.185.38.80: Flags [P.], cksum 0x753d (incorrect -> 0x2b18), seq 354179898:354179975, ack 3265920867, win 216, options [nop,nop,TS val 323632516 ecr 2420503305], length 77: HTTP, length: 77
        GET /ip HTTP/1.1
        Host: httpbin.org
        User-Agent: curl/7.47.0
        Accept: */*

03:10:13.861353 IP (tos 0x0, ttl 234, id 3103, offset 0, flags [DF], proto TCP (6), length 371)
    54.225.185.38.80 > 10.192.122.2.42904: Flags [P.], cksum 0xf0d4 (correct), seq 1:320, ack 77, win 105, options [nop,nop,TS val 2420503425 ecr 323632516], length 319: HTTP, length: 319
        HTTP/1.1 200 OK
        Connection: keep-alive
        Server: gunicorn/19.7.1
        Date: Thu, 26 Apr 2018 00:27:03 GMT
        Content-Type: application/json
        Access-Control-Allow-Origin: *
        Access-Control-Allow-Credentials: true
        X-Powered-By: Flask
        X-Processed-Time: 0
        Content-Length: 32
        Via: 1.1 vegur

        { 
          "origin": "100.100.100.100"
        }
  • We see our wg1 address (10.192.122.2) talking to httpbin’s server at 54.235.230.91
  • The response contains our VPN’s external ip address (100.100.100.100)
  • And oh no, we’re snooping! Or are we?

Don’t worry, in this example we’re sending plaintext to a Wireguard interface and receiving plaintext back, which is what our tcpdump command is showing. However, our Wireguard interface doesn’t actually have the capability to send network data anywhere. It’ll have eth0 transmit the encrypted payload to our VPN server. This means that listening on eth0 but executing the wg1 curl will show encrypted contents. If eth0 can’t read the contents then no one else will either.

We’ll update our tcpdump command as we won’t be communicating TCP over port 80. We know we’ll be communicating with our VPN server, so only capture traffic between us and the server.

tcpdump -n -X -i eth0 host 100.100.100.100

Since we’ll be seeing encrypted packets, they won’t be printable. To display the contents, we’ll view the data hex encoded (which is the -X option).

I’ve also removed the IPv4 header and UDP header, so we can just focus on the data. Below is the our HTTP GET request.

0x0000:  .... .... .... .... .... .... .... ....  ................
0x0010:  .... .... .... .... .... .... 0400 0000  ................
0x0020:  bef2 24a4 0600 0000 0000 0000 002f b736  ..$........../.6
0x0030:  7448 8e01 778f 7e13 adb8 e66c e307 3d39  tH..w.~....l..=9
0x0040:  bfbf f53d a194 211b f0e5 6cab c561 1f5c  ...=..!...l..a.\
0x0050:  2c38 906b 1bec 183a 8e41 8bab 3a59 ca0b  ,8.k...:.A..:Y..
0x0060:  e1ef dda8 e882 4f8a 6590 f517 2d9a 2077  ......O.e...-..w
0x0070:  4830 8673 b26b 1a16 7f3b e358 f3ec b14f  H0.s.k...;.X...O
0x0080:  8d97 b15d 9ad3 3962 3e1f 5d6c 96be 0518  ...]..9b>.]l....

Notice that the data starts with 0400 0000. If we cross reference this with Wireguard’s documented protocol, we can confirm that the data begins with an 8bit 4 followed by 24 bits of 0, so we can rest assured that we’ve set up Wireguard correctly. One could dig a little deeper in the subsequent bits to capture the receiver index part of the protocol, but as a heuristic, 0400 0000 is decent. Keep in mind Wireguard doesn’t try to obsfuscate data, so an internet provider could reasonably try to detect and block Wireguard traffic.

The creator of wireguard had this to say:

WireGuard does not aim to evade DPS [deep packet inspection], unfortunately. There are several things that prevent this from occurring:

  • The first byte, which is a fixed type value.
  • The fact that mac2 is most often all zeros.
  • The fixed length of handshake messages.
  • The unencrypted ephemeral public key.

So Wireguard isn’t the panacea for those trying to evade sophisticated and unfriendly firewalls (and Wireguard never billed itself as that). It’s a great VPN that can be combined with other tools to match one’s desired needs.


Lessons learned: ZFS, databases, and backups

The graves of data without backups

I have a self-hosted Nextcloud for cloud storage installed on a ZFS Raid-6 array. I use rclone to keep my laptop in sync with my cloud. I was setting up a new computer wanted a local copy of the cloud, so I executed rclone sync . nextcloud:. This ended up deleting a good chunk of my cloud files. The correct command was rclone sync nextcloud: .. The manual for rclone sync includes this snippet:

Important: Since this can cause data loss, test first with the –dry-run flag to see exactly what would be copied and deleted.

✅ - Lesson: Prefer rclone copy or rclone copyto where possible as they do not delete files.

Oof. Now that I just deleted a bunch of files, it became a test to see if I could restore them. Since I use zfs-auto-snapshot I figured rolling back to the most recent snapshot would fix the problem. So I logged onto the server to see zfs list

NAME                        USED  AVAIL  REFER  MOUNTPOINT
tank                       1.03T  9.36T   941G  /tank

I have only a single ZFS dataset. So if I rolled back to a snapshot, I’d be rolling back every single application, database, media files to a certain point in time. Since I just executed the erroneous rclone command, I thought it safe to rollback everything to previous snapshot taken a few prior. So I did it.

✅ - Lesson: Use more datasets. Datasets are cheap and configured to have different configuration (sharing, compression, snapshots, etc). The FreeBSD handbook on zfs states:

The only drawbacks to having an extremely large number of datasets is that some commands like zfs list will be slower, and the mounting of hundreds or even thousands of datasets can slow the FreeBSD boot process. […] Destroying a dataset is much quicker than deleting all of the files that reside on the dataset, as it does not involve scanning all of the files and updating all of the corresponding metadata.”

I regretted rolling back. I opened up Nextcloud to see a blank screen. Nextcloud relies on MySQL and logs showed severe MySQL errors. Uh oh, why would MySQL be broken when it had been working at the provided snapshot? MySQL wouldn’t start. Without too much thought I incremented innodb_force_recovery all the way to 5 to get it to start, but then no data was visible. I had no database backups.

✅ - Lesson: Always make database backups using proper database tools (mysqldump, pg_dumpall, .backup). Store these in a snapshotted directory in case you need to rollback the backup.

So I scrapped that database, but why had it gone awry? Here I only have hypotheses. The internet is not abundant in technicians diagnosing why a file system snapshot of a database failed, but here are some good leads. A zfs snapshot is not instantaneous. A database has a data file and several logs that ensure that power loss doesn’t cause any corruption. However, if the database and these logs get out of sync (like they might with a snapshot), you might see the database try and insert data into unavailable space. I say “might” because with a low volume application or snapshotting at just the right time, the files may be in sync and you won’t see this problem.

✅ - Lesson: If you are taking automatic zfs snapshots do not take snapshots of datasets containing databases: zfs set com.sun:auto-snapshot=false tank/containers-db

I went back through the initial installation for Nextcloud. Thankfully, it recognized all the files restored from the snapshot. I thought my troubles were over, but no such luck. I wrote an application called rrinlog that ingests nginx logs and exposes metrics for Grafana (previously blogged: Replacing Elasticsearch with Rust and SQLite). This application uses SQLite with journal_mode=WAL and I started noticing that writes didn’t go through. They didn’t fail, they just didn’t insert! Well, from the application’s perspective, the data appear to insert, but I couldn’t SELECT them. A VACUUM remarked that the database was corrupt.

✅ - Lesson: SQLite, while heavily resistant to corruption, is still susceptible, so don’t forget to backup SQLite databases too!

Maybe it’s a bug in the library that I’m using or maybe it’s a SQLite bug. An error should have been raised somewhere along the way, as I could have caught the issue earlier and not lost as much data. Next step was to recover what data I had left using .backup. Annoyingly, this backup ended with a ROLLBACK statement, so I needed to hand edit the backup.

After these trials I’ve changed my directory structure a little bit and applied all the lessons learned:

|- applications (snapshotted)
|- databases (not snapshotted)
|- database-backups (snapshotted)

It’s always a shame when one has to undergo a bit of stress in order to realize best practices, but the hope is that by having this experience, I should apply these practices in round 1 instead of round 2.


Routing Select Docker Containers through Wireguard VPN

Wouldn’t be a docker post without an image of a ship

Part of the Wireguard series:

Scenario: You have a host running many Docker containers. Several sets of these containers need to route traffic through different VPNs. Below I’ll describe my solution that doesn’t resort to VMs and doesn’t require modification to any docker images.

This post assumes that one has already set up working wireguard servers, and will focus only on client side. For a quick wireguard intro: see WireGuard VPN Walkthrough.

Solution #1

If you’re familiar with the openvpn client trick then this will look familiar. We’re going to create a Wireguard container and link all desired containers to this Wireguard container.

First we’re going to create a Wireguard Dockerfile:

FROM ubuntu:16.04

RUN apt-get update && \
    apt-get install -y software-properties-common debconf-utils iptables curl && \
    add-apt-repository --yes ppa:wireguard/wireguard && \
    apt-get update && \
    echo resolvconf resolvconf/linkify-resolvconf boolean false | debconf-set-selections && \
    apt-get install -y iproute2 wireguard-dkms wireguard-tools curl resolvconf

COPY wgnet0.conf /etc/wireguard/.
COPY startup.sh /.

EXPOSE <PORT-LIST?>

ENTRYPOINT ["/startup.sh"]

Some notes about this Dockerfile:

  • One needs to supply wgnet0.conf (described in previously linked post), as it’ll contain VPN configuration
  • The EXPOSE contains all the applications in the VPN that will need their port exposed. Feel free to omit EXPOSE as it’s completely optional
  • The startup.sh script connects to the VPN, verifies that the container’s IP address is the same VPN server’s, and if not to stop the container. Implementation below.
#!/bin/bash
set -euo pipefail

wg-quick up wgnet0

VPN_IP=$(grep -Po 'Endpoint\s=\s\K[^:]*' /etc/wireguard/wgnet0.conf)

function finish {
    echo "$(date): Shutting down vpn"
    wg-quick down wgnet0
}

# Our IP address should be the VPN endpoint for the duration of the
# container, so this function will give us a true or false if our IP is
# actually the same as the VPN's
function has_vpn_ip {
    curl --silent --show-error --retry 10 --fail http://checkip.dyndns.com/ | \
        grep $VPN_IP
}

# If our container is terminated or interrupted, we'll be tidy and bring down
# the vpn
trap finish TERM INT

# Every minute we check to our IP address
while [[ has_vpn_ip ]]; do
    sleep 60;
done

echo "$(date): VPN IP address not detected"

Some notes:

  • Every minute we check to see what our IP address is from dyndns.com. This is the same endpoint that the dynamic dns client, ddclient, uses.
  • A more efficient kill switch would integrate with ip tables to ensure that all traffic is routed via the VPN. But don’t use the example provided in wg-quick as that will block traffic from the host (192.168.1.x) to the containers. I have a cool trick later on with port forwarding.
  • When the script exits, it brings down the VPN

The best way to see this in action is through a docker compose file. We’ll have grafana traffic routed through the VPN.

version: '3'
services:
  wireguard:
    container_name: 'wireguard'
    build: .
    restart: 'unless-stopped'
    sysctls:
      - "net.ipv4.conf.all.rp_filter=2"
    cap_add:
      - net_admin 
      - sys_module
    ports:
      - '3000:3000' # grafana
  grafana:
    container_name: 'grafana'
    image: 'grafana/grafana'
    restart: 'unless-stopped'
    network_mode: "service:wireguard"

Notes:

  • Just like the OpenVPN solution, we need the NET_ADMIN capability
  • Since Wireguard is a kernel module we need the SYS_MODULE capability too
  • See the sysctls configuration? This affects only the container’s networking.
  • All ports exposed for the wireguard container are the ones that would normally be exposed in other services. For instance, I have wireguard exposing the grafana port 3000.
  • network_mode: "service:wireguard" is the magic that has grafana use the wireguard vpn

When dependant services bind to wireguard’s network they are binding to that container’s id. If you rebuild the wireguard container, you’ll need to rebuild all dependant containers. This is somewhat annoying. Ideally, they would bind to whatever container’s network that had the name of wireguard.

Quick quiz, which of these addresses will resolve to our grafana instance (taken from the host machine)?

localhost
[host-ip:192.168.1.6]
[vpn-ip:10.192.122.2]
[vpn-external-ip]
[wireguard-docker-interface:172.19.0.1]
[docker0:172.17.0.1]

Correct answers:

localhost
[wireguard-docker-interface:172.19.0.1]
[docker0:172.17.0.1]

If I hadn’t ran the experiment, I would have gotten this wrong!

If we log onto the VPN server, we see that only curling only our client IP address will return Grafana. This is good news, it means we are not accidentally exposing services on our VPN’s external IP address. It also allows a cool trick to see the services locally through the host machine without being on the VPN. Normally one would would put in http://host-ip.com:3000 to see Grafana, but as we just discovered, that no longer routes to Grafana because it lives on the VPN. We can, however, ssh into the host machine and port forward localhost:3000 to great success!

The end result gives me a good feeling about the security of the implementation. There is no way someone can access the services routed through the VPN unless they are also on the VPN, they are on the host machine, or port forward to the host machine. We have a rudimentary kill switch as well for some more comfort.

Solution #2

Our second solution will involve installing Wireguard on the host machine. This requires gcc and other build tools, which is annoying as the whole point of docker is to keep hosts disposable, but we’ll see how this solution shakes out as it has some nice properties too.

Initial plans were to follow Wireguard’s official Routing & Network Namespace Integration, as it explicitly mentions docker as a use case, but it’s light on docker instructions. It mentions only using the pid of a docker process. This doesn’t seem like the “docker” approach, as it’s cumbersome. If you are a linux networking guru, I may be missing something obvious, and this may be your most viable solution. For mere mortals like myself, I’ll show a similar approach, but more docker friendly.

First, wg-quick has been my crutch, as it abstracts away some of the routing configuration. Running wg-quick up wgnet0 to have all traffic routed through the Wireguard interface is a desirable property, and it was a struggle to figure out how to route only select traffic.

For those coming from wg-quick we’re going to be doing things manually, so to avoid confusion, I’m going to be creating another interface called wg1. Our beloved DNS and Address configurations found in wgnet0.conf have to be commented out as they are wg-quick specific. These settings are explicitly written in manual invocation:

ip link add dev wg1 type wireguard
wg setconf wg1 /etc/wireguard/wg1.conf
ip address add 10.192.122.2/24 dev wg1
ip link set up dev wg1
printf 'nameserver %s\n' '10.192.122.1' | resolvconf -a tun.wg1 -m 0 -x
sysctl -w net.ipv4.conf.all.rp_filter=2

At this point if your VPN is hosted externally you can test that the Wireguard interface is working by comparing these two outputs:

curl 'http://httpbin.org/ip'
curl --interface wg1 'http://httpbin.org/ip'

For my future self, I’m going to break down what just happened by annotating the commands.

# Create a wireguard interface (device) named `wg1`. The kernel knows what a 
# wireguard interface is as we've already installed the kernel module
ip link add dev wg1 type wireguard

# Point our new wireguard interface at the VPN server and allocate addresses
# for the interface
wg setconf wg1 /etc/wireguard/wg1.conf
ip address add 10.192.122.2/24 dev wg1

# Start the interface and add the VPN server as our DNS nameserver. This is so
# our VPN will resolve hostnames like httpbin.org or google.com.
ip link set up dev wg1
printf 'nameserver %s\n' '10.192.122.1' | resolvconf -a tun.wg1 -m 0 -x

# rp_filter is reverse path filtering. By default it will ensure that the
# source of the received packet belongs to the receiving interface. While a nice
# default, it will block data for our VPN client. By switching it to '2' we only
# drop the packet if it is not routable through any of the defined interfaced.
sysctl -w net.ipv4.conf.all.rp_filter=2

Now for the docker fun. We’re going to create a new docker network for our VPN docker containers:

docker network create docker-vpn0 --subnet 10.193.0.0/16

Now to route traffic for docker-vpn0 through our new wg1 interface:

ip rule add from 10.193.0.0/16 table 200
ip route add default via 10.192.122.2 table 200

My layman understanding is that we mark traffic from our docker subnet as “200”, kinda like fwmark. We then set the default route for the docker subnet to our wg1 interface. The default route allows the docker subnet to query unknown IPs and hosts (ie. everything that is not a docker container in the 10.193.0.0/16 space). By having the route be more specific, as it mentions a table, data is routed through wg1 instead of eth0.

You can test it out with:

docker run -ti --rm --net=docker-vpn0 appropriate/curl http://httpbin.org/ip

Once we docker network remove docker-vpn0, we slim down our docker compose file.

version: '3'
services:
  grafana:
    container_name: 'grafana'
    image: 'grafana/grafana'
    restart: 'unless-stopped'
    ports:
      - '3000:3000'
    dns: '10.192.122.1'
    networks:
      wireguard: {}
networks:
  wireguard:
    ipam:
      config:
        - subnet: 10.193.0.0/16

Now when we bring up grafana, it will automatically be connected through the VPN thanks to the subnet routing.

If we want to bring the VPN up on boot, we need to create /etc/network/interfaces.d/wg1 with encoded commands:

auto wg1
iface wg1 inet manual
pre-up ip link add dev wg1 type wireguard
pre-up wg setconf wg1 /etc/wireguard/wg1.conf
pre-up ip address add 10.192.122.2/24 dev wg1
up ip link set up dev wg1
post-up /bin/bash -c "printf 'nameserver %s\n' '10.192.122.1' | resolvconf -a tun.wg1 -m 0 -x"
post-up ip rule add from 10.193.0.0/16 table 200
post-up ip route add default via 10.192.122.2 table 200
post-up sysctl -w net.ipv4.conf.all.rp_filter=2
post-down ip link del dev wg1

The auto wg1 is what starts the interface automatically on boot, else we’d have to rely on ifup and ifdown. Everything else should look familiar.

The last thing that needs mentioning is the kill switch. We’ve seen calling curl inside our networked docker container. We could use this to periodically check the IP address is as expected. But I know we can do better, but I can’t quite yet formulate a complete solution, so I’ll include my work in progress.

We can deny all traffic from our subnet with the following:

ip route add unreachable 0.0.0.0/0 table 200

But how to run this command when the VPN disintegrates? I’ve thought about putting it in a post-down step for wg1, but I don’t think it’s surefire approach. The scary part is if wg1 goes down, the docker ip table rule is no longer effective, so instead of dropping packets for an interface that no longer exists, they are sent to the next applicable rule which is eth0! We have to be smarter. For reference, the kill switch used as an example in wg-quick:

PostUp = iptables -I OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -j REJECT
PreDown = iptables -D OUTPUT ! -o %i -m mark ! --mark $(wg show %i fwmark) -j REJECT

The end result should look something like this. It will be more complete than any curl kill switch. I didn’t want to bumble through to a halfway decent solution, so I called it a night! If I come across the solution or someone shouts it at me, I’ll update the post.

Conclusion

I think both solutions should be in one’s toolkit. At this stage I’m not sure if there is a clear winner. I ran the first solution for about a week. It can feel like a bit of a hack, but knowing that everything is isolated in the container and managed by docker can be a relief.

I’ve been running the second solution for about a day or so, as I’ve only just figured out how all the pieces fit together. The solution feels more flexible. The apps can be more easily deployed anywhere, as there is nothing encoded about a VPN in the compose file. The only thing that will stand out as different is the networking section. I also find pros and cons to having the VPN managed by the host machine. On one hand, having all the linux tools and wg show readily available to monitor the tunnel is nice. Like collectd, it’s easiest to report stats on top level interfaces. But on the other hand, installing build tools is annoying and managing routing tables makes me anxious if I think too much about it. In testing these solutions, I’ve locked myself out of a VM more than once – forcing a reboot, and I don’t take rebooting actual servers lightly.

Only time will tell which solution is best, but I thought I should document both.