Welcome to NBSoftSolutions, home of the software development company and writings of its main developer: Nick Babcock. If you would like to contact NBSoftSolutions, please see the Contact section of the about page.

IPv6 only networking not viable in 2018

Get on board the IPv6 train

Quick rant about the roadblocks that IPv6 only networking has caused:

Gist of it:

  • launchpad.net is ipv4, so adding Ubuntu apps via apt-add-repository or by an apt list (that reference launchpad.net) will fail. This severely hinders acquiring software that isn’t distributed by Ubuntu.
  • keyserver.ubuntu.com is ipv4 so one can’t use the default method for trusting package keys (something that shouldn’t be taken lightly)
  • thus everything ubuntu documents about packaging software is not possible via ipv6
  • github.com is ipv4 so one needs to either employ a proxy, nat, or manual transfer files

It can be quite crippling to not have access to these resources.

The full story:

I was checking out Vultr for a VPS. They offer a $2.50 a month package – the catch is that it is IPv6 only (inbound and outbound traffic must use IPv6 addresses). I figured this was no big deal. IPv6 has been in draft standard since 1998, and while it only became an internet standard in 2017, people have had 20 years to prepare. It’s not like you even have to abandon IPv4, one can serve a site with IPv6 and IPv4 side by side.

I got a kick out of IPv6 Excuse Bingo, and figured I should give IPv6 only a shot.

I wanted to setup algo, a personal VPN, and since I was just playing around, I wanted to keep costs down (hence the $2.50 a month package).

Following algo’s Ubuntu 18.04 server guide, I hit a snap on the first line:

apt-add-repository ppa:ansible/ansible

The line would error with

Cannot add PPA: 'ppa:~ansible/ubuntu/ansible'.
ERROR: '~ansible' user or team does not exist.

Searching online for this error would reveal questions only related to setting up an apt proxy. This and many other rabbit holes kept me scratching my head. It turns out, this is the error message that is given to users by apt-add-repository when it can’t access either 91.189.89.22 or 91.189.89.23 (bug report). With IPv6 only networking, it is not possible without some intermediate translation to access those addresses.

I soldiered on, maybe I could circumvent this issue.

I go to manually add the ansible apt key

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 7BB9C367

With the lovely error:

gpg: keyserver receive failed: No keyserver available

A bug: keyserver.ubuntu.com has no IPv6 connectivity. The recommended workaround is to switch out the host and execute:

apt-key adv --keyserver pool.sks-keyservers.net --recv-keys 7BB9C367

I’m not familiar with sks-keyservers.net and retrieving keys from a domain I’m not familiar with sets off major red flags. Still, I continue and get ansible installed.

Next issue:

git clone https://github.com/trailofbits/algo

Github doesn’t support IPv6, so I downloaded the repo on another box and transferred accordingly.

I started running algo and then I noticed that it executed the following lines to set up Wireguard (obvious in hindsight):

add-apt-repository ppa:wireguard/wireguard
apt-get update
apt-get install wireguard

These commands fail as add-apt-repository needs to contact an IPv4 address.

And it presented a high enough barrier that I quit (as add-apt-repository won’t work with IPv6). I realized that if I’m having this much trouble setting up the box, I’d have even more trouble when routing traffic through it as a VPN. There were some things I did try or could try to try and soften the problem:

  • Use NAT64, but that would require me to setup another box.
  • Use an apt proxy, but that would also require another box
  • Try appending sixxs.org to domain name for IPv4 exclusive addresses, but sixxs has been sunset.

The solution is simple: upgrade to IPv4. Yes it’ll cost more, but $5 vs $2.50 isn’t something that should cause major headaches.

I can only hope that IPv6 becomes more popular soon.


Satisfying NAS share and backup solution

Data graves are probably more common than backups

A short and to the point article.

I have:

  • A NAS configured with 5 disks in raidz2 with auto snapshots
  • Docker applications like gitea running on the NAS
  • Users who access the NAS over a CIFs share via their client of choice (Mac or Windows)
  • Users who also have documents stored in cloud providers like Microsoft’s Onedrive

This setup is ok, but there are many improvements to make.

Using a cloud file hosting service requires a user to interact with the latest version of a file. This means that accidental overwrites or deletion can result in frustrating data recovery. Rclone will allow us to pull all the user’s file into a local directory on the CIFs share that is automatically snapshotted. After initial configuration, keeping the local directory in sync is as simple as:

rclone sync onedrive:/ /tank/nick/onedrive

Execute hourly or daily. We don’t need realtime syncing of files. The local copy on the NAS isn’t meant to be worked on, as any modifications won’t be replicated out to the cloud (too big of a hassle for a potential of merge conflicts). Thus we can make the network folder readonly. Below I show a service account creating a directory where only the service account has write access. The group is “nas” to allow for nas users access to the location.

sudo -u dumper mkdir --mode=u+rwx,g+rs,o-rwx -p "$location"
sudo -u dumper chown dumper.nas "$location"

Now users can view their cloud data on the network (performance boost) and rest easy knowing that their data can be recovered. Users wishing to add or modify data can still use official clients like the Onedrive client built into Windows.

Cloud file providers have a limited amount of free space (rightfully so) and those that want to store sensitive data may be rubbed the wrong way about “giving up” their data. A fix is to spin up a Nextcloud instance on the NAS and allow users to store documents there (with end-to-end encryption). Users can use the web interface or the official Nextcloud apps to interact with their files.

Nextcloud can be used as a complementary solution. Onedrive still has excellent online editing and sharing capabilities within the Microsoft ecosystem. So keep files on the cloud provider or on the network share as appropriate. You can have your cake and eat it too. We can use rclone to sync a user’s nextcloud directory to the network share as a readonly directory. It might seem odd to rclone a Nextcloud instance on the same box, but it future proofs operations in case Nextcloud is moved offsite and keeps things consistent with all file providers, self hosted or otherwise.

Since we’re self hosting data on our own servers, things are 1000x more likely to go wrong that could result in irreversible data loss (human error, theft, accident, damage, nature, etc). We need a backup solution; restic to the rescue. Restic will obscure file names, encrypt data, deduplicate, and create snapshots. The only thing you need to supply is the directory and the location of the backup. Restic works with a host of providers, so we’ll go with the cheapest: Backblaze B2. Invoking the backup is quite easy:

restic -r b2:<backblaze-id>:/repositories backup /tank/containers/gitea-app/git/repositories

While you can have restic backup everything to the root directory, I’ve decided to break apart backups logically. Such that apps and users get their own directory in case they’d like to encrypt with a different password, but it also cuts down noise when restoring a backup.

Backblaze serves as a great backup solution for docker application data. I haven’t moved to backing up databases there (that’s the next step), but so far it has been working great as a git repo backup.

It should go without saying that one should only back up important data to conserve on costs. Security camera footage, media files (home movies can be exempt!), and any data that can be treated as disposable are fine to not back up. Raidz2 zfs is be enough of a “backup”.

So what do we have:

  • Microsoft Onedrive (free tier):
    • Store office documents, email attachments (auto saved from outlook), and any files that need to be externally shared.
    • Cloned to a readonly directory on the network. This directory is zfs snapshotted to provide additional data recovery
    • Users modify files on Onedrive using official clients
    • Doesn’t need to be backed up to Backblaze B2 (optional)
  • Nextcloud
    • Provides as much storage as your NAS can handles with additional privacy and security benefits
    • Cloned to a readonly directory on the network. This directory is zfs snapshotted to provide additional data recovery
    • Users modify files on Nextcloud using official clients
    • Backed up to Backblaze
  • Network Share (CIFs):
    • Has a complete view of all data (at least read only)
    • zfs snapshot as backup
    • Select folders are backed up to B2 (user data, some container data)
  • Backblaze B2
    • Cheapest object storage out there
    • Everything backed up is encrypted with file names obscured

Migrating to Actix Web from Rocket for Stability

Actors in Actix, actors in life

I previously wrote an article back in November 2017: Replacing Elasticsearch with Rust and SQLite. In it, I needed to create a few HTTP endpoints that ingested JSON, perform a database lookup, and return JSON. Very simple. No query / path parameters, authentication, authorization, H2, or TLS. Any v0.1 microframework would do (Project repo).

I went with Rocket and I knew what I was getting into back then:

[Rocket] has the best testing story, serde support, and contains minimal boilerplate. The downside is that nightly Rust is required.

I became enamored with succint endpoints (a differentiating feature at the time):

#[post("/search", format = "application/json", data = "<data>")]
fn search(data: Json<Search>) -> Json<SearchResponse> {
    debug!("Search received: {:?}", data.0);
    Json(SearchResponse(vec![
        "blog_hits".to_string(),
        "sites".to_string(),
        "outbound_data".to_string(),
    ]))
}

I didn’t understand how stability was such an important feature. I was familiar with needing new versions of the nightly compiler to stay current with clippy and rustfmt, but it was a blindspot when it came to dependencies.

Six Months Later

It’s been six months since the article. There have been a half dozen updates to Rocket and numerous updates to the nightly compiler. Since I only revisit the project about once every other week, I was always met with a compiler error, as I had most likely updated the nightly compiler in the meantime to grab a new rustfmt or clippy version, and the Rocket version wouldn’t work with that nightly. Syncing Rocket and the nightly compiler to the latest version normally fixed the issue. I’m very thankful that Rocket is maintained to this high degree.

There was a time when the latest Rocket broke on nightly because the nightly version broke ring, a dependency. A combination of cross linking, alpha versions, module paths, and herd mentality led to a temporary impasse. The only solution was to pin the nightly version of Rust with a specific version of Rocket. Everyone worked together and eventually resolved the issue. Still, I find myself wary.

This did have the side effect of me pinning the nightly compiler via rustup override. Though, I came back a couple weeks later to update dependencies (including Rocket) and this time my usual incantation of cargo +nightly build failed. Syncing versions didn’t help. It took me an embarrassingly long amount of time to realize that I needed to either update the nightly pin or unpin.

I don’t want a project where I have to remember its unique setup. I like it when a cargo build or cargo build --all is all I need.

Actix-Web

In these last six months, I have been really impressed with the actix.rs project, specifically actix web. It’s a relatively new project, in fact the first released version on crates.io was shortly before I started working on my project. To me, it has everything Rocket has to offer but it also compiles on stable. As a plus, it integrates with tokio for asynchronous endpoints. I don’t utilize this feature, but it’s nice to know actix is closely tracking the future of scalable Rust networking.

I can’t overstate how similar actix web endpoints resemble Rocket endpoints. The following diff is the migration for the endpoint posted earlier:

- #[post("/search", format = "application/json", data = "<data>")]
  fn search(data: Json<Search>) -> Json<SearchResponse> {
      debug!("Search received: {:?}", data.0);
      Json(SearchResponse(vec![
          "blog_hits".to_string(),
          "sites".to_string(),
          "outbound_data".to_string(),
      ]))
  }

That was too easy. How about an endpoint that takes application state too?

- #[post("/query", format = "application/json", data = "<data>")]
- fn query(data: Json<Query>, opt: State<options::Opt>) -> Result<Json<QueryResponse>, Error> {
+ fn query(data: (Json<Query>, State<options::Opt>)) -> Result<Json<QueryResponse>, Error> {
      // endpoint code 
  }

The only thing that changed in the migration were the function signatures! Hats off to the actix project for reaching the same level of ergonomics as Rocket, making it a painless migration, all the while working on stable Rust.

Really, the only code I wrote for the migration was for creating an App

fn create_app(opts: options:Opt) -> App<options:Opt> {
    App::with_state(opts)
        .middleware(Logger::default())
        .resource("/", |r| r.f(index))
        .resource("/search", |r| r.method(http::Method::POST).with(search))
        .resource("/query", |r| r.method(http::Method::POST).with(query))
}

I completed the migration in under an hour with zero experience with actix and only using the official docs. I kicked myself for not doing this sooner, it was almost too easy.

One hiccup, though: integration testing.

Integration Testing

It took me longer to convert a few tests than to go from zero to migrated with actix. Testing with actix has it’s own documentation section which got me 95% of the way.

I was used to Rocket’s way of testing (example taken from Rocket docs):

let client = Client::new(rocket()).expect("valid rocket instance");
let mut response = client.get("/").dispatch();
assert_eq!(response.status(), Status::Ok);
assert_eq!(response.body_string(), Some("Hello world!".into()));

In the example, it’s immediately obvious how to test for the response’s body contents. Actix lacked this example, so I created it. I won’t bore you with why it took me so long to figure it out, so for posterity, here is the same test but with actix.

let mut srv = TestServer::with_factory( create_app);
let request = srv.client(http::Method::GET, "/").finish().unwrap();
let response = srv.execute(request.send()).unwrap();

assert!(response.status().is_success());
assert_eq!(response.content_type(), "text/plain");

let bytes = srv.execute(response.body()).unwrap();
assert_eq!(str::from_utf8(&bytes).unwrap(), "Hello world!");

Other Thoughts

I’m keeping a Rocket branch alive so I can do some comparisons:

Compile times with Rocket

git checkout rocket
cargo clean
time cargo +nightly build --release -p rrinlog-server

real    2m34.228s
user    9m41.905s
sys     0m11.664s

Compile times with Actix-Web (I compile with same nightly to control for any compiler improvements)

git checkout master
cargo clean
time cargo +nightly build --release -p rrinlog-server

real    3m16.306s
user    12m4.985s
sys     0m16.032s

Binary size:

  • Rocket: 9.5MB (5.3MB strip -x)
  • Actix-Web: 13MB (8.5MB strip -x)
  • Actix-Web (no default features): 12MB (7.6MB strip -x)

So Actix-Web results in a slower compile time and a larger executable. Migrating to actix web didn’t result in wins across the board, but it is a fine trade-off. For curiosity, I ran cargo bloat to see if there were obvious culprits:

 File  .text     Size Name
 9.0%  26.1%   1.2MiB [Unknown]
 6.8%  19.7% 961.4KiB std
 3.0%   8.6% 420.8KiB regex_syntax
 2.9%   8.2% 402.2KiB actix_web
 2.1%   6.1% 299.2KiB regex
34.6% 100.0%   4.8MiB .text section size, the file size is 13.8MiB

Eh, nothing stands out. That’s ok.

Last difference that I felt is that Rocket has its built in Rocket.toml, which I used to change the bound address. This was simple enough to move to a commandline argument.

I’m more than happy with the migration and will recommend anyone who uses Rocket and feels the pain of nightly breakage to use actix web. My intent for this article was not to come across like “X is better than Y framework” as both are exceptional, but rather showcase actix for Rocket users. It’s easy, fast, and stable.