September 4, 2019

Managing Server Access with Keybase!

Oh I love Keybase. And you should too - it's an amazing cryptographic chat and collaboration platform that makes many of the world's things easier. You can follow me and chat me here. Keybase client is OSS which is LOVELY. My one complaint is Keybase server is not, so, you do have to trust it/them (hi Chris!).

I also love my new role within Microsoft - more on that later though. One of the reasons I love it? Each month, we're REQUIRED to spend half a day learning and thinking about something. Recently, I read an article on how one could control access to a Linux server using Keybase chatbots and keybase user state in a specific team. This got me thinking - I wonder if I could extend this to Azure? Oh dear, yes indeed, as I learned on my today's lesson of "Derek, go learn something!"

First, the why - why would you WANT to do this. Well, managing access to Linux VMs is a pain. Microsoft is making it better soon by allowing AAD authentication into the VMs. That will be a total trip! There are also other solutions that come to mind like joining Linux VMs to AAD and LDAP and etc third party tools. Bleh, not fun. What is fun, dropping a user into a Keybase team and that automatically granting them access to the VM using not just passwords which leak, but signed certs that go kaboom in relatively short periods of time (ie, 1 hour).

First, we need to have a setup.

  1. For me, this involved a Windows 10 PC where Keybase runs as me all the time. I call this CLIENT.
  2. Next, there's a server running Docker somewhere out there in the cloud that runs the Keybase chat bot that also does Certificate Authority type things. This can be any server anywhere as long as Docker is running. No ports open to it needed - just outbound from it to the internet so it can talk to the Keybase servers. I call this server KEYBASECA.
  3. Next, we will need servers, LOTS OF SERVERS to be our examples of how I can log into a fleet of stuff without nary a password. I call these LOTSOFSERVERS
  4. Here's a pretty picture of what this is gonna do:
Authentication Flow

First, let's set up KEYBASECA:

  1. Have a Linux VM somewhere with Docker running. Could also be in a K8S cluster running in AKS, ehmm eeeeee!
  2. Log in to said machine and get some skills going

curl --remote-name https://prerelease.keybase.io/keybase_amd64.deb
sudo apt install ./keybase_amd64.deb
run_keybase

That there gets Keybase up and running on the VM itself. You'll need a Keybase ID for our bot. DO NOT try to use an existing ID.

Next, we need some bot to work with. Assumes you have Docker up and running.

git clone git@github.com:keybase/bot-sshca.git
cd bot-sshca/docker/
cp env.sh.example env.sh
keybase signup # Creates a new Keybase user to use for the SSH CA bot
keybase paperkey # Generate a new paper key

When you are doing keybase signup it will ask you for an email address. You'll be tempted to use the same email address you used for your regular account - you can't cause it will try recovery. You will then be tempted to use a throw away account. That's fine, just remember what it is in case you need to recover. I like to use mailinator

Now, you'll need to remember (Notepad) your username and the paperkey. The paperkey is a big string of words. Next, on your regular Keybase account (on CLIENT), create a team and some subteams. I did this:

Inside each subteam, I added my bot user (the username from keybase signup). So, I have two users, my CLIENT username and my KEYBASECA username. To control access to servers, I will add and remove users to said subteams. For example, if I want to grant people access to my Google servers (all one of them), I put them there. Etc.

Next, I filled in the important details from the above stuff into env.sh (KEYBASECA username, paperkey and a comma separated list of teams):

My env.sh

Notice the " on each side of the strings and the no spaces in the TEAMS sections. The PAPERKEY does have spaces in it like this: "Fred Simon Chris Happy"

Once I've filled those details in, it's time to generate my Certificate Authority wielding Keybase bot.

make generate
This will output a bunch of stuff as it makes the docker container. Once it does it's stuff, we'll need to grab one of the outputs - the short hand public key. Something like this (the highlighted piece):

The output of make generate

You only want the bits that start with ssh-ed25519 ... that is highlighted in yellow. Ignore all the other instructions...for now.

Now, let's run make serve which will get your bot up and going and it will connect to Keybase quietly. Not much logging happening in the current version, but some goodies are in flight I'm told.

Now, it's time to create some LOTSOFSERVERS! Over in the Azure Portal (I'm not providing you a link cause you should just know), you need to deploy any number of Linux VMs. For me, I am using 18.04LTS.

Something to note here - I'm doing password auth as a "backup" just in case things go squirrly. In the real world, you would want to use a master public/private key auth and a master user. This has nothing to do with our bot though. That comes in a bits.

On the Networking screen, make sure to enable SSH from the internets!

Move on through the screens till you get to the cloud-init screen:

Here's where the magic happens. We're going to take the Cloud init line and pass in a bash script that does some shiny things to our server that makes it trust the KEYBASECA's cert and certs that it generates. We're going to also create a common user (in this case called developer). See here for noms:

#!/bin/bash

useradd developer && mkdir -p /home/developer && chown developer:developer /home/developer
echo "ssh-ed25519 **therestofthekeyfromthehighlightabove**" > /etc/ssh/ca.pub
echo "TrustedUserCAKeys /etc/ssh/ca.pub" >> /etc/ssh/sshd_config
mkdir -p /etc/ssh/auth_principals/
echo "AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u" >> /etc/ssh/sshd_config
chmod g-w /etc
service ssh restart
# Allow anyone in dermarssh.azure to ssh in as the developer user
echo "dermarssh.azure" >> /etc/ssh/auth_principals/developer

The dermarssh.azure on the last line is the subteam whose users I want to be able to auth into my box. If I put a Keybase user into that subteam, BAM, they will get authed into my Linux boxxen. Mmmmm. But we're not done yet.

Paste the code above subtituting out your stuff (team and public key) and then finish creating the VM.

Now, you've created a VM and you want to log into it with a magically rotating SSH public key that the KEYBASECA bot issues if you are in that team.

Over on CLIENT (which is my Windows 10 client which now magically ships with SSH delight cause we're awesome) I am going to download an SSH wrapper. You can get it from here (or build your own cause OSS and security). Stick that sucker in your path somewhere. Open up some POWERSHELL goodness and prepare to be dazzled. Oh, make sure Keybase is running and you're logged in haha.

  1. First - make sure you have a folder in ~/ called .ssh. For me, that's c:\users\derek\.ssh This is where kssh-windows (actually wrapping ssh.exe) will store the goods.
    1. Run this delightful nugget to get ssh-agent up and running (it is disabled by default: Get-Service -Name ssh-agent | Set-Service -StartupType Manual
  2. Run: kssh-windows developer@<IPAddressofYourAzureVM>
BOOM

With a little bit of luck:

  1. kssh-windows will ask Keybase for a public key (which will expire in an hour)
  2. Keybase will ask the bot for a key
  3. The bot will generate a key that is permitted by the subteam membership and send it via Keybase back to kssh-windows
  4. kssh-windows will set that key into ~/.ssh and then issue the appropriate ssh command to the server, passing the key along with it and shazam - logged in you are.

There you go - you can now mass create lots of VMs and depending on what you put in the cloud-init line for authorized team membership, the CA bot will let you in if you are a member. What else could you do with this bit of glorious goo? I'm thinking some neat integration with AAD, or the ability to self provision without cloud-init based on an ARM template deployment, some interaction with KeyVault so the bot doesn't even have the private key, etc. But that's enough for today's half day of learning by Derek.

Bye!