Maintaining Your Boxen Install

@Intridea we use boxen as part of our employee on-boarding process and equipment upgrades. The selling point: having a shared automated process for getting machines up and ready to do real work on…

What Happened?

Recently, a co-worker received a new laptop and I pointed him to our boxen-web instance. The problem is it didn’t work for him. I was confused because running boxen on my machine worked just fine. After digging deep and troubleshooting I learned that the initial install process for boxen is quite different than running the boxen command afterwards.

It had been a few months since anyone installed from scratch with our boxen setup, and the only error message I was getting was pretty vague.

sudo: /opt/boxen/rbenv/shims/gem: command not found sudo: /opt/boxen/rbenv/shims/gem: command not found ... 

I could tell both ruby and git were failing to set-up properly, but it wasn’t much to go on. I didn’t yet know whether the failure was due to Jeff’s configuration or the main boxen configuration. After having him try it with his personal manifest files removed, it was confirmed to be a problem with the main boxen configuration. The next step was to try a fresh install myself.

Fixing our Boxen Setup

First I partitioned my hard drive and installed a fresh copy of 10.9.2. I removed the files for my personal manifest from the boxen repository in order to keep the install as simple as possible. I verified that I was receiving the same error message listed above.

Next, I tried a fresh install from the mainline boxen. This also failed with a different git related error. I’ve mentioned before that the boxen documentation is lacking, and I was only finding issues with people having the same problem and no solution. I was out of options.

I decided to merge mainline boxen with our boxen setup. I found that doing this created conflicts in both the Puppetfile and Puppetfile.lock. I fixed the conflicts in the Puppetfile. The official boxen docs say that it isn’t necessary to resolve conflicts in the Puppetfile.lock. So I tried:

rm Puppetfile.lock bundle exec librarian-puppet install --clean 

However, those steps only gave me this error:

Could not resolve the dependencies. 

So I backed up a step and manually resolved the conflicts in the Puppetfile.lock. This, while extremely tedious, worked.

After merging mainline boxen, I was left with the same error referring to git. I next looked at what version of git our Puppetfile was using and what was available on the puppet-git repository. I found that mainline boxen was using 2.3.0 while 2.3.1 had been released more recently. On a hunch I upgraded to 2.3.1 and performed yet another fresh install of boxen. This time it worked.

Preventative Maintenance for your Boxen

After going through the above hellish process, here’s some recommendations on how to ensure smooth operation in the future.

Keep a fresh install of OS X on an external drive

• You should keep whatever point release you expect to send laptops out with (10.9.2 in our case).

Use the nuke tool

• After an initial boxen run, you can restore the machine to pristine status by using the nuke tool that comes with boxen.

/opt/boxen/repo/script/nuke --force --all 

The above command will remove boxen entirely from your machine. It isn’t necessary to reinstall OS X.

Create a dummy user for your boxen repo

• You will need a github user for your organization which has no personal boxen configuration. This makes it simple to determine whether the problem lies in personal configuration or in your company wide boxen configuration (site.pp).

Test the install often

• In the above scenario, I left out the part about the homebrew version that boxen uses also being broken. Testing more often would have allowed us to catch and fix each of these minor problems one by one instead of as part of a marathon debugging session.

The mainline boxen is not always correct

• In the scenario above, I figured merging the mainline ought to get us to a working state. However, mainline boxen had a broken puppet-git module (2.3.0) at that point, while 2.3.1 was the version that worked. So beware of keeping up to date with mainline too fast.

Got any tips or tricks for using boxen? Send us a tweet or message us on Facebook!