I decided to upgrade my site to the new version of Ubuntu as I haven’t done that for a couple of years. It is always a nice thing to work on when I am on vacation as it is the sort of thing that I don’t really get around to normally when I am busy. What a pain that ended up being.
The Upgrade for the OS itself went very smoothly as it seems to normally do so for Ubuntu. But the upgrade to the newer version of PHP broke everything with my site. As I think back actually I think this happened last time when I went from Ubuntu 14.04 to 16.04 as well and it jumped from php5 to php7. I ended up with about a 3 hour outage trying to sort everything out.
The big issue I saw was the default user that php was using was different than the nginx user so it couldn’t write to the Unix Domain Socket. I also noticed all the configuration advice for nginx was very different than when I set up this site. It seems like things are laid out better with the whole snippets of different configurations instead of sort of everything going in the
default.conf. At some point I may want to start over again from a blank AMI image and import my content into it. Then I could setup nginx a little bit more modern. Seems like there are lets encrypt plugins for it too, so I am wondering if I could have it auto-renew my certificates.
Another thing I could do if I redid the site, would be to switch to mariadb. I have heard it is supposed to be faster than mysql it might be fun to mess with something different. That being said I probably won’t get to that this winter as I am currently working on some content for a talk and I also want to spend a little time doing some machine learning classes on Coursera before I get back to work.
I did take advantage of the time in the config files to figure out how to tighten up my SSL Labs score. I found that I was missing just 1 item to push my key exchange test from a 90 to a 100 so I implemented that. I was hoping to be able to turn on TLSv1.3 as well, but unfortunately Ubuntu 18.04.1 ships with a version of OpenSSL that is too old to support it. I saw on a mailing list that it is supposed to be coming though so hopefully soon I will be able to update to that.
The Upgrade Process
I finally got around to upgrading my web server from Ubuntu 14.04 to Ubuntu 16.04.1. I decided to jot down my experiences in case anyone else runs into these issues.
The first thing I did prior to doing the upgrade was to run updraft plus to backup the Blog’s database and files. After that finished I logged into the Amazon EC2 console and created an image of my AMI. This way in the worst case scenario that I trashed my image I could just boot the backup image and be where I was before I began.
The next thing I did was check to see if I had screen installed. My thinking on these sorts of upgrades is you can easily lose your network connection and it is nice to be able to get right back to where you were without worrying about it if something goes wrong.
Beginning the upgrade
Ubuntu has a handy command:
I invoked the command. It let me know that it would start ssh on an additional port just in case I got disconnected during the upgrade. I said yes to begin and it began downloading all the packages it needed. Amazon seems to have a ton of bandwidth so this step is super fast. Wouldn’t you know it but in the middle of this my network connection dropped. Luckily I was able to ssh back in and reattach to my screen session without issue. There were a couple of questions I had to answer about whether to keep a config file that I had modified or else replace it with the package maintainers. I made the decision for those questions based on looking at the diff to see if the differences in the files were important or not. At the end it asks to reboot to finalize the upgrade so I said yes.
The machine booted up quickly and as soon as I was able to ssh in I logged into the website to see if everything was still working. I was promptly greated with a 502 Bad Gateway error. I started poking around and I could see that my nginx config hadn’t been changed and the service seemed to be running. At that point I figured it must be an issue with php-fpm. It turns out that it didn’t even have the package installed for php-fpm and the php packages that I did have were upgraded from 5.0 to 7.0. After googling around I was able to find out that the unix domain socket name and location have changed in the new version of php. I edited my default.conf for nginx and pointed it to the new socket location:
That being done I needed to make some changes on the php-fpm side. The config file has moved where previous it was located under /etc/php5/fpm/pool.d/www.conf the new location is /etc/php/7.0/fpm/pool.d/www.conf
In this file I needed to update the following fields:
listen = /run/php/php7.0-fpm.sock
listen.owner = nginx
listen.group = nginx
In this case I didn’t change the listen value I just used that value above to update my default.conf under nginx. After setting those values and restarting php7.0-fpm and nginx services my site was back up. Hopefully this will be helpful to anyone else who hits the same issue after they upgrade.