My site was down for much of today as I finally took the plunge and rebuilt my image from ground up. This was something I had considered doing for some time now. The old site had been through a couple of ubuntu upgrades and was chugging along, but I always thought it would be good to do a fresh install. However I never really wanted to dump the time into it. Then I saw this article: https://www.anandtech.com/show/14284/amazon-offers-another-amd-epyc-powered-instance-t3a
I have been interested in AMD’s latest processors as they seem to be killing it with their Ryzen line. I checked the pricing and saw that the price for running my site was about half of the cost of what I was paying for my old t2 instance. Additionally at the same price level I was getting more CPU cores with the new instances. I hadn’t even realized that Amazon had rolled out their regular T3s or these new T3a instances. I figured it would be an easy switch, stop my instance, change the instance type and start up the new instance.
I attempted that with no luck. It seems that since I setup my site Amazon had come out with a new Network card driver, which my Linux kernel didn’t have. I switched back to my T2, and decided I would start building my new image on the side. When I had time I would spin up my new image and start working on the Amazon setup. I was happy to see there is now IPv6 support available. All in all there was just lots of new stuff to play with. I finally reached the point yesterday where I decided to backup my site and finally go for it. I had hit a stopping point where I brought my nginx config over (which had all my SSL settings), but I didn’t have my DNS pointed to the new machine in order to pull down new certificates. Without the certificates nginx wouldn’t launch, and at the same time because of HSTS preloading I can’t test an unencrypted version of my site. I pointed to the new IP this morning in my DNS settings and then proceeded to start working on restoring the backup of my site while I waited for the old DNS cache entries to expire so that I would pull down my new certificates.
Right around that time I had to depart as we had mother’s day plans and thus the site needed to remain down. When I got back on tonight I finally finished restoring my backup of wordpress and we are back. Now that I am back up I realized that my SSL Labs score dropped so I have been tweaking my configuration to get back up to my high A+ score.
Future projects now will include figuring out IPv6 routing. I was able to assign an IPv6 address to my instance and VPC, but when I try to SSH to that address it seems to time out. When I get that working I would like to setup DNS to allow getting to my site solely with IPv6. Anyway that is all for now as I need to go to bed, but more updates coming soon hopefully.
I decided to upgrade my site to the new version of Ubuntu as I haven’t done that for a couple of years. It is always a nice thing to work on when I am on vacation as it is the sort of thing that I don’t really get around to normally when I am busy. What a pain that ended up being.
The Upgrade for the OS itself went very smoothly as it seems to normally do so for Ubuntu. But the upgrade to the newer version of PHP broke everything with my site. As I think back actually I think this happened last time when I went from Ubuntu 14.04 to 16.04 as well and it jumped from php5 to php7. I ended up with about a 3 hour outage trying to sort everything out.
The big issue I saw was the default user that php was using was different than the nginx user so it couldn’t write to the Unix Domain Socket. I also noticed all the configuration advice for nginx was very different than when I set up this site. It seems like things are laid out better with the whole snippets of different configurations instead of sort of everything going in the
default.conf. At some point I may want to start over again from a blank AMI image and import my content into it. Then I could setup nginx a little bit more modern. Seems like there are lets encrypt plugins for it too, so I am wondering if I could have it auto-renew my certificates.
Another thing I could do if I redid the site, would be to switch to mariadb. I have heard it is supposed to be faster than mysql it might be fun to mess with something different. That being said I probably won’t get to that this winter as I am currently working on some content for a talk and I also want to spend a little time doing some machine learning classes on Coursera before I get back to work.
I did take advantage of the time in the config files to figure out how to tighten up my SSL Labs score. I found that I was missing just 1 item to push my key exchange test from a 90 to a 100 so I implemented that. I was hoping to be able to turn on TLSv1.3 as well, but unfortunately Ubuntu 18.04.1 ships with a version of OpenSSL that is too old to support it. I saw on a mailing list that it is supposed to be coming though so hopefully soon I will be able to update to that.
The Upgrade Process
I finally got around to upgrading my web server from Ubuntu 14.04 to Ubuntu 16.04.1. I decided to jot down my experiences in case anyone else runs into these issues.
The first thing I did prior to doing the upgrade was to run updraft plus to backup the Blog’s database and files. After that finished I logged into the Amazon EC2 console and created an image of my AMI. This way in the worst case scenario that I trashed my image I could just boot the backup image and be where I was before I began.
The next thing I did was check to see if I had screen installed. My thinking on these sorts of upgrades is you can easily lose your network connection and it is nice to be able to get right back to where you were without worrying about it if something goes wrong.
Beginning the upgrade
Ubuntu has a handy command:
I invoked the command. It let me know that it would start ssh on an additional port just in case I got disconnected during the upgrade. I said yes to begin and it began downloading all the packages it needed. Amazon seems to have a ton of bandwidth so this step is super fast. Wouldn’t you know it but in the middle of this my network connection dropped. Luckily I was able to ssh back in and reattach to my screen session without issue. There were a couple of questions I had to answer about whether to keep a config file that I had modified or else replace it with the package maintainers. I made the decision for those questions based on looking at the diff to see if the differences in the files were important or not. At the end it asks to reboot to finalize the upgrade so I said yes.
The machine booted up quickly and as soon as I was able to ssh in I logged into the website to see if everything was still working. I was promptly greated with a 502 Bad Gateway error. I started poking around and I could see that my nginx config hadn’t been changed and the service seemed to be running. At that point I figured it must be an issue with php-fpm. It turns out that it didn’t even have the package installed for php-fpm and the php packages that I did have were upgraded from 5.0 to 7.0. After googling around I was able to find out that the unix domain socket name and location have changed in the new version of php. I edited my default.conf for nginx and pointed it to the new socket location:
That being done I needed to make some changes on the php-fpm side. The config file has moved where previous it was located under /etc/php5/fpm/pool.d/www.conf the new location is /etc/php/7.0/fpm/pool.d/www.conf
In this file I needed to update the following fields:
listen = /run/php/php7.0-fpm.sock
listen.owner = nginx
listen.group = nginx
In this case I didn’t change the listen value I just used that value above to update my default.conf under nginx. After setting those values and restarting php7.0-fpm and nginx services my site was back up. Hopefully this will be helpful to anyone else who hits the same issue after they upgrade.