My site was down for much of today as I finally took the plunge and rebuilt my image from ground up. This was something I had considered doing for some time now. The old site had been through a couple of ubuntu upgrades and was chugging along, but I always thought it would be good to do a fresh install. However I never really wanted to dump the time into it. Then I saw this article: https://www.anandtech.com/show/14284/amazon-offers-another-amd-epyc-powered-instance-t3a
I have been interested in AMD’s latest processors as they seem to be killing it with their Ryzen line. I checked the pricing and saw that the price for running my site was about half of the cost of what I was paying for my old t2 instance. Additionally at the same price level I was getting more CPU cores with the new instances. I hadn’t even realized that Amazon had rolled out their regular T3s or these new T3a instances. I figured it would be an easy switch, stop my instance, change the instance type and start up the new instance.
I attempted that with no luck. It seems that since I setup my site Amazon had come out with a new Network card driver, which my Linux kernel didn’t have. I switched back to my T2, and decided I would start building my new image on the side. When I had time I would spin up my new image and start working on the Amazon setup. I was happy to see there is now IPv6 support available. All in all there was just lots of new stuff to play with. I finally reached the point yesterday where I decided to backup my site and finally go for it. I had hit a stopping point where I brought my nginx config over (which had all my SSL settings), but I didn’t have my DNS pointed to the new machine in order to pull down new certificates. Without the certificates nginx wouldn’t launch, and at the same time because of HSTS preloading I can’t test an unencrypted version of my site. I pointed to the new IP this morning in my DNS settings and then proceeded to start working on restoring the backup of my site while I waited for the old DNS cache entries to expire so that I would pull down my new certificates.
Right around that time I had to depart as we had mother’s day plans and thus the site needed to remain down. When I got back on tonight I finally finished restoring my backup of wordpress and we are back. Now that I am back up I realized that my SSL Labs score dropped so I have been tweaking my configuration to get back up to my high A+ score.
Future projects now will include figuring out IPv6 routing. I was able to assign an IPv6 address to my instance and VPC, but when I try to SSH to that address it seems to time out. When I get that working I would like to setup DNS to allow getting to my site solely with IPv6. Anyway that is all for now as I need to go to bed, but more updates coming soon hopefully.
The Upgrade Process
I finally got around to upgrading my web server from Ubuntu 14.04 to Ubuntu 16.04.1. I decided to jot down my experiences in case anyone else runs into these issues.
The first thing I did prior to doing the upgrade was to run updraft plus to backup the Blog’s database and files. After that finished I logged into the Amazon EC2 console and created an image of my AMI. This way in the worst case scenario that I trashed my image I could just boot the backup image and be where I was before I began.
The next thing I did was check to see if I had screen installed. My thinking on these sorts of upgrades is you can easily lose your network connection and it is nice to be able to get right back to where you were without worrying about it if something goes wrong.
Beginning the upgrade
Ubuntu has a handy command:
I invoked the command. It let me know that it would start ssh on an additional port just in case I got disconnected during the upgrade. I said yes to begin and it began downloading all the packages it needed. Amazon seems to have a ton of bandwidth so this step is super fast. Wouldn’t you know it but in the middle of this my network connection dropped. Luckily I was able to ssh back in and reattach to my screen session without issue. There were a couple of questions I had to answer about whether to keep a config file that I had modified or else replace it with the package maintainers. I made the decision for those questions based on looking at the diff to see if the differences in the files were important or not. At the end it asks to reboot to finalize the upgrade so I said yes.
The machine booted up quickly and as soon as I was able to ssh in I logged into the website to see if everything was still working. I was promptly greated with a 502 Bad Gateway error. I started poking around and I could see that my nginx config hadn’t been changed and the service seemed to be running. At that point I figured it must be an issue with php-fpm. It turns out that it didn’t even have the package installed for php-fpm and the php packages that I did have were upgraded from 5.0 to 7.0. After googling around I was able to find out that the unix domain socket name and location have changed in the new version of php. I edited my default.conf for nginx and pointed it to the new socket location:
That being done I needed to make some changes on the php-fpm side. The config file has moved where previous it was located under /etc/php5/fpm/pool.d/www.conf the new location is /etc/php/7.0/fpm/pool.d/www.conf
In this file I needed to update the following fields:
listen = /run/php/php7.0-fpm.sock
listen.owner = nginx
listen.group = nginx
In this case I didn’t change the listen value I just used that value above to update my default.conf under nginx. After setting those values and restarting php7.0-fpm and nginx services my site was back up. Hopefully this will be helpful to anyone else who hits the same issue after they upgrade.
About a month ago I got an email from Amazon that they had introduced a nano instance. This was a very timely email as I was just nearing the end of my free year of AWS (your first year they give you enough of an EC2 budget to run a t2.micro instance for free). I had been running this site on the micro instance (as why not when you aren’t paying for it), but yesterday I got a bill for my EC2 usages. It was $11.46 for the month of December.
This site is mostly idle and only has a few readers so it is an ideal candidate to go to that new t2.nano instance. A few minutes ago I made the switch over it was fairly easy to make the switch. You just stop your instance and then in the EC2 console change the instance type and start it back up and you are good to go. Of course, since I was worried about something going wrong, I first backed up the blog using the UpdraftPlus plugin and stored it in my google drive. I then made a snapshot of the EBS Volume in EC2 just to be safe. But it was a super easy operation and I made the switch in about 1 minute. That will cut my hourly cost from $0.013 per hour to $0.0065 per hour, so half the price. Looking forward to a bill of around $5.50 next month, which it is hard to find hosting for less than that, especially with the level of control you get in EC2.
If you are running a low volume site look into it and see if you can save some money.