I decided to upgrade my site to the new version of Ubuntu as I haven’t done that for a couple of years. It is always a nice thing to work on when I am on vacation as it is the sort of thing that I don’t really get around to normally when I am busy. What a pain that ended up being.
The Upgrade for the OS itself went very smoothly as it seems to normally do so for Ubuntu. But the upgrade to the newer version of PHP broke everything with my site. As I think back actually I think this happened last time when I went from Ubuntu 14.04 to 16.04 as well and it jumped from php5 to php7. I ended up with about a 3 hour outage trying to sort everything out.
The big issue I saw was the default user that php was using was different than the nginx user so it couldn’t write to the Unix Domain Socket. I also noticed all the configuration advice for nginx was very different than when I set up this site. It seems like things are laid out better with the whole snippets of different configurations instead of sort of everything going in the
default.conf. At some point I may want to start over again from a blank AMI image and import my content into it. Then I could setup nginx a little bit more modern. Seems like there are lets encrypt plugins for it too, so I am wondering if I could have it auto-renew my certificates.
Another thing I could do if I redid the site, would be to switch to mariadb. I have heard it is supposed to be faster than mysql it might be fun to mess with something different. That being said I probably won’t get to that this winter as I am currently working on some content for a talk and I also want to spend a little time doing some machine learning classes on Coursera before I get back to work.
I did take advantage of the time in the config files to figure out how to tighten up my SSL Labs score. I found that I was missing just 1 item to push my key exchange test from a 90 to a 100 so I implemented that. I was hoping to be able to turn on TLSv1.3 as well, but unfortunately Ubuntu 18.04.1 ships with a version of OpenSSL that is too old to support it. I saw on a mailing list that it is supposed to be coming though so hopefully soon I will be able to update to that.
Recently Let’s Encrypt announced that they would be supporting wildcard certs. I was pretty excited to hear about this as many times I would like to get certs for machines that might not be accessible on the internet. Currently I didn’t see an easy way to do this. With the new certs you could get a cert on your web server for your domain and use that cert on all the other machines in your domain that need TLS as well.
I decided to try it out and see how easy it was to do. I updated my certbot client to version 0.22 and did some google around and found out that you have to specify the new acme version 2 servers on the client command line in order to generate the wildcard cert. So I found the command and fired it up:
./certbot-auto --server https://acme-v02.api.letsencrypt.org/directory -d *.haskovec.com --manual --preferred-challenges dns-01 certonly
The command runs and asks you a few questions and then presents a DNS challenge. They give you a TXT record that you need to update in your DNS server to prove that you control the domain. I added the record and waited a couple of minutes and next thing you know it generated my new cert.
I updated my NGINX config to point to the new cert restarted the server and hit my site. Next thing I see is an SSL error. It turns out if you only have *.haskovec.com in the cert that isn’t a valid server for the base domain of haskovec.com. So I reran the command again and specified the following:
./certbot-auto --server https://acme-v02.api.letsencrypt.org/directory -d haskovec.com -d *.haskovec.com --manual --preferred-challenges dns-01 certonly
This time it asked me if I want to expand my cert to include the new domain name. I said yes. Next it had 2 challenges that I needed to insert into my DNS TXT record. I added them both and finished generating the new cert. When I restarted NGINX my site was back. I ran the https://www.ssllabs.com/ security test on my site and I am still at an A+.
All in all a very easy process and I recommend people give it a try.
Recently I ported my domain hosting from Godaddy to Google Domains. My main reason for doing so was to save money. Domain names on Godaddy cost $3 more per year, plus they charge you for privacy on whois searches whereas Google includes that for free. It was a fairly easy process to transfer my domain names in, but configuring the DNS was a little bit weird as their zone file editing interface was different that godaddy’s. However I thought I had it all good and working so I was happy with my setup.
Then last night one of my friends mentioned that he had just renewed his SSL certificate. That got me thinking I only had about 2 weeks left on my certificate and I needed to do that as well. I had mentioned previously that I switched my SSL certificates to Let’s Encrypt. The great thing about Let’s Encrypt is that it is free, and once you get it setup, less hassle to renew that other free certificate sites that I had used in the past. The drawback seems to be that they only issue certificates that are good for 90 days. I updated my Let’s Encrypt software (it is based out of a Github repository so there is normally a new version when you need to renew.) When I ran the renew command it failed on www.haskovec.com. That is when I realized that I had misconfigured my cname record on Google Domains DNS setting and www. was completely broken only haskovec.com worked. It took me a little while to figure out. On Godaddy I think I would do the cname of www and point it to @ or * I don’t recall which they used. Google doesn’t let you point the cname to @. Aftering some googling I found out that on their DNS setup you have to point to your hostname that is registered at @. So it ends up being cname www and it points to haskovec.com. Once I got that taken care of my certificate renewed without any issue.
While I was in there messing around I decided I would disable TLS 1.0. Doing so means dropping support for a ton of browsers including IE10. But it is widely considered as the next protocol to be hacked and at this point pretty much everyone supports 1.2 and the handful of readers I have I expect to be running current browsers (whether on their phones or computers.) I reran the Qualys SSL Test to make sure that I hadn’t broken anything. All looked well with the higher score now on the protocol section and many more test browsers that failed. In the course of running that test I noticed the HSTS preloading test that they are doing now. I didn’t even realize such a thing existed. I did some research and added the preload header to my HSTS header on my server and put my site on the preload list for Chrome. We shall see if that works or if I meet all the requirements, but I think I do. While I was editing my headers I noticed that I was doing the domain name rewriting wrong if the person came in through https://www.haskovec.com/ The code was working if they either came in without the www or they came in on www without the https. So it ended up being a useful night as I found 2 issues in my server config I was able to fix. In the course of writing this post I realized I should add the other cnames I have registered for this domain to my SSL certificate so that will be the next thing on my agenda.
I received an email a week or 2 ago that I was accepted into the EFF’s Let’s Encrypt Beta program to try out their new SSL certificate generation service. It uses the Automated Certificate Management Environment (ACME) protocol. I have been really interested in this program since it was announced as in the past when I have used Start SSL’s certificate system I found their whole validation system to be a little clunky. The idea of a nice automated program that does all the work for me sounded very appealing.
The first thing I had to do was to clone the git repository from github with the scripts to run their program. The first thing I discovered is that yeah this thing is really in beta, I ran the ./letsencrypt-auto command and discovered that nginx isn’t supported yet for plugging the certificate into automatically. This ends up not being that big of a deal since you can just edit your config file and point at the certificate directory.
The biggest weakness I found so far is that I had to shut down nginx to run the let’s encrypt client as it wants to bind to port 80. I think what it is doing is listening on port 80 so that the remote server can verify that I actually own this domain and it is okay to issue this certificate to me. That is nice from a security standpoint but in the documentation they mention running this out of a cron job to update certificates, which if you have to take down the web server might not be ideal. That gets me to the next biggest weakness which is that the certificates expire in 90 days. So far I would say running their app is easier than validating on startssl, but startssl doesn’t need me to take down my web server. They idea I think is that since this is all supposed to be automated you can easily script this out in cron and your system deals with getting new certs and updating them with minimal to no end user interaction after you get it running.
The great things about it is it is pretty fast. Much faster than getting a certificate any other route that I have tried so that sort of offsets having to do it 4 times as much. The price is right cause it is also free. They allow you to have multiple names in your certificate. So now my new cert supports both haskovec.com and www.haskovec.com. I think this process also makes things convenient enough that I hope everyone will start encrypting all their servers by default and using this service.
Whenever I mention something at work about someone using an internal certificate that is self signed I always say they should get a real cert. Hopefully as this gets built out and put into production it will make certificate management so easy and fast that people will just do it by default, and I think that is really when this program is going to pay off for the EFF.
I saw a post on twitter about Security Headers. Basically Security Headers will scan your website and check for some common HTTP Headers that you should be including to make your site more secure. They also include helpful links as to how to fix the issues it finds. On my first scanned it warned me of the following: MISSING Content-Security-Policy, MISSING X-Frame-Options, MISSING X-Xss-Protection, MISSING X-Content-Type-Options, MISSING Public-Key-Pins, and X-Powered-By. After going through their documentation I added all of those headers except for Public Key Pins. I am not 100% on that, my concern is when your certificate expires and you replace it (which on a free certificate happens every year) do you end up with people getting an error on your website for the next week cause they have an old key pinned? Not sure enough about that to actually enable it, which is why I hadn’t previously enabled it. The other headers though I didn’t realize could be an issue so I promptly corrected them. The one thing I couldn’t easily fix was the Server header as apparently that is compiled into NGINX and I wasn’t feeling like compiling my own from source. I was able to remove the version string though. All in all they give some very easy configuration changes you can make to help prevent attacks against your website and I strongly recommend giving their tool a look.
So a little while back when I had been playing with Pagespeed I somehow managed to break certificate stapling on my server. So when I ran the Qualys SSL Server Test my score had fallen to a B! I messed around and tried a few things and I had no luck getting it to work. One of my friends said the site started to give weird errors under Chrome on Android. Then I was reading this CertSimple Blog entry yesterday and they mentioned the Mozilla Server Side TLS Project, which I don’t think I had heard of. Basically what it does is you enter your server version and your OpenSSL version and how aggressive you want your security settings and it will generate a sample config for you. It will tell you based on how aggressive your settings are what the minimum browser versions are. For example of of the differences between Intermediate and Modern is that they drop support for TLSv1 in Modern and only support TLSv1.1 and TLSv1.2. For most browsers this doesn’t seem to be an issue but if you are running IE that means the minimum browser version is IE 11. I debated whether I should drop TLSv1 support or not, but I figure if I keep it I can support IE back to 7, though I can’t imagine any software engineers that might check out this blog using IE anyway. For now I have kept it but one of these days I will drop it because given the rate of SSL issues with Freak and Logjam lately, it is only a matter of time before someone finds a hole in TLSv1.
As for my issue Mozilla in their example config said my ssl_certificate setting should point to my signed key plus intermediates, whereas previously I only had my signed key there. I had intermediates in the ssl_trusted_certificate with the root certificate, and that was working prior to my gzip changes but for some reason now it wants them in both places or else it does a separate download on the intermediate certificate and drops me to a B. So I am back to my coveted A+ rank, and I think the lesson I learned is one a coworker mentioned to me. They said that they put all of their config files in git so that any time they make a change if there are issues they can look at all previous revisions. In the future if I make any changes here on my server config I think I may do the same and setup a config file repository before I touch anything again just to have easy version control and knowing how to revert if things get ugly.
The Electronic Freedom Foundation has released a nice post on how one should configure their server to pass the Qualsys SSL Labs test with an A+. Given that when I initially setup my site it took me like 3 hours of messing around to get my site to pass with an A+ I figured I should share this post with everyone as it is so useful. Check it out here!
I was checking the Spring Blog today to see what was new after taking much of the week off. I came upon the following entry. Of course I was very interested as Spring Security 4.0 has been hyped for a few months now so I figured I would check out the migration guide from 3.2 to see what will be involved for us to upgrade. I found this is the new feature section. They have added a feature which will now automatically prepend ROLE_ to any roles you use in Spring security if your role doesn’t start with that. So if you have a role called ROLE_USER for a standard user you can now just say @PreAuthorize(“hasRole(‘USER’)”).
The problem I have with this, is it assumes that you name all your roles with ROLE_. Unfortunately we don’t. When we migrated to Spring Security for our new architecture we already had a system that had over 700 different user rights in it. We mapped our user right system into Spring Security by implementing GrantedAuthority on our UserRights. This allowed us to maintain compatibility with our legacy architecture which some of our system runs on while seamlessly migrating other parts over to the new architecture. This is one of those opinionated things about the framework that is going to cause us a lot of issues. Now we are forced to conform to what Spring Security thinks we should name things instead of it just checking our collection of granted authorities to see if we have permission to carry out an operation. This is one of those changes that risks breaking things for many people with what gain you don’t have to type 5 characters? I don’t really get it and am annoyed as it means maybe we have to do some ugly hack like have our getter prepend that so we can keep our legacy name, but still satisfy the new “feature”. If nothing else it means it will be longer before we think about making this change as why add extra pain for yourself, unless you see a big benefit which at that point I don’t see a super compelling reason why I need to upgrade right now. So maybe we will get around to upgrading in the next year time will tell.
I just wanted to mention signal has been released so for all the iPhone users out there it is definitely worth installing. Signal is an implementation of Text Secure on IOS. Given the insecurity of text messages and how many other messengers have varying degrees of security Open WhisperSystems has released Signal. Some popular chat programs like WhatsApp are starting to encrypt but they aren’t always encrypted. One of the biggest benefits of Signal is they released the source code so that anyone can audit the code. Granted if you are installing it from the iTunes store there is still the risk of a back door being in there, but one inclined and with a developers license could build their own and install it on their phone. Of course that also assumes that your copy of XCode hasn’t been tampered with. At the end of the day if you are a target they are probably going to get your stuff, but the benefits of widespread use of secure products would be to disallow wholesale metadata and data collection that has been alleged to have been happening so that innocent people are left alone. The EFF has a nice scorecard that ranks the security aspects of different messenger apps. Here is another story suggesting the use of this app. If you are on Android Signal is an implementation of TextSecure.