Coursera Stanford Machine Learning Course

I just signed up to take the Stanford Machine Learning Course for free on Coursera. Anyone who wants to take the course with me is welcome to join me over here: https://www.coursera.org/learn/machine-learning. They are only accepting enrollments until January 12th I think it said so not much time left to sign up.

2018 Year End Review

Recap for 2018

As is my tradition it is time to review 2018 and see how my year unfolded. The first thing that I always do is review my themes for the year and see how many of them I hit.

In general 2018 was an interesting year. The year started out with me on vacation. When I was on vacation I realized that I would have to change jobs in 2018 and luckily it all worked out for the better. I can say making that move was one of the best decisions I have made in my career. The difference between the first half of 2018 and the last half was a study in extreme opposites on the career front. Probably the biggest takeaway from that whole situation is that company culture may be the most important thing when it comes to choosing a position.

Without further delay here is how I did for 2018:

Health

I didn’t drop as much weight as I had hoped to at the start of the year. I think I lost 20 pounds in the first quarter and kind of slacked off after that. There will be another push for that again this year as I always like to start Q1 as a quarter to fast since Q4 for me is a quarter to feast. I haven’t figured out what approach I Am going to take? In the past I have done the Primal Prescription, Slow Carb, and Intermittent Fasting / Time Restricted Feeding. I will consider what approach I am going to take over the upcoming days. I also did DNAFit this year so it would be good to try to incorporate their diet ideas for my specific genes in whatever I decide to do. I should also push for more regular activity, so maybe in the end I will need to break down and get an apple watch just to use the data to drive accountability for myself. Not sure yet I have been holding out for it to do more than it currently does before I take the plunge. I do like that the series 4 at least has a larger display.

I suspect 2019 will involve more public speaking for me based on how things are looking so I also want get healthier for that as it is nice to look good if you are on a stage or the center of attention.

More Blogging

This theme was an epic fail. As I check my posts not counting this post there were a paltry 13 blog posts this year. Barely an average of 1 per month (vs my goal of 52 in general). This one I don’t even mind missing this year because it was for a good reason. I had so much stuff to learn for my new job this was one of the easiest things to cut to get more time. As I have settled in I hope to do better this year, but in general if I am pushing for a sprint deadline this is an easy thing to cut so I can get more code across the line. Once we get our product to market I hope to be able to plan more reasonable sprints.

Learning

I nailed this theme. It has been a great year for learning. Instead of the learning being books I read it has been practical things. First I learned Go Lang and implemented a micro serviced base architecture in it. I am still doing a lot of learning in Go. I haven’t yet figured out really the ideal patterns yet. One thing that I miss from Spring is how opinionated the framework is. There is a remarkable amount of consistency in code when you have the strongly defined stereotypes like that. When you see Go on the internet it is all over the place. Sometimes it feels like node code, sometimes it feels like C code, but it never really feels like people are using the same consistent patterns. I have played around with sort of the controller / service / repository layers and not sure that is the correct idiom. I have played around with just general event handlers and middleware patterns. I think this year will still be me trying to figure out what is the best patterns to use for backend services.

On the business front there was also a lot of learning for me. For the first time in my career I built an agile process that our team uses, and got a brand new team that didn’t know each other at the start off to a very good working cadence. Our process has worked so well that other parts of the organization have adopted it as their baseline and started iterating on it to adapt it to their teams. I have also applied some of the things I learned from domain driven development to build a model that allows the business and the technology teams to communicate with the same language.

Architectural Changes

This was underway when I left my previous company. We managed to convert some of our apps to Java from Groovy. We were doing this on a piece by piece basis at first going after the easy parts. We were also doing some restructuring to address some performance problems in the architecture around the time of my departure. This I would probably only give 50% to. Had I remained in that role I am sure we would have made major changes by the end of the year, but the organization had become too political for me and I got the opportunity of a lifetime at the same time.

Conclusion

All in all 2018 was another great year. The year ended much stronger than it started and I feel like I had a lot of personal growth this year. Looking forward to 2019 it is going to be a huge year for our company and for me personally.

Site Upgrade

I decided to upgrade my site to the new version of Ubuntu as I haven’t done that for a couple of years. It is always a nice thing to work on when I am on vacation as it is the sort of thing that I don’t really get around to normally when I am busy. What a pain that ended up being.

The Upgrade for the OS itself went very smoothly as it seems to normally do so for Ubuntu. But the upgrade to the newer version of PHP broke everything with my site. As I think back actually I think this happened last time when I went from Ubuntu 14.04 to 16.04 as well and it jumped from php5 to php7. I ended up with about a 3 hour outage trying to sort everything out.

The big issue I saw was the default user that php was using was different than the nginx user so it couldn’t write to the Unix Domain Socket. I also noticed all the configuration advice for nginx was very different than when I set up this site. It seems like things are laid out better with the whole snippets of different configurations instead of sort of everything going in the default.conf. At some point I may want to start over again from a blank AMI image and import my content into it. Then I could setup nginx a little bit more modern. Seems like there are lets encrypt plugins for it too, so I am wondering if I could have it auto-renew my certificates.

Another thing I could do if I redid the site, would be to switch to mariadb. I have heard it is supposed to be faster than mysql it might be fun to mess with something different. That being said I probably won’t get to that this winter as I am currently working on some content for a talk and I also want to spend a little time doing some machine learning classes on Coursera before I get back to work.

I did take advantage of the time in the config files to figure out how to tighten up my SSL Labs score. I found that I was missing just 1 item to push my key exchange test from a 90 to a 100 so I implemented that. I was hoping to be able to turn on TLSv1.3 as well, but unfortunately Ubuntu 18.04.1 ships with a version of OpenSSL that is too old to support it. I saw on a mailing list that it is supposed to be coming though so hopefully soon I will be able to update to that.

Reactive Streams Talk

Of all the regular conference speakers on the Java circuit Venkat is my favorite. He has the ability to break down complex topics and make them very simple to understand. In addition to all of that he also shows you why you should care, and how whatever he is presenting can make your development life better. I always hope that when I am explaining something that I can do it 1/10th as good as he does as then it will probably come across pretty fairly understandable to the person that I am talking to.

Venkat has done it again, I was looking at some news in my blog feed and one of the stories had a link to a talk he did at Devoxx Belgium (a conference I have always dreamed of attending) on Reactive streams. In 2014 I was at Venkat’s talk at SpringOne and it completely blew my mind and showed me why I needed to embrace Java 8 Lambda’s and Streams. With this talk on Reactive he is taking the same approach. Breaking reactive down in a way that you can relate to the Java 8 streams features and showing you why it matters. If you have a spare 2.5 hours this talk is well worth it:

Google Kubernetes Engine

Introduction

I have been messing around with Google Kubernetes Engine for the last few weeks now (as we are deploying my new app to it) and I have to say overall I am impressed. There has been a lot of talk about Kubernetes for a while and at first I was wondering if it was just the next piece of tech being over-hyped like so many things. Having used it now for a month I can say I understand why people are so excited about it. The learning curve is steep, but once you climb it, you will really appreciate the power of the platform.

As stated in my previous post, I have been building a Micro-service architecture in Golang and for deployment we decided to go with GKE. Go seems to be extremely friendly for docker containers, I have been using Alpine as my base and the container size of each service has been really tiny (around 10MB or less). This is quite a difference compared to Java containers that end up very large when you think of all the jar files that go into a typical spring boot app. There are a few things that you need to do to build your Go app for docker. You need to disable CGO so and tell the build to use Go’s networking for DNS resolution and not to rely on GLIBC’s as Alpine is built on musl libc. The other great thing about Go is the cross compiler is built into it, so you are an environment variable away from being able to compile your app for Linux even when running on a Mac as I do. The only other thing I do is add ca-certificates to my alpine base for SSL connections.

History

Initially when I started building my backend I was using Spring Cloud Gateway and Consul to handle load balancing and service discovery. When I went to bring our services to the cloud I discovered those would no longer be needed as Kubernetes has built in load balancing and service discovery. The integration between Spring Cloud and Consul is great, and what happens is you can just register your app name in consul and spring cloud will automatically route to it based on name. I have an Auth service named auth and I would hit spring cloud at http://localhost:8080/auth and it would look up the auth service in consul and route the request there automatically to my auth service which was running on 9999 at the /. I wanted to keep the same sort of approach in my Ingress and service discovery in Kubernetes.

Kubernetes the beginning…

 Initially I started using the default Google Ingress which behind the scenes provisions a Google Cloud Load Balancer (HTTP/HTTPS) load balancer and then should route the different requests to the different services based on your kubernetes service and Ingress. I was having issues getting the Google Ingress to work correctly with the URL rewriting that I wanted to maintain from my previous design and I discovered a second issue with it, which was it doesn’t support Websockets. We are planning on doing a lot of app communication between the backend and front end via a  websocket and this was going to be a painful limitation for us. We considered using Server Sent Events to push events to the client and rest calls for the messages from the client, but this wasn’t ideal.

I did some digging around and discovered that you can install NGINX as your ingress and behind the scenes it provisions a Google Cloud Load Balancer that is a TCP Loadbalancer. The advantage of this is that we can now send websockets into our cluster. Instead of our SSL terminating at the GCLB, it would now terminate at our Ingress when we were ready for it. As soon as I switched to NGINX all the URL rewriting issues I was seeing went away.

TLS Configuration

Once I had reached a point of having traffic into my cluster from the outside, I decided it was time to provision TLS. I discovered Certmanager which allowed me to configure my certificates through Let’s Encrypt my preferred certificate issuer. When I ran the SSL Labs test against my cluster I discovered that SSL was configured really well and I scored an A+ on their test. The only issue I am having is trying to figure out how to enable support for TLSv1.3 I haven’t been able to get the Ingress to support that even after messing with the NGINX configmap. It is supposed in the current version of the NGINX Ingress, but it is disabled by default and I am still fighting that part of the config.

Configuration examples

The final challenge that I faced was supporting multiple URLs for my cluster and just routing the Ingress based on what URL was being requested. I created a fanout Ingress which worked great but I struggled to find a config that would allow me to have multiple secrets for TLS depending on the URL. I finally found that config and thought I would share it here (which sensitive details changed).

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/rewrite-target
: "/"
nginx.org/websocket-services
: haskovec-api-service
name: nginx-haskovec-api-ingress
spec:
rules:
- host: api.haskovec.com
http:
paths:
- backend:
serviceName: haskovec-api-service
servicePort: 9999
path: /auth
- backend:
serviceName: haskovec-api-service
servicePort: 9998
path: /service2
- host: service2.haskovec.com
http:
paths:
- backend:
serviceName: service2-service
servicePort: 9999
path: /auth
- backend:
serviceName: service2-service
servicePort: 9998
path: /service2
# This section is only required if TLS is to be enabled for the Ingress
tls:
- secretName: api-certificate-secret
hosts:
- api.haskovec.com
- secretName: service2-certificate-secret
hosts:
- service2.haskovec.com


The above Ingress shows using multiple hosts connecting to multiple pods behind a service and routing based on the URL and domain name. It also shows what multiple TLS certificate secret stores look like.

To create your certificates with Certmanger you will need to configure an issuer  as below (again details changed):

apiVersion: certmanager.k8s.io/v1alpha1
kind: Issuer
metadata:
name: letsencrypt-production
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
#Email address for acme registration
email: jeff@haskovec.com
#Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-production
#Enable the http-01 challenge provider
http01: {}

And a certificate:

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
name: api.haskovec.com
spec:
secretName: api-certificate-secret
commonName: api.haskovec.com
dnsNames:
- api.haskovec.com
issuerRef:
name: letsencrypt-production
kind: Issuer
acme:
config:
- http01:
ingressClass: nginx
domains:
- api.haskovec.com

You configure an additional certificate as above for each domain name that you are getting certificates for. And just like that certmanager automagically goes out to let’s encrypt gets certs for that domain name and stores it in the referenced secret.

Conclusion

All things considered I am blown away by working in a Kubernetes environment. Google takes away the pain of actually provisioning your cluster so you can focus on your app. Deployments are a breeze, I just push all my new docker containers, updated my deployment yaml file and apply it, and kubernetes does a rolling update of all my services. This is allowing us to basically have infrastructure like we are a company with a huge devops team when in reality we have no devops engineers. I will definitely be using this on my projects going forward!