Categories
general

Our Engineering Process

I haven’t been active on here lately just due to how busy we have been in getting our product on the market. That being said I did end up writing a post for our company blog about the engineering process we have built up over the last couple of years. I figured I would share it on here as well. Check it out here: https://wndyr.com/is-what-were-doing-driving-value-as-engineers/

I am a strong proponent of Agile development and I like the mind shift about thinking about value creation and who is the ultimate customer for what I am building. I hope that it is useful for you as well.

Categories
Golang

Running Sonar from Bitbucket pipelines on your Go apps

In the past when I was a Java developer we would run Sonar on our projects for static analysis. I have always liked the dashboard view it provides and the way it can find all sorts of problems in a code base that are often overlooked. When I learned that Sonar supported Go I knew that I would eventually integrate it into our environment. Since I had already built out our continuous integration pipeline in Bitbucket, I figured it would be easy to integrate Sonar into our builds. Little did I know that there wasn’t much documentation out there on the internet showing how to do so.

I knew I need to run the sonar-scanner-cli against my project, but the only example I could find was this helpful blog post here. I started out with that approach but had an immediate problem. Before I integrated my sonar step in my pipeline, my builds took about 4 minutes from the time a feature was merged into master until the software was running on Kubernetes in the cloud. After I followed that blog post my builds were taking an extra 3.5 minutes. Given that you are billed based on build minutes and the amount of code we ship this was going to be unacceptable for us as we were going to chew through too much build time.

Setting up your pipeline

Before we go any further though let’s setup a build pipeline for go. My post assumes that you are using Go modules as it makes the pipeline much simpler. My initial pipelines were built before I modularized my code and there was more setup. Given that in Go 1.14 modules are the default I am not going to document the old GOPATH approach.

Start with the top of your pipeline file and you will define it like so:

image: golang:1.14.1

definitions:
  caches:
    go: $GOPATH/pkg
    sonar-cache: .sonar

We will use the golang docker container as our base container for our pipeline. We will then define caches for our Go module dependencies and our .sonar directories. This will speed up subsequent builds and as mentioned above in bitbucket pipelines time is money.

Build Steps

Once we have that defined we define our build steps we will look at the main two steps for our objective here:

 steps:
    - step: &build
        name: Build the app
        caches:
          - go
        script:
          - git config --global url."git@bitbucket.org:".insteadOf "https://bitbucket.org/"
          - go build
          - go test -cover --coverprofile=coverage.out ./...
          - go vet ./...
        artifacts:
          - coverage.out
    - step: &sonar
        name: Sonar code analysis
        caches:
          - sonar-cache
        image: sonarsource/sonar-scanner-cli:4.3
        script:
          - export SONAR_LOGIN=$SONAR_API_TOKEN
          - export SONAR_PROJECT_BASE_DIR=.
          - /opt/sonar-scanner/bin/sonar-scanner -Dsonar.login=$SONAR_API_TOKEN

The first thing we do is a git configuration. This is because if you use libraries in a private bitbucket repository go get by default will fail authenticate. This tells it that any requests to bitbucket should be by ssh instead of https. You will then need to have ssh keys configured on your pipeline.

The next thing we do is build our code and run the unit tests. We use the built in Go code coverage and output the coverage to a file. We also run go vet which is great for finding issues in your code. At the end of that build step it saves the coverage file for use in later steps and also saves the Go cache so that all the dependencies that were downloaded for your build don’t need to be downloaded for each run.

The second step above is the sonar step. Since downloading the entire cli tool each time was too slow, I figured I would start with the docker container published by sonar source. This loads much faster than copying the tool down. Once that image starts up we set some environmental variables and then invoke our scanner. This setup assumes that you have created an API token for your sonar instance and that you have configured it as a pipeline variable in bitbucket.

Sonar config

There is just 1 thing missing now and that is our sonar config. In the root of your project create a sonar-project.properties file to configure your settings. Mine looks similar to this:

sonar.projectKey=haskovec:MyProject
sonar.projectName=MyProject
sonar.host.url=https://sonar.haskovec.com

sonar.sources=.
sonar.exclusions=**/*_test.go,**/vendor/**,**/mocks/**

sonar.tests=.
sonar.test.inclusions=**/*_test.go
sonar.test.exclusions=**/vendor/**
sonar.go.coverage.reportPaths=coverage.out

sonar.sourceEncoding=UTF-8

Here we configure a project name and key and point to the URL of our sonar instance. Then we configure which sources to scan and which to exclude and which tests to scan. We also configure our code coverage file here. Once we have this we can finish our pipeline. Once we have our steps declared we have our pipeline run on master as shown below:

pipelines:
  default:
    - step: *build

  branches:
    master:
      - step: *build
      - step: *sonar

This says whenever anything happens (someone pushes a commit to a branch) we will run our build step. We then setup a branch specific rule that whenever anyone merges code into master we will build our code and run the sonar analysis on our code and publish it to our sonar server.

Conclusion

That is all you need to do to get a very basic pipeline up and running on bitbucket which will build your Go app and run sonar against all new features merged into your master branch. I hope this helps and saves you a bunch of time as it took me a while to figure out how to get it running well.

Categories
general

Happy New Decade

The 2000s

With the turn of the new year it got me thinking a bit about the change in the decade. Hard to believe it has been over 20 years since I was celebrating the change of the century with my friends in Minneapolis. Things have changed drastically since then. At that time I was 6 months into my career. I had joined a telecom startup and moved to Texas after graduating from the University of Minnesota. The 2000s were a decade of change and growth for me. It had me working really hard to get established in my career. Going through a couple of acquisitions and finally moving into my second startup. I met my wife then and got married, and we transitioned from apartment life to having a house. We traveled as much as we could. Mexico was much safer back then and we explored a lot of central Mexico as well as trips to Europe every year.

The 2010s

Along came the 2010s and it was another big shift for us. The 2010s were the decade of having children with 4 children to join our family. It is also when I got really career focused as well and decided that I didn’t just want to be a coder, but I also wanted to really move up and make an impact at work. I started focusing on what I could do to drive more value individually which lead me to moving into architect roles, team leads and technical leadership type opportunities. I also left my 2nd startup and took a corporate role which taught me many things that I needed to be a more effective technical lead and a better software development process. I then left that corporate role for startup 3. This was the first real turnaround startup that I did and we successfully turned the company around and were able to bring it to an exit. I also worked on the highest performing software team of my career at that point there and really saw how to build an engineering machine that could consistently crank out software and value for a business.

After we sold that company I needed to land a job as we were expecting our 4th child. That resulted in me taking another corporate job. That role had me with a great team, but in a business that had all sorts of issues. My main learning there was in many ways things not to do. I left there with some really good friends, and really happy to be out of an unhealthy work environment for me to take on startup 4.

Startup 4 was the first time I was the first engineer, and really got to build a complex system from a founder vision into reality. It has been extremely rewarding and I have learned a ton in the process. We took a vision and have driven it to an early access release that we are evolving into a full featured product.

The 2020s

That leads me into where we are now. I have started to think about what the focus of this decade will be. This next year is going to be crazy and we are going to accomplish big things as we hit product market fit and our startup takes off. In many ways this is going to be a hyper focused decade when it comes to career. I am at a point where I can make a huge impact and I need to really dig deep in my growth areas and help bring us to the next level. I think your 40s are really your prime earning years as you have so much experience you can really drive huge value to a business. Ideally we will have a huge success in my current company and maybe another startup by the end of the decade. Whatever happens I am sure it will be interesting.

Categories
Uncategorized

2019 Year in Review

Wow what a year. It has flown by and as you can see I haven’t gotten to the blog since I upgraded the site earlier this year. Looking in my drafts it looks like I was intending to write a post about travel, but I didn’t get back to it. Anyway given that the year is rapidly ending I figured it was time to start looking back on it.

I never imagined how busy this year would be. We went from the start of the year where we created a plan on what we were going to build for the year to drive to an MVP. We were building out what we thought would be our alpha release, but discovered in user testing that we needed to pivot. Along the way one of my employees left. We then drove the new concept to an early access release grew the engineering team and it has been an amazing ride. 2020 is going to be focused on product market fit and hopefully upon achieving that, rocketing into a growth phase. Given how crazy it was to build what we have built on such a lean team a lot of my themes for the year fell to the wayside especially this blog. When you are super busy and tired this is an easy thing to drop.

I feel like I have gotten much stronger in Go this year which has been nice growth. I no longer feel slow in it as compared to Java like I did for so long. I appreciate many aspects of the language, but also find a few areas lacking. I have gotten very comfortable with Kubernetes and Bitbucket pipelines and built out a nice CI/CD pipeline this year to allow us to run faster at work.

We also spent June traveling which was nice. I took advantage of working for a remote company by working from Mexico for a month and for the family taking a 2600 mile road trip in June. It was great to spend time in places I haven’t been for 10-15 years as well as spending time at my in-laws so the kids could hang out with their grand parents and some of their cousins. I would like to incorporate more remote work in the future.

The big challenge I see going forward in 2020 is moving myself out of the operator mode and into a more strategic role. Most of my past roles have been about execution getting on the ground and grinding out the product to drive us forward. For 2020 I need to be more much strategic in the business and not just on the day to day to really help drive the vision forward. Without further delay let’s review my themes for the year.

Theme 1 – Health

I did fast through Easter and lost about 20 pounds. Some of that has slowly crept back but it was a good start to the year. I think for 2020 I need to go broader and think more along fitness goals. This needs to be some amount of walking and maybe some amount of strength training. I need to think on it yet over the next few days. I have many times made better choices food wise this year. I eat more salads that I used to, but I probably need to continue this and extend it more.

Theme 2 – Learning

There has been a lot of learning in 2019, but not in the way I envisioned. I did most of the Coursera Machine Learning course, but got busy near the end of it and didn’t finish it. I thought the course seemed too much like a CS course in that you are building everything from the ground up. While most of my degree was done that way that isn’t what I am looking for anymore. In reality I am not every going to build all these ML techniques from scratch, I am going to use a library or tool that someone else built. I think a more valuable class would be an overview of what the different available techniques are and when you would try different approaches to get different results. Also more time needs to be spent on teaching people data exploration and sort of things you can try on a given set of data.

I also didn’t get all the reading done that I had set out to do. I found that I was so tired in my evenings it was difficult to do a ton of reading though I did get some things read.

I think most of my learning in 2019 was really focused on learning how to build a team, and build a product from 0, and really learning how businesses work. I really appreciate the view I am getting and what I am learning about building a company and getting to see the broader view of how sales, marketing, finance, and customer all really come together to drive a business forward. We also focused on leadership training this year and I was fortunate to go through that program.

Theme 3 – Security

This theme was a resounding success, I have gone all in on the password managers with really long complex passwords that I don’t know and 2 factor authentication. It was a good change and I can’t imagine going back to the old way of doing things.

Theme 4 – Public Speaking

This theme didn’t happen and was an epic failure. I started the year out noodling around an idea for a talk and putting some stuff together, but ended up getting way to busy again to devote time to it. I think this is something I will need to revisit in 2020 as it will be important to get our name out there in the community as I think this could really lead to us getting a nice hiring pipeline. The perfect scenario would be there are so many people that want to come and work with us, that when we have an opening there are tons of people competing for the role and we can continue building out a strong team of really talented people quickly without some of the hiring lag I faced this year on getting the right candidates. The only way we are going to get to that is if people around here no who we are and what we do and want to be a part of it.

Conclusion

All in all it was a bit of a mixed bag on my plan for the year. It was a good year though, and now I need to start thinking about my themes for 2020. I am hoping to get some more rest in before I get back at it in January unfortunately it has been a pretty busy holiday so far so I am hoping I can slow down for a bit for a couple of days.

Categories
general

Site Upgrade

My site was down for much of today as I finally took the plunge and rebuilt my image from ground up. This was something I had considered doing for some time now. The old site had been through a couple of ubuntu upgrades and was chugging along, but I always thought it would be good to do a fresh install. However I never really wanted to dump the time into it. Then I saw this article: https://www.anandtech.com/show/14284/amazon-offers-another-amd-epyc-powered-instance-t3a

I have been interested in AMD’s latest processors as they seem to be killing it with their Ryzen line. I checked the pricing and saw that the price for running my site was about half of the cost of what I was paying for my old t2 instance. Additionally at the same price level I was getting more CPU cores with the new instances. I hadn’t even realized that Amazon had rolled out their regular T3s or these new T3a instances. I figured it would be an easy switch, stop my instance, change the instance type and start up the new instance.

I attempted that with no luck. It seems that since I setup my site Amazon had come out with a new Network card driver, which my Linux kernel didn’t have. I switched back to my T2, and decided I would start building my new image on the side. When I had time I would spin up my new image and start working on the Amazon setup. I was happy to see there is now IPv6 support available. All in all there was just lots of new stuff to play with. I finally reached the point yesterday where I decided to backup my site and finally go for it. I had hit a stopping point where I brought my nginx config over (which had all my SSL settings), but I didn’t have my DNS pointed to the new machine in order to pull down new certificates. Without the certificates nginx wouldn’t launch, and at the same time because of HSTS preloading I can’t test an unencrypted version of my site. I pointed to the new IP this morning in my DNS settings and then proceeded to start working on restoring the backup of my site while I waited for the old DNS cache entries to expire so that I would pull down my new certificates.

Right around that time I had to depart as we had mother’s day plans and thus the site needed to remain down. When I got back on tonight I finally finished restoring my backup of wordpress and we are back. Now that I am back up I realized that my SSL Labs score dropped so I have been tweaking my configuration to get back up to my high A+ score.

Future projects now will include figuring out IPv6 routing. I was able to assign an IPv6 address to my instance and VPC, but when I try to SSH to that address it seems to time out. When I get that working I would like to setup DNS to allow getting to my site solely with IPv6. Anyway that is all for now as I need to go to bed, but more updates coming soon hopefully.

Categories
Uncategorized

The Chug

So far this year I have not been keeping to my blogging schedule. Since posting my themes for the year I have been AWOL too busy working on launching a product in my current role. That being said a friend of mine just launched a new show on YouTube called The Chug. If you are into craft beer like I am check it out at: http://thechug.life

Categories
general

Themes for 2019

I am late to the party. Normally I tried to take off until Epiphany before I return to work, but I needed to start working on the 2nd so I wasn’t able to get to this post until now. As is my tradition I set my themes for the year that I want to focus on. I find doing so helps me stay on track and make sure that I am growing and improving myself.

Theme 1 – Health

I am carrying this one over from last year but it is always a good one for every year. In my feasting for the holidays I put on a little weight so I am going to attempt to lose 20 pounds before Easter. I hadn’t decided what I would do at the start of the year, but on Epiphany I decided I would do intermittent fasting and cut back on beer (and try to eat fewer carbs when I am eating). So far I am down 9 pounds 2 weeks in, so I should be able to exceed my minimum by Easter since the fasting will continue until then.

In addition to the health benefits of losing weight, I also would like to look better. Later on this year I will be involved in pitching for a series A and I want to look my best in those meetings. Ideally I would like to be more active this year too. When I was testing the Apple Watch that Sofi had I must admit the gamification aspect of closing rings was getting my to walk more and when I was walking was pushing me to walk faster to keep the heart rate up and close the exercise circle. So maybe I will take the plunge this year on one. Though at this point if I do that I would probably hold out for the series 5 in the fall.

Theme 2 – Learning

I have a couple of things in this bucket for the year. First is the machine learning course that I mentioned a few weeks ago. I am just about finished with the week 2 work and I can already see the benefits. It is already kicking my brain into gear thinking about the problems that we are trying to solve at work. Additionally I feel like the base knowledge will enable me to ask the right questions and help keep the ball rolling on that part of our system.

The second area of learning is going to be focused on is reading. Traditionally this is something I really push myself on, but I have fallen away from it a bit in 2018 just with interrupted sleep due to the ages of my children. Now that I am past that stage 2019 is going to be refocusing on getting through a bunch of books. Currently I am reading Leaders Eat Last which was a Christmas gift from my boss. I want to get back into my old pattern where I would try to read 10% of a book a day so I can get through a book in 10 days

Theme 3 – Security

Given all the sites that are constantly in the news each year with getting data stolen, I have decided it is time to take my password security to the next level. Instead of hard passwords I memorize I have switched to using a password manager and generating really long and random passwords that I don’t know. This makes is possible now to have no repeated passwords anywhere and I have found that it has already simplified my life.

I am also considering going the hardware key route instead of just using an authenticator app for 2 factor passwords, but I haven’t yet decided if I want to do that.

Theme 4 – Public Speaking

I have never been very comfortable speaking in public. Anyone who knows me knows that I am very talkative after I get to know a person, but something about being on a stage with all that attention directed at me has always made me uncomfortable. That being said it is a skill that I need to develop to go where I want to go in my career. Beginning this month I will be speaking at our board meeting, and then with needing to fund raise later on it is time for me to conquer this.

Fortunately the timing couldn’t be better in that I was also invited to speak at a developer meetup this year. I can’t think of a better way to practice this skill than to be speaking to a friendly audience of developers discussing technical topics. As that is an area in which I always enjoy speaking. Over my holiday break I began work on a talk, but when I realized it was going to take too much time I pivoted over to the Machine Learning class first as that is a greater need for me. Once I finish that class I intend to focus my time on putting this talk together and then I will get myself on the Calendar as a speaker (hopefully by Q2).

Conclusion

Those are my areas of focus for the year. As always I will do the run down at the end of the year and see how they turned out, as well as what popped up that I didn’t predict for the year. May everyone have a happy and productive year.

Categories
general

Coursera Stanford Machine Learning Course

I just signed up to take the Stanford Machine Learning Course for free on Coursera. Anyone who wants to take the course with me is welcome to join me over here: https://www.coursera.org/learn/machine-learning. They are only accepting enrollments until January 12th I think it said so not much time left to sign up.

Categories
general

2018 Year End Review

Recap for 2018

As is my tradition it is time to review 2018 and see how my year unfolded. The first thing that I always do is review my themes for the year and see how many of them I hit.

In general 2018 was an interesting year. The year started out with me on vacation. When I was on vacation I realized that I would have to change jobs in 2018 and luckily it all worked out for the better. I can say making that move was one of the best decisions I have made in my career. The difference between the first half of 2018 and the last half was a study in extreme opposites on the career front. Probably the biggest takeaway from that whole situation is that company culture may be the most important thing when it comes to choosing a position.

Without further delay here is how I did for 2018:

Health

I didn’t drop as much weight as I had hoped to at the start of the year. I think I lost 20 pounds in the first quarter and kind of slacked off after that. There will be another push for that again this year as I always like to start Q1 as a quarter to fast since Q4 for me is a quarter to feast. I haven’t figured out what approach I Am going to take? In the past I have done the Primal Prescription, Slow Carb, and Intermittent Fasting / Time Restricted Feeding. I will consider what approach I am going to take over the upcoming days. I also did DNAFit this year so it would be good to try to incorporate their diet ideas for my specific genes in whatever I decide to do. I should also push for more regular activity, so maybe in the end I will need to break down and get an apple watch just to use the data to drive accountability for myself. Not sure yet I have been holding out for it to do more than it currently does before I take the plunge. I do like that the series 4 at least has a larger display.

I suspect 2019 will involve more public speaking for me based on how things are looking so I also want get healthier for that as it is nice to look good if you are on a stage or the center of attention.

More Blogging

This theme was an epic fail. As I check my posts not counting this post there were a paltry 13 blog posts this year. Barely an average of 1 per month (vs my goal of 52 in general). This one I don’t even mind missing this year because it was for a good reason. I had so much stuff to learn for my new job this was one of the easiest things to cut to get more time. As I have settled in I hope to do better this year, but in general if I am pushing for a sprint deadline this is an easy thing to cut so I can get more code across the line. Once we get our product to market I hope to be able to plan more reasonable sprints.

Learning

I nailed this theme. It has been a great year for learning. Instead of the learning being books I read it has been practical things. First I learned Go Lang and implemented a micro serviced base architecture in it. I am still doing a lot of learning in Go. I haven’t yet figured out really the ideal patterns yet. One thing that I miss from Spring is how opinionated the framework is. There is a remarkable amount of consistency in code when you have the strongly defined stereotypes like that. When you see Go on the internet it is all over the place. Sometimes it feels like node code, sometimes it feels like C code, but it never really feels like people are using the same consistent patterns. I have played around with sort of the controller / service / repository layers and not sure that is the correct idiom. I have played around with just general event handlers and middleware patterns. I think this year will still be me trying to figure out what is the best patterns to use for backend services.

On the business front there was also a lot of learning for me. For the first time in my career I built an agile process that our team uses, and got a brand new team that didn’t know each other at the start off to a very good working cadence. Our process has worked so well that other parts of the organization have adopted it as their baseline and started iterating on it to adapt it to their teams. I have also applied some of the things I learned from domain driven development to build a model that allows the business and the technology teams to communicate with the same language.

Architectural Changes

This was underway when I left my previous company. We managed to convert some of our apps to Java from Groovy. We were doing this on a piece by piece basis at first going after the easy parts. We were also doing some restructuring to address some performance problems in the architecture around the time of my departure. This I would probably only give 50% to. Had I remained in that role I am sure we would have made major changes by the end of the year, but the organization had become too political for me and I got the opportunity of a lifetime at the same time.

Conclusion

All in all 2018 was another great year. The year ended much stronger than it started and I feel like I had a lot of personal growth this year. Looking forward to 2019 it is going to be a huge year for our company and for me personally.

Categories
general Security

Site Upgrade

I decided to upgrade my site to the new version of Ubuntu as I haven’t done that for a couple of years. It is always a nice thing to work on when I am on vacation as it is the sort of thing that I don’t really get around to normally when I am busy. What a pain that ended up being.

The Upgrade for the OS itself went very smoothly as it seems to normally do so for Ubuntu. But the upgrade to the newer version of PHP broke everything with my site. As I think back actually I think this happened last time when I went from Ubuntu 14.04 to 16.04 as well and it jumped from php5 to php7. I ended up with about a 3 hour outage trying to sort everything out.

The big issue I saw was the default user that php was using was different than the nginx user so it couldn’t write to the Unix Domain Socket. I also noticed all the configuration advice for nginx was very different than when I set up this site. It seems like things are laid out better with the whole snippets of different configurations instead of sort of everything going in the default.conf. At some point I may want to start over again from a blank AMI image and import my content into it. Then I could setup nginx a little bit more modern. Seems like there are lets encrypt plugins for it too, so I am wondering if I could have it auto-renew my certificates.

Another thing I could do if I redid the site, would be to switch to mariadb. I have heard it is supposed to be faster than mysql it might be fun to mess with something different. That being said I probably won’t get to that this winter as I am currently working on some content for a talk and I also want to spend a little time doing some machine learning classes on Coursera before I get back to work.

I did take advantage of the time in the config files to figure out how to tighten up my SSL Labs score. I found that I was missing just 1 item to push my key exchange test from a 90 to a 100 so I implemented that. I was hoping to be able to turn on TLSv1.3 as well, but unfortunately Ubuntu 18.04.1 ships with a version of OpenSSL that is too old to support it. I saw on a mailing list that it is supposed to be coming though so hopefully soon I will be able to update to that.