Java 9 Upgrade

After upgrading my test app to Spring Boot 2.0 yesterday I decided to see how difficult the Java 9 upgrade was from there. I am happy to report that it was fairly trivial. I upgraded my maven pom to set the Java version to 9 and did a mvn clean install.

Immediately I see some no class def exceptions around javax.transaction.Transaction. I did some quick google searching and discovered the problem seems to be in the Maven Surefire plugin. I found a work around that said to set the version to 2.20.1 and added a command line flag of –add-modules javax.transaction. After doing that I was seeing errors around java.xml.bind. Doing some more searching I then added a second –add-modules java.xml.bind. This fixed the issue. In the course of doing so I found a link to the issue on apache’s website. Reading through the comments I ended up with a final configuration of 2.21.0 with the following options:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.21.0</version>
            <configuration>
                <argLine>--add-modules java.xml.bind</argLine>
            </configuration>
        </plugin>
    </plugins>
</build>
Once I dropped that configuration into my pom everything built and ran correctly. So once you get your app to Spring Boot 2.0 the jump to Java 9 is pretty seamless. My follow up change now will be to switch to the new factory methods on List and Set in the app to take full advantage of the features. Once Java 10 drops in a couple of weeks, I will take the App to 10 and see how that goes.

Spring Boot 2.0

Spring Boot 2.0 has finally arrived. Unfortunately we aren’t yet in a position at the office to be able to begin the upgrade so I decided to start playing around with it on one of my projects at home as I didn’t want to wait until we were ready to update our app.

The first thing I noticed about it they removed findOne() from CrudRepository. This to me is a bad change for all the users of the framework. It is one of the most commonly used methods in Spring Data, and not people either have to refactor their code to use findById() which returns an Optional<> of the type and deal with the optional instead of a null or you have to add a findOne with an @Query to all of your repositories. I have worked on projects in the past that have hundreds of tables and Repositories. This change is forcing them to update hundreds of classes just to upgrade to Spring Boot 2.0. It seems to me a better choice would have been to @Deprecated on findOne and encourage people to upgrade to the new findById method.

The next breaking change I found was also in Spring Data. They removed the delete() method from CrudRepository that takes an ID as the type. It has been replaced by a deleteById() call. There is no good reason for this change. In Java we have method overloading so the language can determine whether to invoke delete() with the Entity or delete() with an ID. It seems like they were trying to be consistent with findById() but they just end up breaking a bunch of code in the process. The only benefit that I can see with this change is groovy will get happier as it seems to be very poor at understand overloaded methods.

The next change I noticed was when calling new PageRequest(page, size) it was telling me that constructor was deprecated. To me this is the right way to handle the situation. I clicked into the method and it suggested to use PageRequest.of() instead. That was an easy fix that I made and seems more consistent with Effective Java about using static factory methods instead of constructors.

Finally I got all my unit tests updated to replace the findOne calls with the findById changes I had made and I went to launch the application and Hibernate fell over. It looks like the new version of Hibernate requires me to specify a GenerationType of Identity with MySQL where before it didn’t force me to specify it.

All in all it wasn’t too bad to upgrade my toy application. I am really looking forward to using Spring Boot 2.0 going forward, but a little disappointed by some of the Spring Data breaking changes. I feel like this could have been transitioned in to make it easier to upgrade existing Spring Boot 1.5.x applications. It is interesting that 2.0 is the first release to support running under Java 9, and it is finally released a month before Java 9 goes away and Java 10 is released. Up next for my test application will be switching it to Java 9 to see if I see any issues with that.

Upgrading to Java 9

Upgrading to Java 9

Ever since Java 9 was released last fall, I have been wanting to upgrade our software at work to the new platform. I am not interested in the new module stuff, mostly I just want the convenience methods like List.of(), and the platform improvements. I think G1 by default looks good, the new representation for strings to save memory looks like a huge win, and all the performance numbers that I have seen show it to be a big win. Unfortunately this is not as straight forward as one should hope.

Step 1 – Spring Support

Even though Spring 5.0 was released back in December I think which fully supports Java 9, Spring Boot 2.0 is not yet released. I believe it is in RC2 now and about to be released in the next week or so, but at this point we are less than a month away from Java 10, before we will have proper Spring Boot support in Java 9. All of our microservices are currently running on Spring Boot 1.5.10 except for one. It is still running on the 1.3.x version of Spring boot, which brings us to our second Spring issue to resolve. Some contractors that originally started that service did a hack to lazy load @Formula s in hibernate. While this worked in Hibernate 4.x in Hibernate 5 the hack no longer worked so we can’t even update this service until we refactor those entities and move that data into views. That is still a work in progress. The other microservices could be moved to 2.0 when it comes out, but I think their is a consensus that we don’t want to move to 2.0 until we have all our services to a place where we could do that.

Step 2 – Gradle

We are currently running gradle 3.5.1. Going to Java 9 is going to require Gradle 4.2 (which is also a requirement for the Spring Boot 2.0 Gradle plugin). Normally this is no big deal upgrade the gradle.build file edit the wrapper portion to the new version run ./gradlew wrapper and check in the new gradle wrapper. But in this case we still have a developer running on IntelliJ 2016.1 and another on 2016.2. They would have to upgrade to at least 2017.2 in order for the new gradle to work with their IDE. So we will need to convince the whole team to update their IDE if they haven’t already ideally at that point to 2018.1.

Step 3 – FlywayDB

We are currently using Flyway DB 4.2.0. This didn’t work with Java 9 when I was testing it, so we need to go to 5.0.x. Additionally Spring Boot 2.0 requires Flyway 5.0.x. Ideally we want to upgrade this ahead of Spring to minimize risk. Bring the new flyway into production and then once we are satisfied that it doesn’t break anything upgrade Spring Boot.

Step 4 – Docker Containers

Once we have all that software in place we then will have to update our docker containers to pull in the new JVM. It sounds like in Java 9 Linux, OpenJDK ships with an empty certificate store. So that means finding a Oracle or Azul docker container as our base. We then have to verify that New Relic works with Java 9 so we have our container monitoring. After doing all that we can update our base VM to be Java 9.

Step 5 – Update Jenkins

We will need to add a new VM to Jenkins so that as different builds switch to the new Java 9 container we also switch to compiling them with the Java 9 VM (though with the Java 8 flags still at this point).

Will the pain never end…

Finally after doing everything listed above we could change our compiler flags to 9 and start using the features. It is still a painful road to walk to get us there. At this point it feels like by the time we finish the work that needs to be done (as obviously we need to be shipping features and bug fixes, while we lay the foundation for this on the side) Java 11 will be out and we will be moving to that.

Even though Java 9 support is so close to being out in Spring Boot I still feel like I am at least 6 months away from getting to use any of it. This feels much more painful than when I moved an App from Java 7 -> 8. At that point I think it was just a matter of coordinating a JBoss container upgrade from 6.0.1 to 6.4. After we got that into production we updated the VM to Java 8 (still with everything compiling on 7). Then we upgraded our Jenkins machine to have Java 8 for the compiler, and finally we updated our maven pom to use source and target of 8. I wonder if anyone Spring shops will even make it to 9 or if everyone is holding out for 11 at this point (also to get the LTS release). While I really like the sound of the 6 month release cadence actually getting the dependencies in place and lined up to do an upgrade will probably mean most enterprises just end up going from LTS release to LTS release and I would guess that Java 9 and 10 see very little use running production code and mostly will just be things for developers to play with on their machines.

Recap for 2016

End of year recap

It’s that time again. Time to reflect on my themes for the year and see how I did on them. This is an annual tradition of mine to see how I am doing in general. I find it is a useful accountability tool for myself to make sure I am not wasting too much time on unimportant things and am spending the time that I would like on personal development.

Theme 1 -Regular Blog Updates

This theme was an epic failure. I count 21 posts for the year. Less than half of last years posts. I could make a couple of excuses but it all comes down to the major changes in my life in 2016. It started in February with the birth of my 3rd child. The first 3 months of a new baby wipes you out mentally and physically so that already made things a challenge.

After that I made another major life change, I took a new position as a Principal Engineer at a startup company. This was a great change for me and I am really happy to be working on a new exciting project. Back in April I was working on a major architectural project at my previous company and all of a sudden 3 different recruiters hit me up with 3 interesting opportunities. Like most people in software development I see about 3 – 4 job inquiries per week from various recruiters. I would say 95% of these positions sound terrible, so it is pretty easy to dismiss them generally. I was in a very good position in my previous company so it is also easy to be selective about what you are willing to take. So to see 3 really good opportunities at the same time where the work sounds exciting, the companies sound interesting and the pay is very competitive is most unusual for me.

I started down the road with the first company, when the second recruiter contacted me. So I started talking with them and it is at this time I realized that the first company wasn’t going to be a culture fit for me. Their culture would require that I fly to Portland every 6 weeks to meet up with the other development team there and that was just too much travel when I have 3 young children. The morning that I was going to do my onsite interview with the second company a recruiter that I have talked with for about 8 or 9 years dropped the 3rd opportunity into my lap. Another startup position which had me interested. I arrived for my onsite interview at the second company and we spent some time discussing the type of Software Development that I enjoy doing as well as what I would like to do ideally with a software architecture. The interview went great and about 3 hours after I returned home from that interview I was contacted by my recruiter and he said they wanted to move forward with an offer. I was pretty excited by everyone I had met so I didn’t end up moving forward with the 3rd company as I had found what I was looking for.

Theme 2 – Read Lots of Books

As expected this fell off for the year compared to the previous year with all the new changes in my life. But I did really get back into it in the fall here and have read some good stuff. On the religious side I have been working on Last Testament by Pope Benedict XVI, and The Confessions by St Augustine. On the programming side I have been working on Functional Programming in Java by Venkat. I can’t recommend this book enough it is absolutely amazing. In Non Fiction I read Hillbilly Elegy by J.D. Vance. This book is also amazing to understand a part of the country that is often ignored. Anyone who was confused by the election results this year should really read this book as I think it lays out why middle America is so angry. Anyway that is a small selection of what I have read or am reading.

Theme 3 – Work Stuff

My third theme was some work stuff that I wanted to get done. That didn’t happen due to me changing jobs, but at my new company I have a new list of things that I want to get done for 2017.

Theme 4 – Swift

While I didn’t get around to Swift or iOS development this year I did spend some time in a different language. For the last 2 months we have been doing a lot of front end work at the office, so I have been learning angular.js and lodash. In the past I haven’t found JavaScript that interesting, but I have to say with using angular and lodash it is growing on me. Lodash is ridiculously powerful and I feel like the key to being productive in the JS world is to know this library. Angular brings a lot of patterns that remind me of Spring development which is also nice coming to it from my background.

Theme 5 – More work stuff

Again I didn’t hit this theme as I left my previous role before I got to this, but I did do something sort of similar. When I was interviewing for my current role I mentioned that I would like try doing a micro service architecture built around Spring Boot. I felt like this might make for more maintainable code long term. One week into my new job I was tasked with rolling out this architecture and designing and implementing the first micro service. We had 2 sprints to get it done. It took a big push, but we made it and it was a huge success. Later this summer we rolled out the second micro service. The other developers have been very happy with this architecture and it looks great from both a scale-able application standpoint and from a devops standpoint as Spring Boot is built with devops in mind which really makes it easy to roll out new services.  We will be beginning our 3rd Spring Boot app starting with the first sprint of 2017.

The other aspect of this theme which is relevant to my new role is there are certain aspects of our legacy app’s architecture that I would like to evolve and improve and I have already begun working on that. I am reworking the persistence layer to more heavily lean on Spring Data JPA and less raw Hibernate. I am also reworking the Controllers to use the latest Spring annotations like GetMapping and PostMapping as well as the way we wire the services with the goals of just making all new development on the older platform faster and more lightweight.

Theme 6 – Open Source Contributions

This was a failure. I did some reporting on issues so I at least tried to work through issues I saw, but with the new baby and new job I was far too short on free time and mental energy to dive into another code base and try to contribute to it.

Closing thoughts

If we just want to score this on what I hit it looks pretty bad. That being said I am really happy with my year. It has been a great year. As I said in the post laying out the theme, it doesn’t really matter to me whether I do these things what I care about is personal growth. That is part of the reason for themes and not concrete goals. And on the growth concept this has been a fantastic year. The year turned out completely differently than it started out and I am really happy with where I am. Over the next few days I will lay out the themes for 2017 for me.

MacOS Sierra Slowdown update

I have an update on my slowdown issues on Sierra. It appears the real problem lies in the AWS Java SDK. After talking to the spring boot people via github they were able to narrow it down to an Amazon issue. I opened an issue on github with Amazon and they responded that the version of the SDK that ships in the current spring cloud has this issue in it, and it has been fixed in a newer version of the SDK. One of the big value propositions of Spring Boot to me and the release train concept of Spring Cloud or Spring Data is that it is a collection of dependencies that have all been tested together, which lowers my risk of using them together. So I opened a request with Spring Cloud AWS to upgrade their SDK. Unfortunately they don’t seem very timely in responding to issues as I notice it looks like there are no responses on any of the issues raised in the last 2 weeks.

In the meantime I have a work around that doesn’t involve having to manually bump up your AWS SDK version and that is to set the following property in your application.properties file of your Spring Boot App:

cloud.aws.credentials.instanceProfile=false

Obviously setting this flag for your app running on AWS that uses the IAM profiles isn’t good, but it is a good local workaround on your development machine until the SDK gets updated in Spring Cloud.

MacOS Sierra massive slowdown in Java networking

I upgraded to MacOS Sierra, and have really been enjoying the shared clipboard. I haven’t really noticed any other new features that impact my day to day development, however I would advise Java developers to avoid it for the near future. I have searched and searched and I wasn’t coming up with any results. Then I found this blog post. This is definitely part of the problem. I made the host file changes and got a slight boost, but not enough to explain the whole thing.

First our original application is a monolithic Spring 4.2 app running in Tomcat. On my MacBook Pro it was taking 21-25 seconds to launch. After going to Sierra it started taking 75 seconds. Once I did the host file fix in the blog post above it started launching in 59 seconds. Still much worse than before.

Another Spring Boot micro service we have that does a lot of external api calls and also communicates through Amazon SQS would previously launch in 11 seconds on my laptop and now takes 141 seconds (150 prior to the fix outlined above). Even worse the messages to and from SQS and the third party API’s are painfully slow, sometimes taking 30 seconds or more to go through when previously they were almost instantaneous.

I am still searching for a fix for the issue, but it is practically unusable. I am debating if I am going to roll back to El Capitan, but at this point I would recommend people hold off on upgrading until there is a good understanding of what the slowdown is and how to fix it. Java itself when not hitting the network seems as fast as ever.

A Two Month Recap

It has been a crazy couple of months. Since I last posted I made a trip out to San Francisco to meet the rest of the team I work with (and had a great time). If you live in Texas there is no better time to visit San Francisco then at the end of July. It was a welcome break from the heat. I had a great time and realized when I was out there that I hadn’t been out there since 2005, so I was over due for a trip. I had forgotten how much I love that city it is a really fun place to hang out (though not a place I would really want to live).

We are also about to launch our second Spring Boot Microservice. This one I am really excited about it is even more ambitious than our first one and should provide huge value to the business. The interesting new technology we used in this one is Amazon’s SQS (Simple Queue Service). I am going to say off the bat that the documentation for Spring Cloud is kind of crap, so I spent a lot of time debugging the framework to figure out how to integrate this thing. But now that we have it working it seems to work great. It definitely is pretty light on features compared to something like RabbitMQ, but it seems to get the job done and the Spring Integration is great. I am hoping to put out a blog post on how to configure it as we got hung up a few places with it.

One of the nice things about going to SQS is it forced me to upgrade our Monolith to Spring 4.2 from 4.0 as the current Spring Cloud Brixton release train won’t run on Spring 4.0. I am hoping to roll out Java 8 there as well in the next couple of weeks so that we will be sitting on a fairly current framework on our oldest app and our Microservices running the latest and greatest of everything.

We already have 2 more microservices planned out, it is just a matter of figuring out how they will fit into the product roadmap. I think once we get that delivered we will have cut our monolith down quite a bit and I hope at that point it will be small enough for me to convert it over to boot and simplify it. As I see lot of opportunities to really get everything we are doing lean which should let us turn around features for the business even faster.

Anyway that is the latest update, hopefully I can carve out some time to work on a SQS configuration post. Until next time…

We are live and MySQL settings

We’ll do it live..

The big news is our new Spring Boot micro service went into production! It has been running in production for just over a week and we have had 0 issues with it, everything just works! It ended up being two crazy sprints to get it done, but we shipped it this week with no production issues after it went live. My first big project at the new company couldn’t have gone any better. The team really pulled together to get it done which makes it even more rewarding.

MySQL Database Connections…

We did have one near miss though. It turns out that MySQL drops database connections that have been idle for 8 hours. We discovered that the morning we were about to go live. I really think this should be more widely publicized. It seems like the sensible opinionated default of Spring Boot should be to have the connection pool test the connections so that people don’t end up in this state. Anyway luckily additional testing the morning of uncovered this and I was able to find out how to fix it with some digging around. Just so I remember for the future the magic settings are:

The settings…

spring.datasource.remove-abandoned=true
spring.datasource.remove-abandoned-timeout=600
spring.datasource.testOnBorrow=true
spring.datasource.testWhileIdle=true
spring.datasource.timeBetweenEvictionRunsMillis=60000
spring.datasource.validationQuery=SELECT 1

What it all means…

So breaking it down, testOnBorrow checks the connection before the connection pool gives it to you. This is my first fix that I started with as it it has the connection pool test the connection with the validation query prior to returning it back to the application. This is good enough to fix the issue, but if all connections are killed like if the database had gone down and came back up it is a little slow the first time the app goes for a connection as it has to run through everything in the pool before it allocates a new connection for you.

The second thing I found was the test while idle and the time between eviction runs. With that every minute we test idle connections. This is great for 2 reasons, it keeps the connections alive with MySQL and if someone gets messed up with one of the connections we can be proactive about cleaning it up.

The third set of options I discovered in all this is the remove-abandoned option. This protects you from leaking connections in the application. Above it basically says if we haven’t returned a connection within 600 seconds consider that connection to be abandoned and remove it from the pool. I think with all of these options you are going to make your application much more resilient to database connection issues.

IntelliJ Idea Bug in 2016.1.2

I hit a bug in Idea 2016.1.2 that I wanted to share in case anyone else is hitting the same issue. I was trying to stand up a new Spring Boot project last week and when I what try to launch the app through the IDE the embedded tomcat server would throw an exception.

org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.context.ApplicationContextException: Unable to start EmbeddedWebApplicationContext due to missing EmbeddedServletContainerFactory bean.
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:133) ~[spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:532) ~[spring-context-4.2.6.RELEASE.jar:4.2.6.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:118) ~[spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:766) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at org.springframework.boot.SpringApplication.createAndRefreshContext(SpringApplication.java:361) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1191) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1180) [spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at com.chooseenergy.auth.AuthApplication.main(AuthApplication.java:14) [classes/:na]Caused by: org.springframework.context.ApplicationContextException: Unable to start EmbeddedWebApplicationContext due to missing EmbeddedServletContainerFactory bean.
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.getEmbeddedServletContainerFactory(EmbeddedWebApplicationContext.java:185) ~[spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.createEmbeddedServletContainer(EmbeddedWebApplicationContext.java:158) ~[spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:130) ~[spring-boot-1.3.5.RELEASE.jar:1.3.5.RELEASE]
... 8 common frames omitted

The weird thing was if I ran the maven spring-boot:run plugin the app ran perfectly. I did all sorts of things with no luck to work around it. I starting worrying about it, thinking if I can’t find a solution to this, it is going to be hard to sell using Spring Boot for our Microservices in my new role as we can’t even properly debug them. Finally after no luck I opened a bug against Idea over here. As is often the case, just the simple act of reporting the bug gets me thinking about possibilities and I solved the issue before I heard back from JetBrains. It turns out that the provided scope is broken in Idea and by default the spring-boot-starter-tomcat is marked scope provided in the generated maven pom file from the Spring Boot Initializer. As soon as I removed the provided scope Idea got happy. Jetbrains later updated the ticket as a duplicate of a bug with provided scope not working. So if you see that stack trace in Idea try removing the provided scope to work around it.

Gotta love open source and github

The Problem

I have been working on this project to make our app run in any Java Container. Currently we run in JBoss, but ideally I would like the app to work in JBoss or Tomcat, or TomEE or Wildfly. One of the challenges in making this change is to remove JBoss specific dependencies from our app and pull those libs into the webapp as part of our project. We did the first piece of this a couple of years ago when we stopped using JBoss’ version of Hibernate and pulled a newer version into our app. We have since upgraded JBoss versions so this is somewhat moot since the bundled version and our version are the same, but it is one less thing that I will have to deal with as part of this project.

The first part that I have decided to tackle is not using JBoss’ built in transaction manager and instead bundling one. Looking around it seems like my choices are Atomikos, Bitronix, or JBossTM in standalone mode. Our situation is somewhat more complicated based on the fact that we are not running an XA jdbc driver. Originally the company was using one, but at some point ripped it out because Microsoft SQLServer wouldn’t work correctly or scale with it in there. We are aware of the limitations of not running in XA mode.

Bitronix

That alone rules out Bitronix as they pretty much require XA to do their thing. I had also sort of ruled them out given that the commercial company behind them ceased operating some time ago, and ideally we want to be in a position to pay a company for support. I will say one nice thing about Bitronix is it does have good Spring Boot support.

Atomikos

Atomikos does have a paid supported version if we found we needed it down the line and they have support for a Non XA datasource. Another thing Atomikos has going for it, is that we are already using it for our persistence unit tests, though not with the NonXADataSourceBean but rather with Apache Commons DBCP. I switched over to the NonXADataSourceBean version and most of the unit tests seemed to work which was promising.

JBossTM aka Narayana

At the end of the day though my thinking was the lowest risk solution would be to use JBossTM in standalone mode. Given that we are already using JBoss’ built in version of it this seems to be the least amount of change we can do. The first challenge is actually finding anything out about the project. Just searching JbossTM or JBoss Transactions doesn’t find much for you. I eventually discovered it is called Narayana. Ignore the horribly designed website, I just found that tonight, I originally found it on the JBoss home pages and on github. JBoss has a quickstarts example github repository showing you how to use their stuff as well. The Spring example was sort of what I wanted, but not exactly so I figured why not just do a quick Spring Boot app as a proof of concept. So I forked the repository and added a Spring Boot app. I hit a problem though, I created a unit test to try out reverts my insert wasn’t reverting on a Runtime exception. Just to make sure I wasn’t completely crazy I created a quick Spring Boot app doing the same thing using Atomikos. That test app worked perfectly.

At this point I feared I would be forced into switching to Atomikos as that was all that I could prove was working with a trivial test. I went over to the JBoss forums as a last ditch effort and posted a question about it. Back in the day when I would post things to forums I would often get very little response or things would be vague cause it is hard sometimes on the other side of a forum to be clear what a person is asking about. In this case I could just include the links to the two github repositories as examples of what I was trying to do. I posted it and didn’t hear anything back on the day. The next morning I woke up and saw that at 3am a Redhat Engineer in China had forked my project and provided a fix. Immediately I could see what I was missing and he made my test pass right away. Back in the day I probably would have given up and just gone Atomikos, but with github it is so easy to share sample code of what you are trying to do and Spring boot makes it so fast to stand up a sample app there is really no reason not to do so anymore.

 One more note while I was working on this experiment in testing how to configure different JTA managers the Narayana guys submitted code to Spring Boot 1.4 Milestone 2 to include a Narayana Quickstarter so now it will be possible to easily wire that into a Spring Boot app. I know I talk about the greatness of Spring Boot all the time, but once again it saved me a ton of time by allowing me to prototype out a few options before I go too far down the road of solving the problem, and with github allowing me to collaborate with people around the world, we truly live in some amazing times.