Jackson Bug Update

Well that was fast. I didn’t expect the day after the post to have an update but here it is. One thing I love about open source it is possible to find answers quickly. So it turns out that this is not in fact a bug. Apparently Jackson 2 aims for a Java 6 compatibility level. As a result of this they can’t include Java 8 features in the baseline as that would break compatibility. So they have a module to use the Java 8 features that you can bring into Maven with the following settings:


<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-parameter-names</artifactId>
</dependency>

Once I did that everything started working. I appreciate how quickly the Jackson team responded to my ticket to help me out.

Bug in Jackson JSON serialization of Immutable objects

The bug

I have been using Jackson for JSON processing for many years now. It is a great framework and mostly just works. It is the default framework in Spring Boot and mostly it just sits behind the scenes and drives most micro services in Java these days. That is why I was surprised when we hit a bug in the framework as I hadn’t really had any issues in the past.

The backstory

Anyone who has coded with me knows that I tend to favor immutability for most java objects. By default I make everything final and then only has non final things when they are necessary. This principle is documented in Effective Java Second Edition Item #15. As a side note everyone should read that book. Even senior developers benefit greatly flipping through the book once a year and reviewing the items. It may be the best Java book I have ever read for writing very good code. My hope is that Josh Bloch does a 3rd edition to cover all the new functional operations in the language.

The best way that I have found to create Immutable Data Transfer Objects in Java with Jackson is to have all the fields be final values, and to annotate the constructor with @JsonCreator and each property in the constructor with @JsonProperty("propertyName"). Just like that you have an object that won’t change state and works seamlessly between your Java Service and your JavaScript front end or between 2 Java back end services.

The problem

The problem we found with this approach occurred in an integration with another partner. Our developer was trying to use Java best practices so the variables are camel case and he is using @JsonCreator in the constructor and naming the properties there. While that works great for an inbound object when trying to serialize that object out to JSON Jackson is throwing away those property names and serializing off of the field name. Normally our field name matches our property name so this is no issue. But in this case the property on the other side was something like first_name, but we record the variable as firstName. Jackson is serializing that as firstName if you only use the Constructor approach. If you add @JsonProperty("first_name") to the field it will work. So you are basically left either double annotating the field or going with a mutable POJO object to get the correct output. I have created a project on github demonstrating the problem:

https://github.com/haskovec/jackson-serialization-bug

Additionally I have opened an issue on the Jackson project for anyone who is interested in following this issue. So far I haven’t heard anything back on it.

Closing thoughts

So the real question is how did this thing go through without getting caught. My suspicion is that most developers don’t actually create Immutable objects in general. The other possibility is that people don’t often have an output that doesn’t match the variable names. We could take the ugly approach and declare final String first_name; to fix it but that doesn’t make me happy either.

I will be interested in seeing if anyone responds to the issue and what they say. I think it is always a good test of a project to see how active it is by seeing how long it takes for the developers of the project to at least comment on a given issue found in their code. I did a little bit of debugging into the Jackson code and I can see where it is throwing away the correct property values, but I am not sure what the correct fix is. It feels like a method that if I tried to fix I could easily break some other use case.

Update

Turns out it wasn’t a bug and I got great help on the team to fix the issue.

The road to Spring Framework 4.1

Earlier this year Spring Framework 4.1 was released. I was excited to try out the new features in our project at work and having previously upgraded us from Spring 3.1 to Spring 3.2 and from Spring 3.2 to Spring 4.0 I was expecting this to be another routine Spring update, but alas that was not to be.

One of the first things I do when looking to do a major version upgrade is to check all the dependency versions for the libraries to make sure all of our libraries are new enough to do the upgrade. In this case I discovered that Spring 4.1 requires Jackson 2.x and we were running Jackson 1.9.x.

In theory this isn’t a huge deal. The big breaking change in Jackson 2 is a new code namespace.  So it was basically change some imports fix a few settings where you declare your ObjectMappers and you are good to go. So I made all the said changes and all looked well. Unfortunately a few of our unit tests broke.  It was then that I discovered that one of our data structures being written to Cassandra had a couple of Joda Dates in them that instead of being written out as a nice time stamp string it was actually serializing out the internal structure of the date time class.  And was Jackson 1.9 would deserialize that without issue the 2.x wouldn’t. To make matters worse we had over 40 million records written to our Cassandra database in this incompatible format. Given the size of the data and the inability to take an outage to fix the data structures I needed to come up with a solution to this problem.

At that point I had to revert the changes and they couldn’t go into our build for that week. As it happens I was fortunate enough to attend SpringOne 2GX 2014 so while there I attended a talk about upgrading to Spring 4.1. After the talk I went up to the speaker and explained my dilemma, but he didn’t seem to have any useful advice for me.  I think there was maybe a suggestion about writing a process that deserializes with the old library and then rewrites the record with the new library which is doable given the new namespace. However that solution wouldn’t work for us as we can’t break the functionality while the data is being updated to the new format.

The solution that I settled upon when returning to the office was instead to write a custom Deserializer for DateTime objects. My custom Deserializer understood both the old and the new formats and would look at the content of the JSON and based on that pull the relevant data to construct our DateTime Object. Then on the serialization side I used the Jackson plugin for JodaTime to serialize. So what happens is anytime a record is written in the Cassandra table it is written in the new format and if it is read we can read both the formats. This was the only workable solution I could come up with given our requirements. At that point I noticed with my new change set we were still getting the old Jackson 1.9 libraries in our build as well as the new version. Upon investigation I found that our old 1.2 Cassandra driver was pulling in the old Jackson library. I checked the documentation for the 2.0 driver and it said it was compatible with 1.2 versions of the database. The 2.0 driver did change the way they collect metrics but it wasn’t too large of a change so I made those changes as well so we could finally get rid of all traces of the old version of Jackson.

My code passed all on unit tests and testing I did I my local machine so I pushed it to the code base right after a branch cut to get maximum time for all other developers to run it and test it with me. Everyone ran it all week and it ran great. It was cut in the next weeks build and it sailed through the QA environment. Our staging system ran without a hitch and we were all set to deploy. That week due to unrelated issues the deploy was delayed one night to accommodate some unrelated issues found in the QA process.

And then out of the blue my phone rings at 1am.  It is my boss.  It turns out they are having issues with the deploy. Now for me to be called at 1am means that there have been other issues in this deploy since at that point we are 2 hours into the call. Needless to say this wasn’t a good night from an operations standpoint. Even though this sailed through all test environments in our production system we couldn’t connect to the Cassandra cluster. I was asked whether it was safe to just change the Cassandra version in the pom back to the old version or if we had to revert the entire changeset (including all the Jackson fixes). My first thought in the fog that I was still in from being abruptly woken up is we could just change that lib and then we remembered all the metrics gathering code that had changed. So at that point we had to do a revert of the whole changeset and spin a new build to finish the deploy. At 2:15am I returned to bed and once again we were still on the old Jackson.

It was determined that the Production System was running a different OS version than our Staging Server and that was believed to be the cause of the Cassandra driver failure. So the next week I added back in all my Jackson work, but I left the old Cassandra driver. That finally did it and we solved the first issue we would face on this path to upgrade to Spring 4.1. When I did some followup research I found that on the Cassandra 1.2 driver page it says that the 2.0 version is not compatible with the 1.2 database, even though both the 2.0 and 2.1 drivers documentation pages say they are. Needless to say Datastax has some inconsistencies in what they are saying about the product. Unfortunately the saga isn’t over yet, a new issue has emerged that is being worked up so we can upgrade to 4.1, but I will follow up with a post on that at a later date.