Jackson Bug Update

Well that was fast. I didn’t expect the day after the post to have an update but here it is. One thing I love about open source it is possible to find answers quickly. So it turns out that this is not in fact a bug. Apparently Jackson 2 aims for a Java 6 compatibility level. As a result of this they can’t include Java 8 features in the baseline as that would break compatibility. So they have a module to use the Java 8 features that you can bring into Maven with the following settings:


<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-parameter-names</artifactId>
</dependency>

Once I did that everything started working. I appreciate how quickly the Jackson team responded to my ticket to help me out.

Bug in Jackson JSON serialization of Immutable objects

The bug

I have been using Jackson for JSON processing for many years now. It is a great framework and mostly just works. It is the default framework in Spring Boot and mostly it just sits behind the scenes and drives most micro services in Java these days. That is why I was surprised when we hit a bug in the framework as I hadn’t really had any issues in the past.

The backstory

Anyone who has coded with me knows that I tend to favor immutability for most java objects. By default I make everything final and then only has non final things when they are necessary. This principle is documented in Effective Java Second Edition Item #15. As a side note everyone should read that book. Even senior developers benefit greatly flipping through the book once a year and reviewing the items. It may be the best Java book I have ever read for writing very good code. My hope is that Josh Bloch does a 3rd edition to cover all the new functional operations in the language.

The best way that I have found to create Immutable Data Transfer Objects in Java with Jackson is to have all the fields be final values, and to annotate the constructor with @JsonCreator and each property in the constructor with @JsonProperty("propertyName"). Just like that you have an object that won’t change state and works seamlessly between your Java Service and your JavaScript front end or between 2 Java back end services.

The problem

The problem we found with this approach occurred in an integration with another partner. Our developer was trying to use Java best practices so the variables are camel case and he is using @JsonCreator in the constructor and naming the properties there. While that works great for an inbound object when trying to serialize that object out to JSON Jackson is throwing away those property names and serializing off of the field name. Normally our field name matches our property name so this is no issue. But in this case the property on the other side was something like first_name, but we record the variable as firstName. Jackson is serializing that as firstName if you only use the Constructor approach. If you add @JsonProperty("first_name") to the field it will work. So you are basically left either double annotating the field or going with a mutable POJO object to get the correct output. I have created a project on github demonstrating the problem:

https://github.com/haskovec/jackson-serialization-bug

Additionally I have opened an issue on the Jackson project for anyone who is interested in following this issue. So far I haven’t heard anything back on it.

Closing thoughts

So the real question is how did this thing go through without getting caught. My suspicion is that most developers don’t actually create Immutable objects in general. The other possibility is that people don’t often have an output that doesn’t match the variable names. We could take the ugly approach and declare final String first_name; to fix it but that doesn’t make me happy either.

I will be interested in seeing if anyone responds to the issue and what they say. I think it is always a good test of a project to see how active it is by seeing how long it takes for the developers of the project to at least comment on a given issue found in their code. I did a little bit of debugging into the Jackson code and I can see where it is throwing away the correct property values, but I am not sure what the correct fix is. It feels like a method that if I tried to fix I could easily break some other use case.

Update

Turns out it wasn’t a bug and I got great help on the team to fix the issue.

Kegs & code

I attended the kegs & code last night. It was a code challenge and party with cash prizes that was hosted by Saltt Ventures. I had never attended a code challenge or hack-a-thon or anything like that, but I figured it is good to get out of your comfort zone every now and then and try something new. Plus when they have free beer that is a pretty big perk. The beer was supplied by BrainDead Brewery which I hadn’t had prior to this event. The event started out as a happy hour with pizza and beer and then at some point we setup and the challenge began. It was a race to solve 10 problems in the quickest amount of time with first place getting $500, second $250 and third $100.

The scoreboard
The scoreboard

I setup my laptop and opened up a new IntelliJ project to work in. Looking around I saw a whole mix of people doing different things. The one contestant that I knew Rocco was using a macbook and I suspect programming in Ruby since he is a Rails programmer. I saw someone programming on a little netbook which is funny to see as he then pulled out a huge keyboard and mouse, so his keyboard was much larger than his machine. I saw some people with Python on their screen as well. So it is pretty much pick whatever tools you are comfortable in to solve the problems. The rules that were given to us were that if you submit 10 wrong answers you are disqualified so up front you already have to be somewhat conservative and double check the questions.

The start screen
Waiting for the challenge to begin…

The contest began and the first question displayed a sha1 hash that was a result from the word kegs with a 4 digit hexadecimal value appended to it. I quickly went out and downloaded Apache Commons Codec so I could use its DigestUtils.sha1Hex class, to spit out a hex string after you run sha1 on it. I executed my code and quickly came up with a result of c0de.  The original hash presented was bdcb2013c555b6ff440a0b51ddf9c49284f48d9e

Next we were presented a json map that had integer values as a key and text as a value. You were instructed to decode it. Basically if you appended all the keys in the right order you ended up with a sentence that was the answer. Here is the original json: [[7,\" t\"],[33,\"n.\"],[14,\" B\"],[30,\"du\"],[28,\" p\"],[0,\"I \"],[23,\"te\"],[25,\" i\"],[24,\"st\"],[2,\"n'\"],[17,\"he\"],[10,\"my\"],[1,\"do\"],[26,\"t \"],[29,\"ro\"],[32,\"io\"],[19,\"I \"],[18,\"n \"],[11,\" c\"],[12,\"od\"],[15,\"ut\"],[16,\" w\"],[13,\"e.\"],[8,\"es\"],[22,\"I \"],[21,\", \"],[31,\"ct\"],[4,\"al\"],[27,\"in\"],[9,\"t \"],[6,\"ys\"],[3,\"t \"],[5,\"wa\"],[20,\"do\"]]

With this problem I considered using gson or Jackson. I haven’t actually written any gson code in over 3 years and I am used to using Jackson in the context of Spring where everything is setup for you.  I quickly determined that it would take me longer to set those libs up to do the problem than to just solve it by looking at it, so I did that to move on. But I chalk that up to a lesson learned for next time on preparation for this type of event.

The next challenge we were presented with the following string: gv qrxvy V qan yevt n qrffvx V We were asked to rot13 the string and then reverse the string and replace the world girl with squirrel and then upper case that string as our answer. I googled up a rot13 algorithm that I found on stack overflow and used.

The next challenge question was a block of text:

"Ecstatic advanced and procured civility not absolute put continue. Overcame breeding or my concerns removing desirous so absolute. My melancholy unpleasing imprudence considered in advantages so impression. Almost unable put piqued talked likely houses her met. Met any nor may through resolve entered. An mr cause tried oh do shade happy.\n" +
"\n" +
"She travelling acceptance men unpleasant her especially entreaties law. Law forth but end any arise chief arose. Old her say learn these large. Joy fond many ham high seen this. Few preferred continual sir led incommode neglected. Discovered too old insensible collecting unpleasant but invitation.\n" +
"\n" +
"Savings her pleased are several started females met. Short her not among being any. Thing of judge fruit charm views do. Miles mr an forty along as he. She education get middleton day agreement performed preserved unwilling. Do however as pleased offence outward beloved by present. By outward neither he so covered amiable greater. Juvenile proposal betrayed he an informed weddings followed. Precaution day see imprudence sympathize principles. At full leaf give quit to in they up.\n" +
"\n" +
"Living valley had silent eat merits esteem bed. In last an or went wise as left. Visited civilly am demesne so colonel he calling. So unreserved do interested increasing sentiments. Vanity day giving points within six not law. Few impression difficulty his use has comparison decisively.\n" +
"\n" +
"At every tiled on ye defer do. No attention suspected oh difficult. Fond his say old meet cold find come whom. The sir park sake bred. Wonder matter now can estate esteem assure fat roused. Am performed on existence as discourse is. Pleasure friendly at marriage blessing or.\n" +
"\n" +
"Examine she brother prudent add day ham. Far stairs now coming bed oppose hunted become his. You zealously departure had procuring suspicion. Books whose front would purse if be do decay. Quitting you way formerly disposed perceive ladyship are. Common turned boy direct and yet.\n" +
"\n" +
"Way nor furnished sir procuring therefore but. Warmth far manner myself active are cannot called. Set her half end girl rich met. Me allowance departure an curiosity ye. In no talking address excited it conduct. Husbands debating replying overcame blessing he it me to domestic.\n" +
"\n" +
"Consider now provided laughter boy landlord dashwood. Often voice and the spoke. No shewing fertile village equally prepare up females as an. That do an case an what plan hour of paid. Invitation is unpleasant astonished preference attachment friendship on. Did sentiments increasing particular nay. Mr he recurred received prospect in. Wishing cheered parlors adapted am at amongst matters.\n" +
"\n" +
"Assure polite his really and others figure though. Day age advantages end sufficient eat expression travelling. Of on am father by agreed supply rather either. Own handsome delicate its property mistress her end appetite. Mean are sons too sold nor said. Son share three men power boy you. Now merits wonder effect garret own.";

We were instructed to count the number of words in the block of text that only used letters from the first half of the alphabet. For this problem I found myself wishing I was better with Java 8 Lambas and Streams as this seemed like a perfect problem to stream through the test and then do reductions until we are left with what we want to count. However since I am still a beginner on the Java 8 functional style I slammed out the traditional imperative style. I split the block of text into words. I iterated the array of words and converted them to lower case and then iterated through each letter. I took advantage of the fact that a char in Java is a 16 bit unsigned integer. I found the ascii value for the letter m which is 109 (for lowercase) and then said if the value of the char was > 109 then we break out of the loop and don’t count the word otherwise we increment the count. That worked great and it was on to the next challenge.

I had actually forgotten about this question until just now when I am reviewing my code from the event. This question was given a list of numbers convert them all to binary and count how many one’s there are in those binary numbers. Then take that count and convert that number to binary and that is the answer.  The numbers we had to convert were as follows: 533, 239, 1144, 51, 217, 247, 312, 0, 941, 1060, 805, 158, 1020, 1038, 748, 834, 36, 460, 688, 276, 2, 112, 91, 297, 954, 245, 885, 37, 1120, 803, 551, 257, 654, 1101, 948, 1050, 524, 607, 336, 883, 17, 524, 66, 399, 343, 368, 389, 741, 841, 196, 303, 20, 190, 828, 940, 825, 941, 258, 385, 14, 650, 427, 118, 44, 152, 709, 112, 554, 183, 22, 711, 772, 865, 986, 399, 957, 358, 759, 306, 94, 577, 980, 396, 451, 441, 829, 145, 62, 566, 138, 899, 1010, 383, 449, 745, 560, 586, 489, 117, 56, 508 For this problem I just slammed all the numbers into an array iterated the array and appended the binary value to a StringBuilder using Integer.toBinaryString(). Then I went character by character through the string builder and counted the ones.

The next challenge was to count the number of numbers between 0 and 10^8 that divide evenly by 13 and also if you reverse the number the reversed number also divided by 13 evenly. So for example if 1234 divided evenly by 13 check if 4321 also divided evenly by 13, and if so count it. This was a pretty easy problem to solve basically iterate in a loop and do a couple of modulus operations.

Then I hit the next question. It was a database table of 20 ids and values. You were presented a db query that was something like SELECT SUM(VALUE) FROM TABLE WHERE value / key > 20 AND value / 2 NOT EQUAL 0. I quickly wrote up some java to do the algorithm and slammed the values into a map and spun there and did the result. I got the result of 1620 and I entered it and was told it was incorrect. I double and triple checked both my values and my algorithm and they looked correct. So I loaded all the data into a SQLServer database that I have on my laptop. I again got the same result 1620 which they said was incorrect. At that point I was basically blocked as I had printed out all the values at each step in the java program and could verify it was doing the right thing. After banging my head on this problem for probably 15 minutes then all the money prizes were taken. They offered to give me the value if I wanted to complete the rest of the challenges, but my thought was if I was out of the money I would rather grab a beer and chat with some people, so that is what I did. I ended up completing 60% of the challenge and had a lot of fun messing around with it.

I was discussing the problems with Rocco afterwards (whom placed 2nd). He said there was also a question about pulling something out of an Amazon S3 which I haven’t done, so that would have taken a bit of research but was probably something I could quickly google. The winner of the challenge interestingly enough actually got more than 10 wrong answers. By the rules we were given at the start of the contest he should have been disqualified but they changed the rules in the middle and said they were going to allow 25 incorrect answers. Rocco said the winner beat him by about 5 minutes.

Would I do it again? For sure, however I think before the next one I would do a little prep. First I would have a new IntelliJ project setup and just preload common libraries you are probably going to need to solve most problems. So next time I will have a bunch of things like Apache Commons downloaded and ready to go and some JSON library as well that I will have practiced with. A question worth asking is would I have an advantage programming in a more dynamic language than Java? I think if you have the advantage of a great IDE like Idea and you had a preloaded project with most common libs you are going to need for anything like this type of challenge I don’t feel you are at a disadvantage using Java. One of the biggest strengths of Java is just how rich the libraries are that you can solve most of these things with very few lines of code. Maybe if I had access to the Java 9 REPL shell it could be faster, but coding inside of my IDE is very fast. I saw some people brought external keyboards and mice. I type fast enough on my laptop I can’t see bringing an external keyboard, but I will definitely bring an external mouse next time I could speed things up with that.