A spike on `robust` JSON handling in .NET

When working with JSON formatted data in .NET I have always found it frustrating that I am losing something valuable if I want to use some form of contract to help reason about the code and improve understanding about the entities being worked upon.

What did I just receive?

The first issue I often come across is that after I have deserialized the data I simply can't tell the difference between null and undefined (or missing/absent) e.g. if we had the following entity

public class User
{
  string Name { get; set; }
  int? Age { get; set; }
}

then if we deserialized either of the following JSON objects into it

{
  "Name" : "Arthur Dent",
  "Age" : null
}

or,

{
  "Name" : "Arthur Dent"
}

then in both cases, after deserialization, Age is null. Should we wish to serialize that entity we can either serialize Age as null or we can omit it, but, we can't do both i.e. we can never be 100% sure we are creating the same representation that was received and we could be adding or losing information that may have a detrimental effect on a downstream system that consumes the data.

I don't know about you yet!

Another issue I often encounter is if the JSON object received has more information than expected e.g.

{
  "Name" : "Arthur Dent",
  "Age" : 42,
  "Address" : "ZZ9 Plural Z Alpha"
}

If we deserialize and serialize again, then we will now lose the Address field and this is something we probably want to avoid.

In all these cases, there are ways to handle them but I find them tedious to implement and it probably leaves an ungodly mess behind for someone else to pick up.

Coupling of services

In the last example, where an unknown field is potentially lost, this is often solved by ensuring all services have the same contract. When you add/remove fields from a contract we often use a shared package or file and this leads to laborious upgrading of each service or client that uses that contact and then we have to deal with the subsequent deployment and synchronization issues that then ensue. This just feels wrong as we should be developing (micro)services that can be independently deployed but now we have created a dependency between these services and have effectively created a distributed monolith. This problem is probably better explained in this Channel 9 talk "Data/Contract Coupling in Messaging".

Robustness Principle

We have better things to do with our time. What I needed was a way to take a typed contract e.g. using an interface that describes the entities I want to work with, that I can deserialize JSON objects into but be tolerant of what it reads whilst preserving the original JSON structure unless I have actually modified it; this sounds similar to the robustness principle.

The Robustness Principle is also known as Postel's Law and simply states:

Be conservative in what you do, be liberal in what you accept from others

In the world of services (and now micro-services I suppose) this is sometimes referred to as the Tolerant Reader where we need to be tolerant of what we read so that our contracts can simply evolve without holding us back.

Introducing Hale.NET

Now there was no way I am going to be able to create a better Json.NET library for this purpose but I can use it to do the heavy lifting when handling JSON. I also feel I should try some AOP techniques such as interceptors (e.g. Castle Windsor Dynamic Proxy) or weaving (e.g. Fody) to remove some of the repetitiveness.

The spike

Since my spike is showing potential and that other people have expressed an interest I thought it would be better to share now rather than just tinker myself when time permits. The spike (definitely not production ready) can be found on Github and I have released it under the MIT licence. It currently only handles the most basic of object structures (no arrays or hierarchies) but it does

1. preserve the underlying data by using a JObject and an interceptor to handle the get_ and set_ of our properties.
2. has some basic handling for dealing with null vs undefined using exceptions and some extension methods.

Assert.Null(user.GetValueOrDefault(u => u.Age));
Assert.False(user.IsReferenced(u => u.Age));

Next steps

The next step is to handle hierarchies of objects and arrays because without the ability to do this, it is not going to be of any use whatsoever.