Saturday, January 9, 2016

Using C# 4.0 and dynamic to parse JSON


UPDATE: take a look at http://json.codeplex.com/ before you try and roll your own, it supports dynamic now...

I recently had to get a JSON feed from the web and traverse it. Like any good developer I started off with a Bing search and stumbled across a few posts.

The first by Nikhil Kothari looked interesting but his implementation did way too much, all I needed to do was read a JSON file, did I really need all that code.

The second post I saw by Alex Ghiondea again looked interesting but relied on a library from codeplex (JSON.NET, http://json.codeplex.com/) and again I thought this was a bit overkill for my project.

But Nikhil’s post did get me thinking that this might be a good place to use the new dynamic “type” in C# 4.0. With some further research I dug up that in .NET 3.5 we got a new class called the JavaScriptSerializer, MSDN describes it as:
The JavaScriptSerializer class is used internally by the asynchronous communication layer to serialize and deserialize the data that is passed between the browser and the Web server. You cannot access that instance of the serializer. However, this class exposes a public API. Therefore, you can use the class when you want to work with JavaScript Object Notation (JSON) in managed code.

To use the JavaScriptSerializer (from the System.Web.Extensions dll) you have to implement a JavaScriptConverter. I can do that. :) After poking around a bit with the JavaScriptSerializer I realized that it gives you a few things. A value, an array list of values, a dictionary of key value pairs, or an ArrayList containing an array of dictionary of key value pairs. What I want to do is if I see a dictionary I want my user to be able to use the key as a property on my dynamic, if I see an ArrayList I want the user to be able to iterate over the collection and see its members, finally if I see a value I want to just return it.
Lets start with a simple JSON file, that I borrowed from json.org:
{
    "glossary": {
        "title": "example glossary",
        "GlossDiv": {
            "title": "S",
            "GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
                    "SortAs": "SGML",
                    "GlossTerm": "Standard Generalized Markup Language",
                    "Acronym": "SGML",
                    "Abbrev": "ISO 8879:1986",
                    "GlossDef": {
                        "para": "A meta-markup language, used to create markup languages such as DocBook.",
                        "GlossSeeAlso": ["GML", "XML"]
                    },
                    "GlossSee": "markup"
                }
            }
        }
    }
}

So with this the above JSON I want to be able to write “glossaryEntry.glossary.title” or “glossaryEntry.glossary.GlossDiv.GlossList.GlossEntry.ID” and get back “example glossary” and “SGML” respectively. All without having to actually create a static “glossary” object.
So the first step is to create our DynamicObject. I called mine DynamicJsonObject and as you can see it inherits from the base DynamicObject class.

public class DynamicJsonObject : DynamicObject
{
    private IDictionary<string, object> Dictionary { get; set; }
 
    public DynamicJsonObject(IDictionary<string, object> dictionary)
    {
        this.Dictionary = dictionary;
    }
 
    public override bool TryGetMember(GetMemberBinder binder, out object result)
    {
        result = this.Dictionary[binder.Name];
 
        if (result is IDictionary<string, object>)
        {
            result = new DynamicJsonObject(result as IDictionary<string, object>);
        }
        else if (result is ArrayList && (result as ArrayList) is IDictionary<string, object>)
        {
            result = new List<DynamicJsonObject>((result as ArrayList).ToArray().Select(x => new DynamicJsonObject(x as IDictionary<string, object>)));
        }
        else if (result is ArrayList)
        {
            result = new List<object>((result as ArrayList).ToArray());
        }
 
        return this.Dictionary.ContainsKey(binder.Name);
    }
}
The tricky bit is in the “TryGetMember” Method. The binder.Name is the property value you used in your code. As you can see I am looking that guy up in the current internal dictionary. I then test its type. If it is a dictionary, I wrap it in another DynamicJsonObject and give it back. If it is an ArrayList of Dictionaries then I wrap each item as an DynamicJsonObject and send back the List. If it is just an Array List I convert it to a List and send it back. Finally we have to tell .NET if we found it at all.
The only other thing we need is an implementation of the JavaScriptConverter.
    public class DynamicJsonConverter : JavaScriptConverter
    {
        public override object Deserialize(IDictionary<string, object> dictionary, Type type, JavaScriptSerializer serializer)
        {
            if (dictionary == null)
                throw new ArgumentNullException("dictionary");
 
            if (type == typeof(object))
            {
                return new DynamicJsonObject(dictionary);
            }
 
            return null;
        }
 
        public override IDictionary<string, object> Serialize(object obj, JavaScriptSerializer serializer)
        {
            throw new NotImplementedException();
        }
 
        public override IEnumerable<Type> SupportedTypes
        {
            get { return new ReadOnlyCollection<Type>(new List<Type>(new Type[] { typeof(object) })); }
        }
    }
As you can see by the above code we are only supporting the deserialize method, and the only thing we do is wrap the dictionary with our DynamicJsonObject. Finally since we are using dynamic we can support any object type with this converter.
Now for the fun bit, lets use it.
            JavaScriptSerializer jss = new JavaScriptSerializer();
            jss.RegisterConverters(new JavaScriptConverter[] { new DynamicJsonConverter() });
 
            dynamic glossaryEntry = jss.Deserialize(json, typeof(object)) as dynamic;
 
            Console.WriteLine("glossaryEntry.glossary.title: " + glossaryEntry.glossary.title);
            Console.WriteLine("glossaryEntry.glossary.GlossDiv.title: " + glossaryEntry.glossary.GlossDiv.title);
            Console.WriteLine("glossaryEntry.glossary.GlossDiv.GlossList.GlossEntry.ID: " + glossaryEntry.glossary.GlossDiv.GlossList.GlossEntry.ID);
            Console.WriteLine("glossaryEntry.glossary.GlossDiv.GlossList.GlossEntry.GlossDef.para: " + glossaryEntry.glossary.GlossDiv.GlossList.GlossEntry.GlossDef.para);
            foreach (var also in glossaryEntry.glossary.GlossDiv.GlossList.GlossEntry.GlossDef.GlossSeeAlso)
            {
                Console.WriteLine("glossaryEntry.glossary.GlossDiv.GlossList.GlossEntry.GlossDef.GlossSeeAlso: " + also);
            }
To use our converter we create a serializer, register our converter with it and then call the deserialize method. Now we can use our standard “dot” notation to traverse our JSON object without the need for external libraries and in under 100 lines of code. Now JSON in .NET feels a lot like JSON in JavaScript. . . When I heard that C# was getting this dynamic stuff I was unsure of its power and relevance, now I have “seen the light” and can really see where this was a great addition to the language.

String to Json




var input = ["Fred-Jim-Bob", "Fred-Jim", "Fred-Thomas-Rob", "Fred"];
var output = [];
for (var i = 0; i < input.length; i++) {
    var chain = input[i].split("-");
    var currentNode = output;
    for (var j = 0; j < chain.length; j++) {
        var wantedNode = chain[j];
        var lastNode = currentNode;
        for (var k = 0; k < currentNode.length; k++) {
            if (currentNode[k].name == wantedNode) {
                currentNode = currentNode[k].children;
                break;
            }
        }
        // If we couldn't find an item in this list of children
        // that has the right name, create one:
        if (lastNode == currentNode) {
            var newNode = currentNode[k] = {name: wantedNode, children: []};
            currentNode = newNode.children;
        }
    }
}


output[{
    "name": "Fred",
    "children": [{
        "name": "Jim",
        "children": [{
            "name": "Bob",
            "children": []
        }]
    }, {
        "name": "Thomas",
        "children": [{
            "name": "Rob",
            "children": []
        }]
    }]
}]

Evaluating JavaScript code from C#


As was going through my JSON Serialization code the other day I kept re-thinking this idea I had some time ago: Why not use Microsoft’s built in JavaScript engine to perform the JavaScript and JSON deserialization. After all there is an official Microsoft JavaScript .NET language and it should be possible to take advantage of this from within an application. I finally had some time to check this out...

I started searching for how to do this and there’s not a heck of a lot of information available on this so I started playing around with this on my own. The good news is that it’s pretty straight forward to evaluate JavaScript code, the bad news is that for what I really need it for it doesn’t work quite right.

In order to use JScript you need to add a reference to the the Microsoft.vsa assembly.

<add assembly="Microsoft.Vsa, Version=8.0.0.0, Culture=neutral, PublicKeyToken=B03F5F7F11D50A3A"/></assemblies>

First here are a few eval scenarios and how you can call them from a bit of C# code in ASP.NET:

using Microsoft.JScript;

public partial class JavaScriptEval :  System.Web.UI.Page
{
    public static Microsoft.JScript.Vsa.VsaEngine Engine = Microsoft.JScript.Vsa.VsaEngine.CreateEngine();

    public static object EvalJScript(string JScript)
    {
        object Result = null;
        try
        {
            Result = Microsoft.JScript.Eval.JScriptEvaluate(JScript, Engine);
        }
        catch (Exception ex)
        {
            return ex.Message;
        }

        return Result;
    }
   
    public void Page_Load(object sender,EventArgs e)
    {
        // *** String Value
        object Result = EvalJScript(@"('hello world: ' + new Date())");
        Response.Write( Result.ToString() );
        Response.Write(" Type: " + Result.GetType().Name);
        Response.Write("<hr>");

        // *** Return an integer or numeric - no conversion required
        Result = EvalJScript(@"( 11 * 12)");
        Response.Write(Result.ToString());
        Response.Write(" Type: " + Result.GetType().Name);
        Response.Write("<hr>");
       
        // *** Date value - must be converted!
        Result = EvalJScript(@"(new Date())");
        Response.Write(Result);
        Response.Write(" - Type: " + Result.GetType().Name + "<br>");
       
        // *** Must convert from DateObject to DateTime by coercing
        DateObject dateObject = Result as DateObject;
              
        Result = Microsoft.JScript.Convert.Coerce(Result,typeof(DateTime));
        Response.Write("Converted to DateTime: "  + Result);
        Response.Write("<hr>");
                      
       
        // *** Block of code (last assignment is the return value)
        Result = EvalJScript(@"var out = 'hello';   for ( var x = 1; x < 10; x++) { out = out + 'Line ' + x  + '<br>'; }");
        Response.Write(Result);
        Response.Write(" - Type: " + Result.GetType().Name);
        Response.Write("<hr>");

        /// *** Closure - calling a JavaScript Function with return value
        Result = EvalJScript(@"( function Test(inputParm) {  return 'hello world ' + inputParm; } )");

        /// *** Invoke the function and retrieve the result
        Closure close = Result as Closure;

        // *** This requires full trust
        Result = close.Invoke(close, new object[1] { "Start with this bub..." });
        Response.Write(Result);
        Response.Write(" - Type: " + Result.GetType().Name);
        Response.Write("<hr>");

        // *** JSON style object
        Result = EvalJScript(@"( {""timeString"":'Time is: ' + new Date(),""dateValue"":new Date()} )");

        JSObject obj = Result as JSObject;
        Response.Write(Result);
       
        // *** JavaScript style property array access can be used
        object val = obj["dateValue"];
       
        Response.Write(Microsoft.JScript.Convert.Coerce(val,typeof(DateTime)));

        val = obj["timeString"];
       
        Response.Write(val);        
    }       
}

As you can see execution of simple bits of code is pretty straight forward. If you need to evaluate an expression that returns a simple type like string, bool or numeric you can simply eval the expression and you get the right value and type back.

If you eval something that results in a date however things get more tricky already. You basically need to coerce the JavaScript date into a .NET date. It’s fairly easy to do this luckily – and it might be a decent solution to the lousy kind of code you have to write to deal with all the different date formats available in JSON (new Date() syntax allows for string values, numeric values all in various format). This is somewhat useful.

More interesting is the ability to execute a whole block of code. There are a few ways to do this. You can simply pass the code directly and execute it which works just fine. The problem with this is that you can’t directly return a value. Instead the last assignment ends up as your return value. This is similar to the behavior you see in event handlers like onclick in Web pages where you sometimes get unexpected results if you don’t explicitly return a value. This is probably the best choice even though it’s a bit unconventional.

Result = EvalJScript(@"( function Test(inputParm) {  return 'hello world ' + inputParm; } )");

What happens here is that rather than returning you an evaluated result the parser returns you the closure – ie. essentially a function pointer. You can now use Reflection to execute the Result which is a little more involved:

Closure close = Result as Closure;
Result = close.Invoke(close, new object[1] { "Start with this bub..." });

This works fine, but it requires full Reflection rights in order to be able to access this anonymous function. The closure is treated like a private/internal/protected member by Reflection and so you need Full trust in ASP.NET to execute this code. Bummer – this would be the most natural way to execute code because you’d have the ability to pass in parameter and get a return value using standard .NET Reflection logic. But given the rights issues this is probably not an option. I can’t figure out any other way to pass a parameter into the parser.

Next, there’s the actual issue I was after: Being able to pass an arbitrary object (in JSON style format) in and get this object back in some way to that can be parsed in .NET.

Result = EvalJScript(@"( {""timeString"":'Time is: ' + new Date(),""dateValue"":new Date()} )");

This returns a JavaScript object which can then be parsed with typical JavaScript member arrays syntax:

Result = EvalJScript(@"( {""timeString"":'Time is: ' + new Date(),""dateValue"":new Date()} )");

JSObject obj = Result as JSObject;
foreach ( string FieldName in obj)
{
   object FieldVal = obj[FieldName];
   Response.Write( FieldVal.ToString() + " - "  + FieldVal.GetType().ToString() + "<br>");
}

This works and you can potentially parse out the different types , but that still leaves the issue of how to get an actual object from the JavaScript object or more specifically how to ‘cast’ this object to another object.

I’m just thinking out loud here <s>:

Say you have a JSON method you call and you have a method which takes an object as a strongly typed object parameter:

Public bool SaveCustomer(CustomerEntity Customer)

I suppose one could use the Customer object as the ‘schema’ and walk through that object with Reflection and then pull values from the JavaScript with the code above and then assign them.

I took a closer look at what MS Ajax is doing – and well they’re manually parsing the javascript which is fine, but MS AJAX doesn’t work with a lot of types and doesn’t work of generic type value returned at all – any object structure must exist on the client before it can be passed back (that’s what that long Proxy block for a class definition does basically – it creates type definitions for any parameters and return values).


I have my own JSON parsing code and it works reasonably well against a wide variety of types generically using a mechanism similar to the above but by parsing string values. But the code is pretty messy and it can easily be broken by some funky JSON formatting. It would be much nicer to leave this sort of thing to a  JavaScript parser.

OTOH, manually parsing is probably a safer solution. Using the JSCript parser as shown above actually executes code on the server. If you take a JSON input and do that there’s a lot of potential for security issues.

Actually it’s surprising that the Jscript Eval works at all under a limited trust environment.

Anyway it’s an interesting experiment but for my needs probably a dead end. But these samples might help someone trying to add JavaScript support to a .NET app out or at least get them started.....


Rick,
I used a similar technique in ASP.Net 1.1. When I converted to 2.0 I noticed the VSA library was marked obsolete. So I searched for a way to eval basic javascript expressions. (I was looking for something that would simply return True or False base on whatever I passed in to it.)

We had a problem with the 'Microsoft.JScript.Vsa.VsaEngine' code in ValruleFilters because Microsoft deprecated it and made it obsolete.

They recommended you use the ICodeDomCompiler interface to build code on the fly instead. 
I took a look at it and it is very complex.

I came up with a much simpler solution instead.

I built a .dll file using javascript and then compiled it using the javascript compiler which is available in a VS2005 command prompt.

Once we have the .dll we simply add it to the \Support folder and then referenced it in the project which needed to eval javascript stateemnts.

We call it like this:

Dim jScriptEvaluator As New EvalClass
Dim objResult As Object
objResult = jScriptEvaluator.Evaluate(“1==1 && 2==2”)

This is *much* simpler and safer than ICodeDom.
======================================================================
Warning

'Microsoft.JScript.Vsa.VsaEngine' is obsolete: 
'Use of this type is not recommended because it is being deprecated in Visual Studio 2005; 
there will be no replacement for this feature. Please see the ICodeCompiler documentation for additional help.'

======================================================================

Detailed Steps to create a .dll:

1. Create a file in Notepad with only these contents:

class EvalClass { function Evaluate(expression: String) { return eval(expression); } } 

2. Save the file as C:\MyEval.js

3. Open a VS2005 Command Prompt (Start, Programs, VS2005, VS2005 Tools)

4. Type Cd\ to get to C:\

5. Type 
jsc /t:library C:\MyEval.js

6. A new file is created named MyEval.dll.

7. Copy MyEval.dll to the project and reference it (also reference Microsoft.Jscript.dll).

8. Then you should be able to call it like this:

Dim jScriptEvaluator As New EvalClass
Dim objResult As Object
objResult = jScriptEvaluator.Evaluate(“1==1 && 2==2”)

objResult is True.