XQuery 3.1 Arrays and JSON Support

The current development version of eXistdb includes full support for the array data type and related features from the XQuery 3.1 Candidate Recommendation. In combination with maps, arrays allow for a more "natural" representation of JSON in XQuery. Processing JSON or interfacing with external services returning JSON has become a lot more straightforward.

But even if you are only mildly interested in JSON, arrays are a welcome addition to the XQuery language, mainly because unlike sequences, arrays can be nested. I guess most XQuery programmers have encountered a situation in which it would have been nice to return a sequence of sequences from a function. And sometimes you may want to indicate that particular items in a result sequence are empty. With arrays you can do all that. Arrays may contain other arrays or maps, sequences or even the empty sequence as members.

Recorded presentation of this article during the eXistdb user pre-conference in Prague

Array Constructors

Array constructors come in two flavors: square and curly constructors. The square constructor will look familiar to most people:

let $array := [1, (), (3, 4)] return $array(3)

Within the square constructor, the "," is just a separator (like in a function call), so the resulting array will correspond to the comma-separated members. In the example above we're retrieving the third member using $array(3), which is the sequence containing numbers 3 and 4. Getting the second member with $array(2) will return the empty sequence accordingly.

The curly constructor behaves slightly different: it takes a sequence of items and creates an array member from each of them:

let $array := array { 1, (), (3, 4) } return $array(3)

Above query returns "4"! See the difference? The "," in this case is the XQuery sequence constructor, so the sequence from which the array is built is 1, 3, 4.

As already announced, you can arbitrarily mix sequences, arrays and maps, resulting e.g. in:

let $books := [ map { "title": "eXist", "author": [ [ "Adam", "Retter" ], ["Erik", "Siegel" ] ], "language": "English" }, map { "title": "XQuery", "author": [ [ " Priscilla", "Walmsley" ] ], "language": "English" } ]

Array Lookups

Just like a map, an array is also a function, which accepts a single integer parameter corresponding to the position of the member to retrieve (as always in XQuery, counting starts at 1). We have seen simple examples above. For nested data structures, just chain the function calls, e.g.:

$books(2)("title")

Alternatively, there's a lookup operator, which is often a bit easier to read. It works on arrays as well as maps (but not on other data types):

$books?2?title

The lookup may also appear inside a predicate. In this case, the left hand argument (the array or map) is often skipped and defaults to the context item:

$books?*[?title = "eXist"]?language

The operator excepts an integer, name, parenthesized expression or a wildcard as its right hand argument. So to use e.g. a variable for the lookup, wrap it into parens:

let $field := "title" return $books?2?($field)

Note: because $books is an array, the lookup argument must evaluate to a sequence of integers or you'll see an error. It is possible to look up more than one array item at a time, e.g.: $books?(1 to 2)?title.

The wildcard returns the keys or members of a map or array. When used on a map, it results in a sequence of keys, whereas on an array, you get a sequence of members:

["Hello", "world", "!"]?*

Use the wildcard as a quick way to iterate an array:

for $book in $books?* return $book?title

Function Library

The XQuery 3.1 functions spec also includes a huge library of functions to process and modify arrays. All functions use the prefix array.

array:sizereturns the size of the array
array:headreturns the first member
array:tailan array with all members except the first
array:subarraycreates an array containing a subset of members
array:reversereverse members
array:for-eachiterate over members
array:filterfilter the arrays with a function
array:fold-leftapply function to members and collect results from left to right
array:fold-rightapply function to members and collect results from right to left
array:for-each-pairiterate members pair-wise
array:appendappend a member to an array
array:insert-beforeinsert new member
array:removeremove member
array:joinConcatenates the contents of several arrays into a single array

As all data types in XQuery, arrays are immutable and cannot be modified. The functions above will thus always return a new array. eXist tries to implement this in an efficient way for functions like array:tail, array:append, array:subarray, array:remove without creating redundant copies.

Please note that I did not implement array:sort yet. It will be added later.

Many of the functions mirror other functions already available in the standard function library, but take an array instead of a sequence as input. For example, array:fold-left works like fn:fold-left.

declare function local:price($hoursPerTask as array(xs:integer), $rate as xs:double) as xs:double { array:fold-left($hoursPerTask, 0.0, function($sum, $hours) { $sum + $hours * $rate }) }; local:price([3, 8, 6, 5, 2], 96.0)

In this example we multiply the hours required for some task by our hourly rate and return the sum.

JSON Support

Obviously, representing JSON data within an XQuery has become straightforward using maps and arrays. The function fn:parse-json takes a string of JSON data and returns either a map (for a JSON object), an array, an atomic value (xs:string, xs:double for numbers or xs:boolean), or the empty sequence (corresponding to null in JSON):

let $json := '{"user": "bob", "id": 10, "valid": true}' let $user := parse-json($json) return $user?id

Note that by default parse-json is rather strict about the JSON syntax. For example, strings must use double quotes and duplicate keys generate an error. You can tell the function to be more relaxed about the JSON syntax by passing in a map of options:

let $json := "{'user': 'bob', 'id': 10, 'valid': true}" let $options := map { "liberal": true(), "duplicates": "use-last" } let $user := parse-json($json, $options) return $user?id

To see the function in action on a real-world example, assume we would like to retrieve a list of commits from a git repository, using the HTTP/JSON API provided by github:

xquery version "3.1"; import module namespace http="http://expath.org/ns/http-client"; declare function local:log($json as array(*)) { <table> { for $entry in $json?* let $commit := $entry?commit return <tr> <td>{$commit?committer?date}</td> <td>{$commit?committer?name}</td> <td>{$commit?message}</td> </tr> } </table> }; let $url := "https://api.github.com/repos/eXist-db/exist/commits?since=2015-01-01T00:00:00Z" let $request := <http:request method="GET" href="{$url}" timeout="30"/> let $response := http:send-request($request) return if ($response[1]/@status = "200") then local:log(parse-json(util:binary-to-string($response[2]))) else ()

Here we're using the httpclient module to talk to the github API, which gives us more control over the communication. But there's also a simpler approach, using the fn:json-docfunction:

let $url := "https://api.github.com/repos/eXist-db/exist/commits?since=2015-01-01T00:00:00Z" return local:log(json-doc($url))

fn:json-doc retrieves the contents of the given URI and parses them using fn:parse-json. It works with external resources as well as binary documents stored in eXist. To access stored resources, just use a local path, e.g. /db/test/data.json.

Serialization

eXistdb has supported serialization to JSON for several years, but the old serializer was based on mapping an XML query result to JSON, which caused some difficulties at times, e.g. if you had to produce an array for a certain property, even if it was empty. Contrary to this, the new JSON output method defined by the XQuery 3.1 Serialization spec is straightforward: it takes an array, map, atomic value or empty sequence and produces valid JSON.

The JSON serializer is selected if you set the serialization option method to json. This applies to both, the old and the new serializer. To distinguish between the two while preserving backwards compatibility, we use the following convention:

  • if the sequence to serialize is a single XML element node, the old serializer is used
  • if the sequence contains more than one item or the single item is not an XML element, it will be passed to the new serializer

This convention allows us to run all the old code unchanged without violating the 3.1 specification too much (according to the specs, a single XML element would be serialized to a JSON string).

To see the serializer in action, use the fn:serialize function:

xquery version "3.1"; declare namespace output="http://www.w3.org/2010/xslt-xquery-serialization"; let $array := map { "k1": array { "v1", "v2" }, "k2": "v3" } return serialize($array, <output:serialization-parameters> <output:method>json</output:method> </output:serialization-parameters>)

or save the query and define the serialization method as an output option:

xquery version "3.1"; declare namespace output="http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method "json"; declare option output:media-type "application/json"; map { "k1": array { "v1", "v2" }, "k2": "v3" }

Other Functions Using Arrays

Some applications require calling a function dynamically without knowing the number of arguments it takes in advance. Without arrays, this had been rather difficult to solve: because sequences cannot be nested, passing arguments containing more than one item has been tricky. For example, we solved this in the templating module by using function items. The code becomes rather bloated though.

The newly added fn:apply function makes this straightforward. It takes a function item as first argument and an array containing the parameters as second:

fn:apply(sum#1, [(1, 2, 3)])

Availability

The new features will be available in the eXistdb 2.3 release, but we encourage users to help us testing. We tried to preserve backwards compatibility with existing XQuery code, so most, if not all, apps should work as before.

To experiment with arrays and JSON, just build from source or use a nightly. You may look through the test cases for some inspiration.

Finally, I also recommend watching Dannes' presentation on Mongrel: the MongoDB extension driver for eXistdb, which will rely on the features described in this article.

eXist-db preconference day @ XML Prague 2015

Dear eXistentialist,

if it has escaped you there will be an eXist-db meetup @ XML Prague 2015 according to our tradition.The preconference day is Friday February 13th, see our eXist-db info at http://preconference.info/xmlprague2015/index.html.

We already have a handful confimed presenters wanting to share stuff with you, but the more the merrier and time flies, so if you wish to speak, present, ask or demo something please contact us at mailto:info@preconference.info so that I can put your contribution in the programme.

The preliminary programme is to be found on http://xmlprague2015.preconference.info/xmlprague2015/program.html

Since the preconference day is an official part of the XML Prague conference you need to register for the conference too at http://www.xmlprague.cz/conference-registration/ if you have not already done so.

Welcome!

Leif-Jöran

eXistdb Training in February

http://exist-db.org/exist/apps/homepage/resources/img/mybulb.svg

eXist Solutions is organizing an eXistdb training during the week from February 16 to 20 (the week following XML Prague). There are some spare places, so if you would like to attend, please send us a message before the end of next week (January 30).

The training will be held by Wolfgang Meier at our office not far from Frankfurt/Main airport. For details please refer to our training page.

Automatic form validation in eXistdb

Whenever you need forms in your applications you face the problem of validation. Of course there are a lot of JavaScript solutions out there but that still leaves you with the question how to quickly revalidate the data on the server.

As a first step of our redesign of betterFORM we've developed a facility called 'ModelValidator' that does exactly that but without the need to learn XForms first.

Usually when it comes to user input people start with a plain HTML form as it's the easiest and quickest way to get things running. But once that's done you're left with key/value pairs that you have to check for correct datatype and value range in your server-side scripts. It's certainly no fun task to repeatedly read and check single params before moving on to the interesting part of your application.

Now HTML5 has added some datatypes that have improved the situation but neither all browsers implement them all nor does this give you any safety that the values arriving at the server still adhere to those constraints. Further a simple 'novalidate' put onto the form tag by some unfriendly person before submit disables all checks in the browser and the data are send along as they are.

What is ModelValidator?

'ModelValidator' is a new facility becoming available in eXistdb very soon that improves this situation. It automatically generates a server-side XForms model that reflects the constraints you've put onto your HTML controls.

Once the user submits the form ModelValidator will intercept the request and generate an XForms model that uses the original HTML document as input and will setup all constraints expressed in the HTML for server-side revalidation. Only in case the incoming data conform to the constraints the submission is forwarded to the action url expressed in the HTML form.

In the context of eXistdb you likely want to send the data to some XQuery script. As its tedious to read all the params one by one ModelValidator does more - it will transform the key/value pairs into an XML structure that you can use as input to your XQuery. Now that you can be sure that your data have been validated before reaching your script you can directly work with the XML.

/exist/apps/wiki/blogs/eXist/modelvalidator.png

But ModelValidator can do even more if you are willing to dive into the XForms Model syntax - every time an XForms model is generated for a form it will be stored into the database. When a new request comes in it will check if a model has already been generated and will use it if that's the case. You can now edit the generated model to establish even more rigid constraints that HTML5 does not allow to express.

Last but not least the current prototype does not even need any JavaScript to work - just plain HTML.

What comes next?

The current prototype does not support all HTML5 datatypes yet and there's still some polishing and testing to do before we can roll it out. Further we have to work out a decent deployment into eXistdb. Hopefully we can publish it with the next version of eXistdb.

ModelValidator is just the first step in our redesign of betterFORM. To provide instant validation once the user has entered a value and to offer more fancy ways of alerting the next iteration will add JavaScript again to speak to the server under the hood. We have decided for modernizing the client-side technologies we're using. Instead of Dojo we'll go for Web Components (likely Polymer) and for the transport we'll exchange DWR (Direct Web Remoting) with Websockets.

What will happen to 'full' XForms support?

Though we decided to move away from full XForms support (see betterFORM Blog) in betterFORM 6 we made some effort to further support betterFORM 5 as the fully XForms 1.1 compliant alternative. Thus you will be able to use betterFORM 5 and 6 (once it's there) side-by-side in eXistdb. There won't be any change for those that are using betterFORM 5 in their applications. It will just work as before.

http://exist-db.org/exist/apps/homepage/resources/img/book-cover.jpg

The eXistdb book written by Adam Retter and Erik Siegel is now avaible in its final version as ebook or paperback. Go get it.

There's also a in-depth review of the book by Joe Wicentowski.