App Updates: eXide 2.0.9

The 2.2RC2 release of eXist also includes updates to most of the core apps and one new app ("monex"). We'll cover the updates and additions in a series of articles. eXide will be first: version 2.0.9 has a number of new features which may not be obvious and need some explanation:

Live Reload

Enabling live reload

When developing a web application, one usually runs it in a separate tab or window, so after making a change to e.g. an XQuery, you need to switch to the other tab and press reload to get an updated web view.

The "live reload" feature automates this: whenever one saves a resource which belongs to an application package, the tab or window running the app's web view is refreshed automatically.

Enabling live reload

Due to security restrictions of the browser, a refresh is only possible if the app's browser window was initially created from within eXide. To enable live reload you thus have to:

  1. in the editor, switch to or open a resource which is part of the application package you want to run
  2. in the "Application" menu, select "Live Reload"
  3. a dialog will appear if the application has not been run before from within the same eXide instance. Click on the application link within the dialog to open a new tab with the application's web view.
  4. switch back to eXide

The next time you save a resource, the application view should refresh automatically. Depending on your browser and configuration, the application window or tab may either get focus immediately or it is refreshed in the background and you have to switch there to see the change.

Once live reload has been initialized, you can always toggle the feature on/off via the checkbox in the menu without having to open a new application window.

Personally I mostly use live reload in a setup with two screens, having eXide on one screen and the application on the other. It also works great on a smaller laptop screen with multiple open tabs though.

Directory Uploads

Upload dialog

Users often critisized in the past that uploading nested directories is not supported by the web-based collection browsers built into dashboard and eXide. If you had to preserve the directory structure, your main options were to either switch to the Java admin client or write a script (e.g. XQuery or Ant) which creates the target collection hierarchy.

The good news: eXide 2.0.9 finally supports directory uploads and preserves nested directories by creating a corresponding hierarchy of collections. The bad news: the feature is only available in Google Chrome. It is hard to say if other browsers will follow its lead or provide alternative APIs. Right now you definitely have to use Chrome though.

To upload a directory, proceed as usual:

  1. open the collection manager via the "File/Manage" menu
  2. browse to the collection into which you want to upload and press the upload icon in the toolbar
  3. on Chrome, a separate button, "Upload Directory" will appear. This is hidden on other browsers. Click on it and select a directory to upload.
  4. alternatively, you may also drag and drop a directory into the drop area marked with "Drop files here ...", e.g. from a finder window on a Mac.

Drag and Drop of Files into the Editor

When developing an application, one often needs to use an existing file on the file system as a template to create a new resource in the database. Using eXide, simply drag and drop the file into the editor. eXide will create a new, unnamed resource and insert the contents of the dropped file into it.

Other changes

Apart from many bug fixes, other notable changes include:

  • eXide now remembers the edit position for all open tabs and restores it when the browser window is reopened
  • drop the size limits for the validation of XML and XQuery files: previously only files with up to approx. 100k were sent to the server for validation. In particular, if you editied larger XML files, no syntax or validation errors were shown. Those limits have been dropped. Please do not forget that eXide is web-based and not the right tool to edit really large XML documents, but for the average document, the validation features work very well.
  • improved support for "less" stylesheets: when saving a .less file, eXide calls the less pre-processor to generate a corresponding .css file in the same collection. This feature was quite buggy in older releases. In particular, relative paths and imports are now handled properly.

eXist-db 2.2RC2

The second release candidate for eXist 2.2 is available for download.

RC2 contains numerous bug fixes, enhancements and performance improvements. The new range index - introduced with RC1 - has seen a lot of real-world testing and bug fixing to make sure it is properly used by queries, so they benefit from the superior performance of this new index.

Beyond the new range index, a number of other performance bottlenecks have been identified. Each of those can have a huge effect on particular queries: for example, the removal of unnecessary type checks on index-assisted functions and redundant cleanup cycles in the query engine increased performance in some cases by factor 10 or more. NGram indexes have also seen a huge speed up with respect to indexing as well as query times. Finally, the query optimizer missed to descend into some types of expressions, causing them to run without optimization.

Using the NGram index to search Sanskrit texts

Noteworthy other changes include, but are not limited to:

  • Fix: a caching issue led to random index failures for data sets large enough to fill the entire caching space.
  • Fix: faster addition or insertion of nodes using update extensions. In particular, the time to append a node should not increase with the size of the document anymore.
  • Fix: removing a collection containing a large number of documents or subcollections resulted in a huge recovery log being written, which does not only consume disk space, but may eventually also lead to recovery failures.
  • Enhancement: faster consistency checks and thus backups.
  • Enhancement: allow queries to be terminated by user: now effectively stops all types of expressions that have blocked db shutdown before.
Query profiling with Monex

Nearly all apps shipping with eXist have been enhanced as well. There were so many changes and additions that we will need to cover them in a separate article during the next days, so please stand by.

The final release of eXist 2.2 is scheduled for the first half of August.

eXist 2.2.RC1

We are very proud to announce the first release candidate for the next version of eXist, 2.2. The release candidate is feature complete, but not yet recommended for production use. Please let us know of any issues that you encounter so that we may resolve any unexpected bugs and finalise the release within the next month or so.

eXist 2.2 provides a new Range Index, which can accelerate your XQuery code and has proven to be upto 100 times faster than previous versions. This is the fastest release of eXist ever!

You can download the release from sourceforge.

eXist 2.2.RC1

New Range Index

2.2 features a reimplementation of the range index, the most important user-configurable index in eXist. As reported by users, some types of queries can run up to 100 times faster. The most dramatic performance increases have been observed on large data sets (with millions of documents) and queries on frequent strings. However, the new Lucene based index also brings many benefits for those working with smaller data sets.

While updates on the previous index system did not scale well with increasing collection sizes, the new index removes those limitations, thus allowing queries and updates to scale up. As previously reported by some users, problems with slow updates and increased memory usage have disappeared since switching to the new index.

Please refer to the documentation for more information.

Improved Crash Recovery

The new version also features a largely rewritten and simplified crash recovery, leading to a more robust recovery procedure and smaller transaction logs.


eXist 2.2.RC1 further extends it's Unix permission model, and now includes setUid and setGid bits. This both allows stored XQuerys to escalate permissions and enables the controlled sharing of Collection documents with groups os users. This is extremely important as it now makes it possible to call a query as an unpriviledged user and have it switch to a different effective user without providing a target for attacks.

Bug Fixes

2.2.RC1 includes numerous bug fixes, some of the highlights are:

  • Crash Recovery - Exceptions during transaction rollback no longer cause the database recovery to be aborted; Previously this was commonly seen as page not initialized errors.
  • Java Service Wrapper - No longer kills eXist if it takes longer than expected to shut down or start up.
  • Concurrency - Removed consequtive query invocation lockups, and many other small fixes.
  • Memory - Memory Leaks in the full text index were fixed.
  • Optimizer - Now descends into XQuery Update expressions.
  • Java Admin Client - It is no longer possible to accidentally lock out the admin user by mis-changing her password.

XML Prague eXist Preconference

We are happy to announce the annual eXist-db users group meeting at the 'official' pre-conference day of the XML Prague 2014 conference!

The users meetup became a tradition during the past years. It is the best opportunity to meet the eXist-db developers and the eXist-db community.

For more information, head over to our preconference page.

Redesigned Range Index How To

While the structural and full text indexes in eXist-db have seen redesigns during the past two years, range indexes were still largely unchanged. They increasingly became a bottleneck and limited scalability, at least for applications requiring frequent updates. This has changed: the current development version of eXist includes a rewritten, modularized range index. Under the hood it is based on Apache Lucene for super fast lookups. It also provides new optimizations to speed up some types of queries which failed to run efficiently with the old index.

Range indexes are extremely important in eXist-db. Without a proper index, evaluating a general comparison in a filter (like //foo[baz = "xyz"]) requires eXist to do a full scan over the context node set, checking the value of every node against the argument. This is not only slow, it also limits concurrency due to necessary locking and consumes memory for loading each of the nodes. With a well-defined index, queries will usually complete in a few milliseconds instead of taking seconds. The index allows the optimizer to rewrite the expression and process the index lookup in advance, assuming that the number of baz elements with content "xyz" is much smaller than the total number of elements.

The old range indexing code had three main issues though:

  1. Index entries were organized by collection, resulting in an unfortunate dependency between collection size and update speed. In simple words: updating or removing documents became slower as the collection grew. For a long time, the general recommendation was to split large document sets into multiple, smaller sub-collections if update speed was an issue.
  2. Queries on very frequent search strings were quite inefficient: for example, a query //term[@type = "main"][. = "xyz"] could be quite slow despite an index being defined if @type="main" occurred very often. Unfortunately this is a common use of attributes and to make it quick, you had to reformulate the query, e.g. by moving the non-selective step to the back: //term[. = "xyz"][@type = "main"].
  3. Range indexes were baked into the core of eXist-db, making maintenance and bug fixing difficult.

The rewritten range index addresses both issues. First, indexes are now organized by document/node, so collection size does no longer matter when updating an index entry. Concerning storage, the index is entirely based on Apache Lucene instead of the B+-tree which was previously used. Most range indexes tend to be strings, so why not leave the indexing to a technology like Lucene, which is known to scale well and does a highly efficient job on string processing? Since version 4, Lucene has added support for storing numeric data types and binary data into the index, so it seemed to be a perfect match for our requirements.  Lucene is integrated into eXist on a rather low level with direct access to the indexes.

To address the second issue, it is now possible to combine several fields to index into one index definition, so above XPath: //term[@type = "main"] [. = "xyz"] can be evaluated with a single index lookup. We'll see in a minute how to define such an index.

Finally, the new range index is implemented as a pluggable module: a separate component which is not required for the core of eXist-db to work properly. For eXist, the index is a black box: it does not need to know what the index does. If the index is there, it will automatically plug itself into the indexing pipeline as well as the query engine. If it is not, eXist will fall back to default (brute force) query processing.

Index Configuration

We tried to keep the basic index configuration as much backwards compatible as possible. The old range index is still supported to allow existing applications to run unchanged.

To switch the following index definition to the new range index, we simply wrap the create elements into a range element. Here's the old definition:

<collection xmlns=""> <!--from Tamboti--> <index xmlns:mods=""> <fulltext default="none" attributes="no"/> <lucene> <text qname="mods:title"/> </lucene> <!-- Range indexes --> <create qname="mods:namePart" type="xs:string"/> <create qname="mods:dateIssued" type="xs:string"/> <create qname="@ID" type="xs:string"/> </index> </collection>

To use the new range index, wrap the range index definitions into a range element:

<collection xmlns=""> <!--from Tamboti--> <index xmlns:mods=""> <fulltext default="none" attributes="no"/> <lucene> <text qname="mods:title"/> </lucene> <!-- Range indexes --> <range> <create qname="mods:namePart" type="xs:string" case="no"/> <create qname="mods:dateIssued" type="xs:string"/> <create qname="@ID" type="xs:string"/> </range> </index> </collection>

If you store this definition and do a reindex, you should find new index files in the webapp/WEB-INF/data/range directory (or wherever you configured your data directory to be).

Just as the old range index, the new indexes will be used automatically for general or value comparisons as well as string functions like fn:contains, fn:starts-with, fn:ends-with (fn:matches is currently not supported due to limitations in Lucene's regular expression handling). 

Above configuration applies to documents using MODS, a standard for bibliographical metadata. To provide some examples, the following XPath expressions should use the created indexes:

declare namespace mods=""; //mods:mods[mods:name/mods:namePart = "Dennis Ritchie"], //mods:mods[mods:originInfo/mods:dateIssued = "1978"], //mods:mods[mods:name/mods:namePart = "Dennis Ritchie"][mods:originInfo/mods:dateIssued = "1978"]

New Configuration Features

Case insensitive index

Add case="no" to create a case insensitive index on a string. This is a feature many users have asked for. With a case insensitive index on mods:namePart a match will also be found if you query for "dennis ritchie" instead of "Dennis Ritchie".


A collation changes how strings are compared. For example, you can change the strength property of the collation to ignore diacritics, accents or case. So to compare strings ignoring accents or case, you can define an index as follows:

<create qname="mods:namePart" type="xs:string" collation="?lang=en-US&amp;strength=primary"/>

Please refer to the ICU documentation (which is used by eXist) for more information on collations, strength etc.

Combining indexes

If you know you will often use a certain combination of filters, you can combine the corresponding indexes into one to further reduce query times. For example, the mods:name element has an attribute type which qualifies the name as being "personal", "corporate" or another predefined value. To speed up a query like //mods:mods[mods:name[@type = "personal"] [mods:namePart = "Dennis Ritchie"] you could create a combined index on mods:name as follows:

<range> <create qname="mods:name"> <field name="name-type" match="@type" type="xs:string"/> <field name="name-part" match="mods:namePart" type="xs:string"/> </create> </range>

This index will be used whenever the context of the filter expression is a mods:name and it filters on either or both: @type and mods:namePart. Advantage: only one index lookup is required to evaluate such an expression, resulting in a huge performance boost, in particular if the combination of filters does only match a few names out of a large set!

Note that all 3 attributes of the field element are required. The name you give to the field can be arbitrary, but it should be unique within the index configuration document. The match attribute specifies the nodes to include in the field. It should be a simple path relative to the context element. 

You can skip the match attribute if you want to index the content of the context node itself. In this case, an additional attribute: nested="yes|no" can be added to tell the indexer to skip the content of nested nodes to only index direct text children of the context node.

The index is also used if you only query one of the defined fields, e.g.: //mods:mods[mods:name[mods:namePart = "Dennis Ritchie"]]. It is important that the filter expression matches the index definition though, so the following will not be sped up by the index: //mods:mods[mods:name/mods:namePart = "Dennis Ritchie"] because the context of the filter expression here is mods:mods, notmods:name.

You can create as many combined indexes as you like, even if some of them refer to elements which are nested inside other elements having a different index. For example, to index a complete MODS record, we could create one nested index on the root element: mods:mods, and include all attributes or simple descendant elements we may want to query at the same time. mods:name - even though a child of mods:mods - is a complex element, so we want it to have a separate index as shown above. We thus define both indexes:

<range> <create qname="mods:name"> <field name="name-type" match="@type" type="xs:string"/> <field name="name-part" match="mods:namePart" type="xs:string"/> </create> <create qname="mods:mods"> <field name="mods-dateIssued" match="mods:originInfo/mods:dateIssued" type="xs:string"/> <field name="mods-id" match="@ID" type="xs:string"/> <field name="mods-authority" match="@authority" type="xs:string"/> <field name="mods-lang" match="@lang" type="xs:string"/> </create> </range>

This allows a more complex query to be optimized: 

//mods:mods[mods:name[@type = "personal"][mods:namePart = "Dennis Ritchie"]] [mods:originInfo/mods:dateIssued = "1979"]

In this case, the mods:dateIssued lookup will be done first, which presumably returns more hits than the name lookup. For maximum performance it may thus still be faster to split the expression into two parts and do the name check first. Anyway, average performance should be much better compared to the old range index though.

The combined indexing feature was originally created to handle a type of not-so-nice XML which is hard to query efficiently. In the concrete use case, each document consisted of a larger number of parameter elements, each having nothing but a key and value, e.g.: <parameter name="key" value="value"/>. Queries usually looked like this: //parameter[@name="key"][@value="value"] and were pretty slow, even after applying some optimization tricks. Creating a combined index on @name/@value solved this issue.


The rewritten range index is available in the develop branch of eXist-db at github. It will be included in the next release once more people have reported successful adoption.