eXistdb 2.2 Released


eXide 2.1.2

We've been busy since the summer to test and improve everything, so here it is: eXistdb 2.2 final is out! We think this is a major step forward in many areas. To just highlight a few:

  • New range index: the most important user-configurable index in eXist has been entirely rewritten for scalability and performance, now using Apache Lucene:
    • many times faster on large data sets
    • xpath expressions using multiple filters are optimized into a single index lookup
    • updates and queries scale very well with growing collection size
  • App updates:
    • eXide 2.1.2 is more robust than ever and has many new features
    • monex: a new app for monitoring and profiling eXist-db
  • Extended the security model, now including setUid and setGid, to temporarily escalate permissions inside an application in a safe way
  • Improved crash recovery leading to a more robust recovery procedure and smaller transaction logs
  • Enhancements to the query engine and optimizer leading to improved overall performance, in particular for queries using indexes

For those upgrading from 2.2-RC1 or 2.2-RC2, the main changes in 2.2 final are:

  • improvements to backup/restore: sometimes a restore led to duplicate user accounts or invalid user IDs
  • many improvements to the new range index based on user feedback (thanks!)
  • map support in query engine aligned with XQuery 3.1 specification
  • closed memory leaks

LTS Edition

Along with the community release, the LTS edition has also been updated to 2.2. As with every major release, the LTS edition uses the same code base as the community edition, but all updates will be limited to carefully checked, production-ready, quality assured bug fixes and enhancements. With the LTS subscription, we make sure customers build their applications on a stable foundation and benefit from the latest improvements without the risk of using a development version.

To celebrate the 2.2 release, new customers get the LTS edition with a 50% discount during our autumn special. All money earned through the LTS subscription is directly funding eXistdb development. You thus benefit twice: by building your application on a quality assured version and contributing to the advancement of eXistdb.

Another great video tutorial by Loren Cahlander to help you set up basic authentication in your eXistdb app.

You can also check out his source code for the app on github.

Loren Cahlander created a nice introduction presentation on the HTML templating module, which has been part of eXistdb since 2.0. He demonstrates each templating annotation with a simple demo, so if you ever wondered how those generated templating instructions in your eXist app work together, watching the video is recommended:

App Updates: eXide 2.0.9

The 2.2RC2 release of eXist also includes updates to most of the core apps and one new app ("monex"). We'll cover the updates and additions in a series of articles. eXide will be first: version 2.0.9 has a number of new features which may not be obvious and need some explanation:

Live Reload

Enabling live reload

When developing a web application, one usually runs it in a separate tab or window, so after making a change to e.g. an XQuery, you need to switch to the other tab and press reload to get an updated web view.

The "live reload" feature automates this: whenever one saves a resource which belongs to an application package, the tab or window running the app's web view is refreshed automatically.

Enabling live reload

Due to security restrictions of the browser, a refresh is only possible if the app's browser window was initially created from within eXide. To enable live reload you thus have to:

  1. in the editor, switch to or open a resource which is part of the application package you want to run
  2. in the "Application" menu, select "Live Reload"
  3. a dialog will appear if the application has not been run before from within the same eXide instance. Click on the application link within the dialog to open a new tab with the application's web view.
  4. switch back to eXide

The next time you save a resource, the application view should refresh automatically. Depending on your browser and configuration, the application window or tab may either get focus immediately or it is refreshed in the background and you have to switch there to see the change.

Once live reload has been initialized, you can always toggle the feature on/off via the checkbox in the menu without having to open a new application window.

Personally I mostly use live reload in a setup with two screens, having eXide on one screen and the application on the other. It also works great on a smaller laptop screen with multiple open tabs though.

Directory Uploads

Upload dialog

Users often critisized in the past that uploading nested directories is not supported by the web-based collection browsers built into dashboard and eXide. If you had to preserve the directory structure, your main options were to either switch to the Java admin client or write a script (e.g. XQuery or Ant) which creates the target collection hierarchy.

The good news: eXide 2.0.9 finally supports directory uploads and preserves nested directories by creating a corresponding hierarchy of collections. The bad news: the feature is only available in Google Chrome. It is hard to say if other browsers will follow its lead or provide alternative APIs. Right now you definitely have to use Chrome though.

To upload a directory, proceed as usual:

  1. open the collection manager via the "File/Manage" menu
  2. browse to the collection into which you want to upload and press the upload icon in the toolbar
  3. on Chrome, a separate button, "Upload Directory" will appear. This is hidden on other browsers. Click on it and select a directory to upload.
  4. alternatively, you may also drag and drop a directory into the drop area marked with "Drop files here ...", e.g. from a finder window on a Mac.

Drag and Drop of Files into the Editor

When developing an application, one often needs to use an existing file on the file system as a template to create a new resource in the database. Using eXide, simply drag and drop the file into the editor. eXide will create a new, unnamed resource and insert the contents of the dropped file into it.

Other changes

Apart from many bug fixes, other notable changes include:

  • eXide now remembers the edit position for all open tabs and restores it when the browser window is reopened
  • drop the size limits for the validation of XML and XQuery files: previously only files with up to approx. 100k were sent to the server for validation. In particular, if you editied larger XML files, no syntax or validation errors were shown. Those limits have been dropped. Please do not forget that eXide is web-based and not the right tool to edit really large XML documents, but for the average document, the validation features work very well.
  • improved support for "less" stylesheets: when saving a .less file, eXide calls the less pre-processor to generate a corresponding .css file in the same collection. This feature was quite buggy in older releases. In particular, relative paths and imports are now handled properly.

eXist-db 2.2RC2

The second release candidate for eXist 2.2 is available for download.

RC2 contains numerous bug fixes, enhancements and performance improvements. The new range index - introduced with RC1 - has seen a lot of real-world testing and bug fixing to make sure it is properly used by queries, so they benefit from the superior performance of this new index.

Beyond the new range index, a number of other performance bottlenecks have been identified. Each of those can have a huge effect on particular queries: for example, the removal of unnecessary type checks on index-assisted functions and redundant cleanup cycles in the query engine increased performance in some cases by factor 10 or more. NGram indexes have also seen a huge speed up with respect to indexing as well as query times. Finally, the query optimizer missed to descend into some types of expressions, causing them to run without optimization.

Using the NGram index to search Sanskrit texts

Noteworthy other changes include, but are not limited to:

  • Fix: a caching issue led to random index failures for data sets large enough to fill the entire caching space.
  • Fix: faster addition or insertion of nodes using update extensions. In particular, the time to append a node should not increase with the size of the document anymore.
  • Fix: removing a collection containing a large number of documents or subcollections resulted in a huge recovery log being written, which does not only consume disk space, but may eventually also lead to recovery failures.
  • Enhancement: faster consistency checks and thus backups.
  • Enhancement: allow queries to be terminated by user: now effectively stops all types of expressions that have blocked db shutdown before.
Query profiling with Monex

Nearly all apps shipping with eXist have been enhanced as well. There were so many changes and additions that we will need to cover them in a separate article during the next days, so please stand by.

The final release of eXist 2.2 is scheduled for the first half of August.