Adding Basic Authentication in eXistdb Apps

Another great video tutorial by Loren Cahlander to help you set up basic authentication in your eXistdb app.

You can also check out his source code for the app on github.

An Introduction to eXist-db HTML Templating Module

Loren Cahlander created a nice introduction presentation on the HTML templating module, which has been part of eXistdb since 2.0. He demonstrates each templating annotation with a simple demo, so if you ever wondered how those generated templating instructions in your eXist app work together, watching the video is recommended:

App Updates: eXide 2.0.9

The 2.2RC2 release of eXist also includes updates to most of the core apps and one new app ("monex"). We'll cover the updates and additions in a series of articles. eXide will be first: version 2.0.9 has a number of new features which may not be obvious and need some explanation:

Live Reload

eXide-live.png

Enabling live reload

When developing a web application, one usually runs it in a separate tab or window, so after making a change to e.g. an XQuery, you need to switch to the other tab and press reload to get an updated web view.

The "live reload" feature automates this: whenever one saves a resource which belongs to an application package, the tab or window running the app's web view is refreshed automatically.

Enabling live reload

Due to security restrictions of the browser, a refresh is only possible if the app's browser window was initially created from within eXide. To enable live reload you thus have to:

  1. in the editor, switch to or open a resource which is part of the application package you want to run
  2. in the "Application" menu, select "Live Reload"
  3. a dialog will appear if the application has not been run before from within the same eXide instance. Click on the application link within the dialog to open a new tab with the application's web view.
  4. switch back to eXide

The next time you save a resource, the application view should refresh automatically. Depending on your browser and configuration, the application window or tab may either get focus immediately or it is refreshed in the background and you have to switch there to see the change.

Once live reload has been initialized, you can always toggle the feature on/off via the checkbox in the menu without having to open a new application window.

Personally I mostly use live reload in a setup with two screens, having eXide on one screen and the application on the other. It also works great on a smaller laptop screen with multiple open tabs though.

Directory Uploads

eXide-upload.png

Upload dialog

Users often critisized in the past that uploading nested directories is not supported by the web-based collection browsers built into dashboard and eXide. If you had to preserve the directory structure, your main options were to either switch to the Java admin client or write a script (e.g. XQuery or Ant) which creates the target collection hierarchy.

The good news: eXide 2.0.9 finally supports directory uploads and preserves nested directories by creating a corresponding hierarchy of collections. The bad news: the feature is only available in Google Chrome. It is hard to say if other browsers will follow its lead or provide alternative APIs. Right now you definitely have to use Chrome though.

To upload a directory, proceed as usual:

  1. open the collection manager via the "File/Manage" menu
  2. browse to the collection into which you want to upload and press the upload icon in the toolbar
  3. on Chrome, a separate button, "Upload Directory" will appear. This is hidden on other browsers. Click on it and select a directory to upload.
  4. alternatively, you may also drag and drop a directory into the drop area marked with "Drop files here ...", e.g. from a finder window on a Mac.

Drag and Drop of Files into the Editor

When developing an application, one often needs to use an existing file on the file system as a template to create a new resource in the database. Using eXide, simply drag and drop the file into the editor. eXide will create a new, unnamed resource and insert the contents of the dropped file into it.

Other changes

Apart from many bug fixes, other notable changes include:

  • eXide now remembers the edit position for all open tabs and restores it when the browser window is reopened
  • drop the size limits for the validation of XML and XQuery files: previously only files with up to approx. 100k were sent to the server for validation. In particular, if you editied larger XML files, no syntax or validation errors were shown. Those limits have been dropped. Please do not forget that eXide is web-based and not the right tool to edit really large XML documents, but for the average document, the validation features work very well.
  • improved support for "less" stylesheets: when saving a .less file, eXide calls the less pre-processor to generate a corresponding .css file in the same collection. This feature was quite buggy in older releases. In particular, relative paths and imports are now handled properly.

eXist-db 2.2RC2

The second release candidate for eXist 2.2 is available for download.

RC2 contains numerous bug fixes, enhancements and performance improvements. The new range index - introduced with RC1 - has seen a lot of real-world testing and bug fixing to make sure it is properly used by queries, so they benefit from the superior performance of this new index.

Beyond the new range index, a number of other performance bottlenecks have been identified. Each of those can have a huge effect on particular queries: for example, the removal of unnecessary type checks on index-assisted functions and redundant cleanup cycles in the query engine increased performance in some cases by factor 10 or more. NGram indexes have also seen a huge speed up with respect to indexing as well as query times. Finally, the query optimizer missed to descend into some types of expressions, causing them to run without optimization.

sarit.png

Using the NGram index to search Sanskrit texts

Noteworthy other changes include, but are not limited to:

  • Fix: a caching issue led to random index failures for data sets large enough to fill the entire caching space.
  • Fix: faster addition or insertion of nodes using update extensions. In particular, the time to append a node should not increase with the size of the document anymore.
  • Fix: removing a collection containing a large number of documents or subcollections resulted in a huge recovery log being written, which does not only consume disk space, but may eventually also lead to recovery failures.
  • Enhancement: faster consistency checks and thus backups.
  • Enhancement: allow queries to be terminated by user: now effectively stops all types of expressions that have blocked db shutdown before.
profiling.png

Query profiling with Monex

Nearly all apps shipping with eXist have been enhanced as well. There were so many changes and additions that we will need to cover them in a separate article during the next days, so please stand by.

The final release of eXist 2.2 is scheduled for the first half of August.

eXist 2.2.RC1

We are very proud to announce the first release candidate for the next version of eXist, 2.2. The release candidate is feature complete, but not yet recommended for production use. Please let us know of any issues that you encounter so that we may resolve any unexpected bugs and finalise the release within the next month or so.

eXist 2.2 provides a new Range Index, which can accelerate your XQuery code and has proven to be upto 100 times faster than previous versions. This is the fastest release of eXist ever!

You can download the release from sourceforge.

eXist 2.2.RC1

New Range Index

2.2 features a reimplementation of the range index, the most important user-configurable index in eXist. As reported by users, some types of queries can run up to 100 times faster. The most dramatic performance increases have been observed on large data sets (with millions of documents) and queries on frequent strings. However, the new Lucene based index also brings many benefits for those working with smaller data sets.

While updates on the previous index system did not scale well with increasing collection sizes, the new index removes those limitations, thus allowing queries and updates to scale up. As previously reported by some users, problems with slow updates and increased memory usage have disappeared since switching to the new index.

Please refer to the documentation for more information.

Improved Crash Recovery

The new version also features a largely rewritten and simplified crash recovery, leading to a more robust recovery procedure and smaller transaction logs.

Security

eXist 2.2.RC1 further extends it's Unix permission model, and now includes setUid and setGid bits. This both allows stored XQuerys to escalate permissions and enables the controlled sharing of Collection documents with groups os users. This is extremely important as it now makes it possible to call a query as an unpriviledged user and have it switch to a different effective user without providing a target for attacks.

Bug Fixes

2.2.RC1 includes numerous bug fixes, some of the highlights are:

  • Crash Recovery - Exceptions during transaction rollback no longer cause the database recovery to be aborted; Previously this was commonly seen as page not initialized errors.
  • Java Service Wrapper - No longer kills eXist if it takes longer than expected to shut down or start up.
  • Concurrency - Removed consequtive query invocation lockups, and many other small fixes.
  • Memory - Memory Leaks in the full text index were fixed.
  • Optimizer - Now descends into XQuery Update expressions.
  • Java Admin Client - It is no longer possible to accidentally lock out the admin user by mis-changing her password.