Fraunhofer IPSI

IPSI-XQ - Frequently Asked Questions

  1. Miscellaneous
  2. Using advanced features
  3. Querying large files
  4. Using the API

1 Miscellaneous

Where can I get the latest version of IPSI-XQ?

The latest version of IPSI-XQ is always available from our website at http://ipsi.fraunhofer.de/i-info/ipsi-xq/.

Is IPSI-XQ an open-source project?

IPSI-XQ is not an open-source project. Only its binaries are available for free for non-commercial usage. For academic purposes it is possible to get parts of IPSI-XQ's sources to integrate it in own research projects. If you are interested in doing so, a separate contract with us is necessary. Please contact us for further information.

2 Using advanced features

How can I use the schema support?

The usage of schema support is specification conform, so simply read the XQuery specification for further information. In order to import a schema, you always need to use the "at" feature to specify the location of your schema file.

How can I use modules?

The usage of modules is specification conform, so simply read the XQuery specification for further information. In order to import a module, you always need to use the "at" feature to specify the location of your module file.

3 Querying large files

Why is the query engine slowing down?

If you try to query files larger than 100kb you might realize that the query engine is significantly slowing down. The reason for this is the naive(standard conform) implementation of document order. If you have a look at the core query, you will realize this as the distinct-doc-order function calls. These function calls are very costly.

But you can switch off the evaluation of the distinct-doc-order function by setting the Java system property "distinct-doc-order" to "false". Therefore you simply provide the option "-DdistinctDocOrder=false" when starting Java, e.g. in the runGUI.bat file.

Why do I get OutOfMemoryErrors?

The current data model implementation of IPSI-XQ is directly based on DOM. That means every document that you want to query is built-up as DOM structure in main memory. Therefore IPSI-XQ is limited in its possiblities to query large files.

Documents up to 2 MB should be no problem. If you want to query larger files, you could increase the maximum heap size of your JVM, e.g. by setting the -Xmx option. Using this you should be able to query files up to 10 MB. For even larger files, you need to plug-in a database.

4 Using the API

How can I execute an XQuery from my application?

IPSI-XQ currently supports three different methods of using the XQuery engine from your application. The first API is delivered since the first release of IPSI-XQ and provides a very abstract API, which hides all implementation details. You simply provide the query as a String and returns the query result as DOM nodes or Java primitve values. See the following examples:

import de.fraunhofer.ipsi.xquery.api.*;
import de.fraunhofer.ipsi.ipsixq.api.XQIDriver;
import java.util.Properties;
import org.w3c.dom.NodeList;

...

Properties props = new Properties();
props.setProperty("CorePrettyPrint", "false");
props.setProperty("infer", "true");
props.setProperty("eval", "true");

Driver driver = new XQIDriver();
Connection connection = driver.connect(".", props);
QueryResult result = connection.executeQuery(someQuery);
NodeList nodes = result.getNodeList();
				

The second API gives you more control over IPSI-XQ. You also provide a query as String, but now you can choose to get the AST(abstract syntax tree), the result type, or the query result as data model instance. See the following example:

import de.fraunhofer.ipsi.ipsixq.api.*;

...

XQIQueryHandler handler = XQIQueryHandlerFactory.getInstance().getQueryHandler(someQuery);
XQueryNode ast = handler.getXQuery();
String resultType = handler.getInferredType();
AnyType result = handler.getQueryResult();
				

IPSI-XQ release 1.3.3 introduces XQJ, the XQuery API for Java (see JSR 225: XQuery API for JavaTM (XQJ) for further information) as a third option for query execution and result processing. XQJ is an attempt to standardize access to the XQuery datamodel for programs written in the Java programming language. The current implementation of XQJ in IPSI-XQ should be understood as a preview rather than state-of-the-art; the same applies to the API itself. Its development status is marked as 'early draft review', and probably subject to future changes.

IPSI-XQ currently supports most of the datamodel-exposing interfaces of XQJ, i.e. retrieval and iteration of result sequences, extraction of nodes and atomic values from sequences, binding external variables and rudimentary type querying facilities.

The following usage pattern illustrates the most important aspects of IPSI-XQ's XQJ support.

import javax.xml.xquery.*;
import de.fraunhofer.ipsi.ipsixq.api.impl.*;

//define the query
String query = "declare namespace my = \"http://www.example.com/my\";" +
"declare variable $my:elem external;" +
"declare variable $my:attr external;" +
"declare variable $my:float external;" +
"<elem>{ $my:attr, $my:elem }</elem>," +
"(1 to 10), $my:float";

//establish connection to IPSI-XQ engine
XQDatasource dataSource = new IPSIXQDataSource();
XQConnection connection = dataSource.getConnection();

//bind a DOM element node as an external variable in the dynamic context of the connection;
//this variable will be visible in all expressions created via that connection
Document document = DocumentBuilderFactory.newInstance().newDocumentBuilder().newDocument();
connection.bindNode(new QName("http://www.example.com/my", "elem", "my"), document.createElement("myElem"));

//create the expression used to carry out the query
XQExpression expression = connection.createExpression();

//bind additional external variables in the dynamic context of the expression
expression.bindNode(new QName("http://www.example.com/my", "attr", "my"), document.createAttribute("myAttr"));
expression.bindFloat(new QName("http://www.example.com/my", "float", "my"), (float) 20.375);

//execute query, process result sequence items according to their kind/type
XQResultSequence sequence = expression.executeQuery(query);
XQItem item = null;

Transformer transformer = TransformerFactory.newInstance().newTransformer();
while (sequence.next())
{
	item = sequence.getItem();
	if (item.getItemType().getBaseItemType() == XQItemType.XQITEMTYPE_DOCUMENT)
		System.out.println("Document Node:");
	else if (item.getItemType().getBaseItemType() == XQItemType.XQITEMTYPE_ELEMENT)
		System.out.println("Element Node:");
	else if (item.getItemType().getBaseItemType() == XQItemType.XQITEMTYPE_ATTRIBUTE)
		System.out.println("Attribute Node:");
	if  (item.getItemType().getBaseItemType() == XQItemType.XQITEMTYPE_ATOMIC)
	{
		System.out.println("Atomic:");
		XQItemType intType = connection.createItemType(XQItemType.XQITEMTYPE_ATOMIC, null, new QName("http://www.w3.org/2001/XMLSchema","integer","xs"), false);
		if (item.instanceOf(intType))
		{
			System.out.println("xs:integer: " + item.getInt());
		}
		else
			System.out.println("other: " + item.getString());
	}
	else
		transformer.transform(new DOMSource(item.getNode()), new StreamResult(System.out));
}

More information about the various APIs can be found in the corresponding Javadoc documentation.

How can I plug-in my own data sources?

If you want to plug-in your own data sources into IPSI-XQ you should implement your own DocumentManager(de.fraunhofer.ipsi.xquery.io.DocumentManager). A DocumentManager handles the calls to the two input methods (doc(uri), collection(uri)). If you implemented such a DocumentManager, you need to register it with the Java system property "-DdocumentManager=aaa.bbb.ccc.CLASSNAME".

For this you normally also have to implement your own data model classes. The interfaces and some more information about this can be found in the Javadoc documentation.

Inorder to make the XQuery evaluator build your own data model classes, you can implement and set your own DatamodelManager (de.fraunhofer.ipsi.xquery.datamodel.DatamodelManager) with the Java system property "-DdatamodelManager=aaa.bbb.ccc.CLASSNAME".

For even more flexibility you can implement your own XQIQueryHandler (de.fraunhofer.ipsi.ipsixq.api.XQIQueryHandler). This class handles the complete query process and e.g. starts the individual processing steps. The instantiation of an XQIQueryHandler is done in an XQIQueryHandlerFactory, which must be also provided by you and registered with the Java system property "-DxqueryFactory=aaa.bbb.ccc.CLASSNAME".

How can I add external functions?

In the latest working drafts, XQuery allows to define external functions in a query. This is supported by IPSI-XQ. See the following example for provinding own external functions:

import de.fraunhofer.ipsi.xquery.functions.*;
import de.fraunhofer.ipsi.xquery.context.*;
import de.fraunhofer.ipsi.xquery.datamodel.*;
import de.fraunhofer.ipsi.xpathDatatypes.*;

public class MySqrt implements FunctionDefinition
{
	public AnyType evaluate(List values, StaticContext stat, EvaluationContext eval)
	    throws FunctionException
	{
		AnyType val = (AnyType)value.get(0);
		AtomicValue atomic = null;
		if(val instanceof AtomicValue)
		{
			atomic = (AtomicValue)val;
		}
		else if(val instanceof Sequence)
		{
			atomic = (AtomicValue)((Sequence)val).getSequenceImpl().get(0);
		}

		XS_Double input = (XS_Double)atomic.value()

		double result = Math.sqrt(input.doubleValue());

		DatamodelManager manager = DatamodelManagerFactory.getInstance().getDatamodelManager();
		return manager.newAtomicValue(new XS_Double(result));
	}
}

...

ExternalFunctions functions = ExternalFunctions.getInstance();

XS_AnyURI uri = new XS_AnyURI("http://www.example.com/myfunctions");
XS_QName functionName = new XS_QName(uri, new XS_NCName("sqrt"));

functions.registerExternalFunction(functionName, new MySqrt());

...

(: An XQuery that calls my external function :)
declare namespace my = "http://www.example.com/myfunctions"
define function my:sqrt($val as xs:double) as xs:double external

my:sqrt(4)
				

How can I add external variable bindings?

In the latest working drafts, XQuery allows also to define global and external variables in a query. This is also supported by IPSI-XQ. See the following example for providing external variable bindings, employing the second API:

import de.fraunhofer.ipsi.xquery.context.*;
import de.fraunhofer.ipsi.xquery.datamodel.*;
import de.fraunhofer.ipsi.xpathDatatypes.XS_String;
import de.fraunhofer.ipsi.util.QName;

...

XQIQueryHandler handler = XQIQueryHandlerFactory.getInstance().getQueryHandler(query);
EvaluationContext dynamicContext = handler.getDynamicContext();
DatamodelManager manager = DatamodelManagerFactory.getInstance().getDatamodelManager();
AnyType value = manager.newAtomicValue(new XS_String("hello world!"));

dynamicContext.varValue.declare(new QName("myvar"), value);

...

(: An XQuery that uses my external variable :)
declare variable $myvar as xs:string external;

concat("My variable value: ", $myvar)
				
Please see the XQJ sample code for information on how to bind external variables by means of the newly introduced XQJ-API.

How can I use the IPSI-XQ AST(abstract syntax tree)?

The API for the XQuery AST of IPSI-XQ is provided with this release. You find it in the Javadoc in the package de.fraunhofer.ipsi.xquery.tree. This tree has a visitor concept implemented, that means you can write a visitor (implementing VisitorXQuery) and execute this on the AST by calling node.acceptVisitor(visitor). This can be used as framework for several experiments with XQuery without needing to implement your own parser and AST. In theory you can write your own XQuery processor based on this!