Chapter 3. XML databases

Table of Contents

Native XML databases
Schema-free native XML database collections
Indexing
Classification of XML databases based on their contents
Usage of native XML databases
Hybrid systems
Problems of the fully XML based storage layers
Relational repositories with XML wrapping layer
Hybrid repositories
Other considerations

Let’s start with an apparently simple question:

Is an XML document a database by itself?

Just like from the previous chapters, it has become clear that an XML document consists of data as well as metadata. The collection of data is the most basic definition for databases. This collection contains information, more specifically data, like the bookkeeping records of a firm, or even worldwide demographic data. Beyond this, the basic database structure describes what kind of data can be found in it.

What is the structure descriptive data in a database? The answer is, of course, the database's metadata. That metadata of the database is the data from that part of the database which describes the information held in the database. These amount of metadata makes possible the retrieval of data, just like when we know during the reading of the customer or demographic data, that we are reading just the names of the customers or the data about all citizens who live on a particular part of the World. Without metadata the database's data become a holy shit.

The semantics of the data can obviously be implemented into an application. These apps can be programmed to read specific items from a disk, like the name of a client. Contrarily, it is much more simpler if we also hold this information in the database itself. The structure in result is significantly simpler for the programmers and for the applications as well, and also more effective.

All in all, it can be seen from the previous explanations that XML documents also forming databases by themselves!

The question rises:

"Is this an information that held in the XML document but in the XSD (XML Schema Definition)?"
The answer is NOT of course.

An XML document by itself is also a database – exactly what it is. An XSD or a DTD only needed when we want to check the structure of the XML document whether fulfills or not the appropriate implementation created by the developer.

The XML language is a self-describing language because it contains the data and the necessary metadata for describing its structure in one place. This metadata can be found in the name of the attributes and elements, continued in the hierarchy of the XML document itself. The metadata is descriptive in both scalar and structural meaning. XML metadata are double-natured:

If it has became confusing again by mixing up the XSD and DTD way with the relational approach, it is understandable but it is important to point out that the metadata of an XML document are the elements (tables), attributes (fields) and the hierarchy (relations between tables) that come with an XML document – these are the metadata. An XML document does not need a relational table structure to define itself – because it is already a database by itself.

All in all, the usage of XML has a huge benefit which means that an XML document already has a built-in database structure. XML is self-describing in that interpretation that contains both data and metadata. The metadata are also a descriptive in both scalar (as the fields and values) and structural way (like hierarchy). XML is portable because it is clearly interpretable.

On the other hand, the usage of XML as a database has a lot of disadvantages as well. There can be overlaps in XML documents and because of that it can become wordy. In reality, an XML document contains a huge amount of information in one place and often in one file. This property can lead to enormous processing times for huge amount of data. Essentially, the connection to XML documents – in its basic form – requires full reading, interpretation and of course in cases of potentially big text files and huge resources.

Of course, it would be pointless to compare this size with the power of commercial (typically an Oracle or SQL Server) databases where the amount of data can be measured in terabytes. Right now you can’t find such system on the market which could achieve a size more than a few gigabytes, but even if it could, its processing time would be out of sanity.

Native XML databases

By definition, an NXD (Native XML Database) can be an XML document and also an XML datatype as well. An XML datatype is a special data type of relational databases. This means, an NXD can be practically anything that can store XML data - like an XML document. So an XML document represented in the browser is also an NXD. Besides this, the usage of XML datatypes in relational databases like Oracle or SQL Server can enable to reach the properties of the NXD. Essentially, all we need to describe a database to be an NXD or at least NXD capable is to store an XML document like and XML document. This way we can use the relational, object (-relational) or hierarchical databases as well.

The native XML expression actually means that the database is part of the XML. In other words, as we have put down earlier and repeated in the text, the XML document contains both data and metadata. The data is data and the metadata can be found in the structure, or at least bounds some meaning to the data. The enhancement of the metadata can be seen in the example below:

<?xml version="1.0" encoding="utf-8"?>
<account:persons xmlns:information="http://example.org/information" xmlns:account="http://example.org/schema/person">
	<account:person name="Teszt Elek" age="23">
		<account:favourite number="4" dish="Sultkrumpli"/>
		<account:favouriteWebPage>
			<account:address>http://www.w3.org</account:address>
		</account:favouriteWebPage>
		<information:address>
			<information:city>Debrecen</information:city>
			<information:street>Kis utca</information:street>
		</information:address>
	</account:person>
</account:persons>

Schema-free native XML database collections

An NXD often contains and stores more than one XML document like collections of XML fragments. An NXD can contain XML documents from different types where the documents are about different topics containing independent data. For example, one collection consists of the data of a firm's customers while the other collection contains demographic data of the countries from the Earth. In addition, the XML fragments' structure in each collection can differ from the others. Every simple fragment in a collection can have different structure from the other fragment in the given collection. This result structural and scheme independent XML data.

The collections, if validation required for the XML data, can have poses a relation to schemes. Scheme independent XML makes development spectacularly flexible, faster and easier. However, the flexibility comes with low data integrity and with the risk of faults among the data inside the database. In reality, there are uncountable reasons why we should transform our collection of XML documents into an NXD for achieving better performance and storing methods.

Nevertheless, the misunderstanding and misinterpretation of this flexibility can cause problems. The repetition of too many data, too much or not enough structural complexity can lead to surprisingly many faults. The list of potential pitfalls is as long as the possible different variations of a topic made by an XML-like tool's flexibility. We can represent this through a simple example where we would like to describe a region:

<?xml version="1.0"?>
<database name="Africa">
	<collections>
		<collection name="Countries">
		</collection>
	</collections>
</database>

The example above did not contains any data but it consists more collections where every collection stands for a state. In this example the states belong to the African region. We can add new regions to achieve a multilevel database.

<?xml version="1.0"?>
<databases>
	<database name="Africa">
		<collections>
			<collection name="Countries">
			</collection>
		</collections>
	</database>
	<database name="Europe">
		<collections>
			<collection name="Countries">
			</collection>
		</collections>
	</database>
</databases>

In spite of this, we can go back to the way of getting more collection inside the database:

<?xml version="1.0"?>
<database>
	<collections>
		<collection name="Regions">
			<region name="Africa">
				<population>789548670</population>
			</region>
			<region name="Europe">
				<population>488674441</population>
			</region>
		</collection>
		<collection name="Countries">
			<country name="Finland" region="Europe">
				<population>5231372</population>
			</country>
			<country name="Germany" region="Europe">
				<population>82422299</population>
			</country>
		</collection>
	</collections>
</database>

Collections inside of a collection can also be arranged into real hierarchies just like the countries into the appropriate regions:

<?xml version="1.0"?>
<database>
	<collections>
		<collection name="Regions">
			<region name="Africa">
				<population>789548670</population>
			</region>
			<region name="Europe">
				<population>488674441</population>
				<country name="Finland" region="Europe">
					<population>5231372</population>
				</country>
				<country name="Germany" region="Europe">
					<population>82422299</population>
				</country>
			</region>
		</collection>
	</collections>
</database>

When continuing this example we can see how limitless the options for further evolution in these schema-free XML databases. However, they are carrying all the aforementioned risks which can easily turn into problems and faults. Of course, if we assign a scheme to an XML document we loose this flexibility but the other hand, the database system gain the power to enforce structural constraints.

Indexing

We can already be familiar with the techniques of indexing related to relational database systems, where one of our aims is to get the data with the least I/O processes. In case of relational tables the most commonly used indexing methods are binary trees, hash algorithms and bitmaps. When we talk about indexing, there is one important thing that we have to emphasize: there is a sequence in it. That is, when we are reading an index, we are reading the data in the sequence as the index has created them, independent from the order that can be found in the table or in the XML document. However XML document already contain an inner sequence, which is based on the XML structure, so the arranged index is not necessarily an advantage in case of XML data bases. XML data bases have different indexing methods. Probably every native XML data base applies different ways of indexing in their actual implementation.

The most widely used indexing method in case of XML documents in relational databases, even in case of XML data type, is separating those elements that need indexing. This way the index will contain a simple field about every record in the table. This single field contains a name about every region. Furthermore, the index will contain another pointer/indicator, which makes the linking of index and the table (or XML document) possible; it a physical address on I/O level. In other words, the database has to have a physical address about every (indexed) element in the XML document. This way the index contains the address of this pointer. The result is the following: when it finds a region in the index, the indicator related to the index will be given to a function, which finds the record in the table, or in an XML document on the basis of the hard disk address of the entrance record. The hard disk addresses get assigned to the table or the elements of the XML document when the whole pile of data or the index are created. The process and the steps of the process fully depend on the software used to reach the XML database, or which makes a simple XML document or a set of XML documents as one collection (These collections are obviously stored as XML).

Some questions may rise at this point, such as:

Why don’t we store XML documents in XML data type, if indexing the XML data type is problematic, and it may be subordinated to the indexing of the relational data base?

or:

Why don’t we simple store data in relational tables and convert them to XML, if needed?

At first sight, we should examine the second question at first. Converting is not necessary, if the storage is in XML format. What’s more, we will have access to such useful XML tools like XPath and XQuery. Answering the first question: index is index. Some databases have more friendly indexing techniques than others. There is no reason, why XML indexing would be less effective than any other indexing, no matter what kind of relational database we talk about. Naturally, XML data can be stored in relational table. It’s the matter of choice. Both ways have their own advantages and disadvantages. In general we can say, that the bigger the database, the more sensible it is to store data in relational table – and not the XML stored in an XML data type. Unless we use XML data type collections. Keeping these in mind, we know 4 ways of indexing:

  • Structural index

    Index of elements and attributes, and their relative location to the other elements and attributes in a simple XML document. It helps searching for elements.

  • Value index

    If text and attributes are often searched items in an XML document, the best solution is if we index these, or the combination of text and attribute values.

  • Full text index

    This one relates to the quest of a specific value in the collection of XML documents, and returns some portion of this collection. The value of the index is quite big, it contains several XML documents or some parts of a collection.

  • Context index

    This one is a more general type of indexing, maybe a little bit old-fashioned, where many documents are indexed in a way, that the index contains some kind of a value, which identifies the XML document unambiguously. Indexes are stored in a so-called “side” table and this table gets an index. The solution is a quick indexed access in the collection of XML documents. This method is quite tiring. It’s better if the context of the XML document gets an index. It’s like we were using big and disordered XML files instead of smaller and better categorized fragments. All kind of manual categorization is very time and resource consuming in modern databases, simply based on the physical size of the information.

Classification of XML databases based on their contents

XML documents contain data, metadata and some kind of semantics in their inner hierarchical structure. Document centred documents are better for human application. However, they can be data-centred as well. These are more generic, and mainly used for processing by script languages.

Document-centred XML

A document-centred XML, which is also good for human consumption, is hardly interpretable for a computer, if it is possible at all. The document-centred XML is a document, which is normally handwriting – as a Word document, PDF file, or something similar to these notes. These kind of XML documents are usually stored as wholes and they are normally not understandable by programming methods, or by an XML element's content. Sometimes these documents are indexed for index searching, like the library of technical papers. Contrarily, there are some databases that contain technical data, which mixes data and document centrality. For example in the database of a library, which stores technical data of several years of the past, it is worth to categorize data according to authors, subject, time of publishing, or other kind of descriptive information. The content of these documents could be indexed, so that it would give general information about the subject of the given text.

A special type of document-centred native XML is called Content Management System (CMS). Content management systems permit some kind of control and management through XML data written by human beings, which are stored as XML types in a native XML database.

Data-centred XML

A data-centred XML document uses XML data in their simplest way for data transmission among computers. In reality, XML often contains the mixture of data-centric and document-centric features. The part, which has been made by humans and which can be interpreted for human beings belongs to the document-centred features. Something, that is generic, and available for a programme as it is reproducible, belongs to the data-centric part.

Maybe the best examples for data-centric documents can be found on websites, like Amazon or Ebay. These pages contain a mass of information, with the widest range of diversity. For example a book on the Amazon website, or better say every feature of it, such as ISBN number, author, or other kind of data are data-centric. On the other hand, many books on Amazon are available in PDF format. PDF files are document-centric, specific to a given book, and can be programmed correlated to the PDF document, in contrast with the content of the PDF.

Usage of native XML databases

It is important to go through the different aspect of the usage of native XML databases

  • Storage in native XML documents:: XML documents can be stored as character data in databases, as binary object, or some kind of XML data type. Some relational database make it possible to store more than 4000 characters as a string. As the length of XML documents cannot be predicted, this is usually not enough. Binary objects can be simple elements stored binary (BLOB), or special binary elements, big text documents (CLOBS). The size of CLOBS is usually physically limited (in most cases 4 GB) in contrast with the strings. What is more, in contrast with long strings, or BLOBs, CLOBS usually make some kind of textual search or pattern-matching possible. Although we have to say that no CLOB based pattern-matching can be compared to the real XML handling technologies developed for XML documents. Some relational database makes it possible to store XML documents in their own format. As XML data type, with full XML capabilities. Native XML databases that have been created for this reason obviously have to have the functionality of XML handling besides just storing it.

  • Concurrency handling, locking, transaction handling: Native XML databases in relational database as in form of XML data types only support locking on the level of the XML document. That is, the bigger the XML file, the worse the concurrency handling will be and the multi-user aspects as well. The node-level is also an option in XML documents, but imagine the implementation of that. Locking on the node-level definitely requires validation and force a schema too – this way we have thrown XML flexibility away. We can only enhance it knowing that XML documents have hierarchical structure by nature, so any node-locking infers the locking of every parent-node as well. Locking on the node level is not a good solution with today’s technology. The way we structure an XML file highly depends on the requirements of the application. The bigger XML documents we store, the less suitable the database will be in a multi-user environment. A simple XML document, as native XML database is a single user environment in most of the cases

  • Reading native XML databases: reading data of native XML databases, or XML data types requires special tools. These tools are the XPath and XQuery.

  • Changing the content of a native XML database: several tools and functions are available to change the content of an XML document. As for the standards, XQuery makes updates unambiguously possible (despite it is called query, similarly to SQL).