As we have seen XML is perfect for representing structured data, it can be used to store and archive data, similarly to a relational database. The inner structure of the XML makes it possible to represent the sophisticated relations of various data in one unit. XML documents can be categorized into two groups: traditional SGML based descriptive documents, and the more and more popular data-centred documents.
Descriptive documents are text documents, in which markers help in the formatting, indexing, etc. (such as the collection of technical handbooks, web pages, or online newspapers and magazines). These kind of documents are typically split into articles, sections, or some similar separate and meaningful units. Data-centred XML documents rarely have free formed textual data, and they usually contain similar data structures, such as units that consist of fields (or columns, rows).
Let’s see an example of descriptive document structure. We suppose, that surgeons make copies of cards in a handheld device. We have to create a system, where these cards can be stored for future reference, and they can also be shared with other systems (like the billing system of a hospital). At this example, we can use XML to enter the cards on the devices, but we would surely use it to transfer them to the storage place (where storage might be in form of XML, relational database, or a combination of these). It does not matter how the cards are actually stored, we would probably use XML to export data into other subsystems (billing, medical report, HR, etc…) This is what an example card looks like:
<transcript> <physican id="MD-123456" /> <patient id="035-419-876" /> <procedure>Making a 2 cm long cut on the <location>left upper arm</location> skin in order to remove the nuclation with a <tool>lance</tool>. </procedure> </transcript>
We have several options during storage:
Does it make sense to store the XML document as a simple file in the system?
Is it worth to divide it into separate sections or values, and store them in a relational database instead?
And what if stored the data in a native XML database?
Let’s change our example a little bit. We will use the same base structure with one difference, now we create invoice documents within a trading system. The transaction data (invoices) have to be stored for future viewing and in such a form, that it could be shared with other systems. The base structure will essentially stay the same, but this one results in a much more data-centric solution, because it does not contain textual parts.
The architecture of the base system stays the same in terms of information flow, so the question remains the same: how can we optimize the subsystems and the data storage architecture at these two documents?
If an organization decides, that they integrate XML into their architecture on venture-level, XML developers often make the mistake of creating a storage layer, which is purely XML based ( and it contains only single XML files).
In many cases it seem to be a sensible decision. After all, data enter and leave the system in XML form, and we use XML to represent data to the end users (using XML style sheets to show data in various forms of media). They why should we take this document apart and store the data in some other form, such as a relational database? Why should not we store everything in XML?
As it has already turned out, the scope of handling data as XML is quite limited – in most cases an XML representation will not be suitable to handle the processing procedures of data, what we would expect for XML documents.
Searching on the purely XML based storage layer is slow
Except the simplest cases, such a purely XML based storage layer will not be enough. These are only a few files in a folder, all of which corresponds to an XML document. XML is not an indexed medium - there is no way to clearly identify, whether an XML document contains relevant information in terms of a programming task. The only way to find, e.g. a specific invoice is to parse the documents and look for an element that suites the searching criteria.
For example in the mentioned invoice system, those customers who sign in can review their invoice history. To make it happen, the system has to be able to identify which XML document suits the given customer, and use these documents (such as creating an XSLT layer, which shows the customer the information that can be found in these documents). What helps the system to decide which document belongs to which customer? The system has to parse every document on the storage layer looking for the <customer> element with the “id” attribute value, which is the same as the currently signed in customer’s id. So every invoice and customer information has to be checked, which is, in a huge system, where there are many customers, and many documents have to be checked which are irrelevant in terms of our current task, leads to a massive loss in processing time – even with a “streaming” parser like SAX.
Document aggregation is complicated
Similarly, there is no easy way of processing many XML documents at one time to obtain data – all of them has to be parsed separately, the most important information has to be gained and saved, and after that, if it is needed, the information has to be aggregated.
Going back to our former example, let’s say, we have an invoicing system, where we would like to create a cumulative invoice, which contains all the invoices that are related to one single customer. To achieve this, our billing system has to develop the problem mentioned at our previous example and solve it one step further. Not only does it have to identify which invoice belongs to the given customer ID (and not paid), but it has to retrieve the information from these invoices that is needed to the given task (this case to create a cumulative invoice). After that, the whole of that has to be merged into one format (to the whole amount that has to be paid), this step is often referred to as aggregation processing, or document aggregation.
Manipulation of a document is in vain
Another worrying factor with plain XML storage is the way we process XML documents and gain certain data from them. Let’s suppose, we would like to find out in our invoice example, how many of item YCK001-123456 has been sold last month. After we found out, which documents contain reference to YCK001-123456, we have to call the amount attribute in this document. It is not easy to do, if we parse XML documents directly, because the whole document has to be run through to gain the information. It is also true, if we run the data through the memory with the help of a SAX parser, or if we process the whole with DOM.
Solving the problem of indexing
The general solution of the problem is quite obvious – we have to give some kind of indexing to our XML storage mechanism, to enable us to quickly search for and aggregate the information found in the document. Several solutions exist to index XML data, such as: the previously shown XML databases, the relational solution, or object-oriented databases, or the hybrid combination of these.
As we have seen, native XML databases give index information to the XML storage. These might prove to be a good choice, if we want to keep the original XML appearance of the information, with the condition, that we only want to make a limited processing of the data, because currently these are too resource demanding – however, it will reduce with time and they can become a meaningful alternative. Let’s have a look at the hybrid solutions to the problem.
Another approach of the topic of indexing is storing some part of the information, or the whole of it in a relational database, and using wrapping layers to transfer data between the relational and XML representation. This solution can easily be used in case of repositories, where can be easily broken down. At this approach, XML documents are not stored directly. Instead, the information is parsed, then stored in relational database, or deserialised from XML, depending on the circumstances (for example to operate transmitter or representational levels).
This approach is especially effective, if the information that enters the system is not already in form of XML, or it does not have to be ordered into XML form as part of the enterprise guidelines. Searching methods using this architecture could not be simpler. As every piece of information is stored in relational database, we have access to every mechanism which we would normally use for searching in a relational database: table indexing, key relations, etc…. In order to use the online invoice checking, we only have to write an SQL query that searches for invoices that have the same customer ID. The retrieved invoices can be transformed into XML for representation, whether using an inner XML output producer, or a general procedural code.
In contrast with the native XML databases, this approach makes data aggregation and handling easier. As is breaks down the information into pieces, before they are stored in the database, we can use the working SQL aggregation commands to retrieve data directly. In order to create a uniform invoice for example, we should only write one stored procedure that would download all the unpaid invoices from the customer’s account. Another benefit of this is that this information can be serialized into a proper form – we are not bound to the original XML structure in any way. This makes our presentation layer much simpler as well. Of course, there are some, who live this as a disadvantage, but XML is not a wonder substance, and for many databases using relational storage model might be a better solution, than the XML tree structure.
A disadvantage of this relational-wrapper approach that the serialization and deserialization steps of the XML take time. If a given system highly depends on XML data format (for example a specific XML standard is used to transfer data), than the purely relational approach will take much processing time by compiling and decompiling XML documents. Another problem might be the knowledge of programming – instead of XML and XPath the developer has to be familiar with RDBMS and SQL as well. What is more, in case of XML documents with descriptive style it is very hard, or even impossible to store data in relational database in any usable way.
The purely relational storage method (around an XML wrapper relational layer) might be a good solution, if the system is not too XML-centric, does not contain descriptive information, or it does not have to execute huge aggregation changes on the information.
According to the third approach we assign index to a stored XML documents, that we store the index in a relational database. In reality, there are two different methods, both of them have their advantage.
Storing full XML documents and relational data
In case of the first type of hybrid repositories XML documents are saved exactly as they enter the system, in other words, any other structure that we create on the data will stay as in the XML representation stored in the repository. Besides storing the XML document, the system processes it, and creates a copy of the index information in a relational database. Depending on the requirements of the system, the indexing might be minimal (it indexes only one key, such as the customer ID at our invoice example), or extensive (every single element, or attribute is a separately indexed part). The retrieved indexed information is stored in a relational database index, by the help of which we get access to the proper XML documents.
The deep analysis of the system decides the balance between minimal indexing and the much costly “index everything” approach. The requirements, which drive the system analysis might change in time, as the procedure of the process regulates which parts of the data have to be indexed. Thus, when requirements change, and further indexing is needed, the database can easily be rebuilt by changing an XML document and re-indexing data. This one is another example that XML might be useful at maintaining complex systems – it is usually much easier to change a textual configuration document, than the process code.
As at the relational model, searching at a hybrid repository is also easy – providing, that we index the information that we would like to search for. It is similar to the case, when we search for indexes at a relational database – we have to index the information that the system will probably need for the search. The indexed database gets a command to find the identifier of those XML documents, which contain our customer ID (as in our invoice example). Then it should use these data to reach the XML documents. It is important to point out, that this method makes avoiding the serialisation step possible, which would be unescapable at a simple relational method., so increasing the performance.
Aggregation and document manipulation also become much less complicated thanks to this approach. At a good design, the information that is often used for aggregation and document handling gets indexed – that is, the necessary information can be gained without opening the XML documents themselves. This kind of system analysis is very similar to what is used in designing the traditional RDBMS indexing. As at the purely relational approach, we can take these data and build our XML document, in order to match with our output requirements.
One disadvantage of this method is the extra storage place, which is taken over by the repository. Basically we store the information twice: once the original (maybe lengthy) XML format, and in the relational index. One drawback related to this is the subsidiary complexity of the database maintenance, any change done on the data, will have to be changed in the XML document and its relational index copy as well. This way making synchronization more difficult. Here we have to make the traditional compromise: accepting more complex update codes and leaving more storage place at the documents lead to faster processing and easier code retrieval. Whether it is a good choice at a given programming task, depends on the nature of the application, and the software/hardware boundaries of the system.
Storing XML document fragments and relational data
At the other form of hybrid repository every indexed content is in a relational database, as at our first, purely relational example. However, those parts of the document, which does not have to be indexed (because we rarely search or aggregate them) are stored by the system as XML documents instead of breaking them down into single elements or attributes.
Let’s take our invoices again, which appear in an online invoice tracking system. According to the requirements of the system, if it does not have to search each invoice item, we might store the invoices in a structure that looks like this:
Customer table Invoice table Items table -------------- ------------- ------------- custId invoiceId itemId Name date item Address custId invoiceId City Zip
If we need certain information of an item (for example for presentational purposes), we simply ask for the XML fragment, which is stored in the Items table in correlating with a given invoice.
This method, as the first hybrid method also makes searching in index fields easier. We can use traditional relational database mechanisms to get to the indexed fields, and to identify the documents that meet the requirements. Similarly, this solution also makes aggregation and documents handling easier – we shall simply design the indexing of data often used at aggregation and document handling, this way we will not need an aggregation code.
Comparing hybrid approaches
So what is the difference between the two hybrid approaches? The first one needs more storage place (as the data are repeated from the original XML document in the relational indexing), but it does not need further processing time or code to recover the information in its original form.
In case of the second hybrid solution less storage place is needed, as only the non-indexed XML of the original document is stored, and there is no duplication because of the index saving. The huge disadvantage shows however, when we have to restore the original XML structure - it takes code and processing time to restore the XML document from the indexed parts and fragments.
Please note that the industry standards are becoming more and more prevalent. If we choose an arbitrary division, there is a consortium that defines the XML structures, which are used for data exchange between systems and their description. In many cases, engineers design document repositories without the appropriate experience, which simply store XML as they receive them. This is especially true if a system was built on a principle which transmits the information in the same XML format between itself and other partners. There are several drawbacks of this respect.
Problem of original XML's storage
As we mentioned earlier, the problem with XML standards is that they fail to meet as a one to one mapping with our unique personal representation of business information, or any display requirements. As standards evolve, they may meet individual business requirements and this section will be void, but today it is true.
It is important to emphasize that most XML documents that works well for one purpose, are completely useless for another. Relying on a single XML structure will definitely lead to increased code complexity and processing time, as it can cause major problems when it comes to maintain or update the software, which is creating and searching in it.
Consider again the billing system that we talked about earlier. We would like to store individual accounts, as the clients hand them in (as at a typical online transaction processing, also known as OLTP system), and let clients have the opportunity to access this information directly later. What is more, we would like to be able to do some analysis of the administered accounts – such as "how many invoices a client receives a year ", or "how many pieces were ordered of a given product in 2012," and so on... These systems are the most well-known as "online analysis processes, "or OLAP systems. We have already defined an XML format for our personal accounts, and we want to save them in a separate simple XML format. While this is effective in supporting the OLTP system side (since the display is the same as the original input), it is more than ideal from the OLAP perspective. If analysis is needed, the document storage must be adapted to be able to establish the analytic data, which will lead to slower and slower analytic performance as the database grows.
One solution to this problem is to create a database that can perform both OLTP and OLAP functions. However, it is less satisfactory for processing or analytical purposes – as one or the other almost always will be damaged. There is a different kind of solution you can choose, we can design a divided XML storage layer to directly support both the transactional and analytical requirements of the system.
Changing storage according to the requirements of the task
In this solution, the OLTP documents are processed as they enter the system. The information is read from these OLTP documents and is used to update OLAP documents. For example, if we would like to show the total amount of every single product, which have been ordered in our system every month. To do this, we could design a document with its own structure, then, as the invoices enter the system, they are able to be processed and the important information can be used to update the appropriate OLAP documents. Once we have the OLAP document, we can simply rely on a naming convention of files, or create a relational index to the documents.
The disadvantages are clear – more storage space is consumed, the information is repeated or extended processing time will be required to recover the appropriate OLAP information from each document when it arrives. The advantage of the OLAP processing is that it is much simpler and we have direct access to the XML documents without the need for the step of XML serialization.
Consider using customized XML structures that directly support the XML data input and display requirements. This solution depends on three criteria (e.g., storage space, OLTP processing time or OLAP process time), which is the most critical in our personal architecture and business environment.
Difficulties in managing unstructured data
Another problem with the split of the XML documents to relational databases, is the question of narrative elements. Take our medical record example. It will be very difficult, if not impossible, to define a relational database structure that describes the number of possible variations of an inherently descriptive document. Although there are structured islands in the text, they can appear in so many different combinations and sequences so that we would not like to force them into a certain relational structure. Of course, we cannot just leave texts out just because they do not fit into a pre-defined structure, since this data is likely to be needed later during processing. It is clear that a different approach must be selected.
The best solution that is expected is to apply the first version of the hybrid modelling: the content of the XML document is always available, and indexing the information makes quick data capture easy, which we currently need at a given task.
The problem of archiving
About archiving we mean the long-term storage of data where we no longer need direct access (as opposed to, say the online processing), they cannot be completely removed (due to business and / or legal considerations ). A properly designed single XML document is able to give complete picture of a particular data set. As long as you have enough meta-information (in the form of good attributes and element names), the document can be self-contained and yet be interpretable outside the context for any processing software, data dictionary or schema. Moreover, since XML documents are in text format and can be compressed easily, making it possible to store a lot of documents and related meta-information in a single volume for future recovery. However, forcing data archiving in XML – especially if the source information is not XML (like data of a relational data base) – can result in difficulties when you want to access archived data.
The most obvious method by designing an archiving strategy is to ensure that the information is in retrievable in a usable format. In the past, it usually consisted of transhipment the relational data into a file, whose structure is the same as the original database so that the data can easily be loaded (parts thereof or bundles) into their original structure for future work. However, the advent of XML has created an opportunity to make archive documents that are self-sufficient, also known as self- descriptive. This is particularly relevant in today's rapidly changing technological world. No one knows that the tailor-made system that generated all the archived entries will exist in even twenty, ten, or even five years from now. With this in mind, the archiving strategy should be directed to design self-describing documents, which are able to provide all the information that you have access to.
Recovery of archived information
Documents archived in XML can fall into the same error as individual documents stored in XML have to face. These documents are stored in many files and still they should remain readily identifiable to a specific document or documents in certain restore operations.
Fortunately, we already know a way to store information, namely XML. You can use XML for information indexing, transformation and summarization as part of the archiving process. This way we reduce the number of documents to be processed when we access the archived material but improve the archive search and retrieval performance. When different ways to search XML documents were discussed all of them revolved around the theme of indexing– extracting specific information from the documents and using them to return to their original XML document. We can apply a similar strategy to the archives, but we will create our index in XML format.
Several solutions can be found compared to former ones. First, let's look at a way of simply selecting a few important data elements and attach a copy of them next to the correct file reference in the archive, for example, customer or patient's name, date of registration. When a request is received, information obtained under this index that helps to identify to which actual file has to be checked searching for the detailed data.
The alternative to this is a more advanced version of it so you can create documents that provide aggregate information. A typical case where a business processes often receives queries to retrieve aggregated information, and not just for the daily data. In this case we might create aggregated documents as well. These summary documents could take over the role of the archive documents, or (more probably) collaborate with them in an archive. This is similar to the indexing technique which has been previously analysed in this section – the aggregated XML documents efficiently form an upper level index for the sub–documents, thereby allowing the summarized information query without the need to delve into individual archives.
As a closure we have to talk about something that people rarely think about. The displaying code or style sheet for a given XML document. In some cases, there may be a business or legal restriction that the documents should be maintained in a particular format for years but it is not the bigger part. If we take advantage of the desirable decomposition of the semantic meaning and the representation at the XML database design layer, we can face a situation where we may no longer have the sufficient information to restore it to the original form without some code or style sheet. For this reason, it is strongly recommended to archive the style sheets and codes related to all XML documents.