Compile once and evaluate multiple times

When a developer wishes to use the P6R rule engine (reference An XPath Enabled Rule Engine), he can simply create a single instance of the rule engine component, load it with a set of rules (defined in XML), load the stated facts (in XML or JSON), and generate inferred facts by calling the evaluate() method one or more times. This particular use case will be fine for many applications, but what about applications that need to evaluate the same rule set concurrently (e.g., adserving)?

The approach we have taken to support multiple, concurrently, evaluating rule engine instances is that of “Compiled Rule Sets”. (This is similar to the approach we have taken to improve performance for XSLT, described in the article: Compiled Templates in XSLT 2.0.)

Compiled Rule Sets as objects

The P6R rule engine (p6IRuleEngine) can be used in a simple mode of execution where a set of rules are first loaded (via the modifyRules() method) and then evaluated to generate inferred facts (via the evaluate() method). For some applications this mode of execution will be acceptable.

However, for high performance applications the same set of rules will need to be executed concurrently by several threads. Compiling the same set of rules over and over again for each thread is slow and wasteful, thus we support compiling a rule set into a data structure that is a P6R component on its own. Two additional interfaces support this complex mode of execution. First, the p6IRuleSets interface allows the caller to extract out (as well as to load in) a compiled set of rules from a running rule engine instance. Second, the compiled rules are represented by the p6IRuleCompiled interface (which allows the compiled rules to be marked with a name for caching purposes). The compiled set of rules contains almost no execution state and thus can be shared with multiple instances of the p6IRuleEngine component at the same time.
Continue Reading »

Introduction

I’ve had several questions lately asking specifically what a COM component is, what follows is hopefully meant to provide some quick answers to those questions.

Fundamentally, a COM component (or component-object, or COM object) is just a reusable piece of code and data in binary form that has the following 4 attributes:

• Encapsulation – Components must hide the details of how they are implemented.
• Interfaces – Only pointers to interfaces are used to interact with them.
• Language independence – Components written in one language can connect to and use components written in another language.
• Dynamic linking – Components must link dynamically.

Continue Reading »

A problem of performance

One common concern with the use of XSLT has been performance. The need to compile an XSLT template, with its embedded XPath expressions each time a Web page is to be displayed is wasteful. Part of P6R’s implementation of the XSLT 2.0 standard includes an extension that helps address this performance problem. Our XLST processor (see XJR SDK) compiles an XSLT template into an essentially static data structure that can be used over and over again by multiple threads at the same time. This is achieved by removing most of the dynamic, run-time state out of the compiled templates. This article describes our approach in detail.

Compiled XSLT templates as objects

P6R’s XSLT processor (p6IXSLT) can be used in a simple mode of execution where a template is first compiled (via the compileTemplates() method) and then used in the transformation of a source document. This is a typical run-time model for XSLT processors and for some applications this mode of execution will be acceptable.

However, for high performance applications the same template will need to be executed concurrently by several threads.   Two XSLT interfaces support this complex mode of execution. First, the p6IXSLTTemplate interface allows the caller to extract a compiled template object from a running XSLT processor instance. Second, the compiled template object is represented by the p6IXSLTCompiled interface (which allows the template to be marked with a name for caching purposes).  The compiled template contains almost no execution state and thus can be shared with multiple instances of XSLT processors at the same time.

A compiled template contains all XSLT elements in a tree of nodes along with compiled, embedded XPath expressions, and compiled forms of any regular expressions that might appear in both XPath expressions and XSLT elements (i.e., the Non-deterministic Finite State Automation that represents the regex is its compiled form).   Only compiled regular expressions contain run-time state in a compiled XSLT template and each is protected with its own lock.   During run time all other XSLT execution state (e.g., global variables, keys, result document, source document) are all associated with the instance of the XSLT processor object and not with the compiled XSLT template object.
Continue Reading »

Introduction

P6R’s JSON parser provides a C++ implementation of the SAX2 like interface. (See our JSN Streaming JSON Parser Library, which is also included in our XJR SDK.) Our parser implementation is designed to be high performance and to directly support a streaming IO model. The parser can be invoked with the entire JSON document in one buffer, or the JSON document feed into the parser a chunk at a time over multiple calls. This interface allows chaining of components (e.g., filters, sources and sinks). To assist a developer in debugging a detailed parse trace can be turned on programmatically.

Ultimately, our goal was two fold. First to provide a generic JSON parser that applications could use directly. And secondly, to support JSON data in our XPath 2.0 and XSLT 2.0 products. How the second goal was achieved is described in another P6R article: XSLT and XPath for JSON. We should note that our SAX-like JSON parser is fully implemented and is part of our P6Platform product.

P6R’s SAX2 parser and JSON parser reduce the amount of string copying to help improve performance. In the Java definition ‘String’ objects are frequently passed to the application. However, all this object creation and string copying comes at a cost. We have taken a different approach, we return pointers and a length to parsed strings. These pointers point into the application provided buffer that contains the document to be parsed. Pointers to application provided buffers are defined by our P6JSONSTRING type:

typedef struct  
{
    const P6CHAR* pStart; 
    P6UINT32      length;
} P6JSONSTRING;

P6JSONSTRING values are only valid during a callback into an application written content handler. An application that wants to keep a copy of the string must make a copy during a callback. This way an application has total control of how it manages its own memory and related performance concerns.
Continue Reading »

This document was updated on 11 April 2024.

An excellent introduction to Rule Engines can be found at “Chapter1. The Rule Engine” . However, this description is a bit detailed. In general, a rule engine allows the user to define a set of conditional statements (such as “if x then y”), where when the condition (x) is true the result (y) is executed. Both the condition and execution are application specific. Rule engines have been applied to many applications some of which include ad server systems, email filtering, anti-SPAM software, and business management tools.

As an example in the advertising space, a typical rule could be: if “mother’s day” then “show image banner A”, where banner A is a gift for mom.

There is a lot of technical jargon used to describe how rule engines work two of which are important to understand the rest of this article. The first term is “stated facts” which are fixed facts that are given as input to the rule evaluation (e.g., the price of a typical car is $22,000). And the second term is “inferred facts” which are facts that are generated by the rule engine (these come from a rule’s actions, e.g., Henry bought an expensive car since it cost more than $22,000).

What follows are details of what makes our rule engine so powerful. (See our XJR SDK.) While we where designing our rule engine we knew that we needed an expression language to define a rule’s conditional logic. Luckily we already had built XPath 2.0 which is a powerful expression language that is embeddable into XML.

Examples of other Rule Engines include open Source Rule Engines in Java (Drools is a great example of this). Another popular rule engine is from ILOG namely JRules and ILOG Rules and how they are used in Business rule management.
Continue Reading »

Instead of developing separate software to provide XSLT and XPath functionality to documents encoded in JSON we decided to extend our existing XML suite of tools (i.e., XJR) to accommodate more than XML documents. Once we had this ability we took one more step and applied this capability to another XPath enabled software in our product list (namely our XPath enabled Rule engine).

P6R’s suite of XML tools implement the XPath 2.0 and XSLT 2.0 standards. Excellent references to this functionality are: (1) M.Kay, XPath 2.0, Programmer’s Reference, 2004, Wiley Publishing, ISBN 0-7645-6910-4, and (2) M,Kay, XSLT 2.0, 3rd edition, Programmer’s Reference, Wiley Publishing, ISBN 0-7645-6909-0. The 2.0 versions of these standards represent a significant increase in functionality from their original versions (e.g., both have embedded regular expressions).

All of P6R’s products are written in C++. In addition to the XSLT and XPath products P6R’s XML suite consist of a SAX2-like XML parser, and a DOM XML tree parser which holds an XML document in a tree of nodes. For a reference on SAX2 see: D.Brownell, SAX2, O’Reilly, 2002, ISBN 0-596-00237-8. All of these components work together in the following way:

Continue Reading »

This tutorial discusses what feature branches are and how to implement and manage them using Subversion and svnmerge.py. If you’ve made it here, I’m going to assume that you are already interested in using feature branches, so I’ll skip the sales pitch for the most part.

Feature branches offer a very stable development model where the trunk is always buildable and stable. Features are developed on private branches, and only when they are complete and working are they merged to the trunk. This model allows developers to commit “early and often” without fear of breaking the trunk and stalling other developers. Also, changes can easily be reviewed if needed as check-ins are made on the feature branch.

Note that this tutorial assumes that you are using the standard Subversion “branches, tags, trunk” repository layout. If not, you’ll have to translate the paths to your repository layout, but the concepts illustrated here still hold true. All the examples provided use the subversion command line and svnmerge.py.

The layout of the repository that is used in this example looks like the following:

Three steps for initializing bi-directional merging

1. Create your feature branch
2. Initialize merge tracking to the branch
3. Initialize merge tracking to the trunk

Continue Reading »