First, here is the XSLT template that generates JSON:

<?xml version='1.0' encoding='ISO-8859-1'?>
<xsl:stylesheet version='2.0' xmlns:xsl='http://www.w3.org/1999/XSL/Transform'>
<xsl:output method='json'/>
<xsl:variable name='gtlist' select='/staff/employee/name'/>
<xsl:template match='/'>
{{ "employee":
<xsl:text>  [</xsl:text>
<xsl:for-each select='$gtlist'>
        <xsl:sort select='.' />
"<xsl:value-of select='.' />"
<xsl:if test='position() != last()'>,\n  </xsl:if>
</xsl:for-each>
<xsl:text>]\n}}</xsl:text>
</xsl:template>
</xsl:stylesheet>

Notice the necessity of using the double “{{” and “}}” characters because curly brakets are meaningful in XSLT (e.g., attribute value templates). The “{{” produces a single “{” in the output, and the “}}” character sequence produces a single “}” in the output. Also of interest is that our style sheet is using XSLT sorting.

Second, we have two forms of the ’source’ document that we apply to the template above to produce a JSON output document. We have encoded the same information in XML and JSON. Here is the XML form of the source document:

<?xml version='1.0' encoding='UTF-8' ?>
<staff>
<employee>
   <name>Henry Joseph</name>
   <hiredate>2007-01-09</hiredate>
</employee>
<employee>
   <name>Irvin Joseph</name>
   <hiredate>2003-06-12-03:05</hiredate>
</employee>
<employee>
   <name>Izzy Joseph</name>
   <hiredate>1998-11-03</hiredate>
</employee>
<employee>
   <name>Basil Joseph</name>
   <hiredate>1998-11-03</hiredate>
</employee>
<employee>
   <name>Mark Joseph</name>
   <hiredate>2004-07-12-08:00</hiredate>
</employee>
<employee>
   <name>Jim Susoy</name>
   <hiredate>2004-07-12-08:00</hiredate>
</employee>
</staff>

And here is the JSON form of the source document:

{ "staff": {
     "employee": [
        {"name":"Henry Joseph",
           "hiredate":"2007-01-09"},
        {"name":"Irvin Joseph",
          "hiredate":"2003-06-12-03:00"},
        {"name":"Izzy Joseph",
         "hiredate":"1998-11-03:00"},
        {"name":"Basil Joseph",
          "hiredate":"1998-11-03"},
        {"name":"Mark Joseph",
         "hiredate":"2004-07-12-08:00"},
        {"name":"Jim Susoy",
        "hiredate":"2004-07-12-08:00"}  ]
     } }

The result document when running the XSLT template against either of the source documents above is the same sorted list:

{"employee":
  ["Basil Joseph",
   "Henry Joseph",
   "Irvin Joseph",
   "Izzy Joseph",
   "Jim Susoy",
   "Mark Joseph"]
}

(However, as defined in our XSLT and XPath for JSON article when the source document is in JSON the XPath path expressions have to be changed slightly. So in the XSLT template above the XPath path expression ” select=’/staff/employee/name’ ” is changed to ” select=’/JSON-document/staff/employee/name’ “. See the above mentioned article for an explanation.)

Now the cool thing to notice with our JSON-enabled XSLT processor is that the following transformations are possible: JSON to any of the forms { JSON, XML, Text, HTML, XHTML }, and XML to any of the forms { JSON, XML, Text, HTML, XHTML }. (A JSON to JSON transformation refers to taking one JSON document and re-arranging / adding / removing elements to form a new JSON document.)

DOM Object Support

Once a source document (either in XML or JSON) is parsed into one of our DOM objects, the developer has the option of three DOM API functions that support document output. First, the developer can select to output the entire DOM tree via the function:

p6IDOMXML::output( p6IDataStream* pStream, P6DOMOUTPUT flags, ... );

Or the developer can output any sub tree rooted from a node object that is currently selected, via the methods:

For XML output
p6IXMLNode::outputSubTree( p6IDataStream* pOutput, P6DOMCHARSET charSet );
Or for JSON output
p6IXMLNode::outputSubTreeInJSON( p6IDataStream* pOutput, P6DOMCHARSET charSet );

(Note, a p6IDataStream pointer can be to a file, a network socket, etc.) Here a node in the DOM tree is selected either from evaluating an XPath expression or via walking the tree via the native DOM API.

Our DOM API allows the same DOM tree to be output in either XML or JSON format no matter what its original form was. Of course, since JSON does not have ‘element’ attributes any attributes from an XML source will not appear in a JSON output document.

When a developer wishes to use the P6R rule engine (reference An XPath Enabled Rule Engine), he can simply create a single instance of the rule engine component, load it with a set of rules (defined in XML), load the stated facts (in XML or JSON), and generate inferred facts by calling the evaluate() method one or more times. This particular use case will be fine for many applications, but what about applications that need to evaluate the same rule set concurrently (e.g., adserving)?

The approach we have taken to support multiple, concurrently, evaluating rule engine instances is that of “Compiled Rule Sets”. (This is similar to the approach we have taken to improve performance for XSLT, described in the article: Compiled Templates in XSLT 2.0.)

2. Compiled Rule Sets as objects

The P6R rule engine (p6IRuleEngine) can be used in a simple mode of execution where a set of rules are first loaded (via the modifyRules() method) and then evaluated to generate inferred facts (via the evaluate() method). For some applications this mode of execution will be acceptable.

However, for high performance applications the same set of rules will need to be executed concurrently by several threads. Compiling the same set of rules over and over again for each thread is slow and wasteful, thus we support compiling a rule set into a data structure that is a P6R component on its own. Two additional interfaces support this complex mode of execution. First, the p6IRuleSets interface allows the caller to extract out (as well as to load in) a compiled set of rules from a running rule engine instance. Second, the compiled rules are represented by the p6IRuleCompiled interface (which allows the compiled rules to be marked with a name for caching purposes). The compiled set of rules contains almost no execution state and thus can be shared with multiple instances of the p6IRuleEngine component at the same time.

A compiled rule set contains all XML elements (see reference above for a description of the XML language defining rules) in a tree of nodes along with compiled, embedded XPath expressions, and compiled forms of any regular expressions that might appear in XPath expressions (i.e., the Non-deterministic Finite State Automation that represents the regex is its compiled form). Only compiled regular expressions contain run-time state in a compiled rule set and each is protected with its own lock. During run time all other rule engine execution state (e.g., global variables, facts tree) are all associated with the instance of the rule engine component and not with the compiled rule set object.

The simple mode of execution involves the following sequence of statements:

p6IDataStream* pOutStream;  // output results to a stream (a file or socket)
p6IRuleEngine* pRules;  // instance of the rule engine

// call class factory to get a rule engine instance
p6CreateInstance(... IID_p6IRuleEngine, &pRules );
. . . .
// first compile a set of rules
pRules->modifyRules( <buffer containing XML rules> );

// facts can be streamed in via one or more calls
pRules->startFacts( <buffer containing XML or JSON stated facts> );

// generate inferred facts by evaluating all the relevant rules over the given stated facts
pRules->evaluate( <rule set name to start eval> );

// output the results
pRules->outputInferredFacts( pOutStream );
pRules->release();

In the complex mode of execution, one or more “compiled” rule set objects are generated by performing the following sequence of statements:

p6IDataStream* pOutStream;  // output results or error messages
p6IRuleEngine* pRules;  // instance of the rule engine
p6IRuleEngine* pAnotherRules;
p6IRuleEngine* pThirdRules;
p6IRuleSets* pSets1; // compiled rule set interface
p6IRuleSets* pSets2;
p6IRuleSets* pSets3;
p6IRuleCompiled* pCompiled; // compiled rule set object

// call class factory to get rule engine instances
p6CreateInstance(... IID_p6IRuleEngine, &pRules );
p6CreateInstance(... IID_p6IRuleEngine, &pAnotherRules );
p6CreateInstance(... IID_p6IRuleEngine, &pThirdRules );

// get interfaces that support compiled rule set objects
pRules->queryInterface( p6IRuleSets, &pSets1 );
pAnotherRules->queryInterface( p6IRuleSets, &pSets2 );
pThirdRules->queryInterface( p6IRuleSets, &pSets3 );

// extract the compiled rule sets from a running rule engine instance
pRules->modifyRules( <buffer containing XML rules> );
pSets1->getRules( &pCompiled1 );
. . .
// load the compiled rule sets into other instances of the rule engine for execution
pSets2->setRules( pCompiled );
pAnotherRules->startFacts( <buffer containing XML or JSON stated facts> );
pAnotherRules->evaluate( <rule set name to start eval> );
pAnotherRules->outputInferredFacts( pOutStream );  // a separate set of inferred fact results

pSets3->setRules( pCompiled );
pThirdRules->startFacts( <buffer containing XML or JSON stated facts> );
pThirdRules->evaluate( <rule set name to start eval> );
pCompiled1->release();
. . .

Now what happens if any one of the threads sharing the same compiled rules object invoke the “modifyRules” method with a group of commands that add/delete/modify the existing rule set? The p6IRuleCompiled object is protected by a P6R reader/writer lock, thus ensuring that the rules are modified in a consistent way. Just to be clear, if one of the threads modify the shared rule set, then the rules change for all the using threads. When a compiled set of rules is no longer needed the caller invokes the standard “release()” method on the component to free it.

3. Caching the stated facts

P6R’s rule engine has two main interfaces to load the stated facts. The first interface allows the facts to be streamed in (as XML or JSON) via repeaded calls to the methods: p6IRuleEngine::startFacts() and p6IRuleEngine::continueFacts() methods. The second interface requires that the caller first parses the facts document into a DOM tree and then loads that DOM tree directly into the rule engine via the p6IRuleEngine::startFactsWithDOM() method. Clearly once the facts document has been parsed into a P6R DOM tree component (i.e., p6IDOMXML interface),
it can be cached in memory and used over and over again without repeaded parsing thus allowing for faster execution.

4. Summary

P6R’s compiled rule sets are aimed at providing increase performance for highly threaded applicaitons.
With our rule engine a developer can either use our compiled rules or just use the rule engine in a simple mode of operation with a easy to use API. Our goal is to provide a flexible and powerful rule engine solution to all developers.

Fundamentally, a COM component (or component-object, or COM object) is just a reusable piece of code and data in binary form that has the following 4 attributes:

• Encapsulation - Components must hide the details of how they are implemented.
• Interfaces - Only pointers to interfaces are used to interact with them.
• Language independence - Components written in one language can connect to and use components written in another language.
• Dynamic linking - Components must link dynamically.

Encapsulation

Encapsulation prevents the details of how a component is implemented from leaking out. One of the benefits of COM is the ability to load and unload components at runtime which potentially means unloading one component and replacing it with another. If the first component were to expose any details of its implementation, then code which uses that component could make assumptions based on that. Those assumptions may not hold true for the next component whose implementation is different and could result in undefined behavior or a crash. Therefore the implementation details must be hidden and interfaces help do that.

Interfaces

COM components use interfaces to define the interaction between the client and the component object. Interfaces help provide encapsulation and are the sole method of interacting with a component object. COM interfaces are nothing more than a the definition of a specific memory structure that contains an array of function pointers. It’s a binary definition which is all that is needed to interact with the component. Everything else is unnecessary.

Language Independence

Component objects can be implemented in any language that can be made to create and use the array of function pointers. Given a COM interface, any component written on one language can use a component written in another language. However, the component object MUST hide the all of the details of the language that it is implemented in. Exposing the implementation language would create dependencies between the component and the client. Those dependencies could prevent it’s use by clients that were not implemented using the same language.

For example, given the following:
interface IFoo : p6ICom { P6COMMETHOD someMethod(mysmartptr *ptr) = 0 }

Above we see that someMethod() takes an argument which is some sort of C++ smart pointer. It’s a language specific complex type which breaks encapsulation. As a result, any client that uses this interface to access a component object will have to have knowledge of this language specific implementation. Most likely, clients written in another language will not be able to use this component.

Dynamic Linking

Dynamic linking is the foundation of several of COM’s most powerful features. COM introduced an architecture that moved us away from huge, unchanging monolithic applications. Without dynamic linking your components would need to be statically linked together, which is not different than having a monolithic application. When a component changes, dynamic linking (and encapsulation through interfaces) allows a new component to be dropped in at runtime. Without it, you would have to re-link and possibly recompile in order to implement your changes.

One common concern with the use of XSLT has been performance. The need to compile an XSLT template, with its embedded XPath expressions each time a Web page is to be displayed is wasteful. Part of P6R’s implementation of the XSLT 2.0 standard includes an extension that helps address this performance problem. Our XLST processor compiles an XSLT template into an essentially static data structure that can be used over and over again by multiple threads at the same time. This is achieved by removing most of the dynamic, run-time state out of the compiled templates. This article describes our approach in detail.

2. Compiled XSLT templates as objects

P6R’s XSLT processor (p6IXSLT) can be used in a simple mode of execution where a template is first compiled (via the compileTemplates() method) and then used in the transformation of a source document. This is a typical run-time model for XSLT processors and for some applications this mode of execution will be acceptable.

However, for high performance applications the same template will need to be executed concurrently by several threads.   Two XSLT interfaces support this complex mode of execution. First, the p6IXSLTTemplate interface allows the caller to extract a compiled template object from a running XSLT processor instance. Second, the compiled template object is represented by the p6IXSLTCompiled interface (which allows the template to be marked with a name for caching purposes).  The compiled template contains almost no execution state and thus can be shared with multiple instances of XSLT processors at the same time.

A compiled template contains all XSLT elements in a tree of nodes along with compiled, embedded XPath expressions, and compiled forms of any regular expressions that might appear in both XPath expressions and XSLT elements (i.e., the Non-deterministic Finite State Automation that represents the regex is its compiled form).   Only compiled regular expressions contain run-time state in a compiled XSLT template and each is protected with its own lock.   During run time all other XSLT execution state (e.g., global variables, keys, result document, source document) are all associated with the instance of the XSLT processor object and not with the compiled XSLT template object.

The simple mode of execution involves the following sequence of statements:

p6IDataStream*  pOutStream;  // output results to a stream (a file or socket)
p6IXSLT* pXSLT;  // instance of the XSLT processor

// call class factory to get an XSLT processor instance
p6CreateInstance(... IID_p6IXSLT, &pXSLT );
. . . .
// first compile an XSLT template
pXSLT->compileTemplates( <buffer containing XSLT template>, pOutStream );

// transform the source document, place output in pOutStream
pXSLT->startTransform( pOutStream, <buffer with XML (or JSON) data as source document> );

In the complex mode of execution, one or more “compiled” template objects are generated by performing the following sequence of statements:

p6IDataStream* pOutStream;  // output results or error messages
p6IXSLT* pXSLT;  // instance of the XSLT processor
p6IXSLT* pAnotherXSLT;
p6IXSLT* pThirdXSLT;
p6IXSLTTemplate* pTemplate1;  // compiled template interface
p6IXSLTTemplate* pTemplate2;
p6IXSLTTemplate* pTemplate3;
p6IXSLTCompiled* pCompiled1; // compiled template object
p6IXSLTCompiled* pCompiledN;

// call class factory to get XSLT processor instances
p6CreateInstance(... IID_p6IXSLT, &pXSLT );
p6CreateInstance(... IID_p6IXSLT, &pAnotherXSLT );
p6CreateInstance(... IID_p6IXSLT, &pThirdXSLT );

// get interfaces that support compiled templates
pXSLT->queryInterface( p6IXSLTTemplate, &pTemplate1 );
pAnotherXSLT->queryInterface( p6IXSLTTemplate, &pTemplate2 );
pThirdXSLT->queryInterface( p6IXSLTTemplate, &pTemplate3 );

// extract the compiled template from the XSLT processor instance
pXSLT->compileTemplates( <buffer containing XSLT template>, pOutStream  );
pTemplate1->getTemplate( &pCompiled1 );
. . .
pXSLT->compileTemplates( <buffer containing another XSLT template>, pOutStream );
pTemplate1->getTemplate( &pCompiledN );
. . .
// load the compiled template into other instances of the XSLT processor for execution
pTemplate2->setTemplate( pCompiled1 );
pAnotherXSLT->startTransform( pOutStream, <source document in XML or JSON> );
pTemplate3->setTemplate( pCompiled1 );
pThirdXSLT->startTransform( pOutStream3, <source document in XML or JSON> );
pCompiled1->release();
. . .

An application can generate a compiled template for each output “page” it needs to generate.   Each compiled template (i.e., a p6IXSLTCompiled interface) can then be placed into a RAM cache and used over and over again in several transformations.

Next in the complex mode of execution, the compiled templates can be run concurrently across several instances of the XSLT processor.   The p6IXSLTTemplate interface also allows a compiled template to be “loaded” into any instance of an XSLT processor for execution.   In fact, the same compiled template could be loaded to any number of XSLT processors at the same time, where each XSLT processor executes with its own thread.

When a compiled template is no longer needed the caller simply invokes the “release()” method on the object to free it.   All P6R objects are reference counted, so when the last holder of a compiled template releases it the template will free itself.

3. Application defined variables and functions

XPath is described as an “embedded” expression language.   That means that it typically does not live on its own but is part of another parent application, such as XSLT.  When XPath detects a variable or function which it does not recognize it calls out to its parent application to ask it to resolve the reference.   P6R has implemented this in a very flexible and powerful way.   Our XPath processor contains a method that allows its parent application to register a special “connector” object.   This connector object has two methods: lookupvaraible() and executeExternalFunction().   The connector object can be reset at any time between XPath expression evaluations.   So for example, in an XSLT style sheet all references to XSLT variables that appear in a “select” attribute will force the XPath processor to call to its XSLT parent to obtain the variables’ value (by calling a registered connector’s lookupvariable() method).

This connector object design gets extended one step further.  P6R’s XSLT processor allows its calling application to also register its own connector object.   The key point here is that all connector objects are chained.   So as an example, if the XPath processor detects a variable it does not recognize it calls out to its XSLT parent.   However, if the XSLT processor also does not recognize the variable it will then call out to its parent (i.e., the calling application) via the second connector object.   The same thing occurs if the unrecognized reference is to a function rather than to a variable.

This chaining of connector objects allows an application running an XSLT processor to define its own global variables and external functions.    These global variables are different from normal XSLT variables in that they can change their values at any time since they are under control of the calling application.  (Note that the connector interface passes back and forth only XPath supported types.)

This feature by itself is a powerful P6R extension, but it also supports compiled templates as discussed in the next section.

4. Parametrized compiled templates

P6R’s XSLT processor has restricted support for Attribute Value Templates because of our compiled templates feature.  Many XSLT elements allow some of their attributes to be passed in as part of the “environment” before template compilation.   This way an XSLT element can be parametrized.  For example, the XSLT sort element allows many of its attributes (e.g., order = { ascending | descending },  and collation = { uri }) to be determined at compile time via parameters to the compile/transform call.

The problem with this type of value expansion is that it can only happen once with compiled templates.   Using the XSLT sort element as an example, the same template would have to be compiled with both values for the “order” attribute just in case that attribute could be set by a user at a Web UI.   Instead of using these “hard coded” values we allow the calling application to use the XSLT processor’s connector objects described in the previous section above.  These application controlled variables can change at any time and can be defined differently for each instance of a running XSLT processor.  (For example, different values can be returned by a variable based on the type of operating system that is running.)

Again using the sort element for an example, we would change all appearances of the “collation” attribute to reference a global variable that is defined by a connector object.  If the sort element is being used to generate an HTML page per HTTP request, then the collation value returned could be based on the browser’s value returned in the “Accept-language” header.  The key point is that now these once hard coded values can take on dynamic values thus allowing the same compiled template to be used for a wider range of requests.

5. Caching the source document

P6R’s XSLT processor has two main interfaces to perform a transformation.   The first interface takes the XML (or JSON) source data as a stream of data.   Once the stream is read the XML (or JSON) is compiled into a DOM tree.   If most or all of the source document is static then repeated parsing of the source is again a waste of time.   Thus the second interface to perform a transformation takes a P6R DOM tree as input (i.e., p6IDOMXML interface).

All or parts of a source document can be cached in RAM and used over and over again with compiled templates to improve processing speed.   Now what if part of the source document needs to change?   Well we support that requirement by provided a DOM tree merge function.   Given DOM trees A and B, this feature allows DOM tree B to be grafted onto any part of DOM tree A.   So for example, DOM tree A could be a cached part of a source document while DOM tree B could a dynamic portion of a page to be generated.  (Later DOM tree B can be removed from tree A as well.)

6. Summary

P6R’s XSLT extensions provide enhanced functionality (e.g., the connector objects described in Section 3) and address known performance issues.  With our XSLT processor a developer can either use our extensions or just use the processor in a simple mode of operation with a easy to use API.   Our goal is to provide a flexible and powerful XSLT 2.0 solution to all developers.

P6R’s JSON parser provides a C++ implementation of the SAX like interface. Our parser implementation is designed to be high performance and to directly support a streaming IO model. The parser can be invoked with the entire JSON document in one buffer, or the JSON document feed into the parser a chunk at a time over multiple calls. This interface allows chaining of components (e.g., filters, sources and sinks). To assist a developer in debugging a detailed parse trace can be turned on programmatically.

Ultimately, our goal was two fold. First to provide a generic JSON parser that applications could use directly. And secondly, to support JSON data in our XPath 2.0 and XSLT 2.0 products. How the second goal was achieved is described in another P6R article: XSLT and XPath for JSON. We should note that our SAX-like JSON parser is fully implemented and is part of our P6Platform product.

P6R’s SAX2 parser and JSON parser reduce the amount of string copying to help improve performance. In the Java definition ‘String’ objects are frequently passed to the application. However, all this object creation and string copying comes at a cost. We have taken a different approach, we return pointers and a length to parsed strings. These pointers point into the application provided buffer that contains the document to be parsed. Pointers to application provided buffers are defined by our P6JSONSTRING type:

typedef struct
{
    const P6CHAR* pStart;
    P6UINT32      length;
} P6JSONSTRING;

P6JSONSTRING values are only valid during a callback into an application written content handler. An application that wants to keep a copy of the string must make a copy during a callback. This way an application has total control of how it manages its own memory and related performance concerns.

References:
(1) JSON Home Page, (2) RFC 4627, The application/json Media Type for JavaScript Object Notation (JSON), (3) D.Browneww, SAX2, O/Reilly, 2002, ISBN 0-596-00237-8.

P6IJSONReader Interface Reference

Detailed Description

This is the main JSON interface. Notice that there is no separate parse function, because parsing is done through the p6IDataStream interface.

To parse either a single buffer or a stream of JSON buffers perform the following steps:
First get an JSON reader object:
p6IJSONReader *pReader;

err = p6CreateInstance( CID_p6JSONReader, IID_p6IJSONReader, &pReader );

Second, using the JSON reader, get the p6IDataStream interface on that object:
p6IDataStream *pStream;

err = pReader->queryInterface( p6IDataStream, &pStream );

Third, initialize the data stream interface: err = pStream->beginStream();

Pass the buffer(s) to be parsed one at a time:
err = pStream->processStream( buffer, bufSize ); (1st buffer of stream)
. . . . . .
err = pStream->processStream( buffer, bufSize ); (nth buffer of stream)

The input ‘buffer’ to the processStream() method is where the P6JSONSTRING pointers will often point to. The processStream() function can return an “eEndOfFile” error code to indicate that it is done with the buffer provided and that the buffer is incomplete (i.e., the JSON top most object has not yet been closed).

Lastly, close the stream down: err = pStream->endStream();

Public Member Functions

1. P6R::P6ERR initialize( P6JSONFLAGS flags ): Sets up the interface to run properly.

2. P6R::P6ERR getContentHandler( p6IJSONContentHandler **pObject ): Returns the content handler object defined and set by the application.

3. P6R::P6ERR getErrorHandler( p6IJSONErrorHandler **pObject ): Returns the Error handler object defined and set by the application.

4. P6R::P6ERR setContentHandler( p6IJSONContentHandler *pObject ): The calling application uses this method to register a content handler. That handler is called directly by the JSON parser as tokens from the JSON document are recognized.

5. P6R::P6ERR setErrorHandler( p6IJSONErrorHandler *pObject ): The calling application uses this method to register an error event handler.

6. P6R::P6ERR releaseAllHandlers (): This was addded to make it easy for an application to tell the JSON reader to free all set handlers.

P6IJSONContentHandler Interface Reference

Detailed Description

This JSON interface is implemented by an application using P6R’s JSON parser. This is a callback object that is registered with the p6IJSONReader::setContentHandler() method. This is the main parsing interface to the application. It provides a stream of events for each item in the input JSON document. Each event results in a method call to one of the methods below.

Public Member Functions

1. P6R::P6ERR endDocument(): Provides an application the notification when the end of the JSON input document has been reached by the parser.

2. P6R::P6ERR endObject( P6UINT32 nestingLevel ): Provides an application the notification when the most recent object (as defined by a callback of the startObject method below) has ended. Since JSON objects can nest, the ‘nestingLevel’ parameter is provided so an application can match the proper startObject() and endObject() calls.

3. P6R::P6ERR endArray ( P6UINT32 nestingLevel ): Provides an application the notification when the most recent array (as defined by a callback of the startArray method below) has ended. Since JSON arrays can nest, the ‘nestingLevel’ parameter is provided so an application can match the proper startArray() and endArray() calls.

4. P6R::P6ERR setDocumentLocator( p6IJSONLocator *pObject ): This method is called just before the startDocument() method and the application is given a p6IJSONLocator object created by the JSON parser.

5. P6R::P6ERR startDocument(): Provides an application the notification when the start of the JSON input document has been reached by the parser.

6. P6R::P6ERR startObject( P6UINT32 nestingLevel ): Provides an application the notification when a new JSON object has been detected.

7. P6R::P6ERR startPair( P6JSONSTRING *pName ): A JSON object consists of an unordered set of name/value pairs.

8. P6R::P6ERR startArray( P6UINT32 nestingLevel ): Provides an application the notification when a new JSON array has been detected.

9. P6R::P6ERR value( P6JSONVALUE *pValue): Both JSON objects and arrays contain values which can be strings, boolean values, numeric values, nested objects, nested arrays, and the value “null”.

The “P6R::P6ERR value(P6JSONVALUE *pValue)” method is designed after the SAX2 characters(…) method. We extended this concept in JSON so that we could return parsed out values from a JSON string.

P6JSON_TYPE can be one of the following values:
P6JSON_TYPE_NULL -- a JSON 'null' value
P6JSON_TYPE_STR  -- a complete string has been returned in one chunk
P6JSON_TYPE_STRPART -- part of a string has been returned supporting a streaming interface
P6JSON_TYPE_STREND -- end of a string has been returned
P6JSON_TYPE_BOOL -- { P6TRUE, P6FALSE }
P6JSON_TYPE_INTEGER
P6JSON_TYPE_REAL 

struct jsonValueParts
{
    P6JSON_TYPE  type;         // which type is used?
    P6JSONSTRING jstring;      // offset into buffer where string was found
    P6INT32      integer;      // P6JSON_TYPE_INTEGER
    P6FLOAT      real;         // P6JSON_TYPE_REAL
    P6BOOL       boolean;      // P6JSON_TYPE_BOOL
};
typedef struct jsonValueParts P6JSONVALUE;

P6IJSONLocator Interface Reference

Detailed Description

An instance of this interface is passed to the application in 2 ways: a) via the setDocumentLocator() method of the p6IJSONContentHandler interface implemented by an application, and registered with the JSON Reader component (p6IJSONReader). and b) Via the warning(), error(), and fatalError() methods of the p6IJSONErrorHandler interface implemented by an application and registered with the JSON Reader component.

Public Member Functions

1. P6R::P6ERR getColumnNumber( P6INT32 *pNumber ): The application making use of the JSON Reader can use this interface to find out where in the JSON document the parser is currently processing.

2. P6R::P6ERR getLineNumber( P6INT32 *pNumber ): Return the current line number, in the JSON document being parsed, where the JSON parser is currently pointing to. This function is typically used to pinpoint parsing errors.

3. P6R::P6ERR convertToWideString( P6JSONSTRING *pJstring, P6WCHAR *pOut, P6UINT32 *pLength ): (A Helper function.) JSON value strings can be encoded with hex digits in the form “\uhhhh” and have control characters encoded in strings (e.g., tab as “\t”). This function translates all encodings into their wide character representation.

P6IJSONErrorHandler Interface Reference

Detailed Description

This JSON interface is implemented by an application using P6R’s JSON parser. This is a callback object that is registered with the p6IJSONReader::setErrorHandler() method. Once registered the JSON parser will call one of the 3 methods of this object to notify the application of a parsing error. The application can take the error information and write it to a file or to a socket or etc. P6R provides a default implementation of this interface if none is set by the application.

Public Member Functions

1. P6R::P6ERR warning( P6ERR errorCode, p6IJSONLocator *pObject ) OR
   P6R::P6ERR warningEx ( P6ERR errorCode, p6IJSONLocator *pObject, const P6CHAR *pDisplay ): The JSON parser notifies the application of an unusual condition detected during parsing.

2. P6R::P6ERR error ( P6ERR errorCode, p6IJSONLocator *pObject ) OR
   P6R::P6ERR errorEx( P6ERR errorCode, p6IJSONLocator *pObject, const P6CHAR *pDisplay ): An application can typically proceed when receiving a warning. However, on receipt of an error or fatal parsing error the application should stop parsing the document.

3. P6R::P6ERR fatalError( P6ERR errorCode, p6IJSONLocator *pObject ) OR
   P6R::P6ERR fatalErrorEx( P6ERR errorCode, p6IJSONLocator *pObject, const P6CHAR *pDisplay ):The JSON parser notifies the application that a non-recoverable parsing error has been detected.

A Simple Example

Given the very simple JSON document:

{ "menuitem": "File",
   "offsets": [ 44, 99 ]
}

The following sequence of callbacks would happen in the application written instance of the
P6IJSONContentHandler interface:

 startDocument()
 startObject( 1 ) -  {
 startPair()     -  menuitem
 value()         -  File  (P6JSON_TYPE_STR)
 startPair()     -  offsets  (P6JSON_TYPE_STR)
 startArray( 1 ) -  [
 value()         -  44  (P6JSON_TYPE_INTEGER)
 value()         -  99  (P6JSON_TYPE_INTEGER)
 endArray( 1 )   -  ]
 endObject( 1 )  -  }
 endDocument()

Now if we extended this example slightly so that the first string value returned from the JSON parser is broken into two pieces (i.e., each string chunk appeared in a different input buffer) the sequence of callbacks would change to the following:

{ "menuitem": "File with the name henry-the-great-dane.txt",
   "offsets": [ 44, 99 ]
}

 startDocument()
 startObject( 1 ) -  {
 startPair()      -  menuitem
 value()          -  'File with the name '  (P6JSON_TYPE_STRPART)
 value()          -  'henry-the-great-dane.ext'  (P6JSON_TYPE_STREND)
 startPair()      -  offsets  (P6JSON_TYPE_STR)
 startArray( 1 )  -  [
 value()          -  44  (P6JSON_TYPE_INTEGER)
 value()          -  99  (P6JSON_TYPE_INTEGER)
 endArray( 1 )    -  ]
 endObject( 1 )   -  }
 endDocument()

There is a lot of technical jargon used to describe how rule engines work two of which are important to understand the rest of this article. The first term is “stated facts” which are fixed facts that are given as input to the rule evaluation (e.g., the price of a typical car is $22,000). And the second term is “inferred facts” which are facts that are generated by the rule engine (these come from a rule’s actions, e.g., Henry bought an expensive car since it cost more than $22,000).

What follows are details of what makes our rule engine so powerful. While we where designing our rule engine we knew that we need an expression language to define a rule’s conditional logic. Luckily we already had built XPath 2.0 which is a powerful expression language that is embeddable into XML.

Examples of other Rule Engines include open Source Rule Engines in Java (Drools is a great example of this). Another popular rule engine is from ILOG namely JRules and ILOG Rules and how they are used in Business rule management.

Properties of our Rule Engine
1. Our rule engine implements forward chaining where all inferred facts are added to a DOM tree at location under “/P6R:infer/”. The benefit of this is that all inferred facts are accessible via XPath 2.0 expressions.

2. Our XPath 2.0 supports stated facts defined in either XML or JSON , thus our Rule Engine is also JSON enabled.

3. No software programming is required. Our rule engine is controlled by an XML based language we have defined. For example, a basic rule looks like:

<rule name='abc' setname='rover' priority='5'>
   <if test='XPath expression'>
   <then>
      One or more actions (which can also include XPath expressions)
   </then>
</rule>

4. Rules can be added, deleted, modified any time between evaluation calls.

5. The rule engine is easily extensible by any application. This is done via XPath. XPath calls the enclosing application for any function it does not recognize as a predefined XPath function. For example,

<rule name='hotcar' setname='over1'>
   <if test='$g2'/>
   <then>
      <call func="setclickurl( 'www.p6r.com' )" />
      <set-fact name='12345' select="'Hi Henry'" location='/P6R:infer/mealitem' />
   </then>
</rule>

the ’setclickurl’ is not a defined XPath built-in function but is instead defined by the application. Also the variable ‘$g2′ is defined by the application. Our XPath component allows an application to register a special handler object which defines these external functions and variables.

6. Any number of rule sets, rules, and actions in a rule are allowed.

7. Our rule engine has a large set of built-in functionality via a well engineered XML rule language (some of which is presented below and we document in detail in our SDK).

8. Our rule engine has a built-in logging facility. Also XPath 2.0 defines several tracing functions. Both the internal working of the rule engine logic as well as the XPath engine can be traced. For example, XPath 2.0 supports regular expressions. When the XPath log is turned on any regular expression being evaluated is also traced.

9. Rules are compiled so any errors are returned as error codes to the user before the rules are run.

10. Our rule engine is its own object and thus any number of separate rule engine instances can be created for a single or multiple applications. Each instance can be controlled by a different set of rules and can generate a different set of inferred facts.

11. As with all P6R software, our rule engine runs are several platforms which currently include (as of this writting) Windows, Linux, and Solaris.

The following sections are excerpts from our rule engine manual. These small excerpts provide a view into the full expressive power of our engine.

The Rule Engine Language
The basic building block is the rule and it has the following form:

<rule name='abc' setname='rover' priority='5'>
     <if test='XPath expression' >
     <then>
           One or more actions
     </then>
</rule>

Every rule must have an ‘if’ and ‘then’ element. The ‘if’ element defines the conditions when the rule is true and thus can be executed. The ‘then’ element defines one or more actions which are evaluated when a rule is executed.

Rules can be grouped in to rule sets where each set is defined by a unique name. A rule defines which set it belongs do by the use of the optional attribute ’setname’. If no ’setname’ attribute is defined then a rule is placed in the default rule set which is named by ‘#default’. (Note, that the rule set name ‘#default’ is reserved.)

Rules are grouped under a parent element of ‘<rulesets> … </rulesets>’, allowing multiple rules and rule sets to be defined by a single XML document.

Rules can also use the optional attribute of ‘priority’ which defines the relative importance of a rule with all the other rules in the same rule set. If no ‘priority’ attribute is used then a priority of zero (the lowset level) is the default.

In the ‘if’ element, value of the test attribute can be any valid XPath 2.0 expression. These expressions are evaluated against the XML (or JSON) ‘fact’ tree created by a call to the startFacts() API call. This method allows the application to define ’stated’ facts to be used in the evaluation of rules.

When a rules ‘if’ element evaluates to true and it is selected for execution each action defined in its ‘then’ element is executed in order of appearance. There are several possible actions, each is defined in the next section.

The Rule ‘Then’ Element
1) <call func=’XPath Expression’ />
The ‘func’ parameter is required. The func expression should contain a call to a function that is either a built in XPath 2.0 function, or one defined by the calling application via the rule engine setExternalFunctions() method. For example, the XPath expression could be something like “setclick( ‘abc’, $g1 )”, which would be a function defined by the application with two parameters being passed to it. Note, that XPath 2.0 allows any number of parameters to be passed to an application defined function, the parameters to that function must be valid XPath types (e.g., strings, integers, XML node sequences). This action is a good way to call out to application defined functions to execute an external action when a specific condition is satisified (i.e., when the ‘if’ element condition is true).

2) <set-variable name=’$g1′ select=’XPath expression’ />
The ‘name’ and ’select’ parameters are required. This action will first evaluate the XPath expression defined in the select parameter. The result of that expression will be assigned to the global variable designated in the ‘name’ parameter. The rule set that this action is defined in is the rule set used to lookup the global variable (e.g., “rule setname=”example1″ …”, then the ‘example1′ rule set would be used for this action).

3) <set-focus setname=’over’ />
The ’setname’ parameter is required. This action changes the rule set that is currently being evaluated. The change will occur after the current rule is executed. Thus any actions that follow the set-focus action in the same rule will not be affected. If the rule set identified by the setname parameter does not exist then a fatal error will occur and the evaluate() API call will fail and rule execution will stop immediately.

4) <set-fact name=’rover’ select=’XPath expression’ location=’/P6R;infer …’ />
The ‘name’ parameter is required and ’select’ and ‘location’ are optional. This action adds one or more inferred facts onto the inferred fact XML tree under “/P6R:infer”. The value of the ‘name’ parameter defines the XML node to be added so if “name=’rover’” the path to the new inferred fact will be “/P6R:infer/rover”.

However, if the ‘location’ parameter is also defined, for example “location=’/P6R:infer/part1′”, then the new inferred fact will be add at the path “/P6R:infer/part1/rover”, assuming that “part1″ is already in the tree. The ’select’ parameter will evaluate to a valid XPath type value and that will get added to the location in the inferred fact tree. Thus its the value of the select parameter that determines the value of the inferred fact.

The action <set-fact name=’section1′ /> will simply add a new XML node at the path “/P6R:infer/section1″. This is useful for different rule sets to create separate subtrees in the inferred fact tree that is just for that rule set to use. Also the action

“<set-fact name=’paragraph44′ location=’/P6R:infer/section1′ />”,

would create the following nodes in the inferred fact tree “/P6R:infer/section1/paragraph44″. Once this path is created, any number of facts can be “hung off of” any part of that tree (i.e., at the ’section1′ branch or at the ‘paragraph44′ branch).

5) <clear-facts />
This action results in the entire inferred fact tree (i.e., “/P6R:infer”) to be deleted. After this action new inferred facts can be added to a new inferred fact tree. The result of this action is identical to the result of the rule engine API call reset( P6RULE_CLEAR_FACTS ).

6) <delete-fact location=’/P6R:infer …’ />
This action removes all the inferred facts defined in the inferred fact tree located at the path defined by the ‘location’ parameter. After this action, new inferred facts can always be added back onto the tree at the same location that was deleted.

7) <halt />
There are no parameters to this action. This action stops rule engine evaluation immediately. If there are any additional actions that are defined after the ‘halt’ they are ignored. This action results in a successful return of the rule engine’s evaluate() API call (i.e., it does not represent an error condition).

A Simple Example
This example comes directly out of a unit test for our rule engine:

<?xml version='1.0' encoding='ISO-8859-1'?>
<rulesets version='1.0'
   xmlns:P6R='http://www.p6r.com/ruleengine/extensions'>

<variable name='$g1' select='/menu/dessert/item'/>           

-- 'matches' is an XPath 2.0 function implementing Perl Regular Expressions
<rule name='Irvin' prioirty='5'>
   <if test="matches( $g1, '\d\d-\d\d-\d\d', 'i') />
   <then>
     <set-fact name='infer1' select='menu has blueberry tart' />
  </then>
</rule>

<rule name='stoprules'>
<if test='true()'/>
<then>
     <halt/>
</then>
</rule>
</rulesets>

And the stated facts that was input to the above rule set was:
<?xml version='1.0' encoding='UTF-8' ?>
<menu>
   <maincourse>
         <meat>Whole Chiken</meat>
         <soup>green pee</soup>
   </maincourse>
   <dessert id='5'>
       <icecream>chocolate</icecream>
       <item>blueberry 44-34-29 tart</item>
       <item>strawberry 3 layer cake</item>
       <item>vanilla cream pie</item>
   </dessert>
</menu>

Or if your stated facts are in JSON, this input will produce the same result:
 { "menu": {
   "maincourse": {
       "meat": "Whole Chiken",
        "soup": "green pee"
     },
   "dessert": {
        "id": 5,
        "icecream": "chocolate",
        "item": [
             "blueberry 44-34-29 tart",
             "strawberry 3 layer cake",
             "vanilla cream pie"
        ]  } } };

Here are final results when the rules above are evaluated:
<?xml version="1.0" encoding="utf-8" ?>
<P6R:infer xmlns:P6R="http://www.p6r.com/ruleengine">
  <infer1>menu has blueberry tart</infer1>
</P6R:infer>

P6R’s suite of XML tools implement the XPath 2.0 and XSLT 2.0 standards. Excellent references to this functionality are: (1) M.Kay, XPath 2.0, Programmer’s Reference, 2004, Wiley Publishing, ISBN 0-7645-6910-4, and (2) M,Kay, XSLT 2.0, 3rd edition, Programmer’s Reference, Wiley Publishing, ISBN 0-7645-6909-0. The 2.0 versions of these standards represent a significant increase in functionality from their original versions (e.g., both have embedded regular expressions).

All of P6R’s products are written in C++. In addition to the XSLT and XPath products P6R’s XML suite consist of a SAX2-like XML parser, and a DOM XML tree parser which holds an XML document in a tree of nodes. For a reference on SAX2 see: D.Brownell, SAX2, O’Reilly, 2002, ISBN 0-596-00237-8. All of these components work together in the following way:

In figure 1, the XML input document is parsed by the SAX2 parser which sends events to the XML DOM Tree parser. The XML DOM tree has a SAX2 “content handler” that receives the SAX events (e.g., start of element, characters, …). The content handler makes calls to the XML DOM Tree API to create the nodes in the XML Tree. Once the XML document is totally parsed it is available via the XPath 2.0 API and the DOM tree API. It is important to realize that once the data is placed into the DOM tree its original format is basically invisible to the code above.

Next we decided that we wanted a JSON parser and thought that a SAX-like API (i.e., a stream of events such as start of object, and start of array) would be a unique and powerful tool. Such a JSON parser should fit into our XML architecture outlined above. So we built a SAX-like JSON parser and the tool suite became as follows:

In figure 2, the new XML DOM Tree layer is renamed to just be DOM tree. The DOM tree layer is extended to also contain a JSON SAX parser “content handler” that receives the JSON events like “start of array”, and a series of “values”. See the JSON reference: http://www.json.org. This new JSON content handler takes the stream of JSON events and calls the DOM tree API to construct a tree data structure in the same basic way as the XML SAX2 content handler did. The end result is that above the DOM tree layer, the origin of the data (that is either XML or JSON) is mostly invisible (we describe a few exceptions to this below).

An XML document can have namespaces and attributes for its contained elements. See the XML standard reference: http://www.w3.org/TR/REC-xml/. However, its unclear how these concepts map to a JSON document. So in translating a JSON document into a DOM tree we took the approach that there are no attributes in a JSON document. Without some related JSON like “schema” its not possible to infer additional structure. Our goal was to be able to parse and represent as many JSON documents as possible.

An alternative approach could have been to translate a JSON document directly into an XML document and then just run the XML SAX parser against it. This was not chosen for two reasons. First, we wanted to built a native JSON parser anyway and the SAX approach was attractive to us. Second, converting the JSON to XML seems slow and unnecessary since the conversion requires a JSON parser anyway.

At the XPath level, access to an element in a DOM tree, which came from a JSON source, requires a special root path. For example, given the following XML:

<menu>

   <lunch>

       <soup> ... </soup>

       <veggies> ... </veggies>

   </lunch>

</menu>

a typical XPath step expression could be “/menu/lunch/soup”, for the contents of the soup element. However, not all JSON documents have a top level name such as “menu”. For example, the following JSON document is perfectly legal:

{ "soup": "...",

   "veggies: "...",

   "dessert": "..."

}

In the above example, the top most “wrapper” does not have a name associated with it. In order to handle as many JSON documents as possible our implementation of the DOM tree for JSON was to give a name to the top most “{” “}” pair of any JSON document. That name is simply “JSON-document”. And so one XPath step expression to the example above becomes “/JSON-document/soup”. Given another example:

{ "menu": {

    "lunch": {

        "soup": "...",

        "veggies": "..."

    }

}

which is the previous XML example encoded in JSON. Now the similar XPath step expression would be “/JSON-document/menu/lunch/soup”. Other than this special top level path name and the lack of XML element attributes, all the functionality provided by XPATH 2.0 and XSLT 2.0 is available to any JSON encoded document.

XSLT defines several of its own support functions, such as key(), format-date(), document(), etc. The purpose of the “document()” function is to allow an XSLT script to include an external XML document, parse it and return the parsed XML tree back to the rest of the stylesheet. We took the implementation of this function one step further by allowing JSON encoded documents to be included via the document() function. Our implementation detects if the included document is JSON or XML and parses it accordingly. This proper parsing is done by checking the document’s MIME type, or for local files looking for the opening “{” of a JSON document. The cool thing with this feature is that now both XML and JSON documents can freely be mixed in the processing of a single XSLT stylesheet. Any XSLT operation that can be done on an XML document (e.g., xsl:copy) can also be done on an included JSON document.

At this point, we where very happy with the functionality we achieved. However, it became apparent that JSON support would be immediately available to any other software that used our XPath and DOM tree components. P6R’s Rule engine software uses our XPath and DOM tree components because the rules are defined in XML with XPath expressions. For example, a typical rule could look like:

<rule name='yellocar' setname='over1'>
     <if test='$Var1'/>
     <then>
        <set-variable name='$g3' select='/menu/dessert/item'/>
        <call func=\"setclickurl( 'www.p6r.com', $g3 )\" />
        <set-fact name='sunshine' select='15*4' location='/P6R:infer/mealitem' />
     </then>
</rule>

While this rule is defined in XML its input data (or “facts” in rule engine jargon) can be encoded in XML or JSON. And the JSON support comes for free. Without going into details on how the rule engine works the picture of the software layers now becomes:

The main point we are making here is that any software that uses our XPath and DOM tree components are XML and JSON enabled with next to no effort.

Author: Mark K. Joseph, Ph.D.

Feature branches offer a very stable development model where the trunk is always buildable and stable. Features are developed on private branches, and only when they are complete and working are they merged to the trunk. This model allows developers to commit “early and often” without fear of breaking the trunk and stalling other developers. Also, changes can easily be reviewed if needed as check-ins are made on the feature branch.

Note that this tutorial assumes that you are using the standard Subversion “branches, tags, trunk” repository layout. If not, you’ll have to translate the paths to your repository layout, but the concepts illustrated here still hold true. All the examples provided use the subversion command line and svnmerge.py.

The layout of the repository that is used in this example looks like the following:

Three steps for initializing bi-directional merging

1. Create your feature branch
2. Initialize merge tracking to the branch
3. Initialize merge tracking to the trunk

Step 1: Create Your Feature Branch

Creating your feature branch is just like creating any other branch in subversion, you use the subversion copy command. We will branch remotely, so for this operation, no working copy is needed. Here we are branching (copying) the trunk to our private branch.

C:\temp\test>svn copy -m "create my feature branch" https://svn.example.com/svn/test/project1/trunk/ https://svn.example.com/svn/test/project1/branches/private/user1/feature1/ Committed revision 9. C:\temp\test>

The subversion client requires a working copy for just about everything it does, so next we will create a directory to hold the trunk and check it out.

C:\temp\test>mkdir trunk C:\temp\test>cd trunk C:\temp\test>svn co https://svn.example.com/svn/test/project1/trunk . A file1.cpp Checked out revision 9. C:\temp\test\trunk>

We will also check-out our feature branch now as well.

C:\temp\test\trunk>cd .. C:\temp\test>mkdir feature1 C:\temp\test>cd feature1 C:\temp\test\feature1>svn co https://svn.example.com/svn/test/project1/branches/private/user1/feature1 . A trunk A trunk\file1.cpp Checked out revision 9. C:\temp\test\feature1>

In step 1 we have created the feature branch, created local directories to hold both the branches and checked both branches out into the newly created directories. No lets initialize merge tracking.

Step 2: Initialize Merge Tracking To The Branch

In this step we will initialize merge tracking from the trunk to the branch. This will allow the trunk to track changes in the branch. For this operation we need to be in the directory we checked-out the trunk to.

C:\temp\test\feature1>cd ..\trunk C:\temp\test\trunk>svnmerge.py init -f commit.txt https://svn.example.com/svn/test/project1/branches/private/user1/feature1 property 'svnmerge-integrated' set on '.' C:\temp\test\trunk>svn ci -F commit.txt Sending . Committed revision 10. C:\temp\test\trunk>rm commit.txt C:\temp\test\trunk>

The “-f” argument tells svnmerge.py to create a file using the specified name (commit.txt) containing an appropriate check-in comment, and the URL specifies the location of the branch to track (out feature branch in this case). At this point we have merge tracking initialized in one direction. The trunk is tracking changes on the branch. Next we need to initialize merge tracking in the other direction.

Step 3: Initialize Merge Tracking To The Trunk

In this step we will initialize merge tracking from the feature branch to the trunk. This will allow the feature branch to track changes that occur on the trunk and will complete the initialization of bi-directional merging. Initializing merge tracking in this direction is similar to previous step, except we don’t specify a URL for the branch. This causes svnmerge.py to search though the history looking for “copies” to determine where to start tracking changes.

C:\temp\test\trunk>cd ..\feature1 C:\temp\test\feature1>svnmerge.py init -f commit.txt property 'svnmerge-integrated' set on '.' C:\temp\test\feature1>svn ci -F commit.txt Sending . Committed revision 11. C:\temp\test\feature1>

Now we have merge tracking initialized in both directions and can begin merging changes with svnmerge.py.

Setting Up Your Environment

To make things easier, I have a couple of scripts that reduce the amount of typing that is needed when using svnmerge.py. When using svnmerge.py, the URL of the branch that you are merging from must be provided every time. The goal is to be able to type one simple command to merge. I start out by defining two environment variables containing the URLs to the trunk and the private branch.

Under Windows Vista, to add environment variables you can go the control panel and type “environ” in the search bar, then select “Edit environment variables for your account”. Then add the variables under the “users” section. Something like “MYTRUNK=http://…” and “MYBRANCH=http://…”. On Unix’s, you will need to edit your shell’s start up file (eg. .bash_profile for the bash shell) and export the new variables there.

Example lines to add to a .bash_profile:
MYTRUNK=https://svn.example.com/svn/test/project1/trunk/ MYBRANCH=https://svn.example.com/svn/test/project1/branches/private/user1/feature1/ export MYTRUNK MYBRANCH

Next create two batch files (shell scripts on Unix), mergefrombranch.bat and mergefromtrunk.bat.

Contents of mergefrombranch.bat
@svnmerge.py merge -b -S %MYBRANCH%

Contents of mergefromtrunk.bat
@svnmerge.py merge -b -S %MYTRUNK%

The “-b” flag tells svnmerge.py that this is a bidirectional merge so that it knows to filter out reflected revisions from the merge candidates. Without it, you will see reflected revisions (svnmerge’s own commits) and may have other problems as a result. The “-S” flag tells svnmerge.py what URL to use as the source of the merge.

Keeping your feature branch up to date

When using feature branches, all your changes happen on your own private branch. Others are making changes on their private branches and when those changes are ready, they merge them to the trunk. To minimize the amount of work required to merge, you might want to merge trunk changes onto your private branch periodically. I recommend once a week or more often, but you will have to determine what interval is best for your situation. This will help reduce the number of changes in any given merge to a manageable number, making the merge easier and less risky. The longer you wait, the bigger the delta between your branch and the trunk, which will result in more risk that something will conflict or otherwise go wrong.

Merging Changes from the trunk to your branch

This is something that you should do often and the mergefromtrunk.bat script makes it easy. It may be a good idea to see what changes are available in the trunk before you actually merge. To do this, use the following command from a DOS window (or the equivalent in a Unix shell):

Check for available revisions to merge
c:\temp\test\feature1>svnmerge.py avail -b -S %MYTRUNK% 11 c:\temp\test\feature1>

This shows that there is one revision available (revision 11). If there are more, they will all be listed with contiguous revisions being represented by ranges (1-9 11-15 etc). Next we’ll merge revision 11 into our feature branch using the mergefromtrunk.bat script.

Merge revision from the trunk to the feature branch
c:\temp\test\feature1>mergefromtrunk.bat property 'svnmerge-integrated' deleted from '.'. property 'svnmerge-blocked' deleted from '.'. U file1.cpp property 'svnmerge-integrated' set on '.' property 'svnmerge-blocked' deleted from '.'. C:\temp\test\feature1>

You can see that one file (file1.cpp) was updated and several properties which svnmerge.py uses to track revisions were updated. Check to make sure there are no conflicts listed for any files in the output. If there are, you will need to resolve those before proceeding. Next, compile the code if needed, make sure that unit tests pass (you do use unit tests … right?), whatever you do to verify that you have a working, healthy tree. Once that is done, it is time to check-in the merged changes. Note that svnmerge.py also created a file named “svnmerge-commit-message.txt” which contains the check-in comments for all the revisions being merged. This should be used as the check-in comment when we commit the merged changes.

Commit the merged changes to the feature branch
C:\temp\test\feature1>svn ci -F svnmerge-commit-message.txt Sending . Sending file1.cpp Transmitting file data . Committed revision 12. C:\temp\test\feature1>del svnmerge-commit-message.txt C:\temp\test\feature1>

In this example the contents of svnmerge-commit-message.txt look like this:
Merged revisions 8,10-11 via svnmerge from https://svn.example.com/svn/test/project1/trunk ........ r11 | jss | 2008-01-17 22:39:39 -0800 (Thu, 17 Jan 2008) | 1 line added a comment line ........

Merging Changes from the feature branch back to the trunk

You typically do this when you are ready to make your changes/features available to others. The same basic operations that we performed to merge in changes from the trunk are used here, but instead of using the mergefromtrunk.bat script, the mergefrombranch.bat script is used instead.

C:\temp\test\trunk>svnmerge.py avail -b -S %MYBRANCH% 13 C:\temp\test\trunk>mergefrombranch.bat property 'svnmerge-integrated' deleted from '.'. property 'svnmerge-blocked' deleted from '.'. U file1.cpp property 'svnmerge-integrated' set on '.' property 'svnmerge-blocked' deleted from '.'. C:\temp\test\trunk>svn ci -F svnmerge-commit-message.txt Sending . Sending file1.cpp Transmitting file data . Committed revision 14. C:\temp\test\trunk>del svnmerge-commit-message.txt C:\temp\test\trunk>

Now you have completed merges in both directions and should be prepared to happily use your new feature branch. Good luck!