Implementing basic Processing Engine extensions in TRM 6

Applications
Applications covers the world of Teradata apps, including apps offered by Teradata (such as TRM or DCM), as well as best practices and methodologies for building your own Teradata-connected apps.
Teradata Employee

Implementing basic Processing Engine extensions in TRM 6

Processing Engine Extension represents mapping of customized processing engine tasks, properties, and dependencies whose purpose is to change or extend the processing flow of base processing engine capabilities defined in the base definition files. The purpose of this functionality is to allow for flexibility while maintaining the core product, or at least the processing engine configurations, unchanged.

The core processing engine capabilities are set at development time. At deployment or install time, the customizations are defined in an XML file called trm-pe-extension-context.xml located in the application classpath (/WEB-INF/classes). This file serves as a central place for definition of all engine customizations. Multiple customizations should be consolidated in this one file.

Note: When making modifications to the XML file, please use a text editor. Opening and copying the XML contents using a browser is not recommended as special characters are added, resulting in a malformed XML document.

Processing Engine Base Configuration

Currently, these are the XML files where engine capabilities are defined at development time:

Definition Files Engine Type Capabilities
communication-manager-pe-context.xml Communication batch and adhoc engines LGE, LGE w/Output, FullBatch, Intraday, Cycle Leads, Apply Responses
seg-pe-context.xml Segmentation engine  Segment Plan Realization, Segment Quick Count, Waterfall Counts, Acquisition Import 
output-pe-context.xml  Output engine  Output Generation 
analysis-pe-context.xml  Analysis engine  Cross Segment, Affinity, BehaviorTrend, PercentileProfiling 
rule-pe-context.xml  Rules Manager engine  Rules Manager 
report-pe-context.xml  Reports engine  Rules Manager Reporting 

Processing Engine Task

A processing engine task is a unit of work. A collection of processing engine tasks makes up a processing engine capability. The core set of tasks are defined at development time in the engine definition files. Each functional processing engine has very specific module-related functionalities and these are carried out at runtime by the processing engine tasks. There are processing engine tasks currently available for common functionalities such as execution of a pre-defined SQL script or OS-specific batch commands. These are covered in details in a later section of this document.

The following terms and their definitions will be used throughout this document:

1. Base or core processing engine tasks

These are the processing engine tasks found in the engine XML configuration file. These are the only tasks that will be executed if there are no extensions or customizations added to the processing engine.

2. Custom tasks

These are the processing engine tasks that are only defined in the extension configuration file. These tasks are executed along with the base processing engine tasks and are specifically configured to alter the engine behavior. A custom task may be:

  • One of the built-in processing engine tasks that implement common functionalities
  • An application component written in Java, developed by the customer

Enable/Disable Processing Engine Extension

Processing Engine extensions are controlled globally by an MD_SETTING flag called PROCESSING_ENGINE_EXT_IND. The default setting is ‘0’ (it’s a character field) or disabled. This allows the administrator or power user to define the extensions and toggle it on and off using the Metadata Admin Utility or a SQL script. A change to this setting requires either a server restart or Metadata Refresh from the main application.

Customizations

Customizations may be added to alter the behavior of an engine at these levels:

  • At the beginning of a processing engine flow
    • The custom tasks will be executed at the start of processing flow. If one of the custom tasks fails, none of the base tasks will run.
  • At the end of a processing engine flow
    • The custom tasks will be executed after all the defined tasks, core tasks or customized tasks, have completed successfully. If one of the custom tasks added configured to run at the end of the processing flow fails, the entire job will fail.
  • At the middle of a processing engine flow
    • Custom tasks may be injected into the middle of a set of core processing engine tasks, effectively altering the sequence of processing engine flow as they are defined at development time.

The following artifacts may be defined in the PE extension configuration file:

  • Extensions for specific engines
  • Extensions for specific set of tasks (group tasks)
  • Custom processing engine task definitions
  • Dependencies among custom tasks

Below is a sample trm-pe-extension-context.xml:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:aop="http://www.springframework.org/schema/aop"
xmlns:tx="http://www.springframework.org/schema/tx"
xsi:schemaLocation=
"http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://www.springframework.org/schema/aop
http://www.springframework.org/schema/aop/spring-aop.xsd
http://www.springframework.org/schema/tx
http://www.springframework.org/schema/tx/spring-tx.xsd">

<!-- ==== Section 1: Processing Engine extension mappings =================== -->
<bean id="processingEngineExtensions"
class="org.springframework.beans.factory.config.MapFactoryBean">
<property name="sourceMap">
<map>
<entry key="lge" value-ref="lgeExtension"/>
</map>
</property>
</bean>

<!-- ===== Section 2: Module-specific extensions ========================= -->
<bean id="lgeExtension" parent="processingEngineExtension" scope="prototype">
<property name="tasks">
<map>
<entry key="lgeExt6"
value="bean=lgeExtension6 depends=lge006 output=lge007"/>
<entry key="lgeExt9"
value="bean=lgeExtension9 depends=lge009 dependents=lge010"/>
<entry key="lgeExt12"
value="bean=lgeExtension12 dependents=lge013 source=lge012"/>
</map>
</property>
</bean>

<!-- ====== Section 3: Custom task definitions ========================= -->
<bean id="lgeExtension6"
class="com.teradata.trm.pe.common.task.SimpleMessageTask"
scope="prototype">
<property name="message" value="This is Comm PE LGE extension 6"/>
</bean>
<bean id="lgeExtension9"
class="com.teradata.trm.pe.common.task.SimpleMessageTask"
scope="prototype">
<property name="message" value="This is Comm PE LGE extension 9"/>
</bean>
<bean id="lgeExtension12"
class="com.teradata.trm.pe.common.task.SimpleMessageTask"
scope="prototype">
<property name="message" value="This is Comm PE LGE extension 12"/>
</bean>
</beans>

Sections of trm-pe-extension-context.xml

Section 1: Processing engine extension mapping

This section is where the mappings between existing (core) processing engine capabilities and the customized extensions are configured. A mapping is consists of two parts: the capability bean id (from the base definition file) and the id of the module-specific extension from Section 2 of the extension file.

(from the example)

<!-- ==== Section 1: Processing Engine extension mappings =================== -->
<bean id="processingEngineExtensions"
class="org.springframework.beans.factory.config.MapFactoryBean">
<property name="sourceMap">
<map>
<entry key="lge" value-ref="lgeExtension"/>
</map>
</property>
</bean>

The mapping between Lead Generation (LGE) capability and its extension is:

<entry key="lge" value-ref="lgeExtension"/>

where “lge” is the capability id defined in communication-manager-pe-context.xml and “lgeExtension” is the LGE extension defined in Section 2 of the extension file.

The left (key) attribute is always from the base definition file and should be one of the existing processing engine capabilities. The right (value-ref) attribute is the customized module-specific extension of that capability.

Note: For engines other than Communication engine (FullBatch, Intraday, Adhoc), the left (key) attribute is one the values returned by the query below. These texts or bean ids are found in the base definition files for the processing engines.

SELECT Process_Name_Txt 
FROM Cr_Process_Capability
WHERE Module_Cd != 1
ORDER BY Process_Name_Txt

Processing Engines and Capabilities

Communication:

The following are the customizable process capabilities for communication engines. The texts in the first column are the exact texts that should appear in the extension configuration and are case-sensitive:

Process Description
are Apply Responses
cle Cycle Leads
lge Lead Generation
init Initialization. Invoked at start of every Communication engine capability
end Clean Up. Invoked at end of every Communication engine capability

Currently, there are fourteen capabilities configured in the system for the Communication engines. However, only three are currently supported by a UI job submission or scheduling. The other capabilities, although in place, will require new screens or UI support. The capabilities supported by a UI submission or scheduling are:

Lead Generation - Lead generation process is invoked during adhoc Communication submits and also by the Batch engine for a group of submitted Communications. To customize lead generation process, the mapping should look like this:

<entry key="lge" value-ref="lgeExtension"/>

Lead Generation with Output – Output generation takes place after lead generation. Custom tasks may be injected before and after Output process is kicked off by the Communication engine as an on-demand engine. The Output generation process itself is executed by a separate engine, the Output engine. Output customization should be done separately as an extension to the Output engine.

FullBatch – The FullBatch execution involves Apply Responses, Cycle Leads, Lead Generation, and Output, in that sequence, for all Communications submitted to the group. The individual sub processes may be customized using the following configurations.

<entry key="are" value-ref="areExtension"/>
<entry key="cle" value-ref="cleExtension"/>
<entry key="lge" value-ref="lgeExtension"/>

Segmentation:

Process Description
segRealizePlan Segment Plan Realization (Segment Plan Run Now)
segWaterfall Segment Plan Waterfall Count (Segment Plan Count Now)
Segment_independentCounts Segment Quick Count (Segment Count Now)

Segment Plan Realization – This is invoked when the user submits or schedules the SegmentPlan for RunNow in the UI. This process is also kicked off by the Communication engine, as an on-demand engine, in the middle of lead generation. Extension mapping should look like this:

<entry key="segRealizePlan" value-ref="segRealizationExtension"/>

Segment Plan Waterfall Count – This is the Segmentation engine capability that does the counts for the Segment Plan. Segment plans that are submitted using Count Now in the UI will gone through this process. Mapping:

<entry key="segWaterfall" value-ref="segWaterfallCountExtension"/>

Segment Quick Count – This is used to calculate the counts for a segment. Currently, this job is executed immediately after the Segment is submitted for Count Now in the UI and cannot be scheduled.

<entry key="Segment_independentCounts" value-ref="segmentQuickCountExtension"/>

Limitation: Currently, customizations of any of the three Segmentation engine capabilities shown above are limited to start and end of the process. Injection of custom tasks in the middle of processing steps is not possible. For example, it’s not possible to run a SQL script or batch command after de-duplication, or before counts are to be persisted for the segments or for the segment plan. Pick Best, Splitting and Clipping, Store Limits, etc – These sub processes are internally executed by the Segmentation engine and are currently not exposed as customizable processes. Therefore, custom tasks may only be placed before or after the capabilities shown in the chart above.

Analysis:

Process Description
analysisProductAffinity Product Affinity
analysisBehaviorTrend Behavior Trend Analysis
CrossSegmentAnalysisPE Cross Segment Analysis
analysisPercentileProfiling Percentile Profiling

Example extension mappings for the Analysis engine:

<entry key="analysisProductAffinity" value-ref="affinityExtension"/>
<entry key="analysisBehaviorTrend" value-ref="behaviorTrendExtension"/>
<entry key="CrossSegmentAnalysisPE" value-ref="crossSegmentExtension"/>
<entry key="analysisPercentileProfiling" value-ref="percentileProfilingExtension"/>

Output:

Process Description
outputGenerate Output Generation

Output Generation – Output may be generated as part of a Lead Generation job or as an independent submission. In the latter case, lead generation for a Communication must have previously run in order to have some results. Currently, output processing engine runs a five-step set to come up with the final set of data:

  • Query Building
  • Output Generation (file or table)
  • Pre-processing command
  • Delivery (if any)
  • Post-processing command

Below is an example of an Output customization mapping:

<entry key="outputGenerate" value-ref="outputExtension"/>

Limitation: Currently, like the Segmentation engine, the customization of Output processing engine steps is limited to the start and end of the actual Output process. What that means is, custom tasks maybe injected before or after output is built but not before delivery, if any, or not before or after the pre/post processing. There is a current plan to break up Output processing into smaller, distinct, and therefore customizable parts. At this time, there is no target release set for that.

Section 2: Module-specific extensions

This section contains one or more module extensions. An extension consists of one or more custom tasks, defined in Section 3 of the extension file. The dependencies among custom tasks, the dependencies of the custom tasks to the core processing engine tasks, and the dependencies of the core processing engine tasks to the custom tasks, if any, are defined in this section. Below is an example LGE extension:

<!-- ===== Section 2: Module-specific extensions ========================= -->
<bean id="lgeExtension" parent="processingEngineExtension" scope="prototype">
<property name="tasks">
<map>
<entry key="lgeExt6"
value="bean=lgeExtension6 depends=lge006 dependents=lge007"/>
<entry key="lgeExt9"
value="bean=lgeExtension9 depends=lge009 dependents=lge010"/>
<entry key="lgeExt12"
value="bean=lgeExtension12 depends=lge012 dependents=lge013"/>
</map>
</property>
</bean>

In the example above, we have an LGE extension that is made up of three custom tasks namely lgeExt6, lgeExt9, and lgeExt12. Currently, the Communication engine LGE capability has 14 steps. With this extension in place, there will be 17 steps. The chart below displays the sequence of processing flow for LGE without or with the extension as defined above:

Step # Steps without extension Steps with extension
1 lge001 lge001
2 lge002 lge002
3 lge003 lge003
4 lge004 lge004
5 lge005 lge005
6 lge006 lge006
7 lge007 lgeExt6
8 lge008 lge007
9 lge009 lge008
10 lge010 lge009
11 lge011 lgeExt9
12  lge012  lge010 
13  lge013  lge011 
14  lge014  lge012 
15    lgeExt12
16    lge013 
17    lge014 

From the example above, we can take one of the dependency mappings:

<entry key="lgeExt6" 
value="bean=lgeExtension6 depends=lge006 dependents=lge007"/>

The key, "lgeExt6", identifies the mapping definition.

The value, "bean=lgeExtension6 depends=lge006 dependents=lge007", is the dependency definition.

Section 3: Custom task definitions

This is the section where the bean definitions for the custom tasks are placed. In the example below

<!-- ====== Section 3: Custom task definitions ========================= -->
<bean id="lgeExtension6"
class="com.teradata.trm.pe.common.task.SimpleMessageTask"
scope="prototype">
<property name="message" value="This is Comm PE LGE extension 6"/>
</bean>

the id of the custom task, "lgeExtension6", will appear in Section 2 of the extension. The class attribute is the fully-qualified class name of the custom task. The scope attribute should be set to “prototype” to allow the container to create a brand new instance of the custom task each time the engine runs. The property mappings, identified by the <property> elements, encapsulate the properties (and their values) required by the custom task to do meaningful work and execute successfully.

Tags (2)
2 REPLIES

Re: Implementing basic Processing Engine extensions in TRM 6

Amazing~I implemented pe ext. for adhoc communication with channel integration to deliver leads near real time
Teradata Employee

Re: Implementing basic Processing Engine extensions in TRM 6

Nice rehash of the original informaion on PE extensions.