Unity Data Mover 15.10 Release

UDA
The UDA channel is for Teradata’s Unified Data Architecture including the Analytical Ecosystem and other UDA influences. This channel provides information specific to the integration and co-existence of multiple systems, in particular when a mix of Aster, Teradata, and Hadoop are present. It is also meant to support information around the UDA enabling technologies so products like Viewpoint, Data Mover, Connectors, QueryGrid, etc.
Teradata Employee

Unity Data Mover 15.10 Release

We are pleased to announce the General Customer Availability (GCA) of Unity Data Mover 15.10 as of June 2015. With release 15.10, Unity Data Mover is now certified with Teradata Database 15.10. Data Mover 15.10 can utilize QueryGrid/foreign server connections to move data from Hadoop to Teradata (if installed) in addition to using the Teradata Connector for Hadoop (TDCH).  Hadoop support has also been extended to support TDH 2.1.7, Hortonworks 2.1.7 on commodity hardware and Cloudera 4.3 on commodity hardware. Aster support is extended to Aster 6.10.

Data Mover 15.10 New Features - Operational Enhancements

  • Updated Daemon Status Page. This provides an expanded view into the inner workings of the Data Mover deamon and agents. See all jobs currently running and those that are queued. See why jobs are waiting and what resources thay are waiting for. This feature helps debug any system-wide issues with Data Mover jobs.

  • Enhanced Job Status Output. This provides a detailed output of the job steps executed for a particular job and a time stamp for each step. In conjunction with the Daemon Status Page, systemwide performance issues can be investigated and resolved.

     

  • Database Search Box. This was based on customer requests for an improved Viewpoint experience. Just start typing the name of the object and the object browser will show selected database and all child objects. It eliminates need to traverse tree down from DBC to find your database.

  • User ID Pools in the Portlet. This is another feature to improve the user experience with the Viewpoint portlet. User ID Pools allow you to assign user credentials from a pool instead of assigning them directly to the job. This means that if somebody leaves or a password needs to be changed, you change it once in the pool and all Data Mover jobs using the pool are automatically updated. Eliminates the need to change credentials for each individual job.  

Data Mover Overview

Unity Data Mover is a powerful data synchronization product that quickly and reliably copies data between Teradata UDA systems. It delivers the confidence a business needs when moving data to the right place at the right time. This solution promises to automate the data movement process, and accommodate the many situations found within any comprehensive analytical ecosystem. Moves can be easily initiated using the command line interface, or even be done ad-hoc using the Data Mover / Viewpoint 15.00 portlet.

Unity Data Mover is a focused utility; moving tables and other objects from one system to another in the most efficient manner.

Data Mover is able to copy both Full and Partial tables. However, to move partial tables, the table must be instrumented with a time stamp or other indicator. You can then pass Data Mover a “Where-Clause” to grab only those updated records.

Data Mover has a Viewpoint portlet for ad-hoc moves and a Command-Line Interface for regularly scheduled moves. Data Mover does not have a scheduling mechanism, it relies on the customer’s Enterprise Scheduler driving it through the Command-Line Interface.

Other key features include:

  • Uses Teradata Utilities, including the multi-ARC utility.
  • No need to write job scripts
  • Chooses the best method based on the situation.
  • Supports copying between different database versions.

Data Mover Architecture

This chart shows the Data Mover architecture in a Teradata to Teradata configuration.

The user’s intent is entered in one of two ways, either via the Viewpoint portlet or the Command-Line Interface. The Viewpoint portlet is ideal for ad-hoc data moves. Using the portlet the user can highlight the tables they wish to copy, identify the destination system, and have Data Mover execute the copy job.

The Command-Line Interface is typically used for scheduled moves of a large number of tables. It accepts XML input for creating, running, editing, and monitoring Data Mover copy jobs. You can run the commands on an ad-hoc basis, as an alternative to using the portlets, and you can switch between the command-line and portlet interfaces. You can also develop scripts to automate Data Mover commands and to use with UNIX cron or other job-scheduling applications. For example, you might want to have a copy job run automatically every night at 1 a.m.

The Data Mover Daemon takes that intent and turns it into a Data Mover job. As part of the job creation the utility to be used to accomplish the move is chosen. Typically, either ARC or a TPT protocol is chosen for a Teradata to Teradata move. The Data Mover job is then passed to the Data Mover Agent for execution. Note that the data passes through the Data Mover TMS for Teradata to Teradata moves, but it does not land on disk. Performance is very dependent on the network connectivity in and out of the Data Mover TMS.

Data Mover is also tightly integrated with Unity Ecosystem Manager. It can report status to Ecosystem Manager and have Ecosystem Manager send alerts if something has gone wrong. Ecosystem Manager can also trigger Data Mover jobs based on an event. As an example you could have Ecosystem Manager trigger a copy job after an ETL has completed.

Teradata/Aster and Teradata/Hadoop

DM

When doing copies in the larger UDA environment, Data Mover is limited to having a Teradata system as either the source or destination for the data copy. Aster/Aster, Hadoop/Hadoop and Aster/Hadoop data moves are not supported at this time.

The flow is the same as Teradata to Teradata moves. The user enters their intent via the Viewpoint portlet or Command Line Interface. The Data Mover daemon then creates the job steps needed to execute the move. However, the utility to accomplish the data copy is different from a Teradata to Teradata move and depends on whether Aster or Hadoop is involved. Between Teradata and Aster the move is accomplished via the Teradata-Aster Connector. Between Teradata and Hadoop it is accomplished either through the Teradata Connector for Hadoop (TDCH) or in some specific cases by SQL-H. If the customer has Teradata 14.10 or above and has purchased the SQL-H option, it can be used to move data from Hadoop to Teradata, a one way movement, utilizing the SQL-H command.

Looking at Data Mover in an Aster/Hadoop configuration, another key thing to notice is that the data does not pass through the Data Mover agent TMS. The Data Mover daemon creates the SQL commands to execute the copy and then lets the two systems involved complete the process. In this case the connectivity between the source and destination systems is key to performance.

2 REPLIES
Teradata Employee

Re: Unity Data Mover 15.10 Release

Hi everyone,

We have Teradata Unity Data Mover latest version and need to the configuration guide to configure, There are two TMS and need to configure it in the high availibility option.

Will be very thankful if someone share the latest configuration document. I have 14.10 configuration and upgrade guide but it may not be fit with teh latest version. UDM 15.0 or 15.10 will be ok.

Many thansk in advance.

Badshah Rehman Teradata

Teradata Employee

Re: Unity Data Mover 15.10 Release