New Teradata Unity Data Mover 15.00 Available

UDA
The UDA channel is for Teradata’s Unified Data Architecture including the Analytical Ecosystem and other UDA influences. This channel provides information specific to the integration and co-existence of multiple systems, in particular when a mix of Aster, Teradata, and Hadoop are present. It is also meant to support information around the UDA enabling technologies so products like Viewpoint, Data Mover, Connectors, QueryGrid, etc.
Teradata Employee

New Teradata Unity Data Mover 15.00 Available

Announcing Teradata Unity Data Mover 15.00

We are pleased to announce the General Customer Availability (GCA) of Unity Data Mover 15.00. With release 15.00, Unity Data Mover is now certified with Teradata Database 15.00, and supports important database features such as JSON and foreign server definitions. Data Mover can now move foreign server definitions, as well as utilize QueryGrid/foreign server connections to move data from Hadoop to Teradata (if installed). Hadoop support has also been extended to support TDH 2.1 and 1.3.2.

Unity Data Mover is a powerful data synchronization product that quickly and reliably copies data between Teradata UDA systems. It delivers the confidence a business needs when moving data to the right place at the right time. This solution promises to automate the data movement process, and accommodate the many situations found within any comprehensive analytical ecosystem. Moves can be easily initiated using the command line interface, or even be done ad-hoc using the Data Mover / Viewpoint 15.00 portlet.

Data Mover Overview

Unity Data Mover is a focused utility; moving tables and other objects from one system to another in the most efficient manner.

Data Mover is able to copy both Full and Partial tables. However, to move partial tables, the table must be instrumented with a time stamp or other indicator. You can then pass Data Mover a “Where-Clause” to grab only those updated records.

Data Mover has a Viewpoint portlet for ad-hoc moves and a Command-Line Interface for regularly scheduled moves. Data Mover does not have a scheduling mechanism, it relies on the customer’s Enterprise Scheduler driving it through the Command-Line Interface.

Other key features include:

  • Uses Teradata Utilities, including the multi-ARC utility.
  • No need to write job scripts
  • Chooses the best method based on the situation.
  • Supports copying between different database versions.

Data Mover Architecture

This chart shows the Data Mover architecture in a Teradata to Teradata configuration.

The user’s intent is entered in one of two ways, either via the Viewpoint portlet or the Command-Line Interface. The Viewpoint portlet is ideal for ad-hoc data moves. Using the portlet the user can highlight the tables they wish to copy, identify the destination system, and have Data Mover execute the copy job.

The Command-Line Interface is typically used for scheduled moves of a large number of tables. It accepts XML input for creating, running, editing, and monitoring Data Mover copy jobs. You can run the commands on an ad-hoc basis, as an alternative to using the portlets, and you can switch between the command-line and portlet interfaces. You can also develop scripts to automate Data Mover commands and to use with UNIX cron or other job-scheduling applications. For example, you might want to have a copy job run automatically every night at 1 a.m.

The Data Mover Daemon takes that intent and turns it into a Data Mover job. As part of the job creation the utility to be used to accomplish the move is chosen. Typically, either ARC or a TPT protocol is chosen for a Teradata to Teradata move. The Data Mover job is then passed to the Data Mover Agent for execution. Note that the data passes through the Data Mover TMS for Teradata to Teradata moves, but it does not land on disk. Performance is very dependent on the network connectivity in and out of the Data Mover TMS.

Data Mover is also tightly integrated with Unity Ecosystem Manager. It can report status to Ecosystem Manager and have Ecosystem Manager send alerts if something has gone wrong. Ecosystem Manager can also trigger Data Mover jobs based on an event. As an example you could have Ecosystem Manager trigger a copy job after an ETL has completed.

Teradata/Aster and Teradata/Hadoop

DM

When doing copies in the larger UDA environment, Data Mover is limited to having a Teradata system as either the source or destination for the data copy. Aster/Aster, Hadoop/Hadoop and Aster/Hadoop data moves are not supported at this time.

The flow is the same as Teradata to Teradata moves. The user enters their intent via the Viewpoint portlet or Command Line Interface. The Data Mover daemon then creates the job steps needed to execute the move. However, the utility to accomplish the data copy is different from a Teradata to Teradata move and depends on whether Aster or Hadoop is involved. Between Teradata and Aster the move is accomplished via the Teradata-Aster Connector. Between Teradata and Hadoop it is accomplished either through the Teradata Connector for Hadoop (TDCH) or in some specific cases by SQL-H. If the customer has Teradata 14.10 or above and has purchased the SQL-H option, it can be used to move data from Hadoop to Teradata, a one way movement, utilizing the SQL-H command.

Looking at Data Mover in an Aster/Hadoop configuration, another key thing to notice is that the data does not pass through the Data Mover agent TMS. The Data Mover daemon creates the SQL commands to execute the copy and then lets the two systems involved complete the process. In this case the connectivity between the source and destination systems is key to performance.

Compatibility

Data Mover 15.00 conforms to Viewpoint 15.00 portlet requirements. All product portlets need to be upgraded in sync when going from Viewpoint 14.10 to Viewpoint 15.00.  The listing below documents the minimal product versions necessary for Teradata Viewpoint 15.00 compatibility.

Teradata Database = 13.00 to 15.00

Aster Database = 5.00 to 6.00

Hadoop = Teradata Hadoop Appliance 1.3.2 and 2.1

13 REPLIES
N/A

Re: New Teradata Unity Data Mover 15.00 Available

Can datamover unload to intermediate file also with it's dml for future use?or it just can do on the fly unload and load?

Teradata Employee

Re: New Teradata Unity Data Mover 15.00 Available

Data Mover does not land any data to disk so the data being extracted from the source system must always be immediately loaded to the target system. If the target system is down or unavailable, then the Data Mover job won't work.

N/A

Re: New Teradata Unity Data Mover 15.00 Available

Does it capable of identifying  deleted records from source  and do the same on target systems for partial tables data movement?

Re: New Teradata Unity Data Mover 15.00 Available

Can you move data from TD v13.1 to TDv15 using the datamover? 

Teradata Employee

Re: New Teradata Unity Data Mover 15.00 Available

Yes, Data Mover 15.00 can move data between TD databases 13.00 to 15.00. Includes any back and forth combination of those versions.

Teradata Employee

Re: New Teradata Unity Data Mover 15.00 Available

krrish. Sorry did not see your comment earlier. Data Mover cannot transfer deletes between systems. It can move partial tables between systems if the table has been instrumented with a time stamp or other method to identify new or changed records. You give Data Mover a Where clause that identifies those records to be moved and it loads them into a staging table on the target system before inserting them into the destination table.

N/A

Re: New Teradata Unity Data Mover 15.00 Available

Hi Cliff, I have a question with DM's behaviour.

When choosing from userID pool, will DM opt the user whose access rights are granted directly to it instead of the users who have access to an object via role (Or it doesn't matter)?

Teradata Employee

Re: New Teradata Unity Data Mover 15.00 Available

Data Mover is not aware of the permissions each Teradata user has nor how those permissions are granted. When using the user ID pool, Data Mover simply iterates through the list of Teradata users until it finds one that is not currently in use by another DM job. It is assumed that all of the Teradata users in the pool have the required permissions to do the work being asked.

Re: New Teradata Unity Data Mover 15.00 Available

Cliff,

We are using Data Mover 14.10 at our site.

Currently we are using the cron job at the data mover server to automate the data transfer from the prodctuin TD server to DR TD server.

There is a new request from business to setup event based trigger (using Control-M) to run the new data transfer job using DM once the batch is completed.

Does DM14.10 has any client tool (command line tool) or any table that l could use the BTEQ to insert the record and trigger the job?

Cheers,