TPT export into AWS S3 buckets.

Tools & Utilities
Teradata Employee

Re: TPT export into AWS S3 buckets.

If you would like to download the file to a file share and then load it into Teradata, that is certainly an option.

For those that are spinning up EC2 instances and have TPT installed, we provide a method to copy the data from S3 to Teradata.

 

-- SteveF
Enthusiast

Re: TPT export into AWS S3 buckets.

 With this module can we load to Teradata on-prem or to load to Teradata in AWS?

Teradata Employee

Re: TPT export into AWS S3 buckets.

Either on-prem or in AWS.

 

-- SteveF
Enthusiast

Re: TPT export into AWS S3 buckets.

Good to know. How does it send data to on-prem internally? using Amazon CLI or Directocnnect?

Enthusiast

Re: TPT export into AWS S3 buckets.

Lastly one question. The Teradata Parallel Transporter operators and the Teradata Access Module for Amazon S3 run on an EC2 instance. Is there any possibility to use this functionality through AWS serverless way (lambda function)? Maybe Instal TPT,conigconfig files and run TPT through Lambda ?

 

 

Teradata Employee

Re: TPT export into AWS S3 buckets.

TPT cannot be run through Lambda.
-- SteveF

Re: TPT export into AWS S3 buckets.


@feinholz wrote:

Either on-prem or in AWS.

 


hey Steve,

couple of quick questions. is there a way to increase the log level for 'AWS_S3_ACCESS_MODULE'?

I tried the init string, but it just ignored it.

 

We are running into a strange issue, where the file operations are just fine when using AWS CLI, but fails while running thru TPT.

 

----- setup-----

VARCHAR AccessModuleName = 'libs3axsmod.so'

,VARCHAR AccessModuleInitStr = 'log_level=5 S3Bucket=.........

-----setup-----

 

 

 

 

Teradata Parallel Transporter Version 15.10.01.06 64-Bit

Job log: /opt/teradata/client/15.10/tbuild/logs/landingzone_LineCorp_S3export-3201.out

Job id is landingzone_LineCorp_S3export-3201, running on xyz

Found CheckPoint file: /opt/teradata/client/15.10/tbuild/checkpoint/landingzone_LineCorp_S3exportLVCP

This is a restart job; it restarts at step MAIN_STEP.

Teradata Parallel Transporter DataConnector Operator Version 15.10.01.06

Teradata Parallel Transporter Export Operator Version 15.10.01.06

export_opt: private log not specified

file_opt[1]: Instance 1 directing private log report to 'dtacop-xxxxxx-41746-1'.

file_opt[1]: DataConnector Consumer operator Instances: 1

S3AXM [41746]: AWS S3 ACCESS Module -- 'MODULE_NAME=AWS_S3_ACCESS_MODULE'   Version: MODULE_VERS=15.10.01.02.

S3AXM [41746]: COPYRIGHT 2016, Teradata Corporation.  ALL RIGHTS RESERVED.

S3AXM [41746]: Compiled for PLATFORM_ID=Linux x8664 (64 bit) on Dec 21 2016

S3AXM [41746]: pmdcomt header version 'Common 15.10.00.00', packing 'pack (push, 1)'

S3AXM [41746]: pmddamt header version 'Common 15.10.00.00', packing 'pack (push, 1)'

S3AXM [41746]: AXMS3 Library built on Dec 20 2016 21:33:49

S3AXM [41746]: Unsupported keyword: "log_level" -- continuing

S3AXM [41746]: Default AWS Credentials/Profile directory set from $HOME to: /home/xxxxxx/.aws

S3AXM [41746]: Attribute Set Request type 2: 'TBR_OP_HANDLE'

S3AXM [41746]: Attribute Set Request type 1: 'CHARSET_NAME=ASCII'

S3AXM [41746]: Attribute Set Request type 1: 'PRESERVE_RESTART_INFO=YES'

file_opt[1]: ECI operator ID: 'file_opt-41746'

S3AXM [41746]: S3 Parameter        Parameter Value                     Parameter Origin

S3AXM [41746]: -------------       --------------------------------    ---------------------

S3AXM [41746]: S3ConfigDir       = /home/xxxxxx/.aws                   $HOME environment variable

S3AXM [41746]: S3Profile         = default                             Default

S3AXM [41746]: S3Region          = us-west-2                           AWS "config" file

S3AXM [41746]: S3Bucket          xxxxxxxxxxxx                    Module Parameters

S3AXM [41746]: S3Prefix          = yyyyyyyy/yyyyyyyyy/                  Module Parameters

S3AXM [41746]: S3Object          = data_lake_views_line_corp_2017-    Module Parameters

S3AXM [41746]: S3ConnectionCount = 10                                  Default

S3AXM [41746]: S3BufferCount     = 20                                  Computed Default

S3AXM [41746]: S3BufferSize      = 8388608                             Default

S3AXM [41746]: S3SinglePartFile  = True                                Module Parameters

S3AXM [41746]: S3PartsPerFile    = 10000                               Default

S3AXM [41746]: S3AccessID        = ************HNSQ                    AWS "credentials" file

S3AXM [41746]: S3AccessKey       = ************7MIX                    AWS "credentials" file

S3AXM [41746]: -----------------------------------------------------------------------------

Sending request to s3-us-west-2.amazonaws.com:443: GET /ccna-landingzone HTTP/1.1

Sending request to s3-us-west-2.amazonaws.com:443: GET /ccna-landingzone HTTP/1.1

Sending request to s3-us-west-2.amazonaws.com:443: GET /ccna-landingzone HTTP/1.1

Sending request to s3-us-west-2.amazonaws.com:443: GET /ccna-landingzone HTTP/1.1

Sending request to s3-us-west-2.amazonaws.com:443: GET /ccna-landingzone HTTP/1.1

S3AXM [41746]: Error: Unable to open S3 connection: AccessDenied: Access Denied

S3AXM [41746]: Return value for failing opcode Op_Open is pmrcBadParm: Bad parameter passed to API

file_opt[1]: TPT19402 pmOpen failed. Bad parameter passed to Access Module API (2)

file_opt[1]: TPT19304 Fatal error opening file.

file_opt[1]: TPT19015 TPT Exit code set to 12.

export_opt: connecting sessions

export_opt: disconnecting sessions

S3AXM [41746]: AWS S3 ACCESS Module Shutdown Completed

file_opt[1]: Total files processed: 0.

export_opt: Total processor time used = '0.071964 Second(s)'

export_opt: Start : Thu Oct  5 17:57:24 2017

export_opt: End   : Thu Oct  5 17:57:29 2017

Job step MAIN_STEP terminated (status 12)

Job landingzone_LineCorp_S3export terminated (status 12)

Job start: Thu Oct  5 17:57:23 2017

Job end:   Thu Oct  5 17:57:29 2017

Teradata Employee

Re: TPT export into AWS S3 buckets.

The “LoggingFlags” must be passed as the first field of the module parameters and must NOT be preceded by any white space.

 

The syntax is: LoggingFlags=0x<hex value of desired flags>

 

Supported values are (and can be or’d together):

0x01  Log Parsing

0x02  Log AXMS3 calls

0x04  Log TPTAPI interaction

0x08  Log GetPos calls

0x10  Log “Other”  (not currently used)

0x20  Log AXMS3 calls verbosely  -- creates a file called ‘tester.out’ generated by the AXMS3 library in the current directory

0x40  Log TPTAPI interaction verbosely

0xff    Enable ALL Logging (may be very noisy)

 

It is unlikely that you need "Log Parsing" (0x01) which is very noisy.  You could set logging to 0x7e  (LoggingFlags=0x7e).

-- SteveF

Re: TPT export into AWS S3 buckets.

Thanks, Steve. Is there any way we can start lookign at the actual AWS CLI debug information? I have tried all most all the logging level which seem to show more of how TPT is picking the keys, profiles and such.

But for the AWS work, the logging is at a very high level only.

 

cat tester.out

S3_LIB [40022]Disconnect detected: AccessDenied: Access Denied

S3_LIB [40022]Disconnect detected: AccessDenied: Access Denied

S3_LIB [40022]Disconnect detected: AccessDenied: Access Denied

Re: TPT export into AWS S3 buckets.

Just to close the topic on this, looks like the internal commands need access to list the objects at each layer of the bucket/prefix.

After making that change, the process is working fine.

 

Please make a note of it, if you have highly secured setup, where keys are defined for a specific lower level 'folder' only.