Using TPT/Teradata Access Module for S3 with KMS encrypted objects

Tools & Utilities
Highlighted
Tourist

Using TPT/Teradata Access Module for S3 with KMS encrypted objects

Hi all,

 

I am attempting to use the S3 Access Module to move data from CSVs in S3 to Teradata. I am using TTU 16.20. The objects in the bucket are encrypted using KMS and I want to know if this is interfering with TPT's ability to read those files.

 

I have confirmed that my AWS credentials are correct and can view/download the objects in the bucket via the AWS CLI.

 

The job doesn't seem to have any problem connecting to S3 but can't seem to read from the CSVs. See the output of the job below:

 

 

Teradata Parallel Transporter Version 16.20.00.04 64-Bit
Job log: /opt/teradata/client/16.20/tbuild/logs/anna-28.out
Job id is anna-28, running on Darigaaz-VM-Ubuntu
Teradata Parallel Transporter Load Operator Version 16.20.00.04
$LOAD: private log specified: LOAD_OPERATOR_LOG
Teradata Parallel Transporter DataConnector Operator Version 16.20.00.04
$FILE_READER[1]: Instance 1 directing private log report to 'S3W001P2_1-1'.
$FILE_READER[1]: DataConnector Producer operator Instances: 1
14:06:06 S3AXM [12440]: AWS S3 ACCESS Module -- 'MODULE_NAME=AWS_S3_ACCESS_MODULE'   Version: MODULE_VERS=16.20.00.02.
14:06:06 S3AXM [12440]: COPYRIGHT 2016, 2017, 2018, Teradata Corporation.  ALL RIGHTS RESERVED.
14:06:06 S3AXM [12440]: Compiled for PLATFORM_ID=Linux x8664 (64 bit) on Mar 24 2018
14:06:06 S3AXM [12440]: pmdcomt header version 'Common 15.10.00.00', packing 'pack (push, 1)'
14:06:06 S3AXM [12440]: pmddamt header version 'Common 15.10.00.00', packing 'pack (push, 1)'
14:06:06 S3AXM [12440]: AXMS3 Library built on Nov 14 2017 16:50:20
14:06:06 S3AXM [12440]: Default AWS Credentials/Profile directory set from $HOME to: /home/anna/.aws
14:06:06 S3AXM [12440]: Attribute Set Request type 2: 'TBR_OP_HANDLE'
14:06:06 S3AXM [12440]: Attribute Set Request type 1: 'CHARSET_NAME=ASCII'
14:06:06 S3AXM [12440]: Attribute Set Request type 1: 'PRESERVE_RESTART_INFO=YES'
$FILE_READER[1]: ECI operator ID: '$FILE_READER-12440'
14:06:06 S3AXM [12440]: 	S3 Parameter        Parameter Value                     Parameter Origin
14:06:06 S3AXM [12440]: 	-------------       --------------------------------    ---------------------
14:06:06 S3AXM [12440]: 	S3ConfigDir       = /home/anna/.aws                     $HOME environment variable
14:06:06 S3AXM [12440]: 	S3Profile         = {Omitted}    Module Parameters
14:06:06 S3AXM [12440]: 	S3Region          = us-east-1                           AWS "config" file
14:06:06 S3AXM [12440]: 	S3Bucket          = {Omitted}    Module Parameters
14:06:06 S3AXM [12440]: 	S3Prefix          = order1/                             Module Parameters
14:06:06 S3AXM [12440]: 	S3Object          = *.csv                               Module Parameters
14:06:06 S3AXM [12440]: 	S3ConnectionCount = 10                                  Default
14:06:06 S3AXM [12440]: 	S3BufferCount     = 20                                  Computed Default
14:06:06 S3AXM [12440]: 	S3BufferSize      = 8388608                             Default
14:06:06 S3AXM [12440]: 	S3SinglePartFile  = True                                Module Parameters
14:06:06 S3AXM [12440]: 	S3AccessID        = ************7ZAA                    AWS "credentials" file
14:06:06 S3AXM [12440]: 	S3AccessKey       = ************Je0J                    AWS "credentials" file
14:06:06 S3AXM [12440]: 	S3Direction       = ReadFromS3                          TPT Job Parameters
14:06:06 S3AXM [12440]: 	-----------------------------------------------------------------------------
14:06:06 S3AXM [12440]: S3 Connector[1] reading from: 's3://{Omitted}/order1/*.csv'
$LOAD: connecting sessions
$FILE_READER[1]: Operator instance 1 processing file './S3W001DT'.
$LOAD: preparing target table
$LOAD: entering Acquisition Phase
14:06:13 S3AXM [12440]: For offset            0 opened object of length            0: "order1/part-00000-6d5f81ef-9038-4f2e-8e61-19e869a10c0d-c000.csv"
14:06:13 S3AXM [12440]: For offset            0 opened object of length   1027821425: "order1/part-00002-6d5f81ef-9038-4f2e-8e61-19e869a10c0d-c000.csv"
14:06:18 S3AXM [12440]: !Error! Unable to read at specified S3 offset: AccessDenied: Access Denied
14:06:18 S3AXM [12440]: Return value for failing opcode Op_Read is pmrcBadParm: Bad parameter passed to API
$FILE_READER[1]: TPT19402 pmRead failed. Bad parameter passed to Access Module API (2)
$FILE_READER[1]: TPT19305 Fatal error reading data.
$FILE_READER[1]: TPT19015 TPT Exit code set to 12.
$LOAD: disconnecting sessions
$LOAD: Performance metrics:
$LOAD:     MB/sec in Acquisition phase: 0
$LOAD:     Elapsed time from start to Acquisition phase:   13 second(s)
$LOAD:     Elapsed time in Acquisition phase:   0 second
$LOAD:     Elapsed time in Application phase:   0 second
$LOAD:     Elapsed time from Application phase to end: < 1 second
$LOAD: Total processor time used = '0.127911 Second(s)'
$LOAD: Start : Wed Feb 20 14:06:06 2019
$LOAD: End   : Wed Feb 20 14:06:19 2019
14:06:30 S3AXM [12440]: !Error! Unable to close S3 connection: AccessDenied: Access Denied
14:06:30 S3AXM [12440]: Return value for failing opcode Op_Close is pmrcBadParm: Bad parameter passed to API
14:06:30 S3AXM [12440]: AWS S3 ACCESS Module Shutdown Completed
$FILE_READER[1]: Total files processed: 0.
Job step IMPORT_THE_DATA terminated (status 12)
Job anna terminated (status 12)
Job start: Wed Feb 20 14:06:06 2019
Job end:   Wed Feb 20 14:06:30 2019


 

Thanks in advance.

1 REPLY 1
Teradata Employee

Re: Using TPT/Teradata Access Module for S3 with KMS encrypted objects

Our AWS S3 access module does not yet support KMS.

(It is in the works.)

 

-- SteveF