TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

Tools & Utilities
Enthusiast

TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

Hi,

I am testing Named pipe option for TPTexport and TPTLoad in one script by taking just 100 records in the source table. However it is hanging at below step(connecting sessions) for hours. I have tried generic named pipe and also access module 'np_axsmod.so' with no luck. Have used 'mknod mypipe p' to create the pipe. Manual suggested fexp with TPTLoad for named pipe. But I want to test this using TPTexp and TPTload in one single script.(Consumer --> export --> Producer -->Load are the sequence of operators I am invoking in ctl). The same works fine if I use an out file (instead of named pipe). Could anyone please suggest what the issue is. Appreciate your help.

=========

Log:

====

$tbuild -f tptexp_loadpi.ctl pi02

Teradata Parallel Transporter Version 14.10.00.10 

Job log: /opt/teradata/client/14.10/tbuild/logs/pi02-8238703.out

Job id is pi02-8238703, running on xyzfileserver

Teradata Parallel Transporter DataConnector Operator Version 14.10.00.10D2D.23102.2

FILE_WRITER: Instance 1 directing private log report to 'dataconnector_log-1'.

Teradata Parallel Transporter Export Operator Version 14.10.00.10

EXPORT_OPERATOR: private log specified: export_log

FILE_WRITER: DataConnector Consumer operator Instances: 1

FILE_WRITER: ECI operator ID: 'FILE_WRITER-11141658'

EXPORT_OPERATOR: connecting sessions

=========

CTL:

=========

DEFINE OPERATOR FILE_WRITER 

TYPE DATACONNECTOR CONSUMER 

SCHEMA contact_schema

ATTRIBUTES

(

VARCHAR PrivateLogName='dataconnector_log',

VARCHAR DirectoryPath ='/home/tmp/',

VARCHAR FileName = 'mypipe',

VARCHAR Format = 'DELIMITED',

VARCHAR OpenMode = 'Write',

VARCHAR IndicatorMode = 'N',

VARCHAR TextDelimiter = '@#$',

VARCHAR EscapeTextDelimiter = '\',

VARCHAR DateForm = 'ANSIDATE'

);       

DEFINE OPERATOR EXPORT_OPERATOR

TYPE EXPORT

SCHEMA contact_schema

ATTRIBUTES

(.....)

STEP STEP_NAME

(

APPLY 

TO OPERATOR (FILE_WRITER[1])

SELECT *  

FROM OPERATOR(EXPORT_OPERATOR[1]);

);

DEFINE OPERATOR FILE_READER

TYPE DATACONNECTOR PRODUCER

SCHEMA contact_schema_load

ATTRIBUTES

(

VARCHAR PrivateLogName='dataconnector_log',

INTEGER ErrorLimit = 1,

VARCHAR DirectoryPath = '/home/tmp/',

VARCHAR FileName = 'mypipe',

VARCHAR AccessModuleName = '/usr/lib/np_axsmod.so',

VARCHAR AccessModuleInitStr = 'ld=. fd=.',

VARCHAR IndicatorMode = 'N',

VARCHAR Format = 'Delimited',

VARCHAR TextDelimiter = '@#$',

VARCHAR OpenMode = 'Read',

VARCHAR DateForm = 'ANSIDATE'

);

DEFINE OPERATOR LOAD_OPERATOR

TYPE LOAD

SCHEMA *

ATTRIBUTES

(...)

Thanks.

11 REPLIES
Teradata Employee

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

Please explain what you mean by trying to use the Named Pipe.

If you want to export rows from a Teradata table and load them into another, there is no need for Named Pipes.

You just use the Export and Load operators.

DEFINE JOB

(

   APPLY $INSERT TO OPERATOR ($LOAD)

   SELECT * FROM OPERATOR ($EXPORT);

);

-- SteveF
Enthusiast

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

Hi Steve, Thanks. I understand that and I have done exp-load without any file/pipe usage before.The idea behind this is to have Named pipe work for Teradata export and load, and later extend this to Non-Teradta export and Teradata load using Named pipe. (To avaoid intermediate file creations). Hope I made it clear.Kindly get back for any questions.

Thanks.

Enthusiast

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

Fyi, Named pipe works fine when fastexp and TPTload are used. However the same doesn't work with TPTexp and TPTload was used.(it hangs like I said before)

Teradata Employee

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

Yes, the Named Pipe will work when using FastExport, and TPT Load because you have a process writing to a pipe and a process reading from a pipe.

In your script example, you have an Export operator pushing data to the consumer operator (the DataConnector operator using the Named Pipe access module), but you do not have a processing reading from the named pipe.

What is the process you will be using for pulling data out of the non-Teradata database?

And which non-Teradata database will you be using?

Why not use the ODBC operator?

-- SteveF
Enthusiast

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

Well, I have  DATACONNECTOR PRODUCER to read from named pipe (as per the ctl I have shared). Anyway, I have split exp and load ctls and ran them using & in one go. It worked fine.

tbuild -f tptexppi.ctl & tbuild -f tptloadpi.ctl &

I will explore ODBC operator option but my requirement is to pull data from hadoop and vertica. I believe ODBC operator doesn't work in this case(as per prev posts) (1)Please let me know if this is not correct (2)do you suggest any other method other than exp and tptload using namepipe for best throuput during data migration. Thanks.

Teradata Employee

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

I am not sure if this will help you, but the DataConnector operator can pull data from Hadoop.

It can extract directly if you are trying to use HDFS.

If you need other methods, then we to integrate with TDCH.

If you do not want to use TDCH or HDFS, then yes you will need your own process to extract from Hadoop and push the data through named pipes.

However, if you are trying to test out Export-to-Named_Pipe-to-Load (having one TPT job do the writing of data to a named pipe and another TPT job reading from that named pipe and loading ino Teradata), I would say that is probably a waste of your time.

Just get your process working that will push data into a named pipe and then use TPT to read from the named pipe and load Teradata.

-- SteveF
Enthusiast

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

Thanks for the insights. I have used TDCH for hdfs,hive/hcatalog though not for large datasets and it's a nice tool.Will use other options as well for throughput testing.

Enthusiast

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

1) Could you please tell me which is faster to load teradata :  HDFS API vs TDCH-TPT  for larger datasets.If any bench mark was already done.

2)Could someone please help find the Sample Script

TPT UserGuide says: "For the sample script that corresponds to this job, see the following script in the sample/userguide directory:

PTS00029: Read HDFS flat file"

However I am not able to find this example in any of the folders in directory '/opt/teradata/client/14.10/tbuild/sample/userguide'

Thanks.

Teradata Employee

Re: TPTEXP - TPTLOAD - using Named pipe hanging at step 'EXPORT_OPERATOR: connecting sessions'

What version of the User Guide were you looking at?

I am looking at the 14.10 TPT User Guide and see no reference to PTS00029 in it.

And TPT did not support HDFS in 14.10.

-- SteveF