I am testing Named pipe option for TPTexport and TPTLoad in one script by taking just 100 records in the source table. However it is hanging at below step(connecting sessions) for hours. I have tried generic named pipe and also access module 'np_axsmod.so' with no luck. Have used 'mknod mypipe p' to create the pipe. Manual suggested fexp with TPTLoad for named pipe. But I want to test this using TPTexp and TPTload in one single script.(Consumer --> export --> Producer -->Load are the sequence of operators I am invoking in ctl). The same works fine if I use an out file (instead of named pipe). Could anyone please suggest what the issue is. Appreciate your help.
$tbuild -f tptexp_loadpi.ctl pi02
Teradata Parallel Transporter Version 14.10.00.10
Please explain what you mean by trying to use the Named Pipe.
If you want to export rows from a Teradata table and load them into another, there is no need for Named Pipes.
You just use the Export and Load operators.
APPLY $INSERT TO OPERATOR ($LOAD)
SELECT * FROM OPERATOR ($EXPORT);
Hi Steve, Thanks. I understand that and I have done exp-load without any file/pipe usage before.The idea behind this is to have Named pipe work for Teradata export and load, and later extend this to Non-Teradta export and Teradata load using Named pipe. (To avaoid intermediate file creations). Hope I made it clear.Kindly get back for any questions.
Fyi, Named pipe works fine when fastexp and TPTload are used. However the same doesn't work with TPTexp and TPTload was used.(it hangs like I said before)
Yes, the Named Pipe will work when using FastExport, and TPT Load because you have a process writing to a pipe and a process reading from a pipe.
In your script example, you have an Export operator pushing data to the consumer operator (the DataConnector operator using the Named Pipe access module), but you do not have a processing reading from the named pipe.
What is the process you will be using for pulling data out of the non-Teradata database?
And which non-Teradata database will you be using?
Why not use the ODBC operator?
Well, I have DATACONNECTOR PRODUCER to read from named pipe (as per the ctl I have shared). Anyway, I have split exp and load ctls and ran them using & in one go. It worked fine.
tbuild -f tptexppi.ctl & tbuild -f tptloadpi.ctl &
I will explore ODBC operator option but my requirement is to pull data from hadoop and vertica. I believe ODBC operator doesn't work in this case(as per prev posts) (1)Please let me know if this is not correct (2)do you suggest any other method other than exp and tptload using namepipe for best throuput during data migration. Thanks.
I am not sure if this will help you, but the DataConnector operator can pull data from Hadoop.
It can extract directly if you are trying to use HDFS.
If you need other methods, then we to integrate with TDCH.
If you do not want to use TDCH or HDFS, then yes you will need your own process to extract from Hadoop and push the data through named pipes.
However, if you are trying to test out Export-to-Named_Pipe-to-Load (having one TPT job do the writing of data to a named pipe and another TPT job reading from that named pipe and loading ino Teradata), I would say that is probably a waste of your time.
Just get your process working that will push data into a named pipe and then use TPT to read from the named pipe and load Teradata.
Thanks for the insights. I have used TDCH for hdfs,hive/hcatalog though not for large datasets and it's a nice tool.Will use other options as well for throughput testing.
1) Could you please tell me which is faster to load teradata : HDFS API vs TDCH-TPT for larger datasets.If any bench mark was already done.
2)Could someone please help find the Sample Script
TPT UserGuide says: "For the sample script that corresponds to this job, see the following script in the sample/userguide directory:
PTS00029: Read HDFS flat file"
What version of the User Guide were you looking at?
I am looking at the 14.10 TPT User Guide and see no reference to PTS00029 in it.
And TPT did not support HDFS in 14.10.