TPT 14.10 output to named pipe and then gzip to final files

Tools & Utilities
Enthusiast

Re: TPT 14.10 output to named pipe and then gzip to final files

Hi Steven,

It will be great for TPT to generate files like below (instead of abc.gz-3, abc.gz-12, ...)

  • abc.1.gz
  • abc.3.gz
  • abc.8.gz

I will look forward to a fix in 14.10.00.07+

So for now, would you be able to comment on named pipe usage?

Teradata Employee

Re: TPT 14.10 output to named pipe and then gzip to final files

Not sure on the exact naming convention yet.

We have to be careful with backwards compatibility issues.

What specifically do you want to know about named pipes?

Are you using the named pipe access module?

-- SteveF
Teradata Employee

Re: TPT 14.10 output to named pipe and then gzip to final files

Pipe issues resulting in "getpos" errors was fixed in 14.10.00.003 and 14.00.00.011.

-- SteveF
Enthusiast

Re: TPT 14.10 output to named pipe and then gzip to final files

named pipe access module can be used for READER only. it does not work with WRITER.

my 14.10.00.02 clearly has problem of "getpos" against named pipe. I will ask DBA to get the latest TTU soon.

Thanks a lot, Steven. You are always always prompt and helpful!

Enthusiast

Re: TPT 14.10 output to named pipe and then gzip to final files

Teradata Parallel Transporter Version 14.10.00.03

Job log: /opt/teradata/client/14.10/tbuild/logs/fmcs-2014021817-7.out
Job id is fmcs-2014021817-7, running on xxxxxx
Teradata Parallel Transporter FILE_WRITER: TPT19006 Version 14.10.00.03
FILE_WRITER Instance 1 directing private log report to 'indicdata_writor_log-1'.
FILE_WRITER Instance 2 directing private log report to 'indicdata_writor_log-2'.
FILE_WRITER Instance 7 directing private log report to 'indicdata_writor_log-7'.
FILE_WRITER Instance 3 directing private log report to 'indicdata_writor_log-3'.
FILE_WRITER Instance 6 directing private log report to 'indicdata_writor_log-6'.
FILE_WRITER Instance 8 directing private log report to 'indicdata_writor_log-8'.
FILE_WRITER Instance 4 directing private log report to 'indicdata_writor_log-4'.
FILE_WRITER Instance 5 directing private log report to 'indicdata_writor_log-5'.
Teradata Parallel Transporter Export Operator Version 14.10.00.03
EXPORT_OPERATOR: private log specified: DWH_STG.FACT_abc_SEGMENT_log
FILE_WRITER: TPT19007 DataConnector Consumer operator Instances: 8
FILE_WRITER: TPT19003 ECI operator ID: 'FILE_WRITER-29763'
FILE_WRITER: TPT19222 Operator instance 1 processing file '/var/tmp/fact_abc_segment.fifo-1'.
FILE_WRITER: TPT19222 Operator instance 7 processing file '/var/tmp/fact_abc_segment.fifo-7'.
FILE_WRITER: TPT19222 Operator instance 6 processing file '/var/tmp/fact_abc_segment.fifo-6'.
FILE_WRITER: TPT19222 Operator instance 2 processing file '/var/tmp/fact_abc_segment.fifo-2'.
FILE_WRITER: TPT19222 Operator instance 3 processing file '/var/tmp/fact_abc_segment.fifo-3'.
FILE_WRITER: TPT19222 Operator instance 4 processing file '/var/tmp/fact_abc_segment.fifo-4'.
FILE_WRITER: TPT19222 Operator instance 5 processing file '/var/tmp/fact_abc_segment.fifo-5'.
FILE_WRITER: TPT19222 Operator instance 8 processing file '/var/tmp/fact_abc_segment.fifo-8'.
EXPORT_OPERATOR: connecting sessions
EXPORT_OPERATOR: The RDBMS retryable error code list was not found
EXPORT_OPERATOR: The job will use its internal retryable error codes
FILE_WRITER: TPT19456 (from pmGetPos) PIPE files can not be checkpointed or restarted (56).
FILE_WRITER: TPT19307 Fatal error checkpointing data.
FILE_WRITER: TPT19003 TPT Exit code set to 12.
FILE_WRITER: TPT19456 (from pmGetPos) PIPE files can not be checkpointed or restarted (56).
FILE_WRITER: TPT19456 (from pmGetPos) PIPE files can not be checkpointed or restarted (56).
FILE_WRITER: TPT19307 Fatal error checkpointing data.
FILE_WRITER: TPT19307 Fatal error checkpointing data.
FILE_WRITER: TPT19456 (from pmGetPos) PIPE files can not be checkpointed or restarted (56).
FILE_WRITER: TPT19003 TPT Exit code set to 12.
FILE_WRITER: TPT19456 (from pmGetPos) PIPE files can not be checkpointed or restarted (56).
FILE_WRITER: TPT19003 TPT Exit code set to 12.
FILE_WRITER: TPT19307 Fatal error checkpointing data.

Hi Steven,

I upgraded from 14.10.00.02 to 14.10.00.03 today, but the "pmGetPos" is still bugging me.

Enthusiast

Re: TPT 14.10 output to named pipe and then gzip to final files

/opt/teradata/client/14.10/tbuild/lib $ ls -l
total 8404
drwxr-xr-x 2 root root 4096 Feb 18 17:01 gcc_3.x
-r--r--r-- 1 root root 1511843 Dec 10 13:49 libdataconop.so
-r--r--r-- 1 root root 51145 Dec 10 13:49 libddlop.so
-r--r--r-- 1 root root 144107 Dec 10 13:49 libexportop.so
-r--r--r-- 1 root root 1167191 Dec 10 13:49 libfeomop.so
-r--r--r-- 1 root root 1158858 Dec 10 13:49 libflimop.so
-r--r--r-- 1 root root 120692 Dec 10 13:49 libinsertop.so
-r--r--r-- 1 root root 114687 Dec 10 13:49 libloadop.so
-r--r--r-- 1 root root 34653 Dec 10 13:49 liblogview.so
-r--r--r-- 1 root root 1168077 Dec 10 13:49 libmlimop.so
-r--r--r-- 1 root root 279776 Dec 10 13:49 libodbcop.so
-r--r--r-- 1 root root 602532 Dec 10 13:49 libopcommon.so
-r--r--r-- 1 root root 10992 Dec 10 13:49 liboscmdop.so
-r--r--r-- 1 root root 87686 Dec 10 13:49 libptmin.so
-r--r--r-- 1 root root 22291 Dec 10 13:49 libptmos.so
-r--r--r-- 1 root root 121268 Dec 10 13:49 libptmpi.so
-r--r--r-- 1 root root 41666 Dec 10 13:49 libptmpo.so
-r--r--r-- 1 root root 669703 Dec 10 13:49 libpxcommon.so
-r--r--r-- 1 root root 118413 Dec 10 13:49 libpxextrct.so
-r--r--r-- 1 root root 9203 Dec 10 13:49 libpxicu.so
-r--r--r-- 1 root root 43988 Dec 10 13:49 libschemapop.so
-r--r--r-- 1 root root 136933 Dec 10 13:49 libselectop.so
-r--r--r-- 1 root root 331364 Dec 10 19:25 libstreamop.so
-r--r--r-- 1 root root 83530 Dec 10 13:49 libtableqry.so
-r--r--r-- 1 root root 382681 Dec 10 13:49 libtelapi.so
-r--r--r-- 1 root root 136344 Dec 10 13:49 libupdateop.so

-r-xr-xr-x 1 root root 297275 Dec 10 13:49 coordinator
-r-xr-xr-x 1 root root 843745 Dec 10 13:49 executor
-r-xr-xr-x 1 root root 103189 Dec 10 13:49 planexecutor
-r-xr-xr-x 1 root root 10850 Dec 10 13:49 ptmcmd
-r-xr-xr-x 1 root root 50522 Dec 10 13:49 ptmexec
-r-xr-xr-x 1 root root 40322 Dec 10 13:49 ptmgts
-r-xr-xr-x 1 root root 28655 Dec 10 13:49 ptmjlgr
-r-xr-xr-x 1 root root 47976 Dec 10 13:49 ptmlvs
-r-xr-xr-x 1 root root 13144 Dec 10 13:49 ptmtims
-r-xr-xr-x 1 root root 1854238 Dec 10 13:49 tbuildexe
-r-xr-xr-x 1 root root 57344 Dec 10 13:49 tdloadexe
-r-xr-xr-x 1 root root 621 Dec 10 13:49 tdlog
-r-xr-xr-x 1 root root 20324 Dec 10 13:49 tlogviewexe
-r-xr-xr-x 1 root root 10145 Dec 10 13:49 tptlaunch
Teradata Employee

Re: TPT 14.10 output to named pipe and then gzip to final files

What is the command line you are using to run the TPT job?

-- SteveF
Enthusiast

Re: TPT 14.10 output to named pipe and then gzip to final files

Post the shell commands again:

if [[ -z "$TWB_ROOT" ]]
then
TWB_ROOT=`find /opt/teradata/client -type d -name tbuild | sort -n | head -1`
LD_LIBRARY_PATH=$TWB_ROOT/lib
fi

data_path=/mnt/n001/temp/FACT_ABC_SEGMENT
data_date=`date +'%Y%m%d%H'`
fifo_path=/var/tmp
i=1

for i in $(seq 1 8); do
fifo_pipe=$fifo_path/fact_abc_segment.fifo-${i}
mkfifo $fifo_pipe
gzip -3 < $fifo_pipe > ${data_path}/fact_abc_segment.${i}.fastload.gz &
done

echo "Ready to call TPT"
tbuild -z 60 -h 128M -C \
-f ~/tpt/export.fact_abc_segment.sql \
-v ~/tpt/export.fact_abc_segment.jobvars \
-u "DataFileCount=${i}" \
fmcs-$data_date

TPT EXPORT will write to

  • /var/tmp/fact_abc_segment.fifo-1
  • /var/tmp/fact_abc_segment.fifo-2
  • ...
  • /var/tmp/fact_abc_segment.fifo-8
Teradata Employee

Re: TPT 14.10 output to named pipe and then gzip to final files

Ok. You cannot "checkpoint" when using pipes.

Remove the "-z 60" part of the command and your job should be able to run.

-- SteveF
Enthusiast

Re: TPT 14.10 output to named pipe and then gzip to final files

Validated.

  • 14.10.00.02 has the "GetPos" issue even after removing "-z 60" option
  • 14.10.00.03 works fine without "-z 60" option

The multiple gzip named pipes work again now.

Thanks, Steven.