BTEQ export to Named Pipe on Linux

Tools & Utilities
Enthusiast

BTEQ export to Named Pipe on Linux

TPT can smoothly export to Named Pipe, but BTEQ seems to have some issue.

 *** Error: The following occurred during an Access Module open:
 The Named Pipes Access Module currently does not support writing..
            Results will not be written to the export file.

The logic looks like below:

amp_list_string=$(seq -s, $amp_start_num $amp_end_num)
mkfifo $output_file_name_with_path
gzip < ${output_file_name_with_path} > ${final_file_name_with_path} &

bteq << EOBTEQ &
.run file logon_info;
SET QUERY_BAND='JobType=Export;SourceTable=${source_table};Chunk=${export_chunk_id};PID=${BASHPID};' FOR SESSION;

.session charset 'utf8'
.session respbuflen MAX1MB
.width 100
.decimaldigits 18
.indicdata on
.largedatamode on

.messageout file=${message_file_name_with_path}
.export indicdata file=${output_file_name_with_path} AXSMOD np_axsmod.so

select * from TdAmpCopy(ON DWH_STG.${source_table}
USING AMPList(
${amp_list_string}
)) AS amp64;

.export reset
.logoff
EOBTEQ

If I do not specify AXSMOD np_axsmod.so, then the BTEQ will first drop the FIFO file, and then create/open a normal file to write. This prevent me from using Named Pipe to send the data parcels to gzip before land the data to disk.

 

Is there any way to let BTEQ EXPORT write/append into a Named Pipe without recreating the file?
Is there any way to export into a single gz file from BTEQ?

@feinholz do you have any recommendation?


Accepted Solutions
Teradata Employee

Re: BTEQ export to Named Pipe on Linux

Please also try using the very latest efix versions rather than the base GCA versions just to be sure something has not already been fixed.

1 ACCEPTED SOLUTION
9 REPLIES
Teradata Employee

Re: BTEQ export to Named Pipe on Linux

Hi. Which BTEQ version are you using ?

Enthusiast

Re: BTEQ export to Named Pipe on Linux

15.10.00.00, 16.00.00.04 and 16.10.00.01

 

I can clearly see that BTEQ first drops the named pipe files which are created before launching BTEQ, and after 2 seconds the new normal files are created.

 

Fortunately and unfortunately, 14.10.00.03 works fine with named pipe. So something is changed after 15.10!!! Smiley Sad Man Sad

Teradata Employee

Re: BTEQ export to Named Pipe on Linux

I'm understanding you to be saying the exact same script works with 14.10.00.03 but not with 15.10.00.00. I'm looking into it.

Enthusiast

Re: BTEQ export to Named Pipe on Linux

Exactly. The simple BTEQ script runs fine in 14.10 but not in 15.10 or 16.00 or 16.10

 

Please investigate. I will try to open a support ticket for this as well.

Teradata Employee

Re: BTEQ export to Named Pipe on Linux

What makes using the "Teradata Access Module for Named Pipes" different from just directly using a named pipe is that it provides optional checkpoint and restart/recovery operations for the reader process for applications that also actually expose controls for handling those events. Since it appears from your use case that you are using BTEQ as the writer process, its use is not actually needed here -- and eliminating its use may help diagnose what is going on. So I put together the following boiled-down test script and ran it against all the latest supported BTEQ versions...

 

#!/bin/sh
rm -f nptest2bteqs.out
rm -f nptest2bteqs.pipe
mkfifo nptest2bteqs.pipe

bteq < nptest2bteqs.pipe > nptest2bteqs.out &

bteq << EOBTEQ
.LOGON mytdpid/myuid,mypw;
.EXPORT REPORT FILE=nptest2bteqs.pipe
.TITLEDASHES OFF ALL
select '.REMARK "hi"' (TITLE '') (CHAR(12));
select '.QUIT       ' (TITLE '') (CHAR(12));
.EXPORT RESET
.LOGOFF
.QUIT
EOBTEQ

 

The result was successful for each version. The nptest2bteqs.pipe "file" listed afterwards as being empty and the contents of the nptest2bteqs.out stdout file each looked as follows.

 

BTEQ nn.nn.nn.nn Thu Nov 16 16:00:44 2017 PID: 6939

+---------+---------+---------+---------+---------+---------+---------+----
.REMARK "hi"
 hi
+---------+---------+---------+---------+---------+---------+---------+----
.QUIT
 *** Exiting BTEQ...
 *** RC (return code) = 0

 

I removed use of the access module, gzip, shell variables and having BTEQ run in the background just to make sure I could reduce variables. Then I reintroduced using gzip and it still worked. I verified the 2 same records existed in the then decompressed file.

 

Could you try to replicate my approach for what you need to get accomplished and then incrementally add in more of your original approach to see what ends up not working and let me know ?

Teradata Employee

Re: BTEQ export to Named Pipe on Linux

I also successfully tested use of INDICDATA format for the export as well as using a background processes for both reader and writer. But again, in a very boiled down manner.

Teradata Employee

Re: BTEQ export to Named Pipe on Linux

Please also try using the very latest efix versions rather than the base GCA versions just to be sure something has not already been fixed.

Enthusiast

Re: BTEQ export to Named Pipe on Linux

Verified. The named pipe works with 15.10 and 16.10 BTEQ.

 

It must be some error in the script earlier.

Teradata Employee

Re: BTEQ export to Named Pipe on Linux

Glad to hear this. Happy Friday ! :)