TPT - Load Operator consumes from a named pipe provided by a JNI process

Teradata Applications
Enthusiast

TPT - Load Operator consumes from a named pipe provided by a JNI process

Hi Teradata Masters,

I'm trying to provide data to be consumed by a Load Operator in my TPT script throught a named pipe.

My DATACONNECTOR Producer: (all runs local on Windows)

    DEFINE OPERATOR PIPE_READER()

    DESCRIPTION 'Define opcoes de leitura de arquivos'

    TYPE DATACONNECTOR PRODUCER

    SCHEMA T701S020_SCHEMA

    ATTRIBUTES

    (

          VARCHAR FileName                 = '\.\pipe\mynamedpipe'

         , VARCHAR AccessModuleName         = 'np_axsmod.dll'

        , VARCHAR AccessModuleInitStr

        , VARCHAR OpenMode              = 'Read'

        , VARCHAR Format                 = 'TEXT'

    );

My Java application gets the file (one line tested txt file) and creates the pipe, "mynamedpipe", and write on it.

Java must use the JNI to handle named pipes, it load a C compiled DLL which includes a JNI library.

The Java application creates the named pipe e wait for the other application to connect on it.

Both sides of the pipe connected, then Java write while the other side read.

I have tested this named pipe connection with other Java application and all works fine, but when I run my TPT script, the pipe don't connect.

- If runs Java app first (create and wait to write), then, run TPT (Load operation):

TPT exit with errorcode 12, and viewing the log with tlogview the error are descripted:

    PIPE_READER: TPT19434 pmOpen failed. General failure (34): 'Could not open the requested Pipe, "\.\pipe\mynamedpipe", because:  Acesso negado.

Obs.: "Acesso negado" = "Access denied"

- If runs TPT first, then Java app: (I waited for TPT reach the pipe read step, to run Java app)

TPT waits forever while Java app returns an error (an unknowable error =\ )

- If runs TPT first, then runs a modified read only of Java app: (I waited for TPT reach the pipe read step, to run Java app non-pipe-creation version)

TPT waits forever while Java app return an error: "Access denied"

On my C code which I had compile my DLL to Java, have nulls to security required parameters, as I have seen between C, JNI and pipe forums...

In my Google searchs, on over internet, seems which anyone had tried to do this...

Someone help me? (and forgive my horrible grammar...)

8 REPLIES
Enthusiast

Re: TPT - Load Operator consumes from a named pipe provided by a JNI process

It's simple!

My concept of pipes were wrong!

You don't need to create the pipe, the TPT creates it for you.

I tried to Read too, but I tried with the wrong Reader class (RandomAccessFile).

My working PipeLoader constructor:

private FileOutputStream pipe;

public PipeLoader() {

while(true) {

try {

getLogger().info("Trying to connect to " + getConnectionUrl());

pipe = new FileOutputStream(getConnectionUrl());

} catch (FileNotFoundException e) {

getLogger().debug(e.getMessage());

}

}

 

try {

Thread.sleep(6000);

} catch (InterruptedException iExc) {

iExc.printStackTrace();

}

continue;

}

 

getLogger().info(getConnectionUrl() + " conectado.");

break;

}

 My class PipeLoader implements Runnable, and in my run() method I put all my logics and writes:

String line = "";

BufferedReader br = ...

while((line = br.readLine()) != null) {

pipe.write(line.getBytes());

pipe.write("\n".getBytes());

}

Obs.: I can create the pipe from Java using JNI, but maybe, I never will use this.

Is this.

Re: TPT - Load Operator consumes from a named pipe provided by a JNI process

Hi Eric,

How did you figure out that TPT internally provide named pipe files ?

Re: TPT - Load Operator consumes from a named pipe provided by a JNI process

Hi Teradata Team,

I am using TPT v14.10 (64 bit) and trying to read data from windows named pipe. I have a Java Application that loads data on to the Windows pipe (using JNI).

(1)  If I run my Java application first that creates pipe  -> connects to pipe -> waits on TPT end to connect and then run the TPT script, I get the error:

Could not open the requested Pipe, "\.\pipe\mynamedpipe04", because:  Access is denied

(2) I tried the solution given by Eric -> Running the script first that will create pipe and wait on it .. and then on the Java end write to the pipe using output stream. But this solution is working at my end only for the standalone fastload utility. Incase of TPT, if  I run the TPT script first, I get the below error:

pipe_reader: TPT19008 DataConnector Producer operator Instances: 1
pipe_reader: TPT19003 ECI operator ID: pipe_reader-5252
pipe_reader: TPT19127 WARNING file '\.\pipe\mynamedpipe06' not processed.
pipe_reader: TPT19003 The system cannot find the file specified.
pipe_reader: TPT19003 TPT Exit code set to 12.

This is the DATACONNECTOR definition in my TPT Script:

DEFINE OPERATOR pipe_reader

TYPE DATACONNECTOR PRODUCER

SCHEMA employee_schema

ATTRIBUTES

(

VARCHAR AccessModuleName = 'np_axsmod.dll',

AccessModuleInitStr = 'log_directory=E:\.......\Logs  log_level=6   block_size=1355000  pipe_wait=9000000 signature_check=0',

VARCHAR  FileName = '\.\pipe\mynamedpipe',

VARCHAR  Format   = 'Delimited',

VARCHAR  OpenMode = 'Read',

VARCHAR  TextDelimiter =','

);

Can TPT not connect to an already created pipe by other source or is there any additional configuration needed to do so, for the flow in point(1) above ? 

If TPT creates pipe internally, then what can possibly the reason for flow in point(2) not working.

Please help me out here.
Teradata Employee

Re: TPT - Load Operator consumes from a named pipe provided by a JNI process

What (exact) version of TPT 14.10 are you running?

-- SteveF
Teradata Employee

Re: TPT - Load Operator consumes from a named pipe provided by a JNI process

Please refer to the Access Module Reference Manual.

It says:

Note: On Windows systems

The access module creates them automatically, and they are destroyed when the access module closes them.

The reader process must be running and in the wait mode for a pipe read operation before you launch the writer process.

This means you start the TPT job first.

One thing I notice is that the error message says this:

pipe_reader: TPT19127 WARNING file '\.\pipe\mynamedpipe06' not processed.

But the script has this for the pipe name:

VARCHAR  FileName = '\.\pipe\mynamedpipe',

-- SteveF

Re: TPT - Load Operator consumes from a named pipe provided by a JNI process

Steve, Thanks for your response on this.

Exact version of TPT is:      14.10.0.1   

NamedPipesAccessModule is:    14.10

The difference in pipe name in script and error is a copy paste thing. Please ignore it. I am getting the error for the same pipename that is defined in the script.

I am starting the TPT job first but I am getting this error:

pipe_reader: TPT19008 DataConnector Producer operator Instances: 1

pipe_reader: TPT19003 ECI operator ID: pipe_reader-7312

pipe_reader: TPT19127 WARNING file '\.\pipe\mynamedpipe' not processed.

pipe_reader: TPT19003 The system cannot find the file specified.

pipe_reader: TPT19003 TPT Exit code set to 12.

Below is from the namedpipes.log:

ERROR 72 Log Initialize: ProcessName=executor.exe, UtilityID=Unknown Client, FileName=E:\.....\TPT\Logs\namedpipes.log, Log_Level=2

Tue May 10 21:03:25 2016 7312 DEBUG 9999 New Log Level = 6

Tue May 10 21:03:25 2016 7312 TRACE 9999 Request Received: pmiPIDMOptID (9)

Tue May 10 21:03:25 2016 7312 TRACE 9999 Request Received: pmiPIDMOptPutA_A (13)

Tue May 10 21:03:25 2016 7312 ERROR 9999 Attribute name: 'TBR_OP_HANDLE' unknown

Tue May 10 21:03:25 2016 7312 TRACE 9999 Request Received: pmiPIDMOptPutA_A (13)

This is my complete script:

DEFINE JOB employeeJobFload

Description 'PERFORM TPT FASTLOAD READING NAMED PIPE'   

(

DEFINE SCHEMA employee_schema

Description 'SOURCE FILE SCHEMA DEFINITION'  

(

emp_id VARCHAR(50),

emp_name VARCHAR(50)

);

DEFINE OPERATOR setup_employee_tables

TYPE DDL

ATTRIBUTES

(

VARCHAR TdpId          = 'xxx.xxx.xx.xxx',

VARCHAR UserName       = 'dbc',

VARCHAR UserPassword   = 'dbc',

VARCHAR ErrorList = '3807'

);

DEFINE OPERATOR pipe_reader

TYPE DATACONNECTOR PRODUCER

SCHEMA employee_schema

ATTRIBUTES

(

VARCHAR AccessModuleName = 'np_axsmod.dll',

AccessModuleInitStr = 'log_directory=E:\.....\TPT\Logs log_level=6',

VARCHAR  FileName = '\.\pipe\mynamedpipe',

VARCHAR  Format   = 'Delimited',

VARCHAR  OpenMode = 'Read',

VARCHAR  TextDelimiter =','

);

DEFINE OPERATOR fastload_employee

TYPE LOAD

SCHEMA *

ATTRIBUTES

(

VARCHAR TdpId          = 'xxx.xxx.xx.xxx',

VARCHAR UserName       = 'dbc',

VARCHAR UserPassword   = 'dbc',

VARCHAR TargetTable  =  'tdwm.employee',

VARCHAR ErrorTable1    = 'tdwm.employee_err1',

VARCHAR ErrorTable2    = 'tdwm.employee_err2',

VARCHAR LogTable = 'tdwm.employee_log'

);

STEP stSetup_Tables

(

APPLY 

('DROP TABLE   tdwm.employee_err1;'),

('DROP TABLE   tdwm.employee_err2;'),

('DROP TABLE   tdwm.employee_log;'),

('DROP TABLE   tdwm.employee;'),

('CREATE TABLE tdwm.employee(

emp_id VARCHAR(50),

emp_name VARCHAR(50)

);')

TO OPERATOR (setup_employee_tables);

);

STEP stLOAD_FILE_NAME

(

APPLY

('INSERT INTO tdwm.employee VALUES

( :emp_id ,

:emp_name

);')

TO OPERATOR (fastload_employee)

SELECT * FROM OPERATOR(pipe_reader);

);

Let me know if any other details are needed on this.

Re: TPT - Load Operator consumes from a named pipe provided by a JNI process

Hi Teradata Team,

I have provided more details in the above post. Can you please take a look at it and let me know why is TPT not creating the defined named pipe, instead throwing an error that The system cannot find the file specified.

Teradata Employee

Re: TPT - Load Operator consumes from a named pipe provided by a JNI process

Please download and install the latest patches of TPT 14.10 and Named Pipe Access Module 14.10.