Continual load using TPTstream from message queue

[jkk2020]

I am exploring the optioanl of never finishing TPT Stream job that does continous load as and when a message received in a messaging system. While it may be an option to schedule a TPT job  (through Ksh/using scheduler)to run every minute to read from the MQ but the idea is not to go in that batch mode.

Currently if I use access module for MQ/kafka, the stream job finishes when it reads messages from queue.

(Explored latencyinterval option but it's for a different purpose).

Could anyone please share some ideas around this. Even if there is no exact solution, we can discuss and debate. 

Thanks.

[Fred]

Use the Timeout option on the Data Connector Producer operator to specify how long to wait on an empty queue before giving up and sending "end of file" to Stream.

[jkk2020]

Thanks Fred. I tried this attribute in TPTstream job to (a) read from a simple file with few records and (b) read from a message queue. In both the cases the job completed the same way it did without the attribute. Is there anything I am missing here.Reference manual did say the valid values are from 1 to 99999. I am using value 3600 for Timeout.Could you pls suggest. 

I am using Teradata Parallel Transporter Version 16.10.00.01 64-Bit.

DEFINE OPERATOR FILE_READER

TYPE DATACONNECTOR PRODUCER

SCHEMA DATA_SCHEMA

ATTRIBUTES

(

VARCHAR PrivateLogName='/home/xyz/log4.txt',

VARCHAR DirectoryPath = '/home/xyz',

VARCHAR FileName = 'file_10rec.txt',

VARCHAR OpenMode = 'Read',

VARCHAR Format = 'Delimited',

VARCHAR TextDelimiter = '||',

VARCHAR EscapeTextDelimiter = '\',

VARCHAR DateForm = 'ANSIDATE',

VARCHAR AcceptExccessColumns = 'y',

INTEGER Timeout = 3600

Thanks,

[jkk2020]

Alright figured that there is an option in MQ message WAIT that has same purpose to wait for incoming messages for given number of seconds(time). Either way (Timeout in TPT  or WAIT in MQ) the job waits for all incoming messages and once timed out, it loads data into table(at once). So we can't name it as continual load. I belive scheduling of load job every minute is the only solution as of now for this requirement. If you have different opinion or would like to share thoughts, kindly do so.