TPT - Problem loading delimited data with multiple files

Tools & Utilities
Enthusiast

TPT - Problem loading delimited data with multiple files

Hi all,

I got a "EOF encountered before end of record" error when running a TPT script with filename = '*' (file by file runs fine).

I solved a similar problem to skip the first line of each file (skip only the first file fo queue), I had to add a VARCHAR SkipRowsEveryFile = 'Y' to solve this.

I think it's a similar problem, but I have not found the solution.

It's my script:

DEFINE OPERATOR FILE_READER()

   DESCRIPTION 'TERADATA PARALLEL TRANSPORTER DATA CONNECTOR OPERATOR'

   TYPE DATACONNECTOR PRODUCER

   SCHEMA T811_R2_EXCLUIDOS_DETRAF_SCHEMA

   ATTRIBUTES

   (

      VARCHAR DirectoryPath         = 'D:\Programacao\BV-R2\excluidos detraf\Carga de dados',

      VARCHAR FileName                 = '*.REX',

      VARCHAR ArchiveDirectoryPath     = 'D:\Programacao\BV-R2\excluidos detraf\Carregados',

      VARCHAR IndicatorMode         = 'N',

      VARCHAR OpenMode              = 'Read',

      VARCHAR Format                 = 'DELIMITED',

      VARCHAR TextDelimiter         = ';',

      VARCHAR SkipRowsEveryFile     = 'Y',

      INTEGER SkipRows                 = 1

   );

Obs.: ODBC connection

Thanks in advance

7 REPLIES
Teradata Employee

Re: TPT - Problem loading delimited data with multiple files

Did you check to make sure there is an end-of-record marker at the end of the last record of the file?

Sometimes, people forget about that when creating their data files.

You should not have to skip rows in every file for this.

Also, what version of TPT are you using?

I would also request that you upgrade to the latest patch for your release because we are always fixing issues and that may have been an issue that was fixed.

-- SteveF
Enthusiast

Re: TPT - Problem loading delimited data with multiple files

I'm using a 13.10 version

If I don't skip the first line of each/every file I have a "data type mismatch" when TPT read the following header files.

Indeed, some files do not have an EOR on the last record ...

And I don't know where these files came from, they just fall in my directory ... rs

(A good business vision I have)

If I did not find out who generates these files I'll need to treat it, and I do not know if an ECHO. >> File.rex will solve ... (a batch script)

For now, thank you, I'll see if I can solve the problem now that I know.

Teradata Employee

Re: TPT - Problem loading delimited data with multiple files

Please provide me with the "exact" version of the 13.10 release.

This may be an issue we fixed and if I know which version you are running, I can tell you if the fix came in a patch newer than what you are running.

If your files have some type of header as the first record, then yes I can see why you want to skip those records.

That is fine.

I was just not sure if you were skipping those records because you thought it was related to this EOF problem.

-- SteveF
Enthusiast

Re: TPT - Problem loading delimited data with multiple files

How beautiful!!

We can include the newline character in the end of files and the job runs beautifully.

My TPT version is 13.10.00.02, and I'm trying to convince the bosses to adopt TPT in the next projects.

Congratulations to your work supporting TPT.

In some tests with the TPT, I did some simple scripts that were faster than mload/fload with named pipes! I learned to do functional script in 2 days using only these forums and documentation in pdf.

A "perfect world" of the TPT, it would be if it had an API to interface their operators with Java without use the native language (JNI), and if it could unzip these damn zipfiles (gz, rar, zip).

But I think it would be too much to ask...haha

 

Thanks for all.

Teradata Employee

Re: TPT - Problem loading delimited data with multiple files

We will support "gz" and "zip" in TTU14.0.

Also, you should consider upgrading to the latest and greatest efix patch for 13.10.

I believe the latest version is 13.10.00.07.

-- SteveF
Enthusiast

Re: TPT - Problem loading delimited data with multiple files

Hopefully this version will come soon.

Thanks.

Teradata Employee

Re: TPT - Problem loading delimited data with multiple files

That patch has been out for quite a while.

-- SteveF