Executing multiple TPT scripts simultaneously will yield to better perfomance?

Tools & Utilities
Teradata Employee

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

Make sure your jobs each have unique job names.

Use the -j command line option.

 

-- SteveF
Enthusiast

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

Hi Mate,

 

I am exporting multiple tables into files. Currenly doing testing for 23 files.

 

so i have create one file sample.txt having " tdload -j variable.txt" 23 commands written in job.txt file.

Since i wanted to do extraction simultaneously. I have used GNU parallel.

parallel -j 5 --job_log < job.txt

 

In every run last command is getting skipped and last file not generated. I have also tried to use some dummy command such as " echo "Hello" in last line of job.txt. But this command executed successfully. One thing I obeserved that one of the file_writer instances picking one table twice and thrice. I am so stuck, please help

Teradata Employee

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

Can you provide the content of your job.txt file?

I am not familiar with the "parallel" tool, so cannot comment on why the last command would not be executed.

However, if there is a question about what tdload is, or is not, doing correctly, I will assist with that.

 

-- SteveF
Enthusiast

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

Hi,

 

Parallel is nothing but to execute scripts simultaneously. We have a requirement to run Tdload together. I have 23 tables and wanted to pass parameter based on user input how many scripts to be run together.

 

HERE Is my findings.

If I give " parallel -j 12 < job.txt" all 23 files created.

parallel -j 11 < job.txt -- 21 created

parallel -j 13 <job.txt --23 created.

 

Job.txt has tdload command.

>>>>>>>>>.

tdload -j customer.txt

tdload -j aggregation.txt

tdload -j party.txt

 

similarly 23 statements for 23 tables. Is there any way to make scripts runnign together. I have tried running xargs command also. Same result. Only thing which I have noticed in TPT logs.

Initially few tables were picked correctly but in between TPT is running on same file twice. Please assist

 

Teradata Employee

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

For tbuild command, the -j option specifies jobname. for tdload, -j specifies the job variables file and the jobname is a positional parameter (which should be unique if you are running parallel jobs.. For example:

 

tdload -j customer.txt load_customer_job

Enthusiast

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

Thanks, it worked.

 

Is there any option to split the files based on size I mean by mentioning 1GB or 10 MB size limit tdload command

Highlighted
Teradata Employee

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

No. With the tbuild command and -C option you can split the output across multiple Data Connector Consumer instances, but even then it would be based only on number of instances and not size of file.

Teradata Employee

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

We are currently working on an enhancement that will allow us to have the user provide the max size they would like their output files. I am not sure yet when that feature will be available.

-- SteveF
Enthusiast

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

Thanks for the reply.

One more problem I am having. When I am using tdload -f option to generate files and character columns having spaces too in data. Can it be trimmed in tdload command option so that data should be trimmed itself.

 

See data after extraction.

Please look at second column -country column.

"19","ROMANIA                  ","3","ular asymptotes are about"
"15","MOROCCO                  ","0","rns. blithely bold courts among"

 

Teradata Employee

Re: Executing multiple TPT scripts simultaneously will yield to better perfomance?

The Data Connector operator does support a TrimColumns attribute - but I don't believe that attribute is currently exposed via the tdload command, nor in the supplied "simplicity scripting" templates.