Make sure your jobs each have unique job names.
Use the -j command line option.
I am exporting multiple tables into files. Currenly doing testing for 23 files.
so i have create one file sample.txt having " tdload -j variable.txt" 23 commands written in job.txt file.
Since i wanted to do extraction simultaneously. I have used GNU parallel.
parallel -j 5 --job_log < job.txt
In every run last command is getting skipped and last file not generated. I have also tried to use some dummy command such as " echo "Hello" in last line of job.txt. But this command executed successfully. One thing I obeserved that one of the file_writer instances picking one table twice and thrice. I am so stuck, please help
Can you provide the content of your job.txt file?
I am not familiar with the "parallel" tool, so cannot comment on why the last command would not be executed.
However, if there is a question about what tdload is, or is not, doing correctly, I will assist with that.
Parallel is nothing but to execute scripts simultaneously. We have a requirement to run Tdload together. I have 23 tables and wanted to pass parameter based on user input how many scripts to be run together.
HERE Is my findings.
If I give " parallel -j 12 < job.txt" all 23 files created.
parallel -j 11 < job.txt -- 21 created
parallel -j 13 <job.txt --23 created.
Job.txt has tdload command.
tdload -j customer.txt
tdload -j aggregation.txt
tdload -j party.txt
similarly 23 statements for 23 tables. Is there any way to make scripts runnign together. I have tried running xargs command also. Same result. Only thing which I have noticed in TPT logs.
Initially few tables were picked correctly but in between TPT is running on same file twice. Please assist
For tbuild command, the -j option specifies jobname. for tdload, -j specifies the job variables file and the jobname is a positional parameter (which should be unique if you are running parallel jobs.. For example:
tdload -j customer.txt load_customer_job
Thanks, it worked.
Is there any option to split the files based on size I mean by mentioning 1GB or 10 MB size limit tdload command
No. With the tbuild command and -C option you can split the output across multiple Data Connector Consumer instances, but even then it would be based only on number of instances and not size of file.
We are currently working on an enhancement that will allow us to have the user provide the max size they would like their output files. I am not sure yet when that feature will be available.
Thanks for the reply.
One more problem I am having. When I am using tdload -f option to generate files and character columns having spaces too in data. Can it be trimmed in tdload command option so that data should be trimmed itself.
See data after extraction.
Please look at second column -country column.
"19","ROMANIA ","3","ular asymptotes are about" "15","MOROCCO ","0","rns. blithely bold courts among"
The Data Connector operator does support a TrimColumns attribute - but I don't believe that attribute is currently exposed via the tdload command, nor in the supplied "simplicity scripting" templates.