Hello Teradata experts,
I'm now working on optimization of loading of data into Teradata.
Our application allows to load big csv files by means of jdbc connection and executeBatch.
I've run the load with a relatively big dataset (about 6M rows and 1.2 Gb of data). I've started with our default parameters:
1) batch size = 1000 records
2) a target table had a primary key constraint
The load was very slow and could take about 3-4 hours.
When I've increased the batch size to 50000, and made a source table "multitable" and NoPI, the loading time decreased dramatically to about 10-12 min, which is already acceptable for us.
I also wanted to evaluate the "FASTLOAD" tool and set the type=FASTLOAD in our jdbc connection string.
However, when I try to load data with the fastload, it seems to hang after 50K loaded records (and it doesn't matter whether the batch size = 1000 or 50000). I also found a similar topic here: http://forums.teradata.com/forum/connectivity/tera
What could be the reason for such a behavior and how could we avoid it?
Many thanks for any hint!
In my experience with 13.10 & 14.10 calling down FastLoad utility from JDBC connection string end-up in hung state with large record set. We had an Incident to Teradata GSC for 13.10 and the response was use of JDBC for large data set is not recommended in other words its not certified.
This should have been fixed in 15.0 release. Upgrade your jdbc driver to recent 15.x version and give a try. Else I would prefer switching to command line/shall revoke of fastload utility.
>>> We had an Incident to Teradata GSC for 13.10 and the response was use of JDBC for large data set is not recommended in other words its not certified.
That is not correct. JDBC FastLoad is fully supported for any size data set.
When loading large data sets, the primary caveat to be aware of is that JDBC FastLoad does not support the checkpoint feature of TPT and the standalone FastLoad utility; therefore, if a JDBC FastLoad operation is interrupted for some reason, it must be restarted from the beginning.
Thanks for clarification on checkpoint feature for JDBC FastLoad. Have Checkpoint enforement is fixed in 15.0
The addition of checkpoint functionality to JDBC FastLoad would be a new feature, not something to be "fixed".
And No, that functionality is not present in Teradata JDBC Driver 15.0 or 15.10.