Teradata JDBC fastload hangs after 50K records

Connectivity
Enthusiast

Teradata JDBC fastload hangs after 50K records

Hello Teradata experts,

I'm now working on optimization of loading of data into Teradata.

Our application allows to load big csv files by means of jdbc connection and executeBatch.

I've run the load with a relatively big dataset (about 6M rows and 1.2 Gb of data). I've started with our default parameters:

1) batch size = 1000 records

2) a target table had a primary key constraint

The load was very slow and could take about 3-4 hours.

When I've increased the batch size to 50000, and made a source table "multitable" and NoPI, the loading time decreased dramatically to about 10-12 min, which is already acceptable for us.

I also wanted to evaluate the "FASTLOAD" tool and set the type=FASTLOAD in our jdbc connection string.

However, when I try to load data with the fastload, it seems to hang after 50K loaded records (and it doesn't matter whether the batch size = 1000 or 50000). I also found a similar topic here: http://forums.teradata.com/forum/connectivity/teradata-jdbc-fastload-hangs-after-approximately-50-ba...

What could be the reason for such a behavior and how could we avoid it?

Many thanks for any hint!

Katja

8 REPLIES
Enthusiast

Re: Teradata JDBC fastload hangs after 50K records

For information:

jdbc driver 14.00.00.37, Teradata DB version 15.0.1.06.

Teradata Employee

Re: Teradata JDBC fastload hangs after 50K records

If you are a customer, then please open an incident with Teradata Customer Service.

Enthusiast

Re: Teradata JDBC fastload hangs after 50K records

In my experience with 13.10 & 14.10 calling down FastLoad utility from JDBC connection string end-up in hung state with large record set. We had an Incident to Teradata GSC for 13.10 and the response was use of JDBC for large data set is not recommended in other words its not certified.

This should have been fixed in 15.0 release. Upgrade your jdbc driver to recent 15.x version and give a try. Else I would prefer switching to command line/shall revoke of fastload utility.

Thanks!!

Teradata Employee

Re: Teradata JDBC fastload hangs after 50K records

>>> We had an Incident to Teradata GSC for 13.10 and the response was use of JDBC for large data set is not recommended in other words its not certified.

That is not correct. JDBC FastLoad is fully supported for any size data set.

When loading large data sets, the primary caveat to be aware of is that JDBC FastLoad does not support the checkpoint feature of TPT and the standalone FastLoad utility; therefore, if a JDBC FastLoad operation is interrupted for some reason, it must be restarted from the beginning.

Enthusiast

Re: Teradata JDBC fastload hangs after 50K records

Thanks for clarification on checkpoint feature for JDBC FastLoad. Have Checkpoint enforement is fixed in 15.0

Teradata Employee

Re: Teradata JDBC fastload hangs after 50K records

The addition of checkpoint functionality to JDBC FastLoad would be a new feature, not something to be "fixed".

And No, that functionality is not present in Teradata JDBC Driver 15.0 or 15.10.

Enthusiast

Re: Teradata JDBC fastload hangs after 50K records

Thank you Tom

Enthusiast

Re: Teradata JDBC fastload hangs after 50K records

Thanks you for clarification!