I can get TPT working from the Teradata Utilities on Linux, but I have to write this massive script to specify each column name and data type.
I'm trying to get it working from Kafka's JDBC connector, so I can just tell it to pull in data from a table incrementally.
Is is possible to use JDBC with TPT? Or do I just use JDBC with FastExport? Is that as good as TPT in terms of speed?
Do I just need to specify TYPE=FASTEXPORT in the connection string?
>>> Is it possible to use JDBC with TPT?
>>> Or do I just use JDBC with FastExport?
The Teradata JDBC Driver offers a JDBC FastExport feature; however, it is not needed and/or not appropriate for all situations.
>>> Is that as good as TPT in terms of speed?
Typically, No. The Teradata JDBC Driver and TPT are not competitors. The two products work quite differently and serve different purposes. In terms of raw throughput for data transfer, TPT will typically offer better performance than the Teradata JDBC Driver. The Teradata JDBC Driver is a driver, not an application. The Teradata JDBC Driver implements the JDBC API Specification, which defines API methods for column-wise input and output of data values. In contrast, TPT is an application, and it works quite differently from the Teradata JDBC Driver. TPT offers file-oriented and/or stream-oriented input and output. In simple terms, the Teradata JDBC Driver must individually process every single row/column value, whereas TPT can process large blocks of multiple data values at a time.
>>> Do I just need to specify TYPE=FASTEXPORT in the connection string?
Specifying the TYPE=FASTEXPORT connection parameter directs the Teradata JDBC Driver to attempt to use JDBC FastExport for queries that are compatible with the FastExport wire protocol.
Keep in mind that FastExport has significant overhead in terms of server-side Teradata Database resources, and significant initiation time for creating multiple TCP socket connections to the Teradata Database. We only recommend the use of JDBC FastExport for queries that will return a large volume of records.
The Kafka JDBC connector can be configured to operate in bulk mode or incremental modes. Generally speaking, using JDBC FastExport makes more sense with the Kafka JDBC connector's bulk mode. JDBC FastExport is probably less useful in conjunction with the Kafka JDBC connector's incremental modes.
Does anyone have a working Teradata -> Kafka configuration they can share?
I have Kafka, Zookeeper, & a Schema Registry up and running.
But when I run the JDBC connector, the consumer of the topic doesn't seem to show any entries.
I'm testing with a single table in the whitelist, and a quickstart-jdbc.properties file like this: