Error with Teradata connector for Hadoop with HCAT -> TD fastload

Tools & Utilities
Enthusiast

Error with Teradata connector for Hadoop with HCAT -> TD fastload

I have an HCategory table:

CREATE TABLE src.t (
msgtype string
, ts string
, source string
, msgclass int
, msgtext string
) PARTITIONED BY (device_range string, device_id string);

and a TD table:


CREATE SET TABLE tgt.t ,FALLBACK ,
NO BEFORE JOURNAL,
NO AFTER JOURNAL,
CHECKSUM = DEFAULT,
DEFAULT MERGEBLOCKRATIO
(
msgtype VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,
ts VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,
source VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,
msgclass INTEGER,
msgtext VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,
device_range VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC,
device_id VARCHAR(255) CHARACTER SET LATIN NOT CASESPECIFIC)
PRIMARY INDEX ( ts );

After exporting a HADOOP_CLASSPATH of:

export HADOOP_HOME=/usr/lib/hadoop
export HIVE_HOME=/usr/lib/hive
export HCAT_HOME=/usr/lib/hcatalog
export TDCH_HOME=/usr/lib/tdch

export HADOOP_CLASSPATH=$HIVE_HOME/conf:\
${HIVE_HOME}/lib/antlr-runtime-3.4.jar:\
${HIVE_HOME}/lib/commons-dbcp-1.4.jar:\
${HIVE_HOME}/lib/commons-pool-1.5.4.jar:\
${HIVE_HOME}/lib/datanucleus-core-3.0.9.jar:\
${HIVE_HOME}/lib/datanucleus-enhancer-3.0.1.jar:\
${HIVE_HOME}/lib/datanucleus-rdbms-3.0.8.jar:\
${HIVE_HOME}/lib/hive-cli-0.11.0.1.3.2.0-111.jar:\
${HIVE_HOME}/lib/hive-exec-0.11.0.1.3.2.0-111.jar:\
${HIVE_HOME}/lib/hive-metastore-0.11.0.1.3.2.0-111.jar:\
${HIVE_HOME}/lib/jdo2-api-2.3-ec.jar:\
${HIVE_HOME}/lib/libfb303-0.9.0.jar:\
${HIVE_HOME}/lib/libthrift-0.9.0.jar:\
${HIVE_HOME}/lib/mysql-connector-java.jar:\
${HIVE_HOME}/lib/slf4j-api-1.6.1.jar:\
${HCAT_HOME}/usr/lib/hcatalog/share/hcatalog/hcatalog-core-0.11.0.1.3.2.0-111.jar:\
${TDCH_HOME}/hive-builtins-0.9.0.jar

And using the teradata connector for hadoop command:


hadoop jar /usr/lib/tdch/teradata-connector-1.2.jar
com.teradata.hadoop.tool.TeradataExportTool
-libjars /usr/lib/hive/lib/hive-cli-0.11.0.1.3.2.0-111.jar,/usr/lib/hive/lib/hive-exec-0.11.0.1.3.2.0-111.jar,/usr/lib/hive/lib/hive-metastore-0.11.0.1.3.2.0-111.jar,/usr/lib/hive/lib/jdo2-api-2.3-ec.jar,/usr/lib/hive/lib/libfb303-0.9.0.jar,/usr/lib/hive/lib/libthrift-0.9.0.jar,/usr/lib/hive/lib/slf4j-api-1.6.1.jar,/usr/lib/tdch/hive-builtins-0.9.0.jar
-classname com.teradata.jdbc.TeraDriver
-url jdbc:teradata://td_server/DATABASE=tgt
-username myuser
-password mypasswd
-jobtype hcat
-method multiple.fastload
-sourcedatabase src
-sourcetable t
-targettable t

I get the following error:


ERROR tool.TeradataExportTool: java.lang.NoClassDefFoundError: org/apache/hcatalog/mapreduce/HCatInputFormat

I have been playing around with following arguments, but no combination helps right now...



-targettableschema "msgtype VARCHAR(255),ts VARCHAR(255),source VARCHAR(255),msgclass INT,msgtext VARCHAR(255),device_range VARCHAR(255),device_id VARCHAR(255)" 
-targetfieldnames "msgtype,ts,source,msgclass,msgtext,device_range,device_id"
-targetfieldcount "7"
-sourcetableschema "msgtype STRING,ts STRING,source STRING,msgclass INT,msgtext STRING,device_range STRING,device_id STRING"
-sourcefieldnames "msgtype,ts,source,msgclass,msgtext,device_range,device_id"
1 REPLY
Teradata Employee

Re: Error with Teradata connector for Hadoop with HCAT -> TD fastload

Forgive us for the delayed response-

The comma separated list of jars specified alongside the '-libjars' option should include the hcatalog jar, take a look at the Environment Variables subsection of the Use Case Examples Chapter of the TDCH Tutorial for more info. The NoClassDefFoundError is usually thrown when the '-libjars' value, or the HADOOP_CLASSPATH env, is not setup correctly. Let me know if this doesn't resolve your issue? Thanks

Ryan