Hive table export to Teradata using TDCH failing... due to connection reset

Hadoop
N/A

Hive table export to Teradata using TDCH failing... due to connection reset

when exporting 2billion+ records into teradata from hadoop using TDCH (Teradata Connector for Hadoop) using the below command with "batch.insert",

hadoop jar teradata-connector-1.3.2-hadoop210.jar com.teradata.connector.common.tool.ConnectorExportTool \
-D mapreduce.job.queuename=<queuename> \
-libjars ${LIB_JARS} \
-classname com.teradata.jdbc.TeraDriver \
-url <jdbc_connection_string> \
-username <user_id> \
-password "********" \
-jobtype hive \
-sourcedatabase <hive_src_dbase> \
-sourcetable <hive_src_table> \
-fileformat orcfile \
-stagedatabase <stg_db_in_tdata> \
-stagetablename <stg_tbl_in_tdata> \
-targettable <target_tbl_in_tdata> \
-nummappers 25 \
-batchsize 13000 \
-method batch.insert \
-usexviews false \
-keepstagetable true \
-queryband '<queryband>'

Data is loading successfully into stage table but, then the export job fails before inserting the records in stage table into target table saying, "Connection Reset"

Can someone please help me identify the reason for this, and how to fix this. Thanks a lot in advance !!

1 REPLY
Teradata Employee

Re: Hive table export to Teradata using TDCH failing... due to connection reset

Did you set the environment variables? Export these variables first and run your command.

export TDCH_JAR=/usr/lib/tdch/1.3/lib/teradata-connector-1.3.3.jar

export HADOOP_HOME=/usr/lib/hadoop

export HIVE_HOME=/usr/lib/hive

export HADOOP_CLASSPATH=$HIVE_HOME/conf:$HIVE_HOME/lib/antlr-runtime-3.4.jar:$HIVE_HOME/lib/commons-dbcp-1.4.jar:$HIVE_HOME/lib/commons-pool-1.5.4.jar:$HIVE_HOME/lib/datanucleus-api-jdo-3.2.6.jar:$HIVE_HOME/lib/datanucleus-core-3.2.10.jar:$HIVE_HOME/lib/datanucleus-rdbms-3.2.9.jar:$HIVE_HOME/lib/hive-cli-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-exec-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-metastore-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/jdo-api-3.0.1.jar:$HIVE_HOME/lib/libfb303-0.9.0.jar:$HIVE_HOME/lib/libthrift-0.9.0.jar:$HIVE_HOME/lib/slf4j-api-1.6.4.jar:$HIVE_HOME/lib/hive-common-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-shims-common-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-shims-0.23-0.13.1.2.1.11.0-891.jar:$HIVE_HOME/lib/hive-shims-common-secure-0.13.1.2.1.11.0-891.jar

export HIVE_LIB_JARS=$HIVE_HOME/lib/hive-cli-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-exec-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-metastore-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/libfb303-0.9.0.jar,$HIVE_HOME/lib/libthrift-0.9.0.jar,$HIVE_HOME/lib/jdo-api-3.0.1.jar,$HIVE_HOME/lib/slf4j-api-1.6.4.jar,$HIVE_HOME/lib/hive-common-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-shims-common-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-shims-0.23-0.13.1.2.1.11.0-891.jar,$HIVE_HOME/lib/hive-shims-common-secure-0.13.1.2.1.11.0-891.jar

Example of Export command:

hadoop jar $TDCH_JAR com.teradata.connector.common.tool.ConnectorExportTool -D convert.datetype.format="yyyy-MM-dd HH:mm:ss" -D convert.sourceTz="America/Chicago" -D convert.targetTz="America/Chicago" -libjars $HIVE_LIB_JARS  -classname com.teradata.jdbc.TeraDriver  -url jdbc:teradata://tdt/DATABASE=sandbox -username *****  -password ******  -jobtype hive -method internal.fastload  -sourcetable database.table_name  -sourcefieldnames "****(source filed names)  -targetfieldnames "****(your targe field names)" -hiveconf file:///etc/hive/conf.dist/hive-site.xml -fileformat ORCFile -nummappers 18 -sourcerecordschema "int,int,string,StringFMTTZToTimestampTZ(convert.datetype.format, convert.sourceTz, convert.targetTz),string,string,StringFMTTZToTimestampTZ(convert.datetype.format, convert.sourceTz, convert.targetTz)" &> log/query.outut &