FastLoad UTF8 charset issue - for special characters

Tools
Enthusiast

FastLoad UTF8 charset issue - for special characters

Hi,

I am using charset as UTF8 in fastlload, whatever record has special character like box went to Error table (ET). Pls refer the attachment for the sample issued value. Kindly help me how to handle this situation, we need to load data as it is there in source.

Thanks,

Prem.

Tags (3)
5 REPLIES
Teradata Employee

Re: FastLoad UTF8 charset issue - for special characters

In several common fonts, that symbol is used for x'1A' error replacement character. This is an indication that an earlier translation failed (prior to the load to Teradata). Are you certain the character exists in the actual source and was not introduced when the data was extracted?

Teradata Employee

Re: FastLoad UTF8 charset issue - for special characters

The substution character x'1A' is commonly found in both Oracle and IBM DB2 sources. They are not rejected in UTF8 formatted text columns natively for those databases. Teradata, on the other hand, rejects this character outright, and has no Unicode code point for it. You need to replace the 'replacement' character before you can load it into Teradata. There is an access module plug-in that will scan the inbound file for such "un-translatable" characters and replace them with whatever benign character you wish, such as a space. I do not recall the axsmod, but I do know it is undocumented and I had to contact engineering via a opened incident with the GSC.

Good Luck,

Mike

PS, this is more common than one might think and I had to work with another client that was using GoldenGate Replication with this same issue. Luckily, GoldenGate has a mechanism by which you can map characters to other characters that Teradata will accept. In engineering, the main gues working on this is Takashi Takahashi and Bob Hahn.

Teradata Employee

Re: FastLoad UTF8 charset issue - for special characters

Bob has an access module that is used (by PS folks I believe) in the field. We do not (to the best of my knowledge) actually ship it to customers. It is not an official product.

Tak developed a UDF. That UDF is available to the general public. I do not know if Tak uploaded it to the Developer Exchange or if you have to actually ask him for it.

We were going to add Bob's access module functionality into TPT, but when Tak rolled out his UDF, that became a better solution (and the DBS will soon support the Unicode Pass Through feature).

-- SteveF
tak
Teradata Employee

Re: FastLoad UTF8 charset issue - for special characters

I have both translation UDFs and translation access modules in the Unicode tool kit (UTK). 

http://downloads.teradata.com/download/tools/unicode-tool-kit

Although UTK is not a product, it is being supported by GSC. 

Tak

Re: FastLoad UTF8 charset issue - for special characters

Can you connect to the UTK using LDAP?