How-to Skip duplicates

Tools & Utilities

How-to Skip duplicates

Hi Guys,

I'm loading flat files into a staging table using the LOAD operator (no indexes). Once the load is done, i do a select * on that table and insert into a allready populated table with a unique ppi using a DDL Operator.
My problem is that if the insert encounters a duplicate record, the entire batch insert is skipped. So no records are inserted. Is there a way for me to just ignore duplicate records like i can do when using the load operator?

I have changed my script to use an EXPORT and the IMPORT instead of the select * insert into, but that results in slower performance.

Any suggestions would be greatly appreciated.

Tags (1)
3 REPLIES
Highlighted
Enthusiast

Re: How-to Skip duplicates

Rather than kicking of an insert/select *, write the sql with a group by around the PI field(s). It sounds like you're loading into a UPI table, rather than complete duplicate records. The load operator mimics fastload in that it discards duplicate records as it loads. You might also spend some time to understand your source data better, i.e. are those records valid? etc...
Enthusiast

Re: How-to Skip duplicates

Try using fast load. It ignores duplicate rows.
Teradata Employee

Re: How-to Skip duplicates

The Load operator (FastLoad protocol) cannot be used because the target table is already populated. The FastLoad protocol requires an empty table.

Why not use the Update operator? That uses the MultiLoad protocol, might be faster than the "Load to staging table, followed by INS-SEL", unless you are loading a MULTISET table.

-- SteveF