I have many text files with duplicate records. I had been working with fastload but it seems that fastload does not allow duplicates. So now I am trying multiload but getting some technical difficulties.
Issue 1: Data value for each field in my text files (which resembles column value in table) are variable length and terminated by semicolon.
Fastload easily handles this (by vartext ";" and giving the maximum length of each field). But so far I saw in multiload scripts, we have to specify the start position and number of bytes afterwards, Which is now not possible for me since I have 200+ gigabyte data with variable length field values and finding the start postion in hard, moreover I have almost 160+ text files. I believe there might have some ways in multiload to deal with such files.