I have a .txt file which is pipe separated and it has 543895 rows. Duplicate records are avoided by putting a serial number at the end of every row.
Its UTF-8 encoded. On running the fastload script, I had got the error ": EOF encountered before end of record" and on reading another post in this forum I added a new line at the end of my last record, after which the fastload script was running fine. But one of my records is not loaded into table, since it was found to be a duplicate record.
What you descriped is indicating that the serial number process has one dup value.
Can you share the file?