Adding CR/LF to huge datafiles

Database
Enthusiast

Adding CR/LF to huge datafiles

Hello,

I am trying to load data from Pipe delimited flat files (size from 1.5Gb to 6GB) to a table in our database. I tried MLOAD. After reading all the records in the smallest file (1.5GB), it threw an error 'EOD encountered before end of record'. I was able to fix it by opening the file in Textpad and adding a CR at the very end of the file (i went to the last line in the file and pressed an enter).  I am not able to do the same with the other files as they are too huge to open in my computer. Is there a way in which we can handle it at the script level?  Any suggestions will be much appreciated.

Thank you

Naren

Sample Script:

.LOGTABLE ud122.data_q4_2012_log;

.LOGON tdpid/112,pwd;

.begin import mload

tables UD122.data_Q4_1

Errortables

ud122.data_Q4_ERROR1

ud122.data_Q4_ERROR2

CHECKPOINT 500000

ERRLIMIT 50      

SESSIONS 25

AMPCHECK ALL;

.layout olblayout;

.FIELD IDFIELD * varchar(10);

.FIELD CODEFIELD * varchar(10);

.dml label olbodml;

INSERT INTO UD122.data_Q4_1(

:IDFIELD,

:CODEFIELD

);

import infile "c:\uploading\abc.txt"

format VARTEXT '|'

layout olblayout

apply olbodml;

.end mload;

.logoff;

3 REPLIES
Enthusiast

Re: Adding CR/LF to huge datafiles

an awk-command, which is simply printing every line adds the LF after last line.  But I'm not sure if the size of the file might  cause problems with awk..

> awk '{print $0}' < infile.txt >outfile.txt

Senior Supporter

Re: Adding CR/LF to huge datafiles

you might try to use a perl script with write append.

Didn't test it but the script would be something like

 #!/usr/local/bin/perl

 open (MYFILE, '>>yourFileName.dat') || die("This file will not open!");

 print MYFILE "\n";

 close (MYFILE); 

Enthusiast

Re: Adding CR/LF to huge datafiles

Hi,

Thank you for the suggestions. I used a File Splitter and split the files in to 1GB sizes. Now, I am able to open the files in Textpad and add a CR at the end. I did have to merge the truncated row (last record in a file and first record in the next file). I will update on my progress soon. 

Regards

Naren