How to load large volumes into existing tables in real-time
Hi Gurus, I'm quite new to Teradata. I need to provide a real time solution that will be able to load about 10+ billion records a day into a table with about 150 columns, while users are able to query the table. We are currently able to do about 7-8 billion a day using a different database. And we need to do 10+ so that is why we are investigating Teradata.
The data is received in text files that come in through as the day goes by. We need to load the files as they come in.
Is there any suggestions on where to start? I thought of starting with Fastload, but the fact that it needs a an empty table is a dealbreaker. My table holds about 90 days worth of data. The data is dropped once it reaches a certain age.
Re: How to load large volumes into existing tables in real-time
This is not the right place for that kind of question :-)
You should contact your Teradata salesperson, they will be able to provide you with some info about number of nodes neccessary for that kind of load.
Of course they'll ask some more details like:
How many files must be loaded a day?
Do they arrive constantly all over the day or are there peak times?
Large files, small files?
Regarding "real time", what's the maximum allowed delay according to the SLA?
Based on that one possible solution might be a mini-batch approach: Accumulate the files for x minutes, then FastLoad into a staging table, followed by INSERT SELECT or MERGE into the partitioned target table. User select the data using an access lock.