How to extract data from a source that supports REST-based web services

Tools
Tools covers the tools and utilities you use to work with Teradata and its supporting ecosystem. You'll find information on everything from the Teradata Eclipse plug-in to load/extract tools.

How to extract data from a source that supports REST-based web services

Hi,

 

There is a request to extract data from a source, which provides REST bazed services to extract data, and then load it to teradata. Currently we would first need to write a job that would perform this extract and then invoke TPT to load the data.

 

The data can be extracted as CSV or JSON.

 

Is there any option (accessmodule) to do this directly from within the TPT job without having to land the data? Aslo would it handle both CSV and JSON?

 

Thanks,

Amar

2 REPLIES 2
Teradata Employee

Re: How to extract data from a source that supports REST-based web services

Hi Amar,

 

I had a similar project last year, I did mostly as you described.

Ran a python code to gest the rest answer, appended them in a panda dataframe and a tpt load to insert into the database.

I did had to materialize the data on disk, but the volume wasn't an issue (single digit millions rows - 50 MB zipped csv file), so there may be better ways.

 

Go 100% for CSV if you have the choice, it's easier on the code, the loader and also in the database.

Highlighted

Re: How to extract data from a source that supports REST-based web services

Thanks Waldar,

 

Do you still have your code that you can share?

 

Regards,

Amar

amar.yashlaha@teradata.com

 


@Waldar wrote:

Hi Amar,

 

I had a similar project last year, I did mostly as you described.

Ran a python code to gest the rest answer, appended them in a panda dataframe and a tpt load to insert into the database.

I did had to materialize the data on disk, but the volume wasn't an issue (single digit millions rows - 50 MB zipped csv file), so there may be better ways.

 

Go 100% for CSV if you have the choice, it's easier on the code, the loader and also in the database.