Export large dataset using python

Tools & Utilities
Highlighted
Teradata Employee

Export large dataset using python

In python I am currently using pandas read_sql function to load data frame and then move the data to a csv file. Is there a better way to export data to a file using python.

In various forums i have seen people suggesting to use fastexport, but not sure how to implement it using python.

The dataset size i am trying to export can go upto 3 -4 million rows


Accepted Solutions
Teradata Employee

Re: Export large dataset using python

Hi bonyjohn,

 

This doesn't seem efficient to go from Teradata to pandas to csv, except if you do stuff with pandas that you can't do in Teradata.

 

You can go for a TPT export :

  • cheat a little and call os routines (thus tbuild -f...)
  • write a python tpt module (a dear colleague of mine did that)
  • wait for evolution of any of the Teradata python's package
1 ACCEPTED SOLUTION
3 REPLIES
Teradata Employee

Re: Export large dataset using python


@bonyjohn wrote:

In various forums i have seen people suggesting to use fastexport, but not sure how to implement it using python.


 

The Teradata SQL Driver for Python ( https://pypi.org/project/teradatasql/ ) does not yet support the FastExport wire protocol, but we are working on that.

Teradata Employee

Re: Export large dataset using python

Hi bonyjohn,

 

This doesn't seem efficient to go from Teradata to pandas to csv, except if you do stuff with pandas that you can't do in Teradata.

 

You can go for a TPT export :

  • cheat a little and call os routines (thus tbuild -f...)
  • write a python tpt module (a dear colleague of mine did that)
  • wait for evolution of any of the Teradata python's package
Teradata Employee

Re: Export large dataset using python

@Waldar
Thanks. I also did the same thing. 😀