I'm new to TPT so bear with me. I've tried finding information on this in the TPT API manuals as well as the TPT reference and user guide and haven't really found much so I had no choice but to try the forums.... I typically use a third party ETL tool (DataStage) to load to Teradata. DataStage uses the TPT API and all of the applicable libraries to load to Teradata. I have recently started to use TPT (outside of DataStage) and all of the cool utilities that come with it such as TLOGVIEW, TWBSTAT, etc... This got me thinking about why third party ETL tools that use TPT API can't leverage the TPT operational utilities like TLOGVIEW. I have found that logging in DataStage for Teradata related errors isn't very verbose, so in my opinion, it would certainly add value to be able to use TLOGVIEW to provide more verbose logging as well as all the other operational metadata features (i.e. run statistics). Does anyone know if it is possible to enable TPT API applications to use TLOGVIEW or something equivalent?
Ok. I will try to explain why those tool are not available for TPTAPI.
Those tools are only available for the script-based TPT because the operators are controlled by a TPT infrastructure (a set of applications and processes) that control, monitor and coordinate the activities between the operators.
Those tools interect with the TPT infrastructure.
In TPTAPI, we just provide the operators to a customer-written application and it is the customer-written application that is responsible for controlling, monitoring and coordinating the activities of the operators.
Those tools cannot interact with customer-written applications.
As for TLOGVIEW (specifically), with script-based TPT, the operators write out job progress and other operational information (and error information) to disk files (logs). With TPTAPI, we do not write to any logs because we do not have permission to write to those file systems. We provide information to the customer-written applications and it is the responsibility of those applications to log and/or report back to the user.
Thanks for the quick response. I understand what you are saying and agree to a certain extent. But take it for what it's worth...my opinion is that TPT would benefit from providing a standard set of operational metadata (logs, stats, etc...), whether it's being used in a "script-based" mode or as part of TPTAPI. This would provide operational consistency for Teradata utilities and make support simpler, similar to the idea of creating TPT in the first place to replace all of the seperate legacy utilities we used to have (for which support was unique to each one).
A lot of shops today use third party ETL tools such as DataStage or Informatica. If the support for Teradata is different on DataStage vs. Informatica lets say, you could have confusion and frustration on the part of the end users. Beyond that, if ETL vendors choose to make Teradata support difficult, it makes Teradata look like the "bad guy" and it's hard to prove otherwise sometimes. Providing one consistent view of operational metadata could provide Teradata with ammunition/disclaimer in those cases.
I'm not saying that you force ETL vendors to adopt this either. I'm just suggesting that you make it available. That way, the vendor will also have the flexibility to enable or disable this feature. Then let the end customer decide what is easiest by experience and education...
Just my 2 cents...thanks.
I do not disagree with you in principle. However, with TPTAPI, we are just providing the operators to an application so that they have direct control over them. In the original design, with the original partners and ETL vendors, it was stressed upon us that our operators cannot write to disk.
Therefore, we provide the operational metadata to them (through event calls; their app can call the operator and query the information they would like) and it is their responsibility to perform the proper logging/reporting.
I, too, like consistency. But in this case, we are limited by what the ETL vendors will and will not allow us to do.
As to operational metadata in general, we are working on an enhancement to TPT that will load all operational metadata to Teradata tables (similar in theory to DBQL).
This will be supported in script-based TPT and not in TPTAPI (for different technical reasons than those stated above).
BTW, if you are unhappy with the level and detail of statistical and error reporting by the ETL tool, please inform the vendor. We try to push from our end. It helps if the push also comes from their customers.
Thanks for your feedback. I will certainly do what I can to follow up with the ETL tool vendor. I just think it's easy for them to claim that they have no better option. We will get there eventually....hopefully.