Teradata Meta Data Services (a component that is included in the Teradata Utility Pack) helps an organization locate, describe, and share knowledge. Metadata surrounds raw data with context, transforming it into knowledge.
Teradata Meta Data Services (MDS) is a comprehensive solution for managing metadata in a Teradata warehouse environment. Meta Data Services enables users to load, manage, consolidate, locate and navigate warehouse metadata and its associated business metadata, and is the only metadata management solution optimized for and integrated with Teradata.
MDS provides a blueprint of the warehouse data architecture, enables users to view the contents of the warehouse, and allows staff to understand the impact from schema changes prior to these changes actually being executed. Teradata MDS makes it possible to provide both technical and business context for the data in the warehouse.
Visit the MDS Website at: http://trd.td.teradata.com/mds/ (behind the Teradata firewall)
1. What type of solution is MDS?
Teradata Meta Data Services is a client-server based solution. The Teradata Database System is the server that houses the metadata repository. MDS is a client component providing utilities to create, load, search, modify, and administer the MDS metadata repository, as well as programming and scripting interfaces to read and write into the repository.
The metadata repository is a set of tables, views, and macros stored in a Teradata Database System. MDS is an object-oriented solution so metadata items are represented as objects in the repository. It is a central repository that provides persistent store of metadata using an Object Model Design, and now supports versioning of metadata information. MDS' object oriented design provides straightforward and consistent access of the repository, as well as the ability to easily extend or customize the metadata tracked in the repository. The MDS Engine (a client library) performs the necessary mapping between object to relational mapping, generating optimized Teradata SQL to read or write in to the repository.
2. What are the MDS 13.0 system requirements?
• Microsoft Windows XP Professional SP2, Windows 2000 Professional/Server/Advanced Server SP2, Windows Server 2003 SP2, Windows Server 2008, Windows Vista Business Edition or
• UNIX MP-RAS (3.02 and 3.03.01), or
• Novell SUSE Linux SLES or
• Red Hat Linux 4,5
You will need a Microsoft Windows based system to run the MDS graphical administrative utility.
Teradata Server (required):
• Teradata Database 13.0, 12.0, V2R6.2, V2R6.1, and V2R6.0
Teradata ODBC Driver on Client (required):
• The Teradata ODBC Driver requirement is based on the Teradata database being used.
Web Server (optional):
Required if you use MDS' MetaSurf web application to search and view the contents of the repository.
• Microsoft Internet Information Services 5.0 for Windows Server 2000
• Microsoft Internet Information Services 6.0 for Windows Server 2003
• Microsoft Internet Information Services 7.0 for Windows Server 2008 or Windows Vista
• Other web server for Microsoft Windows that supports Active Server Pages (ASP) and Microsoft COM components
• Web clients require Java Runtime Environment (JRE)
Automatic DIM Update Server (optional):
Required if you use the Auto DIM Update feature to dynamically update the MDS repository when changes to the Teradata Data Dictionary are detected.
• Linux or Windows XP/2000/2003/Vista system to run the DDL Gateway.
• This can be an application node on your Teradata Database System or a separate host that has a network connection to the Teradata Database System
3. What are the components that make up MDS?
MDS provides a number of different administrative and support tools and utilities to create, modify, and manage metadata in the MDS repository.
• MDS Repository - a centrally located and maintained set of tables in a Teradata database accessed via ODBC from the MDS Engine on a client
• MDS Engine - library responsible for handling requests to read and write the repository. Converts object calls to relational form, responsible for managing transactions, security, and generating the SQL to access the tables that make up the MDS repository
• Application Information Metamodels (AIMs) - The Database Information Metamodel (DIM) and the Client Load Metamodel (CLM) come predefined in MDS
• Metacreate - command line program creates the initial MDS repository
• Metadelete - command line program drops all MDS repository tables
• Metaload - command line program for loading, unloading, or synchronizing Teradata database metadata
• MetaManager - Windows based Graphical User Interface (GUI) administrative utility that allows you to load, unload, or synchronize Teradata database metadata, define users and groups of users, define and assign security restrictions, set configuration options, and other tasks.
• Metamigrate - command line program migrates an existing repository to a newer MDS repository version
• Metaclient - program for loading metadata from Teradata FastLoad, MultiLoad, or TPump load scripts and output files. Program can be called from the command line, through MetaManager, or through MetaBrowse
• MetaBrowse - Windows based GUI program that allows you browse and search the repository, create new metamodel definitions, or extend existing classes, properties, or relationships
• MetaSurf - Web based application that provides a view of the metadata in the repository. Provides ability to search, drill-down, navigate, and compare metadata in the MDS repository
• MetaXML - program lets you import data from an XML file into the repository or export data to an XML file. The program can be called from the command line, through MetaManager, or through MetaBrowse
• MetaBridge (CWM-XML bridge) - Provides the ability to exchange metadata with leading business information tools that comply with the Object Management Group (OMG) Common Warehouse Metamodel (CWM) XML Metadata Interchange (XMI) format.
• Metaviews - program installs a set of MDS views into the DBC database. Installing these views eliminates the requirement that a Teradata database user configured for a system object must have SELECT privileges on the DBC tables
• Automatic DIM Update - feature is an optional component that that will dynamically keep the MDS repository synchronized to reflect Data Dictionary changes made to the Teradata Database Systems tracked
• Application Programming Interfaces (API) - MDS interfaces to import and export metadata in the repository. Allow you to create, modify, and delete the metadata objects and Application Information Metamodels (AIM) in the MDS repository. The following interfaces are provided:
o Microsoft Component Object Model (COM)
o Java Interface
• Sample Files - MDS provides a number of different sample files that enable you to load metadata or perform extensions to customize a metadata solution
4. What are Application Information Metamodels (AIMs)?
An AIM defines how metadata is stored and associated to other metadata in the MDS repository. AIMs can be defined to store metadata from a wide variety of sources external to Teradata.
An AIM consists of:
• classes which are types of metadata objects in the repository
• properties which are data fields associated with a particular class of metadata
• relationships which are associations between two classes of metadata. Each relationship defines an origin class and a destination class of metadata
The DIM is a predefined AIM for storing the Teradata Data Dictionary information as well as associated business data. The CLM is a predefined AIM for storing metadata obtained from Teradata client FastLoad, MultiLoad, or TPump utility scripts and output files. The CWM_Metamodel is a predefined AIM based on industry standards to enable interchange of warehouse, business intelligence, and modeling metadata.
5. How do I load Teradata metadata into the repository?
Using the MetaManager administrative utility you can load, unload, or synchronize your Teradata warehouse metadata, or you can manually execute the Metaload utility. The MetaManager graphical utility calls Metaload to extract and load the Teradata warehouse metadata into the repository. You do not have the ability to make a choice for a specific table(s).
6. Does MDS allow me to choose what items in the Teradata warehouse should be loaded in the repository?
Yes, MDS allows you to select a database or database(s) from a Teradata system so the metadata can be tracked in the repository. Begining with MDS 13.0 you also have the ability to select the type of warehouse data to be loaded in the repository for the selected databases being loaded. For example you may select to extract and load metadata for tables and views, but avoid extracting and loading metadata for macros, stored procedures, and triggers.
7. Can MDS manage metadata from more than one Teradata Database System?
Yes, MDS is designed to manage metadata from multiple Teradata Database Systems. You can load Teradata warehouse metadata from production, staging, or validation systems into a single MDS repository and identify differences.
8. How do you keep the repository metadata synchronized with the latest contents of the Teradata warehouse?
You can resynchronize your Teradata metadata any time for one or more databases using MetaManager or by directly invoking the Metaload command line utility. Conversely, when the Automatic DIM Update feature is enabled changes made to the Teradata Data Dictionary are dynamically reflected in the MDS repository.
9. What type of impact analysis can MDS provide?
• What data is maintained in this database?
• Do I have redundant data?
• What views are affected if I drop this column?
• What tables are referenced by this stored procedure, macro, or view?
• Where did this view come from and what table columns does it reference?
• Are there any possible broken views, macros, triggers, or stored procedures?
• What was the last Teradata DDL change and when did it take place?
• Who is accessing the repository and what data are they looking for?
• Are these two databases or tables identical?
10. Does MDS support issuing SQL directly against the MDS repository tables to import, export, or access the metadata in the repository?
Due to its object design, the MDS Repository tables are not defined for human readibility.
We recommend use of the MDS interfaces, which employ the MDS engine, to access the repository. The MDS engine is designed to generate optimized SQL for Teradata including the use of JOINs and issuing LOCKs to prevent deadlocks, and enforces MDS Security.
Using the MDS APIs and the MDS engine guarantees compatibility with future releases of MDS. Compatibility with MDS is not guaranteed and you relinquish the enforcement of MDS' security profiles when you bypass the MDS engine. Issuing direct SQL into the MDS tables is NOT encouraged nor supported.
NEW!! MDS 13.0.1 includes new SQL Views that provide access into certain metadata in the MDS repository by issuing SQL.
11. Does MDS maintain metadata versioning information?
Yes, support for metadata object versioning was introduced in MDS 6.1. When you create your MDS repository or migrate an existing repository you have the option to enable object level versioning. This allows you to maintain historical information of your metadata.
12. Can MDS tell me who is changing or accessing the repository and what data is being accessed?
MDS provides logs for this purpose:
• Audit Trail Log - tracks who, when, what changes made to the Teradata Data Dictionary
• Activity Log - tracks who, when, what was accessed in the repository
13. If I make changes to the Teradata metadata in MDS will these changes be automatically propagated back to the Teradata Data Dictionary?
No. The repository can dynamically detect and track changes to the Teradata Data Dictionary but it does not push Data Dictionary changes back to Teradata.
14. Can MDS coexist in a heterogeneous metadata architecture and interchange metadata with other tools or repositories?
Yes. MDS provides an infrastructure for managing Teradata warehouse metadata, but it also provides open and flexible Application Programming Interfaces (APIs) for creating bridges or tools to interchange metadata with operational systems, ETL tools, Business Intelligence tools, and any other metadata sources.
15. Can MDS interchange metadata with third-party tools?
Yes. MDS includes an XML Bridge (MetaBridge) to provide MDS users a convenient way to exchange metadata with business-intelligence, analytic, and modeling tools through the standards based Common Warehouse Metamodel's XML Metadata Interchange (CWM XMI). As a requirement the tool must be able to export its metadata to CWM XMI XML format.
16. What third-party tools does the MDS XML Bridge support?
• Business Objects Data Integrator 11.5
• Cognos Framework Manager 8.2
• SAS Data Integration Studio 9.1.3
• CA AllFusion ERwin Data Modeler 7.2
• Informatica PowerCenter Repository Manager 8.6 (MDS 13.0.1) NEW!!
17. Can MDS store metadata from DBMS other than Teradata?
MDS is optimized for Teradata but it supports storing metadata from other DBMS. You can load metadata from other sources using the MDS APIs, XML scripting, or manually.
18. Are business descriptions supported?
Yes, MDS supports more than just physical metadata in the DIM:
• Every object in the repository has a description property limited to 1024 characters. You can use this field in tables, columns, views, or any other object to store your business descriptions.
• Secondly, the DIM also includes BusinessEntity and BusinessAttribute classes which contain BusinessDefinition and BusinessNotes properties limited to 12500 characters. If your business descriptions extend beyond 1024 characters you should use these business classes to track your business descriptions.
19. Are ERwin Data Models supported?
CA's AllFusion® ERwin® Data Modeler r7 contains a guided wizard that allows you to move model data directly from ERwin into MDS. When the physical metadata referenced in the ERwin model does not exist in the MDS repository the wizard exports both physical and logical model data. Otherwise it only exports the logical model information and associates it to the physical model (tables, columns, views) already stored in MDS.
To move data from an AllFusion ERwin Data Modeler model into MDS, use the Export to External Format wizard.
20. How can I load business descriptions?
There are several ways:
• Manually using MetaBrowse
• Manually using MetaSurf to edit the description field in repository objects
• Using MDS' XML scripting interface
• Using MDS' C++, Microsoft COM, or Java interfaces to create an application that loads business descriptions
• MDS provides samples, including Excel worksheets, to help load business descriptions
• Directly exporting model data from ERwin into MDS
21. Does MDS provide extensibility to define additional metadata classes, properties, or relationships?
MDS is one of the most extensible and customizable metadta solutions. MDS allows users to amend or extend the DIM & CLM metamodels predefined in the repository as well as the ability to define new metamodels to help track industry, application, tool, process, or any other type of metadata. This flexibility allows MDS to act as a corporate data warehouse repository solution if desired.
22. How do I load metadata values for extensions that I created?
There are several ways:
• Manually using MetaBrowse
• Using MDS' XML scripting interface
• Using MDS' C++, Microsoft COM, or Java interfaces to create an application that loads the metadata
• MDS provides scripting and programming samples that show how to create extensions and populate with data
23. If I define extensions in the DIM, what happens after a database is synchronized to match the latest contents of Teradata?
Extensions to the DIM and their data values should not be affected. The only caveat is if the item being tracked in the repository was itself deleted in Teradata. Let's assume you defined new properties for the table or view classes and loaded values for these. If a table or view being tracked in MDS no longer exists in Teradata then when the database is synchronized the corresponding metadata object is deleted from the MDS repository, which included the extension values provided.
24. What are the requirements for the Java interface?
The Java API is provided to support all features of the C++ class library. The "metajava.jar" library is installed under the MDS installation directory (i.e. %METAHOME%/jdk) along with sample code.
At this time, the only platforms that support the MDS Java API interface are Windows and Linux using MDS 12.0.1. Also, since the interface uses JNI as a bridge to the MDS C++ class libraries (using the MDS metajni DLL), the metajava.jar file cannot be moved to a PC that does not have MDS installed.
The API has been certified against the Sun JRE 1.5.0 (update 6 or later), and the IBM JRE for Eclipse 1.5.0 (build pwi32pdev-20070426a). Both are the 32-bit versions of the J2SE.
Security & Permissions
25. How do you limit access to metadata in the repository?
MDS requires a user identity to authenticate access to the MDS repository. A user identity consists of an MDS user name and password. MDS user identities are separate from Teradata users. MDS users do not need to have a Teradata user logon to access the repository. Instead they have an MDS user logon that only allows them access into the contents of the MDS Repository. An MDS group is a combination of MDS users that have been logically grouped together and given a name.
Access of the metadata objects in the repository can be limited through the use of MDS Security Profiles. By defining Security Profiles and assigning permissions to specific MDS users or groups you can achieve "role-based" access into the metadata.
26. Do MDS Security Profiles reference Teradata security settings?
No, MDS security profiles are separate from Teradata permissions. The use of MDS Security Profiles, along with MDS user and group identities, allow you to effectively restrict access into the metadata repository.
27. If permissions are changed for a database object does it trickle down to the table and column objects associated to the database?
When you assign permissions you have the ability to specify if the permissions should only be assigned to a specific object or if the assignment of permissions should also propagate to related objects. The propagation of permissions to related objects depends on how the relationship was defined.
MetaManager has the option to change the security profile for a database object and to propagate the security profile to all related objects. If the database and related objects have the same security profile, then changing the permissions in the security profile will affect all the objects.
MDS allows you to assign Security Profiles at the following levels:
• AIM - metamodel definition
• Class - any class or category of metadata in an AIM (i.e. database class in DIM)
• Object - specific class instance of metadata (i.e. a database named Inventory)
28. Can administrative privilege be assigned to specific users?
MDS comes with a single pre-defined administrative user, but when you create users you have the option to assign administrative privilege to a new user.
29. Is MDS able to recognize the security access defined in the source system and restrict access based on these?
No. MDS only uses permissions assigned in security profiles to restrict access of objects in the repository.
30. Does MDS provide web access?
MetaSurf is a web based application that provides a view of the metadata in the MDS repository in a format easy to understand and navigate for technical or business users. It provides the ability to perform searches, data drill-down or launch stored documents. MetaSurf is based on Microsoft's Active Server Pages (ASP) technology and it uses the MDS COM interface to access the repository.
31. Does MetaSurf allow both fielded and free-text web based searches?
Yes, the user has several search forms to choose. The business search is a free-text search of various pre-defined classes and properties of metadata. There are also standard and advanced search forms that allow the user to select fields, filters, and search values for general or specific values.
32. Can the MetaSurf web application be customized?
Both look and functionality of MetaSurf can definitely be customized to meet customer needs. For example the look can be changed to displays the customer's colors and logos, or completely changed so the interfaces matches an existing application that users are accustomed to using for metadata access. Furthermore the functionality can be modified to display less technical details, more technical details, or to access other web based applications or enterprise documents.
33. Do new upgrades or releases of MDS come with automated repository conversion routines?
MDS includes a tool called Metamigrate, which converts a previous version MDS repository database to a new version database. This utility automatically updates all existing repository data (AIMs, classes, properties, etc.) to conform to the new version MDS schema.
34. Do I have to recreate my extensions and reload my data when upgrading to a newer release of MDS?
No, the Metamigrate utility does not disrupt extensions and their associated data.
35. Is there any MDS training available?
Teradata provides a 2-day MDS instructor led training workshop via Teradata University. The workshop covers MDS installation on client, configuration, creation of repository, loading of metadata, administration, security options, troubleshooting, as well as searching and navigating the metadata using the MetaSurf web application. Visit Teradata University and look up course 12133 Meta Data Services Implementation.
jainayush007, Teradata Mapping Manager (TMM) is used to create mappings between data and/or requirements. It is common for users to create mappings between source systems and a logical data model of a data warehouse during the design phase, then later use TMM features to copy those mappings to the physical data model for the warehouse. TMM is not a data lineage reporting application but you can access all of the mappings via published views of the TMM metadata using SQL queries. If your TMM mappings document all of the ETL data movements and transformations and/or view references then an external application or query tool could be used to perform data lineage reporting.