QueryGrid 2.0 - EMR configuration issues

[igormac]

When I try to propagate the QG node connectors/fabric from Viewpoint/QGManager to an EMR (Elastic Map Reduce - Amazon's Hadoop) cluster  on which runs Presto, I get the message “The OS/Platform for <node IP address> is not supported.” This is newest EMR  5.5.0 with our presto distribution and Java 1.8. I wonder if this configuration (QG->Presto on EMR) is currently supported and to what extent... The OS on EMR is Amazon linux, so this may be a problem. Anyone encountered similar issues?

[akshat]

Hi

Couple of questions.

1) What version of Presto are you using? It is always helpful to know the exact version. Please add these in any future posts.

2) Where do you see this error? In QG Manager or somewhere in Presto logs? If this is not in Presto logs probably this is not the right forum.

Here is a list of software requirements http://teradata.github.io/presto/docs/current/system-requirements/software-requirements.html

And setting up Presto on EMR, http://teradata.github.io/presto/docs/current/server-installation-aws-emr-manual.html

[igormac]

The presto version is 176.

Presto is installed on a EMR cluster - no problems with that.

I configure the presto/EMR system as a system in viewpoint, in the QG portlet.

Then I go to configure the nodes of the presto system in the viewpoint QG portlet (when the QG Manager is pushing the node software to the nodes).  

That's when I get the error. The error message (as mentioned in my previous message) - is "The OS/Platform for 172.31.50.147 is not supported."

Looks like when the node packages are pushed to the node, there is a check for a specific OS/Platform for which the node packages are supported.

Since in the QG Manager documentation it is mentioned that it can only be installed on SLES 11 and 12 and RedHat 6 and 7, I presume the same is true for the node packages. Since EMR is running on Amazon Linux (not on SLES or RedHat), my assumption is that this caused the error, and hence Presto on EMR is currently not supported as a target system for QG. Can you confirm this is not supported, or if it is supported, can you provide guidance on what is missing?

[sd186060]

The supported Teradata distrbution of Presto that works with the latest QueryGrid 2.0 is 167t.  See below for additional compatibility information. 

QG 2.02.00.00	Compatibility
QGM	Redhat 6, 7 - SLES 11, 12
Viewpoint	16.00.00.03
Fabric Software	Redhat 6, 7, SLES 11, 12, EMR 5.4 (for Presto connector only) Java 8 required for driver nodes & Hive connector
Connectors:
Presto connector	Presto 167-t
Hive connector	HDP 2.3, 2.4, 2.5 CDH 5.7, 5.8, 5.9
Teradata connector	15.10.04.09, 15.10.05.03 (not on TSS yet), 16.00.00.09
Aster	Aster on Hadoop 7.00.01.01

[igormac]

Thanks.

This confirms that Presto on EMR does not work *with QG* at the moment, because fabric software (that gets installed on the nodes) requires SLES 11/12 or RedHat 6/7. EMR works on Amazon Linux so the QG fabric packages cannot be installed on the nodes (hence the message “The OS/Platform...is not supported.”).

[jasonhughes]

Igor, try installing Presto 167-t.0.2 on EMR 5.4. This configuration has been tested and works. The fabric software is supported on the Amazon Linux version that is installed on EMR 5.4. Note you have to perform installation of Presto rather than have AWS install it for you, as they don't install the Teradata distribution. 

[igormac]

Thanks Jason.

I already did this - I installed Presto on AWS (our presto 176, and you also have to change the Java version to sun/oracle java instead of amazon's open java), and I with the current release I had to *manually* install the nodes software on EMR. There is also a caveat that the tdqg-node.json file has to be generated, cut/pasted and placed manually in the EMR directory rather than placed automatically by Viewpoint (there is an issue opened on this). The 02.02.00.01 patch should solve both of these issues but is not out yet (should be in the next several weeks).

[kb186039]

> our presto 176

There is no "our presto 176". 

Did you mean Presto 167t?
Did you install it using Presto-Admin tool as described at http://teradata.github.io/presto/docs/current/server-installation-aws-emr-manual.html#installation-o... ???

[igormac]

Yes, presto 176t. There is a version of EMR which already has presto pre-installed by Amazon (presto 170), but that's no good. Regarding a "tool", your link points to instructions on how to install 176t on EMR, not to a "tool". Yes, I followed exactly these instructions, but I still had to manually download the installation files, and run the installation scripts on linux - I'm not aware/didn't find any "tool" there. Can you clarify your comments about a "tool"?

[kb186039]

The tool is called "Presto-Admin" :)

http://teradata.github.io/presto/docs/current/presto-admin/emr.html

Could you please send us the output of running a command: "./presto-admin server status" ?

That would help us verify you have things installed properly. Thanks!