Hadoop profile creation fails

Teradata Studio
Enthusiast

Hadoop profile creation fails

Hello there,

When creating a new Hadoop profile from TD Studio 14.10.01 nothing happens and I get an Unhandled event loop exception logged in the error log view:

java.lang.NullPointerException

at com.teradata.datatools.hadoop.integrator.oozieservice.OozieHadoopService.checkHDFSRoot(OozieHadoopService.java:406)

at com.teradata.datatools.hadoop.integrator.oozieservice.OozieHadoopService.checkHDFSConfiguration(OozieHadoopService.java:386)

at com.teradata.datatools.hadoop.integrator.oozieservice.OozieHadoopService.checkHadoopConfigured(OozieHadoopService.java:321)

at com.teradata.datatools.hadoop.integrator.oozieservice.OozieHadoopService.ping(OozieHadoopService.java:295)

at com.teradata.datatools.hadoop.integrator.HadoopServices.pingHadoopIntegrator(HadoopServices.java:228)

at com.teradata.datatools.hadoop.dialogs.HadoopConnectionDialog$4.run(HadoopConnectionDialog.java:527)

at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)

at com.teradata.datatools.hadoop.dialogs.HadoopConnectionDialog.okPressed(HadoopConnectionDialog.java:506)

at org.eclipse.jface.dialogs.Dialog.buttonPressed(Dialog.java:472)

at org.eclipse.jface.dialogs.Dialog$2.widgetSelected(Dialog.java:624)

at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:240)

at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)

at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4128)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1457)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1480)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1465)

at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1270)

at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3974)

at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3613)

at org.eclipse.jface.window.Window.runEventLoop(Window.java:825)

at org.eclipse.jface.window.Window.open(Window.java:801)

at com.teradata.datatools.hadoop.view.HadoopContainer.handleHadoopAdd(HadoopContainer.java:338)

at com.teradata.datatools.hadoop.view.HadoopContainer$5.widgetSelected(HadoopContainer.java:287)

at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:240)

at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)

at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4128)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1457)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1480)

at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1465)

at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1270)

at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3974)

at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3613)

at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2701)

at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2665)

at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2499)

at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:679)

at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)

at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:668)

at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)

at com.teradata.studio.StudioApplication.start(StudioApplication.java:33)

at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)

at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:344)

at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622)

at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577)

at org.eclipse.equinox.launcher.Main.run(Main.java:1410)

Thanks for helping.

Regards,

Charles

Tags (2)
17 REPLIES
Enthusiast

Re: Hadoop profile creation fails

I forgot to mention this issue prevents me from exporting/importing data from/to Hadoop obviously using Studio

Teradata Employee

Re: Hadoop profile creation fails

Did you download and install the Teradata Connector for Hadoop and Configure Oozie script on your Hadoop System mentioned on the download page ( http://downloads.teradata.com/download/tools/teradata-studio)?

Teradata Employee

Re: Hadoop profile creation fails

The Exception in the first post is the result of a bug in the code that is checking to see that the Hadoop system has been configured to run the import/export jobs. It is indeed a result of the fact that the files that the Configure Oozie script sets up are not present. That script needs to be run before the connection profile can be made.

Enthusiast

Re: Hadoop profile creation fails

Yes indeed. Thanks for pointing this out as I missed this bash script.



Although I overcome this issue I get the following error:



The Oozie service cannot be reached, check the host name and port



I'm pretty sure I have given the right host and port though. I could telnet this hostname on this port.



By the way, for those using CentOS 6.4 you need to modify line 90 of the configureOozie.sh script to:

jarcmd=`ls -1 /usr/jdk64/jdk*/bin/jar 2> /dev/null | tail -1`

 

Thanks,

Charles

Teradata Employee

Re: Hadoop profile creation fails

Is it possible that your Oozie service is not configured to allow your system or user to use the service?

What version of Hadoop are you using? In HDP 1.3.2, the Oozie configuration file oozie-site.xml has properties that allow hosts and users of groups to access the service. The following lines in the oozie-site.xml open the service to everyone:

    <!-- Hue Configuration -->

    <property>

      <name>oozie.service.ProxyUserService.proxyuser.hue.hosts</name>

    <value>*</value>

    </property>

    <property>

      <name>oozie.service.ProxyUserService.proxyuser.hue.groups</name>

      <value>*</value>

    </property>

The value elements can be set to be more restrictive. Could this be your issue?

Enthusiast

Re: Hadoop profile creation fails

Thanks for helping.

I haven't installed Hue and I don't really understand what Hue's got to do with tdch if it's indeed the tool used underneath?

It wouldn't make sense to add these properties if I haven't installed Hue, right?

I haven't experienced any issue running data transfer using tdch from command line.

Enthusiast

Re: Hadoop profile creation fails

Ok Chuckbert.

I overcome this issue by settings those properties below in core-site.xml to * as you suggested:

hadoop.proxyuser.oozie.groups

hadoop.proxyuser.oozie.hosts

However it seems that there is another issue if you haven't installed HDFS and Templeton (WebHCat) on the same node since you can only reference the first hostname in the hadoop profile wizard. After that the system assumes that templeton hostname is the same and you are stuck with this error:

Exception occurred when accessing the Hadoop database.


Can someone at Teradata confirm or pointing out where I'm wrong?

Thanks,

Charles

Enthusiast

Re: Hadoop profile creation fails

I confirm that you need to install WebHCatalog on the namenode in order to create an Hadoop profile.

Teradata Employee

Re: Hadoop profile creation fails

The Oozie Configure Script does indeed set the WebHCatalog and WebHDFS hosts to be the same as the Name Node. A workaround if either is on a host other than the Name Node is to modify the file in HDFS named "/teradata/oozieworkflows/services.json". You can get it out of HDFS and modify the value for the "WebHCatalog" or "WebHDFS" attribute to be the name or IP address of the host of the service. You then can store the file back in its original location (with the same permissions it had).