unable to create hive table with Sentry enable

Presto
Teradata Employee

unable to create hive table with Sentry enable

Hello,

 

On my presto cluster with a hive connector (CDH5.7/KERBEROS/SENTRY), I'm not unable to create table (see below the error).

 

Might there be any limits on presto functionality when HDFS permissions are managed by Sentry ?

 

Presto catalog configuration for hive : 

 

connector.name=hive-cdh5

hive.metastore.uri=thrift://bddev01.int.com:9083
hive.metastore.authentication.type=KERBEROS
hive.metastore.service.principal=hive/bddev01.int.com@BIGDATA_DEV
hive.metastore.client.principal=presto@BIGDATA_DEV
hive.metastore.client.keytab=/etc/security/keytabs/presto.princ.keytab

hive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml

hive.allow-drop-table=true

hive.hdfs.authentication.type=KERBEROS
hive.hdfs.impersonation.enabled=true
hive.hdfs.presto.principal=presto@BIGDATA_DEV
hive.hdfs.presto.keytab=/etc/security/keytabs/presto.princ.keytab
hive.security=legacy

 

Error message :

 

presto:querygrid2> CREATE TABLE "test2" ( "vin" VARCHAR );
Query 20170419_134533_00096_ghi3z failed: Failed to set permission on directory: hdfs://bigdata-ns/user/hive/warehouse/querygrid2.db/test2
com.facebook.presto.spi.PrestoException: Failed to set permission on directory: hdfs://bigdata-ns/user/hive/warehouse/querygrid2.db/test2
	at com.facebook.presto.hive.HiveWriteUtils.createDirectory(HiveWriteUtils.java:476)
	at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$Committer.prepareAddTable(SemiTransactionalHiveMetastore.java:951)
	at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore$Committer.access$300(SemiTransactionalHiveMetastore.java:878)
	at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.commitShared(SemiTransactionalHiveMetastore.java:786)
	at com.facebook.presto.hive.metastore.SemiTransactionalHiveMetastore.commit(SemiTransactionalHiveMetastore.java:730)
	at com.facebook.presto.hive.HiveMetadata.commit(HiveMetadata.java:1596)
	at com.facebook.presto.hive.HiveConnector.commit(HiveConnector.java:177)
	at com.facebook.presto.transaction.TransactionManager$TransactionMetadata$ConnectorTransactionMetadata.commit(TransactionManager.java:578)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
	at io.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. user=PrestoJDBC_Driver is not the owner of inode=test2
	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:195)
	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:181)
	at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:178)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6631)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6613)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6532)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermissionInt(FSNamesystem.java:1791)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1771)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:650)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setPermission(AuthorizationProviderProxyClientProtocol.java:174)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:448)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
	at org.apache.hadoop.hdfs.DFSClient.setPermission(DFSClient.java:2445)
	at org.apache.hadoop.hdfs.DistributedFileSystem$28.doCall(DistributedFileSystem.java:1473)
	at org.apache.hadoop.hdfs.DistributedFileSystem$28.doCall(DistributedFileSystem.java:1469)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.setPermission(DistributedFileSystem.java:1483)
	at com.facebook.presto.hive.HiveWriteUtils.createDirectory(HiveWriteUtils.java:473)
	... 12 more
Caused by: org.apache.hadoop.ipc.RemoteException: Permission denied. user=PrestoJDBC_Driver is not the owner of inode=test2
	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:195)
	at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:181)
	at org.apache.sentry.hdfs.SentryAuthorizationProvider.checkPermission(SentryAuthorizationProvider.java:178)
	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:152)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6631)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:6613)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOwner(FSNamesystem.java:6532)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermissionInt(FSNamesystem.java:1791)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setPermission(FSNamesystem.java:1771)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setPermission(NameNodeRpcServer.java:650)
	at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.setPermission(AuthorizationProviderProxyClientProtocol.java:174)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.setPermission(ClientNamenodeProtocolServerSideTranslatorPB.java:448)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

	at org.apache.hadoop.ipc.Client.call(Client.java:1475)
	at org.apache.hadoop.ipc.Client.call(Client.java:1412)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy169.setPermission(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setPermission(ClientNamenodeProtocolTranslatorPB.java:364)
	at sun.reflect.GeneratedMethodAccessor796.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy170.setPermission(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.setPermission(DFSClient.java:2443)
	... 17 more

 

Tags (2)
2 REPLIES
Teradata Employee

Re: unable to create hive table with Sentry enable

For information, the same command with the same user works in hive :

 

$ klist
Ticket cache: FILE:/tmp/krb5cc_1033
Default principal: PrestoJDBC_Driver@BIGDATA_DEV

Valid starting     Expires            Service principal
04/19/17 12:27:03  04/20/17 12:27:03  krbtgt/BIGDATA_DEV@BIGDATA_DEV
	renew until 04/23/17 16:27:21
$ beeline -u "jdbc:hive2://bddev01.int.com:10000/querygrid2;principal=hive/bddev01.int.com@BIGDATA_DEV" Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 Java HotSpot(TM) 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 scan complete in 4ms Connecting to jdbc:hive2://bddev01.int.com:10000/querygrid2;principal=hive/bddev01.int.adeo.com@BIGDATA_DEV Connected to: Apache Hive (version 1.1.0-cdh5.8.2) Driver: Hive JDBC (version 1.1.0-cdh5.8.2) Transaction isolation: TRANSACTION_REPEATABLE_READ Beeline version 1.1.0-cdh5.8.2 by Apache Hive

0: jdbc:hive2://bddev01.int.com:10000/qu> CREATE TABLE querygrid2.test2 ( vin string );
INFO : Compiling command(queryId=hive_20170419163939_72446a40-4950-4fb2-aa87-0be50feb5a0d): CREATE TABLE querygrid2.test2 ( vin string )
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=hive_20170419163939_72446a40-4950-4fb2-aa87-0be50feb5a0d); Time taken: 0.236 seconds
INFO : Executing command(queryId=hive_20170419163939_72446a40-4950-4fb2-aa87-0be50feb5a0d): CREATE TABLE querygrid2.test2 ( vin string )
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20170419163939_72446a40-4950-4fb2-aa87-0be50feb5a0d); Time taken: 0.069 seconds
INFO : OK
No rows affected (0.33 seconds)

0: jdbc:hive2://bddev01.int.com:10000/qu> describe querygrid2.test2;
INFO : Compiling command(queryId=hive_20170419163939_a1fd3ed5-4bc6-4215-9467-b6e791356226): describe querygrid2.test2
INFO : Semantic Analysis Completed
INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col_name, type:string, comment:from deserializer), FieldSchema(name:data_type, type:string, comment:from deserializer), FieldSchema(name:comment, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20170419163939_a1fd3ed5-4bc6-4215-9467-b6e791356226); Time taken: 0.197 seconds
INFO : Executing command(queryId=hive_20170419163939_a1fd3ed5-4bc6-4215-9467-b6e791356226): describe querygrid2.test2
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20170419163939_a1fd3ed5-4bc6-4215-9467-b6e791356226); Time taken: 0.014 seconds
INFO : OK
+-----------+------------+----------+--+
| col_name | data_type | comment |
+-----------+------------+----------+--+
| vin | string | |
+-----------+------------+----------+--+
1 row selected (0.246 seconds)

 

Teradata Employee

Re: unable to create hive table with Sentry enable

From this error message below

 

 

Caused by: org.apache.hadoop.security.AccessControlException: Permission denied. user=PrestoJDBC_Driver is not the owner of inode=test2 at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkOwner(DefaultAuthorizationProvider.java:195) 

it appears that Presto is accessing HDFS as the user "PrestoJDBC_Driver". Are you setting this user name explicitly in the connection string (assuming you're using the JDBC driver)? Can you share the command you're using to connect to Presto (or the connection string)?