Presto Kerberos Troubleshooting

Presto
Teradata Employee

Presto Kerberos Troubleshooting

First, see the documentation for Kerberos: http://teradata.github.io/presto/docs/current/security/server.html, http://teradata.github.io/presto/docs/current/security/cli.html, and http://teradata.github.io/presto/docs/current/connector/hive-security.html.

 

If that doesn't help, determine whether you have problems with HTTPS, frontend Kerberos, Kerberos for the Hive connector, or some combination. If after going through this troubleshooting guide you still have trouble, make sure to include verification of the steps in the troubleshooting guide when asking for further assistence, as well as hive.properties (if applicable), config.properties, and the CLI script used to connect to Presto.

 

If you see the following error with the JDBC driver:

[Simba][Presto](100073) Error fetching JSON content: No content to map due to end-of-input at [Source: ; line: 1, column: 1]

You probably have an issue with SSL. See the below SSL Debugging section.

 

Next, try querying the tpch connector in Presto via the Presto CLI. The tpch connector allows you to query in-memory TPC-H data, and can be enabled using a configuration file in the catalog directory with the following contents:

connector.name=tpch

 

Run queries on the tpch connector via the CLI (a sample CLI script can be found here: http://teradata.github.io/presto/docs/current/security/cli.html#presto-cli-execution). 

 

If you get the following exception: java.net.ssl.SSLHandshakeException: General SSLEngine problem

You have a problem with SSL. See the below SSL Debugging section.

 

If it fails with a 404 “cannot connect” error, you have an SSL issue (or maybe the Presto server isn't started). Check ./presto-admin server status, and then see the below SSL Debugging section.

 

If you get the following exception: java.nio.channels.ClosedChannelException

You have an problem with either SSL or Kerberos setup. See the below SSL Debugging section, and then General Kerberos Debugging section and the Frontend Kerberos Debugging.

 

If you have an error like “Error starting query at https://severname.company.com:7778/v1/statement returned an invalid response: JsonResponse{statusCode=401, statusMessage=Unauthorized, headers={Content-Length=[0], Date=[Thu, 30 Jun 2016 21:47:01 GMT], WWW-Authenticate=[Negotiate realm="presto"]}, hasValue=false, value=null}“, or any other issues, you probably have a Kerberos set-up issue. See the below General Kerberos Debugging section and the Frontend Kerberos Debugging section.

 

If the tpch connector works, you probably have a problem with how Kerberos is configured for the Hive connector. See the below General Kerberos Debugging section and the Hive Kerberos Debugging section.

 

General Debugging Tips

1. Remember that you must deploy Presto configuration to all the nodes in the cluster, and any config change requires a restart.

2. It may be helpful to enable debug logging on the Presto server:

Create /etc/presto/log.properties with following line:

com.facebook.presto.server.security=DEBUG

Re-deploy the config file to the whole cluster, and restart.

 

SSL Debugging

Make sure that

1. Your keystore file is the same for the CLI/JDBC and the Presto server

2. The keystore file is readable by the `presto` user

3. If you are connecting via IP address, that your keystore allows for it. Java is particularly strict with SSL security, so unless you add additional parameters to the SSL certificate, in addition to making the IP address the Common Name, it will not work.

e.g.

keytool -genkeypair \
   -alias presto \
   -keyalg RSA \
   -keystore /etc/presto/keystore.jks \
   -keypass password \
   -storepass password \
   -dname "CN=<ip address>, OU=, O=, L=, S=, C=" -ext san=ip:<ip address>

4. If connecting via the Simba JDBC driver, make sure that the Presto remote service name is HTTP (specified by http.server.authentication.krb5.service-name in config.properties)

5. Run keytool -list -v -keystore keystore.jks

Make sure that the CN matches the hostname you're connecting to on the CLI. If you would like to be able to connect either via hostname or via IP, you will need two aliases: one with CN=<hostname> and -ext san=ip:<ip address> and another with CN=<ip address> and -ext san=ip:<ip address>, both in the same keystore file.

 

General Kerberos Debugging

1. Ensure that all keytabs and keystore files are readable by the `presto` user.

2. Ensure that all keytabs and keystore files are on all of the nodes of the cluster in the location specified by the configs, and that you can use kinit -kt with the principals on the cluster

3. Turn on Kerberos debugging by adding

-Dsun.security.krb5.debug=true
-Dlog.enable-console=true

to jvm.config and to the CLI command you're running (see http://teradata.github.io/presto/docs/current/security/server.html#troubleshooting)

4 You can connect to the KDC from the Presto coordinator using telnet

telnet kdc.example.com 88

5. The /etc/krb5.conf file is the same on all of the nodes, and that it contains the same realm that you are using for the principals

6. You have either installed the Java Cryptography Extension Policy Files (http://teradata.github.io/presto/docs/current/security/server.html#java-cryptography-extension-polic...) or configured your /etc/krb5.conf file to use smaller keys.

7. Kerberos relies heavily on DNS resolution, so make sure that a) DNS is set up correctly for your cluster and b) your /etc/hosts file is configured properly, in the form:

<ip address> <fqdn> <optional alias>

8. Only lower case hostnames are permitted.

9. Kerberos is highly dependent on your clocks being synchronized -- make sure that the clocks on all of the Presto servers and on the KDC are the same. If not, you may get a failure like the following (visible when Kerberos debugging is turned on):

>>>KRBError:
     cTime is Fri Nov 14 10:31:21 EST 2008 1226655081000
     sTime is Mon Jan 23 15:12:12 EST 2017 1484921532000
     suSec is 360002
     error code is 25

 

Frontend Kerberos Debugging

1. Ensure that the Presto server keytab contains <service principal>/<fqdn of presto coordinator>, where service name is defined by http.server.authentication.krb5.service-name – if you're trying to connect via JDBC, make sure that http.server.authentication.krb5.service-name is HTTP.

2. Make sure your property names are correct: e.g. the keytab property name is http.server.authentication.krb5.keytab and the service name is http.server.authentication.krb5.service-name.

3. Make sure that you are using the correct keytab file to match the latest service principal name. The proper keytab file for a service principal may change if you generate a keytab file with a random key.

 

So, to do this, after kinit-ing as the client user, try kvno with the Presto service principal (in this case, it's HTTP/presto-master-node). This checks if you can get a service ticket.

 

presto-master-node:/etc/presto # kvno HTTP/presto-master-node

HTTP/presto-master-node@REALM: kvno = 5

presto-master-node:/etc/presto # klist -kt /etc/security/keytabs/HTTP.keytab

Keytab name: FILE:/etc/security/keytabs/HTTP.keytab

KVNO Timestamp Principal

---- ----------------- --------------------------------------------------------

4 06/24/16 19:51:20 HTTP/presto-master-node@REALM

4 06/24/16 19:51:20 HTTP/presto-master-node@REALM

4 06/24/16 19:51:20 HTTP/presto-master-node@REALM

4 06/24/16 19:51:20 HTTP/presto-master-node@REALM

presto-master-node:/etc/presto # klist -kt /etc/security/keytabs/http.keytab

Keytab name: FILE:/etc/security/keytabs/http.keytab

KVNO Timestamp Principal

---- ----------------- --------------------------------------------------------

5 06/30/16 16:29:53 HTTP/presto-master-node@REALM

5 06/30/16 16:29:53 HTTP/presto-master-node@REALM

5 06/30/16 16:29:53 HTTP/presto-master-node@REALM

5 06/30/16 16:29:53 HTTP/presto-master-node@REALM

 

You should use the second keytab file (http.keytab) in config.properties for the server, because the kvno is 5, matching the output from the kvno command.

 

Hive Kerberos Debugging

Ensure that each hive-related keytab is headless (e.g. does not specify a host) and specifies the same principal as in the config file. e.g. hive.metastore.presto.principal should match hive.metastore.presto.keytab, hive.hdfs.presto.principal should match hive.hdfs.presto.keytab, etc.. Then, verify that the keytab file can successfully be used to obtain a ticket for that principal

kinit -kt /etc/presto/presto.keytab presto@EXAMPLE.COM

klist

In Presto versions > 0.157-t, if not using headless keytabs, it is possible to use the _HOST keyword in the config file -- e.g., hive.metastore.presto.principal=hive/_HOST@REALM

In Presto versions <= 0.157-t, if not using headless keytabs, it's necessary to specify a different config file on each node, each with the proper host for the principal (e.g. hive.metastore.presto.principal=hive/node-1@REALM on node-1, hive.metastore.presto.principal=hive/node-2@REALM).

 

Whenever asking for help on Kerberos issues, please include:

* The Presto configuration files (config.properties and hive.properties especially)

* All of the steps taken to debug thusfar

* The CLI script being used to connect to Presto

 

2 REPLIES

Re: Presto Kerberos Troubleshooting

When I run the presto cli jar with kerberos and connector.name=tpch, I get the error:

 

Error running command:

Error reading response from server

 

I am running the presto cli jar as the same user 'presto' that runs the presto server.

Can you please tell what went wrong? Thanks, -Arul

 

Below is the screen output:

 

---

[presto@coordinator:~]$ pwd
/home/presto

[presto@coordinator:~]$ id
uid=701(presto) gid=100(users) groups=100(users)

[presto@coordinator:~]$ hostname -f
coordinator.<DOMAIN>

[presto@coordinator:~]$ cat /etc/krb5.conf

[libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = <REALM>
ticket_lifetime = 24h
dns_lookup_realm = false
dns_lookup_kdc = false
#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5


[domain_realm]

.<DOMAIN>.com = <REALM>

<DOMAIN>.com = <REALM>

.<DOMAININC>.com = <REALM>

<DOMAININC>.com = <REALM>

 

[logging]
default = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
kdc = FILE:/var/log/krb5kdc.log

[realms]
<REALM> = {
admin_server = kdc-vip.<DOMAIN>
kdc = kdc-vip.<DOMAIN>
kdc = kdc01.<DOMAIN>
kdc = kdc02.<DOMAIN>
kdc = kdc03.<DOMAIN>
kdc = kdc04.<DOMAIN>
kdc = kdc05.<DOMAIN>
}


[presto@coordinator:~]$ cat .prestoadmin/coordinator/config.properties
query.max-memory=50GB
node-scheduler.include-coordinator=false
discovery.uri=http://coordinator.<DOMAIN>:8080
discovery-server.enabled=true
http-server.http.port=8080
coordinator=true
query.max-memory-per-node=8GB
http-server.authentication.type=KERBEROS
http.server.authentication.krb5.service-name=presto
http.server.authentication.krb5.keytab=/tmp/keytabs/presto.keytab
http.authentication.krb5.config=/etc/krb5.conf
http-server.https.enabled=true
http-server.https.port=7778
http-server.https.keystore.path=/tmp/keytabs/presto_keystore.jks
http-server.https.keystore.key=xxxxxx

[presto@coordinator:~]$ keytool -list -v -keystore /tmp/keytabs/presto_keystore.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: presto
Creation date: Feb 16, 2017
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=coordinator.<DOMAIN>, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
Issuer: CN=coordinator.<DOMAIN>, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
Serial number: 4919f702
Valid from: Thu Feb 16 23:30:56 PST 2017 until: Thu May 18 00:30:56 PDT 2017
Certificate fingerprints:
MD5: A7:A2:D7:8B:73:15:3B:8A:82:BE:7D:F0:CC:75:A4:29
SHA1: 45:A2:B1:E8:DE:BE:97:0E:61:F7:39:26:84:7B:31:BB:85:7D:7C:85
SHA256: 1F:9E:70:45:A9:BA:54:75:06:8B:D2:2D:21:AB:6E:69:F6:F8:B4:03:8B:9B:2D:9F:23:7D:0E:48:18:56:58:A0
Signature algorithm name: SHA256withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: F5 C7 10 2B C7 EF 4E E2 55 F8 26 5C 8B B0 62 71 ...+..N.U.&\..bq
0010: 53 23 4A 10 S#J.
]
]

 

*******************************************
*******************************************


[presto@coordinator:~]$ klist -kt /tmp/keytabs/presto.keytab
Keytab name: FILE:/tmp/keytabs/presto.keytab
KVNO Timestamp Principal
---- ------------------- ------------------------------------------------------
2 02/21/2017 23:24:28 presto@<REALM>
2 02/21/2017 23:24:28 presto@<REALM>
2 02/21/2017 23:24:28 presto@<REALM>
2 02/21/2017 23:24:28 presto@<REALM>
2 02/21/2017 23:24:44 presto/coordinator.<DOMAIN>@<REALM>
2 02/21/2017 23:24:44 presto/coordinator.<DOMAIN>@<REALM>
2 02/21/2017 23:24:44 presto/coordinator.<DOMAIN>@<REALM>
2 02/21/2017 23:24:44 presto/coordinator.<DOMAIN>@<REALM>


[presto@coordinator:~]$ cat .prestoadmin/catalog/tpch.properties
connector.name=tpch

[presto@coordinator:~]$ cat call_presto_tpch
#!/bin/bash

/usr/java/latest/bin/java \
-Dlog.enable-console=true \
-Dsun.security.krb5.debug=true \
-jar ./presto-cli \
--server https://coordinator.<DOMAIN>:7778 \
--enable-authentication \
--krb5-config-path /etc/krb5.conf \
--krb5-principal presto@<REALM> \
--krb5-keytab-path /tmp/keytabs/presto.keytab \
--krb5-remote-service-name presto \
--keystore-path /tmp/keytabs/presto_keystore.jks \
--keystore-password xxxxxx \
--catalog tpch \
--schema tiny
[presto@coordinator:~]$ ./call_presto_tpch
presto:tiny> Java config name: /etc/krb5.conf
Loaded from Java config
>>> KdcAccessibility: reset
>>> KdcAccessibility: reset
>>> KeyTabInputStream, readName(): <REALM>
>>> KeyTabInputStream, readName(): presto
>>> KeyTab: load() entry length: 74; type: 18
>>> KeyTabInputStream, readName(): <REALM>
>>> KeyTabInputStream, readName(): presto
>>> KeyTab: load() entry length: 58; type: 17
>>> KeyTabInputStream, readName(): <REALM>
>>> KeyTabInputStream, readName(): presto
>>> KeyTab: load() entry length: 66; type: 16
>>> KeyTabInputStream, readName(): <REALM>
>>> KeyTabInputStream, readName(): presto
>>> KeyTab: load() entry length: 58; type: 23
>>> KeyTabInputStream, readName(): <REALM>
>>> KeyTabInputStream, readName(): presto
>>> KeyTabInputStream, readName(): coordinator.<DOMAIN>
>>> KeyTab: load() entry length: 108; type: 18
>>> KeyTabInputStream, readName(): <REALM>
>>> KeyTabInputStream, readName(): presto
>>> KeyTabInputStream, readName(): coordinator.<DOMAIN>
>>> KeyTab: load() entry length: 92; type: 17
>>> KeyTabInputStream, readName(): <REALM>
>>> KeyTabInputStream, readName(): presto
>>> KeyTabInputStream, readName(): coordinator.<DOMAIN>
>>> KeyTab: load() entry length: 100; type: 16
>>> KeyTabInputStream, readName(): <REALM>
>>> KeyTabInputStream, readName(): presto
>>> KeyTabInputStream, readName(): coordinator.<DOMAIN>
>>> KeyTab: load() entry length: 92; type: 23
Looking for keys for: presto@<REALM>
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
Looking for keys for: presto@<REALM>
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
Using builtin default etypes for default_tkt_enctypes
default etypes for default_tkt_enctypes: 18 17 16 23.
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=kdc-vip.<DOMAIN> UDP:88, timeout=30000, number of retries =3, #bytes=156
>>> KDCCommunication: kdc=kdc-vip.<DOMAIN> UDP:88, timeout=30000,Attempt =1, #bytes=156
>>> KrbKdcReq send: #bytes read=717
>>> KdcAccessibility: remove kdc-vip.<DOMAIN>
Looking for keys for: presto@<REALM>
Added key: 23version: 2
Added key: 16version: 2
Added key: 17version: 2
Added key: 18version: 2
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsRep cons in KrbAsReq.getReply presto
Found ticket for presto@<REALM> to go to krbtgt/<REALM>@<REALM> expiring on Thu Feb 23 12:23:44 PST 2017
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbKdcReq send: kdc=kdc-vip.<DOMAIN> UDP:88, timeout=30000, number of retries =3, #bytes=699
>>> KDCCommunication: kdc=kdc-vip.<DOMAIN> UDP:88, timeout=30000,Attempt =1, #bytes=699

presto:tiny> >>> KrbKdcReq send: #bytes read=720
>>> KdcAccessibility: remove kdc-vip.<DOMAIN>
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Subject is readOnly;Kerberos Service ticket not stored
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 241686417
Created InitSecContextToken:
0000: 01 00 6E 82 02 6B 30 82 02 67 A0 03 02 01 05 A1 ..n..k0..g......
0010: 03 02 01 0E A2 07 03 05 00 20 00 00 00 A3 82 01 ......... ......
0020: 78 61 82 01 74 30 82 01 70 A0 03 02 01 05 A1 13 xa..t0..p.......
0030: 1B 11 50 41 59 50 41 4C 53 55 50 50 4F 52 54 2E ..mycompanySUPPORT.
0040: 43 4F 4D A2 35 30 33 A0 03 02 01 00 A1 2C 30 2A COM.503......,0*
0050: 1B 06 70 72 65 73 74 6F 1B 20 6C 76 73 68 64 63 ..presto. lvshdc
0060: 31 32 65 6E 30 30 30 31 2E 6C 76 73 2E 70 61 79 12en0001.lvs.pay
0070: 70 61 6C 69 6E 63 2E 63 6F 6D A3 82 01 1B 30 82 palinc.com....0.
0080: 01 17 A0 03 02 01 12 A1 03 02 01 02 A2 82 01 09 ................
0090: 04 82 01 05 FF A0 74 92 9F 6A 53 2D F8 B6 FC 39 ......t..jS-...9
00A0: 36 FE 5D 03 85 3E 89 DD 56 05 B9 40 73 A2 BB 40 6.]..>..V..@s..@
00B0: 85 AD DA 47 8A 00 33 38 83 87 39 5E F8 54 B9 04 ...G..38..9^.T..
00C0: 01 EC 64 BF 00 DF 11 59 71 DA C5 20 66 F7 BE E1 ..d....Yq.. f...
00D0: 00 69 0B 4F 83 97 98 48 07 C6 78 98 AA 97 E5 A0 .i.O...H..x.....
00E0: 72 66 54 86 DC F2 35 64 A7 CC 14 38 8D F2 B5 D7 rfT...5d...8....
00F0: 99 2E 5A DA 5B 14 BA 66 D4 E2 C3 92 DD 50 42 85 ..Z.[..f.....PB.
0100: 9A 1E 48 DE C7 EB 29 9F F8 C8 44 E9 68 4B A2 DD ..H...)...D.hK..
0110: 8B D6 F5 06 93 65 FE 6C 46 C5 CD 9B CB C1 4A 98 .....e.lF.....J.
0120: E6 3A 85 9F 0B 20 1C 20 82 D8 B6 EE 03 89 92 28 .:... . .......(
0130: 2B 95 4F 2D 3C 8F 99 CE 3C B2 05 25 7B 54 B5 89 +.O-<...<..%.T..
0140: 16 D8 0B EE 63 4E C2 33 81 81 4D 0C 82 3D FB 16 ....cN.3..M..=..
0150: C5 06 85 A3 32 C3 07 EF CD E5 7C 23 A3 A0 AE C8 ....2......#....
0160: 64 DB 14 56 85 52 C1 03 F8 D0 B6 9D C5 2B 2D F0 d..V.R.......+-.
0170: 1D C9 F8 CF F4 75 A0 B1 9F 35 CC 98 B3 67 0F 20 .....u...5...g.
0180: 3F 67 D4 82 73 98 EE 4D 1E 2B E1 D6 67 3A 5F A0 ?g..s..M.+..g:_.
0190: 99 B6 63 4E D8 81 A1 23 6F A4 81 D5 30 81 D2 A0 ..cN...#o...0...
01A0: 03 02 01 12 A2 81 CA 04 81 C7 3B EF B1 66 82 E5 ..........;..f..
01B0: BD CF E4 86 B1 BC 2C 86 95 E6 78 E5 E9 8A 3D 62 ......,...x...=b
01C0: 78 EF CF DD 4B 63 09 8E A0 A9 C3 D2 B2 F8 0C 93 x...Kc..........
01D0: D9 26 5C 8A A5 AD E0 D7 89 81 AB FB 03 07 EF 0A .&\.............
01E0: D8 06 ED 8D EE 32 21 E1 6D C6 54 93 53 88 76 79 .....2!.m.T.S.vy
01F0: 30 91 2D 0F 5A A5 70 82 79 47 DB 7F 7E C2 C3 B3 0.-.Z.p.yG......
0200: 59 01 1F E9 8D 97 0B 6C A9 B0 25 53 E9 F2 8A C8 Y......l..%S....
0210: 0A AE 17 C1 FB 30 D4 0E 36 DA 8D 09 0D 39 1E 83 .....0..6....9..
0220: CE B7 9E ED 0B 21 A5 BC D2 7B 90 A8 7B 2B 90 FD .....!.......+..
0230: 4B 0F 68 EF 90 C8 C7 7A B0 03 7F 25 E6 0F 1F FE K.h....z...%....
0240: 49 61 13 24 EA 43 03 EA 68 B4 4C 15 4D FE DF 7D Ia.$.C..h.L.M...
0250: D5 42 13 66 3A 1D 21 FD F3 F0 AB 61 BA C9 39 AD .B.f:.!....a..9.
0260: CD BD D7 11 1E 27 35 CC 45 32 53 F3 DD 6B 0D A0 .....'5.E2S..k..
0270: C9 .

Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbKdcReq send: kdc=kdc-vip.<DOMAIN> UDP:88, timeout=30000, number of retries =3, #bytes=699
>>> KDCCommunication: kdc=kdc-vip.<DOMAIN> UDP:88, timeout=30000,Attempt =1, #bytes=699
>>> KrbKdcReq send: #bytes read=720
>>> KdcAccessibility: remove kdc-vip.<DOMAIN>
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Subject is readOnly;Kerberos Service ticket not stored
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 989839421
Created InitSecContextToken:
0000: 01 00 6E 82 02 6B 30 82 02 67 A0 03 02 01 05 A1 ..n..k0..g......
0010: 03 02 01 0E A2 07 03 05 00 20 00 00 00 A3 82 01 ......... ......
0020: 78 61 82 01 74 30 82 01 70 A0 03 02 01 05 A1 13 xa..t0..p.......
0030: 1B 11 50 41 59 50 41 4C 53 55 50 50 4F 52 54 2E ..mycompanySUPPORT.
0040: 43 4F 4D A2 35 30 33 A0 03 02 01 00 A1 2C 30 2A COM.503......,0*
0050: 1B 06 70 72 65 73 74 6F 1B 20 6C 76 73 68 64 63 ..presto. lvshdc
0060: 31 32 65 6E 30 30 30 31 2E 6C 76 73 2E 70 61 79 12en0001.lvs.pay
0070: 70 61 6C 69 6E 63 2E 63 6F 6D A3 82 01 1B 30 82 palinc.com....0.
0080: 01 17 A0 03 02 01 12 A1 03 02 01 02 A2 82 01 09 ................
0090: 04 82 01 05 4F 92 CD B9 F5 59 1C FD E7 6C 9C 7F ....O....Y...l..
00A0: C3 75 6A 56 93 F9 05 32 F7 54 31 12 6F EE 68 8B .ujV...2.T1.o.h.
00B0: 0E BC 4C 6A 73 BB 0A 71 7F 02 17 4C 00 B5 2C F9 ..Ljs..q...L..,.
00C0: 4E A0 83 18 67 9B A3 AA 3D 1D 8F 14 DD 51 59 98 N...g...=....QY.
00D0: 7E 3C CF 57 01 C7 4D E3 D8 AA 05 2E 0D FC F2 59 .<.W..M........Y
00E0: 50 3E 2B 1D E2 05 9A FF 8D 65 77 57 E6 1B B5 42 P>+......ewW...B
00F0: 9E E8 5C 64 77 E6 8A E8 47 71 A9 42 52 6F DF 35 ..\dw...Gq.BRo.5
0100: B9 0C A1 33 76 1C CF 91 52 A0 7F 2B 5E 78 72 97 ...3v...R..+^xr.
0110: 18 19 09 F8 B6 E4 06 CD 61 F7 95 A4 29 6D E7 77 ........a...)m.w
0120: 8B D6 60 51 53 C2 B4 FC 7A 43 7A B1 AB CB 7E F9 ..`QS...zCz.....
0130: FC E9 D5 7D 4F 1D 65 48 A2 71 1E 43 CC 7B BD 03 ....O.eH.q.C....
0140: 08 81 63 4A 8D F0 B2 D4 3F AC 5E 52 57 BD 1F 7C ..cJ....?.^RW...
0150: 9F 9D 30 4B 66 C1 DB 94 EC 2C 31 CC F4 C5 3D 89 ..0Kf....,1...=.
0160: D6 74 A7 26 0A 9A EF DD 90 BC FE E1 9E BF E6 AB .t.&............
0170: E0 45 B1 E8 15 6E CA 2F 12 BB 17 05 03 D1 43 65 .E...n./......Ce
0180: EB E7 46 9A 0D 29 9B EB 04 50 26 44 28 95 60 62 ..F..)...P&D(.`b
0190: 45 C6 9C AC 3E 84 D4 7C BE A4 81 D5 30 81 D2 A0 E...>.......0...
01A0: 03 02 01 12 A2 81 CA 04 81 C7 10 32 B0 BA 85 0D ...........2....
01B0: 48 E9 B4 36 E6 25 04 B3 E5 A5 6F 53 F7 99 AE 95 H..6.%....oS....
01C0: F6 59 22 F5 CF 83 2D 65 FF F0 3B 8C 1B 84 E4 DD .Y"...-e..;.....
01D0: 59 A2 62 C7 1D EA D7 83 6E AD DA D7 05 90 CC CB Y.b.....n.......
01E0: 6D C9 67 7F 45 3F 47 E9 22 F7 33 89 5C 7A 87 69 m.g.E?G.".3.\z.i
01F0: FC 1B 9A 82 FB 6C 16 65 71 1C 19 62 55 EE 1D C6 .....l.eq..bU...
0200: B5 4D AC E9 80 CB 72 8D F1 3C FA 0F C6 23 07 42 .M....r..<...#.B
0210: 8D D0 16 75 EA 6B 39 81 5B EC EF F5 10 86 BF B4 ...u.k9.[.......
0220: C9 D2 D1 84 8F 42 95 02 80 BD 4C 5B E2 F6 F7 1B .....B....L[....
0230: 3C 9E A6 95 5D 35 71 72 DC A6 7C 96 F9 53 2D 39 <...]5qr.....S-9
0240: 29 D9 FB F0 2D 5F 89 EC 26 4C 57 2A 5F 26 85 53 )...-_..&LW*_&.S
0250: 8F 37 40 B1 2A 98 B6 00 16 AD 98 5D 37 AD 82 9E .7@.*......]7...
0260: AE AF AD F8 54 4D 1A 8B 2E 90 F3 7D F2 C2 F8 94 ....TM..........
0270: 50 P


presto:tiny> select count(*) from lineitem;
Entered Krb5Context.initSecContext with state=STATE_NEW
Service ticket not found in the subject
>>> Credentials acquireServiceCreds: same realm
Using builtin default etypes for default_tgs_enctypes
default etypes for default_tgs_enctypes: 18 17 16 23.
>>> CksumType: sun.security.krb5.internal.crypto.RsaMd5CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbKdcReq send: kdc=kdc-vip.<DOMAIN> UDP:88, timeout=30000, number of retries =3, #bytes=699
>>> KDCCommunication: kdc=kdc-vip.<DOMAIN> UDP:88, timeout=30000,Attempt =1, #bytes=699
>>> KrbKdcReq send: #bytes read=720
>>> KdcAccessibility: remove kdc-vip.<DOMAIN>
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Subject is readOnly;Kerberos Service ticket not stored
>>> KrbApReq: APOptions are 00100000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 73141858
Created InitSecContextToken:
0000: 01 00 6E 82 02 6B 30 82 02 67 A0 03 02 01 05 A1 ..n..k0..g......
0010: 03 02 01 0E A2 07 03 05 00 20 00 00 00 A3 82 01 ......... ......
0020: 78 61 82 01 74 30 82 01 70 A0 03 02 01 05 A1 13 xa..t0..p.......
0030: 1B 11 50 41 59 50 41 4C 53 55 50 50 4F 52 54 2E ..mycompanySUPPORT.
0040: 43 4F 4D A2 35 30 33 A0 03 02 01 00 A1 2C 30 2A COM.503......,0*
0050: 1B 06 70 72 65 73 74 6F 1B 20 6C 76 73 68 64 63 ..presto. lvshdc
0060: 31 32 65 6E 30 30 30 31 2E 6C 76 73 2E 70 61 79 12en0001.lvs.pay
0070: 70 61 6C 69 6E 63 2E 63 6F 6D A3 82 01 1B 30 82 palinc.com....0.
0080: 01 17 A0 03 02 01 12 A1 03 02 01 02 A2 82 01 09 ................
0090: 04 82 01 05 E7 AE 9E B4 A0 8E C9 D0 2B AE 85 E5 ............+...
00A0: 3B F1 1A AE CD A5 C3 67 58 91 48 31 4B 7C 82 B1 ;......gX.H1K...
00B0: 4F 6E 9D 3C 20 DD E2 C7 1F DB 0C 11 7B 6F 03 60 On.< ........o.`
00C0: E9 7C FA 05 23 1C 44 B7 E1 76 AB 72 D8 15 1E DB ....#.D..v.r....
00D0: 8E 68 D3 D1 4B FF 48 19 F8 CF 24 79 F8 5C D6 FB .h..K.H...$y.\..
00E0: 16 BB C7 FE FD 65 76 19 25 4D 53 2C 72 25 7C D8 .....ev.%MS,r%..
00F0: B4 61 6C 25 2F 3A BF 08 CF 08 17 86 2E 96 9A A4 .al%/:..........
0100: 54 52 1A 43 8C 83 48 9F A9 A1 43 B8 FF 26 FB D2 TR.C..H...C..&..
0110: 85 92 C7 5C 9A F4 FF 8C F1 BF 24 83 80 D1 24 AA ...\......$...$.
0120: DB FD BE 45 79 BE 3F 87 CE B5 7C CD 0F 88 AC 57 ...Ey.?........W
0130: A0 91 1D B2 B8 3B 82 9D FE FD 32 8F F5 43 9C 00 .....;....2..C..
0140: 4F C2 A0 F4 9F 81 43 E4 E9 7A BA A6 50 68 A0 5D O.....C..z..Ph.]
0150: 07 FB D9 E2 04 45 0C 42 00 DC 61 23 25 7B 74 13 .....E.B..a#%.t.
0160: 5C CC AC F2 E2 B1 58 F6 86 C1 D1 49 3D CC 27 DC \.....X....I=.'.
0170: 9D 83 1E 43 6B 63 D8 20 0E 5A 4C FB E9 1B FB 00 ...Ckc. .ZL.....
0180: FA CB 90 0B BE 69 68 61 45 5F 16 0A 81 9D B7 6E .....ihaE_.....n
0190: E9 3D 4D 47 2C 04 62 3C 50 A4 81 D5 30 81 D2 A0 .=MG,.b<P...0...
01A0: 03 02 01 12 A2 81 CA 04 81 C7 B8 13 73 95 22 DF ............s.".
01B0: E1 95 DC 8D AE 27 7C 7C 17 A2 CE 4E 98 AE EB CA .....'.....N....
01C0: 8D 01 52 CE 22 2A 28 5C 65 4F EB F7 4E 83 9C 7D ..R."*(\eO..N...
01D0: A4 F8 94 E9 88 05 0A CD 84 93 3C F7 7C B3 DE 37 ..........<....7
01E0: 6B 46 0D EA 6B 4A 68 4A 75 0E C4 0A 0C C2 75 26 kF..kJhJu.....u&
01F0: 03 3E 7D FB 13 3E A1 5D 8F CE BF 79 4A CA B3 EB .>...>.]...yJ...
0200: 62 E2 2B A0 70 3C 6E 33 94 64 2C EA 89 8D 77 9A b.+.p<n3.d,...w.
0210: 51 BE A1 A6 C8 10 EE B7 AF AA DC A3 A7 43 3D 57 Q............C=W
0220: 9C 5E 50 4A 30 62 DD 5F 6E 36 33 52 FC BA D8 34 .^PJ0b._n63R...4
0230: DF 89 3B 10 33 F4 42 33 66 24 9A 78 16 13 8F E1 ..;.3.B3f$.x....
0240: 2F 7C EE 4C 87 F6 DF C9 F3 C4 C0 7E EC AE 87 3F /..L...........?
0250: 80 74 36 70 B5 00 E7 F8 D6 FC 94 21 9D 99 A7 5A .t6p.......!...Z
0260: E8 B1 BA 67 7A 69 6E D8 7F 71 66 EF 44 97 9C 05 ...gzin..qf.D...
0270: D2 .

Error running command:
Error reading response from server

presto:tiny>

---

 

 

Re: Presto Kerberos Troubleshooting

Please ignore this request.

I was not updating the configs under /etc/presto, even though I was making changes to ~/.prestoadmin/<configs>.

This was resolved by doing a

presto-admin configuration deploy [coordinator|workers]

 Thanks