Presto installation running out of disk space

Presto
Teradata Employee

Presto installation running out of disk space

We're hitting a disk space issue when installing presto with presto-admin via presto-admin server install. It starts deploying the rpm via sftp but then errors because it runs out of disk space. I looked in the logs and it looks like it writes it to /opt/prestoadmin/packages, but /opt has 1.1GB available (server rpm is 600MB). thought it might be /tmp but that has 1.8GB available.

 

Where does presto-admin write the temporarily rpm when copying the server package to the nodes?


Accepted Solutions
Teradata Employee

Re: Presto installation running out of disk space

Hi Jason,

 

I think the weird name for the rpm could be a permissions issue.  When presto-admin deploys the rpm to the cluster, it first copies it to a temporary location and names it a random alphanumeric string.  Then using sudo, it copies it over to the final location.  If the location specified is a directory, (which it is), it will add the original filename to the end of the path before doing the move.  However, if the check whether the path is a directory fails (either if it's not a directory or if any error is encountered), it moves the file to the the path it's given without tacking on the file name to the path.  Since in this case the path was actually a directory, the file retains its randomly generated name.    Can you check that the user can stat the destination directory for the rpm?

 

 

 

 

1 ACCEPTED SOLUTION
12 REPLIES
Teradata Employee

Re: Presto installation running out of disk space

Hi Jason

 

Please check ".prestoadmin/packages" folder

 

thanks

Akshat

Teradata Employee

Re: Presto installation running out of disk space

Thanks Akshat, looks like it was writing to ~. after symlinking ~ to a larger partition, we no longer hit the disk space issue.

 

However, the install still hangs for 10+ minutes. The file appears to be copied over fully (all 600MB) but then nothing else seems to happen. From the log file it looks like there may be a timeout issue.

 

The log is 300 lines and I don't see how to attach a file so rather than post it all here, here's an excerpt: 

 

 

2017-03-14 22:23:51,788|26257|139717756725056|prestoadmin.package|INFO|Deploying rpm on rn1-gotut-lapp11.<domain>...
2017-03-14 22:23:51,789|26258|139717756725056|prestoadmin.package|INFO|Deploying rpm on rn1-gotut-lapp10.<domain>...
2017-03-14 22:23:51,791|26259|139717756725056|prestoadmin.package|INFO|Deploying rpm on rn1-gotut-lapp09.<domain>...
2017-03-14 22:23:51,806|26259|139717516793600|paramiko.transport|DEBUG|starting thread (client mode): 0x2a29e10L
2017-03-14 22:23:51,821|26259|139717516793600|paramiko.transport|INFO|Connected (version 2.0, client OpenSSH_6.6.1)
[...]
2017-03-14 22:23:52,587|26258|139717516793600|paramiko.transport|DEBUG|userauth is OK
2017-03-14 22:23:52,588|26257|139717516793600|paramiko.transport|DEBUG|userauth is OK
2017-03-14 22:23:52,644|26259|139717516793600|paramiko.transport|INFO|Authentication (publickey) successful!
2017-03-14 22:23:52,668|26259|139717756725056|paramiko.transport|DEBUG|[chan 0] Max packet in: 32768 bytes
2017-03-14 22:23:52,827|26259|139717516793600|paramiko.transport|DEBUG|[chan 0] Max packet out: 32768 bytes
2017-03-14 22:23:52,828|26259|139717516793600|paramiko.transport|DEBUG|Secsh channel 0 opened.
2017-03-14 22:23:52,836|26259|139717516793600|paramiko.transport|DEBUG|[chan 0] Sesch channel 0 request ok
2017-03-14 22:23:52,839|26259|139717516793600|paramiko.transport|DEBUG|[chan 0] Sesch channel 0 request ok
2017-03-14 22:23:53,372|26258|139717516793600|paramiko.transport|INFO|Authentication (publickey) successful!
2017-03-14 22:23:53,388|26258|139717756725056|paramiko.transport|DEBUG|[chan 0] Max packet in: 32768 bytes
2017-03-14 22:23:53,538|26258|139717516793600|paramiko.transport|DEBUG|[chan 0] Max packet out: 32768 bytes
2017-03-14 22:23:53,539|26258|139717516793600|paramiko.transport|DEBUG|Secsh channel 0 opened.
2017-03-14 22:23:53,560|26258|139717516793600|paramiko.transport|DEBUG|[chan 0] Sesch channel 0 request ok
2017-03-14 22:23:53,564|26258|139717516793600|paramiko.transport|DEBUG|[chan 0] Sesch channel 0 request ok
2017-03-14 22:23:53,999|26257|139717516793600|paramiko.transport|INFO|Authentication (publickey) successful!
2017-03-14 22:23:54,006|26257|139717756725056|paramiko.transport|DEBUG|[chan 0] Max packet in: 32768 bytes
2017-03-14 22:23:54,186|26257|139717516793600|paramiko.transport|DEBUG|[chan 0] Max packet out: 32768 bytes
2017-03-14 22:23:54,186|26257|139717516793600|paramiko.transport|DEBUG|Secsh channel 0 opened.
2017-03-14 22:23:54,216|26257|139717516793600|paramiko.transport|DEBUG|[chan 0] Sesch channel 0 request ok
2017-03-14 22:23:54,221|26257|139717516793600|paramiko.transport|DEBUG|[chan 0] Sesch channel 0 request ok
2017-03-14 22:28:53,402|26259|139717516793600|paramiko.transport|DEBUG|[chan 0] EOF received (0)
2017-03-14 22:28:53,403|26259|139717516793600|paramiko.transport|DEBUG|[chan 0] EOF sent (0)
2017-03-14 22:28:53,425|26259|139717756725056|prestoadmin.fabric_patches|INFO|
COMMAND: mkdir -p /opt/prestoadmin/packages
FULL COMMAND: sudo -S -p 'sudo password:'  /bin/bash -l -c "mkdir -p /opt/prestoadmin/packages"
STDOUT: root@rn1-gotut-lapp09:/ngs/app/e0315422# timed out waiting for input: auto-logout
STDERR: 
2017-03-14 22:28:53,426|26259|139717756725056|paramiko.transport|DEBUG|[chan 1] Max packet in: 32768 bytes
2017-03-14 22:28:53,443|26259|139717516793600|paramiko.transport|DEBUG|[chan 1] Max packet out: 32768 bytes
2017-03-14 22:28:53,443|26259|139717516793600|paramiko.transport|DEBUG|Secsh channel 1 opened.
2017-03-14 22:28:53,460|26259|139717516793600|paramiko.transport|DEBUG|[chan 1] Sesch channel 1 request ok
2017-03-14 22:28:53,468|26259|139717756725056|paramiko.transport.sftp|INFO|[chan 1] Opened sftp connection (server version 3)
2017-03-14 22:28:53,468|26259|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] normalize('.')
2017-03-14 22:28:53,469|26259|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] lstat('/opt/prestoadmin/packages')
2017-03-14 22:28:53,470|26259|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] stat('/opt/prestoadmin/packages')
2017-03-14 22:28:53,471|26259|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] open('05db144014282aba99e4a19cb80be670964a31d0', 'wb')
2017-03-14 22:28:53,471|26259|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] open('05db144014282aba99e4a19cb80be670964a31d0', 'wb') -> 00000000
2017-03-14 22:28:54,131|26258|139717516793600|paramiko.transport|DEBUG|[chan 0] EOF received (0)
2017-03-14 22:28:54,131|26258|139717516793600|paramiko.transport|DEBUG|[chan 0] EOF sent (0)
2017-03-14 22:28:54,147|26258|139717756725056|prestoadmin.fabric_patches|INFO|
COMMAND: mkdir -p /opt/prestoadmin/packages
FULL COMMAND: sudo -S -p 'sudo password:'  /bin/bash -l -c "mkdir -p /opt/prestoadmin/packages"
STDOUT: root@rn1-gotut-lapp10:/ngs/app/e0315422# timed out waiting for input: auto-logout
STDERR: 
2017-03-14 22:28:54,147|26258|139717756725056|paramiko.transport|DEBUG|[chan 1] Max packet in: 32768 bytes
2017-03-14 22:28:54,224|26258|139717516793600|paramiko.transport|DEBUG|[chan 1] Max packet out: 32768 bytes
2017-03-14 22:28:54,224|26258|139717516793600|paramiko.transport|DEBUG|Secsh channel 1 opened.
2017-03-14 22:28:54,257|26258|139717516793600|paramiko.transport|DEBUG|[chan 1] Sesch channel 1 request ok
2017-03-14 22:28:54,262|26258|139717756725056|paramiko.transport.sftp|INFO|[chan 1] Opened sftp connection (server version 3)
2017-03-14 22:28:54,262|26258|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] normalize('.')
2017-03-14 22:28:54,263|26258|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] lstat('/opt/prestoadmin/packages')
2017-03-14 22:28:54,264|26258|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] stat('/opt/prestoadmin/packages')
2017-03-14 22:28:54,265|26258|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] open('7ffb51ad7cff9e8e53dc144827c2c9986b5414c4', 'wb')
2017-03-14 22:28:54,267|26258|139717756725056|paramiko.transport.sftp|DEBUG|[chan 1] open('7ffb51ad7cff9e8e53dc144827c2c9986b5414c4', 'wb') -> 00000000
2017-03-14 22:28:54,763|26257|139717516793600|paramiko.transport|DEBUG|[chan 0] EOF received (0)
2017-03-14 22:28:54,763|26257|139717516793600|paramiko.transport|DEBUG|[chan 0] EOF sent (0)

 

 

Teradata Employee

Re: Presto installation running out of disk space

Hi

 

While the command is running, can you see if the RPM is being copied by loggin into other nodes? And all the nodes have enough space for RPM to be copied in /opt/

 

thanks

Akshat

Teradata Employee

Re: Presto installation running out of disk space

The prestoadmin directory doesn't exist in /opt, but it does have enough space for the rpm to be copied (1.1 GB)

Teradata Employee

Re: Presto installation running out of disk space

That seems like some permission issue then. You can see in the logs that mkdir -p command is being run. That should've created the directory for you.

Teradata Employee

Re: Presto installation running out of disk space

yup, looks like there was a security measure in .bash_profile to not allow what we were trying to do. for now, we're removing it so it now proceeds past that point.

 

next issue :)

 

When it copies the local rpm to each node, it names it a random alphanumeric string. However, when it goes to install the rpm, it uses the original name (presto-server-rpm...) without renaming it to that first. See the log excerpt below. 

 

Is this another permission issue or has this been seen in other uses of presto-admin?

 

2017-03-15 20:46:46,282|19087|140514793973568|prestoadmin.fabric_patches|INFO|
COMMAND: mv "05db144014282aba99e4a19cb80be670964a31d0" "/opt/prestoadmin/packages"
FULL COMMAND: sudo -S -p 'sudo password:'  /bin/bash -l -c "mv \"05db144014282aba99e4a19cb80be670964a31d0\" \"/opt/prestoadmin/packages\""
STDOUT: 
STDERR: 
2017-03-15 20:46:46,282|19087|140514793973568|paramiko.transport.sftp|INFO|[chan 1] sftp session closed.
2017-03-15 20:46:46,282|19087|140514793973568|paramiko.transport|DEBUG|[chan 1] EOF sent (1)
2017-03-15 20:46:46,283|19087|140514793973568|prestoadmin.package|INFO|Installing the rpm
2017-03-15 20:46:46,283|19087|140514793973568|paramiko.transport|DEBUG|[chan 3] Max packet in: 32768 bytes
2017-03-15 20:46:46,308|19087|140514554042112|paramiko.transport|DEBUG|[chan 3] Max packet out: 32768 bytes
2017-03-15 20:46:46,308|19087|140514554042112|paramiko.transport|DEBUG|Secsh channel 3 opened.
2017-03-15 20:46:46,309|19087|140514554042112|paramiko.transport|DEBUG|Ignoring message for dead channel 1
2017-03-15 20:46:46,310|19087|140514554042112|paramiko.transport|DEBUG|Ignoring message for dead channel 1
2017-03-15 20:46:46,318|19087|140514554042112|paramiko.transport|DEBUG|[chan 3] Sesch channel 3 request ok
2017-03-15 20:46:46,321|19087|140514554042112|paramiko.transport|DEBUG|[chan 3] Sesch channel 3 request ok
2017-03-15 20:46:46,919|19087|140514554042112|paramiko.transport|DEBUG|[chan 3] EOF received (3)
2017-03-15 20:46:46,920|19087|140514554042112|paramiko.transport|DEBUG|[chan 3] EOF sent (3)
2017-03-15 20:46:46,926|19087|140514793973568|prestoadmin.fabric_patches|ERROR|Traceback (most recent call last):
  File "/ngs/app/presto/prestoadmin/presto-admin-install/lib/python2.7/site-packages/prestoadmin/fabric_patches.py", line 138, in inner
    submit(task.run(*args, **kwargs))
 [...]
SystemExit: [rn1-gotut-lapp09.rno.apple.com] sudo() received nonzero return code 1 while executing!

Requested: rpm -i /opt/prestoadmin/packages/presto-server-rpm-0.157.1-t.1.x86_64.rpm
Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "export JAVA8_HOME=\"/usr/java/jre64-1.8.0_111\" && rpm -i /opt/prestoadmin/packages/presto-server-rpm-0.157.1-t.1.x86_64.rpm"

 

Teradata Employee

Re: Presto installation running out of disk space

 Seems like the error shows up whiel running this command

 

 

Requested: rpm -i /opt/prestoadmin/packages/presto-server-rpm-0.157.1-t.1.x86_64.rpm

Can you log on to the machine where this error showed up and check if this file /opt/prestoadmin/packages/presto-server-rpm-0.157.1-t.1.x86_64.rpm exists, what is its size and permissions?

 

 

Teradata Employee

Re: Presto installation running out of disk space

It exists but has the generated alphanumeric name. my thought is that this command should move and rename the file rather than just move it:

 

mv "05db144014282aba99e4a19cb80be670964a31d0" "/opt/prestoadmin/packages"
Teradata Employee

Re: Presto installation running out of disk space

Yes those random strings do not make sense, it should be the same name as the RPM file that is being copied. It could be possibly related to a different python library that is installed. 

Did you install an online or offline version of prestoadmin?