Teradata Developer Tier fails deployment on configure step

Teradata Database on VMWare
Enthusiast

Teradata Developer Tier fails deployment on configure step

Hi there!
   I'm moving my line of questioning over here so that I quit hijacking the other guy's thread who was potentially troubleshooting a different issue.

 

   I'm trying to deploy a 2-node Developer Tier environment, with a VP, CMIC, and DSU.  I had trouble getting everything to deploy in one shot using the "deploy" step, so I've been working through each step sequentially manually.  At this point I've got everything deployed and presumably configured, and am trying to run the "configure" step.  I also snapshotted the VMs so that I can roll back without having to redeploy everything.

 

   First of all, if I log in I notice that even though the nodes rebooted, the /etc/resolv.conf wasn't updated, so I had to run "netconfig update" to get it to pull the right DNS servers and suffixes into /etc/resolv.conf from /etc/sysconfig/network/config. 

 

   I have also noticed on a few tries that sysconfig runs, but then restarts the database and it'll only come up on one node or the other.  In such cases, I can see where the software was killed due to NTP not running, and indeed, the ntp service is stopped or crashed...

 

   Once sysinit restarts the database, after a few minutes CPU use spikes up so high that it eventually becomes unresponsive, even to the VMware Tools backdoor that the deploy script uses to monitor for dip.log.  Usually if I run top I can see that the top CPU user is kswap0, at least until the system locks up completely...

 

   Furthermore, I tried running PUT directly via the GUI and got the same result.

 

   I have the thing sitting on flash storage and on pretty nice modern hardware.  Plenty of RAM and CPU, though the VM CPU and Memory sizes are hard-coded by the Developer Tier deployment script...  I know I can modify them through VMware, but...

 

   Anyway, here's my info:

 

common.IT.properties (sanitized):

{
    "TDsystemname": "<my sys name>",
    "domain":  "<my internal domain>.com",
    "public_vswitch":  "Server Network",
    "byn_vswitch1": "TD_byn1",
    "byn_vswitch2": "TD_byn2",
    "cmic_vswitch1": "TD_Management1",
    "dsu_vswitch": "TD_DSC",
    "public_submask":  "255.255.255.0",
    "gateway":  "X.X.X.1",
    "dns1":  "X.X.X.49",
    "dns2":  "X.X.X.59",
    "ntp1":  "X.X.X.49",
    "ntp2":  "X.X.X.59",
    "OSTimeZone": "US/Pacific",
    "pdiskSizeGB":  "100",
    "Temporal":  "FALSE",
    "TemporalDaylightSavings": "FALSE",
    "TemporalTimeDateWZControl": 0,
    "TemporalTimeZoneHour": 0,
    "TemporalTimeZoneMinute": 0,
    "TemporalTimeZoneString": "America Pacific",
    "SecureZones": "FALSE",
    "RowLevelSecurity": "FALSE",
    "Columnar":  "FALSE",
    "MainframeLHID":"-1",
    "SiteID": "",
    "Folder":"",
    "IPAddressSetting":"static"
}

node.IT.properties (sanitized):

[
    {
        "ESXiHost":  "<vm host>",
        "OS_Datastore":  "SSD",
        "vmName":  "node1",
        "eth0":  "X.X.X.77",
        "vmType":  "tpa",
        "Template":  "Teradata_Database_16.20.05.01_SLES11_SP3_on_VMware_20180219161644"
    },
    {
        "ESXiHost":  "<vm host>",
        "OS_Datastore":  "SSD",
        "vmName":  "node2",
        "eth0":  "X.X.X.78",
        "vmType":  "tpa",
        "Template":  "Teradata_Database_16.20.05.01_SLES11_SP3_on_VMware_20180219161644"
    },
    {
        "ESXiHost":  "<vm host>",
        "OS_Datastore":  "SSD",
        "vmName":  "dsu",
        "eth0":  "X.X.X.76",
        "vmType":  "dsu",
        "Template":  "Teradata_DSC_16.10.00.00_SLES11_SP3_on_VMware_20170726112417"
    },
    {
        "ESXiHost":  "<vm host>",
        "OS_Datastore":  "SSD",
        "vmName":  "vp",
        "eth0":  "X.X.X.75",
        "vmType":  "vp",
        "Template":  "Teradata_Viewpoint_16.20.00.02_SLES11_SP3_on_VMware_20180220135303"
    },
    {
        "ESXiHost":  "<vm host>",
        "OS_Datastore":  "SSD",
        "vmName":  "cmic",
        "eth0":  "X.X.X.74",
        "vmType":  "cmic",
        "Template":  "Teradata_CMIC_12.05.03.00_SLES11_SP3_on_VMware_20170807151026-2"
    }
]

And, the log from the last run I took at running configure (also sanitized):

**********************
Windows PowerShell transcript start
Start time: 20180501225545
Username: <my user>
RunAs User: <my user>
Configuration Name: 
Machine: NSIW10-03 (Microsoft Windows NT 10.0.16299.0)
Host Application: C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -noe -c . "C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1" $true
Process ID: 18120
PSVersion: 5.1.16299.98
PSEdition: Desktop
PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1.16299.98
BuildVersion: 10.0.16299.98
CLRVersion: 4.0.30319.42000
WSManStackVersion: 3.0
PSRemotingProtocolVersion: 2.3
SerializationVersion: 1.1.0.1
**********************

Script Version: 03.01.00.04
Enumerate current working directory!
Please choose a commonITproperties property file:
01) C:\teradata\expanded_scripts\Properties\common.IT.properties.2node.json
02) C:\teradata\expanded_scripts\Properties\common.IT.properties.json
03) C:\teradata\expanded_scripts\Properties\node.IT.properties.2node.json
04) C:\teradata\expanded_scripts\Properties\node.IT.properties.json
Read in the commonITproperties file
Enumerate current working directory!
Please choose a nodeITproperties property file:
01) C:\teradata\expanded_scripts\Properties\common.IT.properties.2node.json
02) C:\teradata\expanded_scripts\Properties\common.IT.properties.json
03) C:\teradata\expanded_scripts\Properties\node.IT.properties.2node.json
04) C:\teradata\expanded_scripts\Properties\node.IT.properties.json
Read in the nodeITproperties file
commonITproperties file is: C:\teradata\expanded_scripts\Properties\common.IT.properties.2node.json
  nodeITproperties file is: C:\teradata\expanded_scripts\Properties\node.IT.properties.2node.json

***********************************************************************************************
*
*   5/1/2018 10:55:50 PM   - Pre-validate the properties files
*
***********************************************************************************************
Continuing at user's request!
Using calculated CMIC subnet mask: 255.255.255.248

All TPA ESXi hosts have the following CPU and Memory resources:
CPU Model ( Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz ) and MemoryMB ( 391842 )
EsxiCPUtotal( 80) EsxiCPUreserved% (0.02) EsxiMemTotal (374082) EsxiMemReserved (17760)

VirtualCores memoryMBPerTPA numpdisks pdiskSize subpools/node amps/subpool pdisks/subpool
4            8192           2         100       2             2            1


Chosen step ( configure ) requires a password
Please enter the root password the deployed system is using. Enter blank twice to use the default password.

***********************************************************************************************
*
*   5/1/2018 10:55:57 PM   - Configure Teradata DBS...
*
***********************************************************************************************
5/1/2018 10:55:57 PM  - TDput Configure Teradata Operation on ( node1 ) and will take approximately 20 minutes...
byn( 001-01,001-02) SystemName(nsitd) SystemFamily(000CLV) TotalAmps(8) AmpsPerNode(4)
Step 1 done. Checking for dip log.
invoke-vmscript : 5/1/2018 11:04:19 PM	Invoke-VMScript		The remote server returned an error: (404) Not Found.	
At C:\teradata\expanded_scripts\tdc.Binarydeploy.ps1:3859 char:34
+ ... pt_Output = invoke-vmscript -scripttext        "ls /var/opt/teradata/ ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-VMScript], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_DownloadFileFromGuest_DownloadError,VMware.VimAutomation.ViCor
e.Cmdlets.Commands.InvokeVmScript
invoke-vmscript : 5/1/2018 11:04:19 PM	Invoke-VMScript		The remote server returned an error: (404) Not Found.
At C:\teradata\expanded_scripts\tdc.Binarydeploy.ps1:3859 char:34
+ ... pt_Output = invoke-vmscript -scripttext        "ls /var/opt/teradata/ ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Invoke-VMScript], ViError
    + FullyQualifiedErrorId : Client20_VmGuestServiceImpl_DownloadFileFromGuest_DownloadError,VMware.VimAutomation.ViCore.Cmdlets.Commands.InvokeVmScript

False

***********************************************************************************************
*
*   5/2/2018 12:44:06 AM   - Applying TASM Settings...
*
***********************************************************************************************
no crontab for root
no crontab for root

5/2/2018 12:44:28 AM Status of step:
Error: Error Teradata could not be properly configured. It should have started.

Deployment started  - 5/1/2018 10:55:45 PM
Deployment finished - 5/2/2018 12:44:28 AM
**********************
Windows PowerShell transcript end
End time: 20180502004428
**********************

Anyway, not sure what to do here... Right now I'm really suspicious of NTP problems, especially since I am not enabling temporal features...  But again, I'm not sure that's the direction to take for troubleshooting right away...

 

Thanks in advance for any help you may be able to provide!!

 

 

7 REPLIES
Enthusiast

Re: Teradata Developer Tier fails deployment on configure step

Ok, a few things I'm noticing:

 

1. There's nothing unusual about this install.  Straight download from your site, two node deploy following the instructions.

2. Your scripts somehow don't get netconfig called, so root login takes forever to prompt for a password due to the fact that /etc/resolv.conf still contains Teradata's internal nameservers...

3. I bumped the pdisk size up to 120GB from the 100 that's the default in the common.properties.  Still got the same result where one of the systems hangs.  It's always after sysinit does a restart.  This time /var/log/messages says we ran out of kernel memory on node 2, so the whole mess restarts on node 1 only, and locks it up.  Looks to me like the Developer Tier VMs are severely undersized or the templates shipped are incorrectly configured.

4. Has anyone successfully gotten this to work with 16.20?  I'm thinking some debugging and adjustment of hard-coded values in the tdc.deploy.exe is needed

5. I'll PayPal someone some actual money if they'll just help me get this working...

   Right now, my .properties files look just like the previous post but I changed pdiskSizeGB to 120 and I deleted the other VM types from the node.properties, so only the two TPA nodes get deployed.  What am I doing wrong?

Teradata Employee

Re: Teradata Developer Tier fails deployment on configure step

Just wanted to make sure you are connecting to a VCenter when you do your deployment and not the ESXi server directly. Here is what I did step by step incase it helps you.

 

I launched the TD 16.20 Developer Edition Sucessfully here is what I did:

1) I had already uploaded the the TD 16.20 ova template file to the VCenter and converted it to an OVF Template following KCS004636

2)  I filled in my Property Files as shown below (I covered up some of the personal information with black rectangular boxes)

 

Common.IT.properties.jason file

Common.png

 

 

Node.IT.properties file

 

node.It.properties.png

3) I opened my Power CLI (version 5.5 Release 1 build 1295336 ) and connected to my VCenter (note it must be VCenter that you connect to and not ESXi directly and the scripts were written to only work with VCenter).  Note I am running from a Windows 7 Professional PC running PowerShell version 4.0

 

Connect-VIServer vcenter2              <== Connected to my server named Vcenter2 server and entered my network credentials

cd C:\Users\AC110057\Downloads\TVME_MPP\DeveloperTier\Teradata_Deployer_DT_03v2      <== Changed Directory to where I run tdc.deploy.exe from

 

4) Next I typed in .\tdc.deploy.exe  and selected "deploy", Enumerate picked the proper properties file, selected Enumerate again, picked the next property file,  then took all the defaults for the rest of the setup.

 

 

Teradata Employee

Re: Teradata Developer Tier fails deployment on configure step

The issue is with 03.01.00.04 version of deployer scripts

From the log file, developer scripts version is 03.01.00.04 and database temple is of TD 16.20.05.01.

To configure TD 16.20 database, we need at least 20 GB Memory.

03.01.00.04 version scripts assign 8 GB Memory hence database configuration fails.

Deployer scripts 03.01.01.06  have the fix for this issue.

You should be able to download it from Beta FTP site.

Enthusiast

Re: Teradata Developer Tier fails deployment on configure step

Hi there!

   Thanks for the reply.  I'd love to test the updated deployment script.  How do I access the beta FTP site to obtain it?

 

Thank you!

 

 

Enthusiast

Re: Teradata Developer Tier fails deployment on configure step

Hi there!

   Thank you for detailing your steps in successfully deploying the Developer Tier. 

   I suspect that you're getting lucky, because I'm looking through the config files in /etc, and I see that there are some Teradata corporate network-specific assumptions made in some of them that are not being overwritten.  In particular, some of the NTP stuff and /etc/resolv.conf have Teradata corporation-specific values in them, which match the IP ranges you're using in the properties files you provided.  Do you or any of the developers have the ability to create a little test bubble that uses something other than td.teradata.com and the 153.6[45] IP scheme and see if the scripts work correctly in an environment that is more similar to a customers?

   Thank you!

 

Teradata Employee

Re: Teradata Developer Tier fails deployment on configure step

I don’t have any kind of test bubble outside the TD network to test deploying developer Tier. I am checking with the TDVM Engineering team to see if they have an environment that would be outside of Teradata and more similar to the customer environment.

Teradata Employee

Re: Teradata Developer Tier fails deployment on configure step

Hi,

 

I am trying to find alternate download location for the scripts.

I will not be able to directly share with you.

Instead I will pass this to Arnie and he will take care ot it.

 

Thanks

Srikanth