[TPT 15.10] TPT19140 Unable to allocate nnn bytes of memory.

Tools & Utilities
Enthusiast

[TPT 15.10] TPT19140 Unable to allocate nnn bytes of memory.

Start to run into memory allocation errors randomly in TPT FastLoad from big gzipped CSV. Sometimes, simply rerun will go through, but we really don't know what is the root cause:

CheckPoint No. 1 finished successfully.
**** 19:42:20 Starting to send data to the RDBMS
TPT_INFRA: TPT02610: Error: Could not allocate a data message of size 321396
TPT_INFRA: TPT02610: Error: Could not allocate a data message of size 321396
TPT_INFRA: TPT02284: Error: Cannot create data buffer, Data Stream status = 3
TPT_INFRA: TPT02284: Error: Cannot create data buffer, Data Stream status = 3
TPT_INFRA: TPT02148: Data Block size reduced to 128552 (Buffers/Block: 2) for task SELECT_2[0002]
TPT_INFRA: TPT02610: Error: Could not allocate a data message of size 128580
TPT_INFRA: TPT02284: Error: Cannot create data buffer, Data Stream status = 3
TPT_INFRA: TPT02148: Data Block size reduced to 64280 (Buffers/Block: 1) for task SELECT_2[0007]
TPT_INFRA: TPT02610: Error: Could not allocate a data message of size 321396
TPT_INFRA: TPT02284: Error: Cannot create data buffer, Data Stream status = 3
TPT_INFRA: TPT02610: Error: Could not allocate a data message of size 128580
TPT_INFRA: TPT02284: Error: Cannot create data buffer, Data Stream status = 3
TPT_INFRA: TPT02610: Error: Could not allocate a data message of size 64308
TPT_INFRA: TPT02284: Error: Cannot create data buffer, Data Stream status = 3
$FILE_READER[8]: TPT19140 Unable to allocate 64260 bytes of memory.
$FILE_READER[8]: TPT19015 TPT Exit code set to 12.
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 8, Total Rows Received = 0, Total Rows Sent = 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 0, Total Rows Received = 0, Total Rows Sent = 0
LOADING              SELECT_2[0001]       Success              $FILE_READER            8    1 TERMINATE-Started    19:42:20     0.0000     0.0000      65000          0                0                0   600     0 N Y
LOADING              SELECT_2[0002]       Success              $FILE_READER            8    2 TERMINATE-Started    19:42:20     0.0000     0.0000     128552          0                0                0   600     0 N Y
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 7, Total Rows Received = 0, Total Rows Sent = 0
LOADING              SELECT_2[0003]       Success              $FILE_READER            8    3 TERMINATE-Started    19:42:20     0.0000     0.0000     321368          0                0                0   600     0 N Y
              Instance      Cpu Time     
              ========  ================ 
                   1        0.67 Seconds
                   2        0.52 Seconds
 
**** 19:42:21 Total processor time used = '1.19 Second(s)'
     .        Start : Wed Aug  2 19:42:09 2017
     .        End   : Wed Aug  2 19:42:21 2017
     .        Highest return code encountered = '0'.
$LOAD: Total processor time used = '1.19 Second(s)'
$LOAD: Start : Wed Aug  2 19:42:09 2017
LOADING              APPLY_1[0002]        Success              $LOAD                   2    2 TERMINATE-Ended      19:42:21     1.0000     0.0000      65000          0                0                0   600     0 N Y
$LOAD: End   : Wed Aug  2 19:42:21 2017
**** 19:42:21 This job terminated
STG_SAS_AD_IMPR_TRK-29762480.00-202,179,5,OperatorEndTS,LOADING,$LOAD,1,2017-08-02,2017-08-02 19:42:21,1,0
STG_SAS_AD_IMPR_TRK-29762480.00-202,18,1,OperatorExit,LOADING,$LOAD,1,2017-08-02,0,1,0
LOADING              APPLY_1[0001]        Success              $LOAD                   2    1 TERMINATE-Ended      19:42:21     1.0000     0.0000      65000          0                0                0   600     0 N Y
Job step LOADING terminated (status 8)
Job STG_SAS_AD_IMPR_TRK-29762480.00 terminated (status 8)
Job start: Wed Aug  2 19:41:56 2017
Job end:   Wed Aug  2 19:42:21 2017
Total available memory:          20000676
Largest allocable area:          20000676
Memory use high water mark:      19982928
Free map size:                       1024
Free map use high water mark:          61
Free list use high water mark:          0
     32-bit DataConnector operator for Linux release 2.6.32-431.17.1.el6.x86_64 on xyz.com
     32-bit DataConnector operator for Linux release 2.6.32-431.17.1.el6.x86_64 on xyz.com
     32-bit DataConnector operator for Linux release 2.6.32-431.17.1.el6.x86_64 on xyz.com
      
      
     PXICU................................... 15.10.00.00
     =                  IndicatorMode: 'NO (defaulted)' (=0)                  =
      
     TDICU................................... 15.10.00.00
     TDICU................................... 15.10.00.00
     TDICU................................... 15.10.00.00
     32-bit DataConnector operator for Linux release 2.6.32-431.17.1.el6.x86_64 on lva1-etl01.corp.linkedin.com
     32-bit DataConnector operator for Linux release 2.6.32-431.17.1.el6.x86_64 on lva1-etl01.corp.linkedin.com
     PMPROCS................................. 15.10.00.03
     =                  NullColumns: 'YES (defaulted)' (=1)                   =
     32-bit DataConnector operator for Linux release 2.6.32-431.17.1.el6.x86_64 on lva1-etl01.corp.linkedin.com
     PXICU................................... 15.10.00.00
     PXICU................................... 15.10.00.00
     PXICU................................... 15.10.00.00
     TDICU................................... 15.10.00.00
     TDICU................................... 15.10.00.00
     PMRWFMT................................. 15.00.00.02
     =                        SYSTEM_CharSetId: 'UTF8'                        =
     TDICU................................... 15.10.00.00
     PMPROCS................................. 15.10.00.03
     PMPROCS................................. 15.10.00.03
     PMPROCS................................. 15.10.00.03
     PXICU................................... 15.10.00.00
     PXICU................................... 15.10.00.00
     PMHADOOP................................ 15.10.00.02
     =                                                                        =
     PXICU................................... 15.10.00.00
     PMRWFMT................................. 15.00.00.02
     PMRWFMT................................. 15.00.00.02
     PMRWFMT................................. 15.00.00.02
     PMPROCS................................. 15.10.00.03
     PMPROCS................................. 15.10.00.03
     PMTRCE.................................. 13.00.00.02
     ==========================================================================
     PMPROCS................................. 15.10.00.03
     PMHADOOP................................ 15.10.00.02
     PMHADOOP................................ 15.10.00.02
     PMHADOOP................................ 15.10.00.02
     PMRWFMT................................. 15.00.00.02
     PMRWFMT................................. 15.00.00.02
     PMMM.................................... 15.10.00.02
      
     PMRWFMT................................. 15.00.00.02
     PMTRCE.................................. 13.00.00.02
     PMTRCE.................................. 13.00.00.02
     PMTRCE.................................. 13.00.00.02
     PMHADOOP................................ 15.10.00.02
     PMHADOOP................................ 15.10.00.02
     DCUDDI.................................. 15.10.00.07
     LITTLE ENDIAN platform
     PMHADOOP................................ 15.10.00.02
     PMMM.................................... 15.10.00.02
     PMMM.................................... 15.10.00.02
     PMMM.................................... 15.10.00.02
     PMTRCE.................................. 13.00.00.02
     PMTRCE.................................. 13.00.00.02
     PMHEXDMP................................ 14.10.00.01
     PMTRCE.................................. 13.00.00.02
     DCUDDI.................................. 15.10.00.07
     DCUDDI.................................. 15.10.00.07
     DCUDDI.................................. 15.10.00.07
     PMMM.................................... 15.10.00.02
     PMMM.................................... 15.10.00.02
     PMHDFSDSK............................... 15.10.00.02
     PMMM.................................... 15.10.00.02
     PMHEXDMP................................ 14.10.00.01
     PMHEXDMP................................ 14.10.00.01
     PMHEXDMP................................ 14.10.00.01
     DCUDDI.................................. 15.10.00.07
     DCUDDI.................................. 15.10.00.07
     PMUNXDSK................................ 15.10.00.02
     DCUDDI.................................. 15.10.00.07
     PMHDFSDSK............................... 15.10.00.02
     PMHDFSDSK............................... 15.10.00.02
     PMHDFSDSK............................... 15.10.00.02
     PMHEXDMP................................ 14.10.00.01
     PMHEXDMP................................ 14.10.00.01
      
     WARNING!  NotifyMethod: 'None (default)'
     PMHEXDMP................................ 14.10.00.01
     PMUNXDSK................................ 15.10.00.02
     PMUNXDSK................................ 15.10.00.02
     PMUNXDSK................................ 15.10.00.02
     PMHDFSDSK............................... 15.10.00.02
     PMHDFSDSK............................... 15.10.00.02
     >> Enter DC_DataConFileInfo
     Operator 'dtacop' main source version:'15.10.00.18'
     PMHDFSDSK............................... 15.10.00.02
      
      
      
     PMUNXDSK................................ 15.10.00.02
     PMUNXDSK................................ 15.10.00.02
     Job Type=0
     DirInfo global variable name: 'DirInfo'
     PMUNXDSK................................ 15.10.00.02
     >> Enter DC_DataConFileInfo

The launching command looks like:

tbuild -z 600 -f /export/code/run/lib_etl/tmp/STG_TABLE_NAME-TPT-29762480.00.tpt -v /export/code/run/lib_etl/tmp/STG_TABLE_NAME-TPT-29762480.00.jobvars -u "TargetTdpId='td001.xyz.com',TargetUserName='ETL_TPT',TargetWorkingDatabase='DWH_STG',TargetTable='TABLE_NAME',SourceDirectoryPath='/mnt/n001/data/table_name/2017080218-2017080218',SourceFileName='stg_table_name.csv.gz',SourceTextDelimiter='^z'" -j STG_TABLE_NAME-29762480.00

Any help is appreciated!

Tags (3)
2 REPLIES
Teradata Employee

Re: [TPT 15.10] TPT19140 Unable to allocate nnn bytes of memory.

It appears you have a large number of operator instances and the default shared memory size is too small.

Consider reducing the number of instances and/or increasing shared memory size on the tbuild command, e.g. -h 32M

Enthusiast

Re: [TPT 15.10] TPT19140 Unable to allocate nnn bytes of memory.

Thanks Fred. The Linux server showed 0 free shared memory, but it can still execute 95% of the tbuild jobs successfully. Does this indicate any OS configuration issue?

 

 

$ /sbin/sysctl kernel.shmmni
kernel.shmmni = 4096

$ grep -i sh /proc/meminfo 
Shmem:              9108 kB

$ free -m
             total       used       free     shared    buffers     cached
Mem:         48201      47782        418          0       1354      42364
-/+ buffers/cache:       4062      44138
Swap:        16382          0      16382

$ vmstat -s
     49358220  total memory
     48889544  used memory
     11161800  active memory
     35308412  inactive memory
       468676  free memory
      1387376  buffer memory
     43381532  swap cache
     16776116  total swap
            0  used swap
     16776116  free swap
  22452736542 non-nice user cpu ticks
      2781736 nice user cpu ticks
   1515692866 system cpu ticks
 180724128072 idle cpu ticks
   4717298356 IO-wait cpu ticks
       727825 IRQ cpu ticks
    271514358 softirq cpu ticks
            0 stolen cpu ticks
  21497327765 pages paged in
 122398180063 pages paged out
            0 pages swapped in
            0 pages swapped out
   3911853577 interrupts
   2986681433 CPU context switches
   1413952543 boot time
   1297900967 forks

If we have "-h 32M" or "-h 64M", will the concurrent tbuild jobs request too much shared memory and then fail each other?