TPT Fast export error General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argument)

Tools & Utilities
Highlighted
RB
Enthusiast

TPT Fast export error General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argument)

Hello All,

 

I am trying to epxort a table to a file using TPT - Fast export and I am getting the below error. The table has 160+ columns and has 14 millions records (around 10 GB) and I am trying to copy to a location that has 500 GB free space. What I don't understand is the file size at the time of failure is around 23 GB. How is that possible when the table size is just 10 GB? And when there is enough space, why does it fail (I read online that this error relates to lasck of space). I am unable to add the script as it's long :-(

 

export_item: sending SELECT request

**** 13:13:29 RDBMS Warning: 3705 Request executed in interpretive EVL mode.
RDBMS subsystem or component ID: 0
export_item: RDBMS Warning: 3705 Request executed in interpretive EVL mode.
RDBMS subsystem or component ID: 0
**** 13:13:29 This job will run in the Spool Export mode.
**** 13:13:29 Select execution completed. 7406707 data blocks generated.
Data Block size: 3462320 Buffers/Block: 7 Data Buffer size: 494616
file_writer[1]: TPT19434 pmWrite failed. General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argument)'
file_writer[1]: TPT19306 Fatal error writing data.
file_writer[1]: TPT19015 TPT Exit code

 

set to 12.
Task(INSERT_1[0001]) cannot take EOD checkpoint, status = Fatal Error

TPT_INFRA: TPT02255: Message Buffers Sent/Received = 10935, Total Rows Received
= 0, Total Rows Sent = 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 10939, Total Rows Received
= 0, Total Rows Sent = 0

 

6 REPLIES 6
Teradata Employee

Re: TPT Fast export error General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argumen

Certainly an fwrite error could be due to lack or space (or hitting file limit on Unix, for example). But "byte count error (Invalid argument)" might be due to the data not matching the schema or something else. Have you checked the operator private logs to see if there is any more detail?

 

There are many reasons why the exported data may be substantially larger than the size in-database. There could be compression applied in the database; the external delimited character representation may be larger than internal form (especially for date/time and numeric fields), etc.

RB
Enthusiast

Re: TPT Fast export error General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argumen

Thanks Fred for the response. Here is the logs. I have no idea why the total bytes exported is "Total Bytes Exported From RDBMS = 457854553365" :-(. When it actually did export just 14 M records. The job was basically not ending. I aborted when I saw the file size to be 260 GB. Here is the log. 

TPT_INFRA: TPT04101: Warning: Teradata PT cannot connect to Unity EcoSystem Manager.
The job will continue without event messages being sent to Unity EcoSystem Manager. 
TPT_INFRA: TPT04197: Warning: OMD API failed to initialize
Teradata Parallel Transporter PlanExecutor Version 15.10.01.11

Teradata Parallel Transporter Coordinator Version 15.10.01.11
Teradata Parallel Transporter Executor Version 15.10.01.11
Teradata Parallel Transporter Executor Version 15.10.01.11
Teradata Parallel Transporter DataConnector Operator Version 15.10.01.11
file_writer[1]: TPT19206 Attribute 'TraceLevel' value reset to 'Statistics Only'.
Teradata Parallel Transporter Export Operator Version 15.10.01.11
export_item: private log not specified
file_writer[1]: Instance 1 directing private log report to 'dtacop-rbakthavachalam-18056-1'.
 
     ===================================================================
     =                                                                 =
     =              TERADATA PARALLEL TRANSPORTER 64-BIT               =
     =                                                                 =
     =             EXPORT OPERATOR     VERSION 15.10.01.11             =
     =                                                                 =
     =          OPERATOR SUPPORT LIBRARY VERSION 15.10.01.11           =
     =                                                                 =
     =           COPYRIGHT 2001-2017, TERADATA CORPORATION.            =
     =                      ALL RIGHTS RESERVED.                       =
     =                                                                 =
     =                       Process I.D.: 6324                        =
     =                                                                 =
     ===================================================================

**** 15:20:01 Processing starting at: Fri Mar 22 15:20:01 2019
 
     ===================================================================
     =                                                                 =
     =                      Module Identification                      =
     =                                                                 =
     ===================================================================

     64-bit Export Operator for Windows on HCITDETL01
     ExportMain : 15.10.01.03
     ExportCLI  : 15.10.01.02
     ExportUtil : 14.10.00.01
     PcomCLI    : 15.10.01.12
     PcomMBCS   : 14.10.00.02
     PcomMsgs   : 15.10.01.02
     PcomNtfy   : 15.10.00.01
     PcomPx     : 15.10.01.02
     PcomUtil   : 15.10.01.06
     PXICU      : 15.10.01.11
     TDICU      : 15.10.01.03
file_writer[1]: DataConnector Consumer operator Instances: 1
file_writer[1]: ECI operator ID: 'file_writer-18056'
file_writer[1]: Operator instance 1 processing file 'D:\Enroll_Eligs\enroll_eligs.txt'.
     CLIv2      : 15.10.01.07   
 
     ===================================================================
     =                                                                 =
     =                      Attribute Definitions                      =
     =                                                                 =
     ===================================================================

**** 15:20:04 Options in effect for this job:
              OperatorType:  Producer
              Instances:     1
              Character set: Not specified; will use default
              Checkpoint:    No checkpoint in effect
              Notify:        Not enabled
              Tenacity:      4 hour limit to successfully connect
              Sleep:         6 minute(s) between connect retries
              Date format:   INTEGERDATE
              Blocksize:     Maximum allowable
              OutLimit:      No limit in effect
              SpoolMode:     Spool
 
     ===================================================================
     =                                                                 =
     =                     Column/Field Definition                     =
     =                                                                 =
     ===================================================================

     Column Name                    Offset Length Type      
     ============================== ====== ====== ========================
     IDcode                              0     30 CHAR
     Row_id                             30     30 CHAR
     Indexdate                          60     30 CHAR
     StartDate                          90     30 CHAR
     EndDate                           120     30 CHAR
     age                               150     30 CHAR
     female                            180     30 CHAR
     BirthDate                         210     30 CHAR
     ASO_flg                           240     30 CHAR
     state                             270     30 CHAR
     region_cat                        300     30 CHAR
     Idx_IND                           330     30 CHAR
     Idx_SHOP                          360     30 CHAR
     Idx_SG                            390     30 CHAR
     Ever_IND                          420     30 CHAR
     Ever_SHOP                         450     30 CHAR
     Ever_SG                           480     30 CHAR
     Idx_Rx_Elig                       510     30 CHAR
	 ............(160columns )
     innoutfacsurgcopay              30660     30 CHAR
     innoonabahourlmt2               30690     30 CHAR
     innhometmttherapycopay          30720     30 CHAR
     inntmttherapycopay              30750     30 CHAR
     ============================== ====== ====== ========================
     INDICATOR BYTES NEEDED: 129
     EXPECTED RECORD LENGTH: 30909
 
     ===================================================================
     =                                                                 =
     =                   Control Session Connection                    =
     =                                                                 =
     ===================================================================

export_item: connecting sessions
**** 15:20:04 Connecting to RDBMS:    'Datacore'
**** 15:20:04 Connecting with UserId: 'sysdba'
 
**** 15:20:05 Number of Query Band data bytes sent to the RDBMS: 118
 
     ===================================================================
     =                                                                 =
     =                  Teradata Database Information                  =
     =                                                                 =
     ===================================================================

**** 15:20:05 Teradata Database Version:      '16.20.29.01'
**** 15:20:05 Teradata Database Release:      '16.20.29.01'
**** 15:20:05 Maximum request size supported: 1MB
**** 15:20:05 Session character set:          'ASCII'
**** 15:20:05 Total AMPs available:           864
**** 15:20:05 Data Encryption:                supported
**** 15:20:05 NoSpool Export:                 supported
**** 15:20:05 Enhanced Statement Status Level: 1
**** 15:20:05 Blocksize for this job:         64330 bytes
 
     ===================================================================
     =                                                                 =
     =                   Special Session Connection                    =
     =                                                                 =
     ===================================================================

**** 15:20:07 Number of sessions adjusted due to TASM:      4
 
              Instance Assigned Connected Result                
              ======== ======== ========= ======================
                  1         4        4    Successful
              ======== ======== ========= ======================
                Total       4        4    Successful
Job is running in Buffer Mode
Task(INSERT_1[0001]): checkpoint completed, status = Success
Task(SELECT_2[0001]): checkpoint completed, status = Success
 
     ===================================================================
     =                                                                 =
     =                         Select Request                          =
     =                                                                 =
     ===================================================================

**** 15:20:08 This job is submitting the following request:
 
     select cast(IDcode as char(30)),cast(Row_id as char(30)),cast(Indexdate as char(30)),cast(StartDate as char(30)),......
 
export_item: sending SELECT request
             
**** 15:26:12 RDBMS Warning: 3705 Request executed in interpretive EVL mode. 
RDBMS subsystem or component ID: 0
export_item: RDBMS Warning: 3705 Request executed in interpretive EVL mode. 
RDBMS subsystem or component ID: 0
**** 15:26:12 This job will run in the Spool Export mode.
**** 15:26:12 Select execution completed. 7406707 data blocks generated.
Data Block size: 3462320  Buffers/Block: 7  Data Buffer size: 494616
 
     ===================================================================
     =                                                                 =
     =                        End Export Phase                         =
     =                                                                 =
     ===================================================================

export_item: entering End Export Phase
**** 17:37:52 END EXPORT SUBMITTED
Task(INSERT_1[0001]) gets a checkpoint marker.

**** 17:37:52 END EXPORT COMPLETED
Task(INSERT_1[0001]) ready to checkpoint.
 
     Total Records Exported From RDBMS    =  14812985
Task(INSERT_1[0001]): checkpoint completed, status = Success
export_item: Total Rows Exported:  14812985
Task(INSERT_1[0001]) gets EndOfData marker.

Task(INSERT_1[0001]) ready to take EOD checkpoint, status = End Of Data

        Records exported by instance 1    =  14812985
Task(INSERT_1[0001]): checkpoint completed, status = Success
 
     Total Bytes Exported From RDBMS      =  457854553365
 
**** 17:37:52 Export Phase statistics:
Elapsed time: 02:11:40 (hh:mm:ss)
CPU time:     7880.28 Second(s)
MB/sec:       55.271
MB/cpusec:    55.410
TASK(SELECT_2[0001]) sends internal checkpoint request
Task(SELECT_2[0001]) ready to checkpoint after PXTB_AcceptRow

Task(SELECT_2[0001]): checkpoint completed, status = Success
Task(SELECT_2[0001]) ready to take EOD checkpoint, status = Success
Task(SELECT_2[0001]): checkpoint completed, status = Success
TASK(SELECT_2[0001]) completes EOD checkpoint, status = Success

TASK(INSERT_1[0001]) completes EOD checkpoint, status = Success

TPT_INFRA: TPT02255: Message Buffers Sent/Received = 132259, Total Rows Received = 0, Total Rows Sent = 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 132259, Total Rows Received = 0, Total Rows Sent = 0
 
     ===================================================================
     =                                                                 =
     =                        Logoff/Disconnect                        =
     =                                                                 =
     ===================================================================

export_item: disconnecting sessions
**** 20:49:02 Logging off all sessions
file_writer[1]: Total files processed: 1.
 
              Instance      Cpu Time     
              ========  ================ 
                   1     7881.34 Seconds
 
**** 20:49:02 Total processor time used = '7881.34 Second(s)'
.        Start : Fri Mar 22 15:20:01 2019
.        End   : Sat Mar 23 20:49:02 2019
.        Highest return code encountered = '4'.
export_item: Total processor time used = '7881.34 Second(s)'
export_item: Start : Fri Mar 22 15:20:01 2019
export_item: End   : Sat Mar 23 20:49:02 2019
**** 20:49:02 This job terminated
Job step MAIN_STEP terminated (status 4)
Job rbakthavachalam completed successfully, but with warning(s).
Job start: Fri Mar 22 15:19:58 2019
Job end:   Sat Mar 23 20:49:02 2019
Total available memory:          20000000
Largest allocable area:          20000000
Memory use high water mark:       6936128
Free map size:                       1024
Free map use high water mark:          14
Free list use high water mark:          0

RB
Enthusiast

Re: TPT Fast export error General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argumen

OK, I changed the data type from CHAR(30) to varchar(30) and that did the trick. The job completed in few minutes and exported a 40 GB file. I still don't understand why the size is 40 GB when the table size is 10 GB. Here is the log after changing to VARCHAR.

TPT_INFRA: TPT04101: Warning: Teradata PT cannot connect to Unity EcoSystem Manager.
The job will continue without event messages being sent to Unity EcoSystem Manager. 
TPT_INFRA: TPT04197: Warning: OMD API failed to initialize
Teradata Parallel Transporter PlanExecutor Version 15.10.01.11

Teradata Parallel Transporter Executor Version 15.10.01.11
Teradata Parallel Transporter Coordinator Version 15.10.01.11
Teradata Parallel Transporter Executor Version 15.10.01.11
Teradata Parallel Transporter Export Operator Version 15.10.01.11
export_item: private log not specified
 
     ===================================================================
     =                                                                 =
     =              TERADATA PARALLEL TRANSPORTER 64-BIT               =
     =                                                                 =
     =             EXPORT OPERATOR     VERSION 15.10.01.11             =
     =                                                                 =
     =          OPERATOR SUPPORT LIBRARY VERSION 15.10.01.11           =
     =                                                                 =
     =           COPYRIGHT 2001-2017, TERADATA CORPORATION.            =
     =                      ALL RIGHTS RESERVED.                       =
     =                                                                 =
     =                       Process I.D.: 11180                       =
     =                                                                 =
     ===================================================================

**** 19:42:07 Processing starting at: Sun Mar 24 19:42:07 2019
 
     ===================================================================
     =                                                                 =
     =                      Module Identification                      =
     =                                                                 =
     ===================================================================

     64-bit Export Operator for Windows on HCITDETL01
     ExportMain : 15.10.01.03
     ExportCLI  : 15.10.01.02
     ExportUtil : 14.10.00.01
     PcomCLI    : 15.10.01.12
     PcomMBCS   : 14.10.00.02
     PcomMsgs   : 15.10.01.02
     PcomNtfy   : 15.10.00.01
     PcomPx     : 15.10.01.02
     PcomUtil   : 15.10.01.06
     PXICU      : 15.10.01.11
     TDICU      : 15.10.01.03
Teradata Parallel Transporter DataConnector Operator Version 15.10.01.11
file_writer[1]: TPT19206 Attribute 'TraceLevel' value reset to 'Statistics Only'.
file_writer[1]: Instance 1 directing private log report to 'dtacop-rbakthavachalam-36212-1'.
file_writer[1]: DataConnector Consumer operator Instances: 1
file_writer[1]: ECI operator ID: 'file_writer-36212'
file_writer[1]: Operator instance 1 processing file 'D:\Enroll_Eligs\enroll_eligs.txt'.
     CLIv2      : 15.10.01.07   
 
     ===================================================================
     =                                                                 =
     =                      Attribute Definitions                      =
     =                                                                 =
     ===================================================================

**** 19:42:16 Options in effect for this job:
              OperatorType:  Producer
              Instances:     1
              Character set: Not specified; will use default
              Checkpoint:    No checkpoint in effect
              Notify:        Not enabled
              Tenacity:      4 hour limit to successfully connect
              Sleep:         6 minute(s) between connect retries
              Date format:   INTEGERDATE
              Blocksize:     Maximum allowable
              OutLimit:      No limit in effect
              SpoolMode:     Spool
 
     ===================================================================
     =                                                                 =
     =                     Column/Field Definition                     =
     =                                                                 =
     ===================================================================

     Column Name                    Offset Length Type      
     ============================== ====== ====== ========================
     IDcode                              0     30 VARCHAR
     Row_id                             32     30 VARCHAR
..........   
   innofcsurgspccopay              32544     30 VARCHAR
     innoonbarisurglifetimelimit     32576     30 VARCHAR
     innhmoinptcopaydaymax           32608     30 VARCHAR
     innhmoinptprofcopay             32640     30 VARCHAR
     dlrlmtambulance                 32672     30 VARCHAR
     innoutfacsurgcopay              32704     30 VARCHAR
     innoonabahourlmt2               32736     30 VARCHAR
     innhometmttherapycopay          32768     30 VARCHAR
     inntmttherapycopay              32800     30 VARCHAR
     ============================== ====== ====== ========================
     INDICATOR BYTES NEEDED: 129
     EXPECTED RECORD LENGTH: 32961
 
     ===================================================================
     =                                                                 =
     =                   Control Session Connection                    =
     =                                                                 =
     ===================================================================

export_item: connecting sessions
**** 19:42:16 Connecting to RDBMS:    'Datacore'
**** 19:42:16 Connecting with UserId: 'sysdba'
 
**** 19:42:17 Number of Query Band data bytes sent to the RDBMS: 118
 
     ===================================================================
     =                                                                 =
     =                  Teradata Database Information                  =
     =                                                                 =
     ===================================================================

**** 19:42:17 Teradata Database Version:      '16.20.29.01'
**** 19:42:17 Teradata Database Release:      '16.20.29.01'
**** 19:42:17 Maximum request size supported: 1MB
**** 19:42:17 Session character set:          'ASCII'
**** 19:42:17 Total AMPs available:           864
**** 19:42:17 Data Encryption:                supported
**** 19:42:17 NoSpool Export:                 supported
**** 19:42:17 Enhanced Statement Status Level: 1
**** 19:42:17 Blocksize for this job:         64330 bytes
 
     ===================================================================
     =                                                                 =
     =                   Special Session Connection                    =
     =                                                                 =
     ===================================================================

**** 19:42:19 Number of sessions adjusted due to TASM:      4
 
              Instance Assigned Connected Result                
              ======== ======== ========= ======================
                  1         4        4    Successful
              ======== ======== ========= ======================
                Total       4        4    Successful
Job is running in Buffer Mode
Task(INSERT_1[0001]): checkpoint completed, status = Success
Task(SELECT_2[0001]): checkpoint completed, status = Success
 
     ===================================================================
     =                                                                 =
     =                         Select Request                          =
     =                                                                 =
     ===================================================================

**** 19:42:20 This job is submitting the following request:
 
     select cast(IDcode as VARCHAR(30)),...........
 
export_item: sending SELECT request
             
**** 19:43:10 RDBMS Warning: 3705 Request executed in interpretive EVL mode. 
RDBMS subsystem or component ID: 0
export_item: RDBMS Warning: 3705 Request executed in interpretive EVL mode. 
RDBMS subsystem or component ID: 0
**** 19:43:10 This job will run in the Spool Export mode.
**** 19:43:10 Select execution completed. 939853 data blocks generated.
Data Block size: 3461424  Buffers/Block: 7  Data Buffer size: 494488
 
     ===================================================================
     =                                                                 =
     =                        End Export Phase                         =
     =                                                                 =
     ===================================================================

export_item: entering End Export Phase
**** 19:57:31 END EXPORT SUBMITTED
**** 19:57:31 END EXPORT COMPLETED
 
     Total Records Exported From RDBMS    =  14812985
export_item: Total Rows Exported:  14812985
        Records exported by instance 1    =  14812985
 
     Total Bytes Exported From RDBMS      =  58379335508
 
**** 19:57:31 Export Phase statistics:
Elapsed time: 00:14:21 (hh:mm:ss)
CPU time:     858.723 Second(s)
MB/sec:       64.663
MB/cpusec:    64.834
TASK(SELECT_2[0001]) sends internal checkpoint request
TASK(SELECT_2[0001]) ready to take checkpoint, DS status = 0
Task(SELECT_2[0001]) ready to checkpoint after PXTB_AcceptRow

Task(SELECT_2[0001]): checkpoint completed, status = Success
Task(INSERT_1[0001]) gets a checkpoint marker.

Task(INSERT_1[0001]) ready to checkpoint.
Task(INSERT_1[0001]): checkpoint completed, status = Success
Task(SELECT_2[0001]) ready to take EOD checkpoint, status = Success
Task(INSERT_1[0001]) gets EndOfData marker.

Task(INSERT_1[0001]) ready to take EOD checkpoint, status = End Of Data

Task(INSERT_1[0001]): checkpoint completed, status = Success
Task(SELECT_2[0001]): checkpoint completed, status = Success
TASK(SELECT_2[0001]) completes EOD checkpoint, status = Success

TASK(INSERT_1[0001]) completes EOD checkpoint, status = Success

TPT_INFRA: TPT02255: Message Buffers Sent/Received = 16959, Total Rows Received = 0, Total Rows Sent = 0
TPT_INFRA: TPT02255: Message Buffers Sent/Received = 16959, Total Rows Received = 0, Total Rows Sent = 0
 
     ===================================================================
     =                                                                 =
     =                        Logoff/Disconnect                        =
     =                                                                 =
     ===================================================================

export_item: disconnecting sessions
**** 19:57:34 Logging off all sessions
file_writer[1]: Total files processed: 1.
 
              Instance      Cpu Time     
              ========  ================ 
                   1      859.67 Seconds
 
**** 19:57:34 Total processor time used = '859.675 Second(s)'
.        Start : Sun Mar 24 19:42:07 2019
.        End   : Sun Mar 24 19:57:34 2019
.        Highest return code encountered = '4'.
export_item: Total processor time used = '859.675 Second(s)'
export_item: Start : Sun Mar 24 19:42:07 2019
export_item: End   : Sun Mar 24 19:57:34 2019
**** 19:57:34 This job terminated
Job step MAIN_STEP terminated (status 4)
Job rbakthavachalam completed successfully, but with warning(s).
Job start: Sun Mar 24 19:42:03 2019
Job end:   Sun Mar 24 19:57:34 2019
Total available memory:          20000000
Largest allocable area:          20000000
Memory use high water mark:      10395488
Free map size:                       1024
Free map use high water mark:          15
Free list use high water mark:          0

Teradata Employee

Re: TPT Fast export error General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argumen

Hi RB,

 

There are those two things I like in TPT Export :

  1. You can export your file already compressed by just adding .gz in the FileName
  2. You can define your schema from your select, letting TPT does the casting for you : 

 

define schema <MySchema> from select delimited of operator <MyExportOperator>;

I guess the CHAR datatype will materialize every spaces where VARCHAR doesn't.

 

RB
Enthusiast

Re: TPT Fast export error General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argumen

Hi Waldar,

 

Thanks so much for the tips. The gz as extension, worked really well. The size came down to 3 GB :-). 

Ambassador

Re: TPT Fast export error General failure (34): 'pmunxWBuf: fwrite byte count error (Invalid argumen


RB wrote:

OK, I changed the data type from CHAR(30) to varchar(30) and that did the trick. The job completed in few minutes and exported a 40 GB file. I still don't understand why the size is 40 GB when the table size is 10 GB.

Based on your number the average record size in your table is approx. 675 bytes, a bit more than 4 bytes per column. Your table is compressed, is this an IntelliFlex system? Then the 10GB also include Fallback :-)

When you export CHAR you need almost 40KB per row and 80% are spaces:

     Total Bytes Exported From RDBMS      =  457854553365 -- CHAR
     Total Bytes Exported From RDBMS      =   58379335508 -- VARCHAR

 

Why don't you simply export in VARTEXT mode instead of writing a Select with 160 CASTs?