Changing Hive Dynamic Partition Limits

Symptoms:

Hive enforces limits on the number of dynamic partitions that it creates.  The default is 100 dynamic partitions per node, with a total (default) limit of 1000 dynamic partitions across all nodes. However, this can be tuned.  If your job tries to create too many dynamic partitions, you may see a stack trace similar to the following:

Diagnostic Messages for this Task:

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.

metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:283)

at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)

at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Caused by: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to: 100

at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:747)

at org.apache.hadoop.hive.ql.exec.FileSinkOperator.startGroup(FileSinkOperator.java:829)

at org.apache.hadoop.hive.ql.exec.Operator.defaultStartGroup(Operator.java:498)

at org.apache.hadoop.hive.ql.exec.Operator.startGroup(Operator.java:521)

at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:232)

... 7 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

MapReduce Jobs Launched:

 

Cause: 

Your Hive job tried to create too many dynamic partitions on a node, exceeding the current limits. The number of dynamic partitions is controlled by the settings forhive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode.  The default is 100 dynamic partitions on a single node, with a limit of 1000 overall.

Solution:

Set the values for hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode to higher values.  For example:

SET hive.exec.dynamic.partition=true;

SET hive.exec.max.dynamic.partitions=2048;

SET hive.exec.max.dynamic.partitions.pernode=256;