2021-09-29

记一次pyspark运行错误,报YarnClientSchedulerBackend失败

他这个错误主要是RPC消息lost,然后executor lost。

我打开yarn管理界面8088,看到主节点是不健康的节点,看他为什么不健康,进入到webui告诉我的log里面,看到可能是因为磁盘太小。
然后我清空了yum缓存,删除了mysql,扩张到了5G的大小。
然后就一切顺利了。

Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
[GCC 7.2.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
21/09/29 11:33:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
21/09/29 11:33:06 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
21/09/29 11:33:25 ERROR YarnClientSchedulerBackend: YARN application has exited unexpectedly with state UNDEFINED!
Check the YARN application logs for more details.
21/09/29 11:33:25 ERROR YarnClientSchedulerBackend: Diagnostics message: Shutdown hook called before final status was reported.
21/09/29 11:33:25 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:745)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:191)
at org.apache.spark.SparkContext.(SparkContext.scala:560)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:25 WARN SparkContext: Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext may be running in this JVM (see SPARK-2243). The other SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
py4j.Gateway.invoke(Gateway.java:238)
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
py4j.GatewayConnection.run(GatewayConnection.java:238)
java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:25 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from /192.168.87.112:38424 is closed
21/09/29 11:33:25 ERROR YarnSchedulerBackend Y a r n S c h e d u l e r E n d p o i n t : S e n d i n g R e q u e s t E x e c u t o r s ( 0 , 0 , M a p ( ) , S e t ( ) ) t o A M w a s u n s u c c e s s f u l j a v a . i o . I O E x c e p t i o n : C o n n e c t i o n f r o m / 192.168.87.112 : 38424 c l o s e d a t o r g . a p a c h e . s p a r k . n e t w o r k . c l i e n t . T r a n s p o r t R e s p o n s e H a n d l e r . c h a n n e l I n a c t i v e ( T r a n s p o r t R e s p o n s e H a n d l e r . j a v a : 146 ) a t o r g . a p a c h e . s p a r k . n e t w o r k . s e r v e r . T r a n s p o r t C h a n n e l H a n d l e r . c h a n n e l I n a c t i v e ( T r a n s p o r t C h a n n e l H a n d l e r . j a v a : 108 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 245 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 231 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . f i r e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 224 ) a t i o . n e t t y . c h a n n e l . C h a n n e l I n b o u n d H a n d l e r A d a p t e r . c h a n n e l I n a c t i v e ( C h a n n e l I n b o u n d H a n d l e r A d a p t e r . j a v a : 75 ) a t i o . n e t t y . h a n d l e r . t i m e o u t . I d l e S t a t e H a n d l e r . c h a n n e l I n a c t i v e ( I d l e S t a t e H a n d l e r . j a v a : 277 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 245 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 231 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . f i r e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 224 ) a t i o . n e t t y . c h a n n e l . C h a n n e l I n b o u n d H a n d l e r A d a p t e r . c h a n n e l I n a c t i v e ( C h a n n e l I n b o u n d H a n d l e r A d a p t e r . j a v a : 75 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 245 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 231 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . f i r e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 224 ) a t i o . n e t t y . c h a n n e l . C h a n n e l I n b o u n d H a n d l e r A d a p t e r . c h a n n e l I n a c t i v e ( C h a n n e l I n b o u n d H a n d l e r A d a p t e r . j a v a : 75 ) a t o r g . a p a c h e . s p a r k . n e t w o r k . u t i l . T r a n s p o r t F r a m e D e c o d e r . c h a n n e l I n a c t i v e ( T r a n s p o r t F r a m e D e c o d e r . j a v a : 182 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 245 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 231 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . f i r e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 224 ) a t i o . n e t t y . c h a n n e l . D e f a u l t C h a n n e l P i p e l i n e YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful java.io.IOException: Connection from /192.168.87.112:38424 closed at org.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146) at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.DefaultChannelPipeline YarnSchedulerEndpoint:SendingRequestExecutors(0,0,Map(),Set())toAMwasunsuccessfuljava.io.IOException:Connectionfrom/192.168.87.112:38424closedatorg.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146)atorg.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atio.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atorg.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.DefaultChannelPipelineHeadContext.channelInactive(DefaultChannelPipeline.java:1354)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor 5. r u n ( S i n g l e T h r e a d E v e n t E x e c u t o r . j a v a : 858 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t T h r e a d F a c t o r y 5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory 5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:25 ERROR Utils: Uncaught exception in thread YARN application state monitor
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils . a w a i t R e s u l t ( T h r e a d U t i l s . s c a l a : 226 ) a t o r g . a p a c h e . s p a r k . r p c . R p c T i m e o u t . a w a i t R e s u l t ( R p c T i m e o u t . s c a l a : 75 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . c l u s t e r . C o a r s e G r a i n e d S c h e d u l e r B a c k e n d . r e q u e s t T o t a l E x e c u t o r s ( C o a r s e G r a i n e d S c h e d u l e r B a c k e n d . s c a l a : 574 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . c l u s t e r . Y a r n S c h e d u l e r B a c k e n d . s t o p ( Y a r n S c h e d u l e r B a c k e n d . s c a l a : 98 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . c l u s t e r . Y a r n C l i e n t S c h e d u l e r B a c k e n d . s t o p ( Y a r n C l i e n t S c h e d u l e r B a c k e n d . s c a l a : 164 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . T a s k S c h e d u l e r I m p l . s t o p ( T a s k S c h e d u l e r I m p l . s c a l a : 581 ) a t o r g . a p a c h e . s p a r k . s c h e d u l e r . D A G S c h e d u l e r . s t o p ( D A G S c h e d u l e r . s c a l a : 2040 ) a t o r g . a p a c h e . s p a r k . S p a r k C o n t e x t .awaitResult(ThreadUtils.scala:226) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75) at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:574) at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:98) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:164) at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:581) at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2040) at org.apache.spark.SparkContext .awaitResult(ThreadUtils.scala:226)atorg.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)atorg.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:574)atorg.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:98)atorg.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:164)atorg.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:581)atorg.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2040)atorg.apache.spark.SparkContext a n o n f u n anonfun anonfunstop 6. a p p l y 6.apply 6.applymcV s p ( S p a r k C o n t e x t . s c a l a : 1949 ) a t o r g . a p a c h e . s p a r k . u t i l . U t i l s sp(SparkContext.scala:1949) at org.apache.spark.util.Utils sp(SparkContext.scala:1949)atorg.apache.spark.util.Utils.tryLogNonFatalError(Utils.scala:1340)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1948)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend M o n i t o r T h r e a d . r u n ( Y a r n C l i e n t S c h e d u l e r B a c k e n d . s c a l a : 121 ) C a u s e d b y : j a v a . i o . I O E x c e p t i o n : C o n n e c t i o n f r o m / 192.168.87.112 : 38424 c l o s e d a t o r g . a p a c h e . s p a r k . n e t w o r k . c l i e n t . T r a n s p o r t R e s p o n s e H a n d l e r . c h a n n e l I n a c t i v e ( T r a n s p o r t R e s p o n s e H a n d l e r . j a v a : 146 ) a t o r g . a p a c h e . s p a r k . n e t w o r k . s e r v e r . T r a n s p o r t C h a n n e l H a n d l e r . c h a n n e l I n a c t i v e ( T r a n s p o r t C h a n n e l H a n d l e r . j a v a : 108 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 245 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 231 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . f i r e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 224 ) a t i o . n e t t y . c h a n n e l . C h a n n e l I n b o u n d H a n d l e r A d a p t e r . c h a n n e l I n a c t i v e ( C h a n n e l I n b o u n d H a n d l e r A d a p t e r . j a v a : 75 ) a t i o . n e t t y . h a n d l e r . t i m e o u t . I d l e S t a t e H a n d l e r . c h a n n e l I n a c t i v e ( I d l e S t a t e H a n d l e r . j a v a : 277 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 245 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 231 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . f i r e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 224 ) a t i o . n e t t y . c h a n n e l . C h a n n e l I n b o u n d H a n d l e r A d a p t e r . c h a n n e l I n a c t i v e ( C h a n n e l I n b o u n d H a n d l e r A d a p t e r . j a v a : 75 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 245 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 231 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . f i r e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 224 ) a t i o . n e t t y . c h a n n e l . C h a n n e l I n b o u n d H a n d l e r A d a p t e r . c h a n n e l I n a c t i v e ( C h a n n e l I n b o u n d H a n d l e r A d a p t e r . j a v a : 75 ) a t o r g . a p a c h e . s p a r k . n e t w o r k . u t i l . T r a n s p o r t F r a m e D e c o d e r . c h a n n e l I n a c t i v e ( T r a n s p o r t F r a m e D e c o d e r . j a v a : 182 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 245 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . i n v o k e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 231 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . f i r e C h a n n e l I n a c t i v e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 224 ) a t i o . n e t t y . c h a n n e l . D e f a u l t C h a n n e l P i p e l i n e MonitorThread.run(YarnClientSchedulerBackend.scala:121) Caused by: java.io.IOException: Connection from /192.168.87.112:38424 closed at org.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146) at org.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75) at org.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231) at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224) at io.netty.channel.DefaultChannelPipeline MonitorThread.run(YarnClientSchedulerBackend.scala:121)Causedby:java.io.IOException:Connectionfrom/192.168.87.112:38424closedatorg.apache.spark.network.client.TransportResponseHandler.channelInactive(TransportResponseHandler.java:146)atorg.apache.spark.network.server.TransportChannelHandler.channelInactive(TransportChannelHandler.java:108)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atio.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)atorg.apache.spark.network.util.TransportFrameDecoder.channelInactive(TransportFrameDecoder.java:182)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)atio.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)atio.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)atio.netty.channel.DefaultChannelPipelineHeadContext.channelInactive(DefaultChannelPipeline.java:1354)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor 5. r u n ( S i n g l e T h r e a d E v e n t E x e c u t o r . j a v a : 858 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t T h r e a d F a c t o r y 5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory 5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:25 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
21/09/29 11:33:37 ERROR Inbox: Ignoring error
java.lang.NullPointerException
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend D r i v e r E n d p o i n t DriverEndpoint DriverEndpoint a n o n f u n anonfun anonfunreceiveAndReply 1. a p p l y O r E l s e ( C o a r s e G r a i n e d S c h e d u l e r B a c k e n d . s c a l a : 232 ) a t o r g . a p a c h e . s p a r k . r p c . n e t t y . I n b o x 1.applyOrElse(CoarseGrainedSchedulerBackend.scala:232) at org.apache.spark.rpc.netty.Inbox 1.applyOrElse(CoarseGrainedSchedulerBackend.scala:232)atorg.apache.spark.rpc.netty.Inbox a n o n f u n anonfun anonfunprocess 1. a p p l y 1.apply 1.applymcV s p ( I n b o x . s c a l a : 105 ) a t o r g . a p a c h e . s p a r k . r p c . n e t t y . I n b o x . s a f e l y C a l l ( I n b o x . s c a l a : 205 ) a t o r g . a p a c h e . s p a r k . r p c . n e t t y . I n b o x . p r o c e s s ( I n b o x . s c a l a : 101 ) a t o r g . a p a c h e . s p a r k . r p c . n e t t y . D i s p a t c h e r sp(Inbox.scala:105) at org.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205) at org.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101) at org.apache.spark.rpc.netty.Dispatcher sp(Inbox.scala:105)atorg.apache.spark.rpc.netty.Inbox.safelyCall(Inbox.scala:205)atorg.apache.spark.rpc.netty.Inbox.process(Inbox.scala:101)atorg.apache.spark.rpc.netty.DispatcherMessageLoop.run(Dispatcher.scala:221)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor W o r k e r . r u n ( T h r e a d P o o l E x e c u t o r . j a v a : 624 ) a t j a v a . l a n g . T h r e a d . r u n ( T h r e a d . j a v a : 748 ) 21 / 09 / 2911 : 33 : 37 W A R N T r a n s p o r t C h a n n e l H a n d l e r : E x c e p t i o n i n c o n n e c t i o n f r o m / 192.168.87.113 : 59216 j a v a . i o . I O E x c e p t i o n : C o n n e c t i o n r e s e t b y p e e r a t s u n . n i o . c h . F i l e D i s p a t c h e r I m p l . r e a d 0 ( N a t i v e M e t h o d ) a t s u n . n i o . c h . S o c k e t D i s p a t c h e r . r e a d ( S o c k e t D i s p a t c h e r . j a v a : 39 ) a t s u n . n i o . c h . I O U t i l . r e a d I n t o N a t i v e B u f f e r ( I O U t i l . j a v a : 223 ) a t s u n . n i o . c h . I O U t i l . r e a d ( I O U t i l . j a v a : 192 ) a t s u n . n i o . c h . S o c k e t C h a n n e l I m p l . r e a d ( S o c k e t C h a n n e l I m p l . j a v a : 380 ) a t i o . n e t t y . b u f f e r . P o o l e d U n s a f e D i r e c t B y t e B u f . s e t B y t e s ( P o o l e d U n s a f e D i r e c t B y t e B u f . j a v a : 288 ) a t i o . n e t t y . b u f f e r . A b s t r a c t B y t e B u f . w r i t e B y t e s ( A b s t r a c t B y t e B u f . j a v a : 1106 ) a t i o . n e t t y . c h a n n e l . s o c k e t . n i o . N i o S o c k e t C h a n n e l . d o R e a d B y t e s ( N i o S o c k e t C h a n n e l . j a v a : 343 ) a t i o . n e t t y . c h a n n e l . n i o . A b s t r a c t N i o B y t e C h a n n e l Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21/09/29 11:33:37 WARN TransportChannelHandler: Exception in connection from /192.168.87.113:59216 java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) at io.netty.channel.nio.AbstractNioByteChannel Worker.run(ThreadPoolExecutor.java:624)atjava.lang.Thread.run(Thread.java:748)21/09/2911:33:37WARNTransportChannelHandler:Exceptioninconnectionfrom/192.168.87.113:59216java.io.IOException:Connectionresetbypeeratsun.nio.ch.FileDispatcherImpl.read0(NativeMethod)atsun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)atsun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)atsun.nio.ch.IOUtil.read(IOUtil.java:192)atsun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)atio.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)atio.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)atio.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)atio.netty.channel.nio.AbstractNioByteChannelNioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor 5. r u n ( S i n g l e T h r e a d E v e n t E x e c u t o r . j a v a : 858 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t T h r e a d F a c t o r y 5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory 5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
21/09/29 11:33:43 ERROR YarnClientSchedulerBackend: YARN application has exited unexpectedly with state FAILED! Check the YARN application logs for more details.
21/09/29 11:33:43 ERROR YarnClientSchedulerBackend: Diagnostics message: Application application_1632884832055_0005 failed 2 times due to AM Container for appattempt_1632884832055_0005_000002 exited with exitCode: -103
For more detailed output, check application tracking page:http://Spark11:8089/proxy/application_1632884832055_0005/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=28319,containerID=container_1632884832055_0005_02_000001] is running beyond virtual memory limits. Current usage: 265.0 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1632884832055_0005_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 28325 28319 28319 28319 (java) 433 138 2359644160 67542 /home/wang/env/jdk1.8.0_191/bin/java -server -Xmx512m -Djava.io.tmpdir=/mnt/x3/dfs/hadooptmp/nm-local-dir/usercache/wang/appcache/application_1632884832055_0005/container_1632884832055_0005_02_000001/tmp -Dspark.yarn.app.container.log.dir=/home/wang/env/hadoop-2.6.5/logs/userlogs/application_1632884832055_0005/container_1632884832055_0005_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg Spark11:40079 --properties-file /mnt/x3/dfs/hadooptmp/nm-local-dir/usercache/wang/appcache/application_1632884832055_0005/container_1632884832055_0005_02_000001/spark_conf/spark_conf.properties
|- 28319 28317 28319 28319 (bash) 0 0 115896320 307 /bin/bash -c /home/wang/env/jdk1.8.0_191/bin/java -server -Xmx512m -Djava.io.tmpdir=/mnt/x3/dfs/hadooptmp/nm-local-dir/usercache/wang/appcache/application_1632884832055_0005/container_1632884832055_0005_02_000001/tmp -Dspark.yarn.app.container.log.dir=/home/wang/env/hadoop-2.6.5/logs/userlogs/application_1632884832055_0005/container_1632884832055_0005_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg ‘Spark11:40079’ --properties-file /mnt/x3/dfs/hadooptmp/nm-local-dir/usercache/wang/appcache/application_1632884832055_0005/container_1632884832055_0005_02_000001/spark_conf/spark_conf.properties 1> /home/wang/env/hadoop-2.6.5/logs/userlogs/application_1632884832055_0005/container_1632884832055_0005_02_000001/stdout 2> /home/wang/env/hadoop-2.6.5/logs/userlogs/application_1632884832055_0005/container_1632884832055_0005_02_000001/stderr

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
21/09/29 11:33:43 ERROR TransportClient: Failed to send RPC RPC 4670801397555361713 to /192.168.87.112:56042: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel A b s t r a c t U n s a f e . w r i t e ( . . . ) ( U n k n o w n S o u r c e ) 21 / 09 / 2911 : 33 : 43 E R R O R Y a r n S c h e d u l e r B a c k e n d AbstractUnsafe.write(...)(Unknown Source) 21/09/29 11:33:43 ERROR YarnSchedulerBackend AbstractUnsafe.write(...)(UnknownSource)21/09/2911:33:43ERRORYarnSchedulerBackendYarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map(),Set()) to AM was unsuccessful
java.io.IOException: Failed to send RPC RPC 4670801397555361713 to /192.168.87.112:56042: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient R p c C h a n n e l L i s t e n e r . h a n d l e F a i l u r e ( T r a n s p o r t C l i e n t . j a v a : 357 ) a t o r g . a p a c h e . s p a r k . n e t w o r k . c l i e n t . T r a n s p o r t C l i e n t RpcChannelListener.handleFailure(TransportClient.java:357) at org.apache.spark.network.client.TransportClient RpcChannelListener.handleFailure(TransportClient.java:357)atorg.apache.spark.network.client.TransportClientStdChannelListener.operationComplete(TransportClient.java:334)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)
at io.netty.channel.AbstractChannel A b s t r a c t U n s a f e . s a f e S e t F a i l u r e ( A b s t r a c t C h a n n e l . j a v a : 987 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l AbstractUnsafe.safeSetFailure(AbstractChannel.java:987) at io.netty.channel.AbstractChannel AbstractUnsafe.safeSetFailure(AbstractChannel.java:987)atio.netty.channel.AbstractChannelAbstractUnsafe.write(AbstractChannel.java:869)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1316)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.access 1900 ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 38 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t 1900(AbstractChannelHandlerContext.java:38) at io.netty.channel.AbstractChannelHandlerContext 1900(AbstractChannelHandlerContext.java:38)atio.netty.channel.AbstractChannelHandlerContextAbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
at io.netty.channel.AbstractChannelHandlerContext W r i t e A n d F l u s h T a s k . w r i t e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 1128 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128) at io.netty.channel.AbstractChannelHandlerContext WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)atio.netty.channel.AbstractChannelHandlerContextAbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor 5. r u n ( S i n g l e T h r e a d E v e n t E x e c u t o r . j a v a : 858 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t T h r e a d F a c t o r y 5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory 5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel A b s t r a c t U n s a f e . w r i t e ( . . . ) ( U n k n o w n S o u r c e ) 21 / 09 / 2911 : 33 : 43 E R R O R U t i l s : U n c a u g h t e x c e p t i o n i n t h r e a d Y A R N a p p l i c a t i o n s t a t e m o n i t o r o r g . a p a c h e . s p a r k . S p a r k E x c e p t i o n : E x c e p t i o n t h r o w n i n a w a i t R e s u l t : a t o r g . a p a c h e . s p a r k . u t i l . T h r e a d U t i l s AbstractUnsafe.write(...)(Unknown Source) 21/09/29 11:33:43 ERROR Utils: Uncaught exception in thread YARN application state monitor org.apache.spark.SparkException: Exception thrown in awaitResult: at org.apache.spark.util.ThreadUtils AbstractUnsafe.write(...)(UnknownSource)21/09/2911:33:43ERRORUtils:UncaughtexceptioninthreadYARNapplicationstatemonitororg.apache.spark.SparkException:ExceptionthrowninawaitResult:atorg.apache.spark.util.ThreadUtils.awaitResult(ThreadUtils.scala:226)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:574)
at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:98)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:164)
at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:581)
at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:2040)
at org.apache.spark.SparkContext$ a n o n f u n anonfun anonfunstop 6. a p p l y 6.apply 6.applymcV s p ( S p a r k C o n t e x t . s c a l a : 1949 ) a t o r g . a p a c h e . s p a r k . u t i l . U t i l s sp(SparkContext.scala:1949) at org.apache.spark.util.Utils sp(SparkContext.scala:1949)atorg.apache.spark.util.Utils.tryLogNonFatalError(Utils.scala:1340)
at org.apache.spark.SparkContext.stop(SparkContext.scala:1948)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend M o n i t o r T h r e a d . r u n ( Y a r n C l i e n t S c h e d u l e r B a c k e n d . s c a l a : 121 ) C a u s e d b y : j a v a . i o . I O E x c e p t i o n : F a i l e d t o s e n d R P C R P C 4670801397555361713 t o / 192.168.87.112 : 56042 : j a v a . n i o . c h a n n e l s . C l o s e d C h a n n e l E x c e p t i o n a t o r g . a p a c h e . s p a r k . n e t w o r k . c l i e n t . T r a n s p o r t C l i e n t MonitorThread.run(YarnClientSchedulerBackend.scala:121) Caused by: java.io.IOException: Failed to send RPC RPC 4670801397555361713 to /192.168.87.112:56042: java.nio.channels.ClosedChannelException at org.apache.spark.network.client.TransportClient MonitorThread.run(YarnClientSchedulerBackend.scala:121)Causedby:java.io.IOException:FailedtosendRPCRPC4670801397555361713to/192.168.87.112:56042:java.nio.channels.ClosedChannelExceptionatorg.apache.spark.network.client.TransportClientRpcChannelListener.handleFailure(TransportClient.java:357)
at org.apache.spark.network.client.TransportClient S t d C h a n n e l L i s t e n e r . o p e r a t i o n C o m p l e t e ( T r a n s p o r t C l i e n t . j a v a : 334 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t P r o m i s e . n o t i f y L i s t e n e r 0 ( D e f a u l t P r o m i s e . j a v a : 507 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t P r o m i s e . n o t i f y L i s t e n e r s N o w ( D e f a u l t P r o m i s e . j a v a : 481 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t P r o m i s e . n o t i f y L i s t e n e r s ( D e f a u l t P r o m i s e . j a v a : 420 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t P r o m i s e . t r y F a i l u r e ( D e f a u l t P r o m i s e . j a v a : 122 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l StdChannelListener.operationComplete(TransportClient.java:334) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at io.netty.channel.AbstractChannel StdChannelListener.operationComplete(TransportClient.java:334)atio.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)atio.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)atio.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)atio.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122)atio.netty.channel.AbstractChannelAbstractUnsafe.safeSetFailure(AbstractChannel.java:987)
at io.netty.channel.AbstractChannel A b s t r a c t U n s a f e . w r i t e ( A b s t r a c t C h a n n e l . j a v a : 869 ) a t i o . n e t t y . c h a n n e l . D e f a u l t C h a n n e l P i p e l i n e AbstractUnsafe.write(AbstractChannel.java:869) at io.netty.channel.DefaultChannelPipeline AbstractUnsafe.write(AbstractChannel.java:869)atio.netty.channel.DefaultChannelPipelineHeadContext.write(DefaultChannelPipeline.java:1316)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:738)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:730)
at io.netty.channel.AbstractChannelHandlerContext.access 1900 ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 38 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t 1900(AbstractChannelHandlerContext.java:38) at io.netty.channel.AbstractChannelHandlerContext 1900(AbstractChannelHandlerContext.java:38)atio.netty.channel.AbstractChannelHandlerContextAbstractWriteTask.write(AbstractChannelHandlerContext.java:1081)
at io.netty.channel.AbstractChannelHandlerContext W r i t e A n d F l u s h T a s k . w r i t e ( A b s t r a c t C h a n n e l H a n d l e r C o n t e x t . j a v a : 1128 ) a t i o . n e t t y . c h a n n e l . A b s t r a c t C h a n n e l H a n d l e r C o n t e x t WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128) at io.netty.channel.AbstractChannelHandlerContext WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1128)atio.netty.channel.AbstractChannelHandlerContextAbstractWriteTask.run(AbstractChannelHandlerContext.java:1070)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
at io.netty.util.concurrent.SingleThreadEventExecutor 5. r u n ( S i n g l e T h r e a d E v e n t E x e c u t o r . j a v a : 858 ) a t i o . n e t t y . u t i l . c o n c u r r e n t . D e f a u l t T h r e a d F a c t o r y 5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory 5.run(SingleThreadEventExecutor.java:858)atio.netty.util.concurrent.DefaultThreadFactoryDefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(…)(Unknown Source)
21/09/29 11:33:43 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:745)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:191)
at org.apache.spark.SparkContext.(SparkContext.scala:560)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)
/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/shell.py:45: UserWarning: Failed to initialize Spark session.
warnings.warn(“Failed to initialize Spark session.”)
Traceback (most recent call last):
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/shell.py”, line 41, in
spark = SparkSession._create_shell_session()
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/sql/session.py”, line 583, in _create_shell_session
return SparkSession.builder.getOrCreate()
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/sql/session.py”, line 173, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/context.py”, line 349, in getOrCreate
SparkContext(conf=conf or SparkConf())
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/context.py”, line 118, in init
conf, jsc, profiler_cls)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/context.py”, line 180, in _do_init
self._jsc = jsc or self._initialize_context(self._conf._jconf)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/pyspark/context.py”, line 288, in _initialize_context
return self._jvm.JavaSparkContext(jconf)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py”, line 1525, in call
answer, self._gateway_client, None, self._fqn)
File “/home/wang/env/spark-2.4.0-bin-hadoop2.6/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py”, line 328, in get_return_value
format(target_id, “.”, name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalStateException: Spark context stopped while waiting for backend
at org.apache.spark.scheduler.TaskSchedulerImpl.waitBackendReady(TaskSchedulerImpl.scala:745)
at org.apache.spark.scheduler.TaskSchedulerImpl.postStartHook(TaskSchedulerImpl.scala:191)
at org.apache.spark.SparkContext.(SparkContext.scala:560)
at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

你可能感兴趣的