[Spark] No space available in any of the local directories 이슈

org.scala-lang.modules#scala-xml_2.11;1.2.0 from spark-list in [default]
        org.slf4j#slf4j-api;1.7.25 from spark-list in [default]
        xml-apis#xml-apis;1.4.01 from spark-list in [default]
        xml-resolver#xml-resolver;1.2 from spark-list in [default]
        :: evicted modules:
        org.slf4j#slf4j-api;1.7.21 by [org.slf4j#slf4j-api;1.7.25] in [default]
        org.slf4j#slf4j-api;1.7.16 by [org.slf4j#slf4j-api;1.7.21] in [default]
        org.slf4j#slf4j-api;1.7.5 by [org.slf4j#slf4j-api;1.7.21] in [default]
        commons-io#commons-io;2.5 by [commons-io#commons-io;2.6] in [default]
        com.databricks#spark-xml_2.11;0.4.1 by [com.databricks#spark-xml_2.11;0.9.0] in [default]
        org.apache.commons#commons-compress;1.18 by [org.apache.commons#commons-compress;1.19] in [default]
        org.apache.poi#poi;4.0.0 by [org.apache.poi#poi;4.1.0] in [default]
        org.apache.poi#poi-ooxml;4.0.0 by [org.apache.poi#poi-ooxml;4.1.0] in [default]
        org.apache.poi#poi-ooxml-schemas;4.0.0 by [org.apache.poi#poi-ooxml-schemas;4.1.0] in [default]
        ---------------------------------------------------------------------
        |                  |            modules            ||   artifacts   |
        |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
        ---------------------------------------------------------------------
        |      default     |   51  |   0   |   0   |   9   ||   42  |   0   |
        ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-fe3a96ab-15df-4ded-8ffc-3459d6c04f0f
        confs: [default]
        0 artifacts copied, 42 already retrieved (0kB/14ms)
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/01/25 21:03:37 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
22/01/25 21:03:42 ERROR cluster.YarnClientSchedulerBackend: The YARN application has already ended! It might have been killed or the Application Master may have failed to start. Check the YARN application logs for more details.
22/01/25 21:03:42 ERROR spark.SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Application application_1641223372724_1184 failed 2 times due to AM Container for appattempt_1641223372724_1184_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://xxxx:8088/proxy/application_1641223372724_1184/Then, click on links to logs of each attempt.
Diagnostics: No space available in any of the local directories.
Failing this attempt. Failing the application.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:94)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:63)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:185)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:501)
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
        at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926)

Spark shell 수행 시 이러한 메시지가 뜨면서 수행이 안되는 문제 발생했다.

"No space available in any of the local directories"

Spark 처리할 때 사용하는 로컬 경로의 용량이 부족한 것이 원인.
일단 기본적으로 spark.local.dir의 기본 경로인 /tmp로 되어있지만 YARN을 사용하기 때문에 YARN의 yarn.nodemanager.local-dirs 를 사용.

해당 장비의 용량은 충분한 상태인데 왜 계속 문제가 발생할까?

알고보니 YARN NodeManager의 상태가 'bad local dirs' 문제로 뻗어있었는데 이 경로와 yarn.nodemanager.local-dirs가 동일한 경로였고 결국은 yarn 계정으로 접근할 수 없는 문제였음. 권한 이슈를 풀어주니 해결.