Spark Event Log

A Spark event log records application execution events used to reconstruct supported Spark UI information. It helps investigate completed applications, including jobs, stages, tasks, executors, and SQL execution.

Enable spark.eventLog.enabled before startup and choose a durable location with spark.eventLog.dir. The application must be able to write there and the History Server must be able to read it. A temporary path on a removed driver is not a shared history store.

Keep the application and attempt identifiers with code, input, and configuration versions. Check that the stored application can actually be opened. Retention policies and lossy compaction can remove task-level evidence.

Event logs do not replace application text logs, source/sink monitoring, input snapshots, or checkpoints. They are diagnostic evidence rather than a recovery mechanism. Limit access and inspect redaction for sensitive plan or configuration values.

See Optimizing Spark Jobs in Production for the measurement and rollout workflow.

Further reading

• Apache Spark documentation