# Changelog

## Score

- CAI 43 → 43 (-0.1)
- Rubric changed (rubric-2026.10.5 → rubric-2026.11.4) — scores are not directly comparable.

## Lenses

- Code Health 34 → 31 (-2.3)
- Architecture 95 → 95 (+0.0)
- Maturity 74 → 75 (+0.9)
- Readiness 48 → 51 (+2.3)
- Security 73 → 69 (-3.8)
- Accessibility 38 → 40 (+2.3)
- Performance 84 → 100 (+15.6)

## Resolved (264)

- ArrowBatchedUDFArrowHandler.run (cyclomatic 17) (python/pyspark/eval_handlers/_arrow.py)
- Change coupling: CTESubstitution.scala ↔ PushdownPredicatesAndPruneColumnsForCTEDef.scala (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala)
- Change coupling: DataSourceV2ScanExecBase.scala ↔ GroupPartitionsExec.scala (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExecBase.scala)
- CheckAnalysis.checkAlterTableCommand (cognitive 32) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala)
- CheckAnalysis.checkAlterTableCommand (cyclomatic 21) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala)
- ClassTooLong: ResolveSessionCatalog (sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala)
- Critical CVE: [GHSA redacted] (core/pom.xml)
- DataStreamWriter.startInternal (cognitive 29) (sql/core/src/main/scala/org/apache/spark/sql/classic/DataStreamWriter.scala)
- Duplicated block (11 lines × 2) (common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java)
- Duplicated block (11 lines × 2) (examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java)
- Duplicated block (11 lines × 2) (examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestClassificationExample.java)
- Duplicated block (11 lines × 2) (mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala)
- Duplicated block (11 lines × 2) (mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala)
- Duplicated block (11 lines × 2) (python/pyspark/ml/evaluation.py)
- Duplicated block (11 lines × 2) (python/pyspark/sql/connect/plan.py)
- Duplicated block (12 lines × 2) (core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala)
- Duplicated block (12 lines × 2) (sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala)
- Duplicated block (12 lines × 3) (sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala)
- Duplicated block (13–14 lines × 2) (examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala)
- Duplicated block (13–14 lines × 3) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala)
- …and 244 more

## New (1418)

- AlterTableValidation.validate (cognitive 32) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlterTableValidation.scala)
- AlterTableValidation.validate (cyclomatic 22) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlterTableValidation.scala)
- AsOfJoinMatchConditionResolution.coerceNearestKeys (cognitive 16) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AsOfJoinMatchConditionResolution.scala)
- AstBuilder.visitJsonQuery (cognitive 18) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala)
- BasicExecutorFeatureStep.configurePod (cyclomatic 18) (resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala)
- BroadcastRangeJoinExec.makeCandidateFn (cognitive 31) (sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastRangeJoinExec.scala)
- BroadcastRangeJoinExec.makeCandidateFn (cyclomatic 17) (sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastRangeJoinExec.scala)
- Change coupling: RewriteMergeIntoTable.scala ↔ RewriteUpdateTable.scala (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala)
- ClassTooLong: NativeLibrary (sql/core/src/main/java/org/apache/spark/sql/execution/datasources/v2/ffi/NativeLibrary.java)
- ClassTooLong: V2ExpressionBuilder (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala)
- ColumnDomain.toString (cognitive 18) (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnDomains.scala)
- ColumnDomains.compare (cognitive 17) (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnDomains.scala)
- ColumnDomains.compare (cyclomatic 16) (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/columnar/ColumnDomains.scala)
- DataSourceV2Utils.loadV2Source (cognitive 17) (sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Utils.scala)
- Duplicated block (10 lines × 2) (sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala)
- Duplicated block (11 lines × 2) (mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala)
- Duplicated block (11 lines × 2) (python/pyspark/eval_handlers/_pandas.py)
- Duplicated block (11 lines × 3) (python/pyspark/eval_handlers/_pandas.py)
- Duplicated block (12 lines × 2) (mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala)
- Duplicated block (12 lines × 2) (sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/PartitionReaderFactory.java)
- …and 1398 more

## Changes since last survey

- 300 commits — 209 feature/other, 91 fixes

## By area

- sql/core — 102 commits
- sql/catalyst — 70 commits
- python/pyspark — 36 commits
- .github/workflows — 16 commits
- core/src — 13 commits
- (root) — 12 commits
- docs/sql-migration-guide.md — 6 commits
- sql/hive — 6 commits
- dev/merge_spark_pr.py — 4 commits
- sql/connect — 4 commits
- common/variant — 3 commits
- dev/sparktestsupport — 3 commits
- mllib/src — 3 commits
- sql/pipelines — 3 commits
- common/utils — 2 commits
- .github/labeler.yml — 1 commit
- binder/Dockerfile — 1 commit
- build/mvn — 1 commit
- connector/avro — 1 commit
- connector/kafka-0-10-sql — 1 commit

## Notable commits

- fix: [MINOR][PYTHON] Fix framed serializer length limit at 2 GiB
- fix: [SPARK-33744][SQL] Fix canonicalization of SortAggregateExec
- fix: [SPARK-48995][PYTHON] Fix intermittent NullPointerException in Column.startswith/endswith with None
- fix: [SPARK-55014][SQL][FOLLOWUP] Fix the version of `spark.sql.optimizer.splitProjectionForExpensiveFilters` to 4.4.0
- fix: [SPARK-60124][SQL] Fix RewriteDistinctAggregates failure on ListAgg with DISTINCT and FILTER
- fix: [SPARK-60421][SQL] Fix overflow and non-finite handling in vector_avg for float vectors
- fix: [SPARK-60522][SPARK-60027][SPARK-60539][SQL] Fix storage-partitioned join failures and wrong results when Spark cannot evaluate a partition transform as is
- fix: [SPARK-60608][ML][CONNECT][TEST][FOLLOWUP] Strengthen nested model allowlist regression coverage
- fix: [SPARK-60651][SQL] Expand decimal remainder regression coverage
- fix: [SPARK-60655][PS][SQL] Fix modulo with an infinite divisor in pandas API on Spark
- fix: [SPARK-60656][PS] Fix NumPy maximum/minimum NaN propagation
- fix: [SPARK-60743][SQL] Fix make_timestamp silently wrapping an out-of-range seconds value
- fix: [SPARK-60780][SQL] Fix internal error when table OPTIONS contain a subquery
- fix: [SPARK-60797][SQL] Fix wrong decimal partition values when the scale differs from the schema
- fix: [SPARK-60801][SQL] Fix wrong results by reading the Hive default partition as NULL
- fix: [SPARK-60803][SQL] Fix auto-filled DSv2 generated columns reading a struct field before the store assignment cast
- fix: [SPARK-60808][SQL] Fix lost join rows when a runtime bloom filter's creation side is non-deterministic
- fix: [SPARK-60814][INFRA] Determine revert vs reapply by wrapper parity in merge_spark_pr.py
- fix: [SPARK-60817][SQL] Fix auto-filled DSv2 generated columns reading a column in a lambda function before the store assignment cast
- fix: [SPARK-60818][SQL] Fix wrong TINYINT and SMALLINT partition values when they are out of range
- …and 280 more

## Architecture

- Unchanged — 0 containers · 14 contexts · 18 edges
