Public report of apache/spark, published 26 October 2026. It has the same score, grades and number of
findings as the full report. The details of security findings (where they are in the code, which known
vulnerabilities and which passwords or keys were found) are left out, so nobody can use them against the project.
Public
Executive reportMeasured under the Code Assurance Index, an open standard for code quality, using its published scoring rules (rubric version 2026.10.5). Assert signed this survey and filed it in the standard's public register, where anyone can look it up by its survey ID. Check this survey →
Signed and filed in the standard's public register, 26 October 2026, 08:55 UTC · Survey ID cd_0ea7a53d8e724b20a5e1ff5e4dd04df2
apache/spark
Measured 26 October 2026, 08:45 UTC, at code version c535eb6
A regression is a drop of 0.5 points or more in the score, or of 2.5 or more out of 10 in a dimension, since the last survey.
Code coverage is not measured: JavaScript/TypeScript suite (CouldNotMeasureTestRunFailed).
CAI scoreAt riskDown 1 since the last survey
The score is built from these lenses. Each lens grades one area of the code, such as how easy it is to change or how well it is protected. The weakest lens counts most, so one weak area pulls the score down instead of being averaged away.
A very large codebase: 1,228,043 lines of product code in 52 projects, tests not counted · Estimated rebuild effort: about 14.8 person-years · Weakest lens: Code Health, 34
Findings by how serious they are
211 critical4803 serious686 minor
14 dimensions are left out of the score. For each one, Watchdog either couldn't measure it in this code, or found no evidence of the safeguard it looks for, such as a tested backup restore. They count neither for nor against the score, and Limitations lists each one with its reason.
This survey was produced by
Watchdog
Producer
Canine Development
Analyzer
Watchdog engine 1.0.0
Measured
26 October 2026, 08:45 UTC
A measurement, not a certificate. The Code Assurance Index records a number
together with the evidence it was worked out from, so anyone can recalculate it, but it doesn't certify or guarantee
the code. The standard was written by Canine Development, which also builds Watchdog. It is open to any tool that
implements it, and today Watchdog is the only one that does.
A critical security finding is open, such as a library with a known critical vulnerability or a password or key found in the code. The score itself is unchanged, but its band shows At risk until the finding is fixed. See the Security and Compliance lens.
The score is deterministic, which means the same code always gets the same score. Watchdog works it out with tools that follow published rules, and an AI model never decides it, so anyone who applies the same rules to the same code gets the same number. How it is done is explained under Methodology ▸
85of 85 dimensions measured by tools.
5657of 5700 findings point to an exact file and line, so you can open each one in the code. The other 43 concern the repository as a whole, such as missing documentation.
85dimensions across 7 lenses, read from 1,228,043 lines of product code in 52 projects.
apache/spark is rated At risk because critical security findings are open, most of them in the build pipeline. It scores 43 out of 100, which the At risk rating leaves unchanged.
Code Health, at 34, is the weakest lens and counts most in the score. The best first step is replacing the two exposed passwords or keys and switching the old ones off, worth about 4 points. Adding type checking would lift Code Health itself, also worth about 4. Raising Code Health to 70 would lift the score to about 51.
101 pipeline tools are referenced by a changeable name, so the build pipeline can be tampered with. Forty-three weaknesses in the code could be used by an attacker, and thirty-eight parts of the code are no longer understood by anyone working on it. Pinning each tool to an exact version, fixing the weaknesses the scanner points to and having someone learn the forgotten files would close these risks. Passing outside input through environment variables would fix the nine pipeline steps that put outside input straight into a shell command, and the six places where the code runs text as code or as a command need a direct call instead. The two form fields without a label need a label.
The product code is 1,228,043 lines, about €2,200,000 to rebuild.
A lens is capped at Adequate while one of its dimensions scores below 4 out of 10, however well the rest of the lens does. That way a serious problem in one place always shows in the lens's rating.
How the score is built: what each lens counts forThe lenses don't count equally. The weakest lens counts most, the next weakest a little less, and so on, so a weak area pulls the score down instead of being averaged away. In the bar below, each part's width shows how much that lens counts and its colour shows its rating. Improving the widest part moves the score most, and as that lens improves, it counts for less, because another lens becomes the weakest.
If Code Health rose from 34 to 70, the lowest score rated Strong, the score would rise from 43 to about 51.
Code composition: what the lines of code are
Product code 50%Tests 50%
New since the last survey: 144 findings
These 144 critical or serious findings weren't in the previous survey, on 12 October 2026. The first 100 are listed here, and the findings file in the download package lists them all.
Watchdog compares every finding with the previous survey's. A finding that has only moved to another line in the same file counts as the same finding, so only new findings appear here. Advisory notes from the AI model are left out, because the model words them afresh in every survey.
Fixed since the last survey: 143 findings
These 143 critical or serious findings were in the previous survey, on 12 October 2026, and are gone from this one. The first 100 are listed here.
€710,000 to €3,600,000, for 11,900 to 37,752 hours of work, about 14.8 person-years for 8 to 28 engineers
Difficulty of the problem
Standard, judged from the kind of software and how it is built. Code that solves a harder problem, such as payments or medical bookings, costs more per line to write.
Quality factor
×0.7 at a score of 43. Better code takes more work to write, so the factor is ×1.0 at a score of 70, lower below it and higher above it.
Size
Very large: 1,228,043 lines of product code
What it is not
Not a quotation: nobody has offered to build it for that. Not a valuation: it says nothing about what the business is worth. Not a certified figure: no professional has signed it as one.
How the estimate is made: Watchdog prices the hours a team would need to write this code again, at €60 to €95 an hour. Declarations and boilerplate are priced at a fast rate and logic at a slower one, and the logic is then multiplied by the difficulty of the problem and by the quality factor. The estimate can be about 30% off either way, and it moves most with the hourly rate and the difficulty. For this code, every line is priced as straight logic, which keeps the estimate on the low side.
Top priorities
The three actions that raise the score most for the work they take. The full ranked list is in the Roadmap.
1
Resolve the 2 Leaked secret finding(s) in REDACTED Scanning — start with REDACTED, REDACTED.
Adopt TypeScript for the frontend: add a typecheck step to the build, then either type-check the existing JavaScript in place (`checkJs`/JSDoc types) or convert the highest-traffic modules first — no file is typed today, so this is an adoption, not a clean-up.
Raises the score by about 4 points · Medium effort · Type Safety
3
Make custom controls keyboard-operable (role + tabindex + key handler), drop positive tabindex, and give anchors a real href.
Each conclusion below combines several dimensions into one point that no single dimension shows, and lists the measurements it rests on.
Where the investment is most exposed · Value at risk
This code would cost about €2,200,000 to build again, about 14.8 person-years of work. Its weakest lens is Code Health, at 34 (Weak). Code Health is about how easy the code is to read and change safely, and code that is hard to read is slow and risky to change, so every future change costs more than it should. Because the weakest lens counts most in the score, work on Code Health also moves the score most.
Based on: the rebuild estimate · the Code Health grade · how much each lens counts
What to do: Start with the Code Health actions in Top priorities.
If you do one thing first · First action
Resolve the 2 Leaked secret finding(s) in REDACTED Scanning. Of all the actions in this report, it raises the score most for the work it takes: about 4 points for low effort, because it clears the 2 REDACTED Scanning findings. The next survey will show the change in the score.
Based on: the ranked actions in Top priorities, by score gain and effort
What hard-to-read code costs each year · Cost
Four dimensions measure how easy the code is to work in: two measure how tangled its logic is, one how much of it is copied, and one how well each part sticks to a single job. In this codebase they average 7.5 out of 10. Code at that level is slower to change than clean code, and Watchdog estimates the extra work at 2% to 5% on every change. The history of the code shows how much it changes: about 430,249 lines over the last 90 days, which is about 1,744,899 lines a year. At that pace, the extra work comes to an estimated 87.2 to 581.6 engineer-days a year, and it grows as more code changes.
How this is estimated: The number of changed lines is measured from the history of the code. The rest rests on two assumptions of Watchdog's own: code scoring 8 out of 10 or more adds no extra work, and each point below 8 adds 2.5% to 5.6% to every change, never less than 2% to 5% and never more than 40% to 60% in all; and an engineer changes 150 to 400 lines in a day. The result is a range for planning, not a measurement.
Based on: Cyclomatic Complexity · Cognitive Complexity · Code Duplication · Cohesion (LCOM4) · the code's change history
What to do: Start with the files listed under Churn × Complexity Hotspots, which change often and are hard to read, because that is where the extra work is paid most often.
Architecture: which parts of the code depend on which
The code is divided into modules, groups of code that each handle one part of the product, and a module depends on another when it uses code from it. The table lists the modules both down the side and across the top, in an order where each module should only use modules listed above it. A number in a cell means the module on that row uses the module in that column, and it counts the pairs of building blocks (types) that connect them. The one thing to look for is a mark above the diagonal line. It means two or more modules use each other in a loop (a dependency cycle), so none of them can be changed, tested or released on its own.
1349 modules, with 4477 dependencies between them. 11 loops run through 299 of the modules, marked above the diagonal.
The table shows the 40 modules with the most connections; 1309 more are left out to keep it readable.
Which modules use which. The module on each row uses the module in each column, and the number counts the pairs of types that connect them. A cell above the diagonal is part of a loop.
scheduler uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break scheduler, while a change to scheduler can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: scheduler uses a module listed above it, which is the healthy direction.
scheduler uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break scheduler, while a change to scheduler can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
scheduler uses rdd, so a change to rdd can break scheduler, while a change to scheduler can't break rdd.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
4→39 scheduler uses org.apache.sparkpart of a loop✕
Connections
37 pairs of types, where a type in scheduler uses a type in org.apache.spark. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.connector.catalog uses internal, so a change to internal can break sql.connector.catalog, while a change to sql.connector.catalog can't break internal.
Position
Below the diagonal: sql.connector.catalog uses a module listed above it, which is the healthy direction.
5→17 sql.connector.catalog uses sql.execution.datasources.v2part of a loop✕
Connections
2 pairs of types, where a type in sql.connector.catalog uses a type in sql.execution.datasources.v2.
sql.connector.catalog uses sql.execution.datasources.v2, so a change to sql.execution.datasources.v2 can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.execution.datasources.v2.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→20 sql.connector.catalog uses sql.catalyst.plans.logicalpart of a loop✕
Connections
4 pairs of types, where a type in sql.connector.catalog uses a type in sql.catalyst.plans.logical.
sql.connector.catalog uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→21 sql.connector.catalog uses sql.catalyst.catalogpart of a loop✕
Connections
9 pairs of types, where a type in sql.connector.catalog uses a type in sql.catalyst.catalog.
sql.connector.catalog uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.catalyst.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→22 sql.connector.catalog uses sql.catalyst.analysispart of a loop✕
Connections
3 pairs of types, where a type in sql.connector.catalog uses a type in sql.catalyst.analysis.
sql.connector.catalog uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→28 sql.connector.catalog uses sql.catalogpart of a loop✕
Connections
4 pairs of types, where a type in sql.connector.catalog uses a type in sql.catalog.
sql.connector.catalog uses sql.catalog, so a change to sql.catalog can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→29 sql.connector.catalog uses sqlpart of a loop✕
Connections
2 pairs of types, where a type in sql.connector.catalog uses a type in sql.
sql.connector.catalog uses sql.catalyst, so a change to sql.catalyst can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→33 sql.connector.catalog uses sql.catalyst.analysis.resolverpart of a loop✕
Connections
1 pair of types, where a type in sql.connector.catalog uses a type in sql.catalyst.analysis.resolver.
sql.connector.catalog uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→34 sql.connector.catalog uses sql.typespart of a loop✕
Connections
18 pairs of types, where a type in sql.connector.catalog uses a type in sql.types.
sql.connector.catalog uses sql.types, so a change to sql.types can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→36 sql.connector.catalog uses sql.internalpart of a loop✕
Connections
2 pairs of types, where a type in sql.connector.catalog uses a type in sql.internal.
sql.connector.catalog uses sql.internal, so a change to sql.internal can break sql.connector.catalog, while a change to sql.connector.catalog can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
5→39 sql.connector.catalog uses org.apache.sparkpart of a loop✕
Connections
1 pair of types, where a type in sql.connector.catalog uses a type in org.apache.spark.
sql.connector.catalog uses org.apache.spark, so a change to org.apache.spark can break sql.connector.catalog, while a change to sql.connector.catalog can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
35 pairs of types, where a type in sql.execution.streaming.state uses a type in internal. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.streaming.state uses internal, so a change to internal can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break internal.
Position
Below the diagonal: sql.execution.streaming.state uses a module listed above it, which is the healthy direction.
sql.execution.streaming.state uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: sql.execution.streaming.state uses a module listed above it, which is the healthy direction.
6→10 sql.execution.streaming.state uses sql.execution.streaming.state.StateStoreEncodingpart of a loop✕
Connections
34 pairs of types, where a type in sql.execution.streaming.state uses a type in sql.execution.streaming.state.StateStoreEncoding. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.streaming.state uses sql.execution.streaming.state.StateStoreEncoding, so a change to sql.execution.streaming.state.StateStoreEncoding can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break sql.execution.streaming.state.StateStoreEncoding.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
6→23 sql.execution.streaming.state uses sql.catalyst.expressionspart of a loop✕
Connections
7 pairs of types, where a type in sql.execution.streaming.state uses a type in sql.catalyst.expressions.
sql.execution.streaming.state uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
6→26 sql.execution.streaming.state uses utilpart of a loop✕
Connections
5 pairs of types, where a type in sql.execution.streaming.state uses a type in util.
sql.execution.streaming.state uses util, so a change to util can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break util.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
6→29 sql.execution.streaming.state uses sqlpart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.streaming.state uses a type in sql.
sql.execution.streaming.state uses sql, so a change to sql can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
6→30 sql.execution.streaming.state uses sql.catalystpart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.streaming.state uses a type in sql.catalyst.
sql.execution.streaming.state uses sql.catalyst, so a change to sql.catalyst can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
6→34 sql.execution.streaming.state uses sql.typespart of a loop✕
Connections
27 pairs of types, where a type in sql.execution.streaming.state uses a type in sql.types. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.streaming.state uses sql.types, so a change to sql.types can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
6→36 sql.execution.streaming.state uses sql.internalpart of a loop✕
Connections
4 pairs of types, where a type in sql.execution.streaming.state uses a type in sql.internal.
sql.execution.streaming.state uses sql.internal, so a change to sql.internal can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
6→38 sql.execution.streaming.state uses rddpart of a loop✕
Connections
1 pair of types, where a type in sql.execution.streaming.state uses a type in rdd.
sql.execution.streaming.state uses rdd, so a change to rdd can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break rdd.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
6→39 sql.execution.streaming.state uses org.apache.sparkpart of a loop✕
Connections
66 pairs of types, where a type in sql.execution.streaming.state uses a type in org.apache.spark. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.streaming.state uses org.apache.spark, so a change to org.apache.spark can break sql.execution.streaming.state, while a change to sql.execution.streaming.state can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.pipelines.graph uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.catalyst.trees.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→20 sql.pipelines.graph uses sql.catalyst.plans.logicalpart of a loop✕
Connections
9 pairs of types, where a type in sql.pipelines.graph uses a type in sql.catalyst.plans.logical.
sql.pipelines.graph uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→21 sql.pipelines.graph uses sql.catalyst.catalogpart of a loop✕
Connections
1 pair of types, where a type in sql.pipelines.graph uses a type in sql.catalyst.catalog.
sql.pipelines.graph uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.catalyst.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→22 sql.pipelines.graph uses sql.catalyst.analysispart of a loop✕
Connections
1 pair of types, where a type in sql.pipelines.graph uses a type in sql.catalyst.analysis.
sql.pipelines.graph uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→23 sql.pipelines.graph uses sql.catalyst.expressionspart of a loop✕
Connections
1 pair of types, where a type in sql.pipelines.graph uses a type in sql.catalyst.expressions.
sql.pipelines.graph uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→28 sql.pipelines.graph uses sql.catalogpart of a loop✕
Connections
13 pairs of types, where a type in sql.pipelines.graph uses a type in sql.catalog.
sql.pipelines.graph uses sql.catalog, so a change to sql.catalog can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.pipelines.graph uses sql, so a change to sql can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→30 sql.pipelines.graph uses sql.catalystpart of a loop✕
Connections
46 pairs of types, where a type in sql.pipelines.graph uses a type in sql.catalyst. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.pipelines.graph uses sql.catalyst, so a change to sql.catalyst can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→33 sql.pipelines.graph uses sql.catalyst.analysis.resolverpart of a loop✕
Connections
3 pairs of types, where a type in sql.pipelines.graph uses a type in sql.catalyst.analysis.resolver.
sql.pipelines.graph uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→34 sql.pipelines.graph uses sql.typespart of a loop✕
Connections
16 pairs of types, where a type in sql.pipelines.graph uses a type in sql.types.
sql.pipelines.graph uses sql.types, so a change to sql.types can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→35 sql.pipelines.graph uses ml.attributepart of a loop✕
Connections
1 pair of types, where a type in sql.pipelines.graph uses a type in ml.attribute.
sql.pipelines.graph uses ml.attribute, so a change to ml.attribute can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→36 sql.pipelines.graph uses sql.internalpart of a loop✕
Connections
1 pair of types, where a type in sql.pipelines.graph uses a type in sql.internal.
sql.pipelines.graph uses sql.internal, so a change to sql.internal can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
7→39 sql.pipelines.graph uses org.apache.sparkpart of a loop✕
Connections
1 pair of types, where a type in sql.pipelines.graph uses a type in org.apache.spark.
sql.pipelines.graph uses org.apache.spark, so a change to org.apache.spark can break sql.pipelines.graph, while a change to sql.pipelines.graph can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
77 pairs of types, where a type in ml.feature uses a type in ml.util. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
84 pairs of types, where a type in ml.feature uses a type in sql.types. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.trees uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.catalyst.trees, while a change to sql.catalyst.trees can't break sql.pipelines.graph.
Position
Below the diagonal: sql.catalyst.trees uses a module listed above it, which is the healthy direction.
9→37 sql.catalyst.trees uses sql.internal.WindowFramepart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.trees uses a type in sql.internal.WindowFrame.
sql.catalyst.trees uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.catalyst.trees, while a change to sql.catalyst.trees can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.execution.streaming.state.StateStoreEncoding uses sql.execution.streaming.state, so a change to sql.execution.streaming.state can break sql.execution.streaming.state.StateStoreEncoding, while a change to sql.execution.streaming.state.StateStoreEncoding can't break sql.execution.streaming.state.
Position
Below the diagonal: sql.execution.streaming.state.StateStoreEncoding uses a module listed above it, which is the healthy direction.
streaming.scheduler uses scheduler, so a change to scheduler can break streaming.scheduler, while a change to streaming.scheduler can't break scheduler.
Position
Below the diagonal: streaming.scheduler uses a module listed above it, which is the healthy direction.
11→25 streaming.scheduler uses streamingpart of a loop✕
Connections
16 pairs of types, where a type in streaming.scheduler uses a type in streaming.
streaming.scheduler uses streaming, so a change to streaming can break streaming.scheduler, while a change to streaming.scheduler can't break streaming.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
11→26 streaming.scheduler uses utilpart of a loop✕
Connections
5 pairs of types, where a type in streaming.scheduler uses a type in util.
streaming.scheduler uses org.apache.spark, so a change to org.apache.spark can break streaming.scheduler, while a change to streaming.scheduler can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.catalyst.expressions.codegen uses internal, so a change to internal can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break internal.
Position
Below the diagonal: sql.catalyst.expressions.codegen uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions.codegen uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break sql.pipelines.graph.
Position
Below the diagonal: sql.catalyst.expressions.codegen uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions.codegen uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break sql.catalyst.trees.
Position
Below the diagonal: sql.catalyst.expressions.codegen uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions.codegen uses sql.execution.streaming.state.StateStoreEncoding, so a change to sql.execution.streaming.state.StateStoreEncoding can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break sql.execution.streaming.state.StateStoreEncoding.
Position
Below the diagonal: sql.catalyst.expressions.codegen uses a module listed above it, which is the healthy direction.
12→23 sql.catalyst.expressions.codegen uses sql.catalyst.expressionspart of a loop✕
Connections
24 pairs of types, where a type in sql.catalyst.expressions.codegen uses a type in sql.catalyst.expressions.
sql.catalyst.expressions.codegen uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
12→30 sql.catalyst.expressions.codegen uses sql.catalystpart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.expressions.codegen uses a type in sql.catalyst.
sql.catalyst.expressions.codegen uses sql.catalyst, so a change to sql.catalyst can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
12→34 sql.catalyst.expressions.codegen uses sql.typespart of a loop✕
Connections
9 pairs of types, where a type in sql.catalyst.expressions.codegen uses a type in sql.types.
sql.catalyst.expressions.codegen uses sql.types, so a change to sql.types can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
12→35 sql.catalyst.expressions.codegen uses ml.attributepart of a loop✕
Connections
8 pairs of types, where a type in sql.catalyst.expressions.codegen uses a type in ml.attribute.
sql.catalyst.expressions.codegen uses ml.attribute, so a change to ml.attribute can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
12→36 sql.catalyst.expressions.codegen uses sql.internalpart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.expressions.codegen uses a type in sql.internal.
sql.catalyst.expressions.codegen uses sql.internal, so a change to sql.internal can break sql.catalyst.expressions.codegen, while a change to sql.catalyst.expressions.codegen can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.errors uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.errors, while a change to sql.errors can't break sql.pipelines.graph.
Position
Below the diagonal: sql.errors uses a module listed above it, which is the healthy direction.
sql.errors uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.errors, while a change to sql.errors can't break sql.catalyst.trees.
Position
Below the diagonal: sql.errors uses a module listed above it, which is the healthy direction.
13→18 sql.errors uses sql.catalyst.utilpart of a loop✕
Connections
2 pairs of types, where a type in sql.errors uses a type in sql.catalyst.util.
sql.errors uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.errors, while a change to sql.errors can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
13→21 sql.errors uses sql.catalyst.catalogpart of a loop✕
Connections
3 pairs of types, where a type in sql.errors uses a type in sql.catalyst.catalog.
sql.errors uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.errors, while a change to sql.errors can't break sql.catalyst.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
13→22 sql.errors uses sql.catalyst.analysispart of a loop✕
Connections
4 pairs of types, where a type in sql.errors uses a type in sql.catalyst.analysis.
sql.errors uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.errors, while a change to sql.errors can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
13→23 sql.errors uses sql.catalyst.expressionspart of a loop✕
Connections
13 pairs of types, where a type in sql.errors uses a type in sql.catalyst.expressions.
sql.errors uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.errors, while a change to sql.errors can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.errors uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.errors, while a change to sql.errors can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
13→39 sql.errors uses org.apache.sparkpart of a loop✕
Connections
29 pairs of types, where a type in sql.errors uses a type in org.apache.spark. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.streaming.runtime uses internal, so a change to internal can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break internal.
Position
Below the diagonal: sql.execution.streaming.runtime uses a module listed above it, which is the healthy direction.
sql.execution.streaming.runtime uses scheduler, so a change to scheduler can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break scheduler.
Position
Below the diagonal: sql.execution.streaming.runtime uses a module listed above it, which is the healthy direction.
sql.execution.streaming.runtime uses sql.execution.streaming.state, so a change to sql.execution.streaming.state can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.execution.streaming.state.
Position
Below the diagonal: sql.execution.streaming.runtime uses a module listed above it, which is the healthy direction.
sql.execution.streaming.runtime uses ml.feature, so a change to ml.feature can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break ml.feature.
Position
Below the diagonal: sql.execution.streaming.runtime uses a module listed above it, which is the healthy direction.
sql.execution.streaming.runtime uses sql.execution.streaming.state.StateStoreEncoding, so a change to sql.execution.streaming.state.StateStoreEncoding can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.execution.streaming.state.StateStoreEncoding.
Position
Below the diagonal: sql.execution.streaming.runtime uses a module listed above it, which is the healthy direction.
14→15 sql.execution.streaming.runtime uses sql.execution.datasourcespart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.execution.datasources.
sql.execution.streaming.runtime uses sql.execution.datasources, so a change to sql.execution.datasources can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.execution.datasources.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→16 sql.execution.streaming.runtime uses sql.executionpart of a loop✕
Connections
14 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.execution.
sql.execution.streaming.runtime uses sql.execution, so a change to sql.execution can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.execution.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→17 sql.execution.streaming.runtime uses sql.execution.datasources.v2part of a loop✕
Connections
1 pair of types, where a type in sql.execution.streaming.runtime uses a type in sql.execution.datasources.v2.
sql.execution.streaming.runtime uses sql.execution.datasources.v2, so a change to sql.execution.datasources.v2 can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.execution.datasources.v2.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→20 sql.execution.streaming.runtime uses sql.catalyst.plans.logicalpart of a loop✕
Connections
16 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.catalyst.plans.logical.
sql.execution.streaming.runtime uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→21 sql.execution.streaming.runtime uses sql.catalyst.catalogpart of a loop✕
Connections
1 pair of types, where a type in sql.execution.streaming.runtime uses a type in sql.catalyst.catalog.
sql.execution.streaming.runtime uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.catalyst.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→22 sql.execution.streaming.runtime uses sql.catalyst.analysispart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.catalyst.analysis.
sql.execution.streaming.runtime uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→26 sql.execution.streaming.runtime uses utilpart of a loop✕
Connections
6 pairs of types, where a type in sql.execution.streaming.runtime uses a type in util.
sql.execution.streaming.runtime uses util, so a change to util can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break util.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→28 sql.execution.streaming.runtime uses sql.catalogpart of a loop✕
Connections
4 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.catalog.
sql.execution.streaming.runtime uses sql.catalog, so a change to sql.catalog can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→29 sql.execution.streaming.runtime uses sqlpart of a loop✕
Connections
12 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.
sql.execution.streaming.runtime uses sql, so a change to sql can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→30 sql.execution.streaming.runtime uses sql.catalystpart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.catalyst.
sql.execution.streaming.runtime uses sql.catalyst, so a change to sql.catalyst can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→32 sql.execution.streaming.runtime uses ml.treepart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.streaming.runtime uses a type in ml.tree.
sql.execution.streaming.runtime uses ml.tree, so a change to ml.tree can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break ml.tree.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→34 sql.execution.streaming.runtime uses sql.typespart of a loop✕
Connections
5 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.types.
sql.execution.streaming.runtime uses sql.types, so a change to sql.types can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→35 sql.execution.streaming.runtime uses ml.attributepart of a loop✕
Connections
3 pairs of types, where a type in sql.execution.streaming.runtime uses a type in ml.attribute.
sql.execution.streaming.runtime uses ml.attribute, so a change to ml.attribute can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→36 sql.execution.streaming.runtime uses sql.internalpart of a loop✕
Connections
1 pair of types, where a type in sql.execution.streaming.runtime uses a type in sql.internal.
sql.execution.streaming.runtime uses sql.internal, so a change to sql.internal can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→37 sql.execution.streaming.runtime uses sql.internal.WindowFramepart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.streaming.runtime uses a type in sql.internal.WindowFrame.
sql.execution.streaming.runtime uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→38 sql.execution.streaming.runtime uses rddpart of a loop✕
Connections
1 pair of types, where a type in sql.execution.streaming.runtime uses a type in rdd.
sql.execution.streaming.runtime uses rdd, so a change to rdd can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break rdd.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
14→39 sql.execution.streaming.runtime uses org.apache.sparkpart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.streaming.runtime uses a type in org.apache.spark.
sql.execution.streaming.runtime uses org.apache.spark, so a change to org.apache.spark can break sql.execution.streaming.runtime, while a change to sql.execution.streaming.runtime can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.execution.datasources uses internal, so a change to internal can break sql.execution.datasources, while a change to sql.execution.datasources can't break internal.
Position
Below the diagonal: sql.execution.datasources uses a module listed above it, which is the healthy direction.
sql.execution.datasources uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: sql.execution.datasources uses a module listed above it, which is the healthy direction.
sql.execution.datasources uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.trees.
Position
Below the diagonal: sql.execution.datasources uses a module listed above it, which is the healthy direction.
sql.execution.datasources uses sql.execution.streaming.state.StateStoreEncoding, so a change to sql.execution.streaming.state.StateStoreEncoding can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.execution.streaming.state.StateStoreEncoding.
Position
Below the diagonal: sql.execution.datasources uses a module listed above it, which is the healthy direction.
sql.execution.datasources uses streaming.scheduler, so a change to streaming.scheduler can break sql.execution.datasources, while a change to sql.execution.datasources can't break streaming.scheduler.
Position
Below the diagonal: sql.execution.datasources uses a module listed above it, which is the healthy direction.
sql.execution.datasources uses sql.execution.streaming.runtime, so a change to sql.execution.streaming.runtime can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.execution.streaming.runtime.
Position
Below the diagonal: sql.execution.datasources uses a module listed above it, which is the healthy direction.
15→16 sql.execution.datasources uses sql.executionpart of a loop✕
Connections
10 pairs of types, where a type in sql.execution.datasources uses a type in sql.execution.
sql.execution.datasources uses sql.execution, so a change to sql.execution can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.execution.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→18 sql.execution.datasources uses sql.catalyst.utilpart of a loop✕
Connections
6 pairs of types, where a type in sql.execution.datasources uses a type in sql.catalyst.util.
sql.execution.datasources uses sql.catalyst.util, so a change to sql.catalyst.util can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.util.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→19 sql.execution.datasources uses sql.catalyst.expressions.aggregatepart of a loop✕
Connections
1 pair of types, where a type in sql.execution.datasources uses a type in sql.catalyst.expressions.aggregate.
sql.execution.datasources uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.expressions.aggregate.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→20 sql.execution.datasources uses sql.catalyst.plans.logicalpart of a loop✕
Connections
49 pairs of types, where a type in sql.execution.datasources uses a type in sql.catalyst.plans.logical. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.datasources uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→21 sql.execution.datasources uses sql.catalyst.catalogpart of a loop✕
Connections
23 pairs of types, where a type in sql.execution.datasources uses a type in sql.catalyst.catalog.
sql.execution.datasources uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→22 sql.execution.datasources uses sql.catalyst.analysispart of a loop✕
Connections
8 pairs of types, where a type in sql.execution.datasources uses a type in sql.catalyst.analysis.
sql.execution.datasources uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→23 sql.execution.datasources uses sql.catalyst.expressionspart of a loop✕
Connections
43 pairs of types, where a type in sql.execution.datasources uses a type in sql.catalyst.expressions. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.datasources uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→26 sql.execution.datasources uses utilpart of a loop✕
Connections
2 pairs of types, where a type in sql.execution.datasources uses a type in util.
sql.execution.datasources uses util, so a change to util can break sql.execution.datasources, while a change to sql.execution.datasources can't break util.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→29 sql.execution.datasources uses sqlpart of a loop✕
Connections
20 pairs of types, where a type in sql.execution.datasources uses a type in sql.
sql.execution.datasources uses sql, so a change to sql can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→30 sql.execution.datasources uses sql.catalystpart of a loop✕
Connections
24 pairs of types, where a type in sql.execution.datasources uses a type in sql.catalyst.
sql.execution.datasources uses sql.catalyst, so a change to sql.catalyst can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→32 sql.execution.datasources uses ml.treepart of a loop✕
Connections
3 pairs of types, where a type in sql.execution.datasources uses a type in ml.tree.
sql.execution.datasources uses ml.tree, so a change to ml.tree can break sql.execution.datasources, while a change to sql.execution.datasources can't break ml.tree.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→33 sql.execution.datasources uses sql.catalyst.analysis.resolverpart of a loop✕
Connections
4 pairs of types, where a type in sql.execution.datasources uses a type in sql.catalyst.analysis.resolver.
sql.execution.datasources uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→34 sql.execution.datasources uses sql.typespart of a loop✕
Connections
31 pairs of types, where a type in sql.execution.datasources uses a type in sql.types. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.datasources uses sql.types, so a change to sql.types can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→35 sql.execution.datasources uses ml.attributepart of a loop✕
Connections
14 pairs of types, where a type in sql.execution.datasources uses a type in ml.attribute.
sql.execution.datasources uses ml.attribute, so a change to ml.attribute can break sql.execution.datasources, while a change to sql.execution.datasources can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→36 sql.execution.datasources uses sql.internalpart of a loop✕
Connections
11 pairs of types, where a type in sql.execution.datasources uses a type in sql.internal.
sql.execution.datasources uses sql.internal, so a change to sql.internal can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→37 sql.execution.datasources uses sql.internal.WindowFramepart of a loop✕
Connections
1 pair of types, where a type in sql.execution.datasources uses a type in sql.internal.WindowFrame.
sql.execution.datasources uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.execution.datasources, while a change to sql.execution.datasources can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→38 sql.execution.datasources uses rddpart of a loop✕
Connections
4 pairs of types, where a type in sql.execution.datasources uses a type in rdd.
sql.execution.datasources uses rdd, so a change to rdd can break sql.execution.datasources, while a change to sql.execution.datasources can't break rdd.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
15→39 sql.execution.datasources uses org.apache.sparkpart of a loop✕
Connections
5 pairs of types, where a type in sql.execution.datasources uses a type in org.apache.spark.
sql.execution.datasources uses org.apache.spark, so a change to org.apache.spark can break sql.execution.datasources, while a change to sql.execution.datasources can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.execution uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break sql.execution, while a change to sql.execution can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: sql.execution uses a module listed above it, which is the healthy direction.
sql.execution uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.execution, while a change to sql.execution can't break sql.pipelines.graph.
Position
Below the diagonal: sql.execution uses a module listed above it, which is the healthy direction.
sql.execution uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.execution, while a change to sql.execution can't break sql.catalyst.trees.
Position
Below the diagonal: sql.execution uses a module listed above it, which is the healthy direction.
sql.execution uses sql.execution.streaming.state.StateStoreEncoding, so a change to sql.execution.streaming.state.StateStoreEncoding can break sql.execution, while a change to sql.execution can't break sql.execution.streaming.state.StateStoreEncoding.
Position
Below the diagonal: sql.execution uses a module listed above it, which is the healthy direction.
36 pairs of types, where a type in sql.execution uses a type in sql.catalyst.expressions.codegen. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution uses sql.catalyst.expressions.codegen, so a change to sql.catalyst.expressions.codegen can break sql.execution, while a change to sql.execution can't break sql.catalyst.expressions.codegen.
Position
Below the diagonal: sql.execution uses a module listed above it, which is the healthy direction.
sql.execution uses sql.execution.streaming.runtime, so a change to sql.execution.streaming.runtime can break sql.execution, while a change to sql.execution can't break sql.execution.streaming.runtime.
Position
Below the diagonal: sql.execution uses a module listed above it, which is the healthy direction.
sql.execution uses sql.execution.datasources, so a change to sql.execution.datasources can break sql.execution, while a change to sql.execution can't break sql.execution.datasources.
Position
Below the diagonal: sql.execution uses a module listed above it, which is the healthy direction.
16→17 sql.execution uses sql.execution.datasources.v2part of a loop✕
Connections
2 pairs of types, where a type in sql.execution uses a type in sql.execution.datasources.v2.
sql.execution uses sql.execution.datasources.v2, so a change to sql.execution.datasources.v2 can break sql.execution, while a change to sql.execution can't break sql.execution.datasources.v2.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→19 sql.execution uses sql.catalyst.expressions.aggregatepart of a loop✕
Connections
2 pairs of types, where a type in sql.execution uses a type in sql.catalyst.expressions.aggregate.
sql.execution uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql.execution, while a change to sql.execution can't break sql.catalyst.expressions.aggregate.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→20 sql.execution uses sql.catalyst.plans.logicalpart of a loop✕
Connections
18 pairs of types, where a type in sql.execution uses a type in sql.catalyst.plans.logical.
sql.execution uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.execution, while a change to sql.execution can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→21 sql.execution uses sql.catalyst.catalogpart of a loop✕
Connections
3 pairs of types, where a type in sql.execution uses a type in sql.catalyst.catalog.
sql.execution uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.execution, while a change to sql.execution can't break sql.catalyst.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→22 sql.execution uses sql.catalyst.analysispart of a loop✕
Connections
1 pair of types, where a type in sql.execution uses a type in sql.catalyst.analysis.
sql.execution uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.execution, while a change to sql.execution can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→23 sql.execution uses sql.catalyst.expressionspart of a loop✕
Connections
75 pairs of types, where a type in sql.execution uses a type in sql.catalyst.expressions. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.execution, while a change to sql.execution can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.execution uses sql, so a change to sql can break sql.execution, while a change to sql.execution can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→30 sql.execution uses sql.catalystpart of a loop✕
Connections
83 pairs of types, where a type in sql.execution uses a type in sql.catalyst. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.execution, while a change to sql.execution can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.execution uses sql.types, so a change to sql.types can break sql.execution, while a change to sql.execution can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→35 sql.execution uses ml.attributepart of a loop✕
Connections
55 pairs of types, where a type in sql.execution uses a type in ml.attribute. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution uses ml.attribute, so a change to ml.attribute can break sql.execution, while a change to sql.execution can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→36 sql.execution uses sql.internalpart of a loop✕
Connections
27 pairs of types, where a type in sql.execution uses a type in sql.internal. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.execution, while a change to sql.execution can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.execution uses rdd, so a change to rdd can break sql.execution, while a change to sql.execution can't break rdd.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
16→39 sql.execution uses org.apache.sparkpart of a loop✕
Connections
26 pairs of types, where a type in sql.execution uses a type in org.apache.spark. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution uses org.apache.spark, so a change to org.apache.spark can break sql.execution, while a change to sql.execution can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.execution.datasources.v2 uses internal, so a change to internal can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break internal.
Position
Below the diagonal: sql.execution.datasources.v2 uses a module listed above it, which is the healthy direction.
sql.execution.datasources.v2 uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: sql.execution.datasources.v2 uses a module listed above it, which is the healthy direction.
sql.execution.datasources.v2 uses sql.connector.catalog, so a change to sql.connector.catalog can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.connector.catalog.
Position
Below the diagonal: sql.execution.datasources.v2 uses a module listed above it, which is the healthy direction.
sql.execution.datasources.v2 uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.trees.
Position
Below the diagonal: sql.execution.datasources.v2 uses a module listed above it, which is the healthy direction.
sql.execution.datasources.v2 uses streaming.scheduler, so a change to streaming.scheduler can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break streaming.scheduler.
Position
Below the diagonal: sql.execution.datasources.v2 uses a module listed above it, which is the healthy direction.
sql.execution.datasources.v2 uses sql.catalyst.expressions.codegen, so a change to sql.catalyst.expressions.codegen can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.expressions.codegen.
Position
Below the diagonal: sql.execution.datasources.v2 uses a module listed above it, which is the healthy direction.
sql.execution.datasources.v2 uses sql.execution.datasources, so a change to sql.execution.datasources can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.execution.datasources.
Position
Below the diagonal: sql.execution.datasources.v2 uses a module listed above it, which is the healthy direction.
32 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.execution. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.datasources.v2 uses sql.execution, so a change to sql.execution can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.execution.
Position
Below the diagonal: sql.execution.datasources.v2 uses a module listed above it, which is the healthy direction.
17→18 sql.execution.datasources.v2 uses sql.catalyst.utilpart of a loop✕
Connections
6 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalyst.util.
sql.execution.datasources.v2 uses sql.catalyst.util, so a change to sql.catalyst.util can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.util.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→19 sql.execution.datasources.v2 uses sql.catalyst.expressions.aggregatepart of a loop✕
Connections
1 pair of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalyst.expressions.aggregate.
sql.execution.datasources.v2 uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.expressions.aggregate.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→20 sql.execution.datasources.v2 uses sql.catalyst.plans.logicalpart of a loop✕
Connections
62 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalyst.plans.logical. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.datasources.v2 uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→21 sql.execution.datasources.v2 uses sql.catalyst.catalogpart of a loop✕
Connections
3 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalyst.catalog.
sql.execution.datasources.v2 uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→22 sql.execution.datasources.v2 uses sql.catalyst.analysispart of a loop✕
Connections
21 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalyst.analysis.
sql.execution.datasources.v2 uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→23 sql.execution.datasources.v2 uses sql.catalyst.expressionspart of a loop✕
Connections
52 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalyst.expressions. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.datasources.v2 uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→26 sql.execution.datasources.v2 uses utilpart of a loop✕
Connections
4 pairs of types, where a type in sql.execution.datasources.v2 uses a type in util.
sql.execution.datasources.v2 uses util, so a change to util can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break util.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→27 sql.execution.datasources.v2 uses storagepart of a loop✕
Connections
1 pair of types, where a type in sql.execution.datasources.v2 uses a type in storage.
sql.execution.datasources.v2 uses storage, so a change to storage can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break storage.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→28 sql.execution.datasources.v2 uses sql.catalogpart of a loop✕
Connections
21 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalog.
sql.execution.datasources.v2 uses sql.catalog, so a change to sql.catalog can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→29 sql.execution.datasources.v2 uses sqlpart of a loop✕
Connections
11 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.
sql.execution.datasources.v2 uses sql, so a change to sql can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→30 sql.execution.datasources.v2 uses sql.catalystpart of a loop✕
Connections
94 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalyst. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.datasources.v2 uses sql.catalyst, so a change to sql.catalyst can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→32 sql.execution.datasources.v2 uses ml.treepart of a loop✕
Connections
4 pairs of types, where a type in sql.execution.datasources.v2 uses a type in ml.tree.
sql.execution.datasources.v2 uses ml.tree, so a change to ml.tree can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break ml.tree.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→33 sql.execution.datasources.v2 uses sql.catalyst.analysis.resolverpart of a loop✕
Connections
1 pair of types, where a type in sql.execution.datasources.v2 uses a type in sql.catalyst.analysis.resolver.
sql.execution.datasources.v2 uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→34 sql.execution.datasources.v2 uses sql.typespart of a loop✕
Connections
18 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.types.
sql.execution.datasources.v2 uses sql.types, so a change to sql.types can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→35 sql.execution.datasources.v2 uses ml.attributepart of a loop✕
Connections
57 pairs of types, where a type in sql.execution.datasources.v2 uses a type in ml.attribute. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.datasources.v2 uses ml.attribute, so a change to ml.attribute can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→36 sql.execution.datasources.v2 uses sql.internalpart of a loop✕
Connections
12 pairs of types, where a type in sql.execution.datasources.v2 uses a type in sql.internal.
sql.execution.datasources.v2 uses sql.internal, so a change to sql.internal can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→38 sql.execution.datasources.v2 uses rddpart of a loop✕
Connections
13 pairs of types, where a type in sql.execution.datasources.v2 uses a type in rdd.
sql.execution.datasources.v2 uses rdd, so a change to rdd can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break rdd.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
17→39 sql.execution.datasources.v2 uses org.apache.sparkpart of a loop✕
Connections
5 pairs of types, where a type in sql.execution.datasources.v2 uses a type in org.apache.spark.
sql.execution.datasources.v2 uses org.apache.spark, so a change to org.apache.spark can break sql.execution.datasources.v2, while a change to sql.execution.datasources.v2 can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.catalyst.util uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: sql.catalyst.util uses a module listed above it, which is the healthy direction.
sql.catalyst.util uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.pipelines.graph.
Position
Below the diagonal: sql.catalyst.util uses a module listed above it, which is the healthy direction.
sql.catalyst.util uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.catalyst.trees.
Position
Below the diagonal: sql.catalyst.util uses a module listed above it, which is the healthy direction.
sql.catalyst.util uses sql.execution.streaming.state.StateStoreEncoding, so a change to sql.execution.streaming.state.StateStoreEncoding can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.execution.streaming.state.StateStoreEncoding.
Position
Below the diagonal: sql.catalyst.util uses a module listed above it, which is the healthy direction.
sql.catalyst.util uses sql.execution.datasources.v2, so a change to sql.execution.datasources.v2 can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.execution.datasources.v2.
Position
Below the diagonal: sql.catalyst.util uses a module listed above it, which is the healthy direction.
18→19 sql.catalyst.util uses sql.catalyst.expressions.aggregatepart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.util uses a type in sql.catalyst.expressions.aggregate.
sql.catalyst.util uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.catalyst.expressions.aggregate.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
18→20 sql.catalyst.util uses sql.catalyst.plans.logicalpart of a loop✕
Connections
5 pairs of types, where a type in sql.catalyst.util uses a type in sql.catalyst.plans.logical.
sql.catalyst.util uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
18→22 sql.catalyst.util uses sql.catalyst.analysispart of a loop✕
Connections
6 pairs of types, where a type in sql.catalyst.util uses a type in sql.catalyst.analysis.
sql.catalyst.util uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
18→23 sql.catalyst.util uses sql.catalyst.expressionspart of a loop✕
Connections
18 pairs of types, where a type in sql.catalyst.util uses a type in sql.catalyst.expressions.
sql.catalyst.util uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
18→25 sql.catalyst.util uses streamingpart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.util uses a type in streaming.
sql.catalyst.util uses sql.catalog, so a change to sql.catalog can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.catalyst.util uses sql.catalyst, so a change to sql.catalyst can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
18→34 sql.catalyst.util uses sql.typespart of a loop✕
Connections
35 pairs of types, where a type in sql.catalyst.util uses a type in sql.types. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.util uses ml.attribute, so a change to ml.attribute can break sql.catalyst.util, while a change to sql.catalyst.util can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
18→36 sql.catalyst.util uses sql.internalpart of a loop✕
Connections
3 pairs of types, where a type in sql.catalyst.util uses a type in sql.internal.
sql.catalyst.util uses sql.internal, so a change to sql.internal can break sql.catalyst.util, while a change to sql.catalyst.util can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
18→39 sql.catalyst.util uses org.apache.sparkpart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.util uses a type in org.apache.spark.
sql.catalyst.util uses org.apache.spark, so a change to org.apache.spark can break sql.catalyst.util, while a change to sql.catalyst.util can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.catalyst.expressions.aggregate uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.pipelines.graph.
Position
Below the diagonal: sql.catalyst.expressions.aggregate uses a module listed above it, which is the healthy direction.
63 pairs of types, where a type in sql.catalyst.expressions.aggregate uses a type in sql.catalyst.trees. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions.aggregate uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.catalyst.trees.
Position
Below the diagonal: sql.catalyst.expressions.aggregate uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions.aggregate uses sql.catalyst.expressions.codegen, so a change to sql.catalyst.expressions.codegen can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.catalyst.expressions.codegen.
Position
Below the diagonal: sql.catalyst.expressions.aggregate uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions.aggregate uses sql.errors, so a change to sql.errors can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.errors.
Position
Below the diagonal: sql.catalyst.expressions.aggregate uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions.aggregate uses sql.catalyst.util, so a change to sql.catalyst.util can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.catalyst.util.
Position
Below the diagonal: sql.catalyst.expressions.aggregate uses a module listed above it, which is the healthy direction.
19→20 sql.catalyst.expressions.aggregate uses sql.catalyst.plans.logicalpart of a loop✕
Connections
8 pairs of types, where a type in sql.catalyst.expressions.aggregate uses a type in sql.catalyst.plans.logical.
sql.catalyst.expressions.aggregate uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.catalyst.plans.logical.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
19→22 sql.catalyst.expressions.aggregate uses sql.catalyst.analysispart of a loop✕
Connections
42 pairs of types, where a type in sql.catalyst.expressions.aggregate uses a type in sql.catalyst.analysis. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions.aggregate uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
19→23 sql.catalyst.expressions.aggregate uses sql.catalyst.expressionspart of a loop✕
Connections
200 pairs of types, where a type in sql.catalyst.expressions.aggregate uses a type in sql.catalyst.expressions. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions.aggregate uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
19→26 sql.catalyst.expressions.aggregate uses utilpart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.expressions.aggregate uses a type in util.
sql.catalyst.expressions.aggregate uses util, so a change to util can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break util.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
19→30 sql.catalyst.expressions.aggregate uses sql.catalystpart of a loop✕
Connections
37 pairs of types, where a type in sql.catalyst.expressions.aggregate uses a type in sql.catalyst. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions.aggregate uses sql.catalyst, so a change to sql.catalyst can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
19→34 sql.catalyst.expressions.aggregate uses sql.typespart of a loop✕
Connections
114 pairs of types, where a type in sql.catalyst.expressions.aggregate uses a type in sql.types. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions.aggregate uses sql.types, so a change to sql.types can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
19→36 sql.catalyst.expressions.aggregate uses sql.internalpart of a loop✕
Connections
17 pairs of types, where a type in sql.catalyst.expressions.aggregate uses a type in sql.internal.
sql.catalyst.expressions.aggregate uses sql.internal, so a change to sql.internal can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
19→37 sql.catalyst.expressions.aggregate uses sql.internal.WindowFramepart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.expressions.aggregate uses a type in sql.internal.WindowFrame.
sql.catalyst.expressions.aggregate uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
19→39 sql.catalyst.expressions.aggregate uses org.apache.sparkpart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.expressions.aggregate uses a type in org.apache.spark.
sql.catalyst.expressions.aggregate uses org.apache.spark, so a change to org.apache.spark can break sql.catalyst.expressions.aggregate, while a change to sql.catalyst.expressions.aggregate can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.catalyst.plans.logical uses internal, so a change to internal can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break internal.
Position
Below the diagonal: sql.catalyst.plans.logical uses a module listed above it, which is the healthy direction.
63 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.catalyst.trees. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.plans.logical uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.catalyst.trees.
Position
Below the diagonal: sql.catalyst.plans.logical uses a module listed above it, which is the healthy direction.
sql.catalyst.plans.logical uses sql.execution, so a change to sql.execution can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.execution.
Position
Below the diagonal: sql.catalyst.plans.logical uses a module listed above it, which is the healthy direction.
sql.catalyst.plans.logical uses sql.catalyst.util, so a change to sql.catalyst.util can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.catalyst.util.
Position
Below the diagonal: sql.catalyst.plans.logical uses a module listed above it, which is the healthy direction.
sql.catalyst.plans.logical uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.catalyst.expressions.aggregate.
Position
Below the diagonal: sql.catalyst.plans.logical uses a module listed above it, which is the healthy direction.
20→21 sql.catalyst.plans.logical uses sql.catalyst.catalogpart of a loop✕
Connections
7 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.catalyst.catalog.
sql.catalyst.plans.logical uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.catalyst.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→22 sql.catalyst.plans.logical uses sql.catalyst.analysispart of a loop✕
Connections
43 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.catalyst.analysis. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.plans.logical uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→23 sql.catalyst.plans.logical uses sql.catalyst.expressionspart of a loop✕
Connections
161 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.catalyst.expressions. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.plans.logical uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→29 sql.catalyst.plans.logical uses sqlpart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.
sql.catalyst.plans.logical uses sql, so a change to sql can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→30 sql.catalyst.plans.logical uses sql.catalystpart of a loop✕
Connections
5 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.catalyst.
sql.catalyst.plans.logical uses sql.catalyst, so a change to sql.catalyst can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→32 sql.catalyst.plans.logical uses ml.treepart of a loop✕
Connections
9 pairs of types, where a type in sql.catalyst.plans.logical uses a type in ml.tree.
sql.catalyst.plans.logical uses ml.tree, so a change to ml.tree can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break ml.tree.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→33 sql.catalyst.plans.logical uses sql.catalyst.analysis.resolverpart of a loop✕
Connections
5 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.catalyst.analysis.resolver.
sql.catalyst.plans.logical uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→34 sql.catalyst.plans.logical uses sql.typespart of a loop✕
Connections
27 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.types. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.plans.logical uses sql.types, so a change to sql.types can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→35 sql.catalyst.plans.logical uses ml.attributepart of a loop✕
Connections
130 pairs of types, where a type in sql.catalyst.plans.logical uses a type in ml.attribute. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.plans.logical uses ml.attribute, so a change to ml.attribute can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→36 sql.catalyst.plans.logical uses sql.internalpart of a loop✕
Connections
8 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.internal.
sql.catalyst.plans.logical uses sql.internal, so a change to sql.internal can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
20→37 sql.catalyst.plans.logical uses sql.internal.WindowFramepart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.plans.logical uses a type in sql.internal.WindowFrame.
sql.catalyst.plans.logical uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.catalyst.plans.logical, while a change to sql.catalyst.plans.logical can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.catalyst.catalog uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: sql.catalyst.catalog uses a module listed above it, which is the healthy direction.
sql.catalyst.catalog uses scheduler, so a change to scheduler can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break scheduler.
Position
Below the diagonal: sql.catalyst.catalog uses a module listed above it, which is the healthy direction.
sql.catalyst.catalog uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.catalyst.trees.
Position
Below the diagonal: sql.catalyst.catalog uses a module listed above it, which is the healthy direction.
sql.catalyst.catalog uses sql.errors, so a change to sql.errors can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.errors.
Position
Below the diagonal: sql.catalyst.catalog uses a module listed above it, which is the healthy direction.
sql.catalyst.catalog uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.catalyst.plans.logical.
Position
Below the diagonal: sql.catalyst.catalog uses a module listed above it, which is the healthy direction.
21→22 sql.catalyst.catalog uses sql.catalyst.analysispart of a loop✕
Connections
7 pairs of types, where a type in sql.catalyst.catalog uses a type in sql.catalyst.analysis.
sql.catalyst.catalog uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.catalyst.analysis.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
21→23 sql.catalyst.catalog uses sql.catalyst.expressionspart of a loop✕
Connections
9 pairs of types, where a type in sql.catalyst.catalog uses a type in sql.catalyst.expressions.
sql.catalyst.catalog uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
21→26 sql.catalyst.catalog uses utilpart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.catalog uses a type in util.
sql.catalyst.catalog uses sql.catalyst, so a change to sql.catalyst can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
21→32 sql.catalyst.catalog uses ml.treepart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.catalog uses a type in ml.tree.
sql.catalyst.catalog uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
21→34 sql.catalyst.catalog uses sql.typespart of a loop✕
Connections
9 pairs of types, where a type in sql.catalyst.catalog uses a type in sql.types.
sql.catalyst.catalog uses sql.types, so a change to sql.types can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
21→35 sql.catalyst.catalog uses ml.attributepart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.catalog uses a type in ml.attribute.
sql.catalyst.catalog uses ml.attribute, so a change to ml.attribute can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
21→36 sql.catalyst.catalog uses sql.internalpart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.catalog uses a type in sql.internal.
sql.catalyst.catalog uses sql.internal, so a change to sql.internal can break sql.catalyst.catalog, while a change to sql.catalyst.catalog can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.catalyst.analysis uses internal, so a change to internal can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break internal.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis uses sql.connector.catalog, so a change to sql.connector.catalog can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.connector.catalog.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
40 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.catalyst.trees. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalyst.trees.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis uses sql.catalyst.expressions.codegen, so a change to sql.catalyst.expressions.codegen can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalyst.expressions.codegen.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis uses sql.errors, so a change to sql.errors can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.errors.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis uses sql.execution.datasources, so a change to sql.execution.datasources can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.execution.datasources.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis uses sql.execution.datasources.v2, so a change to sql.execution.datasources.v2 can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.execution.datasources.v2.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis uses sql.catalyst.util, so a change to sql.catalyst.util can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalyst.util.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
192 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.catalyst.plans.logical. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalyst.plans.logical.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalyst.catalog.
Position
Below the diagonal: sql.catalyst.analysis uses a module listed above it, which is the healthy direction.
22→23 sql.catalyst.analysis uses sql.catalyst.expressionspart of a loop✕
Connections
229 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.catalyst.expressions. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalyst.expressions.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→28 sql.catalyst.analysis uses sql.catalogpart of a loop✕
Connections
7 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.catalog.
sql.catalyst.analysis uses sql.catalog, so a change to sql.catalog can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalog.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→29 sql.catalyst.analysis uses sqlpart of a loop✕
Connections
22 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.
sql.catalyst.analysis uses sql, so a change to sql can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→30 sql.catalyst.analysis uses sql.catalystpart of a loop✕
Connections
40 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.catalyst. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis uses sql.catalyst, so a change to sql.catalyst can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→32 sql.catalyst.analysis uses ml.treepart of a loop✕
Connections
5 pairs of types, where a type in sql.catalyst.analysis uses a type in ml.tree.
sql.catalyst.analysis uses ml.tree, so a change to ml.tree can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break ml.tree.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→33 sql.catalyst.analysis uses sql.catalyst.analysis.resolverpart of a loop✕
Connections
9 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.catalyst.analysis.resolver.
sql.catalyst.analysis uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→34 sql.catalyst.analysis uses sql.typespart of a loop✕
Connections
83 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.types. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis uses sql.types, so a change to sql.types can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→35 sql.catalyst.analysis uses ml.attributepart of a loop✕
Connections
44 pairs of types, where a type in sql.catalyst.analysis uses a type in ml.attribute. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis uses ml.attribute, so a change to ml.attribute can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→36 sql.catalyst.analysis uses sql.internalpart of a loop✕
Connections
20 pairs of types, where a type in sql.catalyst.analysis uses a type in sql.internal.
sql.catalyst.analysis uses sql.internal, so a change to sql.internal can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→37 sql.catalyst.analysis uses sql.internal.WindowFramepart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.analysis uses a type in sql.internal.WindowFrame.
sql.catalyst.analysis uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
22→39 sql.catalyst.analysis uses org.apache.sparkpart of a loop✕
Connections
1 pair of types, where a type in sql.catalyst.analysis uses a type in org.apache.spark.
sql.catalyst.analysis uses org.apache.spark, so a change to org.apache.spark can break sql.catalyst.analysis, while a change to sql.catalyst.analysis can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.catalyst.expressions uses internal, so a change to internal can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break internal.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.pipelines.graph.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
138 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.catalyst.trees. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.trees.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions uses sql.execution.streaming.state.StateStoreEncoding, so a change to sql.execution.streaming.state.StateStoreEncoding can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.execution.streaming.state.StateStoreEncoding.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
585 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.catalyst.expressions.codegen. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.catalyst.expressions.codegen, so a change to sql.catalyst.expressions.codegen can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.expressions.codegen.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
48 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.errors. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.errors, so a change to sql.errors can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.errors.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
37 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.catalyst.util. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.catalyst.util, so a change to sql.catalyst.util can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.util.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.expressions.aggregate.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
49 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.catalyst.plans.logical. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.plans.logical.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
sql.catalyst.expressions uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.catalog.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
170 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.catalyst.analysis. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.analysis.
Position
Below the diagonal: sql.catalyst.expressions uses a module listed above it, which is the healthy direction.
23→29 sql.catalyst.expressions uses sqlpart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.
sql.catalyst.expressions uses sql, so a change to sql can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
23→30 sql.catalyst.expressions uses sql.catalystpart of a loop✕
Connections
162 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.catalyst. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.catalyst, so a change to sql.catalyst can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
23→33 sql.catalyst.expressions uses sql.catalyst.analysis.resolverpart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.catalyst.analysis.resolver.
sql.catalyst.expressions uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.catalyst.analysis.resolver.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
23→34 sql.catalyst.expressions uses sql.typespart of a loop✕
Connections
721 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.types. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.types, so a change to sql.types can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
23→35 sql.catalyst.expressions uses ml.attributepart of a loop✕
Connections
27 pairs of types, where a type in sql.catalyst.expressions uses a type in ml.attribute. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses ml.attribute, so a change to ml.attribute can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
23→36 sql.catalyst.expressions uses sql.internalpart of a loop✕
Connections
45 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.internal. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.expressions uses sql.internal, so a change to sql.internal can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
23→37 sql.catalyst.expressions uses sql.internal.WindowFramepart of a loop✕
Connections
9 pairs of types, where a type in sql.catalyst.expressions uses a type in sql.internal.WindowFrame.
sql.catalyst.expressions uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
23→39 sql.catalyst.expressions uses org.apache.sparkpart of a loop✕
Connections
4 pairs of types, where a type in sql.catalyst.expressions uses a type in org.apache.spark.
sql.catalyst.expressions uses org.apache.spark, so a change to org.apache.spark can break sql.catalyst.expressions, while a change to sql.catalyst.expressions can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.execution.python uses internal, so a change to internal can break sql.execution.python, while a change to sql.execution.python can't break internal.
Position
Below the diagonal: sql.execution.python uses a module listed above it, which is the healthy direction.
sql.execution.python uses sql.execution.streaming.state.StateStoreEncoding, so a change to sql.execution.streaming.state.StateStoreEncoding can break sql.execution.python, while a change to sql.execution.python can't break sql.execution.streaming.state.StateStoreEncoding.
Position
Below the diagonal: sql.execution.python uses a module listed above it, which is the healthy direction.
28 pairs of types, where a type in sql.execution.python uses a type in sql.execution. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.python uses sql.execution, so a change to sql.execution can break sql.execution.python, while a change to sql.execution.python can't break sql.execution.
Position
Below the diagonal: sql.execution.python uses a module listed above it, which is the healthy direction.
sql.execution.python uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql.execution.python, while a change to sql.execution.python can't break sql.catalyst.expressions.aggregate.
Position
Below the diagonal: sql.execution.python uses a module listed above it, which is the healthy direction.
sql.execution.python uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.execution.python, while a change to sql.execution.python can't break sql.catalyst.plans.logical.
Position
Below the diagonal: sql.execution.python uses a module listed above it, which is the healthy direction.
54 pairs of types, where a type in sql.execution.python uses a type in sql.catalyst.expressions. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.python uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.execution.python, while a change to sql.execution.python can't break sql.catalyst.expressions.
Position
Below the diagonal: sql.execution.python uses a module listed above it, which is the healthy direction.
24→26 sql.execution.python uses utilpart of a loop✕
Connections
5 pairs of types, where a type in sql.execution.python uses a type in util.
sql.execution.python uses sql.catalyst, so a change to sql.catalyst can break sql.execution.python, while a change to sql.execution.python can't break sql.catalyst.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
24→34 sql.execution.python uses sql.typespart of a loop✕
Connections
21 pairs of types, where a type in sql.execution.python uses a type in sql.types.
sql.execution.python uses sql.types, so a change to sql.types can break sql.execution.python, while a change to sql.execution.python can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
24→35 sql.execution.python uses ml.attributepart of a loop✕
Connections
22 pairs of types, where a type in sql.execution.python uses a type in ml.attribute.
sql.execution.python uses ml.attribute, so a change to ml.attribute can break sql.execution.python, while a change to sql.execution.python can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
24→36 sql.execution.python uses sql.internalpart of a loop✕
Connections
10 pairs of types, where a type in sql.execution.python uses a type in sql.internal.
sql.execution.python uses sql.internal, so a change to sql.internal can break sql.execution.python, while a change to sql.execution.python can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
24→37 sql.execution.python uses sql.internal.WindowFramepart of a loop✕
Connections
1 pair of types, where a type in sql.execution.python uses a type in sql.internal.WindowFrame.
sql.execution.python uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.execution.python, while a change to sql.execution.python can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
24→38 sql.execution.python uses rddpart of a loop✕
Connections
13 pairs of types, where a type in sql.execution.python uses a type in rdd.
sql.execution.python uses rdd, so a change to rdd can break sql.execution.python, while a change to sql.execution.python can't break rdd.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
24→39 sql.execution.python uses org.apache.sparkpart of a loop✕
Connections
26 pairs of types, where a type in sql.execution.python uses a type in org.apache.spark. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.execution.python uses org.apache.spark, so a change to org.apache.spark can break sql.execution.python, while a change to sql.execution.python can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
streaming uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break streaming, while a change to streaming can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: streaming uses a module listed above it, which is the healthy direction.
streaming uses streaming.scheduler, so a change to streaming.scheduler can break streaming, while a change to streaming can't break streaming.scheduler.
Position
Below the diagonal: streaming uses a module listed above it, which is the healthy direction.
streaming uses sql.execution.python, so a change to sql.execution.python can break streaming, while a change to streaming can't break sql.execution.python.
Position
Below the diagonal: streaming uses a module listed above it, which is the healthy direction.
util uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break util, while a change to util can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: util uses a module listed above it, which is the healthy direction.
storage uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break storage, while a change to storage can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql, while a change to sql can't break sql.catalyst.expressions.aggregate.
Position
Below the diagonal: sql uses a module listed above it, which is the healthy direction.
sql.catalyst uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.catalyst, while a change to sql.catalyst can't break sql.pipelines.graph.
Position
Below the diagonal: sql.catalyst uses a module listed above it, which is the healthy direction.
sql.catalyst uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.catalyst, while a change to sql.catalyst can't break sql.catalyst.trees.
Position
Below the diagonal: sql.catalyst uses a module listed above it, which is the healthy direction.
sql.catalyst uses sql.catalyst.util, so a change to sql.catalyst.util can break sql.catalyst, while a change to sql.catalyst can't break sql.catalyst.util.
Position
Below the diagonal: sql.catalyst uses a module listed above it, which is the healthy direction.
sql.catalyst uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.catalyst, while a change to sql.catalyst can't break sql.catalyst.plans.logical.
Position
Below the diagonal: sql.catalyst uses a module listed above it, which is the healthy direction.
sql.catalyst uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.catalyst, while a change to sql.catalyst can't break sql.catalyst.expressions.
Position
Below the diagonal: sql.catalyst uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis.resolver uses internal, so a change to internal can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break internal.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis.resolver uses sql.connector.catalog, so a change to sql.connector.catalog can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.connector.catalog.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis.resolver uses sql.errors, so a change to sql.errors can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.errors.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis.resolver uses sql.catalyst.expressions.aggregate, so a change to sql.catalyst.expressions.aggregate can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.catalyst.expressions.aggregate.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
147 pairs of types, where a type in sql.catalyst.analysis.resolver uses a type in sql.catalyst.plans.logical. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis.resolver uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.catalyst.plans.logical.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
47 pairs of types, where a type in sql.catalyst.analysis.resolver uses a type in sql.catalyst.analysis. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis.resolver uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.catalyst.analysis.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
131 pairs of types, where a type in sql.catalyst.analysis.resolver uses a type in sql.catalyst.expressions. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis.resolver uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.catalyst.expressions.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis.resolver uses sql.catalog, so a change to sql.catalog can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.catalog.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis.resolver uses sql.catalyst, so a change to sql.catalyst can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.catalyst.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
sql.catalyst.analysis.resolver uses ml.tree, so a change to ml.tree can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break ml.tree.
Position
Below the diagonal: sql.catalyst.analysis.resolver uses a module listed above it, which is the healthy direction.
33→34 sql.catalyst.analysis.resolver uses sql.typespart of a loop✕
Connections
5 pairs of types, where a type in sql.catalyst.analysis.resolver uses a type in sql.types.
sql.catalyst.analysis.resolver uses sql.types, so a change to sql.types can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.types.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
33→35 sql.catalyst.analysis.resolver uses ml.attributepart of a loop✕
Connections
23 pairs of types, where a type in sql.catalyst.analysis.resolver uses a type in ml.attribute.
sql.catalyst.analysis.resolver uses ml.attribute, so a change to ml.attribute can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break ml.attribute.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
33→36 sql.catalyst.analysis.resolver uses sql.internalpart of a loop✕
Connections
37 pairs of types, where a type in sql.catalyst.analysis.resolver uses a type in sql.internal. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
sql.catalyst.analysis.resolver uses sql.internal, so a change to sql.internal can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.internal.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
33→37 sql.catalyst.analysis.resolver uses sql.internal.WindowFramepart of a loop✕
Connections
2 pairs of types, where a type in sql.catalyst.analysis.resolver uses a type in sql.internal.WindowFrame.
sql.catalyst.analysis.resolver uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.catalyst.analysis.resolver, while a change to sql.catalyst.analysis.resolver can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.types uses sql.pipelines.graph, so a change to sql.pipelines.graph can break sql.types, while a change to sql.types can't break sql.pipelines.graph.
Position
Below the diagonal: sql.types uses a module listed above it, which is the healthy direction.
sql.types uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.types, while a change to sql.types can't break sql.catalyst.expressions.
Position
Below the diagonal: sql.types uses a module listed above it, which is the healthy direction.
sql.types uses sql.catalyst.analysis.resolver, so a change to sql.catalyst.analysis.resolver can break sql.types, while a change to sql.types can't break sql.catalyst.analysis.resolver.
Position
Below the diagonal: sql.types uses a module listed above it, which is the healthy direction.
34→37 sql.types uses sql.internal.WindowFramepart of a loop✕
Connections
1 pair of types, where a type in sql.types uses a type in sql.internal.WindowFrame.
sql.types uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.types, while a change to sql.types can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.internal uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break sql.internal, while a change to sql.internal can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: sql.internal uses a module listed above it, which is the healthy direction.
sql.internal uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.internal, while a change to sql.internal can't break sql.catalyst.trees.
Position
Below the diagonal: sql.internal uses a module listed above it, which is the healthy direction.
sql.internal uses sql.execution.datasources, so a change to sql.execution.datasources can break sql.internal, while a change to sql.internal can't break sql.execution.datasources.
Position
Below the diagonal: sql.internal uses a module listed above it, which is the healthy direction.
sql.internal uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.internal, while a change to sql.internal can't break sql.catalyst.plans.logical.
Position
Below the diagonal: sql.internal uses a module listed above it, which is the healthy direction.
sql.internal uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.internal, while a change to sql.internal can't break sql.catalyst.catalog.
Position
Below the diagonal: sql.internal uses a module listed above it, which is the healthy direction.
sql.internal uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.internal, while a change to sql.internal can't break sql.catalyst.analysis.
Position
Below the diagonal: sql.internal uses a module listed above it, which is the healthy direction.
sql.internal uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.internal, while a change to sql.internal can't break sql.catalyst.expressions.
Position
Below the diagonal: sql.internal uses a module listed above it, which is the healthy direction.
sql.internal uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.internal, while a change to sql.internal can't break sql.internal.WindowFrame.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
36→39 sql.internal uses org.apache.sparkpart of a loop✕
Connections
7 pairs of types, where a type in sql.internal uses a type in org.apache.spark.
sql.internal uses org.apache.spark, so a change to org.apache.spark can break sql.internal, while a change to sql.internal can't break org.apache.spark.
Position
Above the diagonal, so it is part of a loop. Neither module can be changed, tested or released on its own until one of these dependencies is removed.
sql.internal.WindowFrame uses sql.internal, so a change to sql.internal can break sql.internal.WindowFrame, while a change to sql.internal.WindowFrame can't break sql.internal.
Position
Below the diagonal: sql.internal.WindowFrame uses a module listed above it, which is the healthy direction.
rdd uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break rdd, while a change to rdd can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: rdd uses a module listed above it, which is the healthy direction.
99 pairs of types, where a type in rdd uses a type in org.apache.spark. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
org.apache.spark uses sql.connect.client.SparkConnectClient, so a change to sql.connect.client.SparkConnectClient can break org.apache.spark, while a change to org.apache.spark can't break sql.connect.client.SparkConnectClient.
Position
Below the diagonal: org.apache.spark uses a module listed above it, which is the healthy direction.
29 pairs of types, where a type in org.apache.spark uses a type in scheduler. Showing 25 of them; the rest are in namespace-graph.json in this report's bundle.
org.apache.spark uses sql.pipelines.graph, so a change to sql.pipelines.graph can break org.apache.spark, while a change to org.apache.spark can't break sql.pipelines.graph.
Position
Below the diagonal: org.apache.spark uses a module listed above it, which is the healthy direction.
org.apache.spark uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break org.apache.spark, while a change to org.apache.spark can't break sql.catalyst.expressions.
Position
Below the diagonal: org.apache.spark uses a module listed above it, which is the healthy direction.
org.apache.spark uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break org.apache.spark, while a change to org.apache.spark can't break sql.internal.WindowFrame.
Position
Below the diagonal: org.apache.spark uses a module listed above it, which is the healthy direction.
sql.classic uses sql.catalyst.trees, so a change to sql.catalyst.trees can break sql.classic, while a change to sql.classic can't break sql.catalyst.trees.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
sql.classic uses sql.execution.streaming.runtime, so a change to sql.execution.streaming.runtime can break sql.classic, while a change to sql.classic can't break sql.execution.streaming.runtime.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
sql.classic uses sql.catalyst.util, so a change to sql.catalyst.util can break sql.classic, while a change to sql.classic can't break sql.catalyst.util.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
sql.classic uses sql.catalyst.plans.logical, so a change to sql.catalyst.plans.logical can break sql.classic, while a change to sql.classic can't break sql.catalyst.plans.logical.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
sql.classic uses sql.catalyst.catalog, so a change to sql.catalyst.catalog can break sql.classic, while a change to sql.classic can't break sql.catalyst.catalog.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
sql.classic uses sql.catalyst.analysis, so a change to sql.catalyst.analysis can break sql.classic, while a change to sql.classic can't break sql.catalyst.analysis.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
sql.classic uses sql.catalyst.expressions, so a change to sql.catalyst.expressions can break sql.classic, while a change to sql.classic can't break sql.catalyst.expressions.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
sql.classic uses sql.execution.python, so a change to sql.execution.python can break sql.classic, while a change to sql.classic can't break sql.execution.python.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
sql.classic uses sql.internal.WindowFrame, so a change to sql.internal.WindowFrame can break sql.classic, while a change to sql.classic can't break sql.internal.WindowFrame.
Position
Below the diagonal: sql.classic uses a module listed above it, which is the healthy direction.
Findings mapped to OWASP categories; the specific CVEs/secrets are in the Security dimension cards below and findings.md (redacted only on the public version of this report).
OWASP category
Findings
Severity
A05:2025 — Injection
213
High / Critical
A03:2025 — Software Supply Chain Failures
24
High / Critical
A02:2025 — Security Misconfiguration
6
High / Critical
A07:2025 — Authentication Failures
5
High / Critical
Roadmap
Immediately resolve the two leaked secrets in REDACTED and REDACTED to secure the codebase. Shift the frontend to TypeScript by adding a typecheck step and progressively typing high-traffic modules, while simultaneously making custom controls fully keyboard-operable and fixing anchor links. Refactor the five largest files into smaller, focused modules to improve maintainability, and enforce strict null safety in Scala by enabling explicit null checks and replacing unsafe null assertions with proper Option handling.
Ranked by impact ÷ effort. "Helps" is the estimated gain on the 0–100 health score.
Do this
Helps
Effort
Dimension
Resolve the 2 Leaked secret finding(s) in REDACTED Scanning — start with REDACTED, REDACTED.
Adopt TypeScript for the frontend: add a typecheck step to the build, then either type-check the existing JavaScript in place (`checkJs`/JSDoc types) or convert the highest-traffic modules first — no file is typed today, so this is an adoption, not a clean-up.
On Scala 3, add `-Yexplicit-nulls` to `scalacOptions` so `String` and `String | Null` mean different things, and type each Java API's result as nullable where it can return null; on Scala 2 (or as a first step), enable WartRemover's `Null` wart (`wartremoverErrors += Wart.Null`) or Scalafix's `DisableSyntax.noNulls = true` so the code holds absence in `Option`. Then pay the assertions down rather than adding to them: each `.nn` is a null the compiler could not rule out — wrap the Java value in `Option(…)` at the boundary, pattern-match on `null` once there, or change the API so the null cannot arrive; an `unsafeNulls` import switches the checking off for everything after it.
Enforce accessibility in the toolchain: add an accessibility check that can read your UI — no component framework was detected and your pages are rendered by server-side templates, which neither the JSX/Vue ESLint plugins nor the HTML-template linters can parse; run axe/pa11y over the rendered pages, or assert the accessibility invariants over that rendered HTML in the test suite you already have, then assert with jest-axe in tests, then gate axe/pa11y/Lighthouse in CI.
Per-file score 0–10 — a quality signature. Of 1496 files carrying findings, judged against the Production bar: 1% slop · 54% mixed · 45% near-clean.
File
Score
Band
Worst signal
REDACTED
0.9
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.3
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.3
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.3
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.4
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.5
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.5
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.7
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.7
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.8
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.9
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
2.9
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
3.0
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
3.0
Slop
Static Analysis (SAST): High: REDACTED
REDACTED
3.5
Slop
IaC & Container Security: High IaC: REDACTED
REDACTED
4.0
Mixed
Static Analysis (SAST): Medium: REDACTED
REDACTED
4.0
Mixed
Static Analysis (SAST): Medium: REDACTED
REDACTED
4.0
Mixed
Static Analysis (SAST): Medium: REDACTED
REDACTED
4.0
Mixed
Static Analysis (SAST): Medium: REDACTED
REDACTED
4.0
Mixed
Static Analysis (SAST): Medium: REDACTED
How the grades work
Every finding carries one of four grades. Three say how serious it is. The fourth says this
survey could not settle it — and it is a grade, not a gap.
Critical — 211
A definite problem that already costs you something and drags the score down: a
missing authorisation check, a dependency with a known exploit, a build that does not reproduce. Failure here
tends to cause failures elsewhere.
Serious — 4803
Likely wrong, but not failing yet. It degrades
the codebase over a longer horizon and can cause failures elsewhere — not urgent this week, not something to
carry for two years either.
Minor — 686
Recorded, with no effect on how the codebase functions.
Present so the survey is complete, not because it needs doing.
Could not be resolved — 14
Something this survey could not settle
from the outside, and which could be critical or serious. Either a control was required and no
positive evidence of it exists in the repository — a backup job that nothing shows was ever restored from proves
nothing about restores — or our own analysis could not run over that part of the tree. This is not a clean
result. These are excluded from the score rather than awarded a pass, so the number on the cover neither
rewards nor penalises them: if you act on this survey without resolving them, you carry that risk yourself. Each
one is named under Limitations.
Methodology & how to trust this report
Watchdog is a deep, periodic assessment — run each sprint, monthly, or quarterly, taking the time to go wider and deeper than a quick check and surfacing in one coherent report what you'd otherwise piece together from a dozen separate tools. It scores deterministically: the same commit yields the same score, every run. 85 of 85 evaluated dimensions are computed purely by tools and static analysis (confidence 1.0); none rely on LLM judgement. Overall confidence is 0.9 — the weighted average across measured dimensions; it falls as more of the score leans on LLM-assisted judgement and rises when it's fully tool-backed.
Every figure here is one of three kinds, and we label which: ✓ Measured — a deterministic fact (LoC, complexity, coverage); ~ Modeled — an estimate from a stated model (cost, effort, value-at-risk), always a range with its assumptions, never a precise fact; ◐ Advisory — an LLM prose judgement. We never present a modelled estimate as if it were measured. Perfect or absent scores carry their provenance too (ADR-0011): ✓ Tool-verified means the property itself was measured across the surface; ○ Nothing flagged means the probes came back clean — a claim bounded by what a repository can show; ⊘ Not evidenced means a working control (a tested restore, an automated rollback) showed no positive evidence — absence of evidence is not evidence of a control, so it's excluded from the score rather than awarded a spurious 10; ◐ Sampled · advisory marks an LLM verdict over a bounded sample — advisory, never a deterministic measurement.
What we checked — 85 dimensions across the health lenses
Each chip is a dimension scored from real signals across architecture, testing, dependencies, security & compliance, documentation, git-history and code quality — in one coherent pass. A surface report typically covers a handful.
How to trust any code-health report — three questions
Can you open the finding? Real findings cite a repo-relative file and line you can open at the cited line — never an absolute scratch path. Here, 5657 of 5700 do; the remainder are repo-wide signals — a dimension-level measurement, not a single line. (Every path in this report is repo-relative by construction: paths are normalized at the producer and the report is rejected if any rooted path leaks through.)
Is there a tool behind the number? Every score below names the method that produced it — Roslyn, git, a scanner, or (for a handful of documentation/naming dimensions) an LLM labelled sampled · advisory — not a narrative.
Does re-running give the same result? Run it again on the same commit and the score — and this report, byte for byte — is identical. A report whose numbers move between runs is describing the run, not the code.
This report answers yes to all three. That's the bar to hold any assessment to.
Tools & methods
The actual versions used this run (captured at analysis time) — re-run on the same commit for the identical score.
Method
Backs
Version
Evaluator
Roslyn static analysis
Complexity, cohesion, coupling, dead code, API surface, layering
What ran differently this time — a tool absent, degraded, or that fell back to an estimate. Named openly, not folded silently into the scores. A degraded run also records its exact cause in diagnostics.md.
D10 Test Quality — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. The 31 test(s) behind this row are the ones the JavaScript/TypeScript census could read, and this repository also carries at least 4,246 test source file(s) (.scala, .java, .py) that it cannot: it reads JavaScript/TypeScript test declarations off disk, so a JUnit/pytest-style suite is invisible to it. Skipped tests, zero-assertion tests and the other quality signals on this row are UNMEASURED in that suite — their absence from the counts above is a gap in this analyzer's language coverage, not a finding that those tests are sound.
D11 Test Reliability — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. Test reliability not scored — the JavaScript/TypeScript suite (jest via `npm ci --ignore-scripts` in ui-test/ (0 tests)) ran but surfaced no test cases to the runner, so flakiness couldn't be exercised.
D14 License Compliance — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. This repository declares a Python pyproject.toml/requirements.txt (pip/uv/Poetry), but the licence verdict published here was taken over its gem dependencies. Nothing was read about its Python dependencies' licensing in either direction, and a clean score on this card must not be read as covering them.
D14 License Compliance — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. This repository declares a Maven POM, but the licence verdict published here was taken over its gem dependencies. Nothing was read about its Maven dependencies' licensing in either direction, and a clean score on this card must not be read as covering them.
D14 License Compliance — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. This repository declares package.json, but the licence verdict published here was taken over its gem dependencies. Nothing was read about its npm dependencies' licensing in either direction, and a clean score on this card must not be read as covering them.
D19 Documentation Quality — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. Each ecosystem's manifests (what makes a folder runnable or a published package) need their own proof before D19 judges its units. Until then those folders' READMEs are judged only as documents (boilerplate, structure), never counted as complete.
D19 Documentation Quality — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. This kind did not reach the 95 % precision gate in three adjudicated runs (docs/plans/deterministic-engine/adjudication/D19-sample3-v3.md), so by the pre-registered fallback it publishes no row until a re-measurement shows it does. The candidates are counted here, not reported as findings, and the score does not include them.
D21 Naming Consistency — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. D21's deterministic naming rules are gated on C# first (docs/plans/deterministic-engine.md, P5): each other language is switched on only after its own concept suite and A/B gate pass. Until then its declarations carry no naming verdict this run — this says nothing about whether its names are consistent.
D22 Internal API Consistency — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. D22 identifies the intentionally-exposed surface from `IsPackable` and `.Contracts` project names, MSBuild conventions read off the loaded project set. This target exposed no such projects, so the probe never ran; this says nothing about whether the repository has a public API. This repository commits no C#/VB source at all, so there was never an MSBuild project set to read these conventions off. That is OUR side and it is a COLLECTOR gap, not an environment fault: it declares a published package (REDACTED), but no published-package marker D22 reads admitted any project here, so this ecosystem's public API has no collector, and the remedy is to write one — no change to the scan image can close it.
D24 Comment Value — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. This check's reader does not cover the language this repository's product is written in, so it had nothing of the product to read. That is a gap in this analyzer's language reach — not a finding about this repository.
D27 Navigability — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. Navigability is taken over call sites whose callee resolved. For C# that population includes calls into referenced assemblies; for this language only calls declared inside the repository count, so the abstention threshold reads the repository's use of its standard library as a resolution failure. That is a limit of this analyzer, not a build problem in this repository.
D31 IaC & Container Security — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. Checkov ran but its report could not be read (checkov produced no output — not a clean result). Every misconfiguration only checkov reports — most of its Terraform, CloudFormation, Bicep and Ansible policies — is absent from this card, so its absence here is not evidence those resources are configured correctly.
D32 Data Compliance (PII/GDPR) — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. `common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala`, `core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala`, `core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala`, `core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala`, `core/src/main/scala/org/apache/spark/deploy/master/ui/ApplicationPage.scala`, … (+43 more) produced a parse error, so every rule in this engine's `gdpr.yml` was absent there. That absence is NOT a clean result: these rules detect personal data crossing a boundary into a log sink, a URL or browser storage, and a file that was never parsed cannot report any of the three. The rest of the tree analysed normally and its rows above stand; only these files are unaccounted for. You can widen what we reach: fix the syntax error (or exclude the file deliberately) and re-scan to cover it.
AX1 Captive dependencies — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. This check reads Microsoft.Extensions.DependencyInjection registrations in C# and Spring beans in Java/Kotlin only, and this repository defines its own container in python/pyspark/sql/datasource_internal.py, whose lifetimes are not modelled yet, so it had nothing of this repository's product to read. That is a gap in this analyzer's language reach — not a finding that the repository is free of what this check looks for.
AXR1 Runtime accessibility — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. Render strategy: boot via REDACTED (REDACTED). docker build failed (exit 1) — 22 | LABEL org.opencontainers.image.licenses="Apache-2.0" -------------------- ERROR: failed to solve: eclipse-temurin:25-jre: failed to resolve source metadata for docker.io/library/eclipse-temurin:25-jre: failed to do request: Head "https://registry-1.docker.io/v2/library/eclipse-temurin/manifests/25-jre": Forbidden; runtime evidence skipped.
C1 Data Protection — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. These personal data controls are read from declarative annotations, request middleware, entity/column names and guard methods in a C# source model, and none was loaded on this run, so there was nothing to gather. That is a gap in this analyzer's language reach — not a finding that the repository lacks personal data controls.
C2 Access Controls — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. These authorization controls are read from declarative annotations, request middleware, entity/column names and guard methods in a C# source model, and none was loaded on this run, so there was nothing to gather. That is a gap in this analyzer's language reach — not a finding that the repository lacks authorization controls.
C3 Audit Trail — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. These audit controls are read from declarative annotations, request middleware, entity/column names and guard methods in a C# source model, and none was loaded on this run, so there was nothing to gather. That is a gap in this analyzer's language reach — not a finding that the repository lacks audit controls.
C4 Data Retention — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. These retention controls are read from declarative annotations, request middleware, entity/column names and guard methods in a C# source model, and none was loaded on this run, so there was nothing to gather. That is a gap in this analyzer's language reach — not a finding that the repository lacks retention controls.
C5 Data-Subject Rights — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. These data-subject rights controls are read from declarative annotations, request middleware, entity/column names and guard methods in a C# source model, and none was loaded on this run, so there was nothing to gather. That is a gap in this analyzer's language reach — not a finding that the repository lacks data-subject rights controls.
IC1 Incompleteness & stubs — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. 78 further occurrence(s) are not listed individually; the score already reflects all 118.
S1 Web-Security Posture — not measured this run — Watchdog could not measure this here. That is a gap on our side — a collector, parser or image we have not built yet — and it is neither a defect in this repository nor evidence that the check would have passed. These web-security controls are read from declarative annotations, request middleware, entity/column names and guard methods in a C# source model, and none was loaded on this run, so there was nothing to gather. That is a gap in this analyzer's language reach — not a finding that the repository lacks web-security controls.
X3 Exception handling — measured, with a gap in what it reached — Watchdog measured this, but not all of it. What it did not reach is a gap on our side — a collector, parser or image we have not built yet — so the numbers on that dimension cover less than the repository, and the part left out is not evidence that it would have passed. 74 further occurrence(s) are not listed individually; the score already reflects all 114.
Repo exclusion declarations (.gitattributes linguist-generated/vendored, .editorconfig generated_code): none declared — every source file was scored.
Limitations & what we did not check
Watchdog assesses the repository exactly as committed, and only the repository. By design it does not reach outside the source tree: the live cloud account, the running CI/CD pipeline, the host's branch-protection and approval rules, the production configuration, or a restore actually exercised against a backup are all out of scope. That boundary is a feature, not a gap — a repo-relative, deterministic scan re-runs identically on any commit and every finding opens at a real file and line, where a live audit can neither be reproduced nor traced. The visible consequence is that controls which leave no in-repo evidence are reported as "not evidenced" and excluded from the score rather than awarded a number a static scan cannot justify.
Per-dimension blind spots
For each dimension that was measured, what a static, repo-only scan structurally cannot see — the honest edge of the measurement, not a failure of it.
D1 Cyclomatic Complexity: Cyclomatic complexity counts branches statically — it cannot tell an essential decision tree from accidental tangle, nor see complexity that lives in data or configuration (large switch-case token tables, DSL lexers/parsers, data-as-code rule tables) rather than control flow: a tokenizer's many single-character cases read as high complexity though each branch is trivial.
D2 Cognitive Complexity: Cognitive-complexity heuristics approximate how hard code is to follow; genuine domain difficulty and well-named intent that eases reading are not captured.
D3 God Classes: "God class" is sized by members and responsibilities visible in the type — a deliberately broad facade over a coherent subsystem can read the same as an accidental grab-bag. For front-end JS the file-length check is cohesion-aware (a single-responsibility module — one class/IIFE — earns a 3× threshold), but cohesion is approximated from top-level declarations, not true dependency structure.
D4 Code Duplication: Duplication is token-similarity — an in-process token-stream comparison over sliding windows, with type-aware normalization — so it finds copy-paste, not semantic duplication expressed differently. Committed machine-written code (scaffolded migrations, designer/codegen output, protobuf/OpenAPI stubs, model snapshots) is EXCLUDED — its repetition is the tool's, not the team's — so the score reflects hand-written duplication only.
D5 Coupling: Coupling is measured between projects/assemblies — runtime coupling through DI, reflection, messaging or shared databases is invisible to a static reference graph.
D6 Cohesion (LCOM4): LCOM4 cohesion is syntactic — it infers connectivity from which methods touch which fields/methods by name, not from real runtime behaviour or intent.
D9 Test Distribution: The test-pyramid shape is inferred from project/folder naming and references, with a single test host bucketed per-file by its path tier and content signals — a suite that names tiers unconventionally and gives no per-file signal can still be mis-bucketed.
D10 Test Quality: Assertion density is structural — it cannot tell a meaningful behavioural assertion from a trivial one, only that an assertion is present.
D12 Dependency Hygiene: Dependency health reads manifests and lockfiles — a vulnerability in a vendored/copied dependency, or risk from how a dependency is actually used, is outside this view.
D13 REDACTED Scanning: REDACTED detection is signature- and entropy-based on the current tree — a secret that does not match a known pattern, or one already rotated, will not be flagged (a clean scan is "nothing matched", not "no secrets exist").
D14 License Compliance: License compatibility is checked against declared package metadata and a policy — mislabelled or missing license metadata, and obligations that depend on how you distribute, are not resolved here.
D15 Churn × Complexity Hotspots: Churn hotspots come from git history — a freshly imported or squashed repository has no churn signal, and recent rewrites can mask a historically risky file.
D16 Bus Factor: Bus-factor is a time-decayed model of commit attribution (who has recently, repeatedly worked a file), not comprehension — pairing, review and reading-without-committing spread knowledge it can't see; bot commits and shared accounts still distort it.
D17 Explicit Debt: Acknowledged-debt signals (TODO/FIXME, suppressions, dead code) are textual — undocumented debt that nobody marked, and debt that lives in design rather than annotations, is invisible. Committed machine-written code (scaffolded migrations, designer/codegen output, generated stubs) is excluded — it is never the team's dead code to delete.
D19 Documentation Quality: Documentation quality publishes two things only: a runnable or packaged unit with no example, and a multi-project root README that maps none of its parts. It does not judge clarity, and its other checks (overview, licence, boilerplate, structure, empty sections, XML-doc coverage) are withheld after adjudication: their candidates are counted in the card's gaps, not shown as rows. A false statement in a document is M4's; units are .NET projects only.
D20 ADR Quality: ADR quality is an LLM read of the decision records present — it cannot know about decisions made and never recorded, and its verdict is sampled and advisory.
D26 Project Cohesion: Project focus is sized from members/namespaces per project — a project that is broad by deliberate design reads the same as one that has sprawled.
D28 Secrets (history): Secret-history scanning sweeps the git log for known patterns — a secret that predates the available history, or never matched a signature, is not found (clean means "nothing matched in the history we can see").
D29 Static Analysis (SAST): SAST findings are pattern-based (semgrep) — it finds classes of bug it has rules for; logic flaws, auth/authorization gaps and issues needing runtime context are out of reach (and clean means "no rule matched").
D30 Dependency Vulnerabilities: CVE matching depends on accurate package/version metadata and on the advisory databases — a vulnerability with no published advisory, or in code not declared as a dependency, is not seen. Coverage needs a RESOLVED graph: an unpinned REDACTED, or a pom without a resolved build, yields partial coverage rather than a clean verdict. An ecosystem the analyzer cannot scan is reported as unmeasured, never as clean.
D31 IaC & Container Security: IaC scanning checks Dockerfiles/Terraform/Kubernetes against best-practice rules — it cannot see the live cloud account, runtime configuration, or drift between the committed config and what is actually deployed.
D34 Knowledge Freshness: Freshness is decayed commit RECENCY, not comprehension — code read often but rarely committed reads as orphaned, and stable code that genuinely needs no changes is penalised the same as forgotten code; bot/squash commits distort it like the bus factor.
D35 Change Coupling: Change coupling is co-change in COMMITS — files split across separate commits, or coupled only through a shared config/build step, read as uncoupled, and a sweeping commit (rename/format) is excluded so it doesn't couple everything. It shows that files change together, not WHY: a high coupling can be a healthy cohesive pair as readily as a hidden leak.
D43 Malicious Dependencies: Only packages some vulnerability database has already NAMED as malicious are seen — a compromise published in the last hours, or never reported at all, is invisible here, and this dimension reading 10 is not evidence that a dependency is trustworthy. There is no typosquat or dependency-confusion analysis: a package nobody has reported is simply absent from the feeds. Coverage is the dependency scan's: an ecosystem that could not be scanned is disclosed as unmeasured, never as clean.
D44 Platform End-of-Life: The support table is FROZEN, so it goes out of date by losing RECALL: a release that ended support after the table was written is missed until the table is refreshed, and this dimension reading 10 is not evidence that a platform is current. Only platforms the repository DECLARES in a place this pass reads are seen — a runtime named only in a REDACTED (D31's subject), in a CI workflow (D29's), or in a file this pass does not parse (go.mod, a Gemfile ruby directive) is invisible here, which is why a repository declaring none of them abstains rather than scoring. Only frameworks with a PUBLISHED support policy are tracked: React, Flask and Express publish none, so their age cannot be judged and their absence from a report is not a statement that they are supported.
AC2 Forms & labels: Label association is read from static markup — a label wired up at runtime (JS-set aria-labelledby, framework-injected ids) reads as missing, a present label says nothing about whether its text is correct. A known UI-library field component (e.g. a JSX <TextField>) is now checked conservatively — flagged only when it carries NO label/aria-label/aria-labelledby/id/name — but wrapper/context-labelled libraries (Chakra/Radix FormControl+FormLabel) aren't statically visible (possible false positive) and non-JSX lowercased components are still skipped. A click handler on a plain element is now asked for a name too (it is a control the author declared), but the subtree test that answers it is deliberately generous: any DYNAMIC text expression in the subtree counts as a name, so an icon chosen by a ternary ({cond ? <IconA/> : <IconB/>}) reads as named, and a glyph component from a library the icon-import list does not know still names its parent. A clean result is "no unlabelled control found", not a labelling proof.
AC3 Page structure: Page structure is read from the static markup tree — landmarks, headings and lang injected at runtime aren't seen, heading ORDER is checked structurally (not against the rendered visual hierarchy), and lang/title/main fire only on full documents, never partials, and the data-table check sees header-cell presence (a <th> exists), not whether each header correctly associates with its cells. Static readiness, not conformance.
AC4 Keyboard semantics: Keyboard semantics are inferred from markup attributes — interactivity wired purely in script, focus managed at runtime, and component-level handlers are invisible. A clean result means "no static keyboard-trap shape", not a keyboard-operability proof.
AC5 ARIA correctness: ARIA correctness is checked against the static role/attribute shape — roles/attributes set dynamically aren't seen, a valid role says nothing about whether it matches the element's real behaviour, and required-state checks are suppressed when a JSX spread could supply them. The two-branch toggle check (a control whose state is conveyed only by which of two mutually exclusive branches renders) reads CONDITIONALS THAT ARE ATTRIBUTES — Vue v-if/v-else/v-show and Alpine x-if/x-show — so the same toggle written as a Svelte {#if} block or a JSX ternary is control flow the markup model never projects as a branch and is not seen at all.
AC6 Visual & motion safety: Contrast and motion safety are PARTIAL by construction — literal colours (hex/rgb/hsl/named) in inline styles, in-repo <style> blocks, in-repo .css files, var() tokens, Tailwind neutral utilities and CSS-in-JS top-level declarations are read (same-rule/same-element colour+background pairs only); computed/runtime/theme colour, external-CDN stylesheets, CSS-in-JS dynamic (${…}) and nested-selector colours, cross-element pairs and image contrast stay out of reach, so a clean result is bounded by what the static CSS itself shows.
AC7 A11y enforcement: Enforcement is scored from in-repo config/CI evidence only — an a11y gate enforced in external tooling with no in-repo trace can't be credited, and a configured linter is presence, not proof the rules actually run or block a merge.
AX10 Code composition: Role is inferred from namespace/folder convention, not semantics — a domain concept living in a folder named "Services" reads as application, and the split is lines-of-code, not business value. The business-logic-share score is a SOFT, FLOORED signal: it contributes to the Architecture lens but is floored at the Critical gate, so an infrastructure-heavy design (a gateway, an ETL, a driver) is legitimately low without being nuked to zero.
P4 Deployment & Rollback: Approval/branch-protection rules live in repository settings the scan cannot see — only their in-repo evidence (config files, workflows) is checked, so a control enforced purely in the host's settings reads as "not evidenced".
P6 Release Hygiene: Rollback/observability controls are inferred from repo artefacts (pipelines, dashboards-as-code) — controls configured in external tooling, with no in-repo trace, cannot be credited.
What it measures: How tangled the control flow is — methods with many branches are hard to test and change.
Method: Cyclomatic complexity per method (1 + decision points), computed exhaustively across production source; test projects separated by convention. Deterministic.
502 method(s) exceeded the cyclomatic complexity threshold of 15; the worst was CheckAnalysis.checkAnalysis0 at 157. A further 114 method(s) were over the threshold but excluded as flat dispatchers (a long switch/match over independent cases: many branches, almost no nesting), the largest being SparkConnectPlanner.transformRelation at 66 — they are counted neither in the figure above nor in this dimension's score. 47 files carry no cyclomatic complexity row at all for this reason — every one of their over-threshold methods was excluded, so the exclusion is disclosed nowhere in the file itself: sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementCodes.scala (SqlStatementCodes.classify at 59), sql/api/src/main/scala/org/apache/spark/sql/catalyst/JavaTypeInference.scala (JavaTypeInference.encoderFor at 42), sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DeserializerBuildHelper.scala (DeserializerBuildHelper.createDeserializerDefault at 37), core/src/main/scala/org/apache/spark/scheduler/SparkListenerBus.scala (SparkListenerBus.doPostEvent at 35), sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala (Resolver.resolve at 32), and 42 more not listed here. They are named here because the per-file figures other dimensions report are taken BEFORE this exclusion, so such a file can show a high maximum complexity elsewhere in this report and nothing here, with nothing to reconcile the two.
+ 488 more group(s) — more in Appendix A; the complete list is findings.md.
What to do
Bring the 20 bodies over 60 down to 60 or less in Cyclomatic Complexity — start with CheckAnalysis.checkAnalysis0 (cyclomatic 157), V2ExpressionBuilder.generateExpression (cyclomatic 132), (anonymous) (cyclomatic 132). — This score is capped by its worst body, so a finding fixed alone moves it by almost nothing — the next one down takes its place. Refactoring these 20 together lifts Cyclomatic Complexity from 5.5 to about 6.5/10, projected with the scoring formula itself and assuming each lands exactly at 60; a cleaner split scores higher.
Enforce Cyclomatic Complexity in CI to reach Verified (currently Documented). — Hardens enforcement from Documented toward Prevented — provenance only; does not change the score.
Detailed fixes: d1_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: How hard the code is for a person to follow, beyond raw branching.
Method: Cognitive complexity per method (Sonar-style nesting-penalized score), computed exhaustively over production code, excluding test projects. Deterministic.
+ 1027 more group(s) — more in Appendix A; the complete list is findings.md.
What to do
Bring the 7 bodies over 120 down to 120 or less in Cognitive Complexity — start with SparkSubmit.prepareSubmitEnvironment (cognitive 201), DAGScheduler.handleTaskCompletion (cognitive 198), (anonymous) (cognitive 194). — This score is capped by its worst body, so a finding fixed alone moves it by almost nothing — the next one down takes its place. Refactoring these 7 together lifts Cognitive Complexity from 5.3 to about 5.8/10, projected with the scoring formula itself and assuming each lands exactly at 120; a cleaner split scores higher.
Enforce Cognitive Complexity in CI to reach Verified (currently Documented). — Hardens enforcement from Documented toward Prevented — provenance only; does not change the score.
Detailed fixes: d2_recommendation.md · top locations in Appendix A, every location in findings.md.
Do you agree with this assessment?
D3 · God Classes8.6 / 10Strong✓ Tool-verified
What it measures: Over-large classes that try to do too much ("god classes").
Method: God-class detection by line and method-count thresholds per logical type (partial classes unified), filtered for generated code and registration/contract false positives. Deterministic.
Resolve the 253 FileTooLong finding(s) in God Classes — start with readwriter.py (4), types.py (2), dataframe.py (2). — One of this dimension's main actionable groups (253 warning-level).
Resolve the 149 TooManyMethods finding(s) in God Classes — start with package.scala (4), Dataset.scala (3), base.py (3). — One of this dimension's main actionable groups (149 warning-level).
Resolve the 41 ClassTooLong finding(s) in God Classes — start with SymmetricHashJoinStateManager.scala (2), AstBuilder.scala, SparkConnectPlanner.scala. — One of this dimension's main actionable groups (41 warning-level).
Enforce God Classes in CI to reach Verified (currently Documented). — Hardens enforcement from Documented toward Prevented — provenance only; does not change the score.
Detailed fixes: d3_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Copy-pasted code that should be shared instead.
Method: Code duplication via token-stream sliding windows with type-aware normalization (locals masked, type names preserved), density-scored per KLoC of production code. Deterministic.
1235 duplicated block group(s) detected. A further 72 rows report members as variants of one another; they aggregate block groups already counted above and are not themselves counted. 137 of the 1307 are in trees this repository does not ship — vendored, example/demo, project-template, fixture and benchmark code — and are ranked below the shipped groups rather than excluded from them: the duplication there is real and is still counted in this dimension's score. The dimensions that publish a production-file census leave those trees out of theirs, so this count is deliberately drawn over the wider population.
+ 263 more group(s) — more in Appendix A; the complete list is findings.md.
What to do
Resolve the 87 Duplicated block (5 lines × 2) finding(s) in Code Duplication — start with Matrices.scala (6), NaiveBayes.scala (3), LevelDBTypeInfo.java (3). — One of this dimension's main actionable groups (87 warning-level).
Resolve the 78 Duplicated block (6 lines × 2) finding(s) in Code Duplication — start with __init__.py (4), LinearSVC.scala (3), PythonRDD.scala (2). — One of this dimension's main actionable groups (78 warning-level).
Resolve the 73 Duplicated block (8 lines × 2) finding(s) in Code Duplication — start with UDFRegistration.scala (4), Matrices.scala (3), __init__.py (3). — One of this dimension's main actionable groups (73 warning-level).
Enforce Code Duplication in CI to reach Verified (currently Documented). — Hardens enforcement from Documented toward Verified — provenance only; does not change the score.
Detailed fixes: d4_recommendation.md · top locations in Appendix A, every location in findings.md.
Do you agree with this assessment?
D5 · Coupling10.0 / 10Exemplary✓ Tool-verified
What it measures: Whether volatile projects sit underneath others that depend on them (so their churn ripples upward), and whether project dependencies form cycles. A widely-depended-on but stable shared/kernel project is healthy, not penalised.
Method: Dependency cycles via elementary-DFS over real .csproj references, plus Martin instability (afferent/efferent) per project. Exhaustive over the reference graph, deterministic.
Coverage: Exhaustive · type-level: afferent/efferent coupling + cycles computed over every production type — the population is all types, not a name convention.
52 production modules (Maven+Python+sbt), 0 dependency cycle(s), 0 unstable depended-on module(s). Read from the build's own module declarations; a dependsOn spread (deps: _*) and a project whose directory a helper computes are not visible there; 0 module(s) off the main sequence, with abstractness counted on 45 of the 52 (the rest declare no modelled class or interface, export only macros, declare types in source no model reads, or have no source directory of their own).
What it measures: Whether a class's methods are focused on a single responsibility.
Method: LCOM4 cohesion per production class with at least two methods: connected components of methods sharing state or calls, computed syntactically. Deterministic, not a proxy.
Coverage: Exhaustive · type-level: LCOM4 cohesion computed over every production class — the population is all types, not a name convention.
Resolve the 48 Low cohesion finding(s) in Cohesion (LCOM4) — start with HiveShim.scala (2), YarnSchedulerBackend.scala (2), LDA.scala (2). — One of this dimension's main actionable groups (48 warning-level).
Enforce Cohesion (LCOM4) in CI to reach Verified (currently Documented). — Hardens enforcement from Documented toward Verified — provenance only; does not change the score.
Detailed fixes: d6_recommendation.md · top locations in Appendix A, every location in findings.md.
Do you agree with this assessment?
D9 · Test Distribution9.8 / 10Exemplary✓ Tool-verified
What it measures: Whether the test suite has a healthy mix of unit / integration / end-to-end tests.
Method: Test projects classified (Unit/Integration/BDD/E2E) from compiled metadata; test methods counted exhaustively across projects with placement-agnostic disk fallback. Deterministic.
38559 test methods: 37721 unit, 173 integration, 139 BDD, 526 e2e. The JavaScript/TypeScript suite contributes 31 `it`/`test` case(s) across 11 test file(s) declaring at least one; its tier split is read from package names and paths only. The Python suite contributes 6002 test function(s) across 516 file(s) declaring at least one — every `def test…` in a file pytest or unittest would collect, which is those frameworks' own definition of a case; a parametrize table counts once, so this is a floor. Its tier split is read from file names and paths only. The Scala suite contributes 32526 test case(s) across 2289 file(s) in a test source set declaring at least one — every `test("…")`, `it("…")`, `property("…")` and `"…" in {…}` registration and every JUnit `@Test`, which is how munit, ScalaTest, utest, ZIO Test and specs2 declare a case; a case registered under a computed name is not counted, so this is a floor. Its tier split is INFERRED from paths and file names, except sbt's src/it source set, which the build declares.
✓ On the Gold path — maintain.
Detailed fixes: d9_recommendation.md.
Do you agree with this assessment?
D10 · Test Quality9.5 / 10Stronggated by 1 serious finding✓ Tool-verified
What it measures: Whether the tests truly assert behaviour rather than just running the code.
Method: Per-test assertions, skips, and mock references analyzed via Roslyn; structured skip-reason tags (BUG:/ENV:) separate documented deferrals from debt. Deterministic.
0 skipped, 1 zero-assertion, no mocking-framework packages referenced (hand-written doubles or no mocking) across 31 tests. Measured on the JavaScript/TypeScript suite only — at least 4,246 test source file(s) (.scala, .java, .py) went unread, so its test quality is unmeasured and is not in these counts.
No assertions: All inline scripts use CSP nonceui-test/tests/csp-compliance.test.js:55
What to do
Resolve the 1 No assertions finding(s) in Test Quality — start with csp-compliance.test.js. — One of this dimension's main actionable groups (1 warning-level).
Enforce Test Quality in CI to reach Verified (currently Documented). — Hardens enforcement from Documented toward Prevented — provenance only; does not change the score.
Detailed fixes: d10_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether dependencies are current, secure, and not bloated.
Method: Manifest scan via dotnet list package across all projects; worst-signal-per-package deduction (saturating for vulnerabilities, capped-linear for deprecation/outdated) per KLoC. Exhaustive, deterministic.
2 outdated direct gem(s), 0 pinning defect(s). 4 declaration(s) graded against rubygems.org, 4 of them against a version resolved in a committed Gemfile.lock. Whether any of these gems is DEPRECATED or YANKED is not graded — rubygems.org publishes no deprecation marker, and a yank removes a version rather than flagging it. Whether any is UNMAINTAINED is not graded either: no registry publishes a maintenance status, and release age does not stand in for one. Whether any is UNUSED is a source question, not a registry one. Known CVEs in this dependency graph are D30's question, read from the manifest there.
Outdated: ffi · ×2
✓ On the Gold path — maintain.
Detailed fixes: d12_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether any secrets (keys, tokens, passwords) have leaked into the code.
Method: In-process native secret scanner (entropy plus signature patterns) across all tracked files; no external tool. A clean result is a measured 10, not no-data zero. Deterministic.
Resolve the 2 Leaked secret finding(s) in REDACTED Scanning — start with REDACTED, REDACTED. — One of this dimension's main actionable groups (2 issue-level).
Enforce REDACTED Scanning in CI to reach Verified (currently Documented). — Hardens enforcement from Documented toward Prevented — provenance only; does not change the score.
Detailed fixes: d13_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether the licenses of third-party packages are compatible with your policy.
Method: Third-party package licenses resolved from declared package metadata and checked against the configured policy (allow/deny/copyleft). Deterministic; clean = no incompatible license found at metadata depth.
0 of 35 shipped gem(s) use a banned license. Licences were resolved from rubygems.org over the 35 gem(s) a consumer installs — this repository's runtime declarations closed transitively over its committed Gemfile.lock. Development-group and `add_development_dependency` gems are excluded: they are not distributed with this repository. ★ COVERAGE OF THIS VERDICT: it grades this repository's gem dependencies and nothing else. The repository also declares a Python pyproject.toml/requirements.txt (pip/uv/Poetry), a Maven POM and package.json, and the licences of those dependency graphs were NOT read by this pass — a gap in this engine's coverage, not a statement about them. So this result says the graded closure carries no banned licence; it does NOT say this repository's licensing is clear.
What it measures: Files that change often and are also complex — the riskiest hotspots.
Method: Per production file churn times cyclomatic complexity over a rolling window, computed from git and Roslyn/JS/Razor analysis. Exhaustive, deterministic per commit date.
Resolve the 76 Hotspot finding(s) in Churn × Complexity Hotspots — start with types.py (3), CheckAnalysis.scala, Analyzer.scala. — One of this dimension's main actionable groups (76 warning-level).
Resolve the 1 Repeated repair finding(s) in Churn × Complexity Hotspots — start with TransformExpression.scala. — One of this dimension's main actionable groups (1 warning-level).
Detailed fixes: d15_recommendation.md · top locations in Appendix A, every location in findings.md.
Do you agree with this assessment?
D16 · Bus Factor9.1 / 10Exemplary✓ Tool-verified
What it measures: Whether knowledge is concentrated in too few people (the "bus factor").
Method: Living knowledge per author via time-decayed commit attribution (6-month half-life, focus weighting) across largest source files. Deterministic, avoids blame's mechanical-refactor false positives.
250 source file(s) have their living knowledge concentrated in one author (≥90% of recent, decayed contribution). The largest is core/src/main/scala/org/apache/spark/deploy/master/Master.scala. Counted over 2854 of the 4632 production source files in this repository: 1727 are under the ~2,400-byte size floor this dimension measures over, and the remaining 51 have no attributable history left to measure.
Off-boarding risk: anonymized user #1 · ×22
Further sole-owners (lower concentration)
✓ On the Gold path — maintain.
Detailed fixes: d16_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Acknowledged debt left in the code — TODOs, dead code, suppressed warnings.
Method: Roslyn syntactic debt markers (suppressions/TODO/FIXME/HACK/empty-catch/commented-code/Obsolete) plus SymbolFinder dead-code analysis; weighted-debt-per-KLoC density deducted 2.0x per unit. Deterministic, exhaustive.
1118 deducted task-comment markers across 1228043 LoC (0.1/KLoC) → score 9.8. Task comments only: this repository's language is read without a compiler, so D17's suppression, dead-code and commented-out-code arms did not run and this score counts fewer marker kinds than a .NET repository's would.
Resolve the 1075 TodoComment finding(s) in Explicit Debt — start with frame.py (20), DAGScheduler.scala (18), series.py (16). — One of this dimension's main actionable groups (1075 warning-level).
Resolve the 32 HackComment finding(s) in Explicit Debt — start with BlockInfoManagerSuite.scala (5), REDACTED (4), REDACTED (3). — One of this dimension's main actionable groups (32 warning-level).
Resolve the 10 FixmeComment finding(s) in Explicit Debt — start with test_stat.py (3), REDACTED (2), ThriftCLIServiceClient.java. — One of this dimension's main actionable groups (10 warning-level).
Enforce Explicit Debt in CI to reach Verified (currently Documented). — Hardens enforcement from Documented toward Prevented — provenance only; does not change the score.
Detailed fixes: d17_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether the project's documentation is clear, complete, and useful.
Method: Every README, docs/ document and root guide parsed by its markup and measured in Unicode letters. Published: a runnable or packaged .NET project whose README (or a document it links, or a root guide) shows no example, and a root README naming none of three or more projects. Overview, licence, boilerplate, wall-of-text, empty-section and XML-doc rows are measured and withheld (counted in gaps) after adjudication. Deterministic, exhaustive, advisory.
1 of 1 documentation facets pass in full. 369 document(s) read across 1 unit(s).
✓ On the Gold path — maintain.
Detailed fixes: d19_recommendation.md.
Do you agree with this assessment?
D20 · ADR Quality0.0 / 10Critical✓ Tool-verified
What it measures: Whether architecture decisions are recorded well (context, decision, consequences).
Method: Per-ADR judgment by language model at low temperature with two-pass stability; confidence is share of ADRs evaluated; enforcement-field presence detected deterministically. Advisory.
10 of 51 build units (Bundler, Maven, Python, npm) flagged as possibly oversized/incoherent.
Projects may be oversized for their cohesion
What to do
Resolve the 1 Projects may be oversized for their cohesion finding(s) in Project Cohesion. — One of this dimension's main actionable groups (1 recommendation-level).
Detailed fixes: d26_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether any secrets were ever committed — scanned across the full git history, not just now.
Method: REDACTED scan via TWO gitleaks detect passes in an isolated checkout — the full git history, then a second --no-git pass over the working tree as it stands — merged and de-duplicated by (rule, file, line); each match flagged High. Both invocations are recorded in the audit trail. Exhaustive; when the tool is absent, or when its output cannot be parsed into the expected shape, the dimension is WITHHELD as an explicit measurement gap on our side — unscored and excluded from the lens, never a hedged middling score.
2 finding(s): 0 critical, 2 high, 0 medium, 0 low. Remediation for historically-committed secrets is credential rotation — they remain in history regardless of later deletion.
REDACTED
REDACTED
What to do
Resolve the 2 REDACTED finding(s) in Secrets (history) — start with REDACTED (2). — One of this dimension's main actionable groups (2 issue-level).
Resolve the 1 Rotate the exposed credentials finding(s) in Secrets (history). — One of this dimension's main actionable groups (1 recommendation-level).
Detailed fixes: d28_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Real static-analysis (SAST) findings — likely security bugs in the code, any language.
Method: Polyglot static analysis via semgrep across the repo using the pinned, image-baked p/security-audit + p/owasp-top-ten rulesets (no scan-time registry fetch); severity rules (ERROR/WARNING/INFO) map to a full-band severity-weighted score. Exhaustive, deterministic; degrades on parse failure.
Coverage: semgrep pattern rules over all files — exhaustive for the rule set, blind to classes of bug without a rule (clean = no rule matched).
213 finding(s): 0 critical, 160 high, 40 medium, 13 low. 93 unpinned-GitHub-Actions row(s) are reported here but scored by D36 (supply-chain provenance), which measures that posture as `pinned_actions` — one pinning decision is charged once, not once per lens. semgrep hit a parse error in 72 file(s) — `R/run-tests.sh` (lines 20–75), `common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala`, `core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala`, `core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala`, `core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala`, … (+67 more) — so no absence of findings in the named regions is evidence of anything; rows reported elsewhere in those files are real. Fix the syntax error (or exclude the file deliberately) and re-scan to cover them. Separately, one or more rules could not re-parse an embedded snippet in 6 file(s) (e.g. a workflow `run:` block read as shell). Those files WERE scanned and their other rows are unaffected; only those rules' view of those snippets is missing.
REDACTED
REDACTED
REDACTED
REDACTED
REDACTED
+ 17 more group(s) — more in Appendix A; the complete list is findings.md.
What to do
Resolve the 41 REDACTED finding(s) in Static Analysis (SAST) — start with REDACTED (11), REDACTED (6), REDACTED (6). — One of this dimension's main actionable groups (41 issue-level).
Resolve the 6 REDACTED finding(s) charged to Static Analysis (SAST) — the other 93 are reported here at file:line but scored by D36 (supply-chain provenance), which charges them once. — One of this dimension's main actionable groups (99 issue-level, 6 of them charged here).
Resolve the 9 REDACTED finding(s) in Static Analysis (SAST) — start with REDACTED (5), REDACTED (2), REDACTED (2). — One of this dimension's main actionable groups (9 issue-level).
Detailed fixes: d29_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether any dependency has a known published vulnerability (CVE), direct or transitive, in ANY ecosystem the repository declares — Dart pub, Elixir and Erlang via Hex, Go modules, Java and Kotlin via Maven/Gradle, JavaScript/npm, .NET/NuGet, PHP/Composer, Python/PyPI, RubyGems, Rust/Cargo and Swift.
Method: Dependency-CVE scan across every ecosystem the repository declares, scored ONCE. Three sources are unioned and deduplicated by advisory identity (rule id + alias closure, CVE<->GHSA) scoped to package+version, keeping the worst severity: `osv-scanner --recursive` over osv.dev for Dart pub, Elixir/Hex (and Erlang, whose `rebar.lock` syft first converts to a CycloneDX SBOM the scanner reads, with rows attributed back to the lock), Go, Java and Kotlin via Maven/Gradle (and Scala, whose sbt build's pinned direct declarations are written into a CycloneDX SBOM the scanner reads, with rows attributed back to the build file), npm, PHP/Composer, Python/PyPI, RubyGems, Rust/Cargo and Swift; `trivy fs --scanners vuln` for npm lockfiles; and `dotnet list package --vulnerable --include-transitive` for NuGet (with per-advisory collapse of the project x target-framework fan-out), plus a DECLARED-dependency arm that resolves a published gem's gemspec against rubygems.org where no Gemfile.lock is committed. `SeverityScore(c,h,m,l, normalizer 8.0)`. NotApplicable only when NO ecosystem is readable; if any applicable ecosystem could not be scanned the findings are REPORTED and the score is withheld. Supersedes the npm and OSV arms, retired 2026-09-05.
7 finding(s): 2 critical, 1 high, 4 medium, 0 low. npm was read by osv-scanner from this repository's lockfiles; its second reader, trivy, did not complete (the scanner produced no output at all, so no dependency was actually scanned). Over the 463 advisory/package pairs measured with both, osv-scanner reported every vulnerable npm install trivy did, so the npm reading stands.
REDACTED
REDACTED
REDACTED
What to do
Resolve the 4 Medium CVE finding(s) in Dependency Vulnerabilities — start with REDACTED (3), REDACTED. — One of this dimension's main actionable groups (4 warning-level).
Resolve the 2 Critical CVE finding(s) in Dependency Vulnerabilities — start with REDACTED, REDACTED. — One of this dimension's main actionable groups (2 issue-level).
Resolve the 1 High CVE finding(s) in Dependency Vulnerabilities — start with REDACTED. — One of this dimension's main actionable groups (1 issue-level).
Detailed fixes: d30_recommendation.md · top locations in Appendix A, every location in findings.md.
Resolve the 1 High IaC finding(s) in IaC & Container Security — start with REDACTED. — One of this dimension's main actionable groups (1 issue-level).
Resolve the 5 Medium IaC finding(s) in IaC & Container Security — start with REDACTED (4), REDACTED. — One of this dimension's main actionable groups (5 warning-level).
Detailed fixes: d31_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether anyone still has living knowledge of each file, or it has been orphaned — last understood long ago by someone now gone quiet. The sibling of the bus factor: D16 asks who owns it, D34 asks whether anyone still knows it.
Method: File orphaning as total living-knowledge decay below one focused-commit's worth within a year, computed per-file from the D16 decay model. Exhaustive, deterministic over fixed history.
576 of 2905 significant source file(s) are orphaned — their living knowledge has decayed to nothing, so no one currently understands them. The largest is python/pyspark/mllib/linalg/distributed.py. Counted over 2905 of the 4632 production source files in this repository: the rest are under the ~2,400-byte size floor this dimension measures over.
Resolve the 38 Orphaned knowledge finding(s) in Knowledge Freshness — start with KeyValueGroupedDataset.scala (3), distributed.py, DStream.scala. — One of this dimension's main actionable groups (38 issue-level).
Resolve the 538 Orphaned file finding(s) in Knowledge Freshness — start with MergeIntoWriter.scala (3), fpm.py (2), SQLContext.scala (2). — One of this dimension's main actionable groups (538 recommendation-level).
Resolve the 1 Further orphaned files (smaller) finding(s) in Knowledge Freshness. — One of this dimension's main actionable groups (1 recommendation-level).
Detailed fixes: d34_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether files that change together actually belong together — pairs that repeatedly co-change in git history despite having no explicit code dependency, surfacing the hidden/logical coupling (and boundaries in the wrong place) a static scan can't see.
Method: Pairwise co-occurrence over the per-commit file sets in git history (production source only — tests and generated dropped): Degree-of-Coupling = shared ÷ min individual revisions, reported above noise floors (each file ≥10 revisions, ≥5 shared commits, ≥50% strength); sweeping commits excluded. Deterministic over fixed history.
Coverage: Population: PRODUCTION source files only — test and generated files are dropped before pairing, so a class co-changing with its own test (trivially ~100%) can't drown the real production↔production coupling. Pairs ranked by Degree-of-Coupling. A non-source file is never a coupling PARTICIPANT either: documentation, schemas, config and data files are dropped with the rest, so a code↔docs pair — a command and the reference page that restates it — is not reported however strongly the two co-change; nor is coupling that runs THROUGH a build step or config file.
Resolve the 1 Boundary-crossing change coupling finding(s) in Change Coupling — start with __init__.py. — One of this dimension's main actionable groups (1 issue-level).
Resolve the 4 Change coupling finding(s) in Change Coupling — start with RuleIdCollection.scala, DataSourceV2ScanExecBase.scala, api.py. — One of this dimension's main actionable groups (4 warning-level).
Resolve the 1 Change coupling clique finding(s) in Change Coupling — start with CSVFileFormat.scala. — One of this dimension's main actionable groups (1 warning-level).
Detailed fixes: d35_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether the build pipeline provides supply-chain integrity — generated provenance/attestation, signed artifacts (cosign/sigstore), an SBOM, and pinned build actions. Presence of the configuration, not a runtime guarantee.
Method: Supply-chain provenance/signing read deterministically from CI/build config (.github/workflows, .gitlab-ci.yml, azure-pipelines, Jenkinsfile, .circleci) + the release surface: four signals — generated provenance/attestation (SLSA/in-toto/actions-attest), artifact signing (cosign/sigstore/gitsign), an SBOM (syft/sbom-action/*.spdx.json/*.cdx.json), and SHA-pinned build actions — scored 10·present/denom. NotApplicable without a build pipeline. Detects configuration presence, not runtime enforcement.
+ 3 more group(s) — more in Appendix A; the complete list is findings.md.
What to do
Resolve the 8 REDACTED finding(s) in Supply-chain Provenance & Signing — start with REDACTED, REDACTED, REDACTED. — One of this dimension's main actionable groups (8 warning-level).
Resolve the 3 REDACTED finding(s) in Supply-chain Provenance & Signing — start with REDACTED, REDACTED, REDACTED. — One of this dimension's main actionable groups (3 warning-level).
Resolve the 1 REDACTED finding(s) in Supply-chain Provenance & Signing. — One of this dimension's main actionable groups (1 warning-level).
Detailed fixes: d36_recommendation.md · top locations in Appendix A, every location in findings.md.
What it measures: Whether the repository publishes a coordinated-vulnerability-disclosure policy (SECURITY.md or security.txt) with a reporting contact, so finders know how to report a vulnerability. Presence of a policy file with a contact, not whether the policy is adequate or honoured.
Method: Vulnerability-disclosure policy read deterministically from the repo: a SECURITY.md (root/.github/docs) or .well-known/security.txt / security.txt, regex-checked for a reporting contact (email / URL / mailto). Present + contact → 10; present without a contact → 4; NotApplicable when no policy file exists (it may live off-repo). Detects the policy file's presence + contact, not its adequacy.
What it measures: Whether any dependency the repository declares is published as MALICIOUS rather than merely vulnerable — a package that is an attacker's work, in any ecosystem osv-scanner reads. Scored apart from D30 because the answer is binary: there is no safe version to upgrade to, and the fix is to remove the package and rotate every credential it could have read.
Method: The same dependency scan D30 reads, partitioned on the scanner's own classification rather than rescanned: a row is MALICIOUS when its id is in the `MAL-` space (the ossf/malicious-packages feed) OR its `database_specific.cwe_ids` carries `CWE-506` ("Embedded Malicious Code"). Both channels are structural; the summary text is deliberately NOT read, because a malicious-package record whose summary says only "Critical severity vulnerability" is a real shape ([GHSA redacted]) and a text matcher misses it. Scored BINARY: any surviving row is 0, whatever its severity and however many CVEs sit beside it — a hostile dependency is not a quantity. Applicability and degradation are D30's: NotApplicable only when no ecosystem is readable, and an unscannable ecosystem degrades rather than reading clean. SCORED, not informational.
No dependency in any ecosystem this repository declares is published as malicious. npm was read by osv-scanner from this repository's lockfiles; its second reader, trivy, did not complete (the scanner produced no output at all, so no dependency was actually scanned). Over the 463 advisory/package pairs measured with both, osv-scanner reported every vulnerable npm install trivy did, so the npm reading stands.
What it measures: Whether anyone still ships security patches for the platform this repository RUNS ON — the runtime it pins and the framework majors its own constraints hold it to. Separate from D12 because the question differs: a current Django on an end-of-life Python is perfectly up to date and completely unsupported, and the fix is a migration rather than a version bump. What the repository says it merely SUPPORTS is never charged.
Method: End-of-life PLATFORM read from the repository's own declarations and graded against a FROZEN, dated table of vendor support dates — no network, no feed, no API, so this dimension answers identically inside a closed scan fence. Two subjects: a RUNTIME the project pins (a single or all-end-of-life TargetFramework, a .nvmrc or .python-version, a requires-python CAP) and a FRAMEWORK major a dependency constraint cannot move off (a caret, tilde or exact version; `vue@^2.7.16` pins Vue 2). A FLOOR is deliberately never charged — `requires-python = ">=3.8"` states what a package SUPPORTS, not what it runs on — and a multi-target project is charged only when EVERY target is out of support. Runtime 4.0/product capped 8.0, framework 1.5 capped 4.5. The table is safe to freeze because a statement about support that ended in the past cannot become false: it loses recall as it ages, never precision, and a test asserts every entry predates the freeze date. Disjoint from D31 (a container image's OS layer) and D29 (the toolchain a CI workflow installs). Abstains when the repository declares no platform this pass reads — never scores it clean.
0 end-of-life runtime(s) and 0 end-of-life framework(s), read from 1 platform declaration(s) and 77 dependency declaration(s). This dimension reads what the repository says about ITSELF — a pinned target framework, a version file, a capped requires-python, a Rust toolchain pin, a framework major a constraint cannot move off. A FLOOR is deliberately never charged: `requires-python = ">=3.8"` states what the package SUPPORTS, not what it runs on, and a well-maintained library declares exactly that while running its own CI on a current release. The end-of-life facts are FROZEN and dated, so this dimension needs no network and answers identically inside a closed scan fence; as the table ages it loses recall and never precision, because a statement about support that ended in the past cannot become false. The OS layer of a container image is D31's question and the toolchain a CI workflow installs is D29's; this row is neither.
✓ On the Gold path — maintain.
Detailed fixes: d44_recommendation.md.
Do you agree with this assessment?
Frontend & cross-cutting dimensions
R = React/JS · M = Maturity · P = Readiness.
AC2 · Forms & labels4.9 / 10Weak✓ Tool-verified
Other · Accessibility — Whether form controls have a programmatic label (an associated label, aria-label or aria-labelledby), buttons have text, links have an accessible name, a click handler on a plain element names the control it declares, fieldsets have a non-empty legend, known UI-library field components carry a label prop, and a placeholder isn't used as the only label. Static markup readiness, not a WCAG conformance claim.
Method: Static markup-model scan: inputs/selects/textareas checked for an associated label[for]/wrapping label/aria-label/aria-labelledby (per document), buttons for accessible text, fieldsets for a legend; placeholder-only labelling flagged. Deterministic, hard fact per control.
Coverage: Population: form controls, buttons, links, fieldsets and known UI-library field components in the PARSED MARKUP files only (.html/.htm/.cshtml/.razor/.vue/.svelte/.jsx/.tsx); components, hidden subtrees and spread/dynamic-attribute elements are skipped, so a control whose label arrives through a spread or a runtime expression is deliberately not judged. Markup built in script — tagged-template (html`…`) UIs and hyperscript DOM factories — is not read at all.
A button with no text and no aria-label has no accessible name. Add visible text or an aria-label (an icon-only button still needs one). — core/src/main/resources/org/apache/spark/ui/static/environmentpage.js:206
This control has no associated label. Add a <label for> / wrapping <label> / aria-label / aria-labelledby so assistive tech can name it. — sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:37
This control has only a placeholder — a placeholder is not a label (it vanishes on input and many AT ignore it). Add a <label for>, a wrapping <label>, or aria-label. — REDACTED:468
What to do
Give every control a programmatic label (a <label for> / wrapping <label> / aria-label) and every button text — a placeholder is not a label.
Do you agree with this assessment?
AC3 · Page structure4.6 / 10Weak✓ Tool-verified
Other · Accessibility — Whether pages declare a language (well-formed BCP-47) and a non-empty title, expose exactly one main landmark and a sane heading order with non-empty headings, keep zoom enabled, title their iframes, give data tables header cells, and avoid meta-refresh. Static markup readiness, not a WCAG conformance claim.
Method: Static markup-model scan: html lang, document <title>, a main landmark and heading order on full documents only, plus zoom-disabling viewports, untitled iframes and meta-refresh anywhere. Deterministic, per structural checkpoint.
Coverage: Population: the PARSED MARKUP documents (.html/.htm/.cshtml/.razor/.vue/.svelte/.jsx/.tsx). The page-level checks — lang, title, single main landmark — fire ONCE PER FULL DOCUMENT (an <html> root) and never on a partial or component fragment, so a repo of fragments is assessed only on the per-element checks (heading order, table headers, iframe titles, meta-refresh, zoom). Markup built in script — tagged-template (html`…`) UIs and hyperscript DOM factories — is not read at all.
The page declares no language, so assistive tech can't pick the right pronunciation. Add lang (e.g. lang="en"). — licenses-binary/LICENSE-javassist.html:1
No <main> (or role="main") means no "skip to content" target and a weaker landmark map. Wrap the primary content in <main>. — licenses-binary/LICENSE-javassist.html:1
What to do
Declare <html lang>, a document <title> and a <main> landmark, keep headings in order, leave zoom enabled, title iframes and drop meta-refresh.
Other · Accessibility — Whether interactive behaviour is keyboard-reachable — no click handler on a non-interactive element lacking a role, tabindex and key handler, no element the repo's own CSS styles `cursor: pointer` without giving it any of the three, no unfocusable element whose only binding is a mouse enter/leave pair or a double-click, no positive tabindex, no href-less anchor, no placeholder-href (#/javascript) link acting as a button. Static markup readiness, not a WCAG conformance claim.
Method: Static markup-model scan: click handlers on non-interactive elements lacking role+tabindex+key handler, positive tabindex values, and href-less anchors. Components skipped, spreads suppressed. Deterministic, hard fact per element.
Coverage: Population: elements that pose a keyboard-semantics question, in the PARSED MARKUP files only (.html/.htm/.cshtml/.razor/.vue/.svelte/.jsx/.tsx) — a non-natively-interactive element carrying a click handler, a double-click or hover enter/leave binding, a pointer-only gesture on a tabindex="0" element, or `cursor: pointer` from the repo's own CSS; an href-less or placeholder-href (#, javascript:) anchor; and any element with a positive tabindex. Natively interactive elements used correctly (<button>, <a href>, form controls) are NOT in the population — there is nothing to judge — so a page of nothing but correct controls gives AC4 nothing to measure. Keyboard reachability is judged from the markup, never from a rendered page. ★ An interactive element declared in a tagged-template (html`…`) or hyperscript frontend is NOT in this population — no producer reads either — so an empty population is reported as an analyzer gap, never as "this repository has no interactive elements".
An <a> with no href, role or tabindex isn't focusable or keyboard-activatable. Give it a real href, or use a <button> for an action. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:899
What to do
Make custom controls keyboard-operable (role + tabindex + key handler), drop positive tabindex, and give anchors a real href.
Other · Accessibility — Whether ARIA is used correctly — valid non-abstract roles, the ARIA state a role requires, valid (non-misspelled) aria-* attribute names, in-enum values for token-typed aria-* attributes, and no aria-hidden on (or wrapping) a focusable element. Static markup readiness, not a WCAG conformance claim.
Method: Static markup-model scan: role values checked against the WAI-ARIA role set (abstract/invalid flagged), required ARIA state for a role, and aria-hidden on a focusable element. Deterministic, role/attribute level.
Coverage: Population: elements in the PARSED MARKUP files (.html/.htm/.cshtml/.razor/.vue/.svelte/.jsx/.tsx) that carry a role or an aria-* attribute; roles and token values are checked against the ARIA enums exhaustively within that set. An expression-valued (dynamic) role or aria-* value is skipped rather than guessed, and markup built in script — tagged-template (html`…`) UIs and hyperscript DOM factories — is not read at all.
Other · Accessibility — Whether focus outlines aren't removed without a replacement, motion respects prefers-reduced-motion, and literal CSS colour pairs meet contrast — PARTIAL: inline styles, in-repo <style> blocks, in-repo .css files, var() tokens, Tailwind neutral utilities and CSS-in-JS literals are read (hex/rgb/hsl/named), never computed/runtime/external-CDN colour. Static markup readiness, not a WCAG conformance claim.
Method: Static markup/CSS scan: inline outline:none/0, literal inline colour/background contrast against the 4.5:1 AA floor, and <style>-block animation without a prefers-reduced-motion guard. Deterministic but PARTIAL — only inline styles and in-repo CSS literals are visible.
Coverage: Population: styled elements in the PARSED MARKUP files (.html/.htm/.cshtml/.razor/.vue/.svelte/.jsx/.tsx), plus in-repo <style> blocks, in-repo .css files and CSS-in-JS literals. Colour contrast is computed from LITERAL colour pairs only (hex/rgb/hsl/named, including var() tokens and Tailwind neutral utilities) — computed, runtime-themed and external-CDN colour is never resolved, so this is a partial read of contrast by construction. Markup built in script — tagged-template (html`…`) UIs and hyperscript DOM factories — is not read at all.
`you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ .badge` sets color: #ffffff on background-color: #999999 — 2.8:1, below the 4.5:1 WCAG AA minimum for normal text. Darken or lighten one of them. — docs/css/api-javadocs.css:20
`.nav-trigger+label` moves (`transition: left 0.4s`) and nothing reads prefers-reduced-motion for it — no @media (prefers-reduced-motion) rule for this element, no reset this project loads — so a motion-sensitive user cannot turn it off (WCAG 2.3.3). Declare it under @media (prefers-reduced-motion: no-preference), or neutralise it under reduce. — docs/css/custom.css:907
`.stage-details` moves (`transition: max-height 0.25s ease-out, padding 0.25s ease-out`) and nothing reads prefers-reduced-motion for it — no @media (prefers-reduced-motion) rule for this element, no reset this project loads — so a motion-sensitive user cannot turn it off (WCAG 2.3.3). Declare it under @media (prefers-reduced-motion: no-preference), or neutralise it under reduce. — core/src/main/resources/org/apache/spark/ui/static/webui.css:182
`.stacktrace-details` moves (`transition: max-height 0.25s ease-out, padding 0.25s ease-out`) and nothing reads prefers-reduced-motion for it — no @media (prefers-reduced-motion) rule for this element, no reset this project loads — so a motion-sensitive user cannot turn it off (WCAG 2.3.3). Declare it under @media (prefers-reduced-motion: no-preference), or neutralise it under reduce. — core/src/main/resources/org/apache/spark/ui/static/webui.css:225
`.arrow-open` moves (`transition: transform 0.2s ease`) and nothing reads prefers-reduced-motion for it — no @media (prefers-reduced-motion) rule for this element, no reset this project loads — so a motion-sensitive user cannot turn it off (WCAG 2.3.3). Declare it under @media (prefers-reduced-motion: no-preference), or neutralise it under reduce. — core/src/main/resources/org/apache/spark/ui/static/webui.css:255
What to do
Keep a visible focus style (don't remove the outline without a replacement), guard animation with prefers-reduced-motion, and raise low-contrast colour pairs to at least 4.5:1.
Do you agree with this assessment?
AC7 · A11y enforcement4.0 / 10Weak✓ Tool-verified
Other · Accessibility — Whether accessibility is ENFORCED in the toolchain — an accessibility checker configured over the markup (an a11y lint rule set, e.g. eslint-plugin-jsx-a11y or vuejs-accessibility where the project lints JavaScript) and an automated accessibility assertion wired into tests or CI (axe/pa11y/Lighthouse or an equivalent) — on the Documented→Verified→Prevented ladder.
Method: Repo config/CI scan: an accessibility checker configured over the markup (an a11y lint rule set such as eslint-plugin-jsx-a11y / vuejs-accessibility where JavaScript is linted) and an automated accessibility assertion in tests or CI (axe/pa11y/Lighthouse or equivalent), graded on the Documented→Verified→Prevented rungs. Deterministic, presence/rung detection.
Coverage: Population: the repository's own tooling configuration — lint config, test and CI files — NOT the markup. It is read for a configured accessibility checker and an automated accessibility assertion (axe/pa11y/Lighthouse, or a native-toolkit equivalent), and it credits an INVOCATION, never a mention: a licence filename, an import comment or a doc reference earns no rung. Enforcement configured entirely outside the repository leaves no evidence here and cannot be credited.
No accessibility enforcement found — no a11y linter (an accessibility check that can read your UI — no component framework was detected and your pages are rendered by server-side templates, which neither the JSX/Vue ESLint plugins nor the HTML-template linters can parse; run axe/pa11y over the rendered pages, or assert the accessibility invariants over that rendered HTML in the test suite you already have) and no axe/pa11y/Lighthouse in tests or CI. Start by running that check over your rendered pages in CI. What was searched, so you can tell an absence from a miss: the 9 markup file(s) this pass actually assessed, the linter configuration checked in beside them, and this repository's test and CI files — matched by name against the accessibility checkers this dimension carries. An audit run outside the repository, a hosted scanner, or a check whose name is not one of those, is not seen here.
What to do
Enforce accessibility in the toolchain: add an accessibility check that can read your UI — no component framework was detected and your pages are rendered by server-side templates, which neither the JSX/Vue ESLint plugins nor the HTML-template linters can parse; run axe/pa11y over the rendered pages, or assert the accessibility invariants over that rendered HTML in the test suite you already have, then assert with jest-axe in tests, then gate axe/pa11y/Lighthouse in CI.
Other · Architecture — How the codebase splits by code ROLE — domain, application, infrastructure, test, generated. The significance map behind the knowledge/coupling weighting, and a DDD signal in its own right: a thin domain core under fat infrastructure is the anemic-domain smell, quantified. How each file's role is decided, because the split is only as good as that: a generated name or a build-output tree makes it Generated, a test project makes it Test, and otherwise the file's NAMESPACE and PATH words are matched against fixed vocabularies in a fixed ORDER — domain, then infrastructure, then application — so a file whose words hit two layers is counted under the earlier one. A production file matching none of them counts as application, so that share reads 'application or unclassified' rather than a measured application layer. Roles come from naming convention, never from what the code does. On this repository the split was taken from the source tree on disk rather than from a loaded .NET workspace, so a file's role is decided by its PATH segments alone — no declared namespace was available to add to the evidence — and generated output is excluded from the census entirely rather than counted as a generated share.
Method: Roslyn line-count by code ROLE: every source file classified Domain/Application/Infrastructure/Test/Generated by namespace + path convention (the shared CodeRoleClassifier), then significant lines summed per role. Deterministic; the advisory score is the business-logic (domain+application) share of production code.
Coverage: Population: ALL source files, each bucketed into ONE of five roles (Domain/Application/Infrastructure/Test/Generated) by namespace + path convention — a file whose layer isn't named in the convention falls to Application (the neutral default), and the split is line-count, not semantic depth or business value.
What to do
The domain core is a small share of production code, but most of the rest matched no layer vocabulary at all — so this is not yet an anemic-domain finding. The namespace/path convention could not place that code, which makes the composition above a statement about the naming, not about the design. Name the layers (or check that the repository's conventions differ from the ones this check knows) before reading a thin domain into it.
Other · Architecture — Whether the project-reference graph is acyclic (cycles block independent build/deploy and signal eroding boundaries).
Method: Project reference cycles via elementary-DFS over real .csproj references, using the engine shared with D5/D7; cyclic versus acyclic. Exhaustive, deterministic.
Other · Architecture — Whether dependencies point inward (Domain ← Application ← Infrastructure/Web) — the clean-architecture dependency rule, checked across the project graph.
Method: Layer violations by name-segment inference (Domain/Core to Application to Infrastructure/Web) over the project-reference graph. Exhaustive over all projects, deterministic.
Other · Architecture — Whether interfaces stay focused rather than fat — the Interface-Segregation principle (SOLID 'I').
Method: Roslyn scan: public interface declared-member counts (accessors fold into their property/event); fat-interface threshold (over 15 declared members) flagged per type. Each finding also reports the distinct-OPERATION count — members counted by name, so an overload group counts once — which decides whether it states the caller-side ISP harm or the implementer-side burden of an overload set. TypeScript: exported interfaces that declare behaviour, counted over their REQUIRED operations (method signatures and function-typed properties with no `?`, once per signature) — data-shape interfaces and ambient .d.ts declarations are outside the population; JavaScript declares no interfaces and is not applicable. Deterministic, type-level.
`HiveSession` declares 26 abstract methods: `cancelDelegationToken`, `cancelOperation`, `close`, `closeExpiredOperations`, `closeOperation`, `executeStatement`, `executeStatement`, `executeStatementAsync`, `executeStatementAsync`, `fetchResults`, `getCatalogs`, `getColumns`, `getCrossReference`, `getDelegationToken`, `getFunctions`, `getInfo`, `getMetaStoreClient`, `getNoOperationTime`, `getPrimaryKeys`, `getResultSetMetadata`, `getSchemas`, `getTableTypes`, `getTables`, `getTypeInfo`, `open`, `renewDelegationToken`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. Those 26 declarations carry 24 distinct name(s), so part of the width is overloading rather than separate responsibilities — but 24 operations clears the >15 bar on its own, so the row is about responsibilities, not signatures. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces. — sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSession.java:29
`HiveSessionBase` declares 17 abstract methods: `getHiveConf`, `getIpAddress`, `getLastAccessTime`, `getOperationLogSessionDir`, `getPassword`, `getProtocolVersion`, `getSessionHandle`, `getSessionManager`, `getSessionState`, `getUserName`, `getUsername`, `isOperationLogEnabled`, `setIpAddress`, `setOperationLogSessionDir`, `setOperationManager`, `setSessionManager`, `setUserName`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces. — sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionBase.java:32
`ICLIService` declares 27 abstract methods: `cancelDelegationToken`, `cancelOperation`, `closeOperation`, `closeSession`, `executeStatement`, `executeStatement`, `executeStatementAsync`, `executeStatementAsync`, `fetchResults`, `fetchResults`, `getCatalogs`, `getColumns`, `getCrossReference`, `getDelegationToken`, `getFunctions`, `getInfo`, `getOperationStatus`, `getPrimaryKeys`, `getQueryId`, `getResultSetMetadata`, `getSchemas`, `getTableTypes`, `getTables`, `getTypeInfo`, `openSession`, `openSessionWithImpersonation`, `renewDelegationToken`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. Those 27 declarations carry 24 distinct name(s), so part of the width is overloading rather than separate responsibilities — but 24 operations clears the >15 bar on its own, so the row is about responsibilities, not signatures. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces. — sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/ICLIService.java:27
`VectorizedValuesReader` declares 31 abstract methods: `readBinary`, `readBinary`, `readBoolean`, `readBooleans`, `readByte`, `readBytes`, `readDouble`, `readDoubles`, `readFloat`, `readFloats`, `readGeography`, `readGeometry`, `readInteger`, `readIntegers`, `readIntegersWithRebase`, `readLong`, `readLongs`, `readLongsWithRebase`, `readShort`, `readShorts`, `readUnsignedIntegers`, `readUnsignedLongs`, `skipBinary`, `skipBooleans`, `skipBytes`, `skipDoubles`, `skipFixedLenByteArray`, `skipFloats`, `skipIntegers`, `skipLongs`, `skipShorts`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. Those 31 declarations carry 30 distinct name(s), so part of the width is overloading rather than separate responsibilities — but 30 operations clears the >15 bar on its own, so the row is about responsibilities, not signatures. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces. — sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java:30
`SpecializedGetters` declares 20 abstract methods: `get`, `getArray`, `getBinary`, `getBinaryView`, `getBoolean`, `getByte`, `getDecimal`, `getDouble`, `getFloat`, `getInt`, `getInterval`, `getLong`, `getMap`, `getShort`, `getStruct`, `getTimestampLTZNanos`, `getTimestampNTZNanos`, `getUTF8String`, `getVariant`, `isNullAt`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces. — sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/SpecializedGetters.java:31
`ExternalCatalog` declares 40 abstract members: `createDatabase`, `dropDatabase`, `alterDatabase`, `getDatabase`, `databaseExists`, `listDatabases`, `listDatabases`, `setCurrentDatabase`, `createTable`, `dropTable`, `renameTable`, `alterTable`, `alterTableDataSchema`, `alterTableSchema`, `alterTableStats`, `getTable`, `getTablesByName`, `tableExists`, `listTables`, `listTables`, `listViews`, `loadTable`, `loadPartition`, `loadDynamicPartitions`, `createPartitions`, `dropPartitions`, `renamePartitions`, `alterPartitions`, `getPartition`, `getPartitionOption`, `listPartitionNames`, `listPartitions`, `listPartitionsByFilter`, `createFunction`, `dropFunction`, `alterFunction`, `renameFunction`, `getFunction`, `functionExists`, `listFunctions`. Counted over its ABSTRACT members — a `def` with no body or a `val`/`var` with no initializer, once per signature; concrete members, private and protected members, and members of a companion `object` are not counted, because no implementer is forced to write them. Those 40 declarations carry 38 distinct name(s), so part of the width is overloading rather than separate responsibilities — but 38 operations clears the >15 bar on its own, so the row is about responsibilities, not signatures. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces. — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala:34
`LogicalPlanVisitor[T]` declares 24 abstract members: `default`, `visitAggregate`, `visitDistinct`, `visitExcept`, `visitExpand`, `visitFilter`, `visitGenerate`, `visitGlobalLimit`, `visitOffset`, `visitIntersect`, `visitJoin`, `visitLocalLimit`, `visitPivot`, `visitProject`, `visitRepartition`, `visitRepartitionByExpr`, `visitRebalancePartitions`, `visitSample`, `visitScriptTransform`, `visitUnion`, `visitWindow`, `visitTail`, `visitSort`, `visitWithCTE`. Counted over its ABSTRACT members — a `def` with no body or a `val`/`var` with no initializer, once per signature; concrete members, private and protected members, and members of a companion `object` are not counted, because no implementer is forced to write them. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces. — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlanVisitor.scala:23
What to do
Split fat interfaces into focused role-interfaces so clients depend only on what they use.
Other · Code Health — Unreviewed-generation residue in this repository's Scala: methods whose whole body is still `???` or a "not implemented" throw where nothing overrides them, and placeholder text (lorem ipsum, changeme/replace_me…) left in strings of non-test, non-generated code. Scored as a quality signature, never as a claim about authorship.
Method: Roslyn syntax scan: NotImplementedException throws and placeholder string literals in non-test, non-generated shipped code. Deterministic, code-shape signature.
`invokeAll` is a shipped member whose whole body is still a "not implemented" error, and nothing in the repository overrides it — generated scaffolding that was never completed. Implement it or remove the dead surface. (×2) — core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:95, core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:99
`invokeAny` is a shipped member whose whole body is still a "not implemented" error, and nothing in the repository overrides it — generated scaffolding that was never completed. Implement it or remove the dead surface. (×2) — core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:104, core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:107
What to do
Finish or delete the members whose body is still `???` or a "not implemented" throw where nothing overrides them, and replace placeholder text before shipping.
Other · Code Health — Unfinished work detected by code SHAPE, not keywords, in this repository's Scala: methods whose body is `???` or only throws a "not implemented" error, methods that take inputs and return a constant or do nothing, dead `if (false)` branches, and skeleton classes most of whose methods are holes.
Method: Roslyn syntax scan: incompleteness by code shape (constant-returning methods, async-never-await, #if false branches, guards that return what the code already falls through to, tests an earlier guard already decided, comparisons against NaN, skeleton types), not keyword-gated. Deterministic, code-shape heuristic.
`invokeAll` throws an exception whose message says it isn't finished — a not-implemented stub wearing an ordinary exception. Implement it or remove it. (×2) — core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:95, core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:99
`invokeAny` throws an exception whose message says it isn't finished — a not-implemented stub wearing an ordinary exception. Implement it or remove it. (×2) — core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:104, core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:107
`validateLiteral` takes parameters but its body is empty — it accepts inputs and does nothing. Either implement it or remove it. — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ExpressionResolutionValidator.scala:136
A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something. (×20) — common/unsafe/src/test/scala/org/apache/spark/unsafe/types/UTF8StringPropertyCheckSuite.scala:228, connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala:2910, core/src/test/scala/org/apache/spark/DriverSuite.scala:33, …
A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead. (×14) — connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtilsPythonHelper.scala:32, connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentials.scala:106, core/src/main/scala/org/apache/spark/InternalAccumulator.scala:49, …
`apply` has an empty body — confirm it's an intentional no-op and not an unfinished method. — mllib/src/main/scala/org/apache/spark/sql/ml/InternalFunctionRegistration.scala:38
What to do
Finish or delete the unfinished stubs (functions that throw "not implemented", or whose body is empty or a bare constant) — they are dead surface that looks live.
Clear the softer debt: re-enable or delete `ignore`d and `pending` tests, decide whether each empty method or UnsupportedOperationException is deliberate (and say so in a comment), and replace a bare `// scalastyle:off` or `@nowarn` with targeted, justified suppressions.
Maturity · Maturity — Whether the repo and its projects have a README, and whether it's substantive and current.
Method: Filesystem scan: README presence, word count, and headings for depth; git history for staleness. Exhaustive across root and project dirs, deterministic.
What to do
Add an 'Architecture' / 'How it works' section to the root README — the high-level shape.
Add a README to the 42 of 55 project(s) that lack one — worth up to 1.5 pts.
Maturity · Maturity — Whether key decisions (ADRs) and the high-level shape (C4/diagrams) are written down.
Method: Filesystem scan: ADR folder/naming conventions or content, plus Mermaid/PlantUML/C4/architecture.md discovery. Exhaustive, deterministic.
No Architecture Decision Records found — no conventional ADR directory, no numbered `NNNN-title` documents in any markup this check reads, and nothing ADR-shaped by content. Design rationale recorded elsewhere (a design-notes tree, a mailing list, pull-request discussion) is not visible to this check and is not re-findable per decision, so a future maintainer cannot ask why one choice was made and get an answer.
What to do
Record your significant design decisions. Write one short document per decision, giving its date, the situation, the decision and its consequences, and keep them together with your other design documents. A docs/adr folder with each file named by number and title is the easiest place to find them.
Readiness · Readiness — Whether an automated pipeline builds and tests every change.
Method: Filesystem scan: CI workflow files (.github/workflows, .gitlab-ci.yml, etc.) for build and test stages. Exhaustive, deterministic.
Do you agree with this assessment?
P10 · Library API & versioning10.0 / 10Exemplary○ Nothing flagged
Readiness · Readiness — For a library: a deliberate (small) public API surface and explicit semantic versioning so consumers can depend on it safely.
Method: Roslyn scan: public API surface area and semantic-versioning markers (SemVer attributes, changelog entries) for libraries; off .NET, a library is the ecosystem's publication act (an npm package that is not private and names an entry point, a PyPI distribution with a build system, a Rust library crate, a Maven/Gradle module that publishes, a Go module with no package main, a gemspec, a Composer library, a SwiftPM library product, a pub.dev or Hex package), its surface is the share of types the language model records as public (Rust, Swift, Java, Kotlin, Go, Dart; not measured where the model records no type visibility or, as in TypeScript, only module-level export), and its version is read from the manifest, a semver CHANGELOG, release tooling or semver git tags. Exhaustive, deterministic.
Do you agree with this assessment?
P2 · Observability8.4 / 10Strong✓ Tool-verified
Readiness · Readiness — Whether the code is diagnosable in production — structured logging, tracing/metrics, health checks.
Only 23/29 runnable modules use logging (modules with no entry point or server are excluded — they are libraries a runnable module hosts). Silent: `connector/credential-aws-integration-tests`, `dev`, `repl`, `sql/connect/client/jvm`, `sql/connect/server` and 1 more.
What to do
Extend structured logging to the remaining runnable modules so everything you run is diagnosable in production.
Add a health-check endpoint (Spring Boot Actuator's /actuator/health, or a /health route) so orchestrators and load balancers can probe liveness/readiness.
Readiness · Readiness — Whether SAST, secret/dependency scanning and performance benchmarking are wired in (presence, not runtime).
Method: Filesystem scan: SAST configuration, dependency-update automation, secret scanning, and a benchmark harness or benchmark step — in this repository's own ecosystem. Exhaustive, deterministic.
What to do
Your ecosystem's static analyzer is declared as a dependency but is not wired into CI — run it as a required step on push/PR (e.g. `composer phpstan` / `mix sobelow` / `bundle exec brakeman` / the spotbugs Gradle task / `sbt scalafixAll --check` / `golangci-lint run` / `rebar3 lint`) so a regression fails the build instead of relying on someone running it locally.
Enable Dependabot/Renovate or a dependency-review gate.
Add gitleaks/trufflehog in CI to block PRs that introduce committed secrets.
Readiness · Readiness — Whether releases are automated and safely reversible (probes, rolling updates, approval gates) — from manifests/pipeline files, not the live environment.
Method: Filesystem scan: deployment manifests/IaC (K8s YAML, Helm, Terraform) for rolling updates, probes, approval gates, migration hooks. Exhaustive, deterministic.
Deployment automation exists and the service already exposes health probe endpoints (healthz/livez/readyz or a container healthcheck), but no manifest in this repository wires them into readiness/liveness probes, a rolling-update strategy, lifecycle hooks or a migration job — a bad release is harder to detect and reverse. If that wiring lives in a separate ops/Helm repository, document the pointer here so release safety is evidenced.
What to do
Wire the existing health endpoints into readiness/liveness probes and a rolling-update (or blue/green) strategy in the deployment manifests — or document where that wiring lives — so a bad release is caught and rolled back automatically.
Readiness · Readiness — Whether releases are traceable — a maintained changelog and explicit version stamping.
Method: Filesystem scan: changelog file presence and version tags in csproj or git tags. Exhaustive, deterministic.
No CHANGELOG/HISTORY/RELEASES file — what shipped when isn't easy to reconstruct for support or audit. (Versioning/tagging makes releases traceable, but a changelog records the what.)
What to do
Keep a changelog (e.g. Keep-a-Changelog) recording what shipped in each release.
Readiness · Readiness — Whether outbound HTTP calls are wrapped in resilience (retry/timeout/circuit-breaker) so a failing dependency doesn't cascade.
Method: Source scan: outbound HTTP clients and what bounds them — resilience handlers (Polly, AddStandardResilienceHandler) on .NET; on Go, the JVM, Python, JavaScript/TypeScript, Ruby, PHP, Rust, Elixir, Swift, Dart and Erlang, a timeout, deadline, retry or breaker beside each call, or a process-wide client default (a framework-wide deadline such as Drupal core's, Laravel's or actix's awc counts). Exhaustive, deterministic.
`.openConnection(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout. (×6) — core/src/main/scala/org/apache/spark/TestUtils.scala:280, core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:283, core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:291, …
`HttpClient(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout. — core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:306
`urlopen(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout. (×4) — REDACTED:601, REDACTED:686, REDACTED:703, …
What to do
Set connect and read timeouts on every client (`RestTemplateBuilder.connectTimeout/readTimeout`, WebClient `responseTimeout`, `HttpRequest.Builder.timeout`, `HttpURLConnection.setReadTimeout`, Feign `connectTimeout`/`readTimeout` in configuration), and wrap calls to dependencies in resilience4j `@Retry`/`@CircuitBreaker`/`@TimeLimiter` (or Spring Retry).
Pass `timeout=` on every `requests`/`httpx` call (requests has no default and will wait forever), mount an `HTTPAdapter(max_retries=Retry(...))` on shared sessions, and wrap calls to flaky dependencies in `tenacity` retries or a `pybreaker` circuit breaker.
Readiness · Performance — Whether the code protects its performance with benchmarks — a benchmark suite, allocation/memory measurement, and (ideally) a CI gate. Presence is credited as a bonus, never a deduction.
Method: Repo + source scan: BenchmarkDotNet referenced (csproj/source), [Benchmark]/[MemoryDiagnoser] attribute counts, and a benchmark step in CI; off .NET, the same ladder over Go testing.B, Rust criterion/#[bench]/divan, JMH/kotlinx-benchmark, pytest-benchmark/asv/pyperf, tinybench/mitata/vitest bench/benchmark.js and Swift package-benchmark — scored as a bonus ladder (absence is neutral, never a deduction). Deterministic, presence detection.
Readiness · Performance — Whether the code is written to minimise allocations so it doesn't pressure its host's memory manager — buffer/slice views over copies, object pooling, stack or value-type allocation, and buffer writers. Reward-only: credited where present, never penalised where a simpler style is fine.
Method: Production-source scan: density (per 1k LoC) of allocation-aware APIs — in .NET Span/Memory, ArrayPool/ObjectPool, stackalloc, ValueTask, value-type structs, IBufferWriter, string.Create, SkipLocalsInit; off .NET, with comments and strings blanked, Go's sync.Pool, preallocated slices/maps, Grow, strconv.Append* and buf[:0] reuse; the JVM's NIO buffer views, MemorySegment, primitive collections, pools and literal presizes (plus Kotlin primitive arrays and value classes, Scala AnyVal and @specialized); Swift's reserveCapacity, ContiguousArray, withUnsafe* access and ~Copyable. Activated off .NET on the same floor (400 lines, and benchmarks or 8 uses); Rust and garbage-collected scripting languages are not applicable. Reward-only. Deterministic, syntax/text detection.
What to do
Raise allocation-aware density on the hot paths — currently 897 use(s) across 1,063,051 production line(s) (~0.8/1k). More of the idioms below on the allocation-heavy paths climbs this toward 10.
Java: on hot paths, work through buffer views (ByteBuffer/LongBuffer slices, MemorySegment) instead of copies, avoid boxing with primitive collections (fastutil, Eclipse Collections, HPPC, Trove), presize collections and StringBuilders, and pool expensive buffers (Netty's pooled allocator, commons-pool).
Scala: on hot paths, make small wrappers value classes (extends AnyVal), @specialized generic code over primitives, use primitive collections, and presize builders.
Readiness · Performance — Whether asynchronous code stays responsive — it avoids sync-over-async blocking (a .NET .Wait()/.GetAwaiter().GetResult(), a time.sleep or blocking HTTP call inside a Python coroutine, a *Sync call inside an async JavaScript function, block_on inside a Rust async fn, runBlocking inside a Kotlin suspend function, block() inside a Reactor publisher) that stalls a thread or event loop and risks deadlock, and, where the code is a reusable library on .NET, awaits with ConfigureAwait(false) so it never captures and stalls its caller's context.
Method: Production-source scan: sync-over-async blocking counted everywhere — .Wait()/.GetAwaiter().GetResult() in .NET; off .NET, read from the language model, a blocking call inside an async function (Python, TS/JS, Rust, Kotlin, Swift, Dart) or inside a Java method returning a Reactor Mono/Flux or a Scala method returning a Future or effect — and, for a .NET library with ≥5 awaits, the share of awaits using ConfigureAwait(false). Deterministic, syntax/text detection.
Do you agree with this assessment?
R1 · Type Safety0.0 / 10Critical✓ Tool-verified
React / JS · Code Health — How much of the frontend is typed TypeScript vs untyped JavaScript.
Method: Frontend file inventory: the share of typed TypeScript vs untyped JavaScript across the source tree. Deterministic, exhaustive over frontend files.
0 typed · 34 plain JS — the untyped files are core/src/main/resources/org/apache/spark/ui/static/dataTables.rowsGroup.js, core/src/main/resources/org/apache/spark/ui/static/environmentpage.js, core/src/main/resources/org/apache/spark/ui/static/executorspage.js, core/src/main/resources/org/apache/spark/ui/static/flamegraph.js, core/src/main/resources/org/apache/spark/ui/static/historypage-common.js, REDACTED (+28 more).
What to do
Adopt TypeScript for the frontend: add a typecheck step to the build, then either type-check the existing JavaScript in place (`checkJs`/JSDoc types) or convert the highest-traffic modules first — no file is typed today, so this is an adoption, not a clean-up.
React / JS · Code Health — Near-exact copy-pasted blocks of substantial extent across the frontend (the D4 clone algorithm over JS/TS tokens, D-386): a block is reported only where its copies still agree on most of their own identifiers and literals, or were renamed as they were pasted but kept most of their constants, and where the copies carry enough code to stand on their own or the copied extent reaches 30 lines — so a re-implementation sharing neither names nor values, and a small pasted declaration, are both found and deliberately not reported, and a clean R10 is not a claim that nothing was copied.
Method: Near-exact copy-pasted blocks of substantial extent across the frontend (the D4 clone algorithm run over JS/TS tokens). Masking finds the candidates; a block is reported when its copies still agree on most of their own identifiers and literals, or when a renamed copy still agrees on most of its constants, AND the copies carry enough code to stand on their own — or when the copied extent reaches 30 lines. So a re-implementation sharing neither names nor values, and a small pasted declaration, are deliberately not counted. Deterministic.
core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:411 · REDACTED:192 — the two spans are one implementation copied and then locally edited — 296 tokens are still identical, in the same order in both files, with only local edits between them. The copies have already begun to drift, which is this row's finding: an edit made to one and not the other changes behaviour silently. Diff the two spans first to learn what genuinely differs, then extract the shared core into one module both sites use, passing the differences in as parameters — or, if one copy exists only because the other could not be imported from its context, make one of them the single source the other is generated or re-exported from. If one copy is no longer reachable, delete it rather than letting it shadow the live one. — core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:411
core/src/main/resources/org/apache/spark/ui/static/executorspage.js:696 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:851 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:696
core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:108 · core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:209 · core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:289 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. — core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:108
core/src/main/resources/org/apache/spark/ui/static/utils.js:205 · core/src/main/resources/org/apache/spark/ui/static/utils.js:228 · sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/sql-table-utils.js:153 — the 2 copies are spread across 2 directories, so the shared home is a decision rather than an obvious spot: check first whether one of them already owns this behaviour, and otherwise put the extracted module somewhere all of the sites already reach rather than making one of them depend on another. — core/src/main/resources/org/apache/spark/ui/static/utils.js:205
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:871 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:921 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:871
core/src/main/resources/org/apache/spark/ui/static/executorspage.js:621 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:639 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:657 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:675 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:621
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:524 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:543 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:561 — all 3 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:524
core/src/main/resources/org/apache/spark/ui/static/executorspage.js:590 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:819 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:590
REDACTED:286 · REDACTED:301 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. — REDACTED:286
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:964 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:978 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1002 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:964
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:60 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:77 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:60
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:940 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1032 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:940
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:982 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1022 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:982
core/src/main/resources/org/apache/spark/ui/static/executorspage.js:234 · REDACTED:66 · REDACTED:81 — the 3 copies are spread across 2 files, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:234
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:892 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:902 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:892
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1024 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1034 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1024
REDACTED:57 · REDACTED:719 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. — REDACTED:57
core/src/main/resources/org/apache/spark/ui/static/utils.js:114 · core/src/main/resources/org/apache/spark/ui/static/utils.js:224 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/utils.js:114
REDACTED:885 · REDACTED:1100 — the two spans are one implementation copied and then locally edited — 101 tokens are still identical, in the same order in both spans, with only local edits between them. The copies have already begun to drift, which is this row's finding: an edit made to one and not the other changes behaviour silently. Diff the two spans first to learn what genuinely differs, then extract the shared core into one module both sites use, passing the differences in as parameters — or, if one copy exists only because the other could not be imported from its context, make one of them the single source the other is generated or re-exported from. If one copy is no longer reachable, delete it rather than letting it shadow the live one. — REDACTED:885
core/src/main/resources/org/apache/spark/ui/static/executorspage.js:291 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:301 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:291
core/src/main/resources/org/apache/spark/ui/static/executorspage.js:916 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:384 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:392 — the 3 copies are spread across 2 files, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:916
core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:172 · core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:184 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. — core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:172
core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:176 · core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:281 · core/src/main/resources/org/apache/spark/ui/static/structured-streaming-page.js:35 — the 2 copies sit in sibling files in one directory, so check first whether one of them (or an existing module there) already owns this behaviour and the others should call it; otherwise extract it into one module in that directory and have each site call it. — core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:176
core/src/main/resources/org/apache/spark/ui/static/stagepage.js:294 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:308 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:294
core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:150 · core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:259 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:150
core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:191 · core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:198 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported. — core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:191
What to do
Act on each finding's own remediation rather than one rule: the move depends on what recurs. Where the copies are executable blocks, give the shared part one home and call it from each site; where they are declarations, a listing, a specialisation already delegating to its base, or one shape repeated per entity, there is no call site and the move is a shared type, a generated set or a factory — sometimes there is nothing to extract.
React / JS · Code Health — Per-function cyclomatic/cognitive complexity from the token-level function scanner (D-386) — real branching, not a regex heuristic.
Method: Per-function cyclomatic/cognitive complexity from a token-level function scanner (real branching, not a regex heuristic), computed over every frontend function. Deterministic.
(anonymous) has cyclomatic complexity 19 and cognitive complexity 15; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive sits below cyclomatic here, so much of the count is breadth — arms side by side rather than stacked — and splitting per arm would leave a function per arm; group the work between the checks into named steps instead. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:678
updateDetailsPanel has cyclomatic complexity 18 and cognitive complexity 19; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — REDACTED:423
getInnerText has cyclomatic complexity 17 and cognitive complexity 19; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — REDACTED:212
innerSortFunction has cyclomatic complexity 15 and cognitive complexity 23; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — REDACTED:92
(anonymous) has cyclomatic complexity 15 and cognitive complexity 18; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:404
formatMetricValue has cyclomatic complexity 15 and cognitive complexity 16; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — REDACTED:344
(anonymous) has cyclomatic complexity 15 and cognitive complexity 12; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive sits below cyclomatic here, so much of the count is breadth — arms side by side rather than stacked — and splitting per arm would leave a function per arm; group the work between the checks into named steps instead. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — REDACTED:1118
makeSortable has cyclomatic complexity 14 and cognitive complexity 22; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — REDACTED:42
(anonymous) has cyclomatic complexity 14 and cognitive complexity 14; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — core/src/main/resources/org/apache/spark/ui/static/environmentpage.js:164
reselectCheckboxesBasedOnTaskTableState has cyclomatic complexity 12 and cognitive complexity 18; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:287
zoomToNode has cyclomatic complexity 12 and cognitive complexity 9; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive sits below cyclomatic here, so much of the count is breadth — arms side by side rather than stacked — and splitting per arm would leave a function per arm; group the work between the checks into named steps instead. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — REDACTED:1047
(anonymous) has cyclomatic complexity 11 and cognitive complexity 12; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes. — REDACTED:951
What to do
Break down the listed branch-heavy functions; aim P95 cyclomatic ≤ 5.
Do you agree with this assessment?
R3 · Large Files1.3 / 10Critical✓ Tool-verified
React / JS · Code Health — How many source files exceed the large-file threshold.
Method: Components/modules exceeding the large-file threshold, counted exhaustively across the frontend source tree. Deterministic.
core/src/main/resources/org/apache/spark/ui/static/stagepage.js has 1077 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module. — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1
REDACTED has 1065 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module. — REDACTED:1
core/src/main/resources/org/apache/spark/ui/static/executorspage.js has 907 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:1
core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js has 529 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module. — core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:1
REDACTED has 459 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module. — REDACTED:1
What to do
Split each oversized file along the responsibilities already in it, into smaller focused modules in the same package.
Do you agree with this assessment?
R4 · Test Coverage4.3 / 10Weak✓ Tool-verified
React / JS · Readiness — Static test reachability (D-386): the share of production files reachable from any test via the import graph — measured without running anything.
Method: Static test reachability: the share of production files reachable from any test via the import graph — measured without running anything. Deterministic.
No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one. (×12) — core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1, core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:1, REDACTED:1, …
What to do
Add tests that import the unreached modules (directly or through their public entry).
React / JS · Readiness — How outdated the npm dependencies are (a maturity signal). JS/npm CVEs are scored separately in D30 (JS/npm Dependency Vulnerabilities).
Method: npm dependency staleness from manifest/registry metadata (a maturity signal; JS/npm CVEs are scored separately in D30, which answers dependency vulnerabilities for every ecosystem). Deterministic.
Do you agree with this assessment?
R6 · Tooling10.0 / 10Exemplary✓ Tool-verified
React / JS · Readiness — Whether the project wires up test, lint and typecheck — detected from each package.json script's COMMAND (eslint / tsc / vitest / jest / playwright), not just its name, and corroborated against CI-workflow invocations so a tool run only in CI still counts.
Method: package.json scanned for test/lint/typecheck script wiring. Deterministic presence check.
Do you agree with this assessment?
R7 · Dead Code10.0 / 10Exemplary✓ Tool-verified
React / JS · Code Health — Files unreachable from every application/tooling/test entry point, and exports nothing imports (module-graph reachability, D-386).
Method: Dead code: files unreachable from every application/tooling/test entry point plus exports nothing imports, via module-graph reachability. Deterministic, exhaustive over the import graph.
1 file(s) (~48 LoC) were excluded from dead-code analysis — declare main/module/exports or a conventional entry (src/index.*, an index.html script) so reachability can see this package. — dev
11 file(s) (~982 LoC) were excluded from dead-code analysis — declare main/module/exports or a conventional entry (src/index.*, an index.html script) so reachability can see this package. — ui-test
React / JS · Readiness — npm dependency truthfulness (D-386): unused dependencies, imports not declared anywhere, and type-/test-only packages shipped as production deps.
Method: npm dependency truthfulness: unused dependencies, imports declared nowhere, and type-/test-only packages shipped as production deps — from the manifest + import graph. Deterministic.
Declared in ui-test/package.json but never imported anywhere in that package or its workspace members — no static import reaches it. Usually that is dead weight and attack surface, but two shapes are indistinguishable from source and are NOT dead: an optional or native peer that another dependency loads dynamically at runtime, and a package a build, docs or test step installs and invokes separately. Confirm which of the three this is before removing it.
What to do
Remove unused dependencies, declare unlisted imports explicitly, and demote type-/test-only packages to devDependencies.
React / JS · Architecture — Import cycles in the module graph (D-386) — files that can only be understood and changed together.
Method: Import cycles in the module graph, detected exhaustively over JS/TS imports (the same cycle detection as the .NET coupling dimension). Deterministic.
Other · Code Health — Whether the code avoids sync-over-async (deadlock-prone blocking on tasks) and async void.
Method: Roslyn syntax scan: async methods scanned for .Wait()/.GetAwaiter().GetResult() and async-void outside event handlers. Deterministic, hard fact per invocation.
Other · Code Health — Whether any branch is dead by construction — a switch arm whose label can never equal a case-normalised subject, or an `else if` whose predicate the arm above has already swallowed.
Method: Roslyn syntax + semantics: switch labels compared against the subject's own case normaliser, and if/else-if chains checked for a literal an earlier arm's containment test already swallows. Deterministic, provable per finding. Advisory.
Do you agree with this assessment?
X13 · Undrained process stream10.0 / 10Exemplary○ Nothing flagged
Other · Code Health — Whether a child process that has BOTH standard streams redirected drains both — reading one to the end while the other is never read deadlocks once the child fills the unread pipe.
Method: Roslyn syntax + semantics: ProcessStartInfo launches with both streams redirected, checked for a drain of each stream across the enclosing type. On a repository with no .NET source it reads JavaScript/TypeScript off the engine’s own token stream (test, vendored and minified paths not) with the same rule: a `spawn` imported from `child_process` whose stdout and stderr are both pipes (no options, no `stdio`, or `stdio` of `'pipe'`), bound to a local that never leaves its scope, where exactly one of the two streams is read, the other never, and the child’s `close`/`exit` (or the read stream’s end) is awaited; a shell redirect in the call’s arguments or a `kill` of the child suppresses it. The same rule reads Python’s `subprocess.Popen`, Dart’s `Process.start`, Ruby’s `Open3.popen3`, and Scala’s JDK process starts (`new ProcessBuilder(…).start()` with no `redirectErrorStream`/`redirectError`/`redirectOutput`/`inheritIO`, `Runtime.getRuntime.exec`) where one of `getInputStream`/`getErrorStream` is read and the other never while `waitFor()` is called or the read stream is read to its end. Deterministic, provable per finding. Advisory.
Other · Security — Whether a hand-rolled public/private IP check can be walked past — a method that unwraps IPv4-mapped IPv6 but returns the opposite verdict for the same host written as IPv4-compatible, 6to4 or NAT64.
Method: Roslyn syntax + semantics: methods that unwrap IPv4-mapped IPv6 and hand-roll IPv4 range carve-outs, checked for whether the IPv6 branch also accounts for the IPv4-compatible, 6to4 and NAT64 embeddings. Deterministic, provable per finding. Advisory.
Do you agree with this assessment?
X15 · Unvalidated length from an untrusted reader2.5 / 10Weak✓ Tool-verified
Other · Security — Whether a length read out of the stream being parsed is bounded before it is allocated or read — an unchecked count taken from the input lets the input choose the allocation.
Method: Roslyn syntax + semantics: integer lengths read from a BinaryReader and spent on a bulk read or an array allocation, checked for any comparison or bounding call on the value anywhere in the method. Deterministic, provable per finding. Advisory.
`length` is read from the bytes being parsed at line 88 (`getInt`), then used to size a `ByteBuffer.allocate` at line 90 — and nowhere in its scope is `length` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `length` against the bytes actually remaining and a maximum the format allows before allocating. — streaming/src/main/scala/org/apache/spark/streaming/dstream/RawInputDStream.scala:90
`length` is read from the bytes being parsed at line 107 (`readInt`), then used to size a `new Array[Byte]` at line 108 — and nowhere in its scope is `length` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `length` against the bytes actually remaining and a maximum the format allows before allocating. — streaming/src/main/scala/org/apache/spark/streaming/api/python/PythonDStream.scala:108
`functionInBytesLength` is read from the bytes being parsed at line 184 (`readInt`), then used to size a `new Array[Byte]` at line 185 — and nowhere in its scope is `functionInBytesLength` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `functionInBytesLength` against the bytes actually remaining and a maximum the format allows before allocating. — sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala:185
`keySize` is read from the bytes being parsed at line 866 (`readInt`), then used to size a `new Array[Byte]` at line 873 — and nowhere in its scope is `keySize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `keySize` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:873
`valueSize` is read from the bytes being parsed at line 878 (`readInt`), then used to size a `new Array[Byte]` at line 892 — and nowhere in its scope is `valueSize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `valueSize` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:892
`keySize` is read from the bytes being parsed at line 1028 (`readInt`), then used to size a `new Array[Byte]` at line 1035 — and nowhere in its scope is `keySize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `keySize` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:1035
`valueSize` is read from the bytes being parsed at line 1041 (`readInt`), then used to size a `new Array[Byte]` at line 1046 — and nowhere in its scope is `valueSize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `valueSize` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:1046
`keySize` is read from the bytes being parsed at line 506 (`readInt`), then used to size a `new Array[Byte]` at line 516 — and nowhere in its scope is `keySize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `keySize` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala:516
`valueSize` is read from the bytes being parsed at line 518 (`readInt`), then used to size a `new Array[Byte]` at line 523 — and nowhere in its scope is `valueSize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `valueSize` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala:523
`blockSize` is read from the bytes being parsed at line 546 (`readInt`), then used to size a `new Array[Byte]` at line 547 — and nowhere in its scope is `blockSize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `blockSize` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala:547
`size` is read from the bytes being parsed at line 157 (`readInt`), then used to size a `new Array[Byte]` at line 158 — and nowhere in its scope is `size` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `size` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/python/RowQueue.scala:158
`size` is read from the bytes being parsed at line 67 (`readInt`), then used to size a `new Array[Byte]` at line 71 — and nowhere in its scope is `size` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `size` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkDeserializer.scala:71
`messageLen` is read from the bytes being parsed at line 244 (`readInt`), then used to size a `new Array[Byte]` at line 245 — and nowhere in its scope is `messageLen` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `messageLen` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:245
`length` is read from the bytes being parsed at line 602 (`getInt`), then used to size a `new Array[Byte]` at line 603 — and nowhere in its scope is `length` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `length` against the bytes actually remaining and a maximum the format allows before allocating. — sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnType.scala:603
`typeLength` is read from the bytes being parsed at line 766 (`getInt`), then used to size a `new Array[Byte]` at line 767 — and nowhere in its scope is `typeLength` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `typeLength` against the bytes actually remaining and a maximum the format allows before allocating. — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala:767
`len` is read from the bytes being parsed at line 863 (`readInt`), then used to size a `new Array[Int]` at line 864 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating. — mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:864
`len` is read from the bytes being parsed at line 879 (`readInt`), then used to size a `new Array[Long]` at line 880 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating. — mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:880
`len` is read from the bytes being parsed at line 895 (`readInt`), then used to size a `new Array[Float]` at line 896 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating. — mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:896
`len` is read from the bytes being parsed at line 911 (`readInt`), then used to size a `new Array[Double]` at line 912 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating. — mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:912
`len` is read from the bytes being parsed at line 944 (`readInt`), then used to size a `new Array[(]` at line 945 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating. — mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:945
`len` is read from the bytes being parsed at line 1036 (`readInt`), then used to size a `new Array[T]` at line 1037 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating. — mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:1037
`chunksNum` is read from the bytes being parsed at line 106 (`readInt`), then used to size a `new Array[ByteBuffer]` at line 109 — and nowhere in its scope is `chunksNum` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `chunksNum` against the bytes actually remaining and a maximum the format allows before allocating. — core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala:109
`len` is read from the bytes being parsed at line 116 (`readInt`), then used to size a `new Array[Byte]` at line 117 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating. — core/src/main/scala/org/apache/spark/security/SocketAuthHelper.scala:117
`len` is read from the bytes being parsed at line 174 (`readInt`), then used to size a `new Array[Byte]` at line 175 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating. — core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala:175
`valueLength` is read from the bytes being parsed at line 240 (`readInt`), then used to size a `new Array[Byte]` at line 241 — and nowhere in its scope is `valueLength` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `valueLength` against the bytes actually remaining and a maximum the format allows before allocating. — core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala:241
`length` is read from the bytes being parsed at line 255 (`readInt`), then used to size a `new Array[Byte]` at line 256 — and nowhere in its scope is `length` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `length` against the bytes actually remaining and a maximum the format allows before allocating. — core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala:256
`numMetrics` is read from the bytes being parsed at line 82 (`readInt`), then used to size a `new Array[Long]` at line 86 — and nowhere in its scope is `numMetrics` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `numMetrics` against the bytes actually remaining and a maximum the format allows before allocating. — core/src/main/scala/org/apache/spark/scheduler/TaskResult.scala:86
`contentPairsLength` is read from the bytes being parsed at line 64 (`readInt`), then used to size a `new Array[Byte]` at line 65 — and nowhere in its scope is `contentPairsLength` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `contentPairsLength` against the bytes actually remaining and a maximum the format allows before allocating. — core/src/main/scala/org/apache/spark/api/r/RRunner.scala:65
`length` is read from the bytes being parsed at line 251 (`readInt`), then used to size a `new Array[Byte]` at line 252 — and nowhere in its scope is `length` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `length` against the bytes actually remaining and a maximum the format allows before allocating. — core/src/main/scala/org/apache/spark/api/java/JavaRDD.scala:252
`n` is read from the bytes being parsed at line 267 (`readInt`), then used to size a `new byte[]` at line 268 — and nowhere in its scope is `n` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `n` against the bytes actually remaining and a maximum the format allows before allocating. — common/unsafe/src/main/java/org/apache/spark/unsafe/types/BinaryView.java:268
`n` is read from the bytes being parsed at line 289 (`readInt`), then used to size a `new byte[]` at line 290 — and nowhere in its scope is `n` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `n` against the bytes actually remaining and a maximum the format allows before allocating. — common/unsafe/src/main/java/org/apache/spark/unsafe/types/BinaryView.java:290
`numWords` is read from the bytes being parsed at line 107 (`readInt`), then used to size a `new long[]` at line 108 — and nowhere in its scope is `numWords` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `numWords` against the bytes actually remaining and a maximum the format allows before allocating. — common/sketch/src/main/java/org/apache/spark/util/sketch/BitArray.java:108
What to do
Each finding names the read that takes the length from the input and the site that spends it, and states that the value is never compared in between. Confirm from the two, then bound the length against the remaining stream before using it.
Other · Code Health — Whether a loop that shortens a string until it fits a length budget has a floor — one with none grinds the value down to the empty string, or past it into a negative-length `Substring`.
Method: Roslyn syntax + semantics: while/do loops whose body's only effect on a string is to drop its last character, checked for whether anything — a direct comparison on the length, a body guard, a break — bounds that length below. On a repository with no .NET source it reads JavaScript/TypeScript off the engine’s own token stream (test, vendored and minified paths not) with the same rule: a `while`/`do` loop whose body’s one assignment to a value is `x = x.slice(0, -1)` or `x = x.slice|substring|substr(0, x.length - 1)`, driven by a condition that reads `x.length` only as a term of a larger expression — never compared directly, never tested for truthiness, and with no other read of `x` — and whose body has no `break`, `return`, `throw` or `if` naming `x`. Deterministic, provable per finding. Advisory.
Do you agree with this assessment?
X17 · Uncapped recursion over a caller-supplied document1.8 / 10Critical✓ Tool-verified
Other · Security — Whether a walk that recurses through a JSON/XML tree handed in by its caller bounds how deep it will go — an uncapped walk lets the document's nesting choose the stack depth, and the resulting StackOverflowException cannot be caught.
Method: Roslyn syntax + semantics: methods that take a JSON/XML document node and call themselves with a child of it, reachable from an externally-callable member of the same type that accepts a document, checked for any depth parameter, descent counter or threaded arithmetic anywhere in the walk. On a repository with no .NET source it reads TypeScript off the engine’s own token stream (test, vendored, generated and minified paths and `.d.ts` not) with the same rule: a function or method with a parameter typed as a document value — `unknown`, `any`, `object`, `Record<string, unknown|any>`, a JSON alias (`JsonValue`, `JSONObject`, …) or a DOM node global the file does not rebind — that calls itself (bare, or through `this` for a method, or hands itself to a call over the value as in `value.map(walk)`) with something it took out of that value and never through an ancestor accessor such as `closest()` or `parentElement`, that is exported or reached from an exported function (a public method of the same exported class) taking such a value, and that names no depth, level, nesting, recursion, remaining or budget anywhere and threads no `+`/`-` arithmetic through a self-call. Plain JavaScript is not read: with no annotation nothing tells a parsed document from a tree the code built itself. Deterministic, provable per finding. Advisory.
`assert_eq` is public and takes `left: Any` from its caller, and it calls itself at line 487 (`self.assert_eq`) with a piece of `left` — and nowhere in `assert_eq` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `assert_eq` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack. — python/pyspark/testing/pandasutils.py:487
`enum_to_value` is public and takes `value: Any` from its caller, and it calls itself at line 226 (`enum_to_value`) with a piece of `value` — and nowhere in `enum_to_value` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `enum_to_value` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack. — python/pyspark/sql/utils.py:226
`lit` is public and takes `col: Any` from its caller, and it calls itself at line 290 (`lit`) with a piece of `col` — and nowhere in `lit` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `lit` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack. — python/pyspark/sql/functions/builtin.py:290
`lit` is public and takes `col: Any` from its caller, and it calls itself at line 286 (`lit`) with a piece of `col` (and at line 294) — and nowhere in `lit` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `lit` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack. — python/pyspark/sql/connect/functions/builtin.py:286
`from_dict` is public and takes `dct: dict` from its caller, and it calls itself at line 262 (`self.from_dict`) with a piece of `dct` — and nowhere in `from_dict` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `from_dict` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack. — REDACTED:262
What to do
Each finding names the externally-callable member that accepts the document and the self-call that descends, and states that no depth parameter or counter exists anywhere in the walk. Confirm from the two, then thread a depth argument through the walk and reject documents past a fixed maximum, or rewrite it with an explicit stack.
Other · Code Health — Whether a type's disposal matches what it OWNS — releasing what it created, leaving alone what it was handed, and not declaring a finalizer for state that has nothing unmanaged to finalize.
Method: Roslyn syntax + semantics: every assignment to a disposable field is read to decide whether the type CREATED the value or was handed it, and the type's disposal is checked against that answer — an injected interface it disposes, a value it constructed and never releases, a finalizer on a type holding nothing unmanaged, and a disposable local whose every reference is a plain member read. A value handed to a container that disposes its contents (a parent control's `Controls` collection, a component `IContainer`) is released by that container and is not reported; generated code is out of population. On a repository with no .NET source the same ownership questions are read in JavaScript/TypeScript off the engine’s own token stream (test, vendored, generated and minified paths not): a class declaring `dispose()`, `[Symbol.dispose]()` or `[Symbol.asyncDispose]()` that disposes a field it was handed through a constructor parameter typed as a repository interface or resolved by a dependency-injection container; that assigns a field only ever from `new X(…)` of a disposable class and neither releases it anywhere in the class nor names it in its disposal member or a method that member calls, nor hands it to anything else; and a `const`/`let` local built from literals only whose every reference opens a statement operating on a non-release member of it. A class is disposable when every repository declaration of its name declares or inherits a disposal member, or when it is a documented library disposable (`vscode` EventEmitter, CancellationTokenSource and Disposable; three.js geometries, materials, textures, render targets, renderers, controls and composers). The finalizer arm has no JavaScript counterpart: a class cannot declare one. Dart is read for the owned-and-never-released arm: a Flutter `State` (or a class declaring `dispose()`/`close()`) whose library-private field is written only with direct creations of a framework controller, `FocusNode`, `StreamController` or periodic `Timer` (or a `.listen(…)` into a field declared `StreamSubscription`), which nothing in the class or its library releases (`dispose`/`close`/`cancel`) and which is never handed on except as a named argument. Ruby is read for the owned-and-never-released arm and the abandoned-local arm: a class declaring an instance `close`, `dispose`, `shutdown`, `stop`, `disconnect` or `terminate` whose instance variable is written only with a blockless core descriptor creation (`File.open`/`new`, `Tempfile`, `Dir.open`, `IO.popen`, the `socket` and `openssl` socket classes) that no method of the class closes and that is never handed on (passed, returned, stored, exposed by `attr_reader`, delegated) — a class whose superclass or mixin lies outside the repository is not judged — and a blockless creation read through and dropped (`File.open(path).read`) or held in a local that is never closed, returned, passed or stored (a `&:read` block-pass, an `autoclose: false` borrow and a `fileno` hand-over are not abandonment). PHP is read for the same two arms: a class (not a trait, and not one using a trait) declaring an instance `close`, `dispose`, `shutdown`, `stop`, `disconnect` or `terminate` whose `private` property (or a `final` class's `protected` one) is written only with a direct core `fopen`, `tmpfile`, `popen`, `proc_open`, `fsockopen`, `stream_socket_client`/`server` or `opendir` that nothing in the class closes (`fclose`, `pclose`, `closedir`, `unset`, a reset to `null`) and that is never handed on, and a handle held in a function's local, or read through as the argument of a reader, that is never closed, returned, passed, captured or stored; a `php://` stream, `curl_init`, `SplFileObject`, `PDO` and a function the repository declares under the core name are not read. Deterministic, provable per finding. Advisory.
`File.open` is called here without a block, so it hands back an open descriptor the caller must close — and its first and only use reads `.read` through it and drops it. Nothing closes it: it stays open until the garbage collector happens to finalize it, which is one leaked descriptor per call (a child process too, for `IO.popen`), `Errno::EMFILE` under load, and for a written file data that is not flushed when the program next reads it. Use the block form, which closes on every path including a raise (`File.open(…) { |io| … }`), or the one-call reader/writer (`File.read`, `File.write`, `File.foreach`). — docs/_plugins/include_example.rb:49
`new ServerSocket(…)` opens something that has to be given back — a file descriptor, a socket, a directory handle — and it is held in `serverSocket`, and every later use of `serverSocket` in its block reads a member through it — it is never returned, passed, stored, captured by a lambda, reassigned or released, so this expression both creates the value and is the last place that can release it, and does not. Every call leaks one more open handle until the collector happens to reach it: `Too many open files` under load, and on Windows a file that cannot be replaced or deleted while it is held. Open it with `scala.util.Using.resource(new ServerSocket(…)) { r => … }`, which releases it on every path including a throw, or hold it and call `.close()` in a `finally`. — streaming/src/main/scala/org/apache/spark/streaming/util/RawTextSender.scala:63
What to do
Each finding names the value and the span that decides who owns it — the `new` that created it, or the constructor parameter that handed it over. Confirm ownership from that span, then make the disposal match it: release what this type created, leave what it was injected with to whoever created THAT, and drop a finalizer whose type holds nothing unmanaged to release.
Other · Code Health — Whether a method that temporarily changes state belonging to the whole process — the working directory, an environment variable — puts it back on EVERY path: a restore reached only when nothing throws leaks the change to the rest of the process.
Method: Roslyn syntax + semantics: method bodies that write the process working directory or an environment variable and write it back in the same body, checked for whether that restore sits in a `finally`/`catch` or only on the straight-line path. On a repository with no .NET source the same rule reads production JavaScript/TypeScript off the engine’s own token stream (test, vendored and minified paths not): `process.chdir`, `process.env.NAME =`/`["NAME"] =` and `delete process.env.NAME`, and Deno’s `Deno.chdir`/`Deno.env.set`/`Deno.env.delete`, paired per function body (a nested function or arrow is its own body, and module top-level code is none), where the last write puts back a local the body captured from the same global or deletes a variable the first write set, with at least one statement between them; a write in a `catch`/`finally` of that body silences it. Deterministic, provable per finding. Advisory.
Other · Code Health — Whether an argument guard throws the exception its own condition describes — a guard that rejects a value for being EMPTY and reports it as `NullPointerException` (Java, Kotlin, Scala), `ArgumentError.notNull` (Dart) or `ArgumentNullException` (F#'s `nullArg`) tells the caller a parameter was null when it provably was not.
Method: Roslyn syntax: `throw new ArgumentNullException(nameof(p))` statements controlled by an `if`, whose condition is read for a test that is true of a NON-null `p` — an emptiness test that dereferences it (`p.Count == 0`, `!p.Any()`) or a BCL predicate documented true of the empty value (`string.IsNullOrEmpty(p)`). Deterministic, provable per finding. Advisory.
Other · Code Health — Whether a `when` guard is free of side effects — a guard that increments a counter or assigns while deciding whether its arm matches applies that change during PATTERN MATCHING, on an arm that may not be selected, and skips it entirely when a short-circuit to its left answers first.
Method: Roslyn syntax: `when` guards on case labels and switch-expression arms, read for a mutation (`++`/`--`/assignment) sitting in a position the guard's own `&&`/`||`/`??`/`?:`/`?.` can skip. Deterministic, provable per finding. Advisory.
Other · Code Health — Whether a method that TAKES a lock or semaphore and gives it back from a flag-guarded `finally` returns the value that flag implies — reporting success while the guard hands the primitive back admits a second caller the exclusion was there to keep out, and reporting failure while the guard keeps it leaves nothing to ever give it back.
Method: Roslyn syntax: `try` statements whose `finally` releases a synchronisation primitive under a bare local-bool guard, where the method also TOOK that same primitive before the `try`, checked for a `return` of a bool literal whose value disagrees with the flag state the method's own straight-line assignments put it in. Deterministic, provable per finding. Advisory.
Other · Code Health — Whether the work a diagnostic log line costs is paid only when that line is wanted — C# evaluates a call's arguments BEFORE the call, so a trace/debug message joined or projected out of a collection is built in full on every pass, and then discarded by a sink the shipped configuration leaves switched off.
Method: Roslyn syntax: log calls at a diagnostic level (a `Log`-prefixed method naming Trace/Debug/Verbose, or a bare `Debug`/`Trace`/`Verbose` on a receiver named for a logger), whose argument list is read for a call whose cost scales with a sequence — a LINQ operator, a materialisation, `string.Join`, a serializer — with no enclosing level check or conditional-compilation region. On a repository with no .NET source it reads JavaScript/TypeScript off the engine’s own token stream (test, vendored and minified paths not) with the same rule: a `debug`/`trace`/`verbose` call on a receiver named for a logger, or a `log`-prefixed method naming the level, whose argument calls an array operator (`map`/`filter`/`reduce`/`sort`/…), `Array.from`, `Object.keys/values/entries`, `JSON.stringify`/`util.inspect` over anything but a literal, or an array `join` — outside any arrow or function passed as an argument, which the logger calls only when the level is on — with no enclosing `if`, `&&` or `?:` whose condition names a level, a level string, or the `NODE_ENV`/`__DEV__`/`DEV` build switch. Deterministic, provable per finding. Advisory.
Do you agree with this assessment?
X24 · Document value interpolated into markup unescaped6.2 / 10Adequate✓ Tool-verified
Other · Security — Whether text read out of the document being converted is escaped before it is written into generated markup — a value the document's author chose, interpolated into an attribute the surrounding literal delimits, can close that attribute and open another.
Method: Roslyn semantic model over the whole compilation: a string-typed `Value`/`InnerText`/`InnerXml`/`Text` member declared inside `DocumentFormat.OpenXml` or `System.Xml` is a taint SOURCE, propagated through assignments, returns, arguments, tuple elements and string composition to its transitive closure, then read at interpolated-string holes that sit in a markup position the surrounding literal itself delimits. Escaper/encoder calls and enclosing validator conditions cut the flow. Flow- and container-insensitive by construction. A second arm needs no provenance at all and reports a type that CONTRADICTS ITSELF — the same expression escaped at one delimited markup hole and interpolated raw at another hole in the same markup position of the same type, which the type's own escaping proves is a defect without knowing where the value came from. On a repository with no .NET source it reads JavaScript/TypeScript off the token stream with the same rule: a DOM read of raw document text (`getAttribute`, `textContent`, `innerText`, `nodeValue`) is the source, propagated through local bindings and string composition, and judged at template-literal and concatenation holes in the same two delimited markup positions; escapers and validating conditions cut it, and documentation-site, test, vendored and minified scripts are not read. Deterministic, provable per finding. Advisory.
`tableId` is text this program did not author, and it is written into a quoted attribute value through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup. — core/src/main/resources/org/apache/spark/ui/static/environmentpage.js:125
`container.innerHTML` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup. — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:81
`url` is text this program did not author, and it is written into the start of a URL attribute, where it chooses the scheme through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. HTML encoding does not help here — a `javascript:` URL contains nothing it changes; allow only http(s) or a relative path. (×2) — core/src/main/resources/org/apache/spark/ui/static/executorspage.js:88, core/src/main/resources/org/apache/spark/ui/static/executorspage.js:117
`(summary.totalQueries||0)` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup. — sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:61
`formatDurationSql(summary.averageDuration||0)` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup. — sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:62
`(summary.runningQueries||0)` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup. — sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:63
`(summary.failedQueries||0)` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup. — sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:64
`formatMetricValue(val,m.type,showStageTask)` is text this program did not author, and it is written into the text of an element through markup built here with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup. — REDACTED:533
What to do
Each finding names the expression, the attribute it lands in and the line. Confirm the value's origin from the read this analyzer traced, then escape it at the sink or validate it against the shape the attribute accepts before interpolating.
Other · Code Health — Whether a value the caller is invited to supply is the value the type actually uses — a constructor parameter stored in a private field that nothing ever reads while the default it was given is spelled out a second time at the site that should have read it, a keyed lookup that falls back to a different setting than the one its key names while the same type falls back to the matching one for that same key, or a culture-sensitive parse given no format provider by a type that feeds its own settable culture to the same kind of parse elsewhere, or a number an entry point writes to standard output under the host's culture while the same entry point formats with the invariant culture elsewhere. Either way, every caller who supplies a value silently gets something else.
Method: Roslyn syntax: private instance fields of a non-partial type assigned in a constructor from one of its own parameters with a `??` fallback, checked for whether anything in the type body reads the field and whether that same fallback expression is spelled out again outside the constructor; and `??` fallbacks onto a member access from a lookup call carrying exactly one string literal, grouped by that key across the type and checked for a fallback member whose folded name disagrees with the key while a sibling site for the same key agrees with it. On a repository with no .NET source the first two arms read JavaScript/TypeScript off the engine’s own token stream (tests included, bundles and vendored paths not): a `#x`, `private` or `private` parameter-property instance field filled in the constructor from a parameter (or one member of one) through `??`/`||` or a parameter default, never read anywhere in the file by name, whose constructed default is spelled again in the class body; and `lookup("key") ?? s.member` grouped by key per class, or per module outside every class. The culture arm has no JavaScript counterpart: its parses take no locale. Deterministic, provable per finding. Advisory.
Other · Code Health — Whether a value handed from a callback to the body that waits on it crosses on something built to be crossed — a `Queue<T>`/`List<T>`/`Dictionary<K,V>` written inside an event handler and read back outside it is mutated by two flows at once, and the semaphore or completion source beside it orders how MANY items exist while leaving the collection's own head, tail and backing array unprotected.
Method: Roslyn syntax: method, accessor, local-function and lambda bodies that declare BOTH a non-thread-safe generic collection (`Queue`/`Stack`/`List`/`Dictionary`/`HashSet`/`Sorted*`/`LinkedList`) and a synchronisation primitive (`SemaphoreSlim`/`TaskCompletionSource`/`ManualResetEvent(Slim)`/`AutoResetEvent`/`CountdownEvent`) as locals, then read for a `+=`-registered lambda that raises that primitive while the body outside every lambda waits on it — and, in that scope, a mutating call on the collection inside the lambda paired with a mention of it outside. Any `lock` in the scope abstains it. On a repository with no .NET source the same handoff is read in Java off the engine’s own token stream (test source sets, vendored and demonstration paths not): an `ArrayList`/`LinkedList`/`ArrayDeque`/`PriorityQueue`/`Hash*`/`LinkedHash*`/`Tree*` local and a `CountDownLatch`/`Semaphore`/`CompletableFuture` local, a lambda or anonymous class that raises the primitive and mutates the collection, and a wait outside it; any `synchronized` or `lock()` abstains the body. Because each of those primitives orders what the callback wrote before raising it, only a touch that provably overlaps the callback is convicted: one after the registration and before the next wait, or one in a loop registered-before, waiting on every pass and not declaring the collection. Deterministic, provable per finding. Advisory.
Do you agree with this assessment?
X27 · Collection changed while being enumerated10.0 / 10Exemplary○ Nothing flagged
Other · Code Health — Whether an enhanced `for` leaves the collection it is walking alone — a body that adds to or removes from the very collection the loop is iterating invalidates the fail-fast iterator it is holding, so the next `next()` throws `ConcurrentModificationException` and the remaining elements are never seen. Whether a `for` loop leaves the `dict`, `set` or `deque` it is walking alone — CPython's iterators over those check that the collection has not changed size since the loop began, so a body that removes from (or empties) the very collection the loop is walking makes the next step raise `RuntimeError: dictionary changed size during iteration` and the remaining items are never seen. Whether a walk over a `Hash` or `Set` (`each`, `each_pair`, `map`, `for … in` and the other block walks) leaves it alone — Ruby refuses `rehash`, `replace`, `compare_by_identity` and `transform_keys!` on a Hash, and `reset`, `replace`, `compare_by_identity` and `map!` on a Set, while it is being iterated: the call raises `RuntimeError` at once and the remaining entries are never seen. Whether a `for` comprehension or `foreach` leaves the mutable collection it is walking alone — an `ArrayBuffer` or `ListBuffer` counts its changes and its iterator checks the count, an `ArrayDeque` (and `mutable.Queue`/`mutable.Stack`) is walked by index up to its starting length, and a `java.util` collection walked through `asScala` is fail-fast, so a body that adds to or removes from the very collection the loop is walking makes a later step throw `ConcurrentModificationException` (`IndexOutOfBoundsException` for a deque) and the remaining elements are never seen.
Method: Roslyn syntax + semantics: `foreach` statements whose body calls a structural mutator (`Add`/`Remove`/`Clear`/`Insert`/…) on the very expression the loop is enumerating. Two arms. ARM A — the source is a concrete fragile BCL collection, or a live `Keys`/`Values` view over one, and the mutator resolves to that same collection's own member; concurrent and immutable collections and arrays are outside the population by construction, since their enumerators survive a structural change. ARM B — the source is an argument-less accessor CALL on a receiver whose body is in source: the accessor must return a stored field VERBATIM and a sibling member must structurally change that same field, both read off the implementations rather than from the members' names. A mutation the loop provably exits immediately after (`break`/`return`/`throw`/`goto`), or one written inside a nested loop or a lambda, is counted and never reported. Deterministic, provable per finding. Advisory.
Do you agree with this assessment?
X28 · Index access outside its own emptiness guard10.0 / 10Exemplary○ Nothing flagged
Other · Code Health — Whether a condition that tests a value for emptiness indexes that same value only where the test holds — an `||` written one parenthesis too far to the left leaves an index access outside the guard beside it, so the empty case the guard exists to anticipate reaches the index and throws.
Method: Roslyn syntax only, no semantic model: the OUTERMOST `&&`/`||` of every boolean condition, read for a symbol the condition tests for emptiness (`string.IsNullOrEmpty`/`IsNullOrWhiteSpace`, a `Length`/`Count` comparison against a literal, `Any()`, a `Length`/`Count` pattern, or a comparison against `""`) and ALSO indexes. Each `symbol[...]` access is placed by a boolean-reachability walk from the access up to the outermost connective: an access is COVERED when some enclosing step has it in the right operand and the left operand, under the truth value that step forces, proves the symbol non-empty — a recursion over `&&`/`||` whose true- and false-directions are asymmetric. A finding needs BOTH an uncovered access and a covered one on the same symbol in the same condition, which is the agreeing twin that separates a misplaced parenthesis from an unrelated length test. Bare index accesses with no emptiness test in the condition are neither counted nor reported; a non-identifier receiver and a lambda nested inside the condition are outside the population. Deterministic, provable per finding. Advisory.
Do you agree with this assessment?
X29 · Per-element action decided by a fixed element10.0 / 10Exemplary○ Nothing flagged
Other · Code Health — Whether a decision taken once per element is taken ABOUT that element — a test inside a counted loop that reads a fixed subscript of the very collection its guarded statement indexes by the loop variable applies element zero's answer to all of them, so the elements that differ from it are all handled wrongly, and in the same direction.
Method: Roslyn syntax only, no semantic model: every `for` statement declaring exactly ONE loop variable, and every `if` inside its body that is not under a nested loop or a lambda. A site enters the population when the `if`’s condition never mentions the loop variable while the statement it guards indexes some collection by that variable ALONE (`c[i]`; `c[i + 1]` and `c[i, j]` are outside it). A finding additionally needs the AGREEING TWIN at the same-collection grain: the condition must read THAT SAME collection at a subscript that does not move — written into the condition, or reached through a local declared BEFORE the loop, so an alias bound inside the body is not followed. Both collection expressions must be simple identifiers. On a repository with no .NET source the same rule reads JavaScript/TypeScript off the engine’s own token stream (tests included, bundles and vendored paths not): a `for (let|var|const x = …; …; …)` with one declarator and a braced body, an alias followed only when it is declared before the loop in a block that encloses it and never assigned inside the loop. Deterministic, provable per finding. Advisory.
Other · Code Health — Whether exceptions are handled rather than silently swallowed or rethrown with lost stack traces.
Method: Roslyn syntax scan: every catch clause counted; empty catches and bare rethrows flagged. Population is all catch clauses, not estimated. Deterministic, hard fact.
An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it. (×15) — streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala:406, sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala:130, sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:143, …
`case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter. (×16) — sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/AutoCdcAuxiliaryTable.scala:88, sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:272, sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala:140, …
`case NonFatal(e)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter. — sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/memory.scala:96
An empty catch block silently discards the error — failures vanish with no log and no rethrow. On a teardown path letting it propagate is not an option (throwing out of `close()` masks the failure already in flight and abandons the rest of the cleanup), so make the swallow deliberate instead: narrow the catch to the exception this release can actually raise, and record it through whatever this codebase already uses to report problems — or, if it truly cannot matter, say why in a comment on the catch. — sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:186
`catch (e)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter. — REDACTED:1059
`case _: Exception` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter. (×4) — sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:167, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1616, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:44, …
`case _: RuntimeException` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter. — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:1377
`case _: Throwable` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter. — sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala:507
What to do
Swallowed exception (empty catch)
Do you agree with this assessment?
X30 · Support guard that admits what it rejects10.0 / 10Exemplary○ Nothing flagged
Other · Code Health — Whether a guard written as a NEGATED `||` says what its author meant — `!(a || b || x != k)` is `!a && !b && x == k` by De Morgan, so a bail-out that mixes capabilities the code needs with a fault it refuses turns inside out: it fires only where the capabilities are ABSENT, and lets every value the fault term names walk straight into the body that cannot handle it.
Method: Roslyn syntax only, no semantic model: every logical-not whose operand is a parenthesised `||` chain of two or more disjuncts, flattened (a left-nested `a || b || c` read once would see `(a || b)` as one disjunct). A site enters the population on that shape alone. A finding additionally needs the disjuncts to DISAGREE in polarity: at least one bare boolean read — an identifier or member access, never an invocation, which is a predicate rather than a capability flag — and at least one `x != <constant>`, the only form that negates into an exact-value pin (`== null` negates into a looser requirement and is outside the fault set). Consistently-polarised disjunctions, all-fault or all-capability, are counted and never reported; a negated `&&` is outside the population entirely. No same-receiver gate: it was measured to cost a real defect and remove no false positive. Deterministic, provable per finding. Advisory.
Other · Code Health — Whether log calls use message templates (queryable) rather than interpolated strings.
Method: Roslyn syntax scan: every log call-site counted; interpolated-string first-argument violations flagged. Population is all log calls, not estimated. On a repository with no .NET source it reads JavaScript/TypeScript off the engine’s own token stream (test, vendored and minified paths not) with the same rule: a `trace`/`debug`/`info`/`warn`/`error`/`fatal`/`verbose`/`silly`/`http`/`log` call on a receiver named for a logger, in a package whose own or an enclosing `package.json` declares a logger that carries values as fields (pino, pino-http, nestjs-pino, Fastify, winston, bunyan, tslog, LogTape, roarr), whose first deciding argument is a template literal with substitutions; an object-literal fields argument is stepped over, a plain string message clears the call. Python is read the same way: a `debug`/`info`/`warning`/`error`/`exception`/`critical`/`log` call on a logger in a file that imports the standard library’s `logging` or structlog (loguru, which formats at the call, is not judged), whose message is a whole f-string with a non-constant field; a `%`-formatted, `.format()`-ed or concatenated message is not convicted, as in C#. PHP is read the same way: a PSR-3 level call (`emergency`…`debug`, `log`) on a logger (`$logger`, `$this->logger`, Laravel’s `Log::`, `logger()`, `\Drupal::logger()`) in a package whose `composer.json` requires a PSR-3 logger or a framework that hands one out (psr/log, Monolog, Laravel, Symfony, Drupal, CakePHP), or a file that imports one, whose message is a double-quoted string interpolating a variable; a single-quoted message clears the call, and `sprintf` or `.` concatenation is not convicted, as in C#. Dart is read the same way: a `trace`/`debug`/`info`/`warn`/`error`/`fatal` call on Sentry’s structured logger (`Sentry.logger`) whose message is a whole string literal interpolating a value (`$name`, `${…}`); the `Sentry.logger.fmt` template API clears the call. Dart that logs only through loggers with one message and no fields (package:logging, package:logger, talker, dart:developer) is not applicable rather than a reach gap. Ruby is read the same way: a `trace`/`debug`/`info`/`warn`/`error`/`fatal` call on the console gem’s `Console` (declared by a Gemfile, gemspec or Gemfile.lock, or required) or, where SemanticLogger, Ougai or dry-logger is declared or required, on a receiver named for a logger (`logger`, `@logger`, `Rails.logger`, `log`), whose first deciding argument is, as the whole argument, a double-quoted string interpolating a non-constant value (`#{…}`, `#@ivar`); a plain literal clears the call, keyword arguments are the fields, and a block, `%`-format or `+` concatenation is not convicted. Ruby that logs only through the standard Logger (Rails.logger) is not applicable rather than a reach gap. Deterministic.
`debug` at line 314 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:314
`debug` at line 324 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:324
`debug` at line 1248 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1248
`debug` at line 1267 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1267
`debug` at line 1286 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1286
`debug` at line 1413 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1413
`debug` at line 1427 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1427
`debug` at line 1445 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1445
`debug` at line 1469 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1469
`debug` at line 1759 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1759
`debug` at line 1777 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1777
`warning` at line 1804 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.warning("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.warning("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1804
`debug` at line 1858 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1858
`debug` at line 1902 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1902
`debug` at line 1963 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:1963
`debug` at line 2446 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:2446
`debug` at line 2716 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:2716
`debug` at line 2746 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/sql/connect/client/core.py:2746
`warning` at line 232 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.warning("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.warning("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/ml/torch/distributor.py:232
`info` at line 637 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.info("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.info("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/ml/torch/distributor.py:637
`info` at line 646 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.info("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.info("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/ml/torch/distributor.py:646
`info` at line 856 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.info("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.info("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/ml/torch/distributor.py:856
`info` at line 872 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.info("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.info("order moved", order_id=order_id, status=status)` with structlog. — python/pyspark/ml/torch/distributor.py:872
`debug` at line 317 is handed a message assembled at the call — a string concatenated with a value, or `String.format` — through SLF4J or log4j 2, so the logger receives one finished string: the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the level is filtered out. Pass a constant pattern and the values beside it instead: `log.debug("Order {} moved to {}", orderId, status)`, which keeps the pattern and its arguments apart on the event for a structured encoder (logstash-logback-encoder, log4j 2's JsonTemplateLayout, OpenTelemetry), or `log.atInfo().addKeyValue("order_id", orderId).log("order moved")` with SLF4J 2's fluent API. An exception goes last, as the throwable argument, not into the text. — sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/LogDivertAppender.java:317
Other · Code Health — Whether Scala's null is held to the type system and not undermined by assertions — a file compiled with `-Yexplicit-nulls` is one where `String` excludes null and `String | Null` admits it, a file under WartRemover's `Null` wart or Scalafix's `DisableSyntax.noNulls` is one whose code cannot write `null` and must say `Option`, and each `.nn`, `import scala.language.unsafeNulls` or null-lint suppression is the author overruling that.
Method: Roslyn compiler-options scan: NullableContextOptions per project; null-forgiving (!) suppression density per 1k syntax nodes. Deterministic, adoption plus suppression penalty.
No build over this repository's 3346 Scala product file(s) passes `-Yexplicit-nulls`, enables WartRemover's `Null` wart or runs Scalafix's `DisableSyntax.noNulls`, so every reference type admits null and nothing reports a `null` written into the code or returned from a Java API — the first one to reach a method call is a `NullPointerException`. `Option` by convention is a habit no tool here holds the code to.
What to do
On Scala 3, add `-Yexplicit-nulls` to `scalacOptions` so `String` and `String | Null` mean different things, and type each Java API's result as nullable where it can return null; on Scala 2 (or as a first step), enable WartRemover's `Null` wart (`wartremoverErrors += Wart.Null`) or Scalafix's `DisableSyntax.noNulls = true` so the code holds absence in `Option`. Then pay the assertions down rather than adding to them: each `.nn` is a null the compiler could not rule out — wrap the Java value in `Option(…)` at the boundary, pattern-match on `null` once there, or change the API so the null cannot arrive; an `unsafeNulls` import switches the checking off for everything after it.
Do you agree with this assessment?
WCAG coverage — what static analysis assessed
Statically assessed 12 of 55 WCAG 2.2 Level A/AA success criteria (22%; ≈24% of the 50 WCAG 2.1 AA criteria for EN 301 549). The other 43 require runtime or manual evaluation. Partial signal only (a clean result is necessary, not sufficient; static analysis fully verifies none). This is accessibility readiness, not a conformance claim — a WCAG conformance claim requires manual evaluation (WCAG-EM 1.0).
The score is the rank-weighted fold of these lenses (worst-heaviest), each including its meta-dimensions; a lens with a Critical contributor is capped at Fair (its band reads "gated by …") and is never the strongest area however high its average.
Unscored — 2 check(s) recorded observations but carry no score
These checks ran and found something, but they do not carry a score — either by design (an advisory check reports evidence rather than grading it) or because they could not be scored here. They are excluded from the score for that reason, not because there was nothing to see.
X10 Duplicated predicate — 25 observation(s) recorded · Advisory — this card reports evidence and never carries a score.
X7 Silent fallback defaults — 7 observation(s) recorded · Advisory — this card reports evidence and never carries a score.
Not evidenced — 4 control(s) we could not find positive evidence for
These checks grade a working control, and the repository shows no evidence of one. That is deliberately not scored as a zero: a repository cannot show an ops runbook, a database TTL or an infrastructure-side audit log, so absence of evidence here is not evidence the control is missing. It is also not a statement that the check is irrelevant to this codebase — the thing it grades applies; we just could not see it. Excluded from the score either way.
C3 Audit Trail — Not assessed: these audit controls are read from a source model (declarative annotations, request middleware, entity/column names, guard methods) that was not loaded for this repository — because the repository is written in a language this check does not yet model, or because its projects failed to load. Absence of an idiom this check recognises is NOT evidence that this repository lacks audit controls: it may implement them entirely in its own ecosystem. This is a gap in the analyzer's language coverage, not a finding about this repository.
C4 Data Retention — Not assessed: these retention controls are read from a source model (declarative annotations, request middleware, entity/column names, guard methods) that was not loaded for this repository — because the repository is written in a language this check does not yet model, or because its projects failed to load. Absence of an idiom this check recognises is NOT evidence that this repository lacks retention controls: it may implement them entirely in its own ecosystem. This is a gap in the analyzer's language coverage, not a finding about this repository.
C5 Data-Subject Rights — Not assessed: these data-subject rights controls are read from a source model (declarative annotations, request middleware, entity/column names, guard methods) that was not loaded for this repository — because the repository is written in a language this check does not yet model, or because its projects failed to load. Absence of an idiom this check recognises is NOT evidence that this repository lacks data-subject rights controls: it may implement them entirely in its own ecosystem. This is a gap in the analyzer's language coverage, not a finding about this repository.
P5 DR & Backup — not evidenced — repo shows no backup/RTO/RPO controls; absence of evidence is not evidence of a working control
Not included — 41 check(s) not relevant to this codebase
These checks had nothing to measure here (no tests, no git history, the codebase is small, or the architecture style doesn't apply), so they're omitted above rather than scored low.
AC1 Text alternatives — No image/media element found in the parsed markup — AC1 not applicable here.
AX1 Captive dependencies — not analysed — these correctness checks read a source model that was not loaded for this repository, because the repository is written in a language this check does not yet model, or because its projects failed to load. This is a gap in the analyzer, not a finding about this repository
AX2 Stateful singletons — No container singleton was found, so there is no shared instance for concurrent requests to race on. No class in this Java code is a Spring, JSR-330 or CDI singleton bean. No function in this Python code is served by a threaded web framework (a Flask, Bottle or FastAPI route, a Django or Pyramid view), so no module is shared between request threads. No class in this Ruby code is a Rails controller, a Sinatra, Grape or Roda app, a Hanami action or a Rack app (an instance `call(env)`), so no request handler shares state between request threads. No class in this Scala code is a Play controller, a Cask routes object, a Scalatra servlet, a Finatra controller, an Akka/Pekko HTTP routes holder or a JSR-330/Spring singleton, so no instance is shared between request threads. TypeScript/JavaScript runs each process's requests on one event loop, so no two requests write a shared object at the same instant (interleaving across an await is a different defect).
AX5 Architecture & structure — not assessed — architecture style/structure is computed from a project graph (projects, types, module namespaces) that was not loaded for this repository, because the repository is written in a language this check does not yet model, or because its projects failed to load. This is a gap in the analyzer, not a finding about this repository
AX7 Slice cohesion — not applicable — not a vertical-slice architecture
AX8 Test isolation — no test/production split to check
AX9 CQS / query purity — no CQRS query handlers detected — query purity is not applicable to this codebase
AXB2 Runtime readiness — Advisory — this card reports evidence and never carries a score, so there is nothing missing here.
AXR1 Runtime accessibility — docker build failed (exit 1) — 22 | LABEL org.opencontainers.image.licenses="Apache-2.0" -------------------- ERROR: failed to solve: eclipse-temurin:25-jre: failed to resolve source metadata for docker.io/library/eclipse-temurin:25-jre: failed to do request: Head "https://registry-1.docker.io/v2/library/eclipse-temurin/manifests/25-jre": Forbidden; runtime evidence skipped This is a statement about this run, not a statement about your application: nothing here says the surface is inaccessible, only that it was never rendered.
C1 Data Protection — Not assessed: these personal data controls are read from a source model (declarative annotations, request middleware, entity/column names, guard methods) that was not loaded for this repository — because the repository is written in a language this check does not yet model, or because its projects failed to load. Absence of an idiom this check recognises is NOT evidence that this repository lacks personal data controls: it may implement them entirely in its own ecosystem. This is a gap in the analyzer's language coverage, not a finding about this repository.
C2 Access Controls — Not assessed: these authorization controls are read from a source model (declarative annotations, request middleware, entity/column names, guard methods) that was not loaded for this repository — because the repository is written in a language this check does not yet model, or because its projects failed to load. Absence of an idiom this check recognises is NOT evidence that this repository lacks authorization controls: it may implement them entirely in its own ecosystem. This is a gap in the analyzer's language coverage, not a finding about this repository.
D11 Test Reliability — No tests discovered
D18 Solution Shape — D18 scores the shape of a .NET solution; this repository has no .NET solution or project files, so the dimension does not apply.
D21 Naming Consistency — Naming consistency is measured on C# only; Java, JavaScript, Python, Ruby, Scala, html not measured.
D22 Internal API Consistency — The exposed public-API surface could not be collected — no C#/VB projects loaded.
D23 Boundary Type-Coupling — No bounded-context organisation was detected either — neither a context-shaped layout, nor 2+ sibling source directories each declaring an aggregate root, nor 2+ projects each declaring an entity of their own. Declaring this codebase's bounded contexts (≥2) would let cross-boundary type coupling be assessed. Declare them in `.codehealth/config.yaml` at the repository root (create it if absent), mapping each context name to the module-path or namespace prefixes that belong to it — e.g. `architecture:` → `contexts:` → `Billing: ["src/billing", "Acme.Billing"]`, `Catalog: ["src/catalog", "Acme.Catalog"]`.
D24 Comment Value — D24 reads C# and Visual Basic comments only — this repository's production source is .java, .py, .rb, .scala, which was left unread. Not scored: this is a gap in the analyzer, not a verdict about this repository.
D25 ADR Conformance — no ADRs to check
D27 Navigability — D27 counts a call into the standard library or a dependency as unresolved for this repository's language (.scala), so too few call sites counted as resolved to assess navigability. Not scored: this is a gap in the analyzer, not a verdict about this repository.
D32 Data Compliance (PII/GDPR) — 48 file(s) were not parsed by semgrep — the PII/GDPR ruleset never ran over them
D39 IL Efficiency — D39 measures the IL emitted by a .NET build; this repository has no .NET solution or project files, so the dimension does not apply.
D40 Network Egress Confinement — No Kubernetes/orchestration workloads found in the repository manifests; network egress policy is a cluster-native control that may live at the platform/firewall layer, so there is nothing to assess here.
D41 Kernel & Syscall Confinement — No Kubernetes/orchestration workloads found in the repository manifests; seccomp/AppArmor/SELinux confinement is a workload-level control, so there is nothing to assess here.
D42 Runtime Threat Enforcement — No Kubernetes/orchestration workloads found in the repository manifests; runtime threat-detection and admission-control policy are cluster-level controls, so there is nothing to assess here.
D7 Architectural Integrity — no checkable ADRs, and no project-reference graph for the cycle pass to read — so this dimension makes no claim about dependency cycles in either direction (where this repository's language has an import-cycle lens, cycles are reported there). Architectural integrity not assessed
D8 Code Coverage — Coverage NOT MEASURED: the JavaScript/TypeScript half could not be measured — the jest suite in ui-test/ ran and not one test passed (FAIL tests/utils.test.js), so the coverage would describe the failed run, not the code. Coverage is excluded from the score rather than counted as a near-zero. The named suite step is one the repository's maintainers can perform; once it passes, the real number is measured on the next scan. Alternatively, commit the lcov/Cobertura report your CI produces and it is read without a re-run.
DM1 Domain Modelling — applicable but not scored (4 signals for this style, 2 needed — the count is met but a required primary signal is absent): 35 aggregate root(s) (types guarding their own state behind command methods — this language has no AggregateRoot base to inherit); 1835 value object(s); 21 domain event(s); its 35 aggregate(s) are ORM models only — persistence, not behaviour: a CRUD model, measured, not claimed as DDD; its domain events are published by services or handlers — no domain entity raises one; a Domain/Aggregates/ValueObjects layer; a domain structure with no aggregate, domain event or strongly-typed id in it: a CRUD / clean-architecture model, measured but not claimed as DDD; a top-level examples/ tree: a library or framework whose examples are its consumers — these types are building blocks, not a domain
ED1 Event-Driven — not scored — this repository shows none of the 3 signals this lens looks for
ED5 Idempotency — no mutating command handlers or message consumers detected — idempotency check not applicable
ES1 Event Sourcing — not scored — this repository shows none of the 3 signals this lens looks for
M4 Documentation accuracy — Not applicable: the 200 current-state document(s) make no claim of a kind M4 checks. 212 claim(s) of kinds M4 does not check were not checked (path 41, command argument 5, environment variable 160, setting 6); M4 checks only links and versions.
P12 CI test-gate honesty — Reported, not scored — this card publishes what the CI gate does with the test inventory rather than grading it. The findings above are its output.
P8 Schema migrations — no ORM, schema-migration tool or schema auto-create was found in this repository's dependency manifests or source, so there is no database schema for this check to judge
P9 Domain vs controller coverage — no coverage report found on disk — produce a coverage report in a standard format (`addSbtPlugin("org.scoverage" % "sbt-scoverage" % "<version>")` in `project/plugins.sbt`, then `sbt clean coverage test coverageReport`) and commit it — a hosted scan measures a clone of the repository, so a report that exists only in a working tree, a CI runner's or your own, never reaches it; the artefact is commonly gitignored, so `git add -f` that one file (or un-ignore its path) and commit it alongside the code it measures, or wire coverage collection into CI, to enable this cross-layer check
R11 Import Boundaries — No recognizable feature-sliced/layered src layout — boundary rules not applicable.
S1 Web-Security Posture — Not assessed: these web-security controls are read from a source model (declarative annotations, request middleware, entity/column names, guard methods) that was not loaded for this repository — because the repository is written in a language this check does not yet model, or because its projects failed to load. Absence of an idiom this check recognises is NOT evidence that this repository lacks web-security controls: it may implement them entirely in its own ecosystem. This is a gap in the analyzer's language coverage, not a finding about this repository.
SC1 Supply-chain hygiene — Advisory — this card reports evidence and never carries a score, so there is nothing missing here.
X2 Cancellation propagation — no C# was loaded; this repository's Ruby, Scala, Java and Python were read, and no request in it can be cancelled by a caller — a Ruby request takes no token: Net::HTTP, Faraday and async-http are stopped from outside the call (Async::Task#stop, Thread#raise, Timeout), not through a parameter; a Scala request takes no token: a Future cannot be cancelled, and cats-effect, ZIO and Monix interrupt the fiber that made the request from outside the call, not through a parameter; a Java request off Android takes no token: it is cancelled through the handle it returns (Future.cancel, Call.cancel, a Disposable), ambiently (io.grpc.Context) or by interrupting its thread, not through a parameter; a Python request takes no token: asyncio cancels the task and trio or anyio the scope that made it, from outside the call, and requests, httpx, aiohttp and gRPC take a timeout, not a cancellation
X32 Type resolved by simple name across every loaded assembly — This check is about how a .NET program searches the assemblies loaded into its process for a type, and this repository contains no .NET source, so there is nothing here for it to assess. Not a gap in the analyzer and not a finding about your code.
X6 Hand-rolled structured-format parsing — Advisory — this card reports evidence and never carries a score, so there is nothing missing here.
X9 Subsumed condition operand — Advisory — this card reports evidence and never carries a score, so there is nothing missing here.
Appendix A — Findings (grouped)
The findings behind the scores, grouped by severity, then by dimension and kind. The high-severity issues are enumerated in full below; items per group are capped at 25 with any overflow stated explicitly per group, never silently truncated. The complete machine-readable list of every finding (all severities) is the companion findings.md in this report's bundle.
Orphaned knowledge python/pyspark/mllib/linalg/distributed.py:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge sql/api/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge python/pyspark/pandas/spark/accessors.py:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaPairDStream.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge python/pyspark/ml/functions.py:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge core/src/main/scala/org/apache/spark/status/protobuf/StageDataWrapperSerializer.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge python/pyspark/streaming/dstream.py:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge sql/connect/common/src/main/scala/org/apache/spark/sql/connect/KeyValueGroupedDataset.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge python/pyspark/mllib/feature.py:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge streaming/src/main/scala/org/apache/spark/streaming/api/java/JavaStreamingContext.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge mllib/src/main/scala/org/apache/spark/mllib/random/RandomRDDs.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge python/pyspark/mllib/tree.py:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge python/pyspark/sql/streaming/tws_tester.py:1— No living knowledge remains for this large file — its last meaningful change has decayed away, so if it breaks, no one currently understands it. It does carry its own tests, so the behaviour is pinned even though the understanding is gone: schedule a read-through, using those tests as the specification, before the next change lands here.
Orphaned knowledge sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAOptimizer.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkIntervalUtils.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge python/pyspark/mllib/util.py:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge sql/api/src/main/scala/org/apache/spark/sql/DataFrameReader.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge core/src/main/scala/org/apache/spark/status/storeTypes.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/tupleSketchAgg.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
Orphaned knowledge mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala:1— No living knowledge remains for this large file — its last meaningful change has decayed away; if it breaks, no one currently understands it. Schedule a read-through / add characterisation tests before it bites.
AC2 · Forms & labels· <button> with no accessible text · ×1
<button> with no accessible text core/src/main/resources/org/apache/spark/ui/static/environmentpage.js:206— A button with no text and no aria-label has no accessible name. Add visible text or an aria-label (an icon-only button still needs one).
AC2 · Forms & labels· <select> without a programmatic label · ×1
<select> without a programmatic label sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:37— This control has no associated label. Add a <label for> / wrapping <label> / aria-label / aria-labelledby so assistive tech can name it.
AC2 · Forms & labels· <input> without a programmatic label · ×1
<input> without a programmatic label REDACTED:468— This control has only a placeholder — a placeholder is not a label (it vanishes on input and many AT ignore it). Add a <label for>, a wrapping <label>, or aria-label.
<html> without a lang licenses-binary/LICENSE-javassist.html:1— The page declares no language, so assistive tech can't pick the right pronunciation. Add lang (e.g. lang="en").
Boundary-crossing change coupling: __init__.py ↔ VariantExpressionEvalUtils.scala python/pyspark/sql/functions/__init__.py:1— `python/pyspark/sql/functions/__init__.py` (context python) and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala` (context sql) sit in DIFFERENT parts of the tree yet change together 50% of the time (11 of the 22 commits that touched whichever of the two files changed less often, counting a file under its earlier names as well, and counted over this repository's 10,000 most recent commits rather than its whole history — a repo-wide or module-wide sweep is evidence about the sweep rather than about any pair inside it and is left out of BOTH sides of this ratio, while a dependency bump, a formatter/rename sweep, or a commit whose edit to one of the two files was a tool directive such as //go:generate or whitespace only is left out of the shared count ONLY, so the two sides are not taken over identical commit sets) — the bounded-context boundary may be in the wrong place, or one context is leaking into the other. This is the behavioural boundary violation a static scan can't see. You can check this without leaving the row: of the 11 shared commits counted here, the most recent 3 are `ba4441f3` [SPARK-59130][SQL] Add `variant_pick` expression; `27404938` [SPARK-58062][SQL] Add `variant_strip_nulls` expression; `5d23405f` [SPARK-51975][SQL] Add variant_from_arrays and variant_from_entries — run `git show` on any of them.
TodoComment udf/worker/grpc/src/main/scala/org/apache/spark/udf/worker/grpc/GrpcWorkerSession.scala:140— // TODO [SPARK-55278]: add application-level gRPC flow control in a follow-up,
TodoComment udf/worker/grpc/src/main/scala/org/apache/spark/udf/worker/grpc/GrpcWorkerSession.scala:143— // TODO [SPARK-57324]: expose queue depth as a metric (early warning for a
TodoComment udf/worker/grpc/src/main/scala/org/apache/spark/udf/worker/grpc/GrpcWorkerSession.scala:609— // TODO [SPARK-57640]: close() is a finalizer often already on the task
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala:67— // TODO: Connection pooling -- reuse idle workers across sessions. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala:68— // TODO: Security scope isolation -- partition pool by WorkerSecurityScope. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala:136— // TODO [SPARK-55278]: extract the env state machine + JVM shutdown hook
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala:376— // TODO [SPARK-55278]: Cleanup sessions as well?
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala:377— // TODO [SPARK-55278]: close workers in parallel -- today shutdown is serialised at
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala:401— // TODO: distinguish retriable vs permanent environment failures. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala:420— // the future predicate (see TODO above). — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala:441— // TODO: share one JVM shutdown hook across all dispatchers in the — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerProcess.scala:58— // TODO: idle-timeout tracking and concurrent session capacity. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment streaming/src/main/scala/org/apache/spark/streaming/scheduler/StreamingListenerBus.scala:98— // events (SPARK-12140). TODO Once SPARK-12140 is resolved we should set it to true.
TodoComment streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala:412— // TODO: merge callsites with scopes so we can just reuse the code there — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment sql/gen-sql-functions-docs.py:76— # TODO(SPARK-31499): Needs to add a column to describe arguments and their types
TodoComment sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/graph/SqlPipelineSuite.scala:476— // TODO: streaming file data sources in SQL is not currently supported. If and when it is, — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/util/SchemaMergingUtils.scala:43— // TODO (SPARK-59268): don't drop changes to comments and field metadata `desiredSchema`, or
TodoComment sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala:697— // TODO (SPARK-57670): Property removal is intentionally not handled here: a property dropped
TodoComment sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:1350— // TODO: This GC triggers a full scan of the aux table; revisit whether to GC only — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala:1480— // TODO: Need a better way to drop the view. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/SparkMetadataOperationSuite.scala:713— // TODO(SPARK-36180): add hive table here too
TodoComment sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala:350— // TODO: Shall use TIMESTAMPLOCALTZ_TYPE, keep AS-IS now for — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/CLIServiceClient.java:40— // TODO: provide STATIC default value — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
TodoComment sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/FetchOrientation.java:46— // TODO: Should this really default to FETCH_NEXT? — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FileTooLong: internal/SQLConf.scala sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala— FileTooLong — 8856 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 8356 over it, 17.71× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: pandas/frame.py python/pyspark/pandas/frame.py— FileTooLong — 5867 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 5367 over it, 11.73× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: parser/AstBuilder.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala— FileTooLong — 5010 significant lines (blank, comment-only and punctuation-only lines excluded), about 99% of them inside a single declaration: AstBuilder (68-8541). The bar is 500 significant lines; this is 4510 over it, 10.02× the bar. Moving the declarations that sit BESIDE it into sibling files will not shorten this file. Extract from INSIDE that declaration instead: lift each cohesive group of its body — the parts that share the same inputs and are named together — into its own unit in a sibling file, and have the original call them.
FileTooLong: REDACTED REDACTED— FileTooLong — 4470 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 3970 over it, 8.94× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: expressions/datetimeExpressions.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala— FileTooLong — 3938 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 3438 over it, 7.88× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: errors/QueryCompilationErrors.scala sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala— FileTooLong — 3722 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 3222 over it, 7.44× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: planner/SparkConnectPlanner.scala sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala— FileTooLong — 3393 significant lines (blank, comment-only and punctuation-only lines excluded), about 98% of them inside a single declaration: SparkConnectPlanner (91-4731). The bar is 500 significant lines; this is 2893 over it, 6.79× the bar. Moving the declarations that sit BESIDE it into sibling files will not shorten this file. Extract from INSIDE that declaration instead: lift each cohesive group of its body — the parts that share the same inputs and are named together — into its own unit in a sibling file, and have the original call them.
FileTooLong: functions/builtin.py python/pyspark/sql/connect/functions/builtin.py— FileTooLong — 3320 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 2820 over it, 6.64× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: expressions/stringExpressions.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala— FileTooLong — 3103 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 2603 over it, 6.21× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: config/package.scala core/src/main/scala/org/apache/spark/internal/config/package.scala— FileTooLong — 2957 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 2457 over it, 5.91× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: analysis/Analyzer.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala— FileTooLong — 2745 significant lines (blank, comment-only and punctuation-only lines excluded), about 88% of them inside a single declaration: Analyzer (403-4595). The bar is 500 significant lines; this is 2245 over it, 5.49× the bar. Moving the declarations that sit BESIDE it into sibling files will not shorten this file. Extract from INSIDE that declaration instead: lift each cohesive group of its body — the parts that share the same inputs and are named together — into its own unit in a sibling file, and have the original call them.
FileTooLong: scheduler/DAGScheduler.scala core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala— FileTooLong — 2739 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 2239 over it, 5.48× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: errors/QueryExecutionErrors.scala sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala— FileTooLong — 2698 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 2198 over it, 5.40× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: ml/feature.py python/pyspark/ml/feature.py— FileTooLong — 2505 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 2005 over it, 5.01× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: pandas/series.py python/pyspark/pandas/series.py— FileTooLong — 2382 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1882 over it, 4.76× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: sql/types.py python/pyspark/sql/types.py— FileTooLong — 2328 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1828 over it, 4.66× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: expressions/Cast.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala— FileTooLong — 2163 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1663 over it, 4.33× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: connect/plan.py python/pyspark/sql/connect/plan.py— FileTooLong — 2145 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1645 over it, 4.29× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: ml/classification.py python/pyspark/ml/classification.py— FileTooLong — 2077 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1577 over it, 4.15× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: sql/functions.scala sql/api/src/main/scala/org/apache/spark/sql/functions.scala— FileTooLong — 2060 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1560 over it, 4.12× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: connect/dataframe.py python/pyspark/sql/connect/dataframe.py— FileTooLong — 1991 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1491 over it, 3.98× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: optimizer/Optimizer.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala— FileTooLong — 1952 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1452 over it, 3.90× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: REDACTED REDACTED— FileTooLong — 1913 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1413 over it, 3.83× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: client/core.py python/pyspark/sql/connect/client/core.py— FileTooLong — 1870 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1370 over it, 3.74× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
FileTooLong: pandas/generic.py python/pyspark/pandas/generic.py— FileTooLong — 1857 significant lines (blank, comment-only and punctuation-only lines excluded). The bar is 500 significant lines; this is 1357 over it, 3.71× the bar. To reduce it, split the file along the responsibilities already in it: move each cohesive group of declarations into its own sibling file in the same module or package, so no one file has to be read whole to change one of them.
TooManyMethods: functions sql/api/src/main/scala/org/apache/spark/sql/functions.scala:87— TooManyMethods — 1051 methods. The bar is 30 methods; this is 1021 over it, 35.03× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: QueryCompilationErrors sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:54— TooManyMethods — 642 methods. The bar is 30 methods; this is 612 over it, 21.40× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: QueryExecutionErrors sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala:67— TooManyMethods — 430 methods. The bar is 30 methods; this is 400 over it, 14.33× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: SparkConnectPlanner sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:91— TooManyMethods — 208 methods. The bar is 30 methods; this is 178 over it, 6.93× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: Dataset sql/api/src/main/scala/org/apache/spark/sql/Dataset.scala:124— TooManyMethods — 205 methods. The bar is 30 methods; this is 175 over it, 6.83× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: DataFrame python/pyspark/pandas/frame.py:389— TooManyMethods — 197 methods. The bar is 30 methods; this is 167 over it, 6.57× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: SparkContext core/src/main/scala/org/apache/spark/SparkContext.scala:88— TooManyMethods — 181 methods. The bar is 30 methods; this is 151 over it, 6.03× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: Utils core/src/main/scala/org/apache/spark/util/Utils.scala:97— TooManyMethods — 173 methods. The bar is 30 methods; this is 143 over it, 5.77× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: SessionCatalog REDACTED:87— TooManyMethods — 170 methods. The bar is 30 methods; this is 140 over it, 5.67× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: DAGScheduler core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:124— TooManyMethods — 154 methods. The bar is 30 methods; this is 124 over it, 5.13× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: Dataset sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:230— TooManyMethods — 151 methods. The bar is 30 methods; this is 121 over it, 5.03× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: AstBuilder sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:68— TooManyMethods — 144 methods. The bar is 30 methods; this is 114 over it, 4.80× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: QueryParsingErrors sql/api/src/main/scala/org/apache/spark/sql/errors/QueryParsingErrors.scala:33— TooManyMethods — 141 methods. The bar is 30 methods; this is 111 over it, 4.70× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: RDD python/pyspark/core/rdd.py:197— TooManyMethods — 135 methods. The bar is 30 methods; this is 105 over it, 4.50× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: DataFrame python/pyspark/sql/classic/dataframe.py:113— TooManyMethods — 135 methods. The bar is 30 methods; this is 105 over it, 4.50× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: Series python/pyspark/pandas/series.py:393— TooManyMethods — 132 methods. The bar is 30 methods; this is 102 over it, 4.40× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: DataFrame python/pyspark/sql/connect/dataframe.py:115— TooManyMethods — 131 methods. The bar is 30 methods; this is 101 over it, 4.37× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: RDD core/src/main/scala/org/apache/spark/rdd/RDD.scala:84— TooManyMethods — 127 methods. The bar is 30 methods; this is 97 over it, 4.23× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: Dataset sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala:140— TooManyMethods — 126 methods. The bar is 30 methods; this is 96 over it, 4.20× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: UTF8String common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:55— TooManyMethods — 118 methods. The bar is 30 methods; this is 88 over it, 3.93× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: WritableColumnVector sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java:55— TooManyMethods — 109 methods. The bar is 30 methods; this is 79 over it, 3.63× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: JavaPairRDD core/src/main/scala/org/apache/spark/api/java/JavaPairRDD.scala:47— TooManyMethods — 96 methods. The bar is 30 methods; this is 66 over it, 3.20× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: JsonProtocol core/src/main/scala/org/apache/spark/util/JsonProtocol.scala:66— TooManyMethods — 94 methods. The bar is 30 methods; this is 64 over it, 3.13× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: DateTimeUtils sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala:41— TooManyMethods — 91 methods. The bar is 30 methods; this is 61 over it, 3.03× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
TooManyMethods: Column sql/api/src/main/scala/org/apache/spark/sql/Column.scala:147— TooManyMethods — 89 methods. The bar is 30 methods; this is 59 over it, 2.97× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
Duplicated block (5 lines × 2) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:262— connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:262-266 | sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:191-195 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:262` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (5 lines × 2) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:338— connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:338-342 | connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:348-352 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (5 lines × 2) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala:102— connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala:102-106 | connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala:109-113 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala:102` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:39— core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:39-43 | core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:65-69 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (5 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:988— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:988-992 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:849-853 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:988` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1065— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1065-1069 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:926-930 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1065` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1070— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1070-1074 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:931-935 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1070` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala:204— mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala:204-208 | mllib/src/main/scala/org/apache/spark/ml/regression/FMRegressor.scala:427-431 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala` and `mllib/src/main/scala/org/apache/spark/ml/regression/FMRegressor.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala:204` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (5 lines × 2) mllib/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala:72— mllib/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala:72-76 | mllib/src/main/scala/org/apache/spark/ml/feature/MinMaxScaler.scala:117-121 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (5 lines × 2) mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala:147— mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala:147-151 | mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala:165-169 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala:147` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala:161` calls `getTypeString` and `mllib/src/main/scala/org/apache/spark/ml/param/shared/SharedParamsCodeGen.scala:146` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (5 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:152— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:152-156 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1497-1501 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:152` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:195— mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:195-199 | mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:240-244 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:195` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:201— mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:201-205 | mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:246-250 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:201` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:207` calls `Since` and `mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:252` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (5 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala:827— mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala:827-831 | mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala:930-934 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala:827` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlreadyExistException.scala:45— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlreadyExistException.scala:45-49 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NoSuchItemException.scala:45-50 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala:123— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala:123-127 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala:145-149 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveChangelogTable.scala:123` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTables.scala:45— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTables.scala:45-49 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTables.scala:58-62 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:915— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:915-919 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:947-951 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:921` calls `TimestampAddInterval`, `StringTypeExpression`, `Cast` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:953` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1170— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1170-1174 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1205-1209 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1170` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:482— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:482-486 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:811-815 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:482` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:291— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:291-295 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:617-621 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:291` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note first that the copies are not typed on the same thing: the declarations holding them bind `child` to `Expression) extends UnaryMathExpression(math.cbrt` in one and `Expression) extends UnaryMathExpression(StrictMath.expm1` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:722— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:722-726 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala:809-813 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:722` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala:602— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala:602-606 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala:654-658 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala:602` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala:381— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala:381-385 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala:391-395 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala:381` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 2) sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:450— sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:450-454 | sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:560-564 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:450` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala:421— core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala:421-426 | core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala:490-495 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala:421` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala:447— core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala:447-452 | core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala:516-521 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala:447` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:53— core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:53-58 | core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:56-61 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:53` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2792— core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2792-2797 | core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2872-2877 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (6 lines × 2) core/src/main/scala/org/apache/spark/status/protobuf/StageDataWrapperSerializer.scala:642— core/src/main/scala/org/apache/spark/status/protobuf/StageDataWrapperSerializer.scala:642-647 | core/src/main/scala/org/apache/spark/status/protobuf/TaskDataWrapperSerializer.scala:97-102 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (6 lines × 2) graphx/src/main/scala/org/apache/spark/graphx/impl/ReplicatedVertexView.scala:70— graphx/src/main/scala/org/apache/spark/graphx/impl/ReplicatedVertexView.scala:70-75 | graphx/src/main/scala/org/apache/spark/graphx/impl/ReplicatedVertexView.scala:112-117 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (6 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:265— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:265-270 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:267-272 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:265` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:485— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:485-490 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:418-423 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:198— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:198-203 | mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:522-527 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala` and `mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 65 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:198` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:296— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:296-301 | mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:541-546 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:296` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:302— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:302-307 | mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:547-552 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:302` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:954— mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:954-959 | mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala:296-301 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:954` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:960— mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:960-965 | mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala:302-307 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:960` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:158— mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:158-163 | mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:143-148 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:158` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala:122— mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala:122-127 | mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala:165-172 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala:42— mllib/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala:42-47 | mllib/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala:65-70 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/ml/feature/StandardScaler.scala:72` calls `setDefault` and `mllib/src/main/scala/org/apache/spark/ml/feature/MaxAbsScaler.scala:48` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/linalg/JsonVectorConverter.scala:32— mllib/src/main/scala/org/apache/spark/ml/linalg/JsonVectorConverter.scala:32-37 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:435-440 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/linalg/JsonVectorConverter.scala:32` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:108— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:108-113 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1453-1458 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:227— mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:227-232 | mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:265-270 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:186— mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:186-191 | mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:221-226 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (6 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala:69— mllib/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala:69-74 | mllib/src/main/scala/org/apache/spark/mllib/pmml/export/GeneralizedLinearPMMLModelExport.scala:55-60 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/pmml/export/BinaryClassificationPMMLModelExport.scala:69` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StringLiteralTypeCoercion.scala:31— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StringLiteralTypeCoercion.scala:31-36 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StringLiteralTypeCoercion.scala:39-44 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (6 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:575— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:575-580 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:607-612 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:575` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (6 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala:172— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala:172-177 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/thetasketchesAggregates.scala:232-237 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/datasketchesAggregates.scala:172` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:248— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:248-253 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:255-260 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:392— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala changed 15 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 157 in CheckAnalysis.checkAnalysis0 at line 392. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1741— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala changed 24 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 62 in ResolveReferences.doApply at line 1741. 3 of those changes were fix/bug commits, and the other 21 changed it for other reasons — this file is under both repair and feature pressure. Before the next change lands here, make sure the area it touches is under test, then split that area out of the file so the following change is smaller than this one — a file this often edited pays the complexity back every time. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: REDACTED REDACTED:2332— REDACTED changed 21 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 59 in merge_spark_pr.main at line 2332. 6 of those changes were fix/bug commits, and the other 15 changed it for other reasons — this file is under both repair and feature pressure. Before the next change lands here, make sure the area it touches is under test, then split that area out of the file so the following change is smaller than this one — a file this often edited pays the complexity back every time. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- REDACTED`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:3891— core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala changed 15 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 79 in DAGScheduler.handleTaskCompletion at line 3891. 1 of those changes was a fix/bug commit, and the other 14 changed it for other reasons — this file is under both repair and feature pressure. Before the next change lands here, make sure the area it touches is under test, then split that area out of the file so the following change is smaller than this one — a file this often edited pays the complexity back every time. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala:615— sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala changed 26 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 35 in EnsureRequirements.checkKeyGroupCompatible at line 615. 9 of those changes were fix/bug commits, and the other 17 changed it for other reasons — this file is under both repair and feature pressure. Before the next change lands here, make sure the area it touches is under test, then split that area out of the file so the following change is smaller than this one — a file this often edited pays the complexity back every time. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:130— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala changed 7 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 126 in DataSourceV2Strategy.apply at line 130. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:1099— sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala changed 11 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 74 in BasicOperators.apply at line 1099. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:2600— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala changed 24 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 25 in AstBuilder.withJoinRelation at line 2600. 3 of those changes were fix/bug commits, and the other 21 changed it for other reasons — this file is under both repair and feature pressure. Before the next change lands here, make sure the area it touches is under test, then split that area out of the file so the following change is smaller than this one — a file this often edited pays the complexity back every time. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:191— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala changed 5 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 110 in UnsupportedOperationChecker.checkForStreaming at line 191. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/core/src/main/scala/org/apache/spark/sql/execution/planmerging/PlanMerger.scala sql/core/src/main/scala/org/apache/spark/sql/execution/planmerging/PlanMerger.scala:404— sql/core/src/main/scala/org/apache/spark/sql/execution/planmerging/PlanMerger.scala changed 9 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 55 in PlanMerger.tryMergePlans at line 404. 2 of those changes were fix/bug commits, and the other 7 changed it for other reasons — this file is under both repair and feature pressure. Before the next change lands here, make sure the area it touches is under test, then split that area out of the file so the following change is smaller than this one — a file this often edited pays the complexity back every time. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/core/src/main/scala/org/apache/spark/sql/execution/planmerging/PlanMerger.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:113— core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala changed 11 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 39 in PythonEvalType.toString at line 113. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala:105— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala changed 3 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 132 in V2ExpressionBuilder.generateExpression at line 105. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: python/pyspark/sql/transpile.py python/pyspark/sql/transpile.py:562— python/pyspark/sql/transpile.py changed 7 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 56 in CatalystTranspiler._convert_chunk at line 562. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- python/pyspark/sql/transpile.py`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: REDACTED REDACTED:56— REDACTED changed 3 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 124 in ResolveSessionCatalog.apply at line 56. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- REDACTED`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:260— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala changed 5 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 68 in JacksonParser.makeConverter at line 260. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: python/pyspark/sql/connect/client/core.py python/pyspark/sql/connect/client/core.py:1592— python/pyspark/sql/connect/client/core.py changed 13 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 26 in SparkConnectClient._analyze at line 1592. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- python/pyspark/sql/connect/client/core.py`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:1307— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala changed 18 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 17 in ConvertToCatalyst.applyExpr at line 1307. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: python/pyspark/sql/conversion.py python/pyspark/sql/conversion.py:674— python/pyspark/sql/conversion.py changed 13 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 23 in LocalDataToArrowConversion._create_converter at line 674. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- python/pyspark/sql/conversion.py`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:172— sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala changed 4 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 70 in AvroDeserializer.newWriter at line 172. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:1204— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala changed 18 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 15 in JsonQuery.sqlString at line 1204. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:211— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala changed 6 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 44 in JdbcUtils.getCatalystType at line 211. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala:1192— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala changed 8 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 32 in MicroBatchExecution.runBatch at line 1192. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:1093— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala changed 5 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 49 in ParquetFilters.createFilterHelper at line 1093. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: python/pyspark/sql/types.py python/pyspark/sql/types.py:2673— python/pyspark/sql/types.py changed 5 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 42 in types._parse_datatype_json_value at line 2673. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- python/pyspark/sql/types.py`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Hotspot: sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:628— sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala changed 12 times in last 90 days, and the most complex body those changes touched has cyclomatic complexity 17 in DataSourceV2Relation.transformV2Stats at line 628. Frequent change and high complexity in one file compound: schedule the next change to it to include carving out the part being edited, with the area under test before it moves. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
Duplicated block (8 lines × 2) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufSerializer.scala:89— connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufSerializer.scala:89-96 | sql/core/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala:111-118 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (8 lines × 2) core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala:325— core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala:325-333 | core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala:385-392 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala:325` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (8 lines × 2) core/src/main/scala/org/apache/spark/partial/MeanEvaluator.scala:48— core/src/main/scala/org/apache/spark/partial/MeanEvaluator.scala:48-57 | core/src/main/scala/org/apache/spark/partial/SumEvaluator.scala:73-80 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (8 lines × 2) core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala:619— core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala:619-626 | core/src/main/scala/org/apache/spark/storage/FallbackStorage.scala:181-188 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala:619` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:712— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:712-719 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:642-649 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (8 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:360— mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:360-367 | mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala:314-321 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:360` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala:509— mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala:509-516 | mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:391-399 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (8 lines × 2) mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala:507— mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala:507-514 | mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala:656-663 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (8 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:91— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:91-98 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1436-1443 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (8 lines × 2) mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala:117— mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala:117-124 | mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:137-144 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala:117` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:591— mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:591-598 | mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala:130-137 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala` and `mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 46 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:591` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:145— mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:145-152 | mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:180-187 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:145` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:307— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:307-314 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18236-18243 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:307` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:339— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:339-346 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18268-18275 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:339` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:373— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:373-380 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18302-18309 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:373` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:409— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:409-416 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18339-18346 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:409` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:46— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:46-53 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSetVariable.scala:46-53 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (8 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala:224— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala:224-231 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala:366-373 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (8 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:1713— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:1713-1720 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:2683-2690 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (8 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:351— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:351-358 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala:279-286 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (8 lines × 2) REDACTED:971— REDACTED:971-978 | REDACTED:985-992 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:971` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:107— sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:107-114 | sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:124-131 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:107` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ValidateSparkPlan.scala:44— sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ValidateSparkPlan.scala:44-51 | sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ValidateSparkPlan.scala:53-60 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (8 lines × 2) sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:731— sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:731-738 | sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:746-753 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:731` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 2) sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:1255— sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:1255-1262 | sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:1475-1482 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (9 lines × 2) core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala:272— core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala:272-280 | core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala:349-357 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (9 lines × 2) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:542— core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:542-550 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala:153-161 — before extracting anything, compare `core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala` and `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 95 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (9 lines × 2) core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala:84— core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala:84-92 | core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala:113-121 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/metrics/sink/PrometheusServlet.scala:84` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:506— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:506-514 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:442-450 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (9 lines × 2) mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala:312— mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala:312-320 | mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala:514-522 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala:312` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:139— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:139-147 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1484-1492 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (9 lines × 2) mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala:435— mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala:435-443 | mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala:398-406 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala:435` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala:445` calls `Path` and `mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala:407` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (9 lines × 2) resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala:371— resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala:371-380 | resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala:386-394 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala:371` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:259— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:259-267 | sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:450-458 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:259` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:92— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:92-100 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:249-257 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:92` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1240— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1240-1248 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1304-1312 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1240` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:703— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:703-711 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1196-1204 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:703` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:897— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:897-905 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:916-924 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:897` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala:589— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala:589-597 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala:623-631 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (9 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:1894— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:1894-1902 | sql/core/src/main/scala/org/apache/spark/sql/api/python/PythonSQLUtils.scala:203-211 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:1894` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:124— sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:124-132 | sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:144-152 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:124` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:142` calls `castToImpl` and `sql/core/src/main/scala/org/apache/spark/sql/classic/KeyValueGroupedDataset.scala:123` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (9 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveSubqueries.scala:38— sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveSubqueries.scala:38-46 | sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:462-470 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (9 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:39— sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/CloseCursorExec.scala:39-47 | sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:51-59 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (9 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:217— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:217-225 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:234-242 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:217` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:74— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:74-82 | sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileOperator.scala:138-146 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (9 lines × 2) REDACTED:351— REDACTED:351-359 | REDACTED:369-377 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:351` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1750— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1750-1759 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1838-1846 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1750` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (9 lines × 2) REDACTED:173— REDACTED:173-181 | REDACTED:346-354 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:173` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) streaming/src/main/scala/org/apache/spark/streaming/rdd/MapWithStateRDD.scala:197— streaming/src/main/scala/org/apache/spark/streaming/rdd/MapWithStateRDD.scala:197-205 | streaming/src/main/scala/org/apache/spark/streaming/rdd/MapWithStateRDD.scala:219-227 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `streaming/src/main/scala/org/apache/spark/streaming/rdd/MapWithStateRDD.scala:197` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 2) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1122— core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1122-1130 | core/src/main/scala/org/apache/spark/api/r/BaseRRunner.scala:120-128 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (7 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:160— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:160-166 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala:201-207 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 90 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:160` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) core/src/main/scala/org/apache/spark/deploy/Utils.scala:72— core/src/main/scala/org/apache/spark/deploy/Utils.scala:72-78 | core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:156-162 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/deploy/Utils.scala:72` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `core/src/main/scala/org/apache/spark/deploy/Utils.scala:70` calls `File`, `normalize` and `core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:155` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (7 lines × 2) core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:340— core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:340-346 | core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:411-417 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:340` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:156— core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:156-162 | core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:206-213 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:156` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (7 lines × 2) core/src/main/scala/org/apache/spark/scheduler/ExecutorResourcesAmounts.scala:82— core/src/main/scala/org/apache/spark/scheduler/ExecutorResourcesAmounts.scala:82-88 | core/src/main/scala/org/apache/spark/scheduler/ExecutorResourcesAmounts.scala:106-112 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/scheduler/ExecutorResourcesAmounts.scala:82` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala:471— core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala:471-477 | core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala:488-494 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala:471` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala:495` calls `setClassLoader` and `core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala:478` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (7 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala:204— mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala:204-210 | mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala:279-285 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala:204` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:285— mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:285-291 | mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:248-254 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala` and `mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 39 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:285` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:413— mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:413-419 | mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:361-367 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala` and `mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 39 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (7 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:175— mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:175-181 | mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:210-216 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:175` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala:74— mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala:74-80 | mllib/src/main/scala/org/apache/spark/mllib/evaluation/RegressionMetrics.scala:51-57 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala:74` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1041— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1041-1047 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1123-1129 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (7 lines × 2) REDACTED:4819— REDACTED:4819-4825 | REDACTED:5464-5470 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (7 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala:285— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala:285-291 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala:417-423 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/randomExpressions.scala:285` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (7 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:2802— sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:2802-2808 | sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:2814-2820 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (7 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:3571— sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:3571-3577 | sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:3583-3589 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (7 lines × 2) sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:609— sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:609-615 | sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:632-638 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:609` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:631` calls `withColumn`, `lit` and `sql/connect/client/jdbc/src/main/scala/org/apache/spark/sql/connect/client/jdbc/SparkConnectDatabaseMetaData.scala:609` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (7 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StreamStreamJoinStateHelper.scala:111— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StreamStreamJoinStateHelper.scala:111-117 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StreamStreamJoinStateHelper.scala:127-133 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (7 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:253— sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:253-259 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:289-295 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala:253` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala:71— sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala:71-77 | sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala:134-140 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala:71` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala:49— sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala:49-55 | sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryPage.scala:69-75 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (7 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala:86— sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala:86-92 | sql/core/src/main/scala/org/apache/spark/sql/streaming/ui/StreamingQueryStatisticsPage.scala:95-101 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (7 lines × 2) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:801— sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:801-807 | sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala:1212-1218 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (7 lines × 2) core/src/main/scala/org/apache/spark/MapOutputTracker.scala:669— core/src/main/scala/org/apache/spark/MapOutputTracker.scala:669-675 | core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala:169-176 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (7 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/impl/Utils.scala:91— mllib-local/src/main/scala/org/apache/spark/ml/impl/Utils.scala:91-97 | mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:555-561 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (11 lines × 2) common/utils/src/main/scala/org/apache/spark/util/SparkTestUtils.scala:144— common/utils/src/main/scala/org/apache/spark/util/SparkTestUtils.scala:144-154 | common/utils/src/main/scala/org/apache/spark/util/SparkTestUtils.scala:200-210 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (11 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:115— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:115-125 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala:141-151 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 90 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (11 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:188— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:188-198 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala:214-224 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 153 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:188` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:694— core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:694-704 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala:101-111 — before extracting anything, compare `core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 37 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (11 lines × 2) core/src/main/scala/org/apache/spark/status/protobuf/CachedQuantileSerializer.scala:104— core/src/main/scala/org/apache/spark/status/protobuf/CachedQuantileSerializer.scala:104-114 | core/src/main/scala/org/apache/spark/status/protobuf/StageDataWrapperSerializer.scala:455-465 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (11 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:412— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:412-422 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:395-405 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:412` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:394` calls `logDebug` and `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:410` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (11 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala:197— mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala:197-207 | mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:185-195 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (11 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:463— mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:463-473 | mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:482-492 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:463` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:54— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:54-64 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1399-1409 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1399` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1398` calls `require` and `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:53` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (11 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala:97— mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala:97-107 | mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala:92-102 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (11 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala:189— mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala:189-199 | mllib/src/main/scala/org/apache/spark/mllib/util/SVMDataGenerator.scala:38-48 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (11 lines × 2) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:779— sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:779-789 | sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:791-801 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:779` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:169— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:169-179 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:198-208 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (11 lines × 2) REDACTED:579— REDACTED:579-589 | REDACTED:643-653 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala:110— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala:110-120 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala:81-91 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1090— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1090-1100 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1217-1227 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1090` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala:378— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala:378-388 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala:443-453 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:220— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:220-230 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:270-280 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:220` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:396— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:396-406 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:914-924 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/DecorrelateInnerQuery.scala:682— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/DecorrelateInnerQuery.scala:682-692 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/DecorrelateInnerQuery.scala:744-754 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/DecorrelateInnerQuery.scala:682` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala:105— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala:105-115 | REDACTED:88-98 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AbstractSqlParser.scala:105` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:511— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:511-521 | sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala:1098-1108 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:511` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:767— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:767-777 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:433-443 — before extracting anything, compare `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala` and `sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala` as WHOLE FILES: this scan already matched 9 separate duplicated blocks between them, totalling at least 118 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:767` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:432` calls `by` and `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:766` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (11 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala:131— sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala:131-141 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonTable.scala:66-76 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala:131` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:372— sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:372-382 | sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:390-400 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:372` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:386` calls `StateStoreRestoreExec` and `sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:369` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:166— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:166-177 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala:174-185 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 153 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (12 lines × 2) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/CatalystDataToProtobuf.scala:82— connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/CatalystDataToProtobuf.scala:82-93 | connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDataToCatalyst.scala:175-186 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (12 lines × 2) core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:67— core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:67-78 | core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala:66-77 — before extracting anything, compare `core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala` and `core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 69 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:67` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:453— core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:453-464 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala:87-98 — before extracting anything, compare `core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala` and `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 95 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (12 lines × 2) core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:39— core/src/main/scala/org/apache/spark/rdd/BinaryFileRDD.scala:39-50 | core/src/main/scala/org/apache/spark/rdd/WholeTextFileRDD.scala:42-53 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (12 lines × 2) core/src/main/scala/org/apache/spark/ui/UIUtils.scala:300— core/src/main/scala/org/apache/spark/ui/UIUtils.scala:300-311 | core/src/main/scala/org/apache/spark/ui/UIUtils.scala:375-386 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (12 lines × 2) core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:464— core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:464-475 | core/src/main/scala/org/apache/spark/ui/jobs/PoolPage.scala:55-66 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:464` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:343— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:343-354 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:326-337 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (12 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:472— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:472-483 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:425-436 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (12 lines × 2) mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala:93— mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala:93-104 | mllib/src/main/scala/org/apache/spark/ml/feature/Normalizer.scala:106-117 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/ml/feature/Normalizer.scala:104` calls `Since` and `mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala:90` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:40— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:40-51 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1384-1395 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1398` calls `require` and `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:54` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12 lines × 2) mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:97— mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:97-108 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:88-99 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:97` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:98— mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:98-109 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:89-100 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 37 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:98` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:110` calls `setLabelCol`, `setPredictionCol` and `mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:101` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12 lines × 2) mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:294— mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:294-305 | mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:257-268 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala` and `mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 39 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:294` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:776— mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:776-787 | mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala:282-293 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala` and `mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 46 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:776` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala:132— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala:132-143 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala:87-98 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala:132` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:1664— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:1664-1675 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:1695-1706 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:1664` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:484— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:484-495 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:772-783 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/windowExpressions.scala:484` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:514— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:514-525 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetPartitionReaderFactory.scala:329-340 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:514` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (12 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala:191— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala:191-202 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextPartitionReaderFactory.scala:59-70 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala:191` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextPartitionReaderFactory.scala:72` calls `PartitionReaderWithPartitionValues` and `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala:203` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:800— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:800-811 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:839-850 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:800` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:896— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:896-907 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:950-961 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2Exec.scala:896` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:385— sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:385-396 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:414-425 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:385` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala:349— sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala:349-360 | sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala:376-387 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (12 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala:156— mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Entropy.scala:156-167 | mllib/src/main/scala/org/apache/spark/mllib/tree/impurity/Gini.scala:151-162 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (10 lines × 2) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1720— core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1720-1729 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonUDFRunner.scala:117-126 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1720` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonUDFRunner.scala:128` calls `pythonMetrics` and `core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1731` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (10 lines × 2) core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala:117— core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala:117-126 | core/src/main/scala/org/apache/spark/rdd/SubtractedRDD.scala:76-85 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala:117` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) core/src/main/scala/org/apache/spark/status/protobuf/StageDataWrapperSerializer.scala:433— core/src/main/scala/org/apache/spark/status/protobuf/StageDataWrapperSerializer.scala:433-442 | core/src/main/scala/org/apache/spark/status/protobuf/TaskDataWrapperSerializer.scala:104-113 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (10 lines × 2) mllib/src/main/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluator.scala:126— mllib/src/main/scala/org/apache/spark/ml/evaluation/MultilabelClassificationEvaluator.scala:126-135 | mllib/src/main/scala/org/apache/spark/ml/evaluation/RankingEvaluator.scala:116-125 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (10 lines × 2) mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:234— mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:234-243 | mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala:231-240 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala` and `mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 81 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:234` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:111— mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:111-120 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:113-122 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:112` calls `setBootstrap` and `mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:110` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (10 lines × 2) mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:652— mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:652-661 | mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala:183-192 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala` and `mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 46 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:652` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala:99— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala:99-108 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala:112-121 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (10 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala:195— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala:195-204 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala:209-218 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala:195` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:802— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:802-811 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:1342-1351 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:802` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:302— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:302-311 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:628-637 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:302` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:604— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:604-613 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:635-644 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:604` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala:240— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala:240-249 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala:286-295 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala:234` calls `max`, `BigDecimal`, `Some` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala:285` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (10 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:427— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:427-436 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:530-539 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (10 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:78— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:78-87 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:134-143 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:78` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:365— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:365-374 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:229-238 — before extracting anything, compare `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala` and `sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala` as WHOLE FILES: this scan already matched 9 separate duplicated blocks between them, totalling at least 118 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala:533— sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala:533-542 | sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowGroupLimitExec.scala:82-92 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala:533` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnType.scala:800— sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnType.scala:800-809 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnType.scala:850-859 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:503— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:503-512 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:527-536 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (10 lines × 2) REDACTED:78— REDACTED:78-87 | REDACTED:101-110 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:78` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:101— sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:101-110 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowWindowPythonEvaluatorFactory.scala:76-85 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/TransformWithStateExec.scala:628— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/TransformWithStateExec.scala:628-637 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/TransformWithStateExec.scala:646-655 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:352— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:352-361 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:371-382 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala:495— sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala:495-504 | sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala:536-545 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala:495` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (10 lines × 2) streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala:122— streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala:122-133 | streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala:179-188 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala:122` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Low cohesion: SparkSqlAstBuilder (LCOM4 29) REDACTED:238— SparkSqlAstBuilder's methods fall into 29 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 29 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: Shim_v2_0 (LCOM4 28) sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:250— Shim_v2_0's methods fall into 28 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 28 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: SQLShuffleReadMetricsReporter (LCOM4 17) sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLShuffleMetricsReporter.scala:30— SQLShuffleReadMetricsReporter's methods fall into 17 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 17 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: SparkContext (LCOM4 12) core/src/main/scala/org/apache/spark/SparkContext.scala:88— SparkContext's methods fall into 12 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 12 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: QueryPlan (LCOM4 12) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala:54— QueryPlan's methods fall into 12 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 12 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: SparkHadoopUtil (LCOM4 11) core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala:48— SparkHadoopUtil's methods fall into 11 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 11 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: InMemoryCatalog (LCOM4 11) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala:47— InMemoryCatalog's methods fall into 11 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 11 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: TreeNode (LCOM4 9) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:89— TreeNode's methods fall into 9 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 9 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: SpecificInternalRow (LCOM4 8) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SpecificInternalRow.scala:210— SpecificInternalRow's methods fall into 8 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 8 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: GraphOps (LCOM4 7) graphx/src/main/scala/org/apache/spark/graphx/GraphOps.scala:35— GraphOps's methods fall into 7 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 7 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: StreamingJobProgressListener (LCOM4 7) streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingJobProgressListener.scala:30— StreamingJobProgressListener's methods fall into 7 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 7 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: StandaloneSchedulerBackend (LCOM4 6) core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala:44— StandaloneSchedulerBackend's methods fall into 6 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 6 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: FMClassificationModel (LCOM4 6) mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala:254— FMClassificationModel's methods fall into 6 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 6 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: YarnSchedulerBackend (LCOM4 6) resource-managers/yarn/src/main/scala/org/apache/spark/scheduler/cluster/YarnSchedulerBackend.scala:46— YarnSchedulerBackend's methods fall into 6 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 6 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: TimeTypeConnectOps (LCOM4 6) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/types/ops/TimeTypeConnectOps.scala:45— TimeTypeConnectOps's methods fall into 6 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 6 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: TransformWithStateExecBase (LCOM4 6) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/TransformWithStateExecBase.scala:42— TransformWithStateExecBase's methods fall into 6 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 6 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: SecurityManager (LCOM4 5) core/src/main/scala/org/apache/spark/SecurityManager.scala:47— SecurityManager's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: HadoopMapRedWriteConfigUtil (LCOM4 5) core/src/main/scala/org/apache/spark/internal/io/SparkHadoopWriter.scala:175— HadoopMapRedWriteConfigUtil's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: NewHadoopRDD (LCOM4 5) core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:78— NewHadoopRDD's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: LinearSVCModel (LCOM4 5) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:361— LinearSVCModel's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: LDAModel (LCOM4 5) mllib/src/main/scala/org/apache/spark/ml/clustering/LDA.scala:415— LDAModel's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: LDA (LCOM4 5) mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala:47— LDA's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: Analyzer (LCOM4 5) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:403— Analyzer's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: BinaryArithmetic (LCOM4 5) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:208— BinaryArithmetic's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
Low cohesion: SparkSession (LCOM4 5) sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:91— SparkSession's methods fall into 5 groups that share no field and call none of each other, against a bar of more than 3 for this run (LCOM4, configurable — your repository's bar is the one quoted here). Each group is a set of methods reachable from one another through shared fields or direct calls, so 5 groups means the type has that many internally-connected clusters with nothing tying them together. Types whose shape makes a high count expected — and which would otherwise dominate this list — are excluded before this row is raised, so this is a genuine split candidate rather than a metric reading. It is still a shape, not a defect: confirm the groups match responsibilities you can name before splitting.
ClassTooLong: AstBuilder sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:68— ClassTooLong — 4956 significant lines (blank, comment-only and punctuation-only lines excluded), 144 methods. The bar is 400 significant lines; this is 4556 over it, 12.39× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: SparkConnectPlanner sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:91— ClassTooLong — 3325 significant lines (blank, comment-only and punctuation-only lines excluded), 208 methods. The bar is 400 significant lines; this is 2925 over it, 8.31× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: ParquetVectorUpdaterFactory sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:49— ClassTooLong — 1448 significant lines (blank, comment-only and punctuation-only lines excluded), 20 methods. The bar is 400 significant lines; this is 1048 over it, 3.62× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: RemoteBlockPushResolver common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:97— ClassTooLong — 1330 significant lines (blank, comment-only and punctuation-only lines excluded), 39 methods. The bar is 400 significant lines; this is 930 over it, 3.33× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: UTF8String common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:55— ClassTooLong — 1182 significant lines (blank, comment-only and punctuation-only lines excluded), 118 methods. The bar is 400 significant lines; this is 782 over it, 2.96× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: SparkSqlAstBuilder REDACTED:238— ClassTooLong — 896 significant lines (blank, comment-only and punctuation-only lines excluded), 10 methods. The bar is 400 significant lines; this is 496 over it, 2.24× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: SparkStrategies sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:76— ClassTooLong — 840 significant lines (blank, comment-only and punctuation-only lines excluded), 2 methods. The bar is 400 significant lines; this is 440 over it, 2.10× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: VariantBuilder common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:43— ClassTooLong — 827 significant lines (blank, comment-only and punctuation-only lines excluded), 50 methods. The bar is 400 significant lines; this is 427 over it, 2.07× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: ParquetFilters sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:55— ClassTooLong — 798 significant lines (blank, comment-only and punctuation-only lines excluded), 27 methods. The bar is 400 significant lines; this is 398 over it, 2.00× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: CollationFactory common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:44— ClassTooLong — 792 significant lines (blank, comment-only and punctuation-only lines excluded), 21 methods. The bar is 400 significant lines; this is 392 over it, 1.98× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: CollationAwareUTF8String common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:44— ClassTooLong — 783 significant lines (blank, comment-only and punctuation-only lines excluded), 59 methods. The bar is 400 significant lines; this is 383 over it, 1.96× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: ResolveSessionCatalog REDACTED:50— ClassTooLong — 679 significant lines (blank, comment-only and punctuation-only lines excluded), 8 methods. The bar is 400 significant lines; this is 279 over it, 1.70× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: TransformWithStateInPySparkStateServer sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:62— ClassTooLong — 664 significant lines (blank, comment-only and punctuation-only lines excluded), 13 methods. The bar is 400 significant lines; this is 264 over it, 1.66× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: VectorizedRleValuesReader sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:46— ClassTooLong — 657 significant lines (blank, comment-only and punctuation-only lines excluded), 53 methods. The bar is 400 significant lines; this is 257 over it, 1.64× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: BlockManagerMasterEndpoint core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala:49— ClassTooLong — 619 significant lines (blank, comment-only and punctuation-only lines excluded), 31 methods. The bar is 400 significant lines; this is 219 over it, 1.55× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: RocksDBFileManager sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:130— ClassTooLong — 604 significant lines (blank, comment-only and punctuation-only lines excluded), 38 methods. The bar is 400 significant lines; this is 204 over it, 1.51× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: HiveSessionImpl sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:89— ClassTooLong — 596 significant lines (blank, comment-only and punctuation-only lines excluded), 58 methods. The bar is 400 significant lines; this is 196 over it, 1.49× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: JacksonParser sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:50— ClassTooLong — 590 significant lines (blank, comment-only and punctuation-only lines excluded), 20 methods. The bar is 400 significant lines; this is 190 over it, 1.48× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: ExpressionResolver sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ExpressionResolver.scala:75— ClassTooLong — 582 significant lines (blank, comment-only and punctuation-only lines excluded), 61 methods. The bar is 400 significant lines; this is 182 over it, 1.46× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: BytesToBytesMap core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:71— ClassTooLong — 582 significant lines (blank, comment-only and punctuation-only lines excluded), 30 methods. The bar is 400 significant lines; this is 182 over it, 1.46× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: LateMaterializationParquetRecordReader sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:104— ClassTooLong — 551 significant lines (blank, comment-only and punctuation-only lines excluded), 35 methods. The bar is 400 significant lines; this is 151 over it, 1.38× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: CollationSupport common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:31— ClassTooLong — 550 significant lines (blank, comment-only and punctuation-only lines excluded), 5 methods. The bar is 400 significant lines; this is 150 over it, 1.38× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: WritableColumnVector sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java:55— ClassTooLong — 521 significant lines (blank, comment-only and punctuation-only lines excluded), 109 methods. The bar is 400 significant lines; this is 121 over it, 1.30× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: UnsafeExternalSorter core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java:52— ClassTooLong — 521 significant lines (blank, comment-only and punctuation-only lines excluded), 30 methods. The bar is 400 significant lines; this is 121 over it, 1.30× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
ClassTooLong: ThriftCLIService sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:55— ClassTooLong — 499 significant lines (blank, comment-only and punctuation-only lines excluded), 43 methods. The bar is 400 significant lines; this is 99 over it, 1.25× the bar. To reduce it, group the members that share the same data into a smaller type of their own and delegate to it, so no single type carries every responsibility.
Duplicated block (13 lines × 2) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/CatalystDataToProtobuf.scala:67— connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/CatalystDataToProtobuf.scala:67-79 | connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDataToCatalyst.scala:160-172 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/CatalystDataToProtobuf.scala:67` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:667— core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:667-679 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala:85-97 — before extracting anything, compare `core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 37 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (13 lines × 2) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1366— core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1366-1378 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala:217-229 — before extracting anything, compare `core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 37 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1366` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:552— core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:552-564 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala:164-176 — before extracting anything, compare `core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala` and `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 95 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:552` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:150— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:150-162 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:152-164 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (13 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:322— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:322-334 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:305-317 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:322` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:321` calls `indices` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:304` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (13 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:882— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:882-894 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:748-760 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:882` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1266— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1266-1278 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:1129-1141 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1266` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:168— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:168-180 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1513-1525 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:168` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:115— mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:115-127 | mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala:118-130 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 39 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (13 lines × 2) mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala:85— mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala:85-97 | mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala:81-93 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 40 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (13 lines × 2) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/DeploymentPodsAllocator.scala:124— resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/DeploymentPodsAllocator.scala:124-136 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/StatefulSetPodsAllocator.scala:113-125 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/DeploymentPodsAllocator.scala:124` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1926— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1926-1938 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1966-1978 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Note first that the copies are not typed on the same thing: the declarations holding them bind `target` to `updateCondition) => // Expand star to top level source columns. If source has less columns than` in one and `includeLastResort = canResolveLastResort)) // Expand star to top level source columns. If source has less columns than` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:425— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:425-437 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:475-487 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (13 lines × 2) REDACTED:1829— REDACTED:1829-1841 | REDACTED:1918-1930 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:182— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:182-194 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:654-666 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:182` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:1202— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:1202-1214 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:1220-1232 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1386— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1386-1398 | REDACTED:1732-1744 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1386` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:3927— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:3927-3939 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:3949-3961 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:3927` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala:461— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala:461-473 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala:495-507 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala:461` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:887— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:887-899 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:906-918 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:887` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectGetStatusHandler.scala:183— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectGetStatusHandler.scala:183-195 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectGetStatusHandler.scala:205-219 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectGetStatusHandler.scala:183` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala:283— sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala:283-295 | sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala:317-329 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala:283` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:69— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:69-81 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonWrite.scala:49-61 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:69` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:961— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:961-973 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:994-1006 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:961` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatch.scala:49— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatch.scala:49-62 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala:59-72 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaRelation.scala:74` calls `logInfo` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatch.scala:64` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (14 lines × 2) core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:365— core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:365-378 | core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:325-338 — before extracting anything, compare `core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala` and `core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 50 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:365` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 2) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2777— core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2777-2790 | core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2857-2870 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2777` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (14 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1280— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1280-1293 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:1143-1156 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1280` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1349— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1349-1362 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:1212-1225 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1349` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:720— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:720-733 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:875-888 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:890` calls `Since` and `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:735` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (14 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala:304— mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala:304-317 | mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:304-317 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (14 lines × 2) mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala:420— mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala:420-433 | mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala:383-396 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (14 lines × 2) sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:83— sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:83-96 | sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParmsAstBuilder.scala:120-133 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (14 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:191— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:191-204 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:277-290 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:191` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:190` calls `handleXmlErrorsByParseMode` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:276` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (14 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:364— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:364-377 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlDataSource.scala:441-454 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (14 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:290— sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:290-303 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/TransformWithStateExec.scala:589-602 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:290` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 2) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/AutoCdcAuxiliaryTable.scala:146— sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/AutoCdcAuxiliaryTable.scala:146-159 | sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/AutoCdcAuxiliaryTable.scala:220-233 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (14 lines × 2) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/DeploymentPodsAllocator.scala:81— resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/DeploymentPodsAllocator.scala:81-94 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/StatefulSetPodsAllocator.scala:68-84 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/DeploymentPodsAllocator.scala:81` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (14 lines × 2) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:955— sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:955-968 | sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala:467-480 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:955` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 2) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:529— common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:529-542 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:676-689 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (14 lines × 2) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1539— common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1539-1552 | common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1572-1585 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1539` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (14 lines × 2) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:634— common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:634-647 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:738-751 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:634` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 2) core/src/main/java/org/apache/spark/util/collection/TimSort.java:713— core/src/main/java/org/apache/spark/util/collection/TimSort.java:713-726 | core/src/main/java/org/apache/spark/util/collection/TimSort.java:835-848 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/java/org/apache/spark/util/collection/TimSort.java:713` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `core/src/main/java/org/apache/spark/util/collection/TimSort.java:710` calls `newKey` and `core/src/main/java/org/apache/spark/util/collection/TimSort.java:833` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (14 lines × 2) python/pyspark/ml/common.py:151— python/pyspark/ml/common.py:151-164 | python/pyspark/mllib/common.py:161-174 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (14 lines × 2) python/pyspark/ml/connect/functions.py:53— python/pyspark/ml/connect/functions.py:53-66 | python/pyspark/sql/connect/tvf.py:128-141 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (14 lines × 2) python/pyspark/pandas/frame.py:9084— python/pyspark/pandas/frame.py:9084-9097 | REDACTED:3551-3564 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (14 lines × 2) REDACTED:4074— REDACTED:4074-4087 | python/pyspark/pandas/spark/accessors.py:1250-1263 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:4074` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 2) python/pyspark/sql/connect/readwriter.py:270— python/pyspark/sql/connect/readwriter.py:270-283 | python/pyspark/sql/readwriter.py:689-702 — before extracting anything, compare `python/pyspark/sql/connect/readwriter.py` and `python/pyspark/sql/readwriter.py` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 113 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Note first that the copies are not typed on the same thing: `DataFrame` names `pyspark.sql.connect.dataframe` in one and `pyspark.sql.dataframe` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (14 lines × 2) python/pyspark/sql/connect/readwriter.py:892— python/pyspark/sql/connect/readwriter.py:892-905 | python/pyspark/sql/connect/readwriter.py:928-941 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/connect/readwriter.py:892` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
HackComment sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala:546— // HACK: Hive is too noisy by default. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala:170— // Hack: Hive simply prints the result of a SET command to screen, — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala:55— // Hack to avoid SEGV on some JVM versions. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:523— // !! HACK ALERT !! — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SchemaMergeUtils.scala:47— // !! HACK ALERT !! — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment sql/core/src/main/scala/org/apache/spark/sql/execution/Columnar.scala:209— // Hack because of type erasure — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:81— // Hacky, is there a better way? see if the type is known as a custom — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlInputFormat.scala:130— // Hack: in the uncompressed case (see more below), we must know how much the — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:943— // HACK ALTER! Ideally we should only resolve GROUPING SETS + HAVING when the having condition — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:342— # Hack namedtuple, make it picklable. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:403— # hack the cls already generated by namedtuple. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:413— # hack inplace — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment python/pyspark/sql/column.py:71— # HACK ALERT!! this is to reduce the backward compatibility concern, and returns — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment python/pyspark/sql/dataframe.py:134— # HACK ALERT!! this is to reduce the backward compatibility concern, and returns — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment python/pyspark/sql/connect/plan.py:770— # !!HACK ALERT!! — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:61— # !!HACK ALERT!! — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:122— # !!HACK ALERT!! — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:225— # !!HACK ALERT!! — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment python/pyspark/pandas/frame.py:12988— # hack to use pandas' info as is. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment python/pyspark/mllib/common.py:33— # Hack for support float('inf') in Py4j — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment python/pyspark/ml/common.py:37— # Hack for support float('inf') in Py4j — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:421— # Hack to circumvent non-predictable memoization caused by string interning. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:715— # Hack to circumvent non-predictable memoization caused by string interning. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:744— # Hack to circumvent non-predictable memoization caused by string interning. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
HackComment REDACTED:834— # Hack to circumvent non-predictable memoization caused by string interning. — a workaround marked in source: record what it is compensating for and what would allow its removal (the upstream fix, the API it is waiting on, the invariant it restores), so the next reader can judge whether it is still needed rather than rediscovering why it is there.
+ 7 more in this group — see findings.md.
D4 · Code Duplication· Members sharing a duplicated core (4 members, 50+ identical tokens) · ×32
Members sharing a duplicated core (4 members, 50+ identical tokens) core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:111— core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:111-139 | core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:142-168 | core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:171-197 | core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:200-226 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:779— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:779-834 | mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:858-894 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:656-711 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:722-760 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:575— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:575-585 | mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:719-733 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:720-730 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:874-888 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:291— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:291-348 | mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:939-1031 | mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala:293-346 | mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:538-610 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:125— mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:125-137 | mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala:128-140 | mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala:95-107 | mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala:123-135 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:330— mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:330-360 | mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:372-402 | mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:414-444 | mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:456-486 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1164— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1164-1188 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1199-1230 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1233-1294 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1297-1360 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2501— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2501-2535 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2539-2573 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2575-2610 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2612-2646 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:108— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:108-134 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:304-329 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:123-153 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:278-322 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:130— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:130-144 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:501-515 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:600-612 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:698-710 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) REDACTED:1559— REDACTED:1559-1587 | REDACTED:1697-1725 | REDACTED:2785-2813 | REDACTED:4521-4536 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) REDACTED:4809— REDACTED:4809-4884 | REDACTED:5006-5087 | REDACTED:5231-5325 | REDACTED:5455-5543 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala:122— sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala:122-141 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonTable.scala:59-76 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcTable.scala:53-68 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetTable.scala:53-70 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:809— sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:809-844 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:849-884 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:889-930 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:935-977 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:453— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:453-492 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:518-535 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:573-597 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:636-661 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:121— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:121-134 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java:107-120 | common/network-common/src/main/java/org/apache/spark/network/shuffledb/LevelDBIterator.java:39-52 | common/network-common/src/main/java/org/apache/spark/network/shuffledb/RocksDBIterator.java:45-58 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:302— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:302-312 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:329-395 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:304-314 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:331-397 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) python/pyspark/eval_handlers/_arrow.py:140— python/pyspark/eval_handlers/_arrow.py:140-154 | python/pyspark/eval_handlers/_arrow.py:210-222 | python/pyspark/eval_handlers/_pandas.py:376-388 | python/pyspark/eval_handlers/_pandas.py:456-466 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) python/pyspark/eval_handlers/_arrow.py:277— python/pyspark/eval_handlers/_arrow.py:277-284 | python/pyspark/eval_handlers/_arrow.py:348-356 | python/pyspark/eval_handlers/_arrow.py:545-556 | python/pyspark/eval_handlers/_arrow.py:597-607 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) python/pyspark/mllib/evaluation.py:670— python/pyspark/mllib/evaluation.py:670-688 | python/pyspark/mllib/feature.py:1053-1063 | python/pyspark/mllib/random.py:682-694 | python/pyspark/mllib/regression.py:1039-1050 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) python/pyspark/pandas/accessors.py:202— python/pyspark/pandas/accessors.py:202-430 | python/pyspark/pandas/accessors.py:435-763 | python/pyspark/pandas/frame.py:3019-3343 | python/pyspark/pandas/groupby.py:1825-2192 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) python/pyspark/pandas/plot/core.py:168— python/pyspark/pandas/plot/core.py:168-180 | python/pyspark/pandas/plot/core.py:394-419 | python/pyspark/sql/plot/core.py:502-527 | python/pyspark/sql/plot/core.py:574-585 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) python/pyspark/sql/avro/functions.py:204— python/pyspark/sql/avro/functions.py:204-240 | python/pyspark/sql/connect/avro/functions.py:91-133 | python/pyspark/sql/connect/protobuf/functions.py:116-158 | python/pyspark/sql/protobuf/functions.py:297-335 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) python/pyspark/sql/connect/readwriter.py:656— python/pyspark/sql/connect/readwriter.py:656-699 | python/pyspark/sql/connect/readwriter.py:712-744 | python/pyspark/sql/readwriter.py:1570-1661 | python/pyspark/sql/readwriter.py:1672-1745 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Members sharing a duplicated core (4 members, 50+ identical tokens) python/pyspark/sql/pandas/typehints.py:53— python/pyspark/sql/pandas/typehints.py:53-199 | python/pyspark/sql/pandas/typehints.py:212-325 | python/pyspark/sql/pandas/typehints.py:409-480 | python/pyspark/sql/pandas/typehints.py:502-577 — These 4 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
+ 7 more in this group — see findings.md.
X15 · Unvalidated length from an untrusted reader· Length read from the parsed input is spent without ever being bounded · ×32
Length read from the parsed input is spent without ever being bounded streaming/src/main/scala/org/apache/spark/streaming/dstream/RawInputDStream.scala:90— `length` is read from the bytes being parsed at line 88 (`getInt`), then used to size a `ByteBuffer.allocate` at line 90 — and nowhere in its scope is `length` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `length` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded streaming/src/main/scala/org/apache/spark/streaming/api/python/PythonDStream.scala:108— `length` is read from the bytes being parsed at line 107 (`readInt`), then used to size a `new Array[Byte]` at line 108 — and nowhere in its scope is `length` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `length` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveShim.scala:185— `functionInBytesLength` is read from the bytes being parsed at line 184 (`readInt`), then used to size a `new Array[Byte]` at line 185 — and nowhere in its scope is `functionInBytesLength` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `functionInBytesLength` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:873— `keySize` is read from the bytes being parsed at line 866 (`readInt`), then used to size a `new Array[Byte]` at line 873 — and nowhere in its scope is `keySize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `keySize` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:892— `valueSize` is read from the bytes being parsed at line 878 (`readInt`), then used to size a `new Array[Byte]` at line 892 — and nowhere in its scope is `valueSize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `valueSize` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:1035— `keySize` is read from the bytes being parsed at line 1028 (`readInt`), then used to size a `new Array[Byte]` at line 1035 — and nowhere in its scope is `keySize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `keySize` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:1046— `valueSize` is read from the bytes being parsed at line 1041 (`readInt`), then used to size a `new Array[Byte]` at line 1046 — and nowhere in its scope is `valueSize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `valueSize` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala:516— `keySize` is read from the bytes being parsed at line 506 (`readInt`), then used to size a `new Array[Byte]` at line 516 — and nowhere in its scope is `keySize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `keySize` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala:523— `valueSize` is read from the bytes being parsed at line 518 (`readInt`), then used to size a `new Array[Byte]` at line 523 — and nowhere in its scope is `valueSize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `valueSize` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreChangelog.scala:547— `blockSize` is read from the bytes being parsed at line 546 (`readInt`), then used to size a `new Array[Byte]` at line 547 — and nowhere in its scope is `blockSize` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `blockSize` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/python/RowQueue.scala:158— `size` is read from the bytes being parsed at line 157 (`readInt`), then used to size a `new Array[Byte]` at line 158 — and nowhere in its scope is `size` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `size` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkDeserializer.scala:71— `size` is read from the bytes being parsed at line 67 (`readInt`), then used to size a `new Array[Byte]` at line 71 — and nowhere in its scope is `size` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `size` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:245— `messageLen` is read from the bytes being parsed at line 244 (`readInt`), then used to size a `new Array[Byte]` at line 245 — and nowhere in its scope is `messageLen` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `messageLen` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnType.scala:603— `length` is read from the bytes being parsed at line 602 (`getInt`), then used to size a `new Array[Byte]` at line 603 — and nowhere in its scope is `length` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `length` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala:767— `typeLength` is read from the bytes being parsed at line 766 (`getInt`), then used to size a `new Array[Byte]` at line 767 — and nowhere in its scope is `typeLength` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `typeLength` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:864— `len` is read from the bytes being parsed at line 863 (`readInt`), then used to size a `new Array[Int]` at line 864 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:880— `len` is read from the bytes being parsed at line 879 (`readInt`), then used to size a `new Array[Long]` at line 880 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:896— `len` is read from the bytes being parsed at line 895 (`readInt`), then used to size a `new Array[Float]` at line 896 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:912— `len` is read from the bytes being parsed at line 911 (`readInt`), then used to size a `new Array[Double]` at line 912 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:945— `len` is read from the bytes being parsed at line 944 (`readInt`), then used to size a `new Array[(]` at line 945 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala:1037— `len` is read from the bytes being parsed at line 1036 (`readInt`), then used to size a `new Array[T]` at line 1037 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded core/src/main/scala/org/apache/spark/util/io/ChunkedByteBuffer.scala:109— `chunksNum` is read from the bytes being parsed at line 106 (`readInt`), then used to size a `new Array[ByteBuffer]` at line 109 — and nowhere in its scope is `chunksNum` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `chunksNum` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded core/src/main/scala/org/apache/spark/security/SocketAuthHelper.scala:117— `len` is read from the bytes being parsed at line 116 (`readInt`), then used to size a `new Array[Byte]` at line 117 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala:175— `len` is read from the bytes being parsed at line 174 (`readInt`), then used to size a `new Array[Byte]` at line 175 — and nowhere in its scope is `len` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `len` against the bytes actually remaining and a maximum the format allows before allocating.
Length read from the parsed input is spent without ever being bounded core/src/main/scala/org/apache/spark/scheduler/TaskDescription.scala:241— `valueLength` is read from the bytes being parsed at line 240 (`readInt`), then used to size a `new Array[Byte]` at line 241 — and nowhere in its scope is `valueLength` bounded from above — compared to a limit, clamped, or checked against the input that remains (a zero or sign test, or the loop that counts up to it, bounds nothing). Whoever wrote those bytes therefore chooses the allocation directly: a 32-bit length asks for up to 2 GiB before a single byte of the payload is known to exist, which either exhausts the heap with an `OutOfMemoryError` or, negative, throws `NegativeArraySizeException` out of a parser whose failure path was written for malformed input, not for hostile input. Bound `valueLength` against the bytes actually remaining and a maximum the format allows before allocating.
Duplicated block (16 lines × 2) core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala:115— core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala:115-130 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala:296-311 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/rdd/JdbcRDD.scala:115` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:121— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:121-136 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:123-138 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (16 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:305— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:305-320 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:289-304 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:305` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:321` calls `indices` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:305` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (16 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:357— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:357-372 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:340-355 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (16 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:864— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:864-879 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:730-745 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:864` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:545— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:545-560 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:688-703 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (16 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:588— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:588-603 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:734-749 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:732` calls `Since` and `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:585` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (16 lines × 2) mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:349— mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:349-364 | mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala:431-446 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala` and `mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 81 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (16 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:73— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:73-88 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1418-1433 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:73` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:183— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:183-198 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1528-1543 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (16 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1346— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1346-1361 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1441-1456 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (16 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:1378— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:1378-1393 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:1742-1757 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:1378` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:478— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:478-493 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:564-579 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (16 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:147— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:147-162 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:119-134 — before extracting anything, compare `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala` and `sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala` as WHOLE FILES: this scan already matched 9 separate duplicated blocks between them, totalling at least 118 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:147` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (16 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala:93— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala:93-108 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerSessionPage.scala:74-89 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala:93` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (16 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/MergingSessionsIterator.scala:66— sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/MergingSessionsIterator.scala:66-81 | sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortBasedAggregationIterator.scala:60-75 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (16 lines × 2) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:343— sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:343-358 | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:390-405 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (16 lines × 2) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala:344— sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala:344-359 | sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala:389-404 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/FlowExecution.scala:344` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16 lines × 2) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:104— common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:104-119 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:206-221 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:104` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16 lines × 2) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1360— common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1360-1375 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1593-1608 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (16 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:188— sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:188-203 | sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:194-209 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 93 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:188` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:193` calls `isNullAt` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:187` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (16 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DefaultValue.java:38— sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DefaultValue.java:38-53 | sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/GenerationExpression.java:38-54 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (16 lines × 2) python/pyspark/pandas/testing.py:148— python/pyspark/pandas/testing.py:148-163 | python/pyspark/pandas/testing.py:244-259 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (16 lines × 2) python/pyspark/sql/classic/dataframe.py:1338— python/pyspark/sql/classic/dataframe.py:1338-1353 | python/pyspark/sql/connect/dataframe.py:1233-1248 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own. Note first that the copies are not typed on the same thing: `Observation` names `pyspark.sql` in one and `pyspark.sql.connect.observation` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (16 lines × 2) python/pyspark/sql/connect/observation.py:36— python/pyspark/sql/connect/observation.py:36-51 | python/pyspark/sql/observation.py:90-105 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Swallowed exception (caught, then discarded) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/AutoCdcAuxiliaryTable.scala:88— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/core/src/main/scala/org/apache/spark/sql/execution/BaseScriptTransformationExec.scala:272— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/memory.scala:96— `case NonFatal(e)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/core/src/main/scala/org/apache/spark/sql/execution/debug/package.scala:140— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InMemoryFileIndex.scala:172— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateNamespaceExec.scala:56— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) REDACTED:1059— `catch (e)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:167— `case _: Exception` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala:809— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ProtobufDescriptorFileReader.scala:65— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:1377— `case _: RuntimeException` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/QueryPlan.scala:88— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1616— `case _: Exception` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TryEval.scala:44— `case _: Exception` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:389— `case _: Exception` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:273— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:276— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:287— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:996— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:1025— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:1654— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:1732— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyDefaultCollation.scala:224— `case NonFatal(_)` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
Swallowed exception (caught, then discarded) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala:507— `case _: Throwable` takes every exception and records none of it — the body neither logs it, rethrows it, nor even names it, so the failure is discarded as completely as by an empty catch and only the substituted value survives. Log it through whatever this codebase already uses to report problems, narrow the catch to the exception this call can actually raise, or say in a comment on the catch why the failure genuinely cannot matter.
MethodTooLong: AvroDeserializer.deserialize sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:166— MethodTooLong — deserialize runs 242 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 142 over it, 2.42× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: AvroDeserializer.newWriter sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:172— MethodTooLong — newWriter runs 241 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 141 over it, 2.41× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: HiveInspectors.unwrapperFor sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:550— MethodTooLong — unwrapperFor runs 236 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 136 over it, 2.36× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: HiveInspectors.wrapperFor sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:272— MethodTooLong — wrapperFor runs 193 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 93 over it, 1.93× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: StagePage.makeTimeline core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala:251— MethodTooLong — makeTimeline runs 180 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 80 over it, 1.80× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: ProtobufDeserializer.newWriter connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:249— MethodTooLong — newWriter runs 159 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 59 over it, 1.59× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: ParquetVectorUpdaterFactory.getUpdater sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:75— MethodTooLong — getUpdater runs 149 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 49 over it, 1.49× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: ColumnNodeToProtoConverter.apply sql/connect/common/src/main/scala/org/apache/spark/sql/connect/columnNodeSupport.scala:61— MethodTooLong — apply runs 145 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 45 over it, 1.45× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: SQLExecution.withNewExecutionId0 sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:205— MethodTooLong — withNewExecutionId0 runs 134 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 34 over it, 1.34× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: DataflowGraphTransformer.transformDownNodes sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DataflowGraphTransformer.scala:121— MethodTooLong — transformDownNodes runs 134 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 34 over it, 1.34× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: ParquetFileFormat.buildParquetReader sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:242— MethodTooLong — buildParquetReader runs 124 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 24 over it, 1.24× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: LogisticRegression.train mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:507— MethodTooLong — train runs 122 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 22 over it, 1.22× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: SqlStatementSplitter.split sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:133— MethodTooLong — split runs 122 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 22 over it, 1.22× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: SqlStatementSplitter.split sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:144— MethodTooLong — split runs 120 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 20 over it, 1.20× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: SqlStatementSplitter.splitWithPositions sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:153— MethodTooLong — splitWithPositions runs 116 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 16 over it, 1.16× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: InterpretedUnsafeProjection.generateFieldWriter sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala:134— MethodTooLong — generateFieldWriter runs 112 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 12 over it, 1.12× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: InjectRuntimeFilter.extractSelectiveFilterOverScan sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:103— MethodTooLong — extractSelectiveFilterOverScan runs 110 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 10 over it, 1.10× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: MLExpressionUtils.scaleShift sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ml/MLExpressionUtils.java:194— MethodTooLong — scaleShift runs 109 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 9 over it, 1.09× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: DAGScheduler.handleJobSubmitted core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2619— MethodTooLong — handleJobSubmitted runs 108 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 8 over it, 1.08× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: VariantShreddingWriter.tryTypedShred common/variant/src/main/java/org/apache/spark/types/variant/VariantShreddingWriter.java:155— MethodTooLong — tryTypedShred runs 103 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 3 over it, 1.03× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
MethodTooLong: TransportResponseHandler.handle common/network-common/src/main/java/org/apache/spark/network/client/TransportResponseHandler.java:166— MethodTooLong — handle runs 102 significant lines (blank, comment-only and punctuation-only lines excluded) in one body. The bar is 100 significant lines; this is 2 over it, 1.02× the bar. This is length, not branching: a long straight-line body scores low on complexity and is still read whole to change any part of it, so the complexity numbers beside this row neither confirm nor excuse it. To reduce it, extract each cohesive step of the body — the runs of statements that work on the same values and would earn the same name — into its own named unit, and have this one call them in order.
Duplicated block (18 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:403— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:403-420 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala:259-276 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 90 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:403` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala:277` calls `logDebug` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:422` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (18 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:352— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:352-369 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala:390-407 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 153 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:352` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18 lines × 2) core/src/main/scala/org/apache/spark/SparkContext.scala:1123— core/src/main/scala/org/apache/spark/SparkContext.scala:1123-1140 | sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala:1053-1070 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/SparkContext.scala:1123` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:375— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:375-392 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:358-375 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (18 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:674— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:674-691 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:564-581 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:674` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:563` calls `logDebug` and `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:672` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (18 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1247— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1247-1264 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:1110-1127 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1247` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:478— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:478-495 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:625-642 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (18 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:213— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:213-230 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:337-354 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (18 lines × 2) REDACTED:2617— REDACTED:2617-2634 | REDACTED:2695-2712 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (18 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala:145— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala:145-162 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala:147-164 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala:145` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18 lines × 2) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala:154— sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala:154-171 | sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:496-513 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/common/src/main/scala/org/apache/spark/sql/connect/Dataset.scala:154` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala:242— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala:242-259 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlDataSource.scala:124-141 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (18 lines × 2) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1826— common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1826-1843 | common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1923-1940 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (18 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java:116— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java:116-133 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VariableLengthRowBasedKeyValueBatch.java:128-145 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VariableLengthRowBasedKeyValueBatch.java` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 51 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java:116` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (18 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:83— sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:83-100 | sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:88-105 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 93 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:83` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (18 lines × 2) python/pyspark/ml/linalg/__init__.py:827— python/pyspark/ml/linalg/__init__.py:827-844 | python/pyspark/mllib/linalg/__init__.py:947-964 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (18 lines × 2) python/pyspark/mllib/regression.py:533— python/pyspark/mllib/regression.py:533-550 | python/pyspark/mllib/regression.py:714-731 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/mllib/regression.py:533` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18 lines × 2) python/pyspark/pandas/groupby.py:4197— python/pyspark/pandas/groupby.py:4197-4214 | python/pyspark/pandas/groupby.py:4395-4412 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/groupby.py:4197` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18 lines × 2) python/pyspark/sql/avro/functions.py:98— python/pyspark/sql/avro/functions.py:98-115 | python/pyspark/sql/connect/avro/functions.py:36-53 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Note first that the copies are not typed on the same thing: `ColumnOrName` names `pyspark.sql._typing` in one and `pyspark.sql.connect._typing` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (18 lines × 2) python/pyspark/sql/connect/types.py:414— python/pyspark/sql/connect/types.py:414-431 | python/pyspark/sql/connect/types.py:441-458 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (18 lines × 2) examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:84— examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:84-101 | examples/src/main/scala/org/apache/spark/examples/mllib/MovieLensALS.scala:78-95 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:84` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchPartitionReader.scala:150— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchPartitionReader.scala:150-166 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceRDD.scala:123-139 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaBatchPartitionReader.scala:150` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17 lines × 2) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufSerializer.scala:174— connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufSerializer.scala:174-190 | sql/core/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala:228-244 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (17 lines × 2) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:566— core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:566-582 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala:179-195 — before extracting anything, compare `core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala` and `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 95 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (17 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:60— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:60-76 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:62-78 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (17 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:79— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:79-95 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:87-103 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (17 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:312— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:312-328 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:466-482 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (17 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:689— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:689-705 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:842-858 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (17 lines × 2) mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala:54— mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala:54-70 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:307-323 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: 87% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (17 lines × 2) sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:383— sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:383-399 | sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:402-418 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (17 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:112— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:112-128 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:273-289 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:112` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:245— sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:245-261 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:301-317 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:245` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (17 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java:90— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java:90-106 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java:123-139 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java` as WHOLE FILES: this scan already matched 12 separate duplicated blocks between them, totalling at least 119 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java:90` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java:265— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java:265-281 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java:297-313 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDB.java` as WHOLE FILES: this scan already matched 12 separate duplicated blocks between them, totalling at least 119 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDB.java:265` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (17 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:458— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:458-474 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:460-476 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 223 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:458` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17 lines × 2) python/pyspark/ml/linalg/__init__.py:246— python/pyspark/ml/linalg/__init__.py:246-262 | python/pyspark/mllib/linalg/__init__.py:257-273 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (17 lines × 2) python/pyspark/pandas/frame.py:12693— python/pyspark/pandas/frame.py:12693-12709 | python/pyspark/pandas/frame.py:12828-12844 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:12693` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (17 lines × 2) python/pyspark/sql/streaming/list_state_client.py:82— python/pyspark/sql/streaming/list_state_client.py:82-98 | python/pyspark/sql/streaming/map_state_client.py:225-241 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/streaming/list_state_client.py:82` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17 lines × 2) examples/src/main/scala/org/apache/spark/examples/LocalALS.scala:59— examples/src/main/scala/org/apache/spark/examples/LocalALS.scala:59-75 | examples/src/main/scala/org/apache/spark/examples/SparkALS.scala:63-79 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/LocalALS.scala` and `examples/src/main/scala/org/apache/spark/examples/SparkALS.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 34 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/SparkALS.scala:63` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17 lines × 2) examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala:38— examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala:38-54 | examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala:42-58 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (17 lines × 2) examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java:75— examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java:75-91 | examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKerberizedKafkaWordCount.java:112-128 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java` and `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKerberizedKafkaWordCount.java` as WHOLE FILES: 87% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java:75` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKerberizedKafkaWordCount.java:111` calls `option` and `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java:74` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (15 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:193— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:193-207 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala:237-251 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 90 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (15 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala:288— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala:288-302 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala:494-508 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala:288` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala:112— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala:112-126 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala:252-266 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (15 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:88— sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:88-102 | sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:126-140 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (15 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:822— sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:822-836 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:849-863 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:822` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 2) connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala:158— connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala:158-172 | connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDataToCatalyst.scala:135-152 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala:156` calls `take` and `connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDataToCatalyst.scala:133` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (15 lines × 2) core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala:174— core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala:174-189 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/PythonStreamingSourceRunner.scala:319-333 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (15 lines × 2) core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:671— core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:671-685 | core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:714-728 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:671` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (15 lines × 2) python/pyspark/eval_handlers/_stateful.py:472— python/pyspark/eval_handlers/_stateful.py:472-486 | python/pyspark/eval_handlers/_stateful.py:645-659 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (15 lines × 2) python/pyspark/ml/common.py:87— python/pyspark/ml/common.py:87-101 | python/pyspark/mllib/common.py:77-91 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Note first that the copies are not typed on the same thing: the declarations holding them bind `sc` to `"SparkContext"` in one and `SparkContext` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (15 lines × 2) python/pyspark/ml/linalg/__init__.py:88— python/pyspark/ml/linalg/__init__.py:88-102 | python/pyspark/mllib/linalg/__init__.py:99-113 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (15 lines × 2) python/pyspark/pandas/frame.py:8443— python/pyspark/pandas/frame.py:8443-8457 | python/pyspark/pandas/indexes/multi.py:455-469 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:8443` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 2) python/pyspark/pandas/plot/matplotlib.py:755— python/pyspark/pandas/plot/matplotlib.py:755-769 | python/pyspark/pandas/plot/matplotlib.py:914-928 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/plot/matplotlib.py:755` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (15 lines × 2) python/pyspark/sql/classic/dataframe.py:2230— python/pyspark/sql/classic/dataframe.py:2230-2244 | python/pyspark/sql/connect/dataframe.py:2628-2642 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own.
Duplicated block (15 lines × 2) python/pyspark/sql/connect/functions/builtin.py:2012— python/pyspark/sql/connect/functions/builtin.py:2012-2026 | python/pyspark/sql/connect/functions/builtin.py:2040-2054 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (15 lines × 2) python/pyspark/sql/connect/group.py:555— python/pyspark/sql/connect/group.py:555-569 | python/pyspark/sql/connect/group.py:586-600 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/connect/group.py:555` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 2) python/pyspark/sql/connect/readwriter.py:926— python/pyspark/sql/connect/readwriter.py:926-940 | python/pyspark/sql/readwriter.py:2311-2325 — before extracting anything, compare `python/pyspark/sql/connect/readwriter.py` and `python/pyspark/sql/readwriter.py` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 113 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (15 lines × 2) python/pyspark/sql/streaming/listener.py:457— python/pyspark/sql/streaming/listener.py:457-471 | python/pyspark/sql/streaming/listener.py:735-749 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/streaming/listener.py:457` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 2) examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala:54— examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala:54-68 | examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala:59-73 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 45 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala:54` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 2) examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala:67— examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala:67-81 | examples/src/main/scala/org/apache/spark/examples/streaming/DirectKerberizedKafkaWordCount.scala:107-121 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala` and `examples/src/main/scala/org/apache/spark/examples/streaming/DirectKerberizedKafkaWordCount.scala` as WHOLE FILES: 87% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala:67` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (22 lines × 2) mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala:33— mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala:33-54 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:201-222 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: 87% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (22 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala:205— mllib/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala:205-226 | mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala:403-424 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (22 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:124— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:124-145 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:291-312 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:124` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (22 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1044— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1044-1065 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlInputFormat.scala:208-229 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1044` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (22 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/plugin/SparkConnectPluginRegistry.scala:173— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/plugin/SparkConnectPluginRegistry.scala:173-194 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectInterceptorRegistry.scala:99-120 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (22 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:403— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:403-424 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:250-271 — before extracting anything, compare `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala` and `sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala` as WHOLE FILES: this scan already matched 9 separate duplicated blocks between them, totalling at least 118 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (22 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala:83— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala:83-104 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextWrite.scala:46-67 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (22 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateScanBuilder.scala:166— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateScanBuilder.scala:166-187 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateTable.scala:65-86 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (22 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1326— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1326-1347 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1447-1468 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (22 lines × 2) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:741— sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:741-762 | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:784-805 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:741` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (22 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:58— sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:58-79 | sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:63-84 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 93 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:58` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (22 lines × 2) python/pyspark/eval_handlers/_arrow.py:157— python/pyspark/eval_handlers/_arrow.py:157-178 | python/pyspark/eval_handlers/_arrow.py:225-246 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/eval_handlers/_arrow.py:249` calls `from_batches` and `python/pyspark/eval_handlers/_arrow.py:181` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (22 lines × 2) python/pyspark/ml/linalg/__init__.py:990— python/pyspark/ml/linalg/__init__.py:990-1011 | python/pyspark/mllib/linalg/__init__.py:1166-1187 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (22 lines × 2) python/pyspark/pandas/data_type_ops/num_ops.py:561— python/pyspark/pandas/data_type_ops/num_ops.py:561-582 | python/pyspark/pandas/data_type_ops/num_ops.py:699-720 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (22 lines × 2) python/pyspark/pandas/frame.py:7719— python/pyspark/pandas/frame.py:7719-7740 | python/pyspark/pandas/indexes/base.py:1391-1412 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:7719` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (22 lines × 2) python/pyspark/pandas/plot/plotly.py:220— python/pyspark/pandas/plot/plotly.py:220-241 | python/pyspark/sql/plot/plotly.py:143-164 — before extracting anything, compare `python/pyspark/pandas/plot/plotly.py` and `python/pyspark/sql/plot/plotly.py` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 54 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (22 lines × 2) python/pyspark/sql/profiler.py:285— python/pyspark/sql/profiler.py:285-306 | python/pyspark/sql/profiler.py:369-393 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (22 lines × 2) python/pyspark/sql/profiler.py:407— python/pyspark/sql/profiler.py:407-428 | python/pyspark/sql/profiler.py:431-452 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (20 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:473— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:473-492 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala:392-411 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 90 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:473` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (20 lines × 2) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:465— core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:465-484 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala:106-125 — before extracting anything, compare `core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala` and `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 95 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (20 lines × 2) core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala:237— core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala:237-256 | core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala:322-341 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala:237` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (20 lines × 2) mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:370— mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:370-389 | mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala:452-471 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala` and `mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 81 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (20 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:188— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:188-207 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:353-372 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:188` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (20 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2355— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2355-2374 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2382-2401 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (20 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:61— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:61-80 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:318-337 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:338` calls `forall` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:82` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (20 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:85— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:85-104 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:240-259 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (20 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:578— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:578-597 | sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AQEOptimizer.scala:53-72 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:578` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (20 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala:402— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala:402-421 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlDataSource.scala:226-245 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (20 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:126— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:126-145 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala:148-167 — before extracting anything, compare `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 38 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (20 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:138— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:138-157 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java:124-143 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 100 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (20 lines × 2) python/pyspark/eval_handlers/_pandas.py:344— python/pyspark/eval_handlers/_pandas.py:344-363 | python/pyspark/eval_handlers/_pandas.py:507-526 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (20 lines × 2) python/pyspark/ml/linalg/__init__.py:1179— python/pyspark/ml/linalg/__init__.py:1179-1198 | python/pyspark/mllib/linalg/__init__.py:1374-1393 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (20 lines × 2) python/pyspark/ml/linalg/__init__.py:1289— python/pyspark/ml/linalg/__init__.py:1289-1308 | python/pyspark/mllib/linalg/__init__.py:1486-1505 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (20 lines × 2) python/pyspark/ml/connect/pipeline.py:213— python/pyspark/ml/connect/pipeline.py:213-234 | python/pyspark/ml/pipeline.py:145-164 — before extracting anything, compare `python/pyspark/ml/connect/pipeline.py` and `python/pyspark/ml/pipeline.py` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 57 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Swallowed exception (empty catch) streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala:406— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlanInfo.scala:130— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:143— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:186— An empty catch block silently discards the error — failures vanish with no log and no rethrow. On a teardown path letting it propagate is not an option (throwing out of `close()` masks the failure already in flight and abandons the rest of the cleanup), so make the swallow deliberate instead: narrow the catch to the exception this release can actually raise, and record it through whatever this codebase already uses to report problems — or, if it truly cannot matter, say why in a comment on the catch.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:246— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:300— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:387— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:403— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlDataSource.scala:449— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala:372— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:503— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:509— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/core/src/main/scala/org/apache/spark/sql/classic/Catalog.scala:802— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/PlanCompressionUtils.scala:62— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:749— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Swallowed exception (empty catch) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:1734— An empty catch block silently discards the error — failures vanish with no log and no rethrow. Log it, handle it, or don't catch it.
Duplicated block (19 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:452— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala:452-470 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala:371-389 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSource.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 90 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (19 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:133— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:133-151 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala:141-159 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 153 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:133` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala:492— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala:492-510 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala:522-540 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (19 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:90— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:90-108 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:92-110 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (19 lines × 2) mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:320— mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:320-338 | mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala:402-420 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala` and `mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 81 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (19 lines × 2) mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala:57— mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala:57-75 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:225-243 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/linalg/MatrixUDT.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: 87% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (19 lines × 2) mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala:33— mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala:33-51 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:286-304 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/linalg/VectorUDT.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: 87% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (19 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:396— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:396-414 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:1291-1309 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:396` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:67— sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:67-85 | sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:106-124 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:67` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:125— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:125-143 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:195-213 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:125` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19 lines × 2) python/pyspark/eval_handlers/_stateful.py:846— python/pyspark/eval_handlers/_stateful.py:846-864 | python/pyspark/eval_handlers/_stateful.py:1026-1044 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (19 lines × 2) python/pyspark/pipelines/api.py:87— python/pyspark/pipelines/api.py:87-105 | python/pyspark/pipelines/api.py:445-463 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (19 lines × 2) python/pyspark/sql/pandas/conversion.py:1018— python/pyspark/sql/pandas/conversion.py:1018-1036 | python/pyspark/sql/pandas/conversion.py:1098-1116 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Note first that the copies are not typed on the same thing: `DataFrame` names `pyspark.sql` in one and `pyspark.sql.dataframe` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (19 lines × 2) examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java:61— examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java:61-79 | examples/src/main/java/org/apache/spark/examples/ml/JavaPipelineExample.java:47-65 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaPipelineExample.java` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 48 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java:61` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (21 lines × 2) connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroPartitionReaderFactory.scala:65— connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroPartitionReaderFactory.scala:65-85 | sql/core/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala:120-140 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroPartitionReaderFactory.scala:65` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (21 lines × 2) core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:167— core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:167-187 | core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala:124-144 — before extracting anything, compare `core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala` and `core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 63 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:167` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (21 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:433— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:433-453 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:401-421 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:433` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (21 lines × 2) mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:116— mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:116-136 | mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala:1461-1481 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala` and `mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 117 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/python/MLSerDe.scala:116` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (21 lines × 2) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStep.scala:85— resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStep.scala:85-105 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala:42-62 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStep.scala:85` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (21 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala:265— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala:265-285 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala:338-358 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (21 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala:213— sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala:213-233 | sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionStateBuilder.scala:106-126 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (21 lines × 2) common/sketch/src/main/java/org/apache/spark/util/sketch/Platform.java:114— common/sketch/src/main/java/org/apache/spark/util/sketch/Platform.java:114-134 | common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java:267-287 — before extracting anything, compare `common/sketch/src/main/java/org/apache/spark/util/sketch/Platform.java` and `common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java` as WHOLE FILES: 82% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 1 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (21 lines × 2) python/pyspark/ml/connect/pipeline.py:187— python/pyspark/ml/connect/pipeline.py:187-207 | python/pyspark/ml/pipeline.py:121-141 — before extracting anything, compare `python/pyspark/ml/connect/pipeline.py` and `python/pyspark/ml/pipeline.py` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 57 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Note first that the copies are not typed on the same thing: the declarations holding them bind `dataset` to `Union[DataFrame, pd.DataFrame]` in one and `DataFrame` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (21 lines × 2) python/pyspark/sql/connect/group.py:275— python/pyspark/sql/connect/group.py:275-295 | python/pyspark/sql/pandas/group_ops.py:96-116 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (21 lines × 2) python/pyspark/sql/connect/group.py:316— python/pyspark/sql/connect/group.py:316-336 | python/pyspark/sql/connect/group.py:503-523 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (21 lines × 2) python/pyspark/ml/linalg/__init__.py:1068— python/pyspark/ml/linalg/__init__.py:1068-1088 | python/pyspark/mllib/linalg/__init__.py:1251-1271 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (21 lines × 2) examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java:54— examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java:54-74 | examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKerberizedKafkaWordCount.java:90-110 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java` and `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKerberizedKafkaWordCount.java` as WHOLE FILES: 87% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKerberizedKafkaWordCount.java:111` calls `option` and `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredKafkaWordCount.java:75` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
R4 · Test Coverage· No test reaches this file · ×12
No test reaches this file core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file REDACTED:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file REDACTED:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/sql-table-utils.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file core/src/main/resources/org/apache/spark/ui/static/dataTables.rowsGroup.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file core/src/main/resources/org/apache/spark/ui/static/webui.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/executionpage.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file core/src/main/resources/org/apache/spark/ui/static/historypage-common.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
No test reaches this file core/src/main/resources/org/apache/spark/ui/static/initialize-tooltips.js:1— No test imports this module directly or transitively. Import reachability cannot see a test that executes a file by path instead of importing it, nor one that drives it through a running browser by navigating to a URL — if neither does, no test reaches this one.
Duplicated block (9 lines × 3) mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:70— mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:70-78 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:75-83 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:76-84 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (9 lines × 3) mllib/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala:167— mllib/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala:167-175 | mllib/src/main/scala/org/apache/spark/ml/r/LinearSVCWrapper.scala:153-161 | mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala:208-216 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (9 lines × 3) mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala:191— mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala:191-199 | mllib/src/main/scala/org/apache/spark/mllib/util/LogisticRegressionDataGenerator.scala:69-77 | mllib/src/main/scala/org/apache/spark/mllib/util/SVMDataGenerator.scala:40-48 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala:191` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 3) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:1103— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:1103-1111 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:1136-1144 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:1147-1155 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (9 lines × 3) REDACTED:1775— REDACTED:1775-1783 | REDACTED:2775-2783 | REDACTED:4511-4519 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (9 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:281— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:281-289 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:564-572 | sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala:209-217 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 3 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 3 times. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:281` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 3) REDACTED:187— REDACTED:187-195 | REDACTED:264-272 | REDACTED:347-355 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:187` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 3) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:197— sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:197-205 | sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:203-211 | sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnarRow.java:207-215 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 93 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:197` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:207` calls `getBinaryView` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:212` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (9 lines × 3) python/pyspark/ml/classification.py:2797— python/pyspark/ml/classification.py:2797-2805 | python/pyspark/ml/regression.py:1602-1610 | python/pyspark/ml/regression.py:1998-2006 — before extracting anything, compare `python/pyspark/ml/classification.py` and `python/pyspark/ml/regression.py` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (9 lines × 3) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:201— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:201-212 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:138-146 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:144-152 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (9 lines × 3) examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeRegressionExample.java:66— examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeRegressionExample.java:66-74 | examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java:65-73 | examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestRegressionExample.java:65-73 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Outbound HTTP without resilience core/src/main/scala/org/apache/spark/TestUtils.scala:280— `.openConnection(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:283— `.openConnection(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:291— `.openConnection(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:299— `.openConnection(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience core/src/main/scala/org/apache/spark/ui/JettyUtils.scala:306— `HttpClient(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience REDACTED:601— `urlopen(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience REDACTED:686— `urlopen(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience REDACTED:703— `urlopen(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience REDACTED:2269— `urlopen(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience resource-managers/yarn/src/main/java/org/apache/spark/deploy/yarn/AmIpFilter.java:218— `.openConnection(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
Outbound HTTP without resilience sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:1764— `.openConnection(` makes an outbound HTTP call, and nothing bounds it: no timeout, deadline, retry or circuit breaker is set for it here, and the client has no process-wide default. A slow or failing dependency will hold this service's request, thread or connection until the call gives up on its own — or never, for a client with no default timeout.
FixmeComment sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:99— // FIXME extract the right info type — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/StatefulStreamlineAggregateExec.scala:360— // FIXME: How we can prevent this to be called? — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment REDACTED:2714— # FIXME: better ordering — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `# REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment REDACTED:2727— # FIXME: better ordering — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `# REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment python/pyspark/pandas/tests/series/test_stat.py:171— # FIXME: MultiIndex.value_counts returns wrong indices. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `# REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment python/pyspark/pandas/tests/series/test_stat.py:210— # FIXME: MultiIndex.value_counts returns wrong indices. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `# REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment python/pyspark/pandas/tests/series/test_stat.py:249— # FIXME: MultiIndex.value_counts returns wrong indices. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `# REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment core/src/test/scala/org/apache/spark/shuffle/sort/SortShuffleWriterSuite.scala:200— // FIXME: this can affect other tests (if any) after this set of tests — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment core/src/main/scala/org/apache/spark/util/Utils.scala:3411— // FIXME: We copy the stream on the level of bytes to avoid encoding problems. — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
FixmeComment core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClient.scala:166— // FIXME How to handle the following cases? — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
Duplicated block (26 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:394— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:394-419 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:362-387 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (26 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2241— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2241-2266 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2278-2303 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2241` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (26 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2658— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2658-2683 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2695-2720 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2658` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (26 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala:350— sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala:350-375 | sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala:397-422 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/classic/RelationalGroupedDataset.scala:350` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (26 lines × 2) python/pyspark/eval_handlers/_stateful.py:542— python/pyspark/eval_handlers/_stateful.py:542-567 | python/pyspark/eval_handlers/_stateful.py:867-892 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (26 lines × 2) python/pyspark/pandas/groupby.py:4287— python/pyspark/pandas/groupby.py:4287-4312 | python/pyspark/pandas/groupby.py:4497-4522 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/groupby.py:4287` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (26 lines × 2) python/pyspark/pandas/plot/core.py:355— python/pyspark/pandas/plot/core.py:355-380 | python/pyspark/sql/plot/core.py:764-789 — before extracting anything, compare `python/pyspark/pandas/plot/core.py` and `python/pyspark/sql/plot/core.py` as WHOLE FILES: this scan already matched 8 separate duplicated blocks between them, totalling at least 114 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (26 lines × 2) python/pyspark/pandas/plot/core.py:432— python/pyspark/pandas/plot/core.py:432-457 | python/pyspark/sql/plot/core.py:543-568 — before extracting anything, compare `python/pyspark/pandas/plot/core.py` and `python/pyspark/sql/plot/core.py` as WHOLE FILES: this scan already matched 8 separate duplicated blocks between them, totalling at least 114 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (26 lines × 2) python/pyspark/ml/connect/base.py:240— python/pyspark/ml/connect/base.py:240-267 | python/pyspark/ml/evaluation.py:87-112 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (26 lines × 2) examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:215— examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:215-240 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:223-248 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala` as WHOLE FILES: this scan already matched 11 separate duplicated blocks between them, totalling at least 165 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:215` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 3) mllib/src/main/scala/org/apache/spark/mllib/regression/Lasso.scala:60— mllib/src/main/scala/org/apache/spark/mllib/regression/Lasso.scala:60-66 | mllib/src/main/scala/org/apache/spark/mllib/regression/LinearRegression.scala:60-66 | mllib/src/main/scala/org/apache/spark/mllib/regression/RidgeRegression.scala:60-66 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/regression/Lasso.scala:60` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 3) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1205— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1205-1211 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1243-1249 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1307-1313 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1205` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala:90— sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala:90-96 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala:56-62 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledJoin.scala:133-139 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala:90` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala:98` calls `withNullability` and `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala:63` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (7 lines × 3) mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala:74— mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala:74-80 | mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala:75-81 | mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala:126-132 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala:72` calls `setDefault` and `mllib/src/main/scala/org/apache/spark/ml/feature/StopWordsRemover.scala:123` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (7 lines × 3) mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:182— mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:182-188 | mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:218-224 | mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:256-262 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:182` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 3) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java:383— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java:383-389 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeMapData.java:120-126 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java:679-685 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 46 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (7 lines × 3) common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java:485— common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java:485-491 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:160-166 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java:146-152 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 100 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (7 lines × 3) python/pyspark/pandas/datetimes.py:166— python/pyspark/pandas/datetimes.py:166-172 | python/pyspark/pandas/frame.py:3926-3932 | python/pyspark/pandas/frame.py:4005-4011 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/datetimes.py:166` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (7 lines × 3) python/pyspark/pandas/groupby.py:411— python/pyspark/pandas/groupby.py:411-417 | python/pyspark/pandas/groupby.py:1491-1497 | python/pyspark/pandas/groupby.py:4923-4929 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (7 lines × 3) python/pyspark/pandas/indexes/datetimes.py:171— python/pyspark/pandas/indexes/datetimes.py:171-177 | python/pyspark/pandas/indexes/multi.py:982-988 | python/pyspark/pandas/indexes/timedelta.py:144-151 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (23 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:834— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:834-856 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:977-999 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (23 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:398— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:398-420 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:444-466 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:398` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (23 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1764— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1764-1786 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1851-1873 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1764` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (23 lines × 2) python/pyspark/ml/linalg/__init__.py:1247— python/pyspark/ml/linalg/__init__.py:1247-1269 | python/pyspark/mllib/linalg/__init__.py:1442-1464 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (23 lines × 2) python/pyspark/pandas/indexes/multi.py:527— python/pyspark/pandas/indexes/multi.py:527-549 | python/pyspark/pandas/indexes/multi.py:574-596 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/indexes/multi.py:527` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (23 lines × 2) python/pyspark/pandas/series.py:4546— python/pyspark/pandas/series.py:4546-4568 | python/pyspark/pandas/series.py:4665-4687 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (23 lines × 2) python/pyspark/pandas/series.py:6574— python/pyspark/pandas/series.py:6574-6596 | python/pyspark/pandas/series.py:6649-6671 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (23 lines × 2) python/pyspark/sql/classic/dataframe.py:239— python/pyspark/sql/classic/dataframe.py:239-261 | python/pyspark/sql/connect/dataframe.py:2198-2220 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own.
Duplicated block (23 lines × 2) examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKafkaWordCount.scala:52— examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKafkaWordCount.scala:52-74 | examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKerberizedKafkaWordCount.scala:91-113 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKafkaWordCount.scala` and `examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKerberizedKafkaWordCount.scala` as WHOLE FILES: 85% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKerberizedKafkaWordCount.scala:114` calls `option` and `examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredKafkaWordCount.scala:75` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (8 lines × 3) sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:482— sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:482-489 | sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:492-499 | sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:502-509 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (8 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala:618— sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala:618-625 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala:728-735 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala:834-841 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala:618` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVScan.scala:60— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVScan.scala:60-67 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScan.scala:59-66 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextScan.scala:58-65 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 3 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 3 times.
Duplicated block (8 lines × 3) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:208— sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:208-215 | sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:212-219 | sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnarRow.java:216-223 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 93 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:208` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:207` calls `getBinaryView` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:211` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (8 lines × 3) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:483— sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:483-490 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:497-504 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:542-549 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java:483` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (8 lines × 3) python/pyspark/pandas/frame.py:1615— python/pyspark/pandas/frame.py:1615-1622 | python/pyspark/pandas/groupby.py:4208-4215 | python/pyspark/pandas/groupby.py:4406-4413 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:1615` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaMaxAbsScalerExample.java:41— examples/src/main/java/org/apache/spark/examples/ml/JavaMaxAbsScalerExample.java:41-48 | examples/src/main/java/org/apache/spark/examples/ml/JavaMinMaxScalerExample.java:41-48 | examples/src/main/java/org/apache/spark/examples/ml/JavaNormalizerExample.java:40-47 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaMaxAbsScalerExample.java:41` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaTrainValidationSplitExample.java:42— examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaTrainValidationSplitExample.java:42-49 | examples/src/main/java/org/apache/spark/examples/ml/JavaNaiveBayesExample.java:35-43 | examples/src/main/java/org/apache/spark/examples/ml/JavaOneVsRestExample.java:41-49 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (8 lines × 3) examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java:34— examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java:34-41 | examples/src/main/java/org/apache/spark/examples/mllib/JavaKMeansExample.java:34-41 | examples/src/main/java/org/apache/spark/examples/mllib/JavaLatentDirichletAllocationExample.java:39-46 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java:34` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 3) mllib/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala:250— mllib/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala:250-254 | mllib/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala:158-162 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:714-718 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 3 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 3 times. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/feature/FeatureHasher.scala:250` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 3) mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala:122— mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala:122-126 | mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:347-351 | mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:142-146 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala:122` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (5 lines × 3) mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:184— mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:184-188 | mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala:219-223 | mllib/src/main/scala/org/apache/spark/mllib/clustering/PowerIterationClustering.scala:83-88 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (5 lines × 3) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1158— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1158-1162 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1294-1298 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala:254-258 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (5 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/StatefulStreamlineAggregateExec.scala:350— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/StatefulStreamlineAggregateExec.scala:350-354 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/statefulOperators.scala:1047-1051 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/statefulOperators.scala:1313-1317 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (5 lines × 3) mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala:135— mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala:135-139 | mllib/src/main/scala/org/apache/spark/mllib/fpm/FPGrowth.scala:107-111 | mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala:653-657 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 3 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 3 times.
Duplicated block (5 lines × 3) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:552— common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:552-556 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:877-881 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:962-966 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:552` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (5 lines × 3) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:582— common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:582-586 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:640-644 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:744-748 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (5 lines × 3) examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java:70— examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java:70-74 | examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java:69-73 | examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestClassificationExample.java:69-73 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
X24 · Document value interpolated into markup unescaped· Untrusted text written into markup unescaped · ×9
Untrusted text written into markup unescaped core/src/main/resources/org/apache/spark/ui/static/environmentpage.js:125— `tableId` is text this program did not author, and it is written into a quoted attribute value through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup.
Untrusted text written into markup unescaped core/src/main/resources/org/apache/spark/ui/static/executorspage.js:81— `container.innerHTML` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup.
Untrusted text written into markup unescaped core/src/main/resources/org/apache/spark/ui/static/executorspage.js:88— `url` is text this program did not author, and it is written into the start of a URL attribute, where it chooses the scheme through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. HTML encoding does not help here — a `javascript:` URL contains nothing it changes; allow only http(s) or a relative path.
Untrusted text written into markup unescaped core/src/main/resources/org/apache/spark/ui/static/executorspage.js:117— `url` is text this program did not author, and it is written into the start of a URL attribute, where it chooses the scheme through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. HTML encoding does not help here — a `javascript:` URL contains nothing it changes; allow only http(s) or a relative path.
Untrusted text written into markup unescaped sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:61— `(summary.totalQueries||0)` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup.
Untrusted text written into markup unescaped sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:62— `formatDurationSql(summary.averageDuration||0)` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup.
Untrusted text written into markup unescaped sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:63— `(summary.runningQueries||0)` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup.
Untrusted text written into markup unescaped sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:64— `(summary.failedQueries||0)` is text this program did not author, and it is written into the text of an element through `innerHTML` with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup.
Untrusted text written into markup unescaped REDACTED:533— `formatMetricValue(val,m.type,showStageTask)` is text this program did not author, and it is written into the text of an element through markup built here with nothing encoding it for that place. Whoever controls the value controls the markup: it can close the element or attribute it sits in and open a script of its own. Encode it for HTML at this site, render it through the framework's escaping output, or sanitise it if it must stay markup.
D4 · Code Duplication· Members sharing a duplicated core (6 members, 50+ identical tokens) · ×8
Members sharing a duplicated core (6 members, 50+ identical tokens) mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:125— mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:125-138 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:108-121 | mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:132-145 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:116-129 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:135-149 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:118-132 — These 6 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
Members sharing a duplicated core (6 members, 50+ identical tokens) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:94— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:94-141 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:146-193 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala:90-138 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:70-115 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:118-164 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerSessionPage.scala:71-118 — These 6 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
Members sharing a duplicated core (6 members, 50+ identical tokens) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:126— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:126-145 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:272-289 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:555-572 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala:148-167 | sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala:118-140 | sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala:199-217 — These 6 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
Members sharing a duplicated core (6 members, 50+ identical tokens) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:529— common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:529-568 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:571-625 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:628-728 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:732-825 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:828-897 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:937-982 — These 6 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
Members sharing a duplicated core (6 members, 50+ identical tokens) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:38— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:38-113 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:124-183 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:194-257 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:265-308 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:316-359 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:397-444 — These 6 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
Members sharing a duplicated core (6 members, 50+ identical tokens) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:68— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:68-132 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:66-135 | examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala:53-101 | examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala:58-109 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:67-141 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala:70-154 — These 6 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
Members sharing a duplicated core (6 members, 50+ identical tokens) examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala:30— examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala:30-67 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRegressionExample.scala:30-66 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingClassificationExample.scala:30-67 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingRegressionExample.scala:30-66 | examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala:29-67 | examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestRegressionExample.scala:29-67 — These 6 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
Members sharing a duplicated core (6 members, 50+ identical tokens) examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java:48— examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java:48-109 | examples/src/main/java/org/apache/spark/examples/ml/JavaMaxAbsScalerExample.java:40-71 | examples/src/main/java/org/apache/spark/examples/ml/JavaMinHashLSHExample.java:48-110 | examples/src/main/java/org/apache/spark/examples/ml/JavaMinMaxScalerExample.java:40-73 | examples/src/main/java/org/apache/spark/examples/ml/JavaNormalizerExample.java:39-73 | examples/src/main/java/org/apache/spark/examples/ml/JavaVarianceThresholdSelectorExample.java:43-78 — These 6 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
Duplicated block (25 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:752— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:752-776 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:684-708 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (25 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:84— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:84-108 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcWrite.scala:48-72 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:84` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcWrite.scala:46` calls `set`, `getOrcSchemaString` and `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:82` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (25 lines × 2) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:186— sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:186-210 | sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:221-245 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:186` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (25 lines × 2) python/pyspark/pandas/internal.py:767— python/pyspark/pandas/internal.py:767-791 | python/pyspark/pandas/internal.py:795-819 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (25 lines × 2) python/pyspark/sql/classic/dataframe.py:1770— python/pyspark/sql/classic/dataframe.py:1770-1794 | python/pyspark/sql/connect/dataframe.py:1713-1737 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own.
Duplicated block (25 lines × 2) python/pyspark/sql/connect/dataframe.py:1405— python/pyspark/sql/connect/dataframe.py:1405-1429 | python/pyspark/sql/connect/dataframe.py:1484-1508 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (25 lines × 2) python/pyspark/sql/connect/readwriter.py:226— python/pyspark/sql/connect/readwriter.py:226-250 | python/pyspark/sql/connect/readwriter.py:441-465 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/connect/readwriter.py:226` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (25 lines × 2) examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala:41— examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala:41-65 | examples/src/main/scala/org/apache/spark/examples/streaming/DirectKerberizedKafkaWordCount.scala:80-104 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala` and `examples/src/main/scala/org/apache/spark/examples/streaming/DirectKerberizedKafkaWordCount.scala` as WHOLE FILES: 87% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/streaming/DirectKafkaWordCount.scala:41` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–15 lines × 2) core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:234— core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:234-247 | core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:277-291 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:234` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–15 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:1746— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:1746-1760 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:1771-1784 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:1746` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–15 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:308— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:308-321 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:1507-1521 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:308` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–15 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:320— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:320-333 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:336-350 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:320` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (14–15 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonIncrementalAggregateExec.scala:88— sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonIncrementalAggregateExec.scala:88-101 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonIncrementalAggregateExec.scala:222-236 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (14–15 lines × 2) examples/src/main/scala/org/apache/spark/examples/SparkHdfsLR.scala:81— examples/src/main/scala/org/apache/spark/examples/SparkHdfsLR.scala:81-94 | examples/src/main/scala/org/apache/spark/examples/SparkLR.scala:76-90 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/SparkHdfsLR.scala:81` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–15 lines × 2) examples/src/main/scala/org/apache/spark/examples/graphx/Analytics.scala:76— examples/src/main/scala/org/apache/spark/examples/graphx/Analytics.scala:76-90 | examples/src/main/scala/org/apache/spark/examples/graphx/Analytics.scala:110-123 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (14–15 lines × 2) examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala:33— examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala:33-46 | examples/src/main/scala/org/apache/spark/examples/mllib/MulticlassMetricsExample.scala:32-46 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassificationMetricsExample.scala:49` calls `clearThreshold` and `examples/src/main/scala/org/apache/spark/examples/mllib/MulticlassMetricsExample.scala:49` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12–13 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:124— mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:124-135 | mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:147-159 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala` and `mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 30 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:124` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12–13 lines × 2) mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala:189— mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala:189-200 | mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala:179-191 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala:189` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala:192` calls `checkNumericType` and `mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala:201` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12–13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala:140— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala:140-151 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala:162-174 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (12–13 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala:432— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala:432-444 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:3633-3644 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala:432` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala:431` calls `replaceGenerator` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:3632` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12–13 lines × 2) REDACTED:190— REDACTED:190-202 | REDACTED:365-376 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:190` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12–13 lines × 2) python/pyspark/pandas/frame.py:7695— python/pyspark/pandas/frame.py:7695-7707 | python/pyspark/pandas/indexes/base.py:1368-1379 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:7695` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately. Note first that the copies are not typed on the same thing: `Index` names `pyspark.pandas.indexes` in one and `pyspark.pandas.indexes.base` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (12–13 lines × 2) python/pyspark/sql/streaming/stateful_processor_api_client.py:426— python/pyspark/sql/streaming/stateful_processor_api_client.py:426-438 | python/pyspark/sql/streaming/stateful_processor_api_client.py:445-456 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/streaming/stateful_processor_api_client.py:426` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (12–13 lines × 2) examples/src/main/java/org/apache/spark/examples/ml/JavaVectorAssemblerExample.java:53— examples/src/main/java/org/apache/spark/examples/ml/JavaVectorAssemblerExample.java:53-64 | examples/src/main/java/org/apache/spark/examples/ml/JavaVectorSizeHintExample.java:65-77 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaVectorAssemblerExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaVectorSizeHintExample.java` as WHOLE FILES: 84% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaVectorAssemblerExample.java:53` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7–8 lines × 2) core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala:78— core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala:78-85 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/PythonStreamingSourceRunner.scala:103-109 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala:78` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `core/src/main/scala/org/apache/spark/api/python/StreamingPythonRunner.scala:87` calls `put` and `sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/PythonStreamingSourceRunner.scala:110` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (7–8 lines × 2) core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:657— core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:657-664 | sql/core/src/main/scala/org/apache/spark/sql/execution/ui/ExecutionPage.scala:442-448 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (7–8 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:339— sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:339-346 | sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:352-358 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (7–8 lines × 2) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:304— sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:304-310 | sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:362-369 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:304` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (7–8 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:224— sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:224-231 | sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:228-234 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 93 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:224` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (7–8 lines × 2) python/pyspark/sql/connect/functions/builtin.py:3395— python/pyspark/sql/connect/functions/builtin.py:3395-3402 | python/pyspark/sql/functions/builtin.py:20230-20236 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note first that the copies are not typed on the same thing: `Column` names `pyspark.sql` in one and `pyspark.sql.column` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (7–8 lines × 2) examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:46— examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:46-52 | examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java:47-54 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:46` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java:44` calls `create` and `examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:44` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (7–8 lines × 2) examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:52— examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:52-58 | examples/src/main/java/org/apache/spark/examples/ml/JavaTokenizerExample.java:53-60 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:52` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (30 lines × 2) core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:211— core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:211-240 | core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:191-220 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:211` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (30 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:841— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:841-870 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:727-756 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (30 lines × 2) REDACTED:4836— REDACTED:4836-4865 | REDACTED:5026-5055 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:4836` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (30 lines × 2) REDACTED:5058— REDACTED:5058-5087 | REDACTED:5296-5325 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:5058` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (30 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:183— sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:183-212 | sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/VariableAssignmentUtils.scala:57-86 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:183` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (30 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:70— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:70-99 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java:62-91 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 100 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:70` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (30 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:477— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:477-506 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:479-508 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 223 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:477` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11–12 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:210— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:210-220 | mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:553-564 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala` and `mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 65 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (11–12 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:107— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:107-117 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:205-216 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:107` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (11–12 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:225— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:225-235 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:257-268 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:257` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (11–12 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/statefulprocessor/StatefulProcessorHandleImpl.scala:255— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/statefulprocessor/StatefulProcessorHandleImpl.scala:255-265 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/statefulprocessor/StatefulProcessorHandleImpl.scala:304-315 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/statefulprocessor/StatefulProcessorHandleImpl.scala:255` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11–12 lines × 2) python/pyspark/pandas/frame.py:7169— python/pyspark/pandas/frame.py:7169-7180 | python/pyspark/pandas/frame.py:7183-7193 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:7169` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/pandas/frame.py:7181` calls `tuple`, `list`, `cast` and `python/pyspark/pandas/frame.py:7194` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (11–12 lines × 2) python/pyspark/pandas/plot/plotly.py:135— python/pyspark/pandas/plot/plotly.py:135-146 | python/pyspark/sql/plot/plotly.py:243-253 — before extracting anything, compare `python/pyspark/pandas/plot/plotly.py` and `python/pyspark/sql/plot/plotly.py` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 54 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/plot/plotly.py:135` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on. Note first that the copies are not typed on the same thing: the declarations holding them bind `data` to `Union["ps.DataFrame", "ps.Series"]` in one and `"DataFrame"` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (11–12 lines × 2) examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java:45— examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java:45-55 | examples/src/main/java/org/apache/spark/examples/ml/JavaPipelineExample.java:39-50 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaPipelineExample.java` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 48 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaModelSelectionViaCrossValidationExample.java:45` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (10 lines × 3) REDACTED:5007— REDACTED:5007-5016 | REDACTED:5232-5241 | REDACTED:5456-5465 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:5007` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `REDACTED:5017` calls `freshName` and `REDACTED:5466` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (10 lines × 3) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:82— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:82-91 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:58-67 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerSessionPage.scala:59-68 — before extracting anything, compare `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala` and `sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala` as WHOLE FILES: this scan already matched 9 separate duplicated blocks between them, totalling at least 118 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (10 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala:220— sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala:220-229 | sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala:521-530 | sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala:553-562 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala:216` calls `extractIdentifier`, `getTableProviderCatalog` and `sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala:549` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (10 lines × 3) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:204— common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:204-213 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:242-251 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:280-289 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (10 lines × 3) sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Avg.java:34— sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Avg.java:34-43 | sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Count.java:34-43 | sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Sum.java:34-43 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Count.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/connector/expressions/aggregate/Sum.java` as WHOLE FILES: 86% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 1 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (10 lines × 3) examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala:53— examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala:53-62 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingClassificationExample.scala:52-61 | examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala:53-62 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala` and `examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 32 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (10 lines × 3) examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java:48— examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java:48-57 | examples/src/main/java/org/apache/spark/examples/streaming/JavaSqlNetworkWordCount.java:50-59 | examples/src/main/java/org/apache/spark/examples/streaming/JavaStatefulNetworkWordCount.java:53-62 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (6 lines × 4) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2501— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2501-2506 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2539-2544 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2575-2580 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2612-2617 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2501` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 4) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:174— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:174-179 | mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala:356-362 | mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala:448-454 | mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:649-655 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 4 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 4 times.
Duplicated block (6 lines × 4) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/StatefulStreamlineAggregateExec.scala:118— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/StatefulStreamlineAggregateExec.scala:118-123 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/flatmapgroupswithstate/FlatMapGroupsWithStateExec.scala:208-213 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/statefulOperators.scala:794-799 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/statefulOperators.scala:1192-1197 — there are 4 copies across 3 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 4 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/flatmapgroupswithstate/FlatMapGroupsWithStateExec.scala:208` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (6 lines × 4) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:303— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:303-308 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:330-336 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:305-310 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:332-338 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 223 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:303` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note first that the copies are not typed on the same thing: `WriteBatch` names `org.iq80.leveldb.WriteBatch` in one and `org.rocksdb.WriteBatch` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (6 lines × 4) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:597— common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:597-602 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:666-671 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:773-778 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:871-876 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:597` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (6 lines × 4) python/pyspark/mllib/evaluation.py:683— python/pyspark/mllib/evaluation.py:683-688 | python/pyspark/mllib/feature.py:1058-1063 | python/pyspark/mllib/random.py:689-694 | python/pyspark/mllib/regression.py:1045-1050 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 4 call sites, so a change lands once.
Duplicated block (6 lines × 4) examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:45— examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:45-50 | examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:51-56 | examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java:46-52 | examples/src/main/java/org/apache/spark/examples/ml/JavaTokenizerExample.java:52-58 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 4 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:45` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java:44` calls `create` and `examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:44` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (44 lines × 2) core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:128— core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:128-171 | core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:108-151 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:128` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (44 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:539— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:539-582 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:455-498 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:539` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (44 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:791— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:791-834 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:668-711 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:791` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (44 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/AsyncCommitLog.scala:107— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/AsyncCommitLog.scala:107-150 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/AsyncOffsetSeqLog.scala:137-180 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (44 lines × 2) python/pyspark/ml/linalg/__init__.py:363— python/pyspark/ml/linalg/__init__.py:363-406 | python/pyspark/mllib/linalg/__init__.py:418-461 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note first that the copies are not typed on the same thing: the declarations holding them bind `other` to `Iterable[float]` in one and `"VectorLike"` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (44 lines × 2) python/pyspark/ml/linalg/__init__.py:1201— python/pyspark/ml/linalg/__init__.py:1201-1244 | python/pyspark/mllib/linalg/__init__.py:1396-1439 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (31 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:995— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:995-1025 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:856-886 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:995` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (31 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowWindowPythonEvaluatorFactory.scala:322— sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowWindowPythonEvaluatorFactory.scala:322-352 | sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactory.scala:116-146 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowWindowPythonEvaluatorFactory.scala:322` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (31 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:331— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:331-361 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:333-363 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 223 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:331` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note first that the copies are not typed on the same thing: `WriteBatch` names `org.iq80.leveldb.WriteBatch` in one and `org.rocksdb.WriteBatch` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (31 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:365— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:365-395 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:367-397 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 223 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:365` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note first that the copies are not typed on the same thing: `WriteBatch` names `org.iq80.leveldb.WriteBatch` in one and `org.rocksdb.WriteBatch` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (31 lines × 2) python/pyspark/sql/connect/udf.py:316— python/pyspark/sql/connect/udf.py:316-346 | python/pyspark/sql/udf.py:889-919 — before extracting anything, compare `python/pyspark/sql/connect/udf.py` and `python/pyspark/sql/udf.py` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 142 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (31 lines × 2) python/pyspark/testing/pandasutils.py:212— python/pyspark/testing/pandasutils.py:212-242 | python/pyspark/testing/pandasutils.py:267-297 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (12 lines × 3) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala:86— mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala:86-97 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala:90-101 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala:92-103 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala` and `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 36 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (12 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:186— sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:186-197 | sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/SetVariableExec.scala:72-83 | sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/VariableAssignmentUtils.scala:60-71 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/command/v2/FetchCursorExec.scala:186` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 3) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:204— sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:204-215 | sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:239-250 | sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:416-427 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:204` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 3) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/tupleSketchAgg.scala:292— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/tupleSketchAgg.scala:292-303 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/tupleUnionAgg.scala:229-240 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/tupleUnion.scala:328-339 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 3 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 3 times.
Duplicated block (12 lines × 3) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:86— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:86-97 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:166-177 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:239-250 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:86` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 3) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:535— sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:535-546 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:555-566 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:618-629 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:535` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10–11 lines × 2) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala:231— resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala:231-241 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala:257-266 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackend.scala:231` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10–11 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectHashAggregateExec.scala:132— sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectHashAggregateExec.scala:132-141 | sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala:321-331 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (10–11 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala:113— sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala:113-123 | sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala:143-152 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (10–11 lines × 2) streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala:79— streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala:79-88 | streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala:191-201 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala:79` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10–11 lines × 2) python/pyspark/pandas/groupby.py:1372— python/pyspark/pandas/groupby.py:1372-1382 | python/pyspark/pandas/groupby.py:3710-3719 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/groupby.py:1372` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10–11 lines × 2) examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java:57— examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java:57-67 | examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java:55-64 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java` and `examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 31 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java:56` calls `getTreeStrategy`, `setNumClasses` and `examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java:54` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (9–10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:418— sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:418-427 | sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:560-568 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:418` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9–10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/StateStoreColumnFamilySchemaUtils.scala:189— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/StateStoreColumnFamilySchemaUtils.scala:189-197 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/StateStoreColumnFamilySchemaUtils.scala:228-237 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/StateStoreColumnFamilySchemaUtils.scala:189` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9–10 lines × 2) python/pyspark/pandas/indexing.py:660— python/pyspark/pandas/indexing.py:660-669 | python/pyspark/pandas/indexing.py:752-760 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/indexing.py:660` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (9–10 lines × 2) python/pyspark/sql/streaming/value_state_client.py:85— python/pyspark/sql/streaming/value_state_client.py:85-94 | python/pyspark/sql/streaming/value_state_client.py:101-109 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/streaming/value_state_client.py:85` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9–10 lines × 2) examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala:114— examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala:114-123 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala:131-139 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala:114` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9–10 lines × 2) examples/src/main/java/org/apache/spark/examples/ml/JavaFPGrowthExample.java:42— examples/src/main/java/org/apache/spark/examples/ml/JavaFPGrowthExample.java:42-50 | examples/src/main/java/org/apache/spark/examples/ml/JavaWord2VecExample.java:36-45 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaFPGrowthExample.java:42` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 3) mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala:190— mllib/src/main/scala/org/apache/spark/ml/feature/ChiSqSelector.scala:190-195 | mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala:372-377 | mllib/src/main/scala/org/apache/spark/ml/feature/VarianceThresholdSelector.scala:210-215 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (6 lines × 3) mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:196— mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:196-201 | mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:233-238 | mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala:272-277 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (6 lines × 3) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:572— common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:572-577 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:629-634 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:733-738 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:572` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (6 lines × 3) sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/streaming/ReadMaxBytes.java:47— sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/streaming/ReadMaxBytes.java:47-52 | sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/streaming/ReadMaxFiles.java:46-51 | sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/streaming/ReadMaxRows.java:46-51 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (6 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java:89— examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java:89-94 | examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderExample.java:50-56 | examples/src/main/java/org/apache/spark/examples/ml/JavaTfIdfExample.java:47-52 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java:89` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (6 lines × 3) examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java:43— examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java:43-48 | examples/src/main/java/org/apache/spark/examples/mllib/JavaKMeansExample.java:43-48 | examples/src/main/java/org/apache/spark/examples/mllib/JavaLatentDirichletAllocationExample.java:48-53 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/mllib/JavaGaussianMixtureExample.java:43` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
D4 · Code Duplication· Members sharing a duplicated core (7 members, 50+ identical tokens) · ×5
Members sharing a duplicated core (7 members, 50+ identical tokens) mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:142— mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:142-154 | mllib/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala:164-177 | mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala:145-157 | mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala:112-124 | mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala:140-152 | mllib/src/main/scala/org/apache/spark/ml/r/LinearSVCWrapper.scala:150-163 | mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala:205-218 — These 7 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 7 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 7 times.
Members sharing a duplicated core (7 members, 50+ identical tokens) mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:75— mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:75-106 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:68-99 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:73-107 | mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala:70-99 | mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala:59-87 | mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala:77-109 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:74-109 — These 7 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 7 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 7 times.
Members sharing a duplicated core (7 members, 50+ identical tokens) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:506— sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:506-526 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:570-587 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:592-609 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:633-654 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:658-675 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:901-918 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:923-942 — These 7 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 7 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 7 times.
Members sharing a duplicated core (7 members, 50+ identical tokens) examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:33— examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:33-103 | examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeRegressionExample.java:34-88 | examples/src/main/java/org/apache/spark/examples/ml/JavaFMClassifierExample.java:34-104 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java:34-104 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java:35-89 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestClassifierExample.java:34-100 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java:35-89 — These 7 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 7 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 7 times.
Members sharing a duplicated core (7 members, 50+ identical tokens) examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java:38— examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java:38-78 | examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeRegressionExample.java:38-78 | examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java:38-79 | examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java:38-79 | examples/src/main/java/org/apache/spark/examples/mllib/JavaNaiveBayesExample.java:33-54 | examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestClassificationExample.java:37-79 | examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestRegressionExample.java:37-79 — These 7 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 7 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 7 times.
Duplicated block (35 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:338— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:338-372 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:492-526 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (35 lines × 2) python/pyspark/ml/linalg/__init__.py:106— python/pyspark/ml/linalg/__init__.py:106-140 | python/pyspark/mllib/linalg/__init__.py:117-151 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (35 lines × 2) python/pyspark/pipelines/api.py:175— python/pyspark/pipelines/api.py:175-209 | python/pyspark/pipelines/api.py:279-313 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pipelines/api.py:175` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (35 lines × 2) python/pyspark/sql/connect/udf.py:81— python/pyspark/sql/connect/udf.py:81-115 | python/pyspark/sql/udf.py:118-152 — before extracting anything, compare `python/pyspark/sql/connect/udf.py` and `python/pyspark/sql/udf.py` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 142 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/connect/udf.py:81` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (35 lines × 2) python/pyspark/sql/readwriter.py:215— python/pyspark/sql/readwriter.py:215-261 | python/pyspark/sql/streaming/readwriter.py:217-251 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/streaming/readwriter.py:217` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (13–14 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingForeachBatchHelper.scala:307— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingForeachBatchHelper.scala:307-320 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingQueryListenerHelper.scala:95-107 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/StreamingForeachBatchHelper.scala:307` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (13–14 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:357— sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:357-369 | sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggUtils.scala:565-578 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (13–14 lines × 2) common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java:70— common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java:70-83 | common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java:103-115 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleServiceMetrics.java:103` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (13–14 lines × 2) python/pyspark/ml/feature.py:4828— python/pyspark/ml/feature.py:4828-4840 | python/pyspark/ml/feature.py:4871-4884 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/ml/feature.py:4828` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13–14 lines × 2) examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:90— examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:90-102 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala:97-110 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:90` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala:96` calls `text` and `examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:90` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (13 lines × 3) core/src/main/scala/org/apache/spark/deploy/Utils.scala:81— core/src/main/scala/org/apache/spark/deploy/Utils.scala:81-93 | core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:165-177 | core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala:122-134 — before extracting anything, compare `core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala` and `core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 63 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (13 lines × 3) mllib/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala:147— mllib/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala:147-159 | mllib/src/main/scala/org/apache/spark/ml/r/LinearSVCWrapper.scala:133-145 | mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala:188-200 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (13 lines × 3) python/pyspark/pandas/plot/plotly.py:204— python/pyspark/pandas/plot/plotly.py:204-216 | python/pyspark/pandas/plot/plotly.py:226-238 | python/pyspark/sql/plot/plotly.py:149-161 — before extracting anything, compare `python/pyspark/pandas/plot/plotly.py` and `python/pyspark/sql/plot/plotly.py` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 54 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/plot/plotly.py:204` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 3) python/pyspark/sql/connect/client/core.py:2136— python/pyspark/sql/connect/client/core.py:2136-2148 | python/pyspark/sql/connect/client/core.py:2152-2164 | python/pyspark/sql/connect/client/core.py:2168-2180 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (13 lines × 3) examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredComplexSessionization.java:120— examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredComplexSessionization.java:120-132 | examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCountWindowed.java:70-82 | examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredSessionization.java:48-60 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (9 lines × 4) core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:67— core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:67-75 | core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala:66-74 | core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:121-129 | core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala:81-89 — before extracting anything, compare `core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala` and `core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 69 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (9 lines × 4) sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18233— sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18233-18241 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18264-18272 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18297-18305 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18333-18341 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18233` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 4) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:381— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:381-389 | mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:738-746 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:348-356 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:669-677 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:679` calls `Since` and `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:748` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (9 lines × 4) REDACTED:58— REDACTED:58-66 | REDACTED:284-292 | REDACTED:112-120 | REDACTED:111-119 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 4 call sites, so a change lands once.
Duplicated block (9 lines × 4) examples/src/main/scala/org/apache/spark/examples/streaming/CustomReceiver.scala:41— examples/src/main/scala/org/apache/spark/examples/streaming/CustomReceiver.scala:41-50 | examples/src/main/scala/org/apache/spark/examples/streaming/HdfsWordCount.scala:37-45 | examples/src/main/scala/org/apache/spark/examples/streaming/NetworkWordCount.scala:38-47 | examples/src/main/scala/org/apache/spark/examples/streaming/SqlNetworkWordCount.scala:42-51 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 4 call sites, so a change lands once.
Duplicated block (7–9 lines × 2) core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala:54— core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala:54-60 | core/src/main/scala/org/apache/spark/ui/jobs/TaskThreadDumpPage.scala:62-70 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala:54` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (7–9 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala:175— mllib/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala:175-183 | mllib/src/main/scala/org/apache/spark/mllib/classification/SVM.scala:98-104 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/classification/LogisticRegression.scala:175` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (7–9 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1159— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1159-1167 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/maskExpressions.scala:255-261 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1159` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (7–9 lines × 2) core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:240— core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:240-248 | core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:364-370 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:240` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7–9 lines × 2) python/pyspark/pandas/frame.py:11532— python/pyspark/pandas/frame.py:11532-11540 | python/pyspark/pandas/frame.py:11657-11663 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:11532` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (8–9 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java:150— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java:150-157 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java:180-188 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (8–9 lines × 2) python/pyspark/eval_handlers/_arrow.py:363— python/pyspark/eval_handlers/_arrow.py:363-370 | python/pyspark/eval_handlers/_pandas.py:659-667 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/eval_handlers/_arrow.py:363` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8–9 lines × 2) python/pyspark/pandas/frame.py:8088— python/pyspark/pandas/frame.py:8088-8096 | python/pyspark/pandas/frame.py:8219-8226 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (8–9 lines × 2) python/pyspark/pandas/plot/core.py:170— python/pyspark/pandas/plot/core.py:170-178 | python/pyspark/sql/plot/core.py:576-583 — before extracting anything, compare `python/pyspark/pandas/plot/core.py` and `python/pyspark/sql/plot/core.py` as WHOLE FILES: this scan already matched 8 separate duplicated blocks between them, totalling at least 114 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (8–9 lines × 2) examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java:89— examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java:89-96 | examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderExample.java:50-58 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaBucketizerExample.java:89` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
X17 · Uncapped recursion over a caller-supplied document· Recursive walk over a caller-supplied document has no depth cap · ×5
Recursive walk over a caller-supplied document has no depth cap python/pyspark/testing/pandasutils.py:487— `assert_eq` is public and takes `left: Any` from its caller, and it calls itself at line 487 (`self.assert_eq`) with a piece of `left` — and nowhere in `assert_eq` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `assert_eq` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack.
Recursive walk over a caller-supplied document has no depth cap python/pyspark/sql/utils.py:226— `enum_to_value` is public and takes `value: Any` from its caller, and it calls itself at line 226 (`enum_to_value`) with a piece of `value` — and nowhere in `enum_to_value` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `enum_to_value` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack.
Recursive walk over a caller-supplied document has no depth cap python/pyspark/sql/functions/builtin.py:290— `lit` is public and takes `col: Any` from its caller, and it calls itself at line 290 (`lit`) with a piece of `col` — and nowhere in `lit` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `lit` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack.
Recursive walk over a caller-supplied document has no depth cap python/pyspark/sql/connect/functions/builtin.py:286— `lit` is public and takes `col: Any` from its caller, and it calls itself at line 286 (`lit`) with a piece of `col` (and at line 294) — and nowhere in `lit` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `lit` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack.
Recursive walk over a caller-supplied document has no depth cap REDACTED:262— `from_dict` is public and takes `dct: dict` from its caller, and it calls itself at line 262 (`self.from_dict`) with a piece of `dct` — and nowhere in `from_dict` is there a depth parameter, a descent counter, or any other cap. The walk therefore spends one stack frame per LEVEL of the value it is handed, so the nesting of the caller's document — not this code — decides how deep the stack goes. Python's parsers accept far more nesting than its stack allows: `json.loads` turns 3,000 bytes of brackets into 1,500 levels and `xml.etree.ElementTree` parses tens of thousands, while a walk like this one raises `RecursionError` once it passes the interpreter's default limit of 1,000 frames. Wherever that exception lands, the work fails: a request errors, and outside any `try` the worker or the program exits with it. Raising `sys.setrecursionlimit` only moves the number the document has to beat. Thread a depth argument through `from_dict` and reject the value past a fixed maximum, or rewrite the descent with an explicit stack.
AC6 · Visual & motion safety· Animation without a prefers-reduced-motion guard · ×4
Animation without a prefers-reduced-motion guard docs/css/custom.css:907— `.nav-trigger+label` moves (`transition: left 0.4s`) and nothing reads prefers-reduced-motion for it — no @media (prefers-reduced-motion) rule for this element, no reset this project loads — so a motion-sensitive user cannot turn it off (WCAG 2.3.3). Declare it under @media (prefers-reduced-motion: no-preference), or neutralise it under reduce.
Animation without a prefers-reduced-motion guard core/src/main/resources/org/apache/spark/ui/static/webui.css:182— `.stage-details` moves (`transition: max-height 0.25s ease-out, padding 0.25s ease-out`) and nothing reads prefers-reduced-motion for it — no @media (prefers-reduced-motion) rule for this element, no reset this project loads — so a motion-sensitive user cannot turn it off (WCAG 2.3.3). Declare it under @media (prefers-reduced-motion: no-preference), or neutralise it under reduce.
Animation without a prefers-reduced-motion guard core/src/main/resources/org/apache/spark/ui/static/webui.css:225— `.stacktrace-details` moves (`transition: max-height 0.25s ease-out, padding 0.25s ease-out`) and nothing reads prefers-reduced-motion for it — no @media (prefers-reduced-motion) rule for this element, no reset this project loads — so a motion-sensitive user cannot turn it off (WCAG 2.3.3). Declare it under @media (prefers-reduced-motion: no-preference), or neutralise it under reduce.
Animation without a prefers-reduced-motion guard core/src/main/resources/org/apache/spark/ui/static/webui.css:255— `.arrow-open` moves (`transition: transform 0.2s ease`) and nothing reads prefers-reduced-motion for it — no @media (prefers-reduced-motion) rule for this element, no reset this project loads — so a motion-sensitive user cannot turn it off (WCAG 2.3.3). Declare it under @media (prefers-reduced-motion: no-preference), or neutralise it under reduce.
Change coupling: RuleIdCollection.scala ↔ TreePatterns.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleIdCollection.scala:1— `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleIdCollection.scala` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreePatterns.scala` change together 59% of the time (17 of the 29 commits that touched whichever of the two files changed less often, counting a file under its earlier names as well, and counted over this repository's 10,000 most recent commits rather than its whole history — a repo-wide or module-wide sweep is evidence about the sweep rather than about any pair inside it and is left out of BOTH sides of this ratio, while a dependency bump, a formatter/rename sweep, or a commit whose edit to one of the two files was a tool directive such as //go:generate or whitespace only is left out of the shared count ONLY, so the two sides are not taken over identical commit sets) with no explicit dependency — the edge is real but nothing declares it. Read the pair before acting: if one registers itself into the other through a hook or an initialiser, the missing dependency is DELIBERATE — the registration is the link, and it is meant not to be an import — and the thing to add is a comment on each side naming the other, not a merge; if they simply belong together, co-locate them; if neither holds, the coupling is hidden and worth breaking. You can check this without leaving the row: of the 17 shared commits counted here, the most recent 3 are `3bc29d9d` [SPARK-58933][SQL] Resolve expressions in INSERT target IDENTIFIER cl…; `16801417` [SPARK-55206][PYTHON][SQL] Transpilation minimal functional implement…; `bcdcd47c` [SPARK-57133][SQL] Add BIN BY relation operator parsing and resolution — run `git show` on any of them.
Change coupling: DataSourceV2ScanExecBase.scala ↔ GroupPartitionsExec.scala sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExecBase.scala:1— `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2ScanExecBase.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala` change together 50% of the time (6 of the 12 commits that touched whichever of the two files changed less often, counting a file under its earlier names as well, and counted over this repository's 10,000 most recent commits rather than its whole history — a repo-wide or module-wide sweep is evidence about the sweep rather than about any pair inside it and is left out of BOTH sides of this ratio, while a dependency bump, a formatter/rename sweep, or a commit whose edit to one of the two files was a tool directive such as //go:generate or whitespace only is left out of the shared count ONLY, so the two sides are not taken over identical commit sets). They sit in the same directory, and in this ecosystem sibling files there normally share one namespace/package — so a direct reference between them needs no import and this pass cannot see whether one exists. Read the pair before acting: if one file only DECLARES what the other consumes (a constants/types file beside its user), the co-change is definitional and the question is whether the split earns its keep; if they duplicate structure, extract the common part into a shared function or type they both call; if neither holds, the coupling is hidden and worth breaking. You can check this without leaving the row: of the 6 shared commits counted here, the most recent 3 are `ad175d80` [SPARK-59981][SQL] Derive the partition-key ordering when a V2 scan's…; `1af0063e` [SPARK-59995][SQL] Drop sort orders that hold a partition transform f…; `1b8ef5fa` [SPARK-59285][SQL] Hold a KeyedPartitioning's shared partition layout… — run `git show` on any of them.
Change coupling: api.py ↔ spark_connect_graph_element_registry.py python/pyspark/pipelines/api.py:1— `python/pyspark/pipelines/api.py` and `python/pyspark/pipelines/spark_connect_graph_element_registry.py` change together 50% of the time (6 of the 12 commits that touched whichever of the two files changed less often, counting a file under its earlier names as well, and counted over this repository's 10,000 most recent commits rather than its whole history — a repo-wide or module-wide sweep is evidence about the sweep rather than about any pair inside it and is left out of BOTH sides of this ratio, while a dependency bump, a formatter/rename sweep, or a commit whose edit to one of the two files was a tool directive such as //go:generate or whitespace only is left out of the shared count ONLY, so the two sides are not taken over identical commit sets) with no explicit dependency between them. They sit in the same directory, but in this ecosystem each file is its own module — a sibling reference still needs an import — so the missing import edge is real: the coupling runs through shared behaviour, not a declared dependency. If they duplicate structure, extract the common part into one unit; otherwise the coupling is hidden and worth breaking. You can check this without leaving the row: of the 6 shared commits counted here, the most recent 3 are `324dfbe4` [SPARK-59447][CONNECT][PYTHON] Add ignore-null updates to the Auto CD…; `3c28a9c0` [SPARK-57594][SDP] Add spark_confs support to create_auto_cdc_flow; `3d593617` [SPARK-58247][CONNECT][SDP] Support SCD Type 2 in the Auto CDC Python… — run `git show` on any of them.
Change coupling: CTESubstitution.scala ↔ PushdownPredicatesAndPruneColumnsForCTEDef.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala:1— `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushdownPredicatesAndPruneColumnsForCTEDef.scala` change together 50% of the time (5 of the 10 commits that touched whichever of the two files changed less often, counting a file under its earlier names as well, and counted over this repository's 10,000 most recent commits rather than its whole history — a repo-wide or module-wide sweep is evidence about the sweep rather than about any pair inside it and is left out of BOTH sides of this ratio, while a dependency bump, a formatter/rename sweep, or a commit whose edit to one of the two files was a tool directive such as //go:generate or whitespace only is left out of the shared count ONLY, so the two sides are not taken over identical commit sets) with no explicit dependency — the edge is real but nothing declares it. Read the pair before acting: if one registers itself into the other through a hook or an initialiser, the missing dependency is DELIBERATE — the registration is the link, and it is meant not to be an import — and the thing to add is a comment on each side naming the other, not a merge; if they simply belong together, co-locate them; if neither holds, the coupling is hidden and worth breaking. You can check this without leaving the row: of the 5 shared commits counted here, the most recent 3 are `6f8fa6b0` [SPARK-52100][SQL] Enable overriding the recursion level limit in syntax; `88addf46` [SPARK-51337][SQL] Add maxRows to CTERelationDef and CTERelationRef; `3b114af4` [SPARK-50739][SQL] Recursive CTE. Analyzer changes to unravel and res… — run `git show` on any of them.
D4 · Code Duplication· Members sharing a duplicated core (5 members, 50+ identical tokens) · ×4
Members sharing a duplicated core (5 members, 50+ identical tokens) mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:120— mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:120-152 | mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:143-182 | mllib/src/main/scala/org/apache/spark/ml/regression/DecisionTreeRegressor.scala:117-140 | mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:330-438 | mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:137-167 — These 5 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 5 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 5 times.
Members sharing a duplicated core (5 members, 50+ identical tokens) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:112— sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:112-221 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowEvalPythonEvaluatorFactory.scala:112-167 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonEvaluatorFactory.scala:82-148 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonUDTFExec.scala:55-145 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonIncrementalAggregateExec.scala:353-369 — These 5 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 5 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 5 times.
Members sharing a duplicated core (5 members, 50+ identical tokens) examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:62— examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:62-103 | examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala:40-63 | examples/src/main/scala/org/apache/spark/examples/mllib/MovieLensALS.scala:52-97 | examples/src/main/scala/org/apache/spark/examples/mllib/MultivariateSummarizer.scala:41-64 | examples/src/main/scala/org/apache/spark/examples/mllib/SampledRDDs.scala:38-59 — These 5 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 5 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 5 times.
Members sharing a duplicated core (5 members, 50+ identical tokens) REDACTED:57— REDACTED:57-80 | examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java:55-97 | examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java:94-139 | examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java:47-72 | examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java:113-158 — These 5 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 5 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 5 times.
Duplicated block (28 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1375— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1375-1402 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1470-1497 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (28 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:892— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:892-919 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:1046-1073 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:892` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:921` calls `close` and `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:1074` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (28 lines × 2) common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java:64— common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java:64-91 | common/network-common/src/main/java/org/apache/spark/network/protocol/SslMessageEncoder.java:67-94 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java:64` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (28 lines × 2) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1425— common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1425-1452 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1505-1532 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1425` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17–19 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:4005— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:4005-4023 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:4083-4099 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:4005` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17–19 lines × 2) python/pyspark/pandas/frame.py:8978— python/pyspark/pandas/frame.py:8978-8996 | REDACTED:3418-3434 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `REDACTED:3416` calls `isinstance`, `ValueError` and `python/pyspark/pandas/frame.py:8977` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (17–19 lines × 2) python/pyspark/pandas/frame.py:10950— python/pyspark/pandas/frame.py:10950-10966 | python/pyspark/pandas/frame.py:10982-11000 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (17–19 lines × 2) python/pyspark/shuffle.py:356— python/pyspark/shuffle.py:356-374 | python/pyspark/shuffle.py:803-819 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (15 lines × 3) REDACTED:1566— REDACTED:1566-1580 | REDACTED:1704-1718 | REDACTED:2792-2806 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:1566` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:200— sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:200-214 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:244-258 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:300-314 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:200` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:299` calls `GenericInternalRow` and `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:199` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (15 lines × 3) python/pyspark/sql/connect/functions/builtin.py:2429— python/pyspark/sql/connect/functions/builtin.py:2429-2443 | python/pyspark/sql/connect/functions/builtin.py:2450-2464 | python/pyspark/sql/connect/functions/builtin.py:2471-2485 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (15 lines × 3) python/pyspark/pandas/missing/groupby.py:22— python/pyspark/pandas/missing/groupby.py:22-36 | python/pyspark/pandas/missing/resample.py:23-37 | python/pyspark/pandas/missing/window.py:22-72 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (13 lines × 4) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:786— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:786-798 | mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:859-871 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:663-675 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:725-737 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:786` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 4) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:122— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:122-134 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java:108-120 | common/network-common/src/main/java/org/apache/spark/network/shuffledb/LevelDBIterator.java:40-52 | common/network-common/src/main/java/org/apache/spark/network/shuffledb/RocksDBIterator.java:46-58 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 100 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (13 lines × 4) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:133— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:133-145 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:203-215 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:268-280 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:319-331 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:133` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 4) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:76— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:76-88 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:74-86 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:75-87 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala:82-94 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (10 lines × 4) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:576— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:576-585 | mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:723-732 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:721-730 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:878-887 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:723` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:732` calls `Since` and `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:587` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (10 lines × 4) mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:77— mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:77-86 | mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala:72-81 | mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala:61-70 | mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala:79-88 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 39 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (10 lines × 4) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala:127— sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala:127-136 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonTable.scala:62-71 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/orc/OrcTable.scala:56-65 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetTable.scala:58-67 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 4 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 4 times. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala:127` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 4) mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala:363— mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala:363-372 | mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:488-497 | mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:1405-1414 | mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala:385-394 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala` and `mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 65 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (5 lines × 4) mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala:237— mllib/src/main/scala/org/apache/spark/ml/feature/Binarizer.scala:237-241 | mllib/src/main/scala/org/apache/spark/ml/feature/Bucketizer.scala:223-227 | mllib/src/main/scala/org/apache/spark/ml/feature/Imputer.scala:309-313 | mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoder.scala:401-405 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 4 call sites, so a change lands once.
Duplicated block (5 lines × 4) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:536— common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:536-540 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:580-584 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:856-860 | common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:943-947 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:536` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:586` calls `equals` and `common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:542` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (5 lines × 4) python/pyspark/eval_handlers/_arrow.py:280— python/pyspark/eval_handlers/_arrow.py:280-284 | python/pyspark/eval_handlers/_arrow.py:352-356 | python/pyspark/eval_handlers/_arrow.py:548-552 | python/pyspark/eval_handlers/_arrow.py:601-605 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (5 lines × 4) python/pyspark/pandas/plot/core.py:168— python/pyspark/pandas/plot/core.py:168-172 | python/pyspark/pandas/plot/core.py:395-399 | python/pyspark/sql/plot/core.py:503-507 | python/pyspark/sql/plot/core.py:574-578 — before extracting anything, compare `python/pyspark/pandas/plot/core.py` and `python/pyspark/sql/plot/core.py` as WHOLE FILES: this scan already matched 8 separate duplicated blocks between them, totalling at least 114 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (27 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java:139— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java:139-165 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VariableLengthRowBasedKeyValueBatch.java:156-182 — before extracting anything, compare `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java` and `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VariableLengthRowBasedKeyValueBatch.java` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 51 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java:139` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (27 lines × 2) python/pyspark/pandas/plot/matplotlib.py:431— python/pyspark/pandas/plot/matplotlib.py:431-457 | python/pyspark/pandas/plot/matplotlib.py:552-578 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (27 lines × 2) python/pyspark/sql/connect/functions/builtin.py:2838— python/pyspark/sql/connect/functions/builtin.py:2838-2864 | python/pyspark/sql/functions/builtin.py:16712-16738 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on. Note first that the copies are not typed on the same thing: `Column` names `pyspark.sql` in one and `pyspark.sql.column` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (27 lines × 2) python/pyspark/sql/connect/protobuf/functions.py:39— python/pyspark/sql/connect/protobuf/functions.py:39-65 | python/pyspark/sql/connect/protobuf/functions.py:78-104 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Unfinished stub — body is a "not implemented" error core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:95— `invokeAll` is a shipped member whose whole body is still a "not implemented" error, and nothing in the repository overrides it — generated scaffolding that was never completed. Implement it or remove the dead surface.
Unfinished stub — body is a "not implemented" error core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:99— `invokeAll` is a shipped member whose whole body is still a "not implemented" error, and nothing in the repository overrides it — generated scaffolding that was never completed. Implement it or remove the dead surface.
Unfinished stub — body is a "not implemented" error core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:104— `invokeAny` is a shipped member whose whole body is still a "not implemented" error, and nothing in the repository overrides it — generated scaffolding that was never completed. Implement it or remove the dead surface.
Unfinished stub — body is a "not implemented" error core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:107— `invokeAny` is a shipped member whose whole body is still a "not implemented" error, and nothing in the repository overrides it — generated scaffolding that was never completed. Implement it or remove the dead surface.
Unfinished stub — throws a "not implemented" exception core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:95— `invokeAll` throws an exception whose message says it isn't finished — a not-implemented stub wearing an ordinary exception. Implement it or remove it.
Unfinished stub — throws a "not implemented" exception core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:99— `invokeAll` throws an exception whose message says it isn't finished — a not-implemented stub wearing an ordinary exception. Implement it or remove it.
Unfinished stub — throws a "not implemented" exception core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:104— `invokeAny` throws an exception whose message says it isn't finished — a not-implemented stub wearing an ordinary exception. Implement it or remove it.
Unfinished stub — throws a "not implemented" exception core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:107— `invokeAny` throws an exception whose message says it isn't finished — a not-implemented stub wearing an ordinary exception. Implement it or remove it.
Duplicated block (37 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:591— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:591-627 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:506-542 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:591` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (37 lines × 2) python/pyspark/pandas/groupby.py:2480— python/pyspark/pandas/groupby.py:2480-2516 | python/pyspark/pandas/groupby.py:2577-2613 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (37 lines × 2) python/pyspark/pandas/frame.py:8332— python/pyspark/pandas/frame.py:8332-8387 | python/pyspark/pandas/series.py:3272-3308 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (34 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:223— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:223-256 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:225-258 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (34 lines × 2) python/pyspark/pandas/frame.py:12712— python/pyspark/pandas/frame.py:12712-12745 | python/pyspark/pandas/frame.py:12847-12880 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:12712` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (34 lines × 2) python/pyspark/sql/connect/udf.py:255— python/pyspark/sql/connect/udf.py:255-288 | python/pyspark/sql/udf.py:714-747 — before extracting anything, compare `python/pyspark/sql/connect/udf.py` and `python/pyspark/sql/udf.py` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 142 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Note first that the copies are not typed on the same thing: `Column` names `pyspark.sql.column` in one and `pyspark.sql.connect.column` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (32 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:811— sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:811-842 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:851-882 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:811` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (32 lines × 2) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:228— sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:228-259 | sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:338-369 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:228` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (32 lines × 2) python/pyspark/pandas/strings.py:2074— python/pyspark/pandas/strings.py:2074-2105 | python/pyspark/pandas/strings.py:2234-2265 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (29 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:322— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:322-350 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala:359-387 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 153 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:322` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (29 lines × 2) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:182— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:182-210 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:184-212 — before extracting anything, compare `common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java` and `common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 223 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (29 lines × 2) python/pyspark/eval_handlers/_stateful.py:758— python/pyspark/eval_handlers/_stateful.py:758-786 | python/pyspark/eval_handlers/_stateful.py:1047-1075 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (24 lines × 2) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:582— core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:582-605 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala:198-221 — before extracting anything, compare `core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala` and `resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 95 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:582` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (24 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:168— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:168-191 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:170-193 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (24 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:445— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:445-468 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:592-615 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (19–20 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala:164— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala:164-182 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/percentiles.scala:98-117 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproximatePercentile.scala:164` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19–20 lines × 2) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1454— common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1454-1473 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1555-1573 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1454` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (19–20 lines × 2) python/pyspark/pandas/frame.py:9006— python/pyspark/pandas/frame.py:9006-9025 | REDACTED:3467-3485 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:9006` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (17–18 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:90— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:90-107 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:146-162 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:90` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17–18 lines × 2) python/pyspark/ml/tuning.py:1164— python/pyspark/ml/tuning.py:1164-1181 | python/pyspark/ml/tuning.py:1724-1740 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/ml/tuning.py:1164` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17–18 lines × 2) examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java:56— examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java:56-72 | examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java:95-112 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java` and `examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 34 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (16–17 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:96— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:96-112 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:72-87 — before extracting anything, compare `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala` and `sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala` as WHOLE FILES: this scan already matched 9 separate duplicated blocks between them, totalling at least 118 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:96` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16–17 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:244— sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:244-260 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/TransformWithStateExec.scala:515-530 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:244` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16–17 lines × 2) python/pyspark/shuffle.py:376— python/pyspark/shuffle.py:376-392 | python/pyspark/shuffle.py:830-845 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (15–16 lines × 2) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala:123— mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala:123-137 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala:144-159 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/AFTBlockAggregator.scala:123` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15–16 lines × 2) mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:702— mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:702-716 | mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala:202-217 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala` and `mllib/src/main/scala/org/apache/spark/mllib/stat/MultivariateOnlineSummarizer.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 46 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Note first that the copies are not typed on the same thing: the declarations holding them bind `n` to `s"Nothing has been added to this summarizer.") val realSum = Array.ofDim[Double](n) var i = 0 while (i <` in one and `s"Nothing has been added to this summarizer.") val realMean = Array.ofDim[Double](n) var i = 0 while (i <` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (15–16 lines × 2) python/pyspark/ml/tuning.py:1122— python/pyspark/ml/tuning.py:1122-1137 | python/pyspark/ml/tuning.py:1684-1698 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/ml/tuning.py:1122` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–16 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala:142— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2Writes.scala:142-155 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala:798-813 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala:798` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–16 lines × 2) REDACTED:2513— REDACTED:2513-2528 | REDACTED:2629-2642 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:2513` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `REDACTED:2530` calls `get_input` and `REDACTED:2643` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (14–16 lines × 2) python/pyspark/mllib/classification.py:392— python/pyspark/mllib/classification.py:392-407 | python/pyspark/mllib/regression.py:360-373 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/mllib/classification.py:392` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 4) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:830— sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:830-844 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:870-884 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:916-930 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:963-977 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:830` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 4) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeClassificationExample.scala:72— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeClassificationExample.scala:72-86 | examples/src/main/scala/org/apache/spark/examples/ml/FMClassifierExample.scala:72-86 | examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeClassifierExample.scala:75-89 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestClassifierExample.scala:74-88 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeClassifierExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/RandomForestClassifierExample.scala` as WHOLE FILES: 82% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (15 lines × 4) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeRegressionExample.scala:62— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeRegressionExample.scala:62-76 | examples/src/main/scala/org/apache/spark/examples/ml/FMRegressorExample.scala:60-74 | examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeRegressorExample.scala:62-76 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestRegressorExample.scala:61-75 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 4 call sites, so a change lands once. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `examples/src/main/scala/org/apache/spark/examples/ml/FMRegressorExample.scala:53` calls `setStepSize` and `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeRegressionExample.scala:55` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (14 lines × 3) sql/api/src/main/scala/org/apache/spark/sql/Row.scala:496— sql/api/src/main/scala/org/apache/spark/sql/Row.scala:496-509 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala:110-123 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala:113-126 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 3 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 3 times. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/Row.scala:496` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14 lines × 3) python/pyspark/sql/pandas/types.py:338— python/pyspark/sql/pandas/types.py:338-351 | python/pyspark/sql/pandas/types.py:355-368 | python/pyspark/sql/pandas/types.py:372-385 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (14 lines × 3) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:254— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:254-267 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:193-206 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:201-214 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:254` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:192` calls `setMaxIter` and `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:253` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12–14 lines × 2) core/src/main/scala/org/apache/spark/SparkContext.scala:1106— core/src/main/scala/org/apache/spark/SparkContext.scala:1106-1117 | sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala:1030-1043 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (12–14 lines × 2) core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:343— core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:343-354 | core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:285-298 — before extracting anything, compare `core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala` and `core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 50 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:343` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12–14 lines × 2) python/pyspark/pandas/plot/core.py:315— python/pyspark/pandas/plot/core.py:315-328 | python/pyspark/sql/plot/core.py:726-737 — before extracting anything, compare `python/pyspark/pandas/plot/core.py` and `python/pyspark/sql/plot/core.py` as WHOLE FILES: this scan already matched 8 separate duplicated blocks between them, totalling at least 114 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/plot/core.py:315` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (10–12 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala:481— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala:481-490 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala:509-520 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala:481` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10–12 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:356— sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:356-367 | sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:387-396 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:356` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10–12 lines × 2) python/pyspark/mllib/feature.py:1053— python/pyspark/mllib/feature.py:1053-1062 | python/pyspark/mllib/random.py:682-693 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (8–9 lines × 3) core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:203— core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:203-210 | core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:240-248 | core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:284-292 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (8–9 lines × 3) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java:427— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java:427-435 | sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ConstantColumnVector.java:287-294 | sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java:471-479 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 3 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 3 times. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java:427` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8–9 lines × 3) python/pyspark/pandas/groupby.py:2502— python/pyspark/pandas/groupby.py:2502-2509 | python/pyspark/pandas/groupby.py:2599-2606 | python/pyspark/pandas/groupby.py:3779-3787 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/groupby.py:2502` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12–13 lines × 3) python/pyspark/sql/pandas/types.py:1233— python/pyspark/sql/pandas/types.py:1233-1245 | python/pyspark/sql/pandas/types.py:1555-1566 | python/pyspark/sql/pandas/types.py:1588-1599 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/pandas/types.py:1233` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12–13 lines × 3) examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala:32— examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala:32-44 | examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala:31-42 | examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestRegressionExample.scala:31-42 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala` and `examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 32 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (12–13 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:38— examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:38-49 | examples/src/main/java/org/apache/spark/examples/ml/JavaTfIdfExample.java:40-51 | examples/src/main/java/org/apache/spark/examples/ml/JavaTokenizerExample.java:45-57 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:38` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 4) python/pyspark/eval_handlers/_arrow.py:140— python/pyspark/eval_handlers/_arrow.py:140-146 | python/pyspark/eval_handlers/_arrow.py:210-216 | python/pyspark/eval_handlers/_pandas.py:377-383 | python/pyspark/eval_handlers/_pandas.py:457-463 — there are 4 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 4 sites; resolving a subset leaves the remainder to drift apart. Note first that the copies are not typed on the same thing: the declarations holding them bind `eval_conf` to `EvalConf` in one and `"EvalConf"` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (7 lines × 4) python/pyspark/mllib/classification.py:610— python/pyspark/mllib/classification.py:610-618 | python/pyspark/mllib/regression.py:232-238 | python/pyspark/mllib/regression.py:443-449 | python/pyspark/mllib/regression.py:620-626 — there are 4 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 4 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (7 lines × 4) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:89— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:89-95 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:90-96 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:96-102 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala:104-110 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (19 lines × 3) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeRegressionExample.scala:32— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeRegressionExample.scala:32-50 | examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeRegressorExample.scala:31-49 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestRegressorExample.scala:31-49 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (19 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeRegressionExample.java:35— examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeRegressionExample.java:35-53 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java:36-54 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java:36-54 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 48 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (19 lines × 3) examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredComplexSessionization.java:112— examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredComplexSessionization.java:112-130 | examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCount.java:41-59 | examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredSessionization.java:40-58 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (11 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:60— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:60 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:77 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (11 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:940— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:940 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1032 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (11 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:982— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:982 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1022 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (10 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:892— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:892 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:902 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (10 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1024— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1024 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1034 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (10 lines × 2 locations) REDACTED:57— REDACTED:57 · REDACTED:719 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (5 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:294— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:294 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:308 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (5 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:150— core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:150 · core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:259 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (5 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:191— core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:191 · core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:198 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
(anonymous) (cyclomatic 24) REDACTED:109— (anonymous) has cyclomatic complexity 24 (threshold 15). Most of this is not in the body itself: 3 of the 24 points are its own statements and the rest belongs to 16 function items inside it that branch (a function literal at line 126, a function literal at line 291, (anonymous)::(anonymous)::render, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
(anonymous) (cyclomatic 24) REDACTED:1093— (anonymous) has cyclomatic complexity 24 (threshold 15). Most of this is not in the body itself: 7 of the 24 points are its own statements and the rest belongs to 7 function literals inside it that branch (lines 1118, 1139, 1142, …). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
LocalDataToArrowConversion._create_converter (cyclomatic 23) python/pyspark/sql/conversion.py:664— LocalDataToArrowConversion._create_converter has cyclomatic complexity 23 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages. This shape REPEATS in the file: one other method here (LocalDataToArrowConversion._create_converter) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
LocalDataToArrowConversion._create_converter (cyclomatic 23) python/pyspark/sql/conversion.py:674— LocalDataToArrowConversion._create_converter has cyclomatic complexity 23 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages. This shape REPEATS in the file: one other method here (LocalDataToArrowConversion._create_converter) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
DataFrame.replace (cyclomatic 21) python/pyspark/sql/classic/dataframe.py:1549— DataFrame.replace has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataFrame.replace (cyclomatic 21) python/pyspark/sql/connect/dataframe.py:1515— DataFrame.replace has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ArrowTableToRowsConversion._create_converter (cyclomatic 19) python/pyspark/sql/conversion.py:1290— ArrowTableToRowsConversion._create_converter has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages. This shape REPEATS in the file: one other method here (ArrowTableToRowsConversion._create_converter) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
ArrowTableToRowsConversion._create_converter (cyclomatic 19) python/pyspark/sql/conversion.py:1296— ArrowTableToRowsConversion._create_converter has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages. This shape REPEATS in the file: one other method here (ArrowTableToRowsConversion._create_converter) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
Index.addOrRemove (cyclomatic 16) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:323— Index.addOrRemove has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Index.addOrRemove (cyclomatic 16) common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:325— Index.addOrRemove has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
BLAS.gemv (cognitive 30) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:774— BLAS.gemv has cognitive complexity 30 (threshold 15). Drivers by points: if/else 11 (17 pts), loops 4 (11 pts), boolean chains 2 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BLAS.gemv (cognitive 30) mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:651— BLAS.gemv has cognitive complexity 30 (threshold 15). Drivers by points: if/else 11 (17 pts), loops 4 (11 pts), boolean chains 2 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparseVector.argmax (cognitive 30) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:735— SparseVector.argmax has cognitive complexity 30 (threshold 15). Drivers by points: if/else 8 (16 pts), loops 3 (11 pts), boolean chains 3 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparseVector.argmax (cognitive 30) mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:890— SparseVector.argmax has cognitive complexity 30 (threshold 15). Drivers by points: if/else 8 (16 pts), loops 3 (11 pts), boolean chains 3 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Index.addOrRemove (cognitive 25) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:323— Index.addOrRemove has cognitive complexity 25 (threshold 15). Drivers by points: if/else 9 (15 pts), ternaries 3 (6 pts), boolean chains 4 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Index.addOrRemove (cognitive 25) common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:325— Index.addOrRemove has cognitive complexity 25 (threshold 15). Drivers by points: if/else 9 (15 pts), ternaries 3 (6 pts), boolean chains 4 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Index.toKey (cognitive 23) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:457— Index.toKey has cognitive complexity 23 (threshold 15). Drivers by points: if/else 10 (13 pts), loops 3 (6 pts), ternaries 2 (4 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Index.toKey (cognitive 23) common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:459— Index.toKey has cognitive complexity 23 (threshold 15). Drivers by points: if/else 10 (13 pts), loops 3 (6 pts), ternaries 2 (4 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Vectors.norm (cognitive 19) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:336— Vectors.norm has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (11 pts), loops 2 (6 pts), match/switch 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Vectors.norm (cognitive 19) mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:490— Vectors.norm has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (11 pts), loops 2 (6 pts), match/switch 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Pipeline._fit (cognitive 19) python/pyspark/ml/connect/pipeline.py:186— Pipeline._fit has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 3, boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Pipeline._fit (cognitive 19) python/pyspark/ml/pipeline.py:120— Pipeline._fit has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 3, boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparseVector.squared_distance (cognitive 19) python/pyspark/ml/linalg/__init__.py:719— SparseVector.squared_distance has cognitive complexity 19 (threshold 15). Drivers by points: if/else 6 (10 pts), loops 3 (6 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparseVector.squared_distance (cognitive 19) python/pyspark/mllib/linalg/__init__.py:826— SparseVector.squared_distance has cognitive complexity 19 (threshold 15). Drivers by points: if/else 6 (10 pts), loops 3 (6 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LocalDataToArrowConversion._create_converter (cognitive 19) python/pyspark/sql/conversion.py:664— LocalDataToArrowConversion._create_converter has cognitive complexity 19 (threshold 15). Drivers by points: if/else 11 (15 pts), ternaries 1 (3 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (LocalDataToArrowConversion._create_converter) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
LocalDataToArrowConversion._create_converter (cognitive 19) python/pyspark/sql/conversion.py:674— LocalDataToArrowConversion._create_converter has cognitive complexity 19 (threshold 15). Drivers by points: if/else 11 (15 pts), ternaries 1 (3 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (LocalDataToArrowConversion._create_converter) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
SparseMatrix.genRandMatrix (cognitive 18) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:976— SparseMatrix.genRandMatrix has cognitive complexity 18 (threshold 15). Drivers by points: if/else 5 (8 pts), loops 3 (7 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparseMatrix.genRandMatrix (cognitive 18) mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:837— SparseMatrix.genRandMatrix has cognitive complexity 18 (threshold 15). Drivers by points: if/else 5 (8 pts), loops 3 (7 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparseVector.__init__ (cognitive 17) python/pyspark/ml/linalg/__init__.py:545— SparseVector.__init__ has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (15 pts), loops 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (SparseVector.__init__) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
SparseVector.__init__ (cognitive 17) python/pyspark/ml/linalg/__init__.py:556— SparseVector.__init__ has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (15 pts), loops 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (SparseVector.__init__) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
ArrowTableToRowsConversion._create_converter (cognitive 17) python/pyspark/sql/conversion.py:1290— ArrowTableToRowsConversion._create_converter has cognitive complexity 17 (threshold 15). Drivers by points: if/else 11 (16 pts), boolean chains 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (ArrowTableToRowsConversion._create_converter) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
ArrowTableToRowsConversion._create_converter (cognitive 17) python/pyspark/sql/conversion.py:1296— ArrowTableToRowsConversion._create_converter has cognitive complexity 17 (threshold 15). Drivers by points: if/else 11 (16 pts), boolean chains 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (ArrowTableToRowsConversion._create_converter) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
SparseMatrix.fromCOO (cognitive 16) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:918— SparseMatrix.fromCOO has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (8 pts), loops 2 (5 pts), boolean chains 3 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparseMatrix.fromCOO (cognitive 16) mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:779— SparseMatrix.fromCOO has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (8 pts), loops 2 (5 pts), boolean chains 3 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Vectors.sqdist (cognitive 16) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:387— Vectors.sqdist has cognitive complexity 16 (threshold 15). Drivers by points: if/else 4 (6 pts), boolean chains 5, loops 2 (4 pts), match/switch 1 (nesting depth added 4). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
Vectors.sqdist (cognitive 16) mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:534— Vectors.sqdist has cognitive complexity 16 (threshold 15). Drivers by points: if/else 4 (6 pts), boolean chains 5, loops 2 (4 pts), match/switch 1 (nesting depth added 4). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TransportClientFactory.createClient (cognitive 16) common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:183— TransportClientFactory.createClient has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (12 pts), boolean chains 2, error handling 1, ternaries 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
TransportClientFactory.createClient (cognitive 16) common/network-common/src/main/java/org/apache/spark/network/client/TransportClientFactory.java:282— TransportClientFactory.createClient has cognitive complexity 16 (threshold 15). Drivers by points: if/else 10 (13 pts), error handling 2, loops 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition.
SparseVector.__init__ (cognitive 16) python/pyspark/mllib/linalg/__init__.py:612— SparseVector.__init__ has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (14 pts), loops 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (SparseVector.__init__) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
SparseVector.__init__ (cognitive 16) python/pyspark/mllib/linalg/__init__.py:623— SparseVector.__init__ has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (14 pts), loops 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (SparseVector.__init__) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
DataStreamWriter.trigger (cognitive 16) python/pyspark/sql/connect/streaming/readwriter.py:569— DataStreamWriter.trigger has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (13 pts), boolean chains 3 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataStreamWriter.trigger (cognitive 16) python/pyspark/sql/streaming/readwriter.py:1358— DataStreamWriter.trigger has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (13 pts), boolean chains 3 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Near-duplicate member pair (45 shared lines) core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:269— core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:269-394 | core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:198-354 — These two members are variants of one another: 45 of their lines are already reported as duplicated blocks below, spread through both bodies rather than gathered into one. Read them as a single construct written twice. The repair is at the members' grain — factor the shared pipeline into one implementation the two call with their differences as parameters or as an injected step, or, where the difference is systematic (sync against async, one transport against another), generate one from the other. Extracting the individual blocks below is not the same fix: it leaves the two bodies in place and the next edit still has to be made twice.
Near-duplicate member pair (45 shared lines) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1244— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1244-1293 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:1107-1156 — These two members are variants of one another: 45 of their lines are already reported as duplicated blocks below, spread through both bodies rather than gathered into one. Read them as a single construct written twice. The repair is at the members' grain — factor the shared pipeline into one implementation the two call with their differences as parameters or as an injected step, or, where the difference is systematic (sync against async, one transport against another), generate one from the other. Extracting the individual blocks below is not the same fix: it leaves the two bodies in place and the next edit still has to be made twice.
D4 · Code Duplication· Members sharing a duplicated core (9 members, 50+ identical tokens) · ×2
Members sharing a duplicated core (9 members, 50+ identical tokens) mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:132— mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:132-144 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:143-156 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:126-139 | mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:150-163 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:134-147 | mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala:125-137 | mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala:135-147 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:154-167 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:137-150 — These 9 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 9 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 9 times.
Members sharing a duplicated core (9 members, 50+ identical tokens) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:614— sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:614-634 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:638-658 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:662-682 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:686-706 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:710-730 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:734-755 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:759-780 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:923-943 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:972-992 — These 9 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 9 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 9 times.
D4 · Code Duplication· Members sharing a duplicated core (8 members, 50+ identical tokens) · ×2
Members sharing a duplicated core (8 members, 50+ identical tokens) python/pyspark/pandas/generic.py:1229— python/pyspark/pandas/generic.py:1229-1298 | python/pyspark/pandas/generic.py:1307-1407 | python/pyspark/pandas/generic.py:1416-1515 | python/pyspark/pandas/generic.py:1522-1585 | python/pyspark/pandas/generic.py:1590-1654 | python/pyspark/pandas/generic.py:1867-1966 | python/pyspark/pandas/generic.py:1971-2065 | python/pyspark/pandas/generic.py:2198-2309 — These 8 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 8 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 8 times.
Members sharing a duplicated core (8 members, 50+ identical tokens) python/pyspark/sql/connect/functions/builtin.py:4370— python/pyspark/sql/connect/functions/builtin.py:4370-4413 | python/pyspark/sql/connect/functions/builtin.py:4463-4507 | python/pyspark/sql/connect/functions/builtin.py:4586-4609 | python/pyspark/sql/connect/functions/builtin.py:4644-4667 | python/pyspark/sql/functions/builtin.py:28417-28613 | python/pyspark/sql/functions/builtin.py:28661-28869 | python/pyspark/sql/functions/builtin.py:29139-29276 | python/pyspark/sql/functions/builtin.py:29309-29439 — These 8 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 8 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 8 times.
Duplicated block (42 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1306— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1306-1347 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:1169-1210 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:1306` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (42 lines × 2) sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java:1— sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java:1-159 | sql/catalyst/src/main/java/org/apache/spark/sql/connector/write/Write.java:47-88 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/Scan.java:1` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (41 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:920— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:920-960 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala:781-821 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 386 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (41 lines × 2) mllib/src/main/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionWithSGD.scala:61— mllib/src/main/scala/org/apache/spark/mllib/classification/StreamingLogisticRegressionWithSGD.scala:61-101 | mllib/src/main/scala/org/apache/spark/mllib/regression/StreamingLinearRegressionWithSGD.scala:58-117 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (36 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:733— mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:733-768 | mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:610-645 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala` as WHOLE FILES: this scan already matched 26 separate duplicated blocks between them, totalling at least 476 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (36 lines × 2) python/pyspark/sql/readwriter.py:1450— python/pyspark/sql/readwriter.py:1450-1499 | python/pyspark/sql/streaming/readwriter.py:1160-1195 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/streaming/readwriter.py:1160` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (24–25 lines × 2) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:316— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:316-339 | mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala:313-337 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:316` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (24–25 lines × 2) python/pyspark/sql/pandas/types.py:1229— python/pyspark/sql/pandas/types.py:1229-1253 | python/pyspark/sql/pandas/types.py:1584-1607 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/pandas/types.py:1229` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (23–24 lines × 2) graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala:192— graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala:192-214 | graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala:283-306 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala:192` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (23–24 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:439— sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:439-462 | sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/SortAggregateExec.scala:169-191 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:439` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (21–23 lines × 2) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala:36— resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala:36-58 | resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala:38-58 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMasterArguments.scala:36` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (21–23 lines × 2) python/pyspark/sql/classic/dataframe.py:1723— python/pyspark/sql/classic/dataframe.py:1723-1743 | python/pyspark/sql/connect/dataframe.py:1776-1798 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/connect/dataframe.py:1776` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19–22 lines × 2) core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:315— core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:315-333 | core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:254-275 — before extracting anything, compare `core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala` and `core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 50 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:315` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19–22 lines × 2) examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionWithElasticNetExample.scala:29— examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionWithElasticNetExample.scala:29-47 | examples/src/main/scala/org/apache/spark/examples/ml/MulticlassLogisticRegressionWithElasticNetExample.scala:29-50 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (20 lines × 4) core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:38— core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:38-57 | core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala:37-56 | core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:92-111 | core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala:52-71 — before extracting anything, compare `core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala` and `core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 69 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (20 lines × 4) python/pyspark/sql/avro/functions.py:204— python/pyspark/sql/avro/functions.py:204-223 | python/pyspark/sql/connect/avro/functions.py:91-110 | python/pyspark/sql/connect/protobuf/functions.py:116-135 | python/pyspark/sql/protobuf/functions.py:297-316 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 4 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 4 times.
Duplicated block (20 lines × 3) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2502— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2502-2521 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2540-2559 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2576-2595 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2502` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (20 lines × 3) examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredComplexSessionization.scala:96— examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredComplexSessionization.scala:96-115 | examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredNetworkWordCount.scala:38-57 | examples/src/main/scala/org/apache/spark/examples/sql/streaming/StructuredSessionization.scala:41-60 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (16–18 lines × 2) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:102— mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:102-117 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala:98-115 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala` and `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:102` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16–18 lines × 2) python/pyspark/sql/catalog.py:364— python/pyspark/sql/catalog.py:364-379 | python/pyspark/sql/catalog.py:689-706 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (17 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:553— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:553-569 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:615-631 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:682-698 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:553` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17 lines × 3) python/pyspark/sql/readwriter.py:497— python/pyspark/sql/readwriter.py:497-513 | python/pyspark/sql/readwriter.py:906-922 | python/pyspark/sql/readwriter.py:1062-1078 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (16 lines × 3) mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:377— mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:377-392 | mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala:459-474 | mllib/src/main/scala/org/apache/spark/mllib/feature/ChiSqSelector.scala:90-105 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala` and `mllib/src/main/scala/org/apache/spark/ml/feature/UnivariateFeatureSelector.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 81 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/feature/Selector.scala:377` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16 lines × 3) REDACTED:1150— REDACTED:1150-1165 | python/pyspark/pandas/plot/matplotlib.py:766-781 | python/pyspark/pandas/plot/matplotlib.py:930-945 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:1150` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13–14 lines × 3) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:312— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:312-325 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:135-147 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:302-314 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:312` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:317` calls `getOrElse` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/toFromProtobufSqlFunctions.scala:150` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (13–14 lines × 3) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:87— common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:87-99 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:126-139 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:165-177 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (11–12 lines × 4) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:328— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:328-338 | mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:1003-1013 | mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala:326-336 | mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:589-600 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala` and `mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 65 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (11–12 lines × 4) examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:41— examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:41-51 | examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderExample.java:39-50 | examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java:37-48 | examples/src/main/java/org/apache/spark/examples/ml/JavaStringIndexerExample.java:38-48 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 4 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:41` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderExample.java:50` calls `create` and `examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java:48` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (11 lines × 4) mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:347— mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:347-357 | mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:389-399 | mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:431-441 | mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:473-483 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala:347` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 4) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:69— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:69-79 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:67-77 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:68-78 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala:54-64 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:69` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8–10 lines × 2) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:363— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:363-372 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/HiveThriftServer2Listener.scala:291-298 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala:363` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (8–10 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala:410— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala:410-419 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:587-594 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala:410` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 5) mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:130— mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:130-138 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:113-121 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:121-129 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:140-149 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:123-132 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:130` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 5) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:53— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:53-61 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:137-145 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:207-215 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:272-280 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:323-331 — all 5 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:53` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 9) mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:133— mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:133-140 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:144-151 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:127-134 | mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:151-158 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:135-142 | mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala:126-133 | mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala:136-143 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:155-162 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:138-145 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 39 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (8 lines × 9) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java:68— sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetCatalogsOperation.java:68-75 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java:243-250 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetCrossReferenceOperation.java:179-186 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetFunctionsOperation.java:140-147 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetPrimaryKeysOperation.java:124-131 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetSchemasOperation.java:89-96 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetTableTypesOperation.java:85-92 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetTablesOperation.java:136-143 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetTypeInfoOperation.java:135-142 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 9 call sites, so a change lands once. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (8 lines × 5) sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18204— sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18204-18211 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18233-18240 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18264-18271 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18297-18304 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18333-18340 — all 5 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18204` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 5) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:466— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:466-473 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:481-488 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:527-534 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:587-594 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:651-658 — all 5 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:466` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8 lines × 4) mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:145— mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:145-152 | mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala:148-155 | mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala:115-122 | mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala:143-150 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 40 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (8 lines × 4) mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala:354— mllib/src/main/scala/org/apache/spark/ml/classification/FMClassifier.scala:354-361 | mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:479-486 | mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:1396-1403 | mllib/src/main/scala/org/apache/spark/ml/classification/MultilayerPerceptronClassifier.scala:376-383 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala` and `mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 65 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (7 lines × 7) mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:144— mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:144-150 | mllib/src/main/scala/org/apache/spark/ml/r/FMClassifierWrapper.scala:166-172 | mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala:147-153 | mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala:114-120 | mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala:142-148 | mllib/src/main/scala/org/apache/spark/ml/r/LinearSVCWrapper.scala:152-158 | mllib/src/main/scala/org/apache/spark/ml/r/LogisticRegressionWrapper.scala:207-213 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 40 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (7 lines × 7) python/pyspark/pandas/groupby.py:4561— python/pyspark/pandas/groupby.py:4561-4567 | python/pyspark/pandas/window.py:223-229 | python/pyspark/pandas/window.py:1037-1043 | python/pyspark/pandas/window.py:1636-1642 | python/pyspark/pandas/window.py:2261-2267 | python/pyspark/pandas/window.py:2849-2855 | python/pyspark/pandas/window.py:2950-2956 — there are 7 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 7 sites; resolving a subset leaves the remainder to drift apart. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/pandas/window.py:1645` calls `__repr__`, `format` and `python/pyspark/pandas/window.py:2269` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (7 lines × 5) core/src/main/scala/org/apache/spark/deploy/master/ui/ApplicationPage.scala:115— core/src/main/scala/org/apache/spark/deploy/master/ui/ApplicationPage.scala:115-121 | core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:204-210 | core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:241-248 | core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:285-292 | core/src/main/scala/org/apache/spark/deploy/worker/ui/WorkerPage.scala:102-108 — there are 5 copies across 3 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 5 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (7 lines × 5) mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:251— mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala:251-257 | mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala:295-301 | mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala:295-301 | mllib/src/main/scala/org/apache/spark/ml/regression/GBTRegressor.scala:263-269 | mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:227-233 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/DecisionTreeClassifier.scala` and `mllib/src/main/scala/org/apache/spark/ml/classification/RandomForestClassifier.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 30 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/ml/regression/RandomForestRegressor.scala:223` calls `Since` and `mllib/src/main/scala/org/apache/spark/ml/classification/GBTClassifier.scala:292` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (6–7 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:123— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:123-128 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:148-154 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:148` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (6–7 lines × 2) examples/src/main/scala/org/apache/spark/examples/mllib/GaussianMixtureExample.scala:31— examples/src/main/scala/org/apache/spark/examples/mllib/GaussianMixtureExample.scala:31-36 | examples/src/main/scala/org/apache/spark/examples/mllib/LatentDirichletAllocationExample.scala:31-37 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (12 lines × 7) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:512— sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:512-526 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:576-587 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:598-609 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:643-654 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:664-675 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:907-918 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:930-942 — all 7 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java:512` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12 lines × 7) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:108— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:108-119 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:111-122 | examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala:77-88 | examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala:85-96 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:117-128 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala:123-134 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala:71-82 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:108` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (56 lines × 2) python/pyspark/ml/linalg/__init__.py:662— python/pyspark/ml/linalg/__init__.py:662-717 | python/pyspark/mllib/linalg/__init__.py:769-824 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note first that the copies are not typed on the same thing: the declarations holding them bind `other` to `Iterable[float]` in one and `"VectorLike"` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (56 lines × 2) python/pyspark/pandas/groupby.py:2422— python/pyspark/pandas/groupby.py:2422-2477 | python/pyspark/pandas/groupby.py:2519-2574 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (38 lines × 2) python/pyspark/pandas/groupby.py:4716— python/pyspark/pandas/groupby.py:4716-4753 | python/pyspark/pandas/groupby.py:4801-4838 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/groupby.py:4716` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (38 lines × 2) python/pyspark/sql/classic/dataframe.py:1234— python/pyspark/sql/classic/dataframe.py:1234-1271 | python/pyspark/sql/connect/dataframe.py:1015-1052 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own.
Duplicated block (24 lines × 4) python/pyspark/sql/connect/functions/builtin.py:4586— python/pyspark/sql/connect/functions/builtin.py:4586-4609 | python/pyspark/sql/connect/functions/builtin.py:4644-4667 | python/pyspark/sql/functions/builtin.py:29253-29276 | python/pyspark/sql/functions/builtin.py:29416-29439 — there are 4 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 4 sites; resolving a subset leaves the remainder to drift apart. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (24 lines × 4) examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:71— examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:71-94 | examples/src/main/java/org/apache/spark/examples/ml/JavaFMClassifierExample.java:71-94 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java:73-96 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestClassifierExample.java:69-92 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 79 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (21–24 lines × 2) python/pyspark/eval_handlers/_stateful.py:519— python/pyspark/eval_handlers/_stateful.py:519-539 | python/pyspark/eval_handlers/_stateful.py:728-751 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/eval_handlers/_stateful.py:728` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (21–24 lines × 2) python/pyspark/pandas/frame.py:9053— python/pyspark/pandas/frame.py:9053-9073 | REDACTED:3524-3547 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:3524` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15–17 lines × 2) python/pyspark/pandas/frame.py:9140— python/pyspark/pandas/frame.py:9140-9156 | REDACTED:3587-3601 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (15–17 lines × 2) examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCountWindowed.java:71— examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCountWindowed.java:71-85 | examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredSessionization.java:49-65 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/sql/streaming/JavaStructuredNetworkWordCountWindowed.java:71` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8–11 lines × 2) python/pyspark/mllib/linalg/distributed.py:125— python/pyspark/mllib/linalg/distributed.py:125-132 | python/pyspark/mllib/linalg/distributed.py:632-642 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/mllib/linalg/distributed.py:125` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8–11 lines × 2) python/pyspark/pandas/frame.py:13108— python/pyspark/pandas/frame.py:13108-13118 | python/pyspark/pandas/frame.py:13634-13641 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (23–25 lines × 2) examples/src/main/scala/org/apache/spark/examples/ml/BucketedRandomProjectionLSHExample.scala:67— examples/src/main/scala/org/apache/spark/examples/ml/BucketedRandomProjectionLSHExample.scala:67-89 | examples/src/main/scala/org/apache/spark/examples/ml/MinHashLSHExample.scala:64-88 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/BucketedRandomProjectionLSHExample.scala:67` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (23–25 lines × 2) examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java:87— examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java:87-109 | examples/src/main/java/org/apache/spark/examples/ml/JavaMinHashLSHExample.java:86-110 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java:87` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (22 lines × 3 locations) core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:108— core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:108 · core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:209 · core/src/main/resources/org/apache/spark/ui/static/timeline-view.js:289 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (22 lines × 3 locations) core/src/main/resources/org/apache/spark/ui/static/utils.js:205— core/src/main/resources/org/apache/spark/ui/static/utils.js:205 · core/src/main/resources/org/apache/spark/ui/static/utils.js:228 · sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/sql-table-utils.js:153 — the 2 copies are spread across 2 directories, so the shared home is a decision rather than an obvious spot: check first whether one of them already owns this behaviour, and otherwise put the extracted module somewhere all of the sites already reach rather than making one of them depend on another.
X18 · Disposal-pattern correctness· Disposable created and abandoned in the function that made it · ×2
Disposable created and abandoned in the function that made it docs/_plugins/include_example.rb:49— `File.open` is called here without a block, so it hands back an open descriptor the caller must close — and its first and only use reads `.read` through it and drops it. Nothing closes it: it stays open until the garbage collector happens to finalize it, which is one leaked descriptor per call (a child process too, for `IO.popen`), `Errno::EMFILE` under load, and for a written file data that is not flushed when the program next reads it. Use the block form, which closes on every path including a raise (`File.open(…) { |io| … }`), or the one-call reader/writer (`File.read`, `File.write`, `File.foreach`).
Disposable created and abandoned in the function that made it streaming/src/main/scala/org/apache/spark/streaming/util/RawTextSender.scala:63— `new ServerSocket(…)` opens something that has to be given back — a file descriptor, a socket, a directory handle — and it is held in `serverSocket`, and every later use of `serverSocket` in its block reads a member through it — it is never returned, passed, stored, captured by a lambda, reassigned or released, so this expression both creates the value and is the last place that can release it, and does not. Every call leaks one more open handle until the collector happens to reach it: `Too many open files` under load, and on Windows a file that cannot be replaced or deleted while it is held. Open it with `scala.util.Using.resource(new ServerSocket(…)) { r => … }`, which releases it on every path including a throw, or hold it and call `.close()` in a `finally`.
AC3 · Page structure· Page without a main landmark · ×1
Page without a main landmark licenses-binary/LICENSE-javassist.html:1— No <main> (or role="main") means no "skip to content" target and a weaker landmark map. Wrap the primary content in <main>.
AC4 · Keyboard semantics· Anchor without href · ×1
Anchor without href core/src/main/resources/org/apache/spark/ui/static/executorspage.js:899— An <a> with no href, role or tabindex isn't focusable or keyboard-activatable. Give it a real href, or use a <button> for an action.
Low contrast colour pair in CSS (2.8:1) docs/css/api-javadocs.css:20— `you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */ .badge` sets color: #ffffff on background-color: #999999 — 2.8:1, below the 4.5:1 WCAG AA minimum for normal text. Darken or lighten one of them.
AC7 · A11y enforcement· Accessibility enforcement below the top rung · ×1
Accessibility enforcement below the top rung — No accessibility enforcement found — no a11y linter (an accessibility check that can read your UI — no component framework was detected and your pages are rendered by server-side templates, which neither the JSX/Vue ESLint plugins nor the HTML-template linters can parse; run axe/pa11y over the rendered pages, or assert the accessibility invariants over that rendered HTML in the test suite you already have) and no axe/pa11y/Lighthouse in tests or CI. Start by running that check over your rendered pages in CI. What was searched, so you can tell an absence from a miss: the 9 markup file(s) this pass actually assessed, the linter configuration checked in beside them, and this repository's test and CI files — matched by name against the accessibility checkers this dimension carries. An audit run outside the repository, a hosted scanner, or a check whose name is not one of those, is not seen here.
CheckAnalysis.checkAnalysis0 (cyclomatic 157) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:392— CheckAnalysis.checkAnalysis0 has cyclomatic complexity 157 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
V2ExpressionBuilder.generateExpression (cyclomatic 132) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala:105— V2ExpressionBuilder.generateExpression has cyclomatic complexity 132 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
(anonymous) (cyclomatic 132) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:343— (anonymous) has cyclomatic complexity 132 (threshold 15). Most of this is not in the body itself: 1 of the 132 points is its own statement and the rest belongs to 55 function items inside it that branch (a function literal at line 678, a function literal at line 404, (anonymous)::(anonymous)::(anonymous)::data, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
SparkSubmit.prepareSubmitEnvironment (cyclomatic 131) core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:243— SparkSubmit.prepareSubmitEnvironment has cyclomatic complexity 131 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataSourceV2Strategy.apply (cyclomatic 126) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:130— DataSourceV2Strategy.apply has cyclomatic complexity 126 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ResolveSessionCatalog.apply (cyclomatic 124) REDACTED:56— ResolveSessionCatalog.apply has cyclomatic complexity 124 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
UnsupportedOperationChecker.checkForStreaming (cyclomatic 110) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:191— UnsupportedOperationChecker.checkForStreaming has cyclomatic complexity 110 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
V2ExpressionSQLBuilder.build (cyclomatic 98) sql/catalyst/src/main/java/org/apache/spark/sql/connector/util/V2ExpressionSQLBuilder.java:101— V2ExpressionSQLBuilder.build has cyclomatic complexity 98 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
DAGScheduler.handleTaskCompletion (cyclomatic 79) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:3891— DAGScheduler.handleTaskCompletion has cyclomatic complexity 79 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
HiveInspectors.unwrapperFor (cyclomatic 76) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:550— HiveInspectors.unwrapperFor has cyclomatic complexity 76 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Cast.canCast (cyclomatic 75) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:249— Cast.canCast has cyclomatic complexity 75 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
BasicOperators.apply (cyclomatic 74) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:1099— BasicOperators.apply has cyclomatic complexity 74 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ParquetVectorUpdaterFactory.getUpdater (cyclomatic 74) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:75— ParquetVectorUpdaterFactory.getUpdater has cyclomatic complexity 74 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
AvroDeserializer.newWriter (cyclomatic 70) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:172— AvroDeserializer.newWriter has cyclomatic complexity 70 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
JacksonParser.makeConverter (cyclomatic 68) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:260— JacksonParser.makeConverter has cyclomatic complexity 68 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
Cast.canAnsiCast (cyclomatic 67) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:92— Cast.canAnsiCast has cyclomatic complexity 67 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
HiveInspectors.wrapperFor (cyclomatic 63) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:272— HiveInspectors.wrapperFor has cyclomatic complexity 63 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ResolveReferences.doApply (cyclomatic 62) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1741— ResolveReferences.doApply has cyclomatic complexity 62 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ArrowSerializer.serializerForDefault (cyclomatic 62) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala:249— ArrowSerializer.serializerForDefault has cyclomatic complexity 62 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SparkSession.createDataFrame (cyclomatic 61) python/pyspark/sql/connect/session.py:508— SparkSession.createDataFrame has cyclomatic complexity 61 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
typehints.as_spark_type (cyclomatic 60) python/pyspark/pandas/typedef/typehints.py:141— typehints.as_spark_type has cyclomatic complexity 60 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
merge_spark_pr.main (cyclomatic 59) REDACTED:2332— merge_spark_pr.main has cyclomatic complexity 59 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ParquetToSparkSchemaConverter.convertPrimitiveField (cyclomatic 58) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:243— ParquetToSparkSchemaConverter.convertPrimitiveField has cyclomatic complexity 58 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SchemaConverters.structFieldFor (cyclomatic 57) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala:74— SchemaConverters.structFieldFor has cyclomatic complexity 57 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
EvaluatePython.makeFromJavaDefault (cyclomatic 57) sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala:192— EvaluatePython.makeFromJavaDefault has cyclomatic complexity 57 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
LocIndexerLike.__setitem__ (cyclomatic 56) python/pyspark/pandas/indexing.py:593— LocIndexerLike.__setitem__ has cyclomatic complexity 56 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
CatalystTranspiler._convert_chunk (cyclomatic 56) python/pyspark/sql/transpile.py:562— CatalystTranspiler._convert_chunk has cyclomatic complexity 56 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkIntervalUtils.stringToInterval (cyclomatic 55) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkIntervalUtils.scala:134— SparkIntervalUtils.stringToInterval has cyclomatic complexity 55 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
PlanMerger.tryMergePlans (cyclomatic 55) sql/core/src/main/scala/org/apache/spark/sql/execution/planmerging/PlanMerger.scala:404— PlanMerger.tryMergePlans has cyclomatic complexity 55 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
BinaryArithmeticWithDatetimeResolver.resolve (cyclomatic 53) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/BinaryArithmeticWithDatetimeResolver.scala:66— BinaryArithmeticWithDatetimeResolver.resolve has cyclomatic complexity 53 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
DataTypeAstBuilder.visitPrimitiveDataType (cyclomatic 52) sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:292— DataTypeAstBuilder.visitPrimitiveDataType has cyclomatic complexity 52 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DecorrelateInnerQuery.apply (cyclomatic 51) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/DecorrelateInnerQuery.scala:464— DecorrelateInnerQuery.apply has cyclomatic complexity 51 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Shim_v2_0.convertFilters (cyclomatic 51) sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:668— Shim_v2_0.convertFilters has cyclomatic complexity 51 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ScalaReflection.encoderFor (cyclomatic 50) sql/api/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala:258— ScalaReflection.encoderFor has cyclomatic complexity 50 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
PropagateEmptyRelationBase.commonApplyFunc (cyclomatic 50) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala:70— PropagateEmptyRelationBase.commonApplyFunc has cyclomatic complexity 50 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
ArrowDeserializers.deserializerForDefault (cyclomatic 50) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowDeserializer.scala:101— ArrowDeserializers.deserializerForDefault has cyclomatic complexity 50 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ParquetFilters.createFilterHelper (cyclomatic 49) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:1093— ParquetFilters.createFilterHelper has cyclomatic complexity 49 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
namespace.concat (cyclomatic 49) REDACTED:2380— namespace.concat has cyclomatic complexity 49 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkDateTimeUtils.parseTimestampString (cyclomatic 48) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:709— SparkDateTimeUtils.parseTimestampString has cyclomatic complexity 48 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
VariantShreddingWriter.tryTypedShred (cyclomatic 48) common/variant/src/main/java/org/apache/spark/types/variant/VariantShreddingWriter.java:155— VariantShreddingWriter.tryTypedShred has cyclomatic complexity 48 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
types.proto_schema_to_pyspark_data_type (cyclomatic 48) python/pyspark/sql/connect/types.py:226— types.proto_schema_to_pyspark_data_type has cyclomatic complexity 48 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SerializerBuildHelper.createSerializerDefault (cyclomatic 46) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SerializerBuildHelper.scala:346— SerializerBuildHelper.createSerializerDefault has cyclomatic complexity 46 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
namespace.read_csv (cyclomatic 46) REDACTED:263— namespace.read_csv has cyclomatic complexity 46 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
(anonymous) (cyclomatic 46) core/src/main/resources/org/apache/spark/ui/static/executorspage.js:348— (anonymous) has cyclomatic complexity 46 (threshold 15). Most of this is not in the body itself: 1 of the 46 points is its own statement and the rest belongs to 33 function items inside it that branch (a function literal at line 436, a function literal at line 575, (anonymous)::(anonymous)::(anonymous)::data, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
ProtobufSerializer.newConverter (cyclomatic 45) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufSerializer.scala:81— ProtobufSerializer.newConverter has cyclomatic complexity 45 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
AvroSerializer.newConverter (cyclomatic 45) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala:102— AvroSerializer.newConverter has cyclomatic complexity 45 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SchemaConverters.toSqlTypeHelper (cyclomatic 45) sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:149— SchemaConverters.toSqlTypeHelper has cyclomatic complexity 45 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
InProcessArrowEvalPythonEvaluatorFactory.evaluateBatches (cyclomatic 45) sql/core/src/main/scala/org/apache/spark/sql/execution/python/InProcessArrowEvalPythonEvaluatorFactory.scala:115— InProcessArrowEvalPythonEvaluatorFactory.evaluateBatches has cyclomatic complexity 45 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
Master.receive (cyclomatic 44) core/src/main/scala/org/apache/spark/deploy/master/Master.scala:234— Master.receive has cyclomatic complexity 44 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
JdbcUtils.getCatalystType (cyclomatic 44) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:211— JdbcUtils.getCatalystType has cyclomatic complexity 44 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ParquetRowConverter.newConverterDefault (cyclomatic 44) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala:320— ParquetRowConverter.newConverterDefault has cyclomatic complexity 44 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
FunctionArgumentTypeCoercion.apply (cyclomatic 43) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:505— FunctionArgumentTypeCoercion.apply has cyclomatic complexity 43 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ColumnNodeToExpressionConverter.apply (cyclomatic 43) sql/core/src/main/scala/org/apache/spark/sql/classic/columnNodeSupport.scala:47— ColumnNodeToExpressionConverter.apply has cyclomatic complexity 43 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
MLExpressionUtils.scaleShift (cyclomatic 43) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ml/MLExpressionUtils.java:194— MLExpressionUtils.scaleShift has cyclomatic complexity 43 (threshold 15). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
Resampler._bin_timestamp (cyclomatic 43) python/pyspark/pandas/resample.py:138— Resampler._bin_timestamp has cyclomatic complexity 43 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StateSourceOptions.apply (cyclomatic 42) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala:546— StateSourceOptions.apply has cyclomatic complexity 42 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
types._parse_datatype_json_value (cyclomatic 42) python/pyspark/sql/types.py:2673— types._parse_datatype_json_value has cyclomatic complexity 42 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkSubmitArguments.handle (cyclomatic 41) core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala:377— SparkSubmitArguments.handle has cyclomatic complexity 41 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
UnwrapCastInBinaryComparison.simplifyNumericComparison (cyclomatic 40) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala:201— UnwrapCastInBinaryComparison.simplifyNumericComparison has cyclomatic complexity 40 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
InternalFrame.__init__ (cyclomatic 40) python/pyspark/pandas/internal.py:543— InternalFrame.__init__ has cyclomatic complexity 40 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
namespace.merge_asof (cyclomatic 40) REDACTED:3110— namespace.merge_asof has cyclomatic complexity 40 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkParserUtils.unescapeSQLString (cyclomatic 39) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:41— SparkParserUtils.unescapeSQLString has cyclomatic complexity 39 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LocIndexer._select_rows_by_slice (cyclomatic 39) python/pyspark/pandas/indexing.py:1064— LocIndexer._select_rows_by_slice has cyclomatic complexity 39 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ShuffleBlockFetcherIterator.next (cyclomatic 38) core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala:808— ShuffleBlockFetcherIterator.next has cyclomatic complexity 38 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
MLHandler._handleMlCommand (cyclomatic 38) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:195— MLHandler._handleMlCommand has cyclomatic complexity 38 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LiteralExpression._to_value (cyclomatic 38) python/pyspark/sql/connect/expressions.py:381— LiteralExpression._to_value has cyclomatic complexity 38 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DAGScheduler.submitMissingTasks (cyclomatic 37) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:3215— DAGScheduler.submitMissingTasks has cyclomatic complexity 37 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
WeightedLeastSquares.fit (cyclomatic 37) mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala:99— WeightedLeastSquares.fit has cyclomatic complexity 37 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ValidateSubqueryExpression.checkCorrelationsInSubquery (cyclomatic 37) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ValidateSubqueryExpression.scala:246— ValidateSubqueryExpression.checkCorrelationsInSubquery has cyclomatic complexity 37 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
AstBuilder.withPredicate (cyclomatic 37) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:3861— AstBuilder.withPredicate has cyclomatic complexity 37 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SparkToParquetSchemaConverter.convertFieldDefault (cyclomatic 37) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:673— SparkToParquetSchemaConverter.convertFieldDefault has cyclomatic complexity 37 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
OnHeapColumnVector.reserveInternal (cyclomatic 37) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java:596— OnHeapColumnVector.reserveInternal has cyclomatic complexity 37 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.merge (cyclomatic 37) python/pyspark/pandas/frame.py:8811— DataFrame.merge has cyclomatic complexity 37 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
types.pyspark_types_to_proto_types (cyclomatic 37) python/pyspark/sql/connect/types.py:127— types.pyspark_types_to_proto_types has cyclomatic complexity 37 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
LogisticRegression.train (cyclomatic 36) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:507— LogisticRegression.train has cyclomatic complexity 36 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
InterpretedHashFunction.hash (cyclomatic 36) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:707— InterpretedHashFunction.hash has cyclomatic complexity 36 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
IntervalUtils.castStringToDTInterval (cyclomatic 36) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala:275— IntervalUtils.castStringToDTInterval has cyclomatic complexity 36 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
DataTypeProtoConverter.toConnectProtoTypeDefault (cyclomatic 36) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:193— DataTypeProtoConverter.toConnectProtoTypeDefault has cyclomatic complexity 36 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
MultiColumnInSubqueryEvaluator.eval (cyclomatic 36) sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:337— MultiColumnInSubqueryEvaluator.eval has cyclomatic complexity 36 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.pivot_table (cyclomatic 36) python/pyspark/pandas/frame.py:6941— DataFrame.pivot_table has cyclomatic complexity 36 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LocIndexerLike.__getitem__ (cyclomatic 36) python/pyspark/pandas/indexing.py:439— LocIndexerLike.__getitem__ has cyclomatic complexity 36 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
namespace.get_dummies (cyclomatic 36) REDACTED:2153— namespace.get_dummies has cyclomatic complexity 36 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ProtobufDeserializer.newWriter (cyclomatic 35) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:249— ProtobufDeserializer.newWriter has cyclomatic complexity 35 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
Decimal.changePrecision (cyclomatic 35) sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala:387— Decimal.changePrecision has cyclomatic complexity 35 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ValidateSubqueryExpression.apply (cyclomatic 35) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ValidateSubqueryExpression.scala:37— ValidateSubqueryExpression.apply has cyclomatic complexity 35 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
VariantGet.cast (cyclomatic 35) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:762— VariantGet.cast has cyclomatic complexity 35 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SqlStatementSplitter.splitWithPositions (cyclomatic 35) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:153— SqlStatementSplitter.splitWithPositions has cyclomatic complexity 35 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataTypeUtils.canWrite (cyclomatic 35) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/DataTypeUtils.scala:108— DataTypeUtils.canWrite has cyclomatic complexity 35 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
PredicateUtils.toV1 (cyclomatic 35) sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/PredicateUtils.scala:29— PredicateUtils.toV1 has cyclomatic complexity 35 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
EnsureRequirements.checkKeyGroupCompatible (cyclomatic 35) sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala:615— EnsureRequirements.checkKeyGroupCompatible has cyclomatic complexity 35 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Master.receiveAndReply (cyclomatic 34) core/src/main/scala/org/apache/spark/deploy/master/Master.scala:417— Master.receiveAndReply has cyclomatic complexity 34 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Worker.receive (cyclomatic 34) core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala:534— Worker.receive has cyclomatic complexity 34 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
TypeCoercion.implicitCast (cyclomatic 34) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:202— TypeCoercion.implicitCast has cyclomatic complexity 34 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
InterpretedUnsafeProjection.generateFieldWriter (cyclomatic 34) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala:134— InterpretedUnsafeProjection.generateFieldWriter has cyclomatic complexity 34 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
FilterEstimation.evaluateBinaryForTwoColumns (cyclomatic 34) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:700— FilterEstimation.evaluateBinaryForTwoColumns has cyclomatic complexity 34 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ColumnNodeToProtoConverter.apply (cyclomatic 34) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/columnNodeSupport.scala:61— ColumnNodeToProtoConverter.apply has cyclomatic complexity 34 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ArrowColumnVector.initAccessor (cyclomatic 34) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java:192— ArrowColumnVector.initAccessor has cyclomatic complexity 34 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataFrame.melt (cyclomatic 34) python/pyspark/pandas/frame.py:10852— DataFrame.melt has cyclomatic complexity 34 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ToStringBase.castToStringDefault (cyclomatic 33) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala:93— ToStringBase.castToStringDefault has cyclomatic complexity 33 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
OrcSerializer.newConverter (cyclomatic 33) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcSerializer.scala:63— OrcSerializer.newConverter has cyclomatic complexity 33 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SortMergeJoinEvaluator.eval (cyclomatic 33) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:58— SortMergeJoinEvaluator.eval has cyclomatic complexity 33 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
AbstractCommandBuilder.buildClassPath (cyclomatic 33) launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java:142— AbstractCommandBuilder.buildClassPath has cyclomatic complexity 33 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ShreddingUtils.rebuild (cyclomatic 33) common/variant/src/main/java/org/apache/spark/types/variant/ShreddingUtils.java:69— ShreddingUtils.rebuild has cyclomatic complexity 33 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.xs (cyclomatic 33) python/pyspark/pandas/frame.py:3588— DataFrame.xs has cyclomatic complexity 33 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
types.from_arrow_type (cyclomatic 33) python/pyspark/sql/pandas/types.py:388— types.from_arrow_type has cyclomatic complexity 33 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
pandasutils._assert_pandas_almost_equal (cyclomatic 33) python/pyspark/testing/pandasutils.py:127— pandasutils._assert_pandas_almost_equal has cyclomatic complexity 33 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
TaskSchedulerImpl.resourceOffers (cyclomatic 32) core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:573— TaskSchedulerImpl.resourceOffers has cyclomatic complexity 32 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
UnivocityParser.makeConverter (cyclomatic 32) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala:189— UnivocityParser.makeConverter has cyclomatic complexity 32 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
Literal.sql (cyclomatic 32) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala:652— Literal.sql has cyclomatic complexity 32 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: Literal.componentTypeToDataType (cyclomatic 34) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
SimpleMetricsCachedBatchSerializer.buildFilter (cyclomatic 32) sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala:252— SimpleMetricsCachedBatchSerializer.buildFilter has cyclomatic complexity 32 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
MicroBatchExecution.runBatch (cyclomatic 32) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala:1192— MicroBatchExecution.runBatch has cyclomatic complexity 32 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
Series.replace (cyclomatic 32) python/pyspark/pandas/series.py:5010— Series.replace has cyclomatic complexity 32 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
UserDefinedFunction._check_return_type (cyclomatic 32) python/pyspark/sql/udf.py:334— UserDefinedFunction._check_return_type has cyclomatic complexity 32 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SerDe.writeObject (cyclomatic 31) core/src/main/scala/org/apache/spark/api/r/SerDe.scala:290— SerDe.writeObject has cyclomatic complexity 31 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SparkIntervalUtils.toDayTimeIntervalString (cyclomatic 31) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkIntervalUtils.scala:435— SparkIntervalUtils.toDayTimeIntervalString has cyclomatic complexity 31 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkContext._do_init (cyclomatic 31) python/pyspark/core/context.py:225— SparkContext._do_init has cyclomatic complexity 31 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
GroupBy.apply (cyclomatic 31) python/pyspark/pandas/groupby.py:1818— GroupBy.apply has cyclomatic complexity 31 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
types._infer_type (cyclomatic 31) python/pyspark/sql/types.py:2993— types._infer_type has cyclomatic complexity 31 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
TaskRunner.run (cyclomatic 30) core/src/main/scala/org/apache/spark/executor/Executor.scala:815— TaskRunner.run has cyclomatic complexity 30 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataType.nameToType (cyclomatic 30) sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:271— DataType.nameToType has cyclomatic complexity 30 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
V2ExpressionUtils.convertMathFunc (cyclomatic 30) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/V2ExpressionUtils.scala:321— V2ExpressionUtils.convertMathFunc has cyclomatic complexity 30 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
CodegenContext.collectInputs (cyclomatic 30) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:1318— CodegenContext.collectInputs has cyclomatic complexity 30 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
UDFRegistration.registerJava (cyclomatic 30) sql/core/src/main/scala/org/apache/spark/sql/classic/UDFRegistration.scala:141— UDFRegistration.registerJava has cyclomatic complexity 30 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SparkShreddingUtils.buildVariantSchema (cyclomatic 30) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala:559— SparkShreddingUtils.buildVariantSchema has cyclomatic complexity 30 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
StateStoreCoordinator.receiveAndReply (cyclomatic 30) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala:317— StateStoreCoordinator.receiveAndReply has cyclomatic complexity 30 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
HiveInspectors.toInspector (cyclomatic 30) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:1085— HiveInspectors.toInspector has cyclomatic complexity 30 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: HiveInspectors.javaTypeToDataType (cyclomatic 41) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
SparkSubmitCommandBuilder.buildSparkSubmitArgs (cyclomatic 30) launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java:198— SparkSubmitCommandBuilder.buildSparkSubmitArgs has cyclomatic complexity 30 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
types.to_arrow_type (cyclomatic 30) python/pyspark/sql/pandas/types.py:118— types.to_arrow_type has cyclomatic complexity 30 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
AppStatusListener.onTaskEnd (cyclomatic 29) core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:669— AppStatusListener.onTaskEnd has cyclomatic complexity 29 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
Literal.validateLiteralValue (cyclomatic 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala:253— Literal.validateLiteralValue has cyclomatic complexity 29 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: Literal.componentTypeToDataType (cyclomatic 34) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
JsonInferSchema.inferField (cyclomatic 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:155— JsonInferSchema.inferField has cyclomatic complexity 29 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SimplifyConditionals.apply (cyclomatic 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:710— SimplifyConditionals.apply has cyclomatic complexity 29 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
TreeNode.parseToJson (cyclomatic 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:1319— TreeNode.parseToJson has cyclomatic complexity 29 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
StringUtils.splitSemiColon (cyclomatic 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala:286— StringUtils.splitSemiColon has cyclomatic complexity 29 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
JoinSelection.apply (cyclomatic 29) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:235— JoinSelection.apply has cyclomatic complexity 29 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkSQLCLIDriver.main (cyclomatic 29) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala:85— SparkSQLCLIDriver.main has cyclomatic complexity 29 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
makeSortable (cyclomatic 29) REDACTED:42— makeSortable has cyclomatic complexity 29 (threshold 15). Most of this is not in the body itself: 14 of the 29 points are its own statements and the rest belongs to 2 function items inside it that branch (innerSortFunction, a function literal at line 133). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
ResolveSQLFunctions.rewrite (cyclomatic 28) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2979— ResolveSQLFunctions.rewrite has cyclomatic complexity 28 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
UnsafeRowUtils.validateStructuralIntegrityWithReasonImpl (cyclomatic 28) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala:80— UnsafeRowUtils.validateStructuralIntegrityWithReasonImpl has cyclomatic complexity 28 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
BufferSetterGetterUtils.createGetters (cyclomatic 28) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala:40— BufferSetterGetterUtils.createGetters has cyclomatic complexity 28 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ShuffleExchangeExec.prepareShuffleDependency (cyclomatic 28) sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala:359— ShuffleExchangeExec.prepareShuffleDependency has cyclomatic complexity 28 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
WritableColumnVector.appendObjects (cyclomatic 28) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java:806— WritableColumnVector.appendObjects has cyclomatic complexity 28 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
functions._validate_and_transform_prediction_result (cyclomatic 28) python/pyspark/ml/functions.py:283— functions._validate_and_transform_prediction_result has cyclomatic complexity 28 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.dropna (cyclomatic 28) python/pyspark/pandas/frame.py:6098— DataFrame.dropna has cyclomatic complexity 28 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
PandasOnSparkTestUtils.assert_eq (cyclomatic 28) python/pyspark/testing/pandasutils.py:384— PandasOnSparkTestUtils.assert_eq has cyclomatic complexity 28 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
FunctionResolution.validateFunction (cyclomatic 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala:629— FunctionResolution.validateFunction has cyclomatic complexity 27 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ResolveLateralColumnAliasReference.apply0 (cyclomatic 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveLateralColumnAliasReference.scala:139— ResolveLateralColumnAliasReference.apply0 has cyclomatic complexity 27 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
CatalogTable.toJsonLinkedHashMap (cyclomatic 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:642— CatalogTable.toJsonLinkedHashMap has cyclomatic complexity 27 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
UnivocityGenerator.makeConverter (cyclomatic 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityGenerator.scala:79— UnivocityGenerator.makeConverter has cyclomatic complexity 27 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
VariantValueConverter.convertInput (cyclomatic 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala:480— VariantValueConverter.convertInput has cyclomatic complexity 27 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ColumnPruning.apply (cyclomatic 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:1502— ColumnPruning.apply has cyclomatic complexity 27 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
BufferSetterGetterUtils.createSetters (cyclomatic 27) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala:111— BufferSetterGetterUtils.createSetters has cyclomatic complexity 27 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
XSDToSchema.getStructField (cyclomatic 27) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:70— XSDToSchema.getStructField has cyclomatic complexity 27 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
daemon.manager (cyclomatic 27) python/pyspark/daemon.py:115— daemon.manager has cyclomatic complexity 27 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
(anonymous) (cyclomatic 27) sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:22— (anonymous) has cyclomatic complexity 27 (threshold 15). Most of this is not in the body itself: 2 of the 27 points are its own statements and the rest belongs to 12 function items inside it that branch (init::renderSummary, init::render, init::renderSubExecutionsHtml, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
KafkaSourceProvider.validateGeneralOptions (cyclomatic 26) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala:228— KafkaSourceProvider.validateGeneralOptions has cyclomatic complexity 26 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LinearRegression.train (cyclomatic 26) mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:329— LinearRegression.train has cyclomatic complexity 26 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ResolveRelations.apply (cyclomatic 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1255— ResolveRelations.apply has cyclomatic complexity 26 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
BindParameters.apply (cyclomatic 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/parameters.scala:202— BindParameters.apply has cyclomatic complexity 26 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
JsonTableEvaluator.anyMatch (cyclomatic 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:880— JsonTableEvaluator.anyMatch has cyclomatic complexity 26 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
GetJsonObjectEvaluator.evaluatePath (cyclomatic 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:1435— GetJsonObjectEvaluator.evaluatePath has cyclomatic complexity 26 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
LiteralValueProtoConverter.getProtoDataTypeDefault (cyclomatic 26) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/LiteralValueProtoConverter.scala:544— LiteralValueProtoConverter.getProtoDataTypeDefault has cyclomatic complexity 26 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: LiteralValueProtoConverter.toDataType (cyclomatic 34) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
SparkConnectAnalyzeHandler.process (cyclomatic 26) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectAnalyzeHandler.scala:60— SparkConnectAnalyzeHandler.process has cyclomatic complexity 26 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
CreateSQLFunctionCommand.run (cyclomatic 26) sql/core/src/main/scala/org/apache/spark/sql/execution/command/CreateSQLFunctionCommand.scala:71— CreateSQLFunctionCommand.run has cyclomatic complexity 26 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
FileScanRDD.compute (cyclomatic 26) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala:105— FileScanRDD.compute has cyclomatic complexity 26 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
OrcDeserializer.newWriter (cyclomatic 26) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcDeserializer.scala:112— OrcDeserializer.newWriter has cyclomatic complexity 26 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
DataTypeOps.__new__ (cyclomatic 26) python/pyspark/pandas/data_type_ops/base.py:265— DataTypeOps.__new__ has cyclomatic complexity 26 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataFrame.drop (cyclomatic 26) python/pyspark/pandas/frame.py:7748— DataFrame.drop has cyclomatic complexity 26 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.describe (cyclomatic 26) python/pyspark/pandas/frame.py:10023— DataFrame.describe has cyclomatic complexity 26 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkConnectClient._analyze (cyclomatic 26) python/pyspark/sql/connect/client/core.py:1592— SparkConnectClient._analyze has cyclomatic complexity 26 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
CatalystTranspiler._category_uncached (cyclomatic 26) python/pyspark/sql/transpile.py:471— CatalystTranspiler._category_uncached has cyclomatic complexity 26 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
NewHadoopRDD.compute (cyclomatic 25) core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:198— NewHadoopRDD.compute has cyclomatic complexity 25 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
ExecutorPodsAllocator.onNewSnapshots (cyclomatic 25) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala:171— ExecutorPodsAllocator.onNewSnapshots has cyclomatic complexity 25 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
Client.prepareLocalResources (cyclomatic 25) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:527— Client.prepareLocalResources has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ResolveFunctions.apply (cyclomatic 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2483— ResolveFunctions.apply has cyclomatic complexity 25 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
MapObjects.doGenCode (cyclomatic 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:1089— MapObjects.doGenCode has cyclomatic complexity 25 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
VariantExpressionEvalUtils.buildVariant (cyclomatic 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:414— VariantExpressionEvalUtils.buildVariant has cyclomatic complexity 25 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
NullPropagation.apply (cyclomatic 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:1066— NullPropagation.apply has cyclomatic complexity 25 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
AstBuilder.withJoinRelation (cyclomatic 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:2600— AstBuilder.withJoinRelation has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StringUtils.stripComment (cyclomatic 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala:164— StringUtils.stripComment has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ToNumberParser.checkInputDataTypes (cyclomatic 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala:258— ToNumberParser.checkInputDataTypes has cyclomatic complexity 25 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
SchemaConverters.toAvroType (cyclomatic 25) sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:402— SchemaConverters.toAvroType has cyclomatic complexity 25 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
JdbcUtils.makeSetter (cyclomatic 25) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:503— JdbcUtils.makeSetter has cyclomatic complexity 25 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: JdbcUtils.makeGetter (cyclomatic 28) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
V2ScanRelationPushDown.rewriteAggregate (cyclomatic 25) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:632— V2ScanRelationPushDown.rewriteAggregate has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkSQLCLIDriver.processCmd (cyclomatic 25) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala:442— SparkSQLCLIDriver.processCmd has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
HiveFunctionRegistryUtils.getMethodInternal (cyclomatic 25) sql/hive/src/main/java/org/apache/hadoop/hive/ql/exec/HiveFunctionRegistryUtils.java:85— HiveFunctionRegistryUtils.getMethodInternal has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
OperationState.validateTransition (cyclomatic 25) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/OperationState.java:50— OperationState.validateTransition has cyclomatic complexity 25 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ColumnVectorUtils.populate (cyclomatic 25) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java:71— ColumnVectorUtils.populate has cyclomatic complexity 25 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ColumnarBatchRow.get (cyclomatic 25) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:187— ColumnarBatchRow.get has cyclomatic complexity 25 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
TaskMemoryManager.allocatePageInternal (cyclomatic 25) core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:660— TaskMemoryManager.allocatePageInternal has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
merge_spark_pr.resolve_jira_issue (cyclomatic 25) REDACTED:1484— merge_spark_pr.resolve_jira_issue has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.reset_index (cyclomatic 25) python/pyspark/pandas/frame.py:4471— DataFrame.reset_index has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LocIndexer._select_cols_by_iterable (cyclomatic 25) python/pyspark/pandas/indexing.py:1346— LocIndexer._select_cols_by_iterable has cyclomatic complexity 25 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkSubmit.runMain (cyclomatic 24) core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:964— SparkSubmit.runMain has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkSubmitArguments.validateSubmitArguments (cyclomatic 24) core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala:278— SparkSubmitArguments.validateSubmitArguments has cyclomatic complexity 24 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
RowMatrix.computeSVD (cyclomatic 24) mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:341— RowMatrix.computeSVD has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ExtractGenerator.apply (cyclomatic 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:3452— ExtractGenerator.apply has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ColumnResolutionHelper.resolveExpression (cyclomatic 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:112— ColumnResolutionHelper.resolveExpression has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ResolveWithCTE.resolveWithCTE (cyclomatic 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:47— ResolveWithCTE.resolveWithCTE has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SelectedField.selectField (cyclomatic 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SelectedField.scala:83— SelectedField.selectField has cyclomatic complexity 24 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
JacksonParser.parseIterator (cyclomatic 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:908— JacksonParser.parseIterator has cyclomatic complexity 24 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
InjectRuntimeFilter.extractSelectiveFilterOverScan (cyclomatic 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:103— InjectRuntimeFilter.extractSelectiveFilterOverScan has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ParseSqlResult.collectPlanReferences (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala:328— ParseSqlResult.collectPlanReferences has cyclomatic complexity 24 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
HiveResult.toHiveStringDefault (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala:122— HiveResult.toHiveStringDefault has cyclomatic complexity 24 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
DataSource.lookupDataSource (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala:687— DataSource.lookupDataSource has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
InsertIntoHadoopFsRelationCommand.run (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala:83— InsertIntoHadoopFsRelationCommand.run has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
JdbcUtils.savePartition (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:653— JdbcUtils.savePartition has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: JdbcUtils.makeGetter (cyclomatic 28) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
OrcUtils.createAggInternalRowFromFooter (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:613— OrcUtils.createAggInternalRowFromFooter has cyclomatic complexity 24 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ParquetFilters.valueMatchesParquetType (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:958— ParquetFilters.valueMatchesParquetType has cyclomatic complexity 24 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
TransformWithStateInPySparkStateServer.handleStatefulProcessorCall (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:351— TransformWithStateInPySparkStateServer.handleStatefulProcessorCall has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StateStore.submitMaintenanceWorkForProvider (cyclomatic 24) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala:2037— StateStore.submitMaintenanceWorkForProvider has cyclomatic complexity 24 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
MutableColumnarRow.copy (cyclomatic 24) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnarRow.java:57— MutableColumnarRow.copy has cyclomatic complexity 24 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SpecializedGettersReader.read (cyclomatic 24) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/SpecializedGettersReader.java:29— SpecializedGettersReader.read has cyclomatic complexity 24 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ColumnarRow.get (cyclomatic 24) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:192— ColumnarRow.get has cyclomatic complexity 24 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
api.create_auto_cdc_flow (cyclomatic 24) python/pyspark/pipelines/api.py:531— api.create_auto_cdc_flow has cyclomatic complexity 24 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
typehints.infer_pandas_eval_type (cyclomatic 24) python/pyspark/sql/pandas/typehints.py:42— typehints.infer_pandas_eval_type has cyclomatic complexity 24 (threshold 15). To reduce it, name the conditions: bind each compound test to a well-named local or a small predicate function, so the body reads as a sequence of named decisions rather than a chain of operators.
spark-sql-viz.document.addEventListener("DOMContentLoaded") (cyclomatic 24) REDACTED:1093— spark-sql-viz.document.addEventListener("DOMContentLoaded") has cyclomatic complexity 24 (threshold 15). Most of this is not in the body itself: 7 of the 24 points are its own statements and the rest belongs to 4 function literals inside it that branch (lines 1118, 1139, 1142, …). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
IndylambdaScalaClosures.findAccessedFields (cyclomatic 23) common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:916— IndylambdaScalaClosures.findAccessedFields has cyclomatic complexity 23 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Writer.open (cyclomatic 23) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:920— Writer.open has cyclomatic complexity 23 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: PythonEvalType.toString (cyclomatic 39) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
JobPage.render (cyclomatic 23) core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:222— JobPage.render has cyclomatic complexity 23 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SummarizerBuffer.merge (cyclomatic 23) mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:590— SummarizerBuffer.merge has cyclomatic complexity 23 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ResolveReferences.expandStarExpression (cyclomatic 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2205— ResolveReferences.expandStarExpression has cyclomatic complexity 23 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
DeduplicateRelations.dedupRight (cyclomatic 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala:366— DeduplicateRelations.dedupRight has cyclomatic complexity 23 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ResolveCatalogs.apply (cyclomatic 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala:44— ResolveCatalogs.apply has cyclomatic complexity 23 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
UnwrapCastInBinaryComparison.unwrapCast (cyclomatic 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala:112— UnwrapCastInBinaryComparison.unwrapCast has cyclomatic complexity 23 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
RewritePredicateSubquery.apply (cyclomatic 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:144— RewritePredicateSubquery.apply has cyclomatic complexity 23 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ToNumberParser.parse (cyclomatic 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala:401— ToNumberParser.parse has cyclomatic complexity 23 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
StaxXmlParser.convertTo (cyclomatic 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:791— StaxXmlParser.convertTo has cyclomatic complexity 23 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SortMergeJoinExec.doProduce (cyclomatic 23) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala:485— SortMergeJoinExec.doProduce has cyclomatic complexity 23 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ColumnVectorUtils.appendValue (cyclomatic 23) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java:178— ColumnVectorUtils.appendValue has cyclomatic complexity 23 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataFrame.filter (cyclomatic 23) python/pyspark/pandas/frame.py:11938— DataFrame.filter has cyclomatic complexity 23 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
GroupBy.aggregate (cyclomatic 23) python/pyspark/pandas/groupby.py:176— GroupBy.aggregate has cyclomatic complexity 23 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
iLocIndexer._select_rows_by_slice (cyclomatic 23) python/pyspark/pandas/indexing.py:1645— iLocIndexer._select_rows_by_slice has cyclomatic complexity 23 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Builder.getOrCreate (cyclomatic 23) python/pyspark/sql/session.py:435— Builder.getOrCreate has cyclomatic complexity 23 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
spark-sql-viz.updateDetailsPanel (cyclomatic 23) REDACTED:423— spark-sql-viz.updateDetailsPanel has cyclomatic complexity 23 (threshold 15). Of this number, 19 points are the body's own statements and 4 belong to 2 function literals inside it that branch. To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkContext.addJar (cyclomatic 22) core/src/main/scala/org/apache/spark/SparkContext.scala:2478— SparkContext.addJar has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
HadoopRDD.compute (cyclomatic 22) core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:269— HadoopRDD.compute has cyclomatic complexity 22 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
TaskSetManager.handleFailedTask (cyclomatic 22) core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:992— TaskSetManager.handleFailedTask has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SummarizerBuffer.add (cyclomatic 22) mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:499— SummarizerBuffer.add has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Client.monitorApplication (cyclomatic 22) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:1210— Client.monitorApplication has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RelationResolution.tryResolvePersistent (cyclomatic 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala:241— RelationResolution.tryResolvePersistent has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
TypeCoercion.findCommonTypeForBinaryComparison (cyclomatic 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:138— TypeCoercion.findCommonTypeForBinaryComparison has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Cast.castToByte (cyclomatic 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1297— Cast.castToByte has cyclomatic complexity 22 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: Cast.castInternal (cyclomatic 32) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
ProjectionOverSchema.getProjection (cyclomatic 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ProjectionOverSchema.scala:38— ProjectionOverSchema.getProjection has cyclomatic complexity 22 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SchemaOfVariant.schemaOf (cyclomatic 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:2123— SchemaOfVariant.schemaOf has cyclomatic complexity 22 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
StaxXmlParser.convertObject (cyclomatic 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:622— StaxXmlParser.convertObject has cyclomatic complexity 22 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SparkResult.processResponses (cyclomatic 22) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:119— SparkResult.processResponses has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SchemaConverters.toAvroTypeWithDefaults (cyclomatic 22) sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:509— SchemaConverters.toAvroTypeWithDefaults has cyclomatic complexity 22 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
BinaryFileFormat.createFilterFunction (cyclomatic 22) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormat.scala:207— BinaryFileFormat.createFilterFunction has cyclomatic complexity 22 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
DataflowGraphTransformer.transformDownNodes (cyclomatic 22) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DataflowGraphTransformer.scala:121— DataflowGraphTransformer.transformDownNodes has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SortState.mergeHi (cyclomatic 22) core/src/main/java/org/apache/spark/util/collection/TimSort.java:806— SortState.mergeHi has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This shape REPEATS in the file: one other method here (SortState.mergeLo) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
SortState.mergeLo (cyclomatic 22) core/src/main/java/org/apache/spark/util/collection/TimSort.java:686— SortState.mergeLo has cyclomatic complexity 22 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This shape REPEATS in the file: one other method here (SortState.mergeHi) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
functions._validate_vectorized_udf (cyclomatic 22) python/pyspark/sql/pandas/functions.py:882— functions._validate_vectorized_udf has cyclomatic complexity 22 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
transpile._transpile_func (cyclomatic 22) python/pyspark/sql/transpile.py:1208— transpile._transpile_func has cyclomatic complexity 22 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
types._merge_type (cyclomatic 22) python/pyspark/sql/types.py:3296— types._merge_type has cyclomatic complexity 22 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
analyze_udtf._main (cyclomatic 22) python/pyspark/sql/worker/analyze_udtf.py:100— analyze_udtf._main has cyclomatic complexity 22 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
sql-table-utils.getSqlTableColumns (cyclomatic 22) sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/sql-table-utils.js:184— sql-table-utils.getSqlTableColumns has cyclomatic complexity 22 (threshold 15). Most of this is not in the body itself: 2 of the 22 points are its own statements and the rest belongs to 9 function literals inside it that branch (lines 263, 202, 254, …). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
SparkContext.addFile (cyclomatic 21) core/src/main/scala/org/apache/spark/SparkContext.scala:1866— SparkContext.addFile has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
GeneralizedLinearAlgorithm.run (cyclomatic 21) mllib/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala:238— GeneralizedLinearAlgorithm.run has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkDateTimeUtils.stringToDate (cyclomatic 21) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:577— SparkDateTimeUtils.stringToDate has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ResolveFieldNameAndPosition.apply (cyclomatic 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4280— ResolveFieldNameAndPosition.apply has cyclomatic complexity 21 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
CheckAnalysis.checkAlterTableCommand (cyclomatic 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:1308— CheckAnalysis.checkAlterTableCommand has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Cast.castToShort (cyclomatic 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1233— Cast.castToShort has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: Cast.castInternal (cyclomatic 32) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
AstBuilder.withSample (cyclomatic 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:2828— AstBuilder.withSample has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
FilterEstimation.evaluateBinaryForNumeric (cyclomatic 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:501— FilterEstimation.evaluateBinaryForNumeric has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkConnectPlanner.transformReadRel (cyclomatic 21) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:1790— SparkConnectPlanner.transformReadRel has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: SparkConnectPlanner.transformRelation (cyclomatic 66) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
HashAggregateExec.doProduceWithKeys (cyclomatic 21) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:560— HashAggregateExec.doProduceWithKeys has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DetectAmbiguousSelfJoin.apply (cyclomatic 21) sql/core/src/main/scala/org/apache/spark/sql/execution/analysis/DetectAmbiguousSelfJoin.scala:77— DetectAmbiguousSelfJoin.apply has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ParquetReadSupport.findCheapestGroupField (cyclomatic 21) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala:554— ParquetReadSupport.findCheapestGroupField has cyclomatic complexity 21 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
PreprocessTableCreation.apply (cyclomatic 21) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala:141— PreprocessTableCreation.apply has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StreamExecution.runStream (cyclomatic 21) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala:282— StreamExecution.runStream has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
AwsStsCredentialProvider.resolve (cyclomatic 21) connector/credential-aws/src/main/java/org/apache/spark/security/aws/AwsStsCredentialProvider.java:310— AwsStsCredentialProvider.resolve has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
Variant.toJsonImpl (cyclomatic 21) common/variant/src/main/java/org/apache/spark/types/variant/Variant.java:282— Variant.toJsonImpl has cyclomatic complexity 21 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
UTF8String.indexOf (cyclomatic 21) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1320— UTF8String.indexOf has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
UTF8String.toInt (cyclomatic 21) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1922— UTF8String.toInt has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This shape REPEATS in the file: one other method here (UTF8String.toLong) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
UTF8String.toLong (cyclomatic 21) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1825— UTF8String.toLong has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This shape REPEATS in the file: one other method here (UTF8String.toInt) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
DataFrame.align (cyclomatic 21) python/pyspark/pandas/frame.py:13773— DataFrame.align has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
PandasOnSparkBoxPlot.boxplot (cyclomatic 21) python/pyspark/pandas/plot/matplotlib.py:87— PandasOnSparkBoxPlot.boxplot has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
typehints.spark_type_to_pandas_dtype (cyclomatic 21) python/pyspark/pandas/typedef/typehints.py:262— typehints.spark_type_to_pandas_dtype has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ServerPool._reap_retired (cyclomatic 21) python/pyspark/sql/connect/local_server_pool.py:715— ServerPool._reap_retired has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
typehints.infer_arrow_eval_type (cyclomatic 21) python/pyspark/sql/pandas/typehints.py:202— typehints.infer_arrow_eval_type has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
types._check_arrow_array_timestamps_localize (cyclomatic 21) python/pyspark/sql/pandas/types.py:554— types._check_arrow_array_timestamps_localize has cyclomatic complexity 21 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkSession.createDataFrame (cyclomatic 21) python/pyspark/sql/session.py:1427— SparkSession.createDataFrame has cyclomatic complexity 21 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
spark-sql-viz.setupPlanVizSearch (cyclomatic 21) REDACTED:850— spark-sql-viz.setupPlanVizSearch has cyclomatic complexity 21 (threshold 15). Most of this is not in the body itself: 7 of the 21 points are its own statements and the rest belongs to 3 function literals inside it that branch (lines 888, 875, 868). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
spark-sql-viz.runPlanVizSearch (cyclomatic 21) REDACTED:933— spark-sql-viz.runPlanVizSearch has cyclomatic complexity 21 (threshold 15). Most of this is not in the body itself: 6 of the 21 points are its own statements and the rest belongs to 2 function literals inside it that branch (lines 951, 980). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
ShuffleMigrationRunnable.run (cyclomatic 20) core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:103— ShuffleMigrationRunnable.run has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Client.createContainerLaunchContext (cyclomatic 20) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:1033— Client.createContainerLaunchContext has cyclomatic complexity 20 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ArrowUtils.fromArrowField (cyclomatic 20) sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:622— ArrowUtils.fromArrowField has cyclomatic complexity 20 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: ArrowUtils.fromArrowTypeDefault (cyclomatic 27) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
InternalRow.getAccessor (cyclomatic 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/InternalRow.scala:146— InternalRow.getAccessor has cyclomatic complexity 20 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
StreamingJoinHelper.collectTerms (cyclomatic 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala:213— StreamingJoinHelper.collectTerms has cyclomatic complexity 20 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
AstBuilder.withQueryResultClauses (cyclomatic 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1636— AstBuilder.withQueryResultClauses has cyclomatic complexity 20 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
AstBuilder.visitTypeConstructor (cyclomatic 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:4851— AstBuilder.visitTypeConstructor has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
AstBuilder.visitOperatorPipeAggregate (cyclomatic 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:8371— AstBuilder.visitOperatorPipeAggregate has cyclomatic complexity 20 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
StaxXmlGenerator.writeVariantPrimitive (cyclomatic 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlGenerator.scala:388— StaxXmlGenerator.writeVariantPrimitive has cyclomatic complexity 20 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: StaxXmlGenerator.writeElement (cyclomatic 28) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
PipelinesHandler.buildAutoCdcFlow (cyclomatic 20) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala:440— PipelinesHandler.buildAutoCdcFlow has cyclomatic complexity 20 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkConnectPlanner.handleWriteStreamOperationStart (cyclomatic 20) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:3688— SparkConnectPlanner.handleWriteStreamOperationStart has cyclomatic complexity 20 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: SparkConnectPlanner.transformRelation (cyclomatic 66) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
AdaptiveSparkPlanExec.withFinalPlanUpdate (cyclomatic 20) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:343— AdaptiveSparkPlanExec.withFinalPlanUpdate has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
PreprocessTableInsertion.preprocess (cyclomatic 20) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala:461— PreprocessTableInsertion.preprocess has cyclomatic complexity 20 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
V2ScanRelationPushDown.pushVariantExtractions (cyclomatic 20) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:441— V2ScanRelationPushDown.pushVariantExtractions has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
TransformWithStateInPySparkStateServer.handleMapStateRequest (cyclomatic 20) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:595— TransformWithStateInPySparkStateServer.handleMapStateRequest has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StreamingSymmetricHashJoinExec.processPartitions (cyclomatic 20) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala:371— StreamingSymmetricHashJoinExec.processPartitions has cyclomatic complexity 20 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
HadoopTableReader.fillObject (cyclomatic 20) sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala:440— HadoopTableReader.fillObject has cyclomatic complexity 20 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
MutableColumnarRow.get (cyclomatic 20) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnarRow.java:197— MutableColumnarRow.get has cyclomatic complexity 20 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
VectorizedRleValuesReader.readBatchRepeatedInternal (cyclomatic 20) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:437— VectorizedRleValuesReader.readBatchRepeatedInternal has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LateMaterializationParquetRecordReader.evaluateStorageFilter (cyclomatic 20) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:707— LateMaterializationParquetRecordReader.evaluateStorageFilter has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ColumnarBatchRow.copy (cyclomatic 20) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:51— ColumnarBatchRow.copy has cyclomatic complexity 20 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ColumnarRow.copy (cyclomatic 20) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:56— ColumnarRow.copy has cyclomatic complexity 20 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
VariantBuilder.appendWithInsertionImpl (cyclomatic 20) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:627— VariantBuilder.appendWithInsertionImpl has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
VariantBuilder.appendWithSetImpl (cyclomatic 20) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:730— VariantBuilder.appendWithSetImpl has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
TransportResponseHandler.handle (cyclomatic 20) common/network-common/src/main/java/org/apache/spark/network/client/TransportResponseHandler.java:166— TransportResponseHandler.handle has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.select_dtypes (cyclomatic 20) python/pyspark/pandas/frame.py:7452— DataFrame.select_dtypes has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Frame.groupby (cyclomatic 20) python/pyspark/pandas/generic.py:2395— Frame.groupby has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
typehints.infer_return_type (cyclomatic 20) python/pyspark/pandas/typedef/typehints.py:429— typehints.infer_return_type has cyclomatic complexity 20 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LiteralExpression._infer_type (cyclomatic 20) python/pyspark/sql/connect/expressions.py:321— LiteralExpression._infer_type has cyclomatic complexity 20 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
BasePythonRunner.compute (cyclomatic 19) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:644— BasePythonRunner.compute has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages. This is NOT this file's highest cyclomatic complexity: PythonEvalType.toString (cyclomatic 39) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
RBackendHandler.findMatchedSignature (cyclomatic 19) core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala:223— RBackendHandler.findMatchedSignature has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
FsHistoryProvider.doMergeApplicationListingInternal (cyclomatic 19) core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:1055— FsHistoryProvider.doMergeApplicationListingInternal has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
PipedRDD.compute (cyclomatic 19) core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala:68— PipedRDD.compute has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DAGScheduler.handleJobSubmitted (cyclomatic 19) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2619— DAGScheduler.handleJobSubmitted has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages. This is NOT this file's highest cyclomatic complexity: DAGSchedulerEventProcessLoop.doOnReceive (cyclomatic 27) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
BlockStoreUpdater.save (cyclomatic 19) core/src/main/scala/org/apache/spark/storage/BlockManager.scala:536— BlockStoreUpdater.save has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
AllJobsPage.render (cyclomatic 19) core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:278— AllJobsPage.render has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
KMeans.trainWithBlock (cyclomatic 19) mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala:526— KMeans.trainWithBlock has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
MultinomialLogisticBlockAggregator.add (cyclomatic 19) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregator.scala:103— MultinomialLogisticBlockAggregator.add has cyclomatic complexity 19 (threshold 15). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
DecisionTreeMetadata.buildMetadata (cyclomatic 19) mllib/src/main/scala/org/apache/spark/ml/tree/impl/DecisionTreeMetadata.scala:113— DecisionTreeMetadata.buildMetadata has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Row.equals (cyclomatic 19) sql/api/src/main/scala/org/apache/spark/sql/Row.scala:478— Row.equals has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: Row.jsonValue (cyclomatic 28) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
SparkDateTimeUtils.stringToTime (cyclomatic 19) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:1098— SparkDateTimeUtils.stringToTime has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ResolveInsertInto.apply (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1466— ResolveInsertInto.apply has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
BinByResolution.validateAndComputeParameters (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/BinByResolution.scala:63— BinByResolution.validateAndComputeParameters has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DeduplicateRelations.apply (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala:39— DeduplicateRelations.apply has cyclomatic complexity 19 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
TableOutputResolver.resolveColumnsByPosition (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:504— TableOutputResolver.resolveColumnsByPosition has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Cast.castToTimestamp (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:901— Cast.castToTimestamp has cyclomatic complexity 19 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: Cast.castInternal (cyclomatic 32) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
Literal.valueSizeInBytes (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala:313— Literal.valueSizeInBytes has cyclomatic complexity 19 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: Literal.componentTypeToDataType (cyclomatic 34) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
RewriteDistinctAggregates.rewrite (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:259— RewriteDistinctAggregates.rewrite has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
PullupCorrelatedPredicates.rewriteSubQueries (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:570— PullupCorrelatedPredicates.rewriteSubQueries has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
AstBuilder.visitMergeIntoTable (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1274— AstBuilder.visitMergeIntoTable has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ExtractEquiJoinKeys.unapply (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala:187— ExtractEquiJoinKeys.unapply has cyclomatic complexity 19 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
NormalizePlan.normalizeRecursive (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/NormalizePlan.scala:162— NormalizePlan.normalizeRecursive has cyclomatic complexity 19 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ToNumberParser.format (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala:649— ToNumberParser.format has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
XmlInferSchema.compatibleType (cyclomatic 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlInferSchema.scala:867— XmlInferSchema.compatibleType has cyclomatic complexity 19 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
LiteralValueProtoConverter.toLiteralProtoWithTypeDefault (cyclomatic 19) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/LiteralValueProtoConverter.scala:139— LiteralValueProtoConverter.toLiteralProtoWithTypeDefault has cyclomatic complexity 19 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: LiteralValueProtoConverter.toDataType (cyclomatic 34) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
ExecuteGrpcResponseSender.execute (cyclomatic 19) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteGrpcResponseSender.scala:205— ExecuteGrpcResponseSender.execute has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrameWriter.saveCommand (cyclomatic 19) sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala:131— DataFrameWriter.saveCommand has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
HashAggregateExec.doConsumeWithKeys (cyclomatic 19) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:925— HashAggregateExec.doConsumeWithKeys has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
OrcUtils.requestedColumnIds (cyclomatic 19) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:320— OrcUtils.requestedColumnIds has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
TableCapabilityCheck.apply (cyclomatic 19) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableCapabilityCheck.scala:39— TableCapabilityCheck.apply has cyclomatic complexity 19 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
UserDefinedPythonFunction.builder (cyclomatic 19) sql/core/src/main/scala/org/apache/spark/sql/execution/python/UserDefinedPythonFunction.scala:66— UserDefinedPythonFunction.builder has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
OneSideHashJoiner.storeAndJoinWithOtherSide (cyclomatic 19) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala:783— OneSideHashJoiner.storeAndJoinWithOtherSide has cyclomatic complexity 19 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
FileStreamSource.fetchMaxOffset (cyclomatic 19) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/FileStreamSource.scala:171— FileStreamSource.fetchMaxOffset has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SqlGraphRegistrationContext.buildChangeArgs (cyclomatic 19) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala:287— SqlGraphRegistrationContext.buildChangeArgs has cyclomatic complexity 19 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
RDD.histogram (cyclomatic 19) python/pyspark/core/rdd.py:2210— RDD.histogram has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.rename (cyclomatic 19) python/pyspark/pandas/frame.py:12113— DataFrame.rename has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Index.difference (cyclomatic 19) python/pyspark/pandas/indexes/base.py:2031— Index.difference has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
LocIndexer._get_from_multiindex_column (cyclomatic 19) python/pyspark/pandas/indexing.py:1246— LocIndexer._get_from_multiindex_column has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Series._interpolate (cyclomatic 19) python/pyspark/pandas/series.py:2327— Series._interpolate has cyclomatic complexity 19 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ArrowTableToRowsConversion._to_pylist (cyclomatic 19) python/pyspark/sql/conversion.py:1141— ArrowTableToRowsConversion._to_pylist has cyclomatic complexity 19 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
(anonymous) (cyclomatic 19) core/src/main/resources/org/apache/spark/ui/static/environmentpage.js:138— (anonymous) has cyclomatic complexity 19 (threshold 15). Most of this is not in the body itself: 3 of the 19 points are its own statements and the rest belongs to 10 function items inside it that branch (a function literal at line 164, (anonymous)::(anonymous)::createdRow, a function literal at line 186, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
KinesisUtilsPythonHelper.createStream (cyclomatic 18) connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtilsPythonHelper.scala:33— KinesisUtilsPythonHelper.createStream has cyclomatic complexity 18 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
ExecutorAllocationManager.validateSettings (cyclomatic 18) core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:213— ExecutorAllocationManager.validateSettings has cyclomatic complexity 18 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
FsHistoryProvider.checkForLogsInDir (cyclomatic 18) core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:635— FsHistoryProvider.checkForLogsInDir has cyclomatic complexity 18 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
Inbox.process (cyclomatic 18) core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala:86— Inbox.process has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DriverEndpoint.receive (cyclomatic 18) core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:178— DriverEndpoint.receive has cyclomatic complexity 18 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
DriverEndpoint.receiveAndReply (cyclomatic 18) core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:261— DriverEndpoint.receiveAndReply has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
EdgePartition.aggregateMessagesIndexScan (cyclomatic 18) graphx/src/main/scala/org/apache/spark/graphx/impl/EdgePartition.scala:428— EdgePartition.aggregateMessagesIndexScan has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
VectorIndexerModel.getTransformFunc (cyclomatic 18) mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala:369— VectorIndexerModel.getTransformFunc has cyclomatic complexity 18 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
YarnAllocator.processCompletedContainers (cyclomatic 18) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:848— YarnAllocator.processCompletedContainers has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DeduplicateRelations.deduplicate (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala:243— DeduplicateRelations.deduplicate has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: DeduplicateRelations.renewDuplicatedRelations (cyclomatic 19) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
TableOutputResolver.reorderColumnsByName (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:382— TableOutputResolver.reorderColumnsByName has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
JdkCodeCompiler.narrowingVerdict (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:1226— JdkCodeCompiler.narrowingVerdict has cyclomatic complexity 18 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
HiveHashFunction.hash (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:1298— HiveHashFunction.hash has cyclomatic complexity 18 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: HashExpression.computeHashWithTailRec (cyclomatic 19) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
PushdownPredicatesAndPruneColumnsForCTEDef.removePushedDownFilter (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushdownPredicatesAndPruneColumnsForCTEDef.scala:203— PushdownPredicatesAndPruneColumnsForCTEDef.removePushedDownFilter has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StarSchemaDetection.findStarJoins (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/StarSchemaDetection.scala:73— StarSchemaDetection.findStarJoins has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SimplifyBinaryComparison.apply (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:663— SimplifyBinaryComparison.apply has cyclomatic complexity 18 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
JoinEstimation.estimateInnerOuterJoin (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala:55— JoinEstimation.estimateInnerOuterJoin has cyclomatic complexity 18 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
GenericArrayData.equals (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala:90— GenericArrayData.equals has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
XmlTokenizer.readUntilEndElement (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1067— XmlTokenizer.readUntilEndElement has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: StaxXmlParser.castTo (cyclomatic 19) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
SchemaUtils.validateTypeCompatibility (cyclomatic 18) sql/catalyst/src/main/scala/org/apache/spark/sql/util/SchemaUtils.scala:428— SchemaUtils.validateTypeCompatibility has cyclomatic complexity 18 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
UnionLoopExec.doExecute (cyclomatic 18) sql/core/src/main/scala/org/apache/spark/sql/execution/UnionLoopExec.scala:155— UnionLoopExec.doExecute has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataSourceAnalysis.apply (cyclomatic 18) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:144— DataSourceAnalysis.apply has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: DataSourceStrategy.translateLeafNodeFilter (cyclomatic 23) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
OrcUtils.orcTypeDescription (cyclomatic 18) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:456— OrcUtils.orcTypeDescription has cyclomatic complexity 18 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SparkShreddingUtils.extractField (cyclomatic 18) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala:788— SparkShreddingUtils.extractField has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
XSDToSchema.getStructFieldsFromParticle (cyclomatic 18) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:198— XSDToSchema.getStructFieldsFromParticle has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RocksDBFileManager.deleteOldVersions (cyclomatic 18) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:659— RocksDBFileManager.deleteOldVersions has cyclomatic complexity 18 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
AvroStateEncoder.encodePrefixKeyForRangeScan (cyclomatic 18) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:1063— AvroStateEncoder.encodePrefixKeyForRangeScan has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RowSetUtils.toTColumnValue (cyclomatic 18) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/RowSetUtils.scala:187— RowSetUtils.toTColumnValue has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
HiveInspectors.unwrapperFor (cyclomatic 18) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:852— HiveInspectors.unwrapperFor has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: HiveInspectors.javaTypeToDataType (cyclomatic 41) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
VectorizedColumnReader.readBatch (cyclomatic 18) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:257— VectorizedColumnReader.readBatch has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LateMaterializationParquetRecordReader.loadNextRowGroup (cyclomatic 18) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:511— LateMaterializationParquetRecordReader.loadNextRowGroup has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
CommandBuilderUtils.parseOptionString (cyclomatic 18) launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java:129— CommandBuilderUtils.parseOptionString has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
CollationAwareUTF8String.lowercaseTrimRight (cyclomatic 18) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1490— CollationAwareUTF8String.lowercaseTrimRight has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
CollationAwareUTF8String.trimRight (cyclomatic 18) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1588— CollationAwareUTF8String.trimRight has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
connect._convert_exception (cyclomatic 18) python/pyspark/errors/exceptions/connect.py:172— connect._convert_exception has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.__init__ (cyclomatic 18) python/pyspark/pandas/frame.py:552— DataFrame.__init__ has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Frame.to_csv (cyclomatic 18) python/pyspark/pandas/generic.py:600— Frame.to_csv has cyclomatic complexity 18 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
utils.align_diff_frames (cyclomatic 18) python/pyspark/pandas/utils.py:325— utils.align_diff_frames has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataFrame.explain (cyclomatic 18) python/pyspark/sql/classic/dataframe.py:236— DataFrame.explain has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ArtifactManager._parse_artifacts (cyclomatic 18) python/pyspark/sql/connect/client/artifact.py:183— ArtifactManager._parse_artifacts has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
plan_data_source_read.records_to_arrow_batches (cyclomatic 18) python/pyspark/sql/worker/plan_data_source_read.py:55— plan_data_source_read.records_to_arrow_batches has cyclomatic complexity 18 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StringSubstitutor.substitute (cyclomatic 17) common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:44— StringSubstitutor.substitute has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
NettyBlockRpcServer.receive (cyclomatic 17) core/src/main/scala/org/apache/spark/network/netty/NettyBlockRpcServer.scala:51— NettyBlockRpcServer.receive has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ResourceProfileManager.isSupported (cyclomatic 17) core/src/main/scala/org/apache/spark/resource/ResourceProfileManager.scala:69— ResourceProfileManager.isSupported has cyclomatic complexity 17 (threshold 15). To reduce it, name the conditions: bind each compound test to a well-named local or a small predicate function, so the body reads as a sequence of named decisions rather than a chain of operators.
BLAS.gemmImpl (cyclomatic 17) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:523— BLAS.gemmImpl has cyclomatic complexity 17 (threshold 15). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
ParsedRFormula.resolve (cyclomatic 17) mllib/src/main/scala/org/apache/spark/ml/feature/RFormulaParser.scala:37— ParsedRFormula.resolve has cyclomatic complexity 17 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
Word2Vec.doFit (cyclomatic 17) mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala:333— Word2Vec.doFit has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
BLAS.gemm (cyclomatic 17) mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:436— BLAS.gemm has cyclomatic complexity 17 (threshold 15). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
LogisticGradient.compute (cyclomatic 17) mllib/src/main/scala/org/apache/spark/mllib/optimization/Gradient.scala:165— LogisticGradient.compute has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ExecutorPodsLifecycleManager.onNewSnapshots (cyclomatic 17) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala:104— ExecutorPodsLifecycleManager.onNewSnapshots has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ApplyCharTypePaddingHelper.singleNodePaddingForStringComparison (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyCharTypePaddingHelper.scala:81— ApplyCharTypePaddingHelper.singleNodePaddingForStringComparison has cyclomatic complexity 17 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
CTESubstitution.apply (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala:54— CTESubstitution.apply has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
CollationTypeCoercion.findCollationContext (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala:305— CollationTypeCoercion.findCollationContext has cyclomatic complexity 17 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ImplicitTypeCoercion.apply (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:733— ImplicitTypeCoercion.apply has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RoundBase.nullSafeEval (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala:1774— RoundBase.nullSafeEval has cyclomatic complexity 17 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
BaseGenericInternalRow.equals (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala:87— BaseGenericInternalRow.equals has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
JsonInferSchema.compatibleType (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:383— JsonInferSchema.compatibleType has cyclomatic complexity 17 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
GeneratorNestedColumnAliasing.unapply (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala:322— GeneratorNestedColumnAliasing.unapply has cyclomatic complexity 17 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ConvertToCatalyst.applyExpr (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:1307— ConvertToCatalyst.applyExpr has cyclomatic complexity 17 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
RuleExecutor.execute (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala:214— RuleExecutor.execute has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DataSourceV2Relation.transformV2Stats (cyclomatic 17) sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:628— DataSourceV2Relation.transformV2Stats has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkPlan.executeTake (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala:525— SparkPlan.executeTake has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
WholeStageCodegenExec.decideFromTrials (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala:857— WholeStageCodegenExec.decideFromTrials has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DemoteBroadcastHashJoin.selectJoinStrategy (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/DemoteBroadcastHashJoin.scala:41— DemoteBroadcastHashJoin.selectJoinStrategy has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
AggregatePushDownUtils.getSchemaForPushedAggregation (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala:38— AggregatePushDownUtils.getSchemaForPushedAggregation has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
FileSourceStrategy.apply (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:203— FileSourceStrategy.apply has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
JDBCRDD.compute (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala:288— JDBCRDD.compute has cyclomatic complexity 17 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
InferVariantShreddingSchema.mergeSchema (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/InferVariantShreddingSchema.scala:152— InferVariantShreddingSchema.mergeSchema has cyclomatic complexity 17 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
InferVariantShreddingSchema.collectFieldStats (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/InferVariantShreddingSchema.scala:356— InferVariantShreddingSchema.collectFieldStats has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SQLLastAttemptAccumulator.extractStageRDDScopes (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala:295— SQLLastAttemptAccumulator.extractStageRDDScopes has cyclomatic complexity 17 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
StatFunctions.summary (cyclomatic 17) sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala:174— StatFunctions.summary has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
TriggeredGraphExecution.topologicalExecution (cyclomatic 17) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/TriggeredGraphExecution.scala:157— TriggeredGraphExecution.topologicalExecution has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
VectorizedColumnReader.isLazyDecodingSupported (cyclomatic 17) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:191— VectorizedColumnReader.isLazyDecodingSupported has cyclomatic complexity 17 (threshold 15). To reduce it, name the conditions: bind each compound test to a well-named local or a small predicate function, so the body reads as a sequence of named decisions rather than a chain of operators.
RecordBinaryComparator.compare (cyclomatic 17) sql/core/src/main/java/org/apache/spark/sql/execution/RecordBinaryComparator.java:32— RecordBinaryComparator.compare has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SparkLauncher.startApplication (cyclomatic 17) launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java:384— SparkLauncher.startApplication has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
AwsStsCredentialProvider.init (cyclomatic 17) connector/credential-aws/src/main/java/org/apache/spark/security/aws/AwsStsCredentialProvider.java:164— AwsStsCredentialProvider.init has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
UTF8String.levenshteinDistance (cyclomatic 17) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:2228— UTF8String.levenshteinDistance has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ArrowBatchedUDFArrowHandler.run (cyclomatic 17) python/pyspark/eval_handlers/_arrow.py:865— ArrowBatchedUDFArrowHandler.run has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RemoteMLReader.loadInstance (cyclomatic 17) python/pyspark/ml/connect/readwrite.py:222— RemoteMLReader.loadInstance has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataFrame.corrwith (cyclomatic 17) python/pyspark/pandas/frame.py:1764— DataFrame.corrwith has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataFrame._assign (cyclomatic 17) python/pyspark/pandas/frame.py:5851— DataFrame._assign has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
LiteralExpression.__repr__ (cyclomatic 17) python/pyspark/sql/connect/expressions.py:521— LiteralExpression.__repr__ has cyclomatic complexity 17 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: LiteralExpression.__init__ (cyclomatic 19) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
ArrowToPandasConversion._convert_array_numpy (cyclomatic 17) python/pyspark/sql/conversion.py:2075— ArrowToPandasConversion._convert_array_numpy has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
types._to_corrected_pandas_type (cyclomatic 17) python/pyspark/sql/pandas/types.py:920— types._to_corrected_pandas_type has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
UserDefinedFunction.__init__ (cyclomatic 17) python/pyspark/sql/udf.py:168— UserDefinedFunction.__init__ has cyclomatic complexity 17 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
getInnerText (cyclomatic 17) REDACTED:212— getInnerText has cyclomatic complexity 17 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
KafkaSourceProvider.validateBatchOptions (cyclomatic 16) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala:370— KafkaSourceProvider.validateBatchOptions has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
PythonWorkerFactory.startDaemon (cyclomatic 16) core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala:338— PythonWorkerFactory.startDaemon has cyclomatic complexity 16 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
CoarseGrainedExecutorBackend.decommissionSelf (cyclomatic 16) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:318— CoarseGrainedExecutorBackend.decommissionSelf has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
IndexShuffleBlockResolver.writeMetadataFileAndCommit (cyclomatic 16) core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala:391— IndexShuffleBlockResolver.writeMetadataFileAndCommit has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
BlockManagerMasterEndpoint.updateBlockInfo (cyclomatic 16) core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala:828— BlockManagerMasterEndpoint.updateBlockInfo has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top. This is NOT this file's highest cyclomatic complexity: BlockManagerMasterEndpoint.receiveAndReply (cyclomatic 31) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
LogisticRegression.createInitialSolution (cyclomatic 16) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:816— LogisticRegression.createInitialSolution has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
GradientBoostedTrees.boost (cyclomatic 16) mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala:290— GradientBoostedTrees.boost has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RandomForest.runBagged (cyclomatic 16) mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala:118— RandomForest.runBagged has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RandomForest.findBestSplits (cyclomatic 16) mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala:526— RandomForest.findBestSplits has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RowMatrix.columnSimilaritiesDIMSUM (cyclomatic 16) mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:722— RowMatrix.columnSimilaritiesDIMSUM has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Metadata.fromJObject (cyclomatic 16) sql/api/src/main/scala/org/apache/spark/sql/types/Metadata.scala:171— Metadata.fromJObject has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
CollapseGroupedSumOfCount.collapse (cyclomatic 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/CollapseGroupedSumOfCount.scala:53— CollapseGroupedSumOfCount.collapse has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
InlineCTE.inlineCTE (cyclomatic 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:238— InlineCTE.inlineCTE has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
OptimizeJoinCondition.optimizeCondition (cyclomatic 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJoinCondition.scala:38— OptimizeJoinCondition.optimizeCondition has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
CombineUnions.flattenUnion (cyclomatic 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:2340— CombineUnions.flattenUnion has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
ToNumberParser.formatDigitGroups (cyclomatic 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala:788— ToNumberParser.formatDigitGroups has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
util.usePrettyExpression (cyclomatic 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/package.scala:90— util.usePrettyExpression has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
Abbreviator.abbreviateRecursive (cyclomatic 16) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/Abbreviator.scala:39— Abbreviator.abbreviateRecursive has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
Serializer.deserializeMethodArguments (cyclomatic 16) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/Serializer.scala:140— Serializer.deserializeMethodArguments has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
SparkConnectPlanner.handleStreamingQueryCommand (cyclomatic 16) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:3864— SparkConnectPlanner.handleStreamingQueryCommand has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing. This is NOT this file's highest cyclomatic complexity: SparkConnectPlanner.transformRelation (cyclomatic 66) is higher and carries no row of its own — it was excluded as a flat dispatcher (a long switch/match over independent cases: many branches, almost no nesting), which this dimension does not treat as a refactor obligation. It is named here so the ranking you see in this file is not mistaken for the whole of it; the excluded method is counted neither in this dimension's figures nor in its score.
ResolveDataSource.apply (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:45— ResolveDataSource.apply has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
SQLExecution.withNewExecutionId0 (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:205— SQLExecution.withNewExecutionId0 has cyclomatic complexity 16 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
SparkSqlAstBuilder.visitCreateUserDefinedFunction (cyclomatic 16) REDACTED:990— SparkSqlAstBuilder.visitCreateUserDefinedFunction has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Aggregation.apply (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:745— Aggregation.apply has cyclomatic complexity 16 (threshold 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Where every arm is uniform — the same kind of value, with no behaviour of its own — a table keyed by the case is the shorter form; wherever the arms carry different data or different behaviour, keep them as cases, because collapsing those trades an explicit, reviewable set of cases for nothing.
TungstenAggregationIterator.processInputs (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala:208— TungstenAggregationIterator.processInputs has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
TruncateTableCommand.run (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:447— TruncateTableCommand.run has cyclomatic complexity 16 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
EmbeddedArraySplitter.consumeValue (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/EmbeddedArrayJsonDataSource.scala:216— EmbeddedArraySplitter.consumeValue has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StateDataSource.runStateVarChecks (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala:201— StateDataSource.runStateVarChecks has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
XmlRecordReader.readUntilEndElement (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlInputFormat.scala:231— XmlRecordReader.readUntilEndElement has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
StateStoreSaveExec.doExecute (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/statefulOperators.scala:880— StateStoreSaveExec.doExecute has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
RocksDBStateStoreChangeDataReader.getNext (cyclomatic 16) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:1676— RocksDBStateStoreChangeDataReader.getNext has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
MySQLDialect.getCatalystType (cyclomatic 16) REDACTED:144— MySQLDialect.getCatalystType has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
OracleDialect.getCatalystType (cyclomatic 16) REDACTED:155— OracleDialect.getCatalystType has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ProcessIterator.advance (cyclomatic 16) udf/worker/grpc/src/main/scala/org/apache/spark/udf/worker/grpc/GrpcWorkerSession.scala:923— ProcessIterator.advance has cyclomatic complexity 16 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
LateMaterializationParquetRecordReader.initializeLateMaterialization (cyclomatic 16) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:419— LateMaterializationParquetRecordReader.initializeLateMaterialization has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ExpressionImplUtils.aesInternal (cyclomatic 16) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java:277— ExpressionImplUtils.aesInternal has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ExpressionImplUtils.jaroWinklerSimilarity (cyclomatic 16) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java:540— ExpressionImplUtils.jaroWinklerSimilarity has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
VariantShreddingWriter.castShredded (cyclomatic 16) common/variant/src/main/java/org/apache/spark/types/variant/VariantShreddingWriter.java:59— VariantShreddingWriter.castShredded has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
DecryptionHandler.channelRead (cyclomatic 16) common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:414— DecryptionHandler.channelRead has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
instrumentation_utils._attach (cyclomatic 16) python/pyspark/instrumentation_utils.py:121— instrumentation_utils._attach has cyclomatic complexity 16 (threshold 15). To reduce it, separate the branches: extract each independent case into its own named function so the top-level body reads as a short sequence of named decisions.
DataFrame.replace (cyclomatic 16) python/pyspark/pandas/frame.py:6519— DataFrame.replace has cyclomatic complexity 16 (threshold 15). To reduce it, split the body: these branches sit side by side rather than nested inside one another, so extracting each one on its own would leave a function per branch. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataFrame.quantile (cyclomatic 16) python/pyspark/pandas/frame.py:13007— DataFrame.quantile has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
Frame.truncate (cyclomatic 16) python/pyspark/pandas/generic.py:3110— Frame.truncate has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
typehints._new_type_holders (cyclomatic 16) python/pyspark/pandas/typedef/typehints.py:855— typehints._new_type_holders has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
ArrowBatchTransformer.enforce_schema (cyclomatic 16) python/pyspark/sql/conversion.py:230— ArrowBatchTransformer.enforce_schema has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
transpile._get_function_from_ast (cyclomatic 16) python/pyspark/sql/transpile.py:1115— transpile._get_function_from_ast has cyclomatic complexity 16 (threshold 15). To reduce it, separate the cases: extract each independent branch into its own named function, and where the body has guards that only reject input, fold those into early returns at the top.
No assertions: All inline scripts use CSP nonce ui-test/tests/csp-compliance.test.js:55— This method's body runs code, and no assertion call was recognised in it. Recognised by name: Assert*, *Should*/ShouldBe*, Verify, Expect, Throws, Record, Received/DidNotReceive, MustHaveHappened/MustNotHaveHappened, EnsureSuccessStatusCode and *AndEnsure* — so verification routed through a helper of your own naming, through a base-class or callback object whose members hold the assertions, or through a harness that fails by throwing under some other name, is not visible to this check and is not counted here. Read it as 'no assertion this check knows how to see', and if that is right, add one.
Repeated repair: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala:95— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala changed 3 times in last 90 days and 3 of those changes were fix/bug commits, so repair is the majority of this file's churn. Its max cyclomatic complexity is 4 (its worst body is TransformExpression.isCompatible at line 95), UNDER the 15 threshold, so this is deliberately not filed as a churn × complexity hotspot — the difficulty here is in the behaviour the file has to get right, not in its control flow, and refactoring it for complexity would be the wrong move. The repairs counted were: “[SPARK-59887][SPARK-59901][SQL] Fix wrong results when a storage-partitioned join pairs a transform of a join-key expression with one of a column”; “[SPARK-59121][SQL] Fix wrong results when a storage-partitioned join reduces the partition keys of both sides”; “[SPARK-59045][SQL] Fix SPJ ClassCastException when reducer changes partition key data type”. Each one is a case this code did not handle. Before the next change lands here, check that every one of them is pinned by a test that fails without its fix; where the same area keeps coming back, the durable fix is usually at the interface that keeps being misused rather than at the line that was last corrected. Counted over 2026-07-28..2026-10-26, the 90 days ending at the analysed commit. Reproduce with `git log --since='2026-07-28 09:58:39 +02:00' --until='2026-10-26 09:58:39 +02:00' --full-history --no-merges -- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/TransformExpression.scala`: merges are excluded because a merge re-states changes already counted at their own commits, and history is NOT path-simplified because a change that reached the file through a merged branch is still a change to it. That command counts raw commits and can read HIGHER than this row, which counts a cherry-picked re-land, and a revert together with the commit it undoes, once each — a difference of several commits on a file whose history was re-landed or reverted inside the window.
XxxComment streaming/src/test/scala/org/apache/spark/streaming/ReceiverSuite.scala:179— // XXX Checking every block would require an even distribution of messages across blocks, — source code is not a task system: move the work to your tracker and leave a reference instead (e.g. `// REF: #123`), so the task is planned where tasks live and the ticket links back to the code.
SparkSubmit.prepareSubmitEnvironment (cognitive 201) core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:243— SparkSubmit.prepareSubmitEnvironment has cognitive complexity 201 (threshold 15). Drivers by points: if/else 92 (152 pts), boolean chains 32, match/switch 6 (8 pts), loops 3 (7 pts), error handling 1 (2 pts) (nesting depth added 67). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DAGScheduler.handleTaskCompletion (cognitive 198) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:3891— DAGScheduler.handleTaskCompletion has cognitive complexity 198 (threshold 15). Drivers by points: if/else 52 (143 pts), match/switch 10 (25 pts), boolean chains 16, error handling 3 (14 pts) (nesting depth added 117). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
(anonymous) (cognitive 194) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:343— (anonymous) has cognitive complexity 194 (threshold 15). Drivers by points: if/else 107 (136 pts), ternaries 20 (32 pts), boolean chains 25, match/switch 1 (nesting depth added 41). None of this is in the body itself: all 194 points belong to 55 function items inside it that branch (a function literal at line 404, a function literal at line 678, a function literal at line 1082, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
UnsupportedOperationChecker.checkForStreaming (cognitive 145) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/UnsupportedOperationChecker.scala:191— UnsupportedOperationChecker.checkForStreaming has cognitive complexity 145 (threshold 15). Drivers by points: if/else 45 (81 pts), match/switch 21 (38 pts), boolean chains 26 (nesting depth added 53). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CheckAnalysis.checkAnalysis0 (cognitive 141) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:392— CheckAnalysis.checkAnalysis0 has cognitive complexity 141 (threshold 15). Drivers by points: match/switch 25 (62 pts), if/else 32 (59 pts), boolean chains 20 (nesting depth added 64). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DecorrelateInnerQuery.apply (cognitive 140) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/DecorrelateInnerQuery.scala:464— DecorrelateInnerQuery.apply has cognitive complexity 140 (threshold 15). Drivers by points: if/else 51 (102 pts), match/switch 10 (34 pts), boolean chains 4 (nesting depth added 75). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LocIndexerLike.__setitem__ (cognitive 129) python/pyspark/pandas/indexing.py:593— LocIndexerLike.__setitem__ has cognitive complexity 129 (threshold 15). Drivers by points: if/else 45 (108 pts), boolean chains 14, loops 3 (7 pts) (nesting depth added 67). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SparkSession.createDataFrame (cognitive 116) python/pyspark/sql/connect/session.py:508— SparkSession.createDataFrame has cognitive complexity 116 (threshold 15). Drivers by points: if/else 45 (85 pts), ternaries 6 (17 pts), boolean chains 10, loops 1 (4 pts) (nesting depth added 54). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HiveInspectors.unwrapperFor (cognitive 113) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:550— HiveInspectors.unwrapperFor has cognitive complexity 113 (threshold 15). Drivers by points: if/else 57 (110 pts), match/switch 2 (3 pts) (nesting depth added 54). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkDateTimeUtils.parseTimestampString (cognitive 109) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:709— SparkDateTimeUtils.parseTimestampString has cognitive complexity 109 (threshold 15). Drivers by points: if/else 36 (90 pts), boolean chains 16, loops 3 (nesting depth added 54). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
PlanMerger.tryMergePlans (cognitive 109) sql/core/src/main/scala/org/apache/spark/sql/execution/planmerging/PlanMerger.scala:404— PlanMerger.tryMergePlans has cognitive complexity 109 (threshold 15). Drivers by points: match/switch 17 (54 pts), if/else 22 (39 pts), boolean chains 16 (nesting depth added 54). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ParquetVectorUpdaterFactory.getUpdater (cognitive 101) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java:75— ParquetVectorUpdaterFactory.getUpdater has cognitive complexity 101 (threshold 15). Drivers by points: if/else 58 (83 pts), boolean chains 14, ternaries 1 (3 pts), match/switch 1 (nesting depth added 27). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
namespace.read_csv (cognitive 100) REDACTED:263— namespace.read_csv has cognitive complexity 100 (threshold 15). Drivers by points: if/else 39 (83 pts), loops 3 (8 pts), boolean chains 5, ternaries 2 (4 pts) (nesting depth added 51). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
ShuffleBlockFetcherIterator.next (cognitive 99) core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala:808— ShuffleBlockFetcherIterator.next has cognitive complexity 99 (threshold 15). Drivers by points: if/else 27 (76 pts), error handling 3 (11 pts), match/switch 2 (7 pts), boolean chains 4, loops 1 (nesting depth added 62). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
merge_spark_pr.main (cognitive 97) REDACTED:2332— merge_spark_pr.main has cognitive complexity 97 (threshold 15). Drivers by points: if/else 39 (68 pts), loops 7 (10 pts), boolean chains 9, ternaries 3 (6 pts), error handling 2 (4 pts) (nesting depth added 37). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveSessionCatalog.apply (cognitive 96) REDACTED:56— ResolveSessionCatalog.apply has cognitive complexity 96 (threshold 15). Drivers by points: if/else 45 (72 pts), match/switch 8 (15 pts), boolean chains 9 (nesting depth added 34). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
MLExpressionUtils.scaleShift (cognitive 94) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ml/MLExpressionUtils.java:194— MLExpressionUtils.scaleShift has cognitive complexity 94 (threshold 15). Drivers by points: loops 15 (49 pts), if/else 25 (33 pts), boolean chains 6, ternaries 4 (6 pts) (nesting depth added 44). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
SqlStatementSplitter.splitWithPositions (cognitive 92) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementSplitter.scala:153— SqlStatementSplitter.splitWithPositions has cognitive complexity 92 (threshold 15). Drivers by points: if/else 22 (59 pts), loops 8 (23 pts), boolean chains 6, match/switch 1 (4 pts) (nesting depth added 55). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
TaskSchedulerImpl.resourceOffers (cognitive 91) core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala:573— TaskSchedulerImpl.resourceOffers has cognitive complexity 91 (threshold 15). Drivers by points: if/else 25 (71 pts), loops 6 (11 pts), boolean chains 5, match/switch 1 (4 pts) (nesting depth added 54). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
DataSourceV2Strategy.apply (cognitive 91) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:130— DataSourceV2Strategy.apply has cognitive complexity 91 (threshold 15). Drivers by points: if/else 34 (51 pts), match/switch 17 (34 pts), boolean chains 6 (nesting depth added 34). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
Decimal.changePrecision (cognitive 89) sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala:387— Decimal.changePrecision has cognitive complexity 89 (threshold 15). Drivers by points: if/else 29 (75 pts), match/switch 2 (8 pts), boolean chains 6 (nesting depth added 52). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
namespace.concat (cognitive 87) REDACTED:2380— namespace.concat has cognitive complexity 87 (threshold 15). Drivers by points: if/else 39 (66 pts), loops 7 (13 pts), boolean chains 4, ternaries 2 (4 pts) (nesting depth added 35). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
V2ExpressionBuilder.generateExpression (cognitive 86) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala:105— V2ExpressionBuilder.generateExpression has cognitive complexity 86 (threshold 15). Drivers by points: if/else 39 (59 pts), boolean chains 14, match/switch 6 (13 pts) (nesting depth added 27). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SchemaConverters.toSqlTypeHelper (cognitive 86) sql/core/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala:149— SchemaConverters.toSqlTypeHelper has cognitive complexity 86 (threshold 15). Drivers by points: if/else 39 (74 pts), match/switch 5 (10 pts), boolean chains 2 (nesting depth added 40). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AvroDeserializer.newWriter (cognitive 85) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:172— AvroDeserializer.newWriter has cognitive complexity 85 (threshold 15). Drivers by points: if/else 25 (41 pts), match/switch 12 (34 pts), boolean chains 6, loops 2 (4 pts) (nesting depth added 40). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
daemon.manager (cognitive 82) python/pyspark/daemon.py:115— daemon.manager has cognitive complexity 82 (threshold 15). Drivers by points: if/else 20 (45 pts), error handling 6 (22 pts), loops 4 (14 pts), boolean chains 1 (nesting depth added 51). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JacksonParser.makeConverter (cognitive 81) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:260— JacksonParser.makeConverter has cognitive complexity 81 (threshold 15). Drivers by points: match/switch 27 (57 pts), if/else 8 (14 pts), error handling 3 (10 pts) (nesting depth added 43). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
(anonymous) (cognitive 79) core/src/main/resources/org/apache/spark/ui/static/executorspage.js:348— (anonymous) has cognitive complexity 79 (threshold 15). Drivers by points: if/else 53 (64 pts), ternaries 10 (12 pts), boolean chains 3 (nesting depth added 13). None of this is in the body itself: all 79 points belong to 33 function items inside it that branch ((anonymous)::(anonymous)::(anonymous)::data, (anonymous)::(anonymous)::(anonymous)::data, (anonymous)::(anonymous)::(anonymous)::data, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
DataFrame.filter (cognitive 78) python/pyspark/pandas/frame.py:11938— DataFrame.filter has cognitive complexity 78 (threshold 15). Drivers by points: if/else 27 (63 pts), loops 4 (15 pts) (nesting depth added 47). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
Series.replace (cognitive 77) python/pyspark/pandas/series.py:5010— Series.replace has cognitive complexity 77 (threshold 15). Drivers by points: if/else 28 (55 pts), ternaries 5 (15 pts), boolean chains 4, loops 1 (3 pts) (nesting depth added 39). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
VariantShreddingWriter.tryTypedShred (cognitive 76) common/variant/src/main/java/org/apache/spark/types/variant/VariantShreddingWriter.java:155— VariantShreddingWriter.tryTypedShred has cognitive complexity 76 (threshold 15). Drivers by points: if/else 24 (65 pts), match/switch 3 (7 pts), boolean chains 4 (nesting depth added 45). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
WeightedLeastSquares.fit (cognitive 75) mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala:99— WeightedLeastSquares.fit has cognitive complexity 75 (threshold 15). Drivers by points: if/else 36 (58 pts), boolean chains 7, loops 6 (7 pts), error handling 1 (2 pts), match/switch 1 (nesting depth added 24). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.drop (cognitive 75) python/pyspark/pandas/frame.py:7748— DataFrame.drop has cognitive complexity 75 (threshold 15). Drivers by points: if/else 24 (64 pts), ternaries 2 (8 pts), boolean chains 3 (nesting depth added 46). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
LocIndexer._select_rows_by_slice (cognitive 75) python/pyspark/pandas/indexing.py:1064— LocIndexer._select_rows_by_slice has cognitive complexity 75 (threshold 15). Drivers by points: if/else 24 (55 pts), ternaries 4 (8 pts), boolean chains 6, loops 2 (6 pts) (nesting depth added 39). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
CatalystTranspiler._convert_chunk (cognitive 75) python/pyspark/sql/transpile.py:562— CatalystTranspiler._convert_chunk has cognitive complexity 75 (threshold 15). Drivers by points: if/else 21 (49 pts), boolean chains 8, ternaries 3 (7 pts), loops 2 (6 pts), match/switch 3 (5 pts) (nesting depth added 38). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkIntervalUtils.stringToInterval (cognitive 72) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkIntervalUtils.scala:134— SparkIntervalUtils.stringToInterval has cognitive complexity 72 (threshold 15). Drivers by points: if/else 26 (42 pts), match/switch 6 (15 pts), boolean chains 7, error handling 2 (7 pts), loops 1 (nesting depth added 30). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MultiColumnInSubqueryEvaluator.eval (cognitive 72) sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala:337— MultiColumnInSubqueryEvaluator.eval has cognitive complexity 72 (threshold 15). Drivers by points: if/else 23 (47 pts), loops 6 (15 pts), boolean chains 10 (nesting depth added 33). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkSQLCLIDriver.processCmd (cognitive 72) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala:442— SparkSQLCLIDriver.processCmd has cognitive complexity 72 (threshold 15). Drivers by points: if/else 15 (37 pts), match/switch 3 (16 pts), error handling 2 (8 pts), boolean chains 7, loops 1 (4 pts) (nesting depth added 44). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
DataFrame.melt (cognitive 72) python/pyspark/pandas/frame.py:10852— DataFrame.melt has cognitive complexity 72 (threshold 15). Drivers by points: if/else 30 (49 pts), ternaries 8 (21 pts), boolean chains 2 (nesting depth added 32). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ToStringBase.castToStringDefault (cognitive 71) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala:93— ToStringBase.castToStringDefault has cognitive complexity 71 (threshold 15). Drivers by points: if/else 22 (61 pts), loops 3 (9 pts), match/switch 1 (nesting depth added 45). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SortMergeJoinEvaluator.eval (cognitive 71) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:58— SortMergeJoinEvaluator.eval has cognitive complexity 71 (threshold 15). Drivers by points: if/else 15 (46 pts), loops 7 (19 pts), boolean chains 5, match/switch 1 (nesting depth added 43). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DataTypeAstBuilder.visitPrimitiveDataType (cognitive 70) sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala:292— DataTypeAstBuilder.visitPrimitiveDataType has cognitive complexity 70 (threshold 15). Drivers by points: if/else 29 (50 pts), match/switch 7 (20 pts) (nesting depth added 34). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
functions._validate_and_transform_prediction_result (cognitive 69) python/pyspark/ml/functions.py:283— functions._validate_and_transform_prediction_result has cognitive complexity 69 (threshold 15). Drivers by points: if/else 22 (58 pts), loops 2 (6 pts), boolean chains 3, ternaries 1 (2 pts) (nesting depth added 41). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
namespace.merge_asof (cognitive 69) REDACTED:3110— namespace.merge_asof has cognitive complexity 69 (threshold 15). Drivers by points: if/else 34 (56 pts), boolean chains 11, loops 2 (nesting depth added 22). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.merge (cognitive 68) python/pyspark/pandas/frame.py:8811— DataFrame.merge has cognitive complexity 68 (threshold 15). Drivers by points: if/else 31 (54 pts), boolean chains 8, loops 3 (6 pts) (nesting depth added 26). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
namespace.get_dummies (cognitive 67) REDACTED:2153— namespace.get_dummies has cognitive complexity 67 (threshold 15). Drivers by points: if/else 24 (47 pts), ternaries 4 (13 pts), boolean chains 4, loops 2 (3 pts) (nesting depth added 33). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
StateSourceOptions.apply (cognitive 66) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala:546— StateSourceOptions.apply has cognitive complexity 66 (threshold 15). Drivers by points: if/else 38 (57 pts), boolean chains 5, error handling 4 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkResult.processResponses (cognitive 65) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:119— SparkResult.processResponses has cognitive complexity 65 (threshold 15). Drivers by points: if/else 19 (53 pts), loops 2 (5 pts), match/switch 1 (5 pts), boolean chains 2 (nesting depth added 41). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
AbstractCommandBuilder.buildClassPath (cognitive 65) launcher/src/main/java/org/apache/spark/launcher/AbstractCommandBuilder.java:142— AbstractCommandBuilder.buildClassPath has cognitive complexity 65 (threshold 15). Drivers by points: if/else 15 (40 pts), boolean chains 14, loops 4 (11 pts) (nesting depth added 32). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GroupBy.apply (cognitive 65) python/pyspark/pandas/groupby.py:1818— GroupBy.apply has cognitive complexity 65 (threshold 15). Drivers by points: if/else 38 (60 pts), boolean chains 2, ternaries 1 (2 pts), error handling 1 (nesting depth added 23). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AstBuilder.withPredicate (cognitive 63) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:3861— AstBuilder.withPredicate has cognitive complexity 63 (threshold 15). Drivers by points: if/else 23 (35 pts), match/switch 12 (24 pts), boolean chains 4 (nesting depth added 24). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JoinSelection.apply (cognitive 63) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:235— JoinSelection.apply has cognitive complexity 63 (threshold 15). Drivers by points: if/else 34 (53 pts), boolean chains 7, match/switch 2 (3 pts) (nesting depth added 20). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DataFrame.xs (cognitive 63) python/pyspark/pandas/frame.py:3588— DataFrame.xs has cognitive complexity 63 (threshold 15). Drivers by points: if/else 29 (50 pts), ternaries 3 (7 pts), error handling 2 (5 pts), boolean chains 1 (nesting depth added 28). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LocIndexerLike.__getitem__ (cognitive 63) python/pyspark/pandas/indexing.py:439— LocIndexerLike.__getitem__ has cognitive complexity 63 (threshold 15). Drivers by points: if/else 31 (49 pts), boolean chains 9, loops 1 (2 pts), ternaries 1 (2 pts), error handling 1 (nesting depth added 20). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InternalFrame.__init__ (cognitive 63) python/pyspark/pandas/internal.py:543— InternalFrame.__init__ has cognitive complexity 63 (threshold 15). Drivers by points: if/else 28 (41 pts), ternaries 6 (12 pts), boolean chains 10 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveReferences.doApply (cognitive 62) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1741— ResolveReferences.doApply has cognitive complexity 62 (threshold 15). Drivers by points: if/else 20 (33 pts), match/switch 8 (18 pts), boolean chains 11 (nesting depth added 23). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StarSchemaDetection.findStarJoins (cognitive 61) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/StarSchemaDetection.scala:73— StarSchemaDetection.findStarJoins has cognitive complexity 61 (threshold 15). Drivers by points: if/else 17 (38 pts), match/switch 6 (21 pts), boolean chains 2 (nesting depth added 36). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
MLHandler._handleMlCommand (cognitive 61) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:195— MLHandler._handleMlCommand has cognitive complexity 61 (threshold 15). Drivers by points: if/else 23 (33 pts), match/switch 8 (23 pts), error handling 1 (3 pts), boolean chains 2 (nesting depth added 27). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VectorizedRleValuesReader.readBatchRepeatedInternal (cognitive 60) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:437— VectorizedRleValuesReader.readBatchRepeatedInternal has cognitive complexity 60 (threshold 15). Drivers by points: if/else 18 (52 pts), loops 2 (4 pts), boolean chains 2, match/switch 1 (2 pts) (nesting depth added 37). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
ShreddingUtils.rebuild (cognitive 60) common/variant/src/main/java/org/apache/spark/types/variant/ShreddingUtils.java:69— ShreddingUtils.rebuild has cognitive complexity 60 (threshold 15). Drivers by points: if/else 22 (40 pts), loops 3 (10 pts), boolean chains 6, match/switch 1 (4 pts) (nesting depth added 28). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
pandasutils._assert_pandas_almost_equal (cognitive 60) python/pyspark/testing/pandasutils.py:127— pandasutils._assert_pandas_almost_equal has cognitive complexity 60 (threshold 15). Drivers by points: if/else 14 (37 pts), loops 8 (18 pts), boolean chains 5 (nesting depth added 33). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ValidateSubqueryExpression.apply (cognitive 59) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ValidateSubqueryExpression.scala:37— ValidateSubqueryExpression.apply has cognitive complexity 59 (threshold 15). Drivers by points: if/else 21 (35 pts), match/switch 10 (21 pts), boolean chains 3 (nesting depth added 25). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ParquetToSparkSchemaConverter.convertPrimitiveField (cognitive 59) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:243— ParquetToSparkSchemaConverter.convertPrimitiveField has cognitive complexity 59 (threshold 15). Drivers by points: if/else 20 (27 pts), match/switch 10 (22 pts), boolean chains 10 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkSQLCLIDriver.main (cognitive 59) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala:85— SparkSQLCLIDriver.main has cognitive complexity 59 (threshold 15). Drivers by points: if/else 27 (49 pts), error handling 3 (4 pts), boolean chains 3, loops 2, match/switch 1 (nesting depth added 23). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
OnHeapColumnVector.reserveInternal (cognitive 59) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/OnHeapColumnVector.java:596— OnHeapColumnVector.reserveInternal has cognitive complexity 59 (threshold 15). Drivers by points: if/else 27 (49 pts), boolean chains 10 (nesting depth added 22). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Resampler._bin_timestamp (cognitive 59) python/pyspark/pandas/resample.py:138— Resampler._bin_timestamp has cognitive complexity 59 (threshold 15). Drivers by points: if/else 23 (38 pts), boolean chains 15, ternaries 3 (6 pts) (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Master.receive (cognitive 58) core/src/main/scala/org/apache/spark/deploy/master/Master.scala:234— Master.receive has cognitive complexity 58 (threshold 15). Drivers by points: if/else 15 (31 pts), match/switch 7 (13 pts), loops 4 (12 pts), boolean chains 2 (nesting depth added 30). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
BLAS.gemmImpl (cognitive 58) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:523— BLAS.gemmImpl has cognitive complexity 58 (threshold 15). Drivers by points: loops 12 (48 pts), if/else 7 (10 pts) (nesting depth added 39). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
BLAS.gemm (cognitive 58) mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:436— BLAS.gemm has cognitive complexity 58 (threshold 15). Drivers by points: loops 12 (48 pts), if/else 7 (10 pts) (nesting depth added 39). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
DataflowGraphTransformer.transformDownNodes (cognitive 58) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DataflowGraphTransformer.scala:121— DataflowGraphTransformer.transformDownNodes has cognitive complexity 58 (threshold 15). Drivers by points: if/else 15 (31 pts), match/switch 2 (12 pts), error handling 4 (10 pts), loops 2 (3 pts), boolean chains 2 (nesting depth added 33). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
VariantGet.cast (cognitive 57) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:762— VariantGet.cast has cognitive complexity 57 (threshold 15). Drivers by points: if/else 22 (33 pts), loops 3 (9 pts), match/switch 4 (9 pts), error handling 2 (6 pts) (nesting depth added 26). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Master.receiveAndReply (cognitive 56) core/src/main/scala/org/apache/spark/deploy/master/Master.scala:417— Master.receiveAndReply has cognitive complexity 56 (threshold 15). Drivers by points: if/else 21 (46 pts), match/switch 4 (9 pts), boolean chains 1 (nesting depth added 30). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
JsonTableEvaluator.anyMatch (cognitive 56) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:880— JsonTableEvaluator.anyMatch has cognitive complexity 56 (threshold 15). Drivers by points: if/else 15 (34 pts), loops 3 (9 pts), match/switch 5 (9 pts), boolean chains 4 (nesting depth added 29). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
BufferSetterGetterUtils.createGetters (cognitive 56) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala:40— BufferSetterGetterUtils.createGetters has cognitive complexity 56 (threshold 15). Drivers by points: if/else 27 (53 pts), match/switch 1 (2 pts), loops 1 (nesting depth added 27). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
V2ScanRelationPushDown.rewriteAggregate (cognitive 56) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:632— V2ScanRelationPushDown.rewriteAggregate has cognitive complexity 56 (threshold 15). Drivers by points: if/else 17 (29 pts), match/switch 8 (26 pts), boolean chains 1 (nesting depth added 30). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
LocIndexer._select_cols_by_iterable (cognitive 56) python/pyspark/pandas/indexing.py:1346— LocIndexer._select_cols_by_iterable has cognitive complexity 56 (threshold 15). Drivers by points: if/else 18 (43 pts), loops 2 (5 pts), boolean chains 4, ternaries 1 (4 pts) (nesting depth added 31). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
PandasOnSparkBoxPlot.boxplot (cognitive 56) python/pyspark/pandas/plot/matplotlib.py:87— PandasOnSparkBoxPlot.boxplot has cognitive complexity 56 (threshold 15). Drivers by points: if/else 22 (49 pts), loops 2 (6 pts), boolean chains 1 (nesting depth added 31). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SummarizerBuffer.merge (cognitive 55) mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:590— SummarizerBuffer.merge has cognitive complexity 55 (threshold 15). Drivers by points: if/else 19 (51 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 33). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
ResolveWithCTE.resolveWithCTE (cognitive 55) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:47— ResolveWithCTE.resolveWithCTE has cognitive complexity 55 (threshold 15). Drivers by points: if/else 20 (46 pts), match/switch 4 (8 pts), boolean chains 1 (nesting depth added 30). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
HiveFunctionRegistryUtils.getMethodInternal (cognitive 55) sql/hive/src/main/java/org/apache/hadoop/hive/ql/exec/HiveFunctionRegistryUtils.java:85— HiveFunctionRegistryUtils.getMethodInternal has cognitive complexity 55 (threshold 15). Drivers by points: if/else 18 (38 pts), loops 4 (8 pts), ternaries 2 (7 pts), boolean chains 2 (nesting depth added 29). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
WritableColumnVector.appendObjects (cognitive 55) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java:806— WritableColumnVector.appendObjects has cognitive complexity 55 (threshold 15). Drivers by points: if/else 19 (32 pts), loops 9 (23 pts) (nesting depth added 27). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
types._parse_datatype_json_value (cognitive 55) python/pyspark/sql/types.py:2673— types._parse_datatype_json_value has cognitive complexity 55 (threshold 15). Drivers by points: if/else 21 (49 pts), boolean chains 5, ternaries 1 (nesting depth added 28). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SchemaConverters.structFieldFor (cognitive 54) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/utils/SchemaConverters.scala:74— SchemaConverters.structFieldFor has cognitive complexity 54 (threshold 15). Drivers by points: if/else 18 (27 pts), boolean chains 18, match/switch 5 (9 pts) (nesting depth added 13). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DeduplicateRelations.deduplicate (cognitive 54) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala:243— DeduplicateRelations.deduplicate has cognitive complexity 54 (threshold 15). Drivers by points: match/switch 6 (30 pts), if/else 11 (21 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 35). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
PropagateEmptyRelationBase.commonApplyFunc (cognitive 54) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PropagateEmptyRelation.scala:70— PropagateEmptyRelationBase.commonApplyFunc has cognitive complexity 54 (threshold 15). Drivers by points: if/else 19 (34 pts), boolean chains 12, match/switch 4 (8 pts) (nesting depth added 19). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DataFrame.pivot_table (cognitive 54) python/pyspark/pandas/frame.py:6941— DataFrame.pivot_table has cognitive complexity 54 (threshold 15). Drivers by points: if/else 27 (36 pts), boolean chains 8, ternaries 4 (8 pts), loops 1 (2 pts) (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkParserUtils.unescapeSQLString (cognitive 53) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkParserUtils.scala:41— SparkParserUtils.unescapeSQLString has cognitive complexity 53 (threshold 15). Drivers by points: if/else 17 (33 pts), boolean chains 16, loops 2 (3 pts), match/switch 1 (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
FilterEstimation.evaluateBinaryForTwoColumns (cognitive 53) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:700— FilterEstimation.evaluateBinaryForTwoColumns has cognitive complexity 53 (threshold 15). Drivers by points: if/else 17 (38 pts), boolean chains 10, match/switch 3 (5 pts) (nesting depth added 23). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.reset_index (cognitive 53) python/pyspark/pandas/frame.py:4471— DataFrame.reset_index has cognitive complexity 53 (threshold 15). Drivers by points: if/else 17 (30 pts), loops 4 (9 pts), ternaries 3 (8 pts), error handling 1 (4 pts), boolean chains 2 (nesting depth added 26). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
types._infer_type (cognitive 53) python/pyspark/sql/types.py:2993— types._infer_type has cognitive complexity 53 (threshold 15). Drivers by points: if/else 22 (39 pts), loops 3 (9 pts), boolean chains 3, error handling 1 (2 pts) (nesting depth added 24). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BufferSetterGetterUtils.createSetters (cognitive 52) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/udaf.scala:111— BufferSetterGetterUtils.createSetters has cognitive complexity 52 (threshold 15). Drivers by points: if/else 25 (49 pts), match/switch 1 (2 pts), loops 1 (nesting depth added 25). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CreateSQLFunctionCommand.run (cognitive 52) sql/core/src/main/scala/org/apache/spark/sql/execution/command/CreateSQLFunctionCommand.scala:71— CreateSQLFunctionCommand.run has cognitive complexity 52 (threshold 15). Drivers by points: if/else 26 (35 pts), match/switch 6 (14 pts), boolean chains 3 (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
EnsureRequirements.checkKeyGroupCompatible (cognitive 52) sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala:615— EnsureRequirements.checkKeyGroupCompatible has cognitive complexity 52 (threshold 15). Drivers by points: if/else 24 (33 pts), boolean chains 14, loops 2 (3 pts), match/switch 2 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StringSubstitutor.substitute (cognitive 51) common/utils/src/main/scala/org/apache/spark/StringSubstitutor.scala:44— StringSubstitutor.substitute has cognitive complexity 51 (threshold 15). Drivers by points: if/else 13 (39 pts), match/switch 2 (9 pts), boolean chains 2, loops 1 (nesting depth added 33). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
Writer.open (cognitive 51) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:920— Writer.open has cognitive complexity 51 (threshold 15). Drivers by points: if/else 21 (40 pts), error handling 2 (4 pts), match/switch 1 (3 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 24). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RelationResolution.tryResolvePersistent (cognitive 51) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala:241— RelationResolution.tryResolvePersistent has cognitive complexity 51 (threshold 15). Drivers by points: if/else 14 (23 pts), match/switch 4 (9 pts), error handling 2 (8 pts), boolean chains 6, loops 2 (5 pts) (nesting depth added 23). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StateStore.submitMaintenanceWorkForProvider (cognitive 51) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala:2037— StateStore.submitMaintenanceWorkForProvider has cognitive complexity 51 (threshold 15). Drivers by points: if/else 16 (35 pts), match/switch 6 (14 pts), boolean chains 1, error handling 1 (nesting depth added 27). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
TaskMemoryManager.allocatePageInternal (cognitive 51) core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:660— TaskMemoryManager.allocatePageInternal has cognitive complexity 51 (threshold 15). Drivers by points: if/else 18 (36 pts), ternaries 2 (9 pts), boolean chains 3, error handling 1 (2 pts), loops 1 (nesting depth added 26). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.describe (cognitive 51) python/pyspark/pandas/frame.py:10023— DataFrame.describe has cognitive complexity 51 (threshold 15). Drivers by points: if/else 14 (24 pts), loops 6 (13 pts), ternaries 3 (10 pts), boolean chains 4 (nesting depth added 24). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Client.prepareLocalResources (cognitive 50) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:527— Client.prepareLocalResources has cognitive complexity 50 (threshold 15). Drivers by points: if/else 29 (43 pts), match/switch 3 (5 pts), boolean chains 2 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LateMaterializationParquetRecordReader.evaluateStorageFilter (cognitive 50) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:707— LateMaterializationParquetRecordReader.evaluateStorageFilter has cognitive complexity 50 (threshold 15). Drivers by points: if/else 11 (34 pts), error handling 3 (9 pts), loops 4 (6 pts), boolean chains 1 (nesting depth added 31). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JdbcUtils.savePartition (cognitive 49) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:653— JdbcUtils.savePartition has cognitive complexity 49 (threshold 15). Drivers by points: if/else 20 (32 pts), error handling 6 (13 pts), loops 2 (3 pts), boolean chains 1 (nesting depth added 20). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
XSDToSchema.getStructField (cognitive 49) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:70— XSDToSchema.getStructField has cognitive complexity 49 (threshold 15). Drivers by points: match/switch 13 (40 pts), if/else 9 (nesting depth added 27). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
InProcessArrowEvalPythonEvaluatorFactory.evaluateBatches (cognitive 49) sql/core/src/main/scala/org/apache/spark/sql/execution/python/InProcessArrowEvalPythonEvaluatorFactory.scala:115— InProcessArrowEvalPythonEvaluatorFactory.evaluateBatches has cognitive complexity 49 (threshold 15). Drivers by points: if/else 33, boolean chains 9, error handling 3, match/switch 2 (3 pts), loops 1 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
TransportResponseHandler.handle (cognitive 49) common/network-common/src/main/java/org/apache/spark/network/client/TransportResponseHandler.java:166— TransportResponseHandler.handle has cognitive complexity 49 (threshold 15). Drivers by points: if/else 24 (33 pts), error handling 4 (16 pts) (nesting depth added 21). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.dropna (cognitive 49) python/pyspark/pandas/frame.py:6098— DataFrame.dropna has cognitive complexity 49 (threshold 15). Drivers by points: if/else 23 (43 pts), boolean chains 3, ternaries 1 (3 pts) (nesting depth added 22). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DAGScheduler.submitMissingTasks (cognitive 48) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:3215— DAGScheduler.submitMissingTasks has cognitive complexity 48 (threshold 15). Drivers by points: if/else 15 (25 pts), match/switch 10 (14 pts), boolean chains 5, error handling 4 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Word2Vec.doFit (cognitive 48) mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala:333— Word2Vec.doFit has cognitive complexity 48 (threshold 15). Drivers by points: if/else 12 (32 pts), loops 5 (14 pts), boolean chains 2 (nesting depth added 29). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkIntervalUtils.toDayTimeIntervalString (cognitive 48) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkIntervalUtils.scala:435— SparkIntervalUtils.toDayTimeIntervalString has cognitive complexity 48 (threshold 15). Drivers by points: if/else 14 (25 pts), match/switch 6 (20 pts), boolean chains 3 (nesting depth added 25). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RewriteWithExpression.rewriteWithExprAndInputPlans (cognitive 48) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteWithExpression.scala:378— RewriteWithExpression.rewriteWithExprAndInputPlans has cognitive complexity 48 (threshold 15). Drivers by points: if/else 15 (43 pts), match/switch 3 (5 pts) (nesting depth added 30). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
HiveInspectors.wrapperFor (cognitive 48) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:272— HiveInspectors.wrapperFor has cognitive complexity 48 (threshold 15). Drivers by points: match/switch 13 (42 pts), if/else 6 (nesting depth added 29). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SortState.mergeHi (cognitive 48) core/src/main/java/org/apache/spark/util/collection/TimSort.java:806— SortState.mergeHi has cognitive complexity 48 (threshold 15). Drivers by points: if/else 16 (39 pts), loops 3 (5 pts), boolean chains 3, ternaries 1 (nesting depth added 25). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (SortState.mergeLo) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
SortState.mergeLo (cognitive 48) core/src/main/java/org/apache/spark/util/collection/TimSort.java:686— SortState.mergeLo has cognitive complexity 48 (threshold 15). Drivers by points: if/else 16 (39 pts), loops 3 (5 pts), boolean chains 3, ternaries 1 (nesting depth added 25). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (SortState.mergeHi) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
UserDefinedFunction._check_return_type (cognitive 48) python/pyspark/sql/udf.py:334— UserDefinedFunction._check_return_type has cognitive complexity 48 (threshold 15). Drivers by points: error handling 10 (25 pts), if/else 12 (18 pts), boolean chains 5 (nesting depth added 21). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LogisticRegression.train (cognitive 47) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:507— LogisticRegression.train has cognitive complexity 47 (threshold 15). Drivers by points: if/else 29 (34 pts), boolean chains 11, match/switch 2 (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
BindParameters.apply (cognitive 47) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/parameters.scala:202— BindParameters.apply has cognitive complexity 47 (threshold 15). Drivers by points: if/else 11 (21 pts), match/switch 8 (19 pts), boolean chains 7 (nesting depth added 21). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
FileScanRDD.compute (cognitive 47) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala:105— FileScanRDD.compute has cognitive complexity 47 (threshold 15). Drivers by points: if/else 15 (21 pts), error handling 5 (13 pts), match/switch 5 (8 pts), boolean chains 5 (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkContext._do_init (cognitive 47) python/pyspark/core/context.py:225— SparkContext._do_init has cognitive complexity 47 (threshold 15). Drivers by points: if/else 22 (30 pts), loops 6 (9 pts), boolean chains 5, error handling 1 (3 pts) (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AdaptiveSparkPlanExec.withFinalPlanUpdate (cognitive 46) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:343— AdaptiveSparkPlanExec.withFinalPlanUpdate has cognitive complexity 46 (threshold 15). Drivers by points: if/else 11 (27 pts), match/switch 4 (13 pts), error handling 1 (3 pts), boolean chains 2, loops 1 (nesting depth added 27). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TungstenAggregationIterator.processInputs (cognitive 46) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala:208— TungstenAggregationIterator.processInputs has cognitive complexity 46 (threshold 15). Drivers by points: if/else 14 (39 pts), loops 2 (4 pts), boolean chains 3 (nesting depth added 27). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
typehints.as_spark_type (cognitive 46) python/pyspark/pandas/typedef/typehints.py:141— typehints.as_spark_type has cognitive complexity 46 (threshold 15). Drivers by points: boolean chains 26, if/else 10 (18 pts), ternaries 1 (2 pts) (nesting depth added 9). To reduce it, name the conditions: bind each compound test to a well-named local or a small predicate function, so the body reads as a sequence of named decisions rather than a chain of operators.
RBackendHandler.findMatchedSignature (cognitive 45) core/src/main/scala/org/apache/spark/api/r/RBackendHandler.scala:223— RBackendHandler.findMatchedSignature has cognitive complexity 45 (threshold 15). Drivers by points: if/else 8 (26 pts), loops 3 (8 pts), match/switch 1 (6 pts), boolean chains 5 (nesting depth added 28). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataSourceV2Relation.transformV2Stats (cognitive 45) sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:628— DataSourceV2Relation.transformV2Stats has cognitive complexity 45 (threshold 15). Drivers by points: if/else 25 (33 pts), match/switch 4 (12 pts) (nesting depth added 16). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
OrcSerializer.newConverter (cognitive 45) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcSerializer.scala:63— OrcSerializer.newConverter has cognitive complexity 45 (threshold 15). Drivers by points: if/else 22 (37 pts), loops 3 (7 pts), match/switch 1 (nesting depth added 19). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
UTF8String.indexOf (cognitive 45) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1320— UTF8String.indexOf has cognitive complexity 45 (threshold 15). Drivers by points: if/else 13 (28 pts), loops 6 (14 pts), boolean chains 3 (nesting depth added 23). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PandasOnSparkTestUtils.assert_eq (cognitive 45) python/pyspark/testing/pandasutils.py:384— PandasOnSparkTestUtils.assert_eq has cognitive complexity 45 (threshold 15). Drivers by points: if/else 17 (35 pts), boolean chains 8, loops 1 (2 pts) (nesting depth added 19). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SparkContext.addJar (cognitive 44) core/src/main/scala/org/apache/spark/SparkContext.scala:2478— SparkContext.addJar has cognitive complexity 44 (threshold 15). Drivers by points: if/else 20 (36 pts), error handling 2 (3 pts), match/switch 1 (3 pts), boolean chains 2 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskRunner.run (cognitive 44) core/src/main/scala/org/apache/spark/executor/Executor.scala:815— TaskRunner.run has cognitive complexity 44 (threshold 15). Drivers by points: if/else 24 (29 pts), error handling 7 (9 pts), boolean chains 5, match/switch 1 (nesting depth added 7). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
DetectAmbiguousSelfJoin.apply (cognitive 44) sql/core/src/main/scala/org/apache/spark/sql/execution/analysis/DetectAmbiguousSelfJoin.scala:77— DetectAmbiguousSelfJoin.apply has cognitive complexity 44 (threshold 15). Drivers by points: if/else 15 (32 pts), match/switch 4 (8 pts), boolean chains 4 (nesting depth added 21). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
LogisticGradient.compute (cognitive 43) mllib/src/main/scala/org/apache/spark/mllib/optimization/Gradient.scala:165— LogisticGradient.compute has cognitive complexity 43 (threshold 15). Drivers by points: if/else 17 (28 pts), loops 3 (8 pts), match/switch 3 (5 pts), boolean chains 2 (nesting depth added 18). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveReferences.expandStarExpression (cognitive 43) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2205— ResolveReferences.expandStarExpression has cognitive complexity 43 (threshold 15). Drivers by points: match/switch 10 (25 pts), if/else 11 (16 pts), boolean chains 2 (nesting depth added 20). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
UnivocityGenerator.makeConverter (cognitive 43) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityGenerator.scala:79— UnivocityGenerator.makeConverter has cognitive complexity 43 (threshold 15). Drivers by points: if/else 15 (33 pts), loops 3 (9 pts), match/switch 1 (nesting depth added 24). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
UnwrapCastInBinaryComparison.simplifyNumericComparison (cognitive 43) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala:201— UnwrapCastInBinaryComparison.simplifyNumericComparison has cognitive complexity 43 (threshold 15). Drivers by points: match/switch 7 (22 pts), if/else 17 (20 pts), boolean chains 1 (nesting depth added 18). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
RuleExecutor.execute (cognitive 43) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala:214— RuleExecutor.execute has cognitive complexity 43 (threshold 15). Drivers by points: if/else 15 (33 pts), match/switch 2 (6 pts), boolean chains 3, loops 1 (nesting depth added 22). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ToNumberParser.parse (cognitive 43) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala:401— ToNumberParser.parse has cognitive complexity 43 (threshold 15). Drivers by points: if/else 13 (20 pts), match/switch 7 (20 pts), boolean chains 2, loops 1 (nesting depth added 20). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkPlan.executeTake (cognitive 43) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala:525— SparkPlan.executeTake has cognitive complexity 43 (threshold 15). Drivers by points: if/else 16 (28 pts), loops 4 (12 pts), boolean chains 3 (nesting depth added 20). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
EvaluatePython.makeFromJavaDefault (cognitive 43) sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala:192— EvaluatePython.makeFromJavaDefault has cognitive complexity 43 (threshold 15). Drivers by points: match/switch 18 (35 pts), if/else 2 (4 pts), loops 1 (3 pts), boolean chains 1 (nesting depth added 21). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ShuffleMigrationRunnable.run (cognitive 42) core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala:103— ShuffleMigrationRunnable.run has cognitive complexity 42 (threshold 15). Drivers by points: if/else 15 (27 pts), error handling 4 (10 pts), boolean chains 4, loops 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CodegenContext.collectInputs (cognitive 42) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:1318— CodegenContext.collectInputs has cognitive complexity 42 (threshold 15). Drivers by points: if/else 7 (18 pts), boolean chains 14, match/switch 4 (9 pts), loops 1 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
types.proto_schema_to_pyspark_data_type (cognitive 42) python/pyspark/sql/connect/types.py:226— types.proto_schema_to_pyspark_data_type has cognitive complexity 42 (threshold 15). Drivers by points: ternaries 12 (24 pts), if/else 10 (16 pts), loops 1 (2 pts) (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UnsafeRowUtils.validateStructuralIntegrityWithReasonImpl (cognitive 41) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/UnsafeRowUtils.scala:80— UnsafeRowUtils.validateStructuralIntegrityWithReasonImpl has cognitive complexity 41 (threshold 15). Drivers by points: if/else 14 (29 pts), boolean chains 7, match/switch 3 (5 pts) (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
XmlTokenizer.readUntilEndElement (cognitive 41) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1067— XmlTokenizer.readUntilEndElement has cognitive complexity 41 (threshold 15). Drivers by points: if/else 19 (37 pts), boolean chains 3, loops 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InsertIntoHadoopFsRelationCommand.run (cognitive 41) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoHadoopFsRelationCommand.scala:83— InsertIntoHadoopFsRelationCommand.run has cognitive complexity 41 (threshold 15). Drivers by points: if/else 21 (35 pts), boolean chains 4, match/switch 1 (2 pts) (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PreprocessTableCreation.apply (cognitive 41) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala:141— PreprocessTableCreation.apply has cognitive complexity 41 (threshold 15). Drivers by points: if/else 19 (34 pts), match/switch 3 (6 pts), boolean chains 1 (nesting depth added 18). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AvroStateEncoder.encodePrefixKeyForRangeScan (cognitive 41) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:1063— AvroStateEncoder.encodePrefixKeyForRangeScan has cognitive complexity 41 (threshold 15). Drivers by points: if/else 21 (39 pts), match/switch 1 (2 pts) (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Builder.getOrCreate (cognitive 41) python/pyspark/sql/session.py:435— Builder.getOrCreate has cognitive complexity 41 (threshold 15). Drivers by points: if/else 13 (28 pts), boolean chains 7, loops 2 (6 pts) (nesting depth added 19). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SummarizerBuffer.add (cognitive 40) mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala:499— SummarizerBuffer.add has cognitive complexity 40 (threshold 15). Drivers by points: if/else 19 (38 pts), boolean chains 2 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowColumnVector.initAccessor (cognitive 40) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ArrowColumnVector.java:192— ArrowColumnVector.initAccessor has cognitive complexity 40 (threshold 15). Drivers by points: if/else 34 (36 pts), loops 1 (3 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
typehints._new_type_holders (cognitive 40) python/pyspark/pandas/typedef/typehints.py:855— typehints._new_type_holders has cognitive complexity 40 (threshold 15). Drivers by points: if/else 13 (23 pts), ternaries 2 (10 pts), loops 2 (4 pts), boolean chains 3 (nesting depth added 20). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
plan_data_source_read.records_to_arrow_batches (cognitive 40) python/pyspark/sql/worker/plan_data_source_read.py:55— plan_data_source_read.records_to_arrow_batches has cognitive complexity 40 (threshold 15). Drivers by points: if/else 9 (22 pts), loops 6 (15 pts), boolean chains 2, error handling 1 (nesting depth added 22). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
IndylambdaScalaClosures.findAccessedFields (cognitive 39) common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:916— IndylambdaScalaClosures.findAccessedFields has cognitive complexity 39 (threshold 15). Drivers by points: if/else 14 (22 pts), loops 7 (14 pts), boolean chains 3 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Worker.receive (cognitive 39) core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala:534— Worker.receive has cognitive complexity 39 (threshold 15). Drivers by points: if/else 11 (23 pts), error handling 3 (8 pts), match/switch 3 (6 pts), boolean chains 2 (nesting depth added 20). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveLateralColumnAliasReference.apply0 (cognitive 39) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveLateralColumnAliasReference.scala:139— ResolveLateralColumnAliasReference.apply0 has cognitive complexity 39 (threshold 15). Drivers by points: match/switch 7 (18 pts), if/else 10 (17 pts), boolean chains 4 (nesting depth added 18). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
InlineCTE.inlineCTE (cognitive 39) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:238— InlineCTE.inlineCTE has cognitive complexity 39 (threshold 15). Drivers by points: if/else 17 (34 pts), match/switch 3 (5 pts) (nesting depth added 19). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DataTypeUtils.canWrite (cognitive 39) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/DataTypeUtils.scala:108— DataTypeUtils.canWrite has cognitive complexity 39 (threshold 15). Drivers by points: if/else 16 (24 pts), boolean chains 12, match/switch 2 (3 pts) (nesting depth added 9). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StringUtils.splitSemiColon (cognitive 39) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala:286— StringUtils.splitSemiColon has cognitive complexity 39 (threshold 15). Drivers by points: if/else 17 (24 pts), boolean chains 11, loops 4 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AvroSerializer.newConverter (cognitive 39) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroSerializer.scala:102— AvroSerializer.newConverter has cognitive complexity 39 (threshold 15). Drivers by points: if/else 14 (22 pts), match/switch 6 (11 pts), loops 2 (4 pts), boolean chains 2 (nesting depth added 15). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DataFrameWriter.saveCommand (cognitive 39) sql/core/src/main/scala/org/apache/spark/sql/classic/DataFrameWriter.scala:131— DataFrameWriter.saveCommand has cognitive complexity 39 (threshold 15). Drivers by points: if/else 16 (28 pts), match/switch 3 (8 pts), boolean chains 3 (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UnionLoopExec.doExecute (cognitive 39) sql/core/src/main/scala/org/apache/spark/sql/execution/UnionLoopExec.scala:155— UnionLoopExec.doExecute has cognitive complexity 39 (threshold 15). Drivers by points: if/else 13 (23 pts), match/switch 4 (12 pts), boolean chains 3, loops 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HashAggregateExec.doConsumeWithKeys (cognitive 39) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:925— HashAggregateExec.doConsumeWithKeys has cognitive complexity 39 (threshold 15). Drivers by points: if/else 31 (37 pts), match/switch 2 (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SparkShreddingUtils.extractField (cognitive 39) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala:788— SparkShreddingUtils.extractField has cognitive complexity 39 (threshold 15). Drivers by points: if/else 11 (27 pts), boolean chains 4, loops 2 (4 pts), match/switch 1 (4 pts) (nesting depth added 21). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Shim_v2_0.convertFilters (cognitive 39) sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala:668— Shim_v2_0.convertFilters has cognitive complexity 39 (threshold 15). Drivers by points: if/else 17 (18 pts), boolean chains 8, match/switch 8, loops 2 (5 pts) (nesting depth added 4). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
connect._convert_exception (cognitive 39) python/pyspark/errors/exceptions/connect.py:172— connect._convert_exception has cognitive complexity 39 (threshold 15). Drivers by points: if/else 9 (19 pts), ternaries 4 (13 pts), loops 2 (4 pts), boolean chains 3 (nesting depth added 21). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SparkContext.holdExecutors (cognitive 38) core/src/main/scala/org/apache/spark/SparkContext.scala:2177— SparkContext.holdExecutors has cognitive complexity 38 (threshold 15). Drivers by points: if/else 12 (25 pts), match/switch 4 (13 pts) (nesting depth added 22). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
NewHadoopRDD.compute (cognitive 38) core/src/main/scala/org/apache/spark/rdd/NewHadoopRDD.scala:198— NewHadoopRDD.compute has cognitive complexity 38 (threshold 15). Drivers by points: error handling 10 (17 pts), if/else 12 (16 pts), match/switch 3, boolean chains 2 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveFieldNameAndPosition.apply (cognitive 38) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4280— ResolveFieldNameAndPosition.apply has cognitive complexity 38 (threshold 15). Drivers by points: match/switch 10 (24 pts), if/else 9 (11 pts), boolean chains 3 (nesting depth added 16). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
CatalogTable.toJsonLinkedHashMap (cognitive 38) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala:642— CatalogTable.toJsonLinkedHashMap has cognitive complexity 38 (threshold 15). Drivers by points: if/else 29 (37 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
StaxXmlParser.convertObject (cognitive 38) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:622— StaxXmlParser.convertObject has cognitive complexity 38 (threshold 15). Drivers by points: match/switch 5 (19 pts), if/else 7 (11 pts), error handling 2 (6 pts), loops 2 (nesting depth added 22). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
MicroBatchExecution.runBatch (cognitive 38) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala:1192— MicroBatchExecution.runBatch has cognitive complexity 38 (threshold 15). Drivers by points: if/else 14 (22 pts), match/switch 11 (16 pts) (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CommandBuilderUtils.parseOptionString (cognitive 38) launcher/src/main/java/org/apache/spark/launcher/CommandBuilderUtils.java:129— CommandBuilderUtils.parseOptionString has cognitive complexity 38 (threshold 15). Drivers by points: if/else 13 (29 pts), match/switch 2 (6 pts), boolean chains 2, loops 1 (nesting depth added 20). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RocksDBProvider.initRockDB (cognitive 38) common/network-common/src/main/java/org/apache/spark/network/util/RocksDBProvider.java:45— RocksDBProvider.initRockDB has cognitive complexity 38 (threshold 15). Drivers by points: if/else 6 (19 pts), error handling 5 (14 pts), loops 1 (5 pts) (nesting depth added 26). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
merge_spark_pr.resolve_jira_issue (cognitive 38) REDACTED:1484— merge_spark_pr.resolve_jira_issue has cognitive complexity 38 (threshold 15). Drivers by points: if/else 18 (28 pts), error handling 2 (4 pts), boolean chains 3, loops 2, ternaries 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LocIndexer._get_from_multiindex_column (cognitive 38) python/pyspark/pandas/indexing.py:1246— LocIndexer._get_from_multiindex_column has cognitive complexity 38 (threshold 15). Drivers by points: if/else 12 (22 pts), ternaries 3 (9 pts), boolean chains 6, loops 1 (nesting depth added 16). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
ShuffleWriteProcessor.write (cognitive 37) core/src/main/scala/org/apache/spark/shuffle/ShuffleWriteProcessor.scala:44— ShuffleWriteProcessor.write has cognitive complexity 37 (threshold 15). Drivers by points: if/else 11 (25 pts), match/switch 2 (7 pts), error handling 2 (3 pts), boolean chains 2 (nesting depth added 20). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
GeneralizedLinearAlgorithm.run (cognitive 37) mllib/src/main/scala/org/apache/spark/mllib/regression/GeneralizedLinearAlgorithm.scala:238— GeneralizedLinearAlgorithm.run has cognitive complexity 37 (threshold 15). Drivers by points: if/else 24 (30 pts), boolean chains 4, loops 1 (3 pts) (nesting depth added 8). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SeptenaryExpression.eval (cognitive 37) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Expression.scala:1216— SeptenaryExpression.eval has cognitive complexity 37 (threshold 15). Drivers by points: if/else 9 (37 pts) (nesting depth added 28). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
RewriteDistinctAggregates.rewrite (cognitive 37) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteDistinctAggregates.scala:259— RewriteDistinctAggregates.rewrite has cognitive complexity 37 (threshold 15). Drivers by points: if/else 15 (19 pts), match/switch 7 (14 pts), boolean chains 4 (nesting depth added 11). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
FilterEstimation.evaluateBinaryForNumeric (cognitive 37) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:501— FilterEstimation.evaluateBinaryForNumeric has cognitive complexity 37 (threshold 15). Drivers by points: if/else 15 (29 pts), match/switch 3 (7 pts), boolean chains 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PredicateUtils.toV1 (cognitive 37) sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/PredicateUtils.scala:29— PredicateUtils.toV1 has cognitive complexity 37 (threshold 15). Drivers by points: if/else 15 (24 pts), match/switch 5 (9 pts), boolean chains 4 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ErrorUtils.throwableToProtoErrors (cognitive 37) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/utils/ErrorUtils.scala:78— ErrorUtils.throwableToProtoErrors has cognitive complexity 37 (threshold 15). Drivers by points: if/else 11 (29 pts), loops 2 (4 pts), boolean chains 2, match/switch 1 (2 pts) (nesting depth added 21). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HashAggregateExec.doProduceWithKeys (cognitive 37) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/HashAggregateExec.scala:560— HashAggregateExec.doProduceWithKeys has cognitive complexity 37 (threshold 15). Drivers by points: if/else 33 (36 pts), match/switch 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
XSDToSchema.getStructFieldsFromParticle (cognitive 37) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala:198— XSDToSchema.getStructFieldsFromParticle has cognitive complexity 37 (threshold 15). Drivers by points: if/else 16 (30 pts), match/switch 4 (7 pts) (nesting depth added 17). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
RocksDBStateStoreChangeDataReader.getNext (cognitive 37) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:1676— RocksDBStateStoreChangeDataReader.getNext has cognitive complexity 37 (threshold 15). Drivers by points: if/else 17 (30 pts), match/switch 1 (3 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VectorizedColumnReader.readBatch (cognitive 37) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java:257— VectorizedColumnReader.readBatch has cognitive complexity 37 (threshold 15). Drivers by points: if/else 13 (28 pts), boolean chains 6, error handling 1 (2 pts), loops 1 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LevelDBProvider.initLevelDB (cognitive 37) common/network-common/src/main/java/org/apache/spark/network/util/LevelDBProvider.java:42— LevelDBProvider.initLevelDB has cognitive complexity 37 (threshold 15). Drivers by points: if/else 6 (19 pts), error handling 4 (12 pts), loops 1 (5 pts), boolean chains 1 (nesting depth added 25). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
ArrowBatchedUDFArrowHandler.run (cognitive 37) python/pyspark/eval_handlers/_arrow.py:865— ArrowBatchedUDFArrowHandler.run has cognitive complexity 37 (threshold 15). Drivers by points: if/else 10 (22 pts), ternaries 3 (8 pts), error handling 1 (3 pts), loops 2 (3 pts), boolean chains 1 (nesting depth added 20). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConversionMixin._convert_from_pandas (cognitive 37) python/pyspark/sql/pandas/conversion.py:703— SparkConversionMixin._convert_from_pandas has cognitive complexity 37 (threshold 15). Drivers by points: if/else 12 (31 pts), loops 2 (5 pts), boolean chains 1 (nesting depth added 22). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
ExecutorPodsAllocator.onNewSnapshots (cognitive 36) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsAllocator.scala:171— ExecutorPodsAllocator.onNewSnapshots has cognitive complexity 36 (threshold 15). Drivers by points: if/else 24 (29 pts), boolean chains 5, match/switch 2 (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ResolveFunctions.apply (cognitive 36) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2483— ResolveFunctions.apply has cognitive complexity 36 (threshold 15). Drivers by points: match/switch 9 (21 pts), if/else 9 (13 pts), boolean chains 2 (nesting depth added 16). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveSQLFunctions.rewrite (cognitive 36) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2979— ResolveSQLFunctions.rewrite has cognitive complexity 36 (threshold 15). Drivers by points: if/else 12 (18 pts), match/switch 5 (11 pts), boolean chains 7 (nesting depth added 12). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ArrayInsert.nullSafeEval (cognitive 36) REDACTED:5652— ArrayInsert.nullSafeEval has cognitive complexity 36 (threshold 15). Drivers by points: if/else 18 (35 pts), boolean chains 1 (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InjectRuntimeFilter.extractSelectiveFilterOverScan (cognitive 36) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:103— InjectRuntimeFilter.extractSelectiveFilterOverScan has cognitive complexity 36 (threshold 15). Drivers by points: if/else 17 (26 pts), boolean chains 7, match/switch 2 (3 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SimplifyConditionals.apply (cognitive 36) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:710— SimplifyConditionals.apply has cognitive complexity 36 (threshold 15). Drivers by points: if/else 13 (28 pts), loops 1 (3 pts), match/switch 2 (3 pts), boolean chains 2 (nesting depth added 18). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AggregationIterator.generateProcessRow (cognitive 36) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregationIterator.scala:156— AggregationIterator.generateProcessRow has cognitive complexity 36 (threshold 15). Drivers by points: if/else 9 (19 pts), match/switch 6 (15 pts), loops 1 (2 pts) (nesting depth added 20). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DescribeColumnExec.run (cognitive 36) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeColumnExec.scala:36— DescribeColumnExec.run has cognitive complexity 36 (threshold 15). Drivers by points: if/else 18 (31 pts), match/switch 2 (5 pts) (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StateDataSource.getStoreMetadataAndRunChecks (cognitive 36) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala:297— StateDataSource.getStoreMetadataAndRunChecks has cognitive complexity 36 (threshold 15). Drivers by points: if/else 15 (32 pts), boolean chains 2, error handling 1 (2 pts) (nesting depth added 18). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SparkSubmitCommandBuilder.buildSparkSubmitArgs (cognitive 36) launcher/src/main/java/org/apache/spark/launcher/SparkSubmitCommandBuilder.java:198— SparkSubmitCommandBuilder.buildSparkSubmitArgs has cognitive complexity 36 (threshold 15). Drivers by points: if/else 23 (26 pts), boolean chains 7, loops 2 (3 pts) (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
VariantBuilder.appendWithSetImpl (cognitive 36) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:730— VariantBuilder.appendWithSetImpl has cognitive complexity 36 (threshold 15). Drivers by points: if/else 17 (24 pts), loops 4 (8 pts), boolean chains 4 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GroupBy.aggregate (cognitive 36) python/pyspark/pandas/groupby.py:176— GroupBy.aggregate has cognitive complexity 36 (threshold 15). Drivers by points: if/else 17 (29 pts), boolean chains 7 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Logging.initializeLogging (cognitive 35) common/utils/src/main/scala/org/apache/spark/internal/Logging.scala:343— Logging.initializeLogging has cognitive complexity 35 (threshold 15). Drivers by points: if/else 12 (27 pts), match/switch 2 (7 pts), boolean chains 1 (nesting depth added 20). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
KMeans.trainWithBlock (cognitive 35) mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala:526— KMeans.trainWithBlock has cognitive complexity 35 (threshold 15). Drivers by points: if/else 16 (25 pts), match/switch 4 (7 pts), boolean chains 2, loops 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MultinomialLogisticBlockAggregator.add (cognitive 35) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregator.scala:103— MultinomialLogisticBlockAggregator.add has cognitive complexity 35 (threshold 15). Drivers by points: loops 7 (17 pts), if/else 11 (16 pts), boolean chains 1, match/switch 1 (nesting depth added 15). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
Client.monitorApplication (cognitive 35) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:1210— Client.monitorApplication has cognitive complexity 35 (threshold 15). Drivers by points: if/else 10 (19 pts), match/switch 2 (7 pts), boolean chains 4, error handling 2 (4 pts), loops 1 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Row.equals (cognitive 35) sql/api/src/main/scala/org/apache/spark/sql/Row.scala:478— Row.equals has cognitive complexity 35 (threshold 15). Drivers by points: if/else 11 (28 pts), boolean chains 3, match/switch 1 (3 pts), loops 1 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AttributeNameParser.parseAttributeName (cognitive 35) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/AttributeNameParser.scala:28— AttributeNameParser.parseAttributeName has cognitive complexity 35 (threshold 15). Drivers by points: if/else 13 (31 pts), boolean chains 3, loops 1 (nesting depth added 18). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
TableOutputResolver.reorderColumnsByName (cognitive 35) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:382— TableOutputResolver.reorderColumnsByName has cognitive complexity 35 (threshold 15). Drivers by points: if/else 21 (30 pts), match/switch 2 (4 pts), boolean chains 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JsonInferSchema.inferField (cognitive 35) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:155— JsonInferSchema.inferField has cognitive complexity 35 (threshold 15). Drivers by points: if/else 15 (25 pts), loops 2 (4 pts), boolean chains 3, match/switch 2 (3 pts) (nesting depth added 13). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkConnectPlanner.transformReadRel (cognitive 35) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:1790— SparkConnectPlanner.transformReadRel has cognitive complexity 35 (threshold 15). Drivers by points: if/else 15 (28 pts), boolean chains 4, match/switch 2 (3 pts) (nesting depth added 14). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DataSource.lookupDataSource (cognitive 35) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala:687— DataSource.lookupDataSource has cognitive complexity 35 (threshold 15). Drivers by points: if/else 16 (23 pts), boolean chains 4, error handling 3 (4 pts), match/switch 3 (4 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ParquetFilters.createFilterHelper (cognitive 35) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:1093— ParquetFilters.createFilterHelper has cognitive complexity 35 (threshold 15). Drivers by points: boolean chains 14, if/else 9 (13 pts), loops 2 (5 pts), match/switch 2 (3 pts) (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StateStoreCoordinator.receiveAndReply (cognitive 35) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreCoordinator.scala:317— StateStoreCoordinator.receiveAndReply has cognitive complexity 35 (threshold 15). Drivers by points: if/else 9 (20 pts), match/switch 5 (12 pts), error handling 1 (3 pts) (nesting depth added 20). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
LateMaterializationParquetRecordReader.loadNextRowGroup (cognitive 35) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:511— LateMaterializationParquetRecordReader.loadNextRowGroup has cognitive complexity 35 (threshold 15). Drivers by points: if/else 7 (15 pts), ternaries 6 (15 pts), boolean chains 2, error handling 1 (2 pts), loops 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VariantBuilder.appendWithInsertionImpl (cognitive 35) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:627— VariantBuilder.appendWithInsertionImpl has cognitive complexity 35 (threshold 15). Drivers by points: if/else 14 (20 pts), loops 6 (12 pts), boolean chains 3 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
merge_spark_pr.get_input (cognitive 35) REDACTED:541— merge_spark_pr.get_input has cognitive complexity 35 (threshold 15). Drivers by points: if/else 13 (29 pts), loops 3 (6 pts) (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
typehints.infer_return_type (cognitive 35) python/pyspark/pandas/typedef/typehints.py:429— typehints.infer_return_type has cognitive complexity 35 (threshold 15). Drivers by points: if/else 10 (14 pts), ternaries 4 (11 pts), boolean chains 5, loops 2 (5 pts) (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CatalystTranspiler._category_uncached (cognitive 35) python/pyspark/sql/transpile.py:471— CatalystTranspiler._category_uncached has cognitive complexity 35 (threshold 15). Drivers by points: if/else 15 (29 pts), boolean chains 3, ternaries 1 (2 pts), match/switch 1 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DAGScheduler.liveReduceSet (cognitive 34) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2201— DAGScheduler.liveReduceSet has cognitive complexity 34 (threshold 15). Drivers by points: if/else 9 (18 pts), match/switch 5 (14 pts), boolean chains 1, loops 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExternalSorter.writePartitionedMapOutput (cognitive 34) core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala:709— ExternalSorter.writePartitionedMapOutput has cognitive complexity 34 (threshold 15). Drivers by points: if/else 11 (22 pts), loops 4 (11 pts), boolean chains 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
YarnAllocator.processCompletedContainers (cognitive 34) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:848— YarnAllocator.processCompletedContainers has cognitive complexity 34 (threshold 15). Drivers by points: if/else 10 (20 pts), match/switch 3 (8 pts), loops 3 (6 pts) (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveRelations.lookupTableOrView (cognitive 34) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1368— ResolveRelations.lookupTableOrView has cognitive complexity 34 (threshold 15). Drivers by points: match/switch 5 (13 pts), if/else 8 (11 pts), error handling 2 (7 pts), boolean chains 3 (nesting depth added 16). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveInsertInto.apply (cognitive 34) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1466— ResolveInsertInto.apply has cognitive complexity 34 (threshold 15). Drivers by points: if/else 13 (25 pts), boolean chains 5, match/switch 2 (4 pts) (nesting depth added 14). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
VariantValueConverter.convertInput (cognitive 34) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala:480— VariantValueConverter.convertInput has cognitive complexity 34 (threshold 15). Drivers by points: if/else 18 (24 pts), error handling 5 (6 pts), boolean chains 3, match/switch 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InterpretedHashFunction.hash (cognitive 34) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala:707— InterpretedHashFunction.hash has cognitive complexity 34 (threshold 15). Drivers by points: if/else 12 (20 pts), match/switch 4 (7 pts), loops 3 (6 pts), boolean chains 1 (nesting depth added 14). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
FileSourceStrategy.apply (cognitive 34) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala:203— FileSourceStrategy.apply has cognitive complexity 34 (threshold 15). Drivers by points: if/else 12 (19 pts), match/switch 7 (13 pts), boolean chains 2 (nesting depth added 13). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ParquetFilters.shreddedVariantEntries (cognitive 34) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:369— ParquetFilters.shreddedVariantEntries has cognitive complexity 34 (threshold 15). Drivers by points: match/switch 6 (19 pts), if/else 7 (14 pts), boolean chains 1 (nesting depth added 20). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
XmlRecordReader.readUntilEndElement (cognitive 34) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlInputFormat.scala:231— XmlRecordReader.readUntilEndElement has cognitive complexity 34 (threshold 15). Drivers by points: if/else 15 (30 pts), boolean chains 3, loops 1 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MapOutputTracker.convertMapStatuses (cognitive 33) core/src/main/scala/org/apache/spark/MapOutputTracker.scala:1872— MapOutputTracker.convertMapStatuses has cognitive complexity 33 (threshold 15). Drivers by points: if/else 6 (18 pts), loops 5 (12 pts), boolean chains 3 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ReaderInputStream.read (cognitive 33) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1336— ReaderInputStream.read has cognitive complexity 33 (threshold 15). Drivers by points: if/else 11 (28 pts), boolean chains 4, loops 1 (nesting depth added 17). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
VectorIndexerModel.getTransformFunc (cognitive 33) mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala:369— VectorIndexerModel.getTransformFunc has cognitive complexity 33 (threshold 15). Drivers by points: match/switch 4 (12 pts), if/else 7 (11 pts), error handling 2 (7 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 18). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
FunctionResolution.resolveProcedure (cognitive 33) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala:881— FunctionResolution.resolveProcedure has cognitive complexity 33 (threshold 15). Drivers by points: error handling 7 (20 pts), match/switch 2 (6 pts), if/else 3 (5 pts), boolean chains 1, loops 1 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RewritePredicateSubquery.apply (cognitive 33) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:144— RewritePredicateSubquery.apply has cognitive complexity 33 (threshold 15). Drivers by points: match/switch 7 (17 pts), if/else 7 (12 pts), boolean chains 4 (nesting depth added 15). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ExecuteGrpcResponseSender.execute (cognitive 33) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteGrpcResponseSender.scala:205— ExecuteGrpcResponseSender.execute has cognitive complexity 33 (threshold 15). Drivers by points: if/else 13 (23 pts), boolean chains 4, loops 2 (3 pts), match/switch 1 (3 pts) (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Aggregation.apply (cognitive 33) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:745— Aggregation.apply has cognitive complexity 33 (threshold 15). Drivers by points: match/switch 8 (19 pts), if/else 10 (13 pts), boolean chains 1 (nesting depth added 14). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
InferVariantShreddingSchema.collectFieldStats (cognitive 33) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/InferVariantShreddingSchema.scala:356— InferVariantShreddingSchema.collectFieldStats has cognitive complexity 33 (threshold 15). Drivers by points: if/else 13 (23 pts), loops 3 (7 pts), boolean chains 2, match/switch 1 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ColumnVectorUtils.populate (cognitive 33) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java:71— ColumnVectorUtils.populate has cognitive complexity 33 (threshold 15). Drivers by points: if/else 22 (24 pts), ternaries 2 (5 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
DataFrame.isin (cognitive 33) python/pyspark/pandas/frame.py:8681— DataFrame.isin has cognitive complexity 33 (threshold 15). Drivers by points: if/else 8 (15 pts), ternaries 4 (13 pts), loops 2 (4 pts), boolean chains 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
iLocIndexer._select_rows_by_slice (cognitive 33) python/pyspark/pandas/indexing.py:1645— iLocIndexer._select_rows_by_slice has cognitive complexity 33 (threshold 15). Drivers by points: if/else 20 (28 pts), boolean chains 5 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
KafkaSourceProvider.validateGeneralOptions (cognitive 32) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaSourceProvider.scala:228— KafkaSourceProvider.validateGeneralOptions has cognitive complexity 32 (threshold 15). Drivers by points: if/else 21 (29 pts), boolean chains 2, match/switch 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkSubmit.runMain (cognitive 32) core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:964— SparkSubmit.runMain has cognitive complexity 32 (threshold 15). Drivers by points: if/else 15 (20 pts), error handling 4 (5 pts), boolean chains 3, match/switch 2 (3 pts), loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HadoopMapReduceCommitProtocol.commitJob (cognitive 32) core/src/main/scala/org/apache/spark/internal/io/HadoopMapReduceCommitProtocol.scala:189— HadoopMapReduceCommitProtocol.commitJob has cognitive complexity 32 (threshold 15). Drivers by points: if/else 8 (22 pts), loops 3 (8 pts), boolean chains 2 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DAGScheduler.submitStage (cognitive 32) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2887— DAGScheduler.submitStage has cognitive complexity 32 (threshold 15). Drivers by points: if/else 10 (24 pts), loops 1 (5 pts), boolean chains 3 (nesting depth added 18). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
ALS.computeFactors (cognitive 32) mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala:1773— ALS.computeFactors has cognitive complexity 32 (threshold 15). Drivers by points: if/else 8 (21 pts), loops 2 (5 pts), error handling 1 (4 pts), boolean chains 1, match/switch 1 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LinearRegression.train (cognitive 32) mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:329— LinearRegression.train has cognitive complexity 32 (threshold 15). Drivers by points: if/else 18 (21 pts), boolean chains 8, match/switch 3 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
Client.createContainerLaunchContext (cognitive 32) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:1033— Client.createContainerLaunchContext has cognitive complexity 32 (threshold 15). Drivers by points: if/else 25 (29 pts), boolean chains 3 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
CheckAnalysis.checkAlterTableCommand (cognitive 32) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala:1308— CheckAnalysis.checkAlterTableCommand has cognitive complexity 32 (threshold 15). Drivers by points: if/else 10 (20 pts), match/switch 4 (11 pts), boolean chains 1 (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PullupCorrelatedPredicates.rewriteSubQueries (cognitive 32) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:570— PullupCorrelatedPredicates.rewriteSubQueries has cognitive complexity 32 (threshold 15). Drivers by points: if/else 16 (20 pts), match/switch 4 (8 pts), boolean chains 4 (nesting depth added 8). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
GenericArrayData.equals (cognitive 32) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/GenericArrayData.scala:90— GenericArrayData.equals has cognitive complexity 32 (threshold 15). Drivers by points: if/else 10 (24 pts), boolean chains 4, match/switch 1 (3 pts), loops 1 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AdaptiveSparkPlanExec.createNonResultQueryStages (cognitive 32) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:882— AdaptiveSparkPlanExec.createNonResultQueryStages has cognitive complexity 32 (threshold 15). Drivers by points: if/else 12 (24 pts), match/switch 3 (8 pts) (nesting depth added 17). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DemoteBroadcastHashJoin.selectJoinStrategy (cognitive 32) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/DemoteBroadcastHashJoin.scala:41— DemoteBroadcastHashJoin.selectJoinStrategy has cognitive complexity 32 (threshold 15). Drivers by points: if/else 13 (16 pts), match/switch 4 (9 pts), boolean chains 7 (nesting depth added 8). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
HiveInspectors.unwrapperFor (cognitive 32) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala:852— HiveInspectors.unwrapperFor has cognitive complexity 32 (threshold 15). Drivers by points: if/else 18 (27 pts), loops 1 (4 pts), match/switch 1 (nesting depth added 12). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ExpressionImplUtils.aesInternal (cognitive 32) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java:277— ExpressionImplUtils.aesInternal has cognitive complexity 32 (threshold 15). Drivers by points: if/else 15 (28 pts), boolean chains 3, error handling 1 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DecryptionHandler.channelRead (cognitive 32) common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:414— DecryptionHandler.channelRead has cognitive complexity 32 (threshold 15). Drivers by points: if/else 11 (26 pts), boolean chains 3, loops 2 (3 pts) (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RDD.histogram (cognitive 32) python/pyspark/core/rdd.py:2210— RDD.histogram has cognitive complexity 32 (threshold 15). Drivers by points: if/else 12 (23 pts), error handling 3 (6 pts), boolean chains 3 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkContext.addFile (cognitive 31) core/src/main/scala/org/apache/spark/SparkContext.scala:1866— SparkContext.addFile has cognitive complexity 31 (threshold 15). Drivers by points: if/else 19 (23 pts), boolean chains 7, match/switch 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
HadoopRDD.compute (cognitive 31) core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:269— HadoopRDD.compute has cognitive complexity 31 (threshold 15). Drivers by points: if/else 12 (15 pts), error handling 10 (13 pts), match/switch 2, boolean chains 1 (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ExecutorResourcesAmounts.assignAddressesCustomResources (cognitive 31) core/src/main/scala/org/apache/spark/scheduler/ExecutorResourcesAmounts.scala:143— ExecutorResourcesAmounts.assignAddressesCustomResources has cognitive complexity 31 (threshold 15). Drivers by points: if/else 7 (19 pts), loops 3 (9 pts), match/switch 1 (2 pts), boolean chains 1 (nesting depth added 19). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BasicBlockReplicationPolicy.prioritize (cognitive 31) core/src/main/scala/org/apache/spark/storage/BlockReplicationPolicy.scala:154— BasicBlockReplicationPolicy.prioritize has cognitive complexity 31 (threshold 15). Drivers by points: if/else 14 (27 pts), boolean chains 4 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveRelations.apply (cognitive 31) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:1255— ResolveRelations.apply has cognitive complexity 31 (threshold 15). Drivers by points: match/switch 10 (21 pts), if/else 9, boolean chains 1 (nesting depth added 11). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ExtractGenerator.apply (cognitive 31) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:3452— ExtractGenerator.apply has cognitive complexity 31 (threshold 15). Drivers by points: if/else 13 (22 pts), match/switch 4 (7 pts), boolean chains 2 (nesting depth added 12). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveDeserializer.apply (cognitive 31) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4145— ResolveDeserializer.apply has cognitive complexity 31 (threshold 15). Drivers by points: match/switch 7 (24 pts), if/else 5 (7 pts) (nesting depth added 19). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ColumnResolutionHelper.resolveExpression (cognitive 31) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:112— ColumnResolutionHelper.resolveExpression has cognitive complexity 31 (threshold 15). Drivers by points: if/else 14 (20 pts), match/switch 4 (7 pts), boolean chains 3, error handling 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GetJsonObjectEvaluator.evaluatePath (cognitive 31) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:1435— GetJsonObjectEvaluator.evaluatePath has cognitive complexity 31 (threshold 15). Drivers by points: if/else 11 (18 pts), loops 5 (10 pts), match/switch 2 (3 pts) (nesting depth added 13). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
BaseGenericInternalRow.equals (cognitive 31) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/rows.scala:87— BaseGenericInternalRow.equals has cognitive complexity 31 (threshold 15). Drivers by points: if/else 10 (24 pts), boolean chains 3, match/switch 1 (3 pts), loops 1 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GeneratorNestedColumnAliasing.unapply (cognitive 31) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala:322— GeneratorNestedColumnAliasing.unapply has cognitive complexity 31 (threshold 15). Drivers by points: match/switch 6 (16 pts), if/else 8 (11 pts), boolean chains 4 (nesting depth added 13). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
OrcUtils.requestedColumnIds (cognitive 31) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:320— OrcUtils.requestedColumnIds has cognitive complexity 31 (threshold 15). Drivers by points: if/else 15 (26 pts), match/switch 3, boolean chains 2 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkShreddingUtils.buildVariantSchema (cognitive 31) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala:559— SparkShreddingUtils.buildVariantSchema has cognitive complexity 31 (threshold 15). Drivers by points: if/else 11 (16 pts), match/switch 5 (12 pts), boolean chains 3 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
V2ScanRelationPushDown.pushVariantExtractions (cognitive 31) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:441— V2ScanRelationPushDown.pushVariantExtractions has cognitive complexity 31 (threshold 15). Drivers by points: if/else 15 (20 pts), boolean chains 5, match/switch 2 (4 pts), loops 2 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StreamExecution.runStream (cognitive 31) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamExecution.scala:282— StreamExecution.runStream has cognitive complexity 31 (threshold 15). Drivers by points: if/else 14 (19 pts), error handling 4 (6 pts), boolean chains 3, match/switch 2 (3 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RecordBinaryComparator.compare (cognitive 31) sql/core/src/main/java/org/apache/spark/sql/execution/RecordBinaryComparator.java:32— RecordBinaryComparator.compare has cognitive complexity 31 (threshold 15). Drivers by points: if/else 7 (15 pts), ternaries 2 (7 pts), loops 3 (5 pts), boolean chains 4 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VariantShreddingWriter.castShredded (cognitive 31) common/variant/src/main/java/org/apache/spark/types/variant/VariantShreddingWriter.java:59— VariantShreddingWriter.castShredded has cognitive complexity 31 (threshold 15). Drivers by points: if/else 13 (22 pts), loops 3 (7 pts), boolean chains 2 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
merge_spark_pr.compute_merge_default_fix_versions (cognitive 31) REDACTED:201— merge_spark_pr.compute_merge_default_fix_versions has cognitive complexity 31 (threshold 15). Drivers by points: if/else 14 (28 pts), loops 2, boolean chains 1 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
merge_spark_pr.choose_jira_assignee (cognitive 31) REDACTED:1692— merge_spark_pr.choose_jira_assignee has cognitive complexity 31 (threshold 15). Drivers by points: if/else 6 (14 pts), error handling 4 (10 pts), loops 2 (3 pts), ternaries 1 (3 pts), boolean chains 1 (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.droplevel (cognitive 31) python/pyspark/pandas/frame.py:7635— DataFrame.droplevel has cognitive complexity 31 (threshold 15). Drivers by points: if/else 11 (29 pts), loops 1 (2 pts) (nesting depth added 19). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
DataFrame.rename (cognitive 31) python/pyspark/pandas/frame.py:12113— DataFrame.rename has cognitive complexity 31 (threshold 15). Drivers by points: if/else 15 (24 pts), boolean chains 4, loops 1 (3 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UserCredentialManager.resolveCredentials (cognitive 30) core/src/main/scala/org/apache/spark/deploy/security/UserCredentialManager.scala:257— UserCredentialManager.resolveCredentials has cognitive complexity 30 (threshold 15). Drivers by points: if/else 8 (17 pts), error handling 2 (7 pts), match/switch 1 (5 pts), loops 1 (nesting depth added 18). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskSetManager.handleFailedTask (cognitive 30) core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:992— TaskSetManager.handleFailedTask has cognitive complexity 30 (threshold 15). Drivers by points: if/else 17 (27 pts), boolean chains 2, match/switch 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BlockStoreUpdater.save (cognitive 30) core/src/main/scala/org/apache/spark/storage/BlockManager.scala:536— BlockStoreUpdater.save has cognitive complexity 30 (threshold 15). Drivers by points: if/else 17 (24 pts), boolean chains 4, error handling 1 (2 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
EdgePartition.aggregateMessagesIndexScan (cognitive 30) graphx/src/main/scala/org/apache/spark/graphx/impl/EdgePartition.scala:428— EdgePartition.aggregateMessagesIndexScan has cognitive complexity 30 (threshold 15). Drivers by points: if/else 18 (26 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DecisionTreeMetadata.buildMetadata (cognitive 30) mllib/src/main/scala/org/apache/spark/ml/tree/impl/DecisionTreeMetadata.scala:113— DecisionTreeMetadata.buildMetadata has cognitive complexity 30 (threshold 15). Drivers by points: if/else 15 (22 pts), match/switch 5 (8 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RankingMetrics.ndcgAt (cognitive 30) mllib/src/main/scala/org/apache/spark/mllib/evaluation/RankingMetrics.scala:158— RankingMetrics.ndcgAt has cognitive complexity 30 (threshold 15). Drivers by points: if/else 11 (26 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RowMatrix.computeSVD (cognitive 30) mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:341— RowMatrix.computeSVD has cognitive complexity 30 (threshold 15). Drivers by points: if/else 12 (17 pts), loops 3 (6 pts), boolean chains 5, match/switch 2 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RowMatrix.columnSimilaritiesDIMSUM (cognitive 30) mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala:722— RowMatrix.columnSimilaritiesDIMSUM has cognitive complexity 30 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 4 (10 pts), boolean chains 4, match/switch 1 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
FunctionResolution.validateFunction (cognitive 30) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala:629— FunctionResolution.validateFunction has cognitive complexity 30 (threshold 15). Drivers by points: if/else 12 (18 pts), boolean chains 7, match/switch 3 (5 pts) (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveIdentifierClause.apply0 (cognitive 30) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveIdentifierClause.scala:96— ResolveIdentifierClause.apply0 has cognitive complexity 30 (threshold 15). Drivers by points: if/else 10 (21 pts), match/switch 3 (6 pts), boolean chains 3 (nesting depth added 14). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TableOutputResolver.resolveColumnsByPosition (cognitive 30) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:504— TableOutputResolver.resolveColumnsByPosition has cognitive complexity 30 (threshold 15). Drivers by points: if/else 17 (24 pts), boolean chains 3, match/switch 2 (3 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ImplicitTypeCoercion.apply (cognitive 30) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:733— ImplicitTypeCoercion.apply has cognitive complexity 30 (threshold 15). Drivers by points: if/else 15 (23 pts), match/switch 3 (5 pts), boolean chains 2 (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
PushdownPredicatesAndPruneColumnsForCTEDef.removePushedDownFilter (cognitive 30) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushdownPredicatesAndPruneColumnsForCTEDef.scala:203— PushdownPredicatesAndPruneColumnsForCTEDef.removePushedDownFilter has cognitive complexity 30 (threshold 15). Drivers by points: if/else 17 (26 pts), match/switch 2 (3 pts), boolean chains 1 (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ToNumberParser.checkInputDataTypes (cognitive 30) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala:258— ToNumberParser.checkInputDataTypes has cognitive complexity 30 (threshold 15). Drivers by points: if/else 14 (15 pts), match/switch 7 (9 pts), boolean chains 5, loops 1 (nesting depth added 3). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
WorkerInputStream.read (cognitive 30) sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonPlannerRunner.scala:216— WorkerInputStream.read has cognitive complexity 30 (threshold 15). Drivers by points: if/else 9 (23 pts), loops 2 (4 pts), boolean chains 3 (nesting depth added 16). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
InsertIntoHiveTable.processInsert (cognitive 30) sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala:125— InsertIntoHiveTable.processInsert has cognitive complexity 30 (threshold 15). Drivers by points: if/else 10 (24 pts), match/switch 1 (4 pts), boolean chains 2 (nesting depth added 17). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
UTF8String.levenshteinDistance (cognitive 30) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:2228— UTF8String.levenshteinDistance has cognitive complexity 30 (threshold 15). Drivers by points: if/else 13 (20 pts), ternaries 3 (6 pts), loops 3 (4 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
instrumentation_utils._attach (cognitive 30) python/pyspark/instrumentation_utils.py:121— instrumentation_utils._attach has cognitive complexity 30 (threshold 15). Drivers by points: loops 8 (13 pts), if/else 4 (10 pts), error handling 1 (3 pts), ternaries 1 (3 pts), boolean chains 1 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.select_dtypes (cognitive 30) python/pyspark/pandas/frame.py:7452— DataFrame.select_dtypes has cognitive complexity 30 (threshold 15). Drivers by points: if/else 9 (11 pts), error handling 4 (8 pts), loops 5, ternaries 2 (4 pts), boolean chains 2 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
(anonymous) (cognitive 30) REDACTED:109— (anonymous) has cognitive complexity 30 (threshold 15). Drivers by points: if/else 11 (14 pts), ternaries 5 (9 pts), boolean chains 4, loops 2 (3 pts) (nesting depth added 8). Most of this is not in the body itself: 2 of the 30 points are its own statements and the rest belongs to 16 function items inside it that branch (a function literal at line 126, a function literal at line 291, a function literal at line 171, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
FsHistoryProvider.checkForLogsInDir (cognitive 29) core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:635— FsHistoryProvider.checkForLogsInDir has cognitive complexity 29 (threshold 15). Drivers by points: if/else 13 (17 pts), error handling 7 (10 pts), boolean chains 2 (nesting depth added 7). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
NettyBlockRpcServer.receive (cognitive 29) core/src/main/scala/org/apache/spark/network/netty/NettyBlockRpcServer.scala:51— NettyBlockRpcServer.receive has cognitive complexity 29 (threshold 15). Drivers by points: if/else 13 (25 pts), error handling 2, boolean chains 1, match/switch 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AppStatusListener.onTaskEnd (cognitive 29) core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:669— AppStatusListener.onTaskEnd has cognitive complexity 29 (threshold 15). Drivers by points: if/else 23 (24 pts), boolean chains 3, match/switch 2 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
BlockManager.getLocalValues (cognitive 29) core/src/main/scala/org/apache/spark/storage/BlockManager.scala:1151— BlockManager.getLocalValues has cognitive complexity 29 (threshold 15). Drivers by points: if/else 10 (22 pts), boolean chains 3, error handling 1 (3 pts), match/switch 1 (nesting depth added 14). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AllJobsPage.render (cognitive 29) core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:278— AllJobsPage.render has cognitive complexity 29 (threshold 15). Drivers by points: if/else 19 (25 pts), boolean chains 2, match/switch 2 (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ImageSchema.decode (cognitive 29) mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala:131— ImageSchema.decode has cognitive complexity 29 (threshold 15). Drivers by points: if/else 8 (14 pts), loops 4 (14 pts), error handling 1 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RandomForest.findBestSplits (cognitive 29) mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala:526— RandomForest.findBestSplits has cognitive complexity 29 (threshold 15). Drivers by points: if/else 19 (26 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkDateTimeUtils.stringToDate (cognitive 29) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:577— SparkDateTimeUtils.stringToDate has cognitive complexity 29 (threshold 15). Drivers by points: if/else 12 (18 pts), boolean chains 9, error handling 1, loops 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkDateTimeUtils.stringToTime (cognitive 29) sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala:1098— SparkDateTimeUtils.stringToTime has cognitive complexity 29 (threshold 15). Drivers by points: if/else 13 (21 pts), boolean chains 7, error handling 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
FunctionArgumentTypeCoercion.apply (cognitive 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:505— FunctionArgumentTypeCoercion.apply has cognitive complexity 29 (threshold 15). Drivers by points: match/switch 11 (21 pts), boolean chains 7, if/else 1 (nesting depth added 10). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
JacksonParser.parseIterator (cognitive 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:908— JacksonParser.parseIterator has cognitive complexity 29 (threshold 15). Drivers by points: if/else 8 (14 pts), error handling 8, match/switch 3 (4 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InferWindowGroupLimit.apply (cognitive 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InferWindowGroupLimit.scala:89— InferWindowGroupLimit.apply has cognitive complexity 29 (threshold 15). Drivers by points: if/else 10 (20 pts), match/switch 3 (6 pts), boolean chains 3 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AstBuilder.withSample (cognitive 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:2828— AstBuilder.withSample has cognitive complexity 29 (threshold 15). Drivers by points: if/else 17 (25 pts), boolean chains 3, match/switch 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ToNumberParser.format (cognitive 29) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala:649— ToNumberParser.format has cognitive complexity 29 (threshold 15). Drivers by points: if/else 12 (23 pts), boolean chains 3, match/switch 1 (2 pts), loops 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowDeserializers.deserializerForDefault (cognitive 29) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowDeserializer.scala:101— ArrowDeserializers.deserializerForDefault has cognitive complexity 29 (threshold 15). Drivers by points: if/else 13 (18 pts), loops 3 (9 pts), boolean chains 1, match/switch 1 (nesting depth added 11). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveDataSource.apply (cognitive 29) sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDataSource.scala:45— ResolveDataSource.apply has cognitive complexity 29 (threshold 15). Drivers by points: if/else 12 (18 pts), match/switch 5 (9 pts), boolean chains 2 (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DataStreamWriter.startInternal (cognitive 29) sql/core/src/main/scala/org/apache/spark/sql/classic/DataStreamWriter.scala:225— DataStreamWriter.startInternal has cognitive complexity 29 (threshold 15). Drivers by points: if/else 15 (21 pts), match/switch 2 (6 pts), boolean chains 2 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
OrcUtils.createAggInternalRowFromFooter (cognitive 29) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcUtils.scala:613— OrcUtils.createAggInternalRowFromFooter has cognitive complexity 29 (threshold 15). Drivers by points: if/else 17 (22 pts), match/switch 4 (6 pts), error handling 1 (nesting depth added 7). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ParquetReadSupport.findCheapestGroupField (cognitive 29) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala:554— ParquetReadSupport.findCheapestGroupField has cognitive complexity 29 (threshold 15). Drivers by points: if/else 6 (16 pts), boolean chains 5, match/switch 3 (5 pts), loops 1 (3 pts) (nesting depth added 14). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SortMergeJoinExec.doProduce (cognitive 29) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala:485— SortMergeJoinExec.doProduce has cognitive complexity 29 (threshold 15). Drivers by points: if/else 15, match/switch 9 (13 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
UserDefinedPythonFunction.builder (cognitive 29) sql/core/src/main/scala/org/apache/spark/sql/execution/python/UserDefinedPythonFunction.scala:66— UserDefinedPythonFunction.builder has cognitive complexity 29 (threshold 15). Drivers by points: if/else 17 (19 pts), boolean chains 6, match/switch 2 (4 pts) (nesting depth added 4). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
MicroBatchExecution.populateStartOffsets (cognitive 29) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala:849— MicroBatchExecution.populateStartOffsets has cognitive complexity 29 (threshold 15). Drivers by points: if/else 8 (15 pts), match/switch 5 (14 pts) (nesting depth added 16). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
CollationAwareUTF8String.trimRight (cognitive 29) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1588— CollationAwareUTF8String.trimRight has cognitive complexity 29 (threshold 15). Drivers by points: if/else 10 (17 pts), loops 5 (9 pts), boolean chains 3 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LogisticRegressionModel.predict (cognitive 29) python/pyspark/mllib/classification.py:235— LogisticRegressionModel.predict has cognitive complexity 29 (threshold 15). Drivers by points: if/else 11 (20 pts), loops 2 (6 pts), ternaries 1 (3 pts) (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PandasToArrowConversion._convert_column_legacy (cognitive 29) python/pyspark/sql/conversion.py:502— PandasToArrowConversion._convert_column_legacy has cognitive complexity 29 (threshold 15). Drivers by points: if/else 8 (18 pts), error handling 5 (10 pts), ternaries 1 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowTableToRowsConversion._to_pylist (cognitive 29) python/pyspark/sql/conversion.py:1141— ArrowTableToRowsConversion._to_pylist has cognitive complexity 29 (threshold 15). Drivers by points: if/else 10 (17 pts), ternaries 4 (9 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MapOutputTrackerWorker.getStatuses (cognitive 28) core/src/main/scala/org/apache/spark/MapOutputTracker.scala:1592— MapOutputTrackerWorker.getStatuses has cognitive complexity 28 (threshold 15). Drivers by points: if/else 9 (18 pts), error handling 2 (8 pts), boolean chains 2 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Master.schedule (cognitive 28) core/src/main/scala/org/apache/spark/deploy/master/Master.scala:961— Master.schedule has cognitive complexity 28 (threshold 15). Drivers by points: if/else 7 (14 pts), loops 3 (7 pts), match/switch 1 (4 pts), boolean chains 3 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DAGScheduler.cleanupStagesForJob (cognitive 28) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:1738— DAGScheduler.cleanupStagesForJob has cognitive complexity 28 (threshold 15). Drivers by points: if/else 8 (20 pts), loops 2 (7 pts), boolean chains 1 (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LogisticRegression.createBounds (cognitive 28) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:726— LogisticRegression.createBounds has cognitive complexity 28 (threshold 15). Drivers by points: if/else 12 (26 pts), loops 1 (2 pts) (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
NNLS.solve (cognitive 28) mllib/src/main/scala/org/apache/spark/mllib/optimization/NNLS.scala:70— NNLS.solve has cognitive complexity 28 (threshold 15). Drivers by points: if/else 9 (19 pts), loops 4 (7 pts), boolean chains 2 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ApplicationMaster.allocationThreadImpl (cognitive 28) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:587— ApplicationMaster.allocationThreadImpl has cognitive complexity 28 (threshold 15). Drivers by points: if/else 11 (18 pts), error handling 4 (8 pts), boolean chains 1, loops 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CodeCompiler.forEachJavaSpan (cognitive 28) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:552— CodeCompiler.forEachJavaSpan has cognitive complexity 28 (threshold 15). Drivers by points: if/else 9 (13 pts), match/switch 2 (6 pts), boolean chains 5, loops 2 (4 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RewriteCorrelatedScalarSubquery.constructLeftJoins (cognitive 28) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:901— RewriteCorrelatedScalarSubquery.constructLeftJoins has cognitive complexity 28 (threshold 15). Drivers by points: if/else 11 (18 pts), match/switch 3 (8 pts), boolean chains 2 (nesting depth added 12). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AstBuilder.visitMergeIntoTable (cognitive 28) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1274— AstBuilder.visitMergeIntoTable has cognitive complexity 28 (threshold 15). Drivers by points: if/else 20 (24 pts), boolean chains 4 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SchemaUtils.validateTypeCompatibility (cognitive 28) sql/catalyst/src/main/scala/org/apache/spark/sql/util/SchemaUtils.scala:428— SchemaUtils.validateTypeCompatibility has cognitive complexity 28 (threshold 15). Drivers by points: if/else 7 (12 pts), loops 2 (9 pts), boolean chains 4, match/switch 2 (3 pts) (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowSerializer.serializerForDefault (cognitive 28) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala:249— ArrowSerializer.serializerForDefault has cognitive complexity 28 (threshold 15). Drivers by points: if/else 15 (19 pts), match/switch 4 (8 pts), boolean chains 1 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TruncateTableCommand.run (cognitive 28) sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:447— TruncateTableCommand.run has cognitive complexity 28 (threshold 15). Drivers by points: error handling 5 (14 pts), if/else 8 (10 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PullOutVariantExtractions.pushSideAliases (cognitive 28) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PullOutVariantExtractions.scala:230— PullOutVariantExtractions.pushSideAliases has cognitive complexity 28 (threshold 15). Drivers by points: if/else 12 (21 pts), match/switch 2 (5 pts), boolean chains 2 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ParquetRowConverter.newConverterDefault (cognitive 28) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetRowConverter.scala:320— ParquetRowConverter.newConverterDefault has cognitive complexity 28 (threshold 15). Drivers by points: if/else 11 (14 pts), match/switch 5 (8 pts), boolean chains 6 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DistributionAndOrderingUtils.prepareQuery (cognitive 28) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DistributionAndOrderingUtils.scala:33— DistributionAndOrderingUtils.prepareQuery has cognitive complexity 28 (threshold 15). Drivers by points: if/else 14 (24 pts), match/switch 2 (3 pts), boolean chains 1 (nesting depth added 11). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TransformWithStateInPySparkStateServer.handleStatefulProcessorCall (cognitive 28) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:351— TransformWithStateInPySparkStateServer.handleStatefulProcessorCall has cognitive complexity 28 (threshold 15). Drivers by points: if/else 15 (23 pts), match/switch 3 (5 pts) (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
UnsafeRowDataEncoder.encodePrefixKeyForRangeScan (cognitive 28) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:512— UnsafeRowDataEncoder.encodePrefixKeyForRangeScan has cognitive complexity 28 (threshold 15). Drivers by points: if/else 14 (26 pts), match/switch 1 (2 pts) (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TriggeredGraphExecution.topologicalExecution (cognitive 28) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/TriggeredGraphExecution.scala:157— TriggeredGraphExecution.topologicalExecution has cognitive complexity 28 (threshold 15). Drivers by points: if/else 8 (14 pts), boolean chains 5, error handling 2 (4 pts), loops 2 (3 pts), match/switch 1 (2 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MutableColumnarRow.copy (cognitive 28) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnarRow.java:57— MutableColumnarRow.copy has cognitive complexity 28 (threshold 15). Drivers by points: if/else 23 (26 pts), boolean chains 1, loops 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
UTF8String.toInt (cognitive 28) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1922— UTF8String.toInt has cognitive complexity 28 (threshold 15). Drivers by points: if/else 11 (18 pts), boolean chains 6, loops 4 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (UTF8String.toLong) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
UTF8String.toLong (cognitive 28) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1825— UTF8String.toLong has cognitive complexity 28 (threshold 15). Drivers by points: if/else 11 (18 pts), boolean chains 6, loops 4 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (UTF8String.toInt) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
structured_logging_style.main (cognitive 28) dev/structured_logging_style.py:26— structured_logging_style.main has cognitive complexity 28 (threshold 15). Drivers by points: if/else 6 (15 pts), loops 5 (12 pts), boolean chains 1 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.__init__ (cognitive 28) python/pyspark/pandas/frame.py:552— DataFrame.__init__ has cognitive complexity 28 (threshold 15). Drivers by points: if/else 13 (23 pts), ternaries 1 (3 pts), boolean chains 2 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.corrwith (cognitive 28) python/pyspark/pandas/frame.py:1764— DataFrame.corrwith has cognitive complexity 28 (threshold 15). Drivers by points: if/else 15 (19 pts), loops 4 (8 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
DataFrame.align (cognitive 28) python/pyspark/pandas/frame.py:13773— DataFrame.align has cognitive complexity 28 (threshold 15). Drivers by points: if/else 11 (19 pts), boolean chains 4, loops 2 (4 pts), ternaries 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InternalFrame.with_new_columns (cognitive 28) python/pyspark/pandas/internal.py:1275— InternalFrame.with_new_columns has cognitive complexity 28 (threshold 15). Drivers by points: if/else 15 (22 pts), loops 3 (6 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PandasSQLStringFormatter._convert_value (cognitive 28) python/pyspark/pandas/sql_formatter.py:249— PandasSQLStringFormatter._convert_value has cognitive complexity 28 (threshold 15). Drivers by points: if/else 9 (21 pts), loops 2 (7 pts) (nesting depth added 17). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
types.parse_attr_name (cognitive 28) python/pyspark/sql/connect/types.py:363— types.parse_attr_name has cognitive complexity 28 (threshold 15). Drivers by points: if/else 10 (24 pts), boolean chains 3, loops 1 (nesting depth added 14). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SparkConversionMixin._create_from_pandas_with_arrow (cognitive 28) python/pyspark/sql/pandas/conversion.py:907— SparkConversionMixin._create_from_pandas_with_arrow has cognitive complexity 28 (threshold 15). Drivers by points: if/else 10 (18 pts), loops 2 (6 pts), ternaries 2 (3 pts), boolean chains 1 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
(anonymous) (cognitive 28) sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/allexecutionspage.js:22— (anonymous) has cognitive complexity 28 (threshold 15). Drivers by points: if/else 10 (14 pts), boolean chains 13, ternaries 1 (nesting depth added 4). Most of this is not in the body itself: 1 of the 28 points is its own statement and the rest belongs to 12 function items inside it that branch (init::renderSummary, a function literal at line 152, init::render, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
updateDetailsPanel (cognitive 28) REDACTED:423— updateDetailsPanel has cognitive complexity 28 (threshold 15). Drivers by points: if/else 15 (19 pts), boolean chains 7, ternaries 2 (nesting depth added 4). Of this number, 19 points are the body's own statements and 9 belong to 4 function literals inside it that branch. To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ProtobufSerializer.newConverter (cognitive 27) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufSerializer.scala:81— ProtobufSerializer.newConverter has cognitive complexity 27 (threshold 15). Drivers by points: if/else 9 (16 pts), boolean chains 4, loops 2 (4 pts), match/switch 2 (3 pts) (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
FsHistoryProvider.doMergeApplicationListingInternal (cognitive 27) core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:1055— FsHistoryProvider.doMergeApplicationListingInternal has cognitive complexity 27 (threshold 15). Drivers by points: if/else 7 (14 pts), boolean chains 9, error handling 1 (3 pts), match/switch 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskReaper.run (cognitive 27) core/src/main/scala/org/apache/spark/executor/Executor.scala:1274— TaskReaper.run has cognitive complexity 27 (threshold 15). Drivers by points: if/else 11 (19 pts), error handling 1 (4 pts), boolean chains 3, loops 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
IndexShuffleBlockResolver.writeMetadataFileAndCommit (cognitive 27) core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala:391— IndexShuffleBlockResolver.writeMetadataFileAndCommit has cognitive complexity 27 (threshold 15). Drivers by points: if/else 13 (20 pts), error handling 2 (4 pts), boolean chains 3 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BlockManager.doPutIterator (cognitive 27) core/src/main/scala/org/apache/spark/storage/BlockManager.scala:1864— BlockManager.doPutIterator has cognitive complexity 27 (threshold 15). Drivers by points: if/else 11 (20 pts), match/switch 2 (6 pts), boolean chains 1 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JobPage.render (cognitive 27) core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:222— JobPage.render has cognitive complexity 27 (threshold 15). Drivers by points: if/else 19 (22 pts), boolean chains 3, loops 1, match/switch 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SparseVector.slice (cognitive 27) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:792— SparseVector.slice has cognitive complexity 27 (threshold 15). Drivers by points: if/else 6 (16 pts), loops 3 (8 pts), boolean chains 3 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
YarnAllocator.updateResourceRequests (cognitive 27) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:498— YarnAllocator.updateResourceRequests has cognitive complexity 27 (threshold 15). Drivers by points: if/else 11 (21 pts), loops 1 (3 pts), match/switch 1 (2 pts), boolean chains 1 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InterpretedUnsafeProjection.generateFieldWriter (cognitive 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InterpretedUnsafeProjection.scala:134— InterpretedUnsafeProjection.generateFieldWriter has cognitive complexity 27 (threshold 15). Drivers by points: if/else 13 (17 pts), match/switch 5 (10 pts) (nesting depth added 9). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DivModLike.doGenCode (cognitive 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:747— DivModLike.doGenCode has cognitive complexity 27 (threshold 15). Drivers by points: if/else 17 (23 pts), boolean chains 3, match/switch 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JsonTableEvaluator.positionAt (cognitive 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:998— JsonTableEvaluator.positionAt has cognitive complexity 27 (threshold 15). Drivers by points: if/else 9 (18 pts), loops 2 (6 pts), boolean chains 2, match/switch 1 (nesting depth added 13). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
JsonTable.checkInputDataTypes (cognitive 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:456— JsonTable.checkInputDataTypes has cognitive complexity 27 (threshold 15). Drivers by points: match/switch 6 (17 pts), if/else 6 (9 pts), boolean chains 1 (nesting depth added 14). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ReorderAssociativeOperator.apply (cognitive 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:303— ReorderAssociativeOperator.apply has cognitive complexity 27 (threshold 15). Drivers by points: if/else 10 (20 pts), boolean chains 4, match/switch 2 (3 pts) (nesting depth added 11). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
XmlTokenizer.readUntilStartElement (cognitive 27) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1006— XmlTokenizer.readUntilStartElement has cognitive complexity 27 (threshold 15). Drivers by points: if/else 13 (25 pts), boolean chains 1, loops 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkSqlAstBuilder.visitCreateUserDefinedFunction (cognitive 27) REDACTED:990— SparkSqlAstBuilder.visitCreateUserDefinedFunction has cognitive complexity 27 (threshold 15). Drivers by points: if/else 9 (16 pts), loops 3 (6 pts), boolean chains 5 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
WholeStageCodegenExec.decideFromTrials (cognitive 27) sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala:857— WholeStageCodegenExec.decideFromTrials has cognitive complexity 27 (threshold 15). Drivers by points: if/else 11 (16 pts), match/switch 4 (11 pts) (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TransformWithStateInPySparkStateServer.handleMapStateRequest (cognitive 27) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:595— TransformWithStateInPySparkStateServer.handleMapStateRequest has cognitive complexity 27 (threshold 15). Drivers by points: if/else 17 (26 pts), match/switch 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RocksDB.loadWithCheckpointId (cognitive 27) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:474— RocksDB.loadWithCheckpointId has cognitive complexity 27 (threshold 15). Drivers by points: if/else 13 (19 pts), match/switch 1 (4 pts), boolean chains 3, error handling 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ColumnVectorUtils.appendValue (cognitive 27) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVectorUtils.java:178— ColumnVectorUtils.appendValue has cognitive complexity 27 (threshold 15). Drivers by points: if/else 24 (26 pts), boolean chains 1 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
V2ExpressionSQLBuilder.build (cognitive 27) sql/catalyst/src/main/java/org/apache/spark/sql/connector/util/V2ExpressionSQLBuilder.java:101— V2ExpressionSQLBuilder.build has cognitive complexity 27 (threshold 15). Drivers by points: if/else 24 (27 pts) (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
CollationAwareUTF8String.lowercaseIndexOfSlow (cognitive 27) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:836— CollationAwareUTF8String.lowercaseIndexOfSlow has cognitive complexity 27 (threshold 15). Drivers by points: if/else 11 (23 pts), loops 2 (4 pts) (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame._assign (cognitive 27) python/pyspark/pandas/frame.py:5851— DataFrame._assign has cognitive complexity 27 (threshold 15). Drivers by points: if/else 6 (16 pts), loops 4 (5 pts), boolean chains 3, ternaries 3 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Frame.to_csv (cognitive 27) python/pyspark/pandas/generic.py:600— Frame.to_csv has cognitive complexity 27 (threshold 15). Drivers by points: if/else 15 (19 pts), ternaries 2 (4 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.replace (cognitive 27) python/pyspark/sql/classic/dataframe.py:1549— DataFrame.replace has cognitive complexity 27 (threshold 15). Drivers by points: if/else 18 (22 pts), boolean chains 5 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
transpile._transpile_func (cognitive 27) python/pyspark/sql/transpile.py:1208— transpile._transpile_func has cognitive complexity 27 (threshold 15). Drivers by points: if/else 12 (15 pts), boolean chains 5, error handling 2 (4 pts), loops 2 (3 pts) (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
FsHistoryProvider.skipToLineStart (cognitive 26) core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:1787— FsHistoryProvider.skipToLineStart has cognitive complexity 26 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 3 (8 pts), boolean chains 3 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskSetManager.executorLost (cognitive 26) core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:1210— TaskSetManager.executorLost has cognitive complexity 26 (threshold 15). Drivers by points: if/else 7 (13 pts), match/switch 2 (6 pts), boolean chains 4, loops 2 (3 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BlockManagerMasterEndpoint.updateBlockInfo (cognitive 26) core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala:828— BlockManagerMasterEndpoint.updateBlockInfo has cognitive complexity 26 (threshold 15). Drivers by points: if/else 16 (22 pts), boolean chains 2, match/switch 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MemoryStore.putIterator (cognitive 26) core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala:191— MemoryStore.putIterator has cognitive complexity 26 (threshold 15). Drivers by points: if/else 12 (22 pts), boolean chains 3, loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StronglyConnectedComponents.run (cognitive 26) graphx/src/main/scala/org/apache/spark/graphx/lib/StronglyConnectedComponents.scala:38— StronglyConnectedComponents.run has cognitive complexity 26 (threshold 15). Drivers by points: if/else 9 (18 pts), boolean chains 5, loops 2 (3 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Postfix.project (cognitive 26) mllib/src/main/scala/org/apache/spark/mllib/fpm/PrefixSpan.scala:497— Postfix.project has cognitive complexity 26 (threshold 15). Drivers by points: if/else 8 (21 pts), loops 2 (4 pts), boolean chains 1 (nesting depth added 15). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveQualify.apply (cognitive 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4536— ResolveQualify.apply has cognitive complexity 26 (threshold 15). Drivers by points: if/else 11 (21 pts), match/switch 2 (4 pts), boolean chains 1 (nesting depth added 12). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
BinByResolution.validateAndComputeParameters (cognitive 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/BinByResolution.scala:63— BinByResolution.validateAndComputeParameters has cognitive complexity 26 (threshold 15). Drivers by points: if/else 14 (19 pts), error handling 2 (4 pts), match/switch 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CodegenContext.subexpressionEliminationForWholeStageCodegen (cognitive 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala:2138— CodegenContext.subexpressionEliminationForWholeStageCodegen has cognitive complexity 26 (threshold 15). Drivers by points: if/else 12 (19 pts), match/switch 3 (7 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InsertMapSortInAggregate.apply (cognitive 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InsertMapSortExpression.scala:52— InsertMapSortInAggregate.apply has cognitive complexity 26 (threshold 15). Drivers by points: if/else 11 (15 pts), match/switch 4 (9 pts), boolean chains 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AstBuilder.withJoinRelation (cognitive 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:2600— AstBuilder.withJoinRelation has cognitive complexity 26 (threshold 15). Drivers by points: if/else 12 (21 pts), match/switch 3 (5 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AstBuilder.visitTypeConstructor (cognitive 26) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:4851— AstBuilder.visitTypeConstructor has cognitive complexity 26 (threshold 15). Drivers by points: if/else 13 (18 pts), error handling 2 (4 pts), match/switch 2 (3 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectPlanExecution.processAsArrowBatches (cognitive 26) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/SparkConnectPlanExecution.scala:129— SparkConnectPlanExecution.processAsArrowBatches has cognitive complexity 26 (threshold 15). Drivers by points: if/else 8 (11 pts), loops 2 (7 pts), match/switch 3 (6 pts), boolean chains 2 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectPlanner.transformTypedGroupMap (cognitive 26) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:777— SparkConnectPlanner.transformTypedGroupMap has cognitive complexity 26 (threshold 15). Drivers by points: if/else 16 (22 pts), boolean chains 2, match/switch 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
EmbeddedArraySplitter.consumeValue (cognitive 26) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/EmbeddedArrayJsonDataSource.scala:216— EmbeddedArraySplitter.consumeValue has cognitive complexity 26 (threshold 15). Drivers by points: if/else 10 (18 pts), boolean chains 4, loops 2 (4 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StateStoreSaveExec.doExecute (cognitive 26) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/statefulOperators.scala:880— StateStoreSaveExec.doExecute has cognitive complexity 26 (threshold 15). Drivers by points: if/else 9 (15 pts), loops 3 (6 pts), match/switch 2 (3 pts), boolean chains 2 (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
FileStreamSource.fetchMaxOffset (cognitive 26) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/FileStreamSource.scala:171— FileStreamSource.fetchMaxOffset has cognitive complexity 26 (threshold 15). Drivers by points: if/else 13 (17 pts), match/switch 4 (6 pts), boolean chains 3 (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SparkGetSchemasOperation.runInternal (cognitive 26) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkGetSchemasOperation.scala:50— SparkGetSchemasOperation.runInternal has cognitive complexity 26 (threshold 15). Drivers by points: if/else 13 (20 pts), boolean chains 3, match/switch 1 (3 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GetColumnsOperation.runInternal (cognitive 26) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetColumnsOperation.java:133— GetColumnsOperation.runInternal has cognitive complexity 26 (threshold 15). Drivers by points: loops 5 (10 pts), ternaries 2 (8 pts), if/else 3 (6 pts), boolean chains 1, error handling 1 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ParquetColumnVector.<init> (cognitive 26) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetColumnVector.java:64— ParquetColumnVector.<init> has cognitive complexity 26 (threshold 15). Drivers by points: if/else 12 (20 pts), ternaries 1 (3 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VariantBuilder.pickImpl (cognitive 26) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:1046— VariantBuilder.pickImpl has cognitive complexity 26 (threshold 15). Drivers by points: if/else 12 (22 pts), loops 2 (4 pts) (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CollationAwareUTF8String.lowercaseTrimRight (cognitive 26) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1490— CollationAwareUTF8String.lowercaseTrimRight has cognitive complexity 26 (threshold 15). Drivers by points: if/else 13 (18 pts), loops 4 (5 pts), boolean chains 3 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TorchDistributor._execute_command (cognitive 26) python/pyspark/ml/torch/distributor.py:495— TorchDistributor._execute_command has cognitive complexity 26 (threshold 15). Drivers by points: if/else 10 (18 pts), error handling 2 (6 pts), boolean chains 1, loops 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Frame.groupby (cognitive 26) python/pyspark/pandas/generic.py:2395— Frame.groupby has cognitive complexity 26 (threshold 15). Drivers by points: if/else 12 (24 pts), loops 1 (2 pts) (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
typehints.spark_type_to_pandas_dtype (cognitive 26) python/pyspark/pandas/typedef/typehints.py:262— typehints.spark_type_to_pandas_dtype has cognitive complexity 26 (threshold 15). Drivers by points: if/else 14 (24 pts), boolean chains 2 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
utils.align_diff_frames (cognitive 26) python/pyspark/pandas/utils.py:325— utils.align_diff_frames has cognitive complexity 26 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 5 (7 pts), boolean chains 4 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExecutePlanResponseReattachableIterator._has_next (cognitive 26) python/pyspark/sql/connect/client/reattach.py:166— ExecutePlanResponseReattachableIterator._has_next has cognitive complexity 26 (threshold 15). Drivers by points: error handling 3 (11 pts), if/else 4 (8 pts), loops 2 (6 pts), boolean chains 1 (nesting depth added 16). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.replace (cognitive 26) python/pyspark/sql/connect/dataframe.py:1515— DataFrame.replace has cognitive complexity 26 (threshold 15). Drivers by points: if/else 16 (20 pts), boolean chains 5, loops 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
types._check_arrow_array_timestamps_localize (cognitive 26) python/pyspark/sql/pandas/types.py:554— types._check_arrow_array_timestamps_localize has cognitive complexity 26 (threshold 15). Drivers by points: if/else 11 (15 pts), boolean chains 7, ternaries 2 (4 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DefaultPartitionCoalescer.throwBalls (cognitive 25) core/src/main/scala/org/apache/spark/rdd/CoalescedRDD.scala:329— DefaultPartitionCoalescer.throwBalls has cognitive complexity 25 (threshold 15). Drivers by points: loops 5 (12 pts), if/else 6 (11 pts), boolean chains 2 (nesting depth added 12). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
PipedRDD.compute (cognitive 25) core/src/main/scala/org/apache/spark/rdd/PipedRDD.scala:68— PipedRDD.compute has cognitive complexity 25 (threshold 15). Drivers by points: if/else 14 (16 pts), error handling 3 (4 pts), loops 3 (4 pts), match/switch 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
Utils.getCallSite (cognitive 25) core/src/main/scala/org/apache/spark/util/Utils.scala:1482— Utils.getCallSite has cognitive complexity 25 (threshold 15). Drivers by points: if/else 11 (24 pts), boolean chains 1 (nesting depth added 13). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
Utils.nthSmallest (cognitive 25) core/src/main/scala/org/apache/spark/util/Utils.scala:3176— Utils.nthSmallest has cognitive complexity 25 (threshold 15). Drivers by points: if/else 6 (12 pts), loops 4 (12 pts), boolean chains 1 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GradientBoostedTrees.boost (cognitive 25) mllib/src/main/scala/org/apache/spark/ml/tree/impl/GradientBoostedTrees.scala:290— GradientBoostedTrees.boost has cognitive complexity 25 (threshold 15). Drivers by points: if/else 12 (21 pts), boolean chains 3, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GradientDescent.runMiniBatchSGD (cognitive 25) mllib/src/main/scala/org/apache/spark/mllib/optimization/GradientDescent.scala:188— GradientDescent.runMiniBatchSGD has cognitive complexity 25 (threshold 15). Drivers by points: if/else 13 (21 pts), boolean chains 3, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BasicExecutorFeatureStep.configurePod (cognitive 25) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala:111— BasicExecutorFeatureStep.configurePod has cognitive complexity 25 (threshold 15). Drivers by points: if/else 20 (22 pts), match/switch 2, boolean chains 1 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
TableOutputResolver.resolveMapType (cognitive 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableOutputResolver.scala:730— TableOutputResolver.resolveMapType has cognitive complexity 25 (threshold 15). Drivers by points: if/else 17 (23 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InjectRuntimeFilter.extractBeneficialFilterCreatePlan (cognitive 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:306— InjectRuntimeFilter.extractBeneficialFilterCreatePlan has cognitive complexity 25 (threshold 15). Drivers by points: if/else 11 (15 pts), match/switch 3 (7 pts), boolean chains 3 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ConvertToCatalyst.applyExpr (cognitive 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:1307— ConvertToCatalyst.applyExpr has cognitive complexity 25 (threshold 15). Drivers by points: match/switch 4 (12 pts), if/else 8 (9 pts), boolean chains 4 (nesting depth added 9). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
OptimizeUpdateFields.optimizeFieldOps (cognitive 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UpdateFields.scala:42— OptimizeUpdateFields.optimizeFieldOps has cognitive complexity 25 (threshold 15). Drivers by points: match/switch 7 (14 pts), if/else 10 (11 pts) (nesting depth added 8). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AstBuilder.visitCommonSelectQueryClausePlan (cognitive 25) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1936— AstBuilder.visitCommonSelectQueryClausePlan has cognitive complexity 25 (threshold 15). Drivers by points: if/else 15 (17 pts), match/switch 3 (7 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
OptimizeSkewedJoin.tryOptimizeJoinChildren (cognitive 25) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeSkewedJoin.scala:102— OptimizeSkewedJoin.tryOptimizeJoinChildren has cognitive complexity 25 (threshold 15). Drivers by points: if/else 9 (15 pts), loops 3 (6 pts), boolean chains 4 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ShufflePartitionsUtil.coalescePartitionsAndGetSpecs (cognitive 25) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShufflePartitionsUtil.scala:263— ShufflePartitionsUtil.coalescePartitionsAndGetSpecs has cognitive complexity 25 (threshold 15). Drivers by points: if/else 10 (16 pts), boolean chains 6, loops 2 (3 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PartitioningUtils.parsePartitions (cognitive 25) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningUtils.scala:117— PartitioningUtils.parsePartitions has cognitive complexity 25 (threshold 15). Drivers by points: if/else 11 (16 pts), match/switch 2 (4 pts), error handling 1 (3 pts), boolean chains 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkShreddingUtils.getFieldsToExtract (cognitive 25) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala:724— SparkShreddingUtils.getFieldsToExtract has cognitive complexity 25 (threshold 15). Drivers by points: if/else 7 (14 pts), match/switch 2 (5 pts), boolean chains 3, loops 1 (3 pts) (nesting depth added 12). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
V2ScanRelationPushDown.pushDownLimitAndOffset (cognitive 25) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:1172— V2ScanRelationPushDown.pushDownLimitAndOffset has cognitive complexity 25 (threshold 15). Drivers by points: if/else 14 (24 pts), match/switch 1 (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StateDataSource.runStateVarChecks (cognitive 25) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala:201— StateDataSource.runStateVarChecks has cognitive complexity 25 (threshold 15). Drivers by points: if/else 11 (16 pts), match/switch 3 (6 pts), boolean chains 3 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HashJoin.outerJoin (cognitive 25) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala:215— HashJoin.outerJoin has cognitive complexity 25 (threshold 15). Drivers by points: if/else 11 (20 pts), boolean chains 3, loops 1 (2 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HDFSBackedStateStoreProvider.updateFromDeltaFile (cognitive 25) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:848— HDFSBackedStateStoreProvider.updateFromDeltaFile has cognitive complexity 25 (threshold 15). Drivers by points: if/else 11 (23 pts), error handling 1, loops 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ColumnarBatchRow.get (cognitive 25) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:187— ColumnarBatchRow.get has cognitive complexity 25 (threshold 15). Drivers by points: if/else 22, boolean chains 3. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
TaskMemoryManager.acquireExecutionMemory (cognitive 25) core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:197— TaskMemoryManager.acquireExecutionMemory has cognitive complexity 25 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 2 (4 pts), ternaries 1 (4 pts), boolean chains 2 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AwsStsCredentialProvider.resolve (cognitive 25) connector/credential-aws/src/main/java/org/apache/spark/security/aws/AwsStsCredentialProvider.java:310— AwsStsCredentialProvider.resolve has cognitive complexity 25 (threshold 15). Drivers by points: if/else 13 (16 pts), boolean chains 7, error handling 2 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
VariantBuilder.finishWritingObject (cognitive 25) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:425— VariantBuilder.finishWritingObject has cognitive complexity 25 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 4 (8 pts), ternaries 2 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
generate_srs_registry.parse_sql_values (cognitive 25) REDACTED:114— generate_srs_registry.parse_sql_values has cognitive complexity 25 (threshold 15). Drivers by points: if/else 6 (13 pts), loops 4 (9 pts), boolean chains 3 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowGroupedAggIncrementalFinalUDFHandler.run (cognitive 25) python/pyspark/eval_handlers/_arrow.py:558— ArrowGroupedAggIncrementalFinalUDFHandler.run has cognitive complexity 25 (threshold 15). Drivers by points: loops 5 (12 pts), ternaries 2 (8 pts), if/else 1 (5 pts) (nesting depth added 17). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
java_gateway.launch_gateway (cognitive 25) REDACTED:39— java_gateway.launch_gateway has cognitive complexity 25 (threshold 15). Drivers by points: if/else 10 (15 pts), loops 2 (5 pts), ternaries 2 (4 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PandasOnSparkFrameMethods.transform_batch (cognitive 25) python/pyspark/pandas/accessors.py:432— PandasOnSparkFrameMethods.transform_batch has cognitive complexity 25 (threshold 15). Drivers by points: if/else 12 (24 pts), boolean chains 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
IndexOpsMixin.factorize (cognitive 25) python/pyspark/pandas/base.py:1649— IndexOpsMixin.factorize has cognitive complexity 25 (threshold 15). Drivers by points: if/else 14 (20 pts), loops 2 (4 pts), ternaries 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame._map_series_op (cognitive 25) python/pyspark/pandas/frame.py:921— DataFrame._map_series_op has cognitive complexity 25 (threshold 15). Drivers by points: if/else 9 (17 pts), loops 2 (6 pts), boolean chains 2 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.quantile (cognitive 25) python/pyspark/pandas/frame.py:13007— DataFrame.quantile has cognitive complexity 25 (threshold 15). Drivers by points: if/else 8 (13 pts), loops 4 (8 pts), boolean chains 2, ternaries 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.fillna (cognitive 25) python/pyspark/sql/connect/dataframe.py:1365— DataFrame.fillna has cognitive complexity 25 (threshold 15). Drivers by points: if/else 11 (20 pts), loops 2 (5 pts) (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowBatchTransformer.enforce_schema (cognitive 25) python/pyspark/sql/conversion.py:230— ArrowBatchTransformer.enforce_schema has cognitive complexity 25 (threshold 15). Drivers by points: if/else 10 (13 pts), ternaries 2 (6 pts), error handling 1 (3 pts), boolean chains 2, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
runPlanVizSearch (cognitive 25) REDACTED:933— runPlanVizSearch has cognitive complexity 25 (threshold 15). Drivers by points: if/else 11 (14 pts), boolean chains 8, loops 1 (2 pts), ternaries 1 (nesting depth added 4). Most of this is not in the body itself: 6 of the 25 points are its own statements and the rest belongs to 3 function literals inside it that branch (lines 951, 980, 979). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
SparkThrowableHelper.getMessage (cognitive 24) common/utils/src/main/scala/org/apache/spark/SparkThrowableHelper.scala:113— SparkThrowableHelper.getMessage has cognitive complexity 24 (threshold 15). Drivers by points: if/else 8 (20 pts), match/switch 2 (4 pts) (nesting depth added 14). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkSubmitArguments.validateSubmitArguments (cognitive 24) core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala:278— SparkSubmitArguments.validateSubmitArguments has cognitive complexity 24 (threshold 15). Drivers by points: if/else 12 (13 pts), boolean chains 11 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
CoarseGrainedExecutorBackend.decommissionSelf (cognitive 24) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:318— CoarseGrainedExecutorBackend.decommissionSelf has cognitive complexity 24 (threshold 15). Drivers by points: if/else 12 (18 pts), boolean chains 4, error handling 1, loops 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DAGScheduler.handleJobSubmitted (cognitive 24) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:2619— DAGScheduler.handleJobSubmitted has cognitive complexity 24 (threshold 15). Drivers by points: if/else 14 (17 pts), boolean chains 3, error handling 3, match/switch 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
DriverEndpoint.receiveAndReply (cognitive 24) core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala:261— DriverEndpoint.receiveAndReply has cognitive complexity 24 (threshold 15). Drivers by points: if/else 9 (18 pts), loops 2 (4 pts), boolean chains 1, match/switch 1 (nesting depth added 11). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ShuffleBlockPusher.prepareBlockPushRequests (cognitive 24) core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala:395— ShuffleBlockPusher.prepareBlockPushRequests has cognitive complexity 24 (threshold 15). Drivers by points: if/else 8 (22 pts), boolean chains 1, loops 1 (nesting depth added 14). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BlockManagerInfo.updateBlockInfo (cognitive 24) core/src/main/scala/org/apache/spark/storage/BlockManagerMasterEndpoint.scala:1200— BlockManagerInfo.updateBlockInfo has cognitive complexity 24 (threshold 15). Drivers by points: if/else 13 (23 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ShuffleBlockFetcherIterator.fetchUpToMaxBytes (cognitive 24) core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala:1182— ShuffleBlockFetcherIterator.fetchUpToMaxBytes has cognitive complexity 24 (threshold 15). Drivers by points: if/else 9 (14 pts), loops 3 (6 pts), boolean chains 4 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ShuffleBlockFetcherIterator.mergeContinuousShuffleBlockIdsIfNeeded (cognitive 24) core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala:1471— ShuffleBlockFetcherIterator.mergeContinuousShuffleBlockIdsIfNeeded has cognitive complexity 24 (threshold 15). Drivers by points: if/else 10 (21 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 12). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
Utils.splitCommandString (cognitive 24) core/src/main/scala/org/apache/spark/util/Utils.scala:1685— Utils.splitCommandString has cognitive complexity 24 (threshold 15). Drivers by points: if/else 13 (21 pts), boolean chains 2, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparseMatrix.toSparseMatrix (cognitive 24) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala:789— SparseMatrix.toSparseMatrix has cognitive complexity 24 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 2 (7 pts), boolean chains 2 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MinHashLSHModel.keyDistance (cognitive 24) mllib/src/main/scala/org/apache/spark/ml/feature/MinHashLSH.scala:82— MinHashLSHModel.keyDistance has cognitive complexity 24 (threshold 15). Drivers by points: if/else 13 (22 pts), boolean chains 1, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RandomForest.runBagged (cognitive 24) mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala:118— RandomForest.runBagged has cognitive complexity 24 (threshold 15). Drivers by points: if/else 10 (19 pts), loops 2, match/switch 2, boolean chains 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LocalKMeans.kMeansPlusPlus (cognitive 24) mllib/src/main/scala/org/apache/spark/mllib/clustering/LocalKMeans.scala:37— LocalKMeans.kMeansPlusPlus has cognitive complexity 24 (threshold 15). Drivers by points: if/else 7 (12 pts), loops 6 (10 pts), boolean chains 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveAggregateFunctions.buildAggExprList (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:3308— ResolveAggregateFunctions.buildAggExprList has cognitive complexity 24 (threshold 15). Drivers by points: if/else 9 (14 pts), match/switch 4 (10 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ApplyCharTypePaddingHelper.singleNodePaddingForStringComparison (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyCharTypePaddingHelper.scala:81— ApplyCharTypePaddingHelper.singleNodePaddingForStringComparison has cognitive complexity 24 (threshold 15). Drivers by points: match/switch 6 (14 pts), if/else 7 (9 pts), boolean chains 1 (nesting depth added 10). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AssignmentUtils.applyAssignments (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AssignmentUtils.scala:167— AssignmentUtils.applyAssignments has cognitive complexity 24 (threshold 15). Drivers by points: if/else 12 (18 pts), boolean chains 3, match/switch 1 (3 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RewriteMergeIntoTable.apply (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala:46— RewriteMergeIntoTable.apply has cognitive complexity 24 (threshold 15). Drivers by points: match/switch 7 (16 pts), if/else 5, boolean chains 3 (nesting depth added 9). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TypeCoercionHelper.findWiderDateTimeType (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:358— TypeCoercionHelper.findWiderDateTimeType has cognitive complexity 24 (threshold 15). Drivers by points: if/else 10 (19 pts), boolean chains 4, match/switch 1 (nesting depth added 9). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ValidateSubqueryExpression.checkCorrelationsInSubquery (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ValidateSubqueryExpression.scala:246— ValidateSubqueryExpression.checkCorrelationsInSubquery has cognitive complexity 24 (threshold 15). Drivers by points: if/else 10 (12 pts), boolean chains 7, match/switch 4 (5 pts) (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ExternalCatalogUtils.unescapePathName (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalogUtils.scala:100— ExternalCatalogUtils.unescapePathName has cognitive complexity 24 (threshold 15). Drivers by points: if/else 9 (17 pts), boolean chains 5, loops 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ToTimestamp.eval (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala:2155— ToTimestamp.eval has cognitive complexity 24 (threshold 15). Drivers by points: if/else 8 (17 pts), error handling 1 (5 pts), match/switch 1 (2 pts) (nesting depth added 14). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
MapObjects.doGenCode (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala:1089— MapObjects.doGenCode has cognitive complexity 24 (threshold 15). Drivers by points: if/else 12 (14 pts), match/switch 5 (6 pts), boolean chains 4 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
RemoveRedundantAliases.removeRedundantAliases (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:715— RemoveRedundantAliases.removeRedundantAliases has cognitive complexity 24 (threshold 15). Drivers by points: if/else 9 (13 pts), match/switch 5 (10 pts), boolean chains 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
IntervalUtils.castStringToDTInterval (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/IntervalUtils.scala:275— IntervalUtils.castStringToDTInterval has cognitive complexity 24 (threshold 15). Drivers by points: match/switch 5 (11 pts), boolean chains 8, if/else 5 (nesting depth added 6). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
QuantileSummaries.merge (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/QuantileSummaries.scala:154— QuantileSummaries.merge has cognitive complexity 24 (threshold 15). Drivers by points: if/else 9 (17 pts), loops 3 (6 pts), boolean chains 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ToNumberParser.formatDigitGroups (cognitive 24) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/ToNumberParser.scala:788— ToNumberParser.formatDigitGroups has cognitive complexity 24 (threshold 15). Drivers by points: match/switch 4 (12 pts), loops 3 (7 pts), if/else 2 (4 pts), boolean chains 1 (nesting depth added 14). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ArtifactManager.addArtifact (cognitive 24) sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala:193— ArtifactManager.addArtifact has cognitive complexity 24 (threshold 15). Drivers by points: if/else 13 (22 pts), boolean chains 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Decoder.decompress (cognitive 24) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala:485— Decoder.decompress has cognitive complexity 24 (threshold 15). Drivers by points: if/else 9 (19 pts), loops 2 (4 pts), match/switch 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
FilePartitionReader.next (cognitive 24) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FilePartitionReader.scala:41— FilePartitionReader.next has cognitive complexity 24 (threshold 15). Drivers by points: error handling 7 (15 pts), if/else 5 (6 pts), boolean chains 3 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PartitionPruning.prune (cognitive 24) sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala:315— PartitionPruning.prune has cognitive complexity 24 (threshold 15). Drivers by points: if/else 7 (14 pts), boolean chains 5, match/switch 3 (5 pts) (nesting depth added 9). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ShuffleExchangeExec.prepareShuffleDependency (cognitive 24) sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/ShuffleExchangeExec.scala:359— ShuffleExchangeExec.prepareShuffleDependency has cognitive complexity 24 (threshold 15). Drivers by points: if/else 12 (13 pts), boolean chains 7, loops 1 (2 pts), match/switch 2 (nesting depth added 2). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SortMergeJoinScanner.findNextInnerJoinRows (cognitive 24) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala:1213— SortMergeJoinScanner.findNextInnerJoinRows has cognitive complexity 24 (threshold 15). Drivers by points: if/else 11 (17 pts), boolean chains 4, loops 2 (3 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ReuseExchangeAndSubquery.apply (cognitive 24) sql/core/src/main/scala/org/apache/spark/sql/execution/reuse/ReuseExchangeAndSubquery.scala:38— ReuseExchangeAndSubquery.apply has cognitive complexity 24 (threshold 15). Drivers by points: if/else 9 (18 pts), match/switch 2 (5 pts), boolean chains 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RocksDB.rangeScan (cognitive 24) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1798— RocksDB.rangeScan has cognitive complexity 24 (threshold 15). Drivers by points: if/else 16 (22 pts), match/switch 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HiveExternalCatalog.alterTable (cognitive 24) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:609— HiveExternalCatalog.alterTable has cognitive complexity 24 (threshold 15). Drivers by points: if/else 14 (21 pts), error handling 1 (2 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ColumnarBatchRow.copy (cognitive 24) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:51— ColumnarBatchRow.copy has cognitive complexity 24 (threshold 15). Drivers by points: if/else 20 (23 pts), loops 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ColumnarRow.copy (cognitive 24) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:56— ColumnarRow.copy has cognitive complexity 24 (threshold 15). Drivers by points: if/else 20 (23 pts), loops 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ColumnarRow.get (cognitive 24) sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java:192— ColumnarRow.get has cognitive complexity 24 (threshold 15). Drivers by points: if/else 22, boolean chains 2. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SortState.gallopLeft (cognitive 24) core/src/main/java/org/apache/spark/util/collection/TimSort.java:540— SortState.gallopLeft has cognitive complexity 24 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 3 (5 pts), boolean chains 4 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (SortState.gallopRight) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
SortState.gallopRight (cognitive 24) core/src/main/java/org/apache/spark/util/collection/TimSort.java:611— SortState.gallopRight has cognitive complexity 24 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 3 (5 pts), boolean chains 4 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (SortState.gallopLeft) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
base.align_diff_index_ops (cognitive 24) python/pyspark/pandas/base.py:85— base.align_diff_index_ops has cognitive complexity 24 (threshold 15). Drivers by points: if/else 8 (13 pts), ternaries 3 (9 pts), boolean chains 2 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame._reduce_for_stat_function (cognitive 24) python/pyspark/pandas/frame.py:770— DataFrame._reduce_for_stat_function has cognitive complexity 24 (threshold 15). Drivers by points: if/else 8 (18 pts), boolean chains 2, loops 1 (2 pts), ternaries 1 (2 pts) (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame._fillna_with_method (cognitive 24) python/pyspark/pandas/frame.py:6415— DataFrame._fillna_with_method has cognitive complexity 24 (threshold 15). Drivers by points: if/else 11 (18 pts), loops 1 (3 pts), ternaries 1 (3 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
typehints.infer_pandas_eval_type (cognitive 24) python/pyspark/sql/pandas/typehints.py:42— typehints.infer_pandas_eval_type has cognitive complexity 24 (threshold 15). Drivers by points: boolean chains 12, if/else 10 (11 pts), loops 1 (nesting depth added 1). To reduce it, name the conditions: bind each compound test to a well-named local or a small predicate function, so the body reads as a sequence of named decisions rather than a chain of operators.
worker.invoke_udf (cognitive 24) python/pyspark/worker.py:97— worker.invoke_udf has cognitive complexity 24 (threshold 15). Drivers by points: if/else 11 (17 pts), loops 1 (3 pts), error handling 2, boolean chains 1, ternaries 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
spark-sql-viz.document.addEventListener("DOMContentLoaded") (cognitive 24) REDACTED:1093— spark-sql-viz.document.addEventListener("DOMContentLoaded") has cognitive complexity 24 (threshold 15). Drivers by points: if/else 14 (16 pts), boolean chains 6, ternaries 1 (2 pts) (nesting depth added 3). Most of this is not in the body itself: 6 of the 24 points are its own statements and the rest belongs to 4 function literals inside it that branch (lines 1118, 1139, 1142, …). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
(anonymous) (cognitive 24) REDACTED:1093— (anonymous) has cognitive complexity 24 (threshold 15). Drivers by points: if/else 14 (16 pts), boolean chains 6, ternaries 1 (2 pts) (nesting depth added 3). Most of this is not in the body itself: 6 of the 24 points are its own statements and the rest belongs to 7 function literals inside it that branch (lines 1118, 1139, 1142, …). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
MapOutputTrackerMaster.getPreferredLocationsForShuffle (cognitive 23) core/src/main/scala/org/apache/spark/MapOutputTracker.scala:1179— MapOutputTrackerMaster.getPreferredLocationsForShuffle has cognitive complexity 23 (threshold 15). Drivers by points: if/else 12 (21 pts), boolean chains 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BasePythonRunner.createPipelinedDataIn (cognitive 23) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:767— BasePythonRunner.createPipelinedDataIn has cognitive complexity 23 (threshold 15). Drivers by points: if/else 10 (19 pts), error handling 1 (2 pts), boolean chains 1, loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Worker.reregisterWithMaster (cognitive 23) core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala:346— Worker.reregisterWithMaster has cognitive complexity 23 (threshold 15). Drivers by points: if/else 8 (15 pts), error handling 2 (6 pts), match/switch 1 (2 pts) (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ReplayListenerBus.replayEntries (cognitive 23) core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:201— ReplayListenerBus.replayEntries has cognitive complexity 23 (threshold 15). Drivers by points: error handling 8 (11 pts), if/else 4 (10 pts), boolean chains 1, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExternalSorter.mergeWithAggregation (cognitive 23) core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala:411— ExternalSorter.mergeWithAggregation has cognitive complexity 23 (threshold 15). Drivers by points: if/else 6 (13 pts), loops 3 (7 pts), boolean chains 3 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Literal.valueSizeInBytes (cognitive 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala:313— Literal.valueSizeInBytes has cognitive complexity 23 (threshold 15). Drivers by points: loops 4 (9 pts), match/switch 3 (7 pts), if/else 4 (6 pts), boolean chains 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PushDownLeftSemiAntiJoin.apply (cognitive 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushDownLeftSemiAntiJoin.scala:38— PushDownLeftSemiAntiJoin.apply has cognitive complexity 23 (threshold 15). Drivers by points: if/else 8 (15 pts), boolean chains 4, match/switch 2 (4 pts) (nesting depth added 9). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AstBuilder.visitOperatorPipeAggregate (cognitive 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:8371— AstBuilder.visitOperatorPipeAggregate has cognitive complexity 23 (threshold 15). Drivers by points: match/switch 6 (11 pts), if/else 6 (9 pts), boolean chains 3 (nesting depth added 8). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StringUtils.stripComment (cognitive 23) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala:164— StringUtils.stripComment has cognitive complexity 23 (threshold 15). Drivers by points: if/else 8 (17 pts), boolean chains 3, match/switch 1 (2 pts), loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PipelinesHandler.buildAutoCdcFlow (cognitive 23) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala:440— PipelinesHandler.buildAutoCdcFlow has cognitive complexity 23 (threshold 15). Drivers by points: if/else 18, boolean chains 3, match/switch 2. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkSqlAstBuilder.visitManageResource (cognitive 23) REDACTED:714— SparkSqlAstBuilder.visitManageResource has cognitive complexity 23 (threshold 15). Drivers by points: if/else 10 (16 pts), match/switch 4 (6 pts), boolean chains 1 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
FilterExec.doConsume (cognitive 23) sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala:451— FilterExec.doConsume has cognitive complexity 23 (threshold 15). Drivers by points: if/else 8 (10 pts), match/switch 3 (7 pts), boolean chains 6 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SupportsArchiveFormat.readArchiveEntries (cognitive 23) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:375— SupportsArchiveFormat.readArchiveEntries has cognitive complexity 23 (threshold 15). Drivers by points: if/else 6 (9 pts), error handling 3 (6 pts), boolean chains 3, loops 2 (3 pts), match/switch 1 (2 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
OrcDeserializer.newWriter (cognitive 23) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcDeserializer.scala:112— OrcDeserializer.newWriter has cognitive complexity 23 (threshold 15). Drivers by points: if/else 8 (14 pts), loops 3 (6 pts), match/switch 2 (3 pts) (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
InferVariantShreddingSchema.buildSchemaFromStats (cognitive 23) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/InferVariantShreddingSchema.scala:491— InferVariantShreddingSchema.buildSchemaFromStats has cognitive complexity 23 (threshold 15). Drivers by points: if/else 12 (16 pts), match/switch 3 (5 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AnalyzedSchemaProjection.projectToType (cognitive 23) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/AnalyzedSchemaProjection.scala:125— AnalyzedSchemaProjection.projectToType has cognitive complexity 23 (threshold 15). Drivers by points: if/else 15 (22 pts), match/switch 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
V2ScanRelationPushDown.pushDownLimit (cognitive 23) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:1109— V2ScanRelationPushDown.pushDownLimit has cognitive complexity 23 (threshold 15). Drivers by points: if/else 10 (18 pts), match/switch 2 (4 pts), boolean chains 1 (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SchemaUtil.isValidSchema (cognitive 23) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/utils/SchemaUtil.scala:258— SchemaUtil.isValidSchema has cognitive complexity 23 (threshold 15). Drivers by points: if/else 15 (21 pts), match/switch 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GraphValidations.validateFlowStreamingness (cognitive 23) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/GraphValidations.scala:104— GraphValidations.validateFlowStreamingness has cognitive complexity 23 (threshold 15). Drivers by points: if/else 9 (17 pts), match/switch 3 (6 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ParquetBlockAccounting.compressedBytesForRowRanges (cognitive 23) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetBlockAccounting.java:91— ParquetBlockAccounting.compressedBytesForRowRanges has cognitive complexity 23 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 2 (3 pts), boolean chains 2, error handling 1 (2 pts), ternaries 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExpressionImplUtils.jaroWinklerSimilarity (cognitive 23) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionImplUtils.java:540— ExpressionImplUtils.jaroWinklerSimilarity has cognitive complexity 23 (threshold 15). Drivers by points: if/else 8 (13 pts), loops 5 (7 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SpecializedGettersReader.read (cognitive 23) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/SpecializedGettersReader.java:29— SpecializedGettersReader.read has cognitive complexity 23 (threshold 15). Drivers by points: if/else 20, boolean chains 3. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkSubmitOptionParser.parse (cognitive 23) launcher/src/main/java/org/apache/spark/launcher/SparkSubmitOptionParser.java:146— SparkSubmitOptionParser.parse has cognitive complexity 23 (threshold 15). Drivers by points: if/else 9 (22 pts), loops 1 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Distributor._get_num_tasks (cognitive 23) python/pyspark/ml/torch/distributor.py:195— Distributor._get_num_tasks has cognitive complexity 23 (threshold 15). Drivers by points: if/else 9 (23 pts) (nesting depth added 14). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
DataFrame.apply (cognitive 23) python/pyspark/pandas/frame.py:3016— DataFrame.apply has cognitive complexity 23 (threshold 15). Drivers by points: if/else 13 (23 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Series._fillna (cognitive 23) python/pyspark/pandas/series.py:2255— Series._fillna has cognitive complexity 23 (threshold 15). Drivers by points: if/else 13 (20 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame._to_cols (cognitive 23) python/pyspark/sql/connect/dataframe.py:218— DataFrame._to_cols has cognitive complexity 23 (threshold 15). Drivers by points: if/else 10 (20 pts), boolean chains 2, loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
WindowExpression.to_plan (cognitive 23) python/pyspark/sql/connect/expressions.py:1119— WindowExpression.to_plan has cognitive complexity 23 (threshold 15). Drivers by points: if/else 14 (23 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PandasConversionMixin._to_pandas (cognitive 23) python/pyspark/sql/pandas/conversion.py:318— PandasConversionMixin._to_pandas has cognitive complexity 23 (threshold 15). Drivers by points: if/else 11 (16 pts), error handling 2 (5 pts), ternaries 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UserDefinedFunction.__call__ (cognitive 23) python/pyspark/sql/udf.py:573— UserDefinedFunction.__call__ has cognitive complexity 23 (threshold 15). Drivers by points: if/else 10 (17 pts), boolean chains 4, loops 1 (2 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
makeSortable::innerSortFunction (cognitive 23) REDACTED:92— makeSortable::innerSortFunction has cognitive complexity 23 (threshold 15). Drivers by points: if/else 5 (10 pts), loops 5 (7 pts), ternaries 3 (5 pts), boolean chains 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MavenUtils.resolveMavenCoordinatesInternal (cognitive 22) common/utils/src/main/scala/org/apache/spark/util/MavenUtils.scala:633— MavenUtils.resolveMavenCoordinatesInternal has cognitive complexity 22 (threshold 15). Drivers by points: if/else 11 (20 pts), boolean chains 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExecutorAllocationManager.doUpdateRequest (cognitive 22) core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:558— ExecutorAllocationManager.doUpdateRequest has cognitive complexity 22 (threshold 15). Drivers by points: if/else 10 (19 pts), error handling 1 (2 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkEnv.create (cognitive 22) core/src/main/scala/org/apache/spark/SparkEnv.scala:689— SparkEnv.create has cognitive complexity 22 (threshold 15). Drivers by points: if/else 21, boolean chains 1. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
BasePythonRunner.compute (cognitive 22) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:644— BasePythonRunner.compute has cognitive complexity 22 (threshold 15). Drivers by points: if/else 16 (17 pts), boolean chains 3, error handling 1 (2 pts) (nesting depth added 2). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
Inbox.process (cognitive 22) core/src/main/scala/org/apache/spark/rpc/netty/Inbox.scala:86— Inbox.process has cognitive complexity 22 (threshold 15). Drivers by points: if/else 7 (14 pts), error handling 1 (3 pts), boolean chains 2, match/switch 1 (2 pts), loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DAGScheduler.checkPipelinedShuffleRepeatedReads (cognitive 22) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:1531— DAGScheduler.checkPipelinedShuffleRepeatedReads has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (15 pts), match/switch 2 (5 pts), loops 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HighlyCompressedMapStatus.apply (cognitive 22) core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala:314— HighlyCompressedMapStatus.apply has cognitive complexity 22 (threshold 15). Drivers by points: if/else 14 (17 pts), boolean chains 4, loops 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
Utils.startServiceOnPort (cognitive 22) core/src/main/scala/org/apache/spark/util/Utils.scala:2269— Utils.startServiceOnPort has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (17 pts), boolean chains 2, error handling 1 (2 pts), loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LSHModel.approxNearestNeighbors (cognitive 22) mllib/src/main/scala/org/apache/spark/ml/feature/LSH.scala:115— LSHModel.approxNearestNeighbors has cognitive complexity 22 (threshold 15). Drivers by points: if/else 10 (19 pts), loops 1 (3 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RandomForest.binsToBestSplit (cognitive 22) mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala:915— RandomForest.binsToBestSplit has cognitive complexity 22 (threshold 15). Drivers by points: if/else 14 (18 pts), loops 2 (4 pts) (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
BisectingKMeans.runWithWeight (cognitive 22) mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala:157— BisectingKMeans.runWithWeight has cognitive complexity 22 (threshold 15). Drivers by points: if/else 12 (16 pts), loops 2 (4 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Word2Vec.createBinaryTree (cognitive 22) mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala:227— Word2Vec.createBinaryTree has cognitive complexity 22 (threshold 15). Drivers by points: if/else 8 (14 pts), loops 6 (8 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExecutorPodsLifecycleManager.onNewSnapshots (cognitive 22) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala:104— ExecutorPodsLifecycleManager.onNewSnapshots has cognitive complexity 22 (threshold 15). Drivers by points: if/else 11 (17 pts), boolean chains 3, match/switch 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExecutorPodsSnapshot.toState (cognitive 22) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsSnapshot.scala:76— ExecutorPodsSnapshot.toState has cognitive complexity 22 (threshold 15). Drivers by points: match/switch 3 (11 pts), if/else 7 (9 pts), boolean chains 2 (nesting depth added 10). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ApplyCharTypePaddingHelper.padAttrLitCmp (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ApplyCharTypePaddingHelper.scala:164— ApplyCharTypePaddingHelper.padAttrLitCmp has cognitive complexity 22 (threshold 15). Drivers by points: if/else 11 (20 pts), match/switch 1 (2 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CTESubstitution.resolveCTERelations (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala:264— CTESubstitution.resolveCTERelations has cognitive complexity 22 (threshold 15). Drivers by points: if/else 12 (19 pts), boolean chains 2, loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
FunctionResolution.lookupFunctionType (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala:569— FunctionResolution.lookupFunctionType has cognitive complexity 22 (threshold 15). Drivers by points: if/else 6 (9 pts), error handling 2 (6 pts), match/switch 1 (3 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TimeWindowing.apply (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala:63— TimeWindowing.apply has cognitive complexity 22 (threshold 15). Drivers by points: match/switch 5 (13 pts), if/else 5 (8 pts), boolean chains 1 (nesting depth added 11). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident. This shape REPEATS in the file: one other method here (SessionWindowing.apply) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
SessionWindowing.apply (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTimeWindows.scala:114— SessionWindowing.apply has cognitive complexity 22 (threshold 15). Drivers by points: match/switch 5 (13 pts), if/else 5 (8 pts), boolean chains 1 (nesting depth added 11). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident. This shape REPEATS in the file: one other method here (TimeWindowing.apply) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
WidenSetOperationTypes.apply (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionBase.scala:192— WidenSetOperationTypes.apply has cognitive complexity 22 (threshold 15). Drivers by points: if/else 10 (16 pts), boolean chains 5, match/switch 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
Pmod.doGenCode (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1213— Pmod.doGenCode has cognitive complexity 22 (threshold 15). Drivers by points: if/else 13 (18 pts), boolean chains 3, match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JdkCodeCompiler.narrowingVerdict (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:1226— JdkCodeCompiler.narrowingVerdict has cognitive complexity 22 (threshold 15). Drivers by points: if/else 11 (12 pts), boolean chains 8, error handling 2 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
MultiGetJsonObjectEvaluator.copyCurrentStructure (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:1741— MultiGetJsonObjectEvaluator.copyCurrentStructure has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (12 pts), error handling 2 (3 pts), match/switch 1 (3 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StarSchemaDetection.isUnique (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/StarSchemaDetection.scala:178— StarSchemaDetection.isUnique has cognitive complexity 22 (threshold 15). Drivers by points: if/else 7 (14 pts), match/switch 3 (6 pts), boolean chains 2 (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
UnionEstimation.computeColumnStats (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/UnionEstimation.scala:69— UnionEstimation.computeColumnStats has cognitive complexity 22 (threshold 15). Drivers by points: if/else 11 (15 pts), match/switch 4 (6 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CatalogV2Util.applySchemaChanges (cognitive 22) sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala:221— CatalogV2Util.applySchemaChanges has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (11 pts), match/switch 5 (11 pts) (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AlterTableChangeColumnCommand.run (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/command/ddl.scala:388— AlterTableChangeColumnCommand.run has cognitive complexity 22 (threshold 15). Drivers by points: if/else 13 (18 pts), boolean chains 2, match/switch 1 (2 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataSourceAnalysis.apply (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala:144— DataSourceAnalysis.apply has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (16 pts), boolean chains 5, match/switch 1 (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DynamicPartitionDataConcurrentWriter.write (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala:458— DynamicPartitionDataConcurrentWriter.write has cognitive complexity 22 (threshold 15). Drivers by points: if/else 12 (19 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PreprocessTableInsertion.preprocess (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala:461— PreprocessTableInsertion.preprocess has cognitive complexity 22 (threshold 15). Drivers by points: if/else 10 (12 pts), boolean chains 6, match/switch 2 (3 pts), error handling 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SchemaUtil.unifyMapStateRowPair (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/utils/SchemaUtil.scala:141— SchemaUtil.unifyMapStateRowPair has cognitive complexity 22 (threshold 15). Drivers by points: if/else 10 (18 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SortMergeAsOfJoinScanner.findNearest (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:437— SortMergeAsOfJoinScanner.findNearest has cognitive complexity 22 (threshold 15). Drivers by points: if/else 8 (18 pts), boolean chains 3, loops 1 (nesting depth added 10). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
TransformWithStateInPySparkStateServer.initializeStateVariable (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:693— TransformWithStateInPySparkStateServer.initializeStateVariable has cognitive complexity 22 (threshold 15). Drivers by points: if/else 12 (21 pts), match/switch 1 (nesting depth added 9). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StreamingSymmetricHashJoinExec.processPartitions (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala:371— StreamingSymmetricHashJoinExec.processPartitions has cognitive complexity 22 (threshold 15). Drivers by points: if/else 14, match/switch 5 (8 pts) (nesting depth added 3). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SlidingWindowMinMaxFunctionFrame.write (cognitive 22) sql/core/src/main/scala/org/apache/spark/sql/execution/window/SlidingWindowMinMaxFunctionFrame.scala:139— SlidingWindowMinMaxFunctionFrame.write has cognitive complexity 22 (threshold 15). Drivers by points: if/else 6 (10 pts), loops 5 (9 pts), boolean chains 3 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ProcessIterator.advance (cognitive 22) udf/worker/grpc/src/main/scala/org/apache/spark/udf/worker/grpc/GrpcWorkerSession.scala:923— ProcessIterator.advance has cognitive complexity 22 (threshold 15). Drivers by points: if/else 6 (11 pts), match/switch 2 (5 pts), error handling 1 (3 pts), boolean chains 2, loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HiveFunctionRegistryUtils.filterMethodsByTypeAffinity (cognitive 22) sql/hive/src/main/java/org/apache/hadoop/hive/ql/exec/HiveFunctionRegistryUtils.java:304— HiveFunctionRegistryUtils.filterMethodsByTypeAffinity has cognitive complexity 22 (threshold 15). Drivers by points: if/else 6 (16 pts), loops 2 (5 pts), boolean chains 1 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkLauncher.startApplication (cognitive 22) launcher/src/main/java/org/apache/spark/launcher/SparkLauncher.java:384— SparkLauncher.startApplication has cognitive complexity 22 (threshold 15). Drivers by points: if/else 10 (13 pts), boolean chains 5, ternaries 1 (2 pts), error handling 1, loops 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UTF8String.soundex (cognitive 22) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:2334— UTF8String.soundex has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (16 pts), boolean chains 5, loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExternalBlockHandler.handleMessage (cognitive 22) common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java:184— ExternalBlockHandler.handleMessage has cognitive complexity 22 (threshold 15). Drivers by points: if/else 14 (19 pts), error handling 1 (2 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PushBlockStreamCallback.onComplete (cognitive 22) common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:1533— PushBlockStreamCallback.onComplete has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (15 pts), error handling 2 (6 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LevelDBIterator.loadNext (cognitive 22) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:227— LevelDBIterator.loadNext has cognitive complexity 22 (threshold 15). Drivers by points: if/else 6 (12 pts), ternaries 3 (7 pts), error handling 1 (2 pts), loops 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataTypeOps.__new__ (cognitive 22) python/pyspark/pandas/data_type_ops/base.py:265— DataTypeOps.__new__ has cognitive complexity 22 (threshold 15). Drivers by points: if/else 12 (18 pts), boolean chains 4 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame._update_internal_frame (cognitive 22) python/pyspark/pandas/frame.py:668— DataFrame._update_internal_frame has cognitive complexity 22 (threshold 15). Drivers by points: if/else 8 (18 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.update (cognitive 22) python/pyspark/pandas/frame.py:9372— DataFrame.update has cognitive complexity 22 (threshold 15). Drivers by points: if/else 8 (12 pts), ternaries 2 (6 pts), loops 2 (3 pts), boolean chains 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GroupBy._prepare_return (cognitive 22) python/pyspark/pandas/groupby.py:3724— GroupBy._prepare_return has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (19 pts), loops 1 (3 pts) (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
api.create_auto_cdc_flow (cognitive 22) python/pyspark/pipelines/api.py:531— api.create_auto_cdc_flow has cognitive complexity 22 (threshold 15). Drivers by points: if/else 13 (14 pts), boolean chains 7, loops 1 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
shuffle.get_used_memory (cognitive 22) python/pyspark/shuffle.py:63— shuffle.get_used_memory has cognitive complexity 22 (threshold 15). Drivers by points: if/else 9 (17 pts), loops 1 (3 pts), boolean chains 1, error handling 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LiteralExpression._to_value (cognitive 22) python/pyspark/sql/connect/expressions.py:381— LiteralExpression._to_value has cognitive complexity 22 (threshold 15). Drivers by points: boolean chains 15, if/else 4 (7 pts) (nesting depth added 3). To reduce it, name the conditions: bind each compound test to a well-named local or a small predicate function, so the body reads as a sequence of named decisions rather than a chain of operators.
SparkSession.createDataFrame (cognitive 22) python/pyspark/sql/session.py:1427— SparkSession.createDataFrame has cognitive complexity 22 (threshold 15). Drivers by points: if/else 10 (13 pts), boolean chains 4, error handling 3, ternaries 1 (2 pts) (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
makeSortable (cognitive 22) REDACTED:42— makeSortable has cognitive complexity 22 (threshold 15). Drivers by points: if/else 16 (40 pts), loops 8 (21 pts), ternaries 3 (11 pts), boolean chains 2 (nesting depth added 45). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ClosureCleaner.clean (cognitive 21) common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:233— ClosureCleaner.clean has cognitive complexity 21 (threshold 15). Drivers by points: if/else 11 (18 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
KafkaDataConsumer.get (cognitive 21) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala:400— KafkaDataConsumer.get has cognitive complexity 21 (threshold 15). Drivers by points: if/else 10 (17 pts), error handling 1 (2 pts), boolean chains 1, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ProtobufDeserializer.checkAnyJsonRecursion (cognitive 21) connector/protobuf/src/main/scala/org/apache/spark/sql/protobuf/ProtobufDeserializer.scala:113— ProtobufDeserializer.checkAnyJsonRecursion has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (20 pts), loops 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ClientEndpoint.reportDriverStatus (cognitive 21) core/src/main/scala/org/apache/spark/deploy/Client.scala:157— ClientEndpoint.reportDriverStatus has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (13 pts), match/switch 3 (8 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DAGScheduler.classifyJobShuffleShape (cognitive 21) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:1248— DAGScheduler.classifyJobShuffleShape has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (16 pts), match/switch 1 (3 pts), boolean chains 1, loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UIUtils.makeProgressBar (cognitive 21) core/src/main/scala/org/apache/spark/ui/UIUtils.scala:509— UIUtils.makeProgressBar has cognitive complexity 21 (threshold 15). Drivers by points: if/else 20, boolean chains 1. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
RDDOperationGraph.makeOperationGraph (cognitive 21) core/src/main/scala/org/apache/spark/ui/scope/RDDOperationGraph.scala:131— RDDOperationGraph.makeOperationGraph has cognitive complexity 21 (threshold 15). Drivers by points: if/else 13 (17 pts), match/switch 2 (3 pts), boolean chains 1 (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
LogisticRegression.createInitialSolution (cognitive 21) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:816— LogisticRegression.createInitialSolution has cognitive complexity 21 (threshold 15). Drivers by points: if/else 12 (16 pts), boolean chains 2, loops 1 (2 pts), match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ALS.train (cognitive 21) mllib/src/main/scala/org/apache/spark/ml/recommendation/ALS.scala:1024— ALS.train has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (14 pts), loops 2 (4 pts), boolean chains 1, error handling 1, match/switch 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SlidingRDD.getPartitions (cognitive 21) mllib/src/main/scala/org/apache/spark/mllib/rdd/SlidingRDD.scala:68— SlidingRDD.getPartitions has cognitive complexity 21 (threshold 15). Drivers by points: if/else 7 (14 pts), loops 2 (6 pts), boolean chains 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InTypeCoercion.apply (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala:450— InTypeCoercion.apply has cognitive complexity 21 (threshold 15). Drivers by points: if/else 7 (10 pts), match/switch 4 (9 pts), boolean chains 2 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
WidenStatefulOperatorAttributeNullability.apply (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/WidenStatefulOperatorAttributeNullability.scala:135— WidenStatefulOperatorAttributeNullability.apply has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (13 pts), match/switch 3 (6 pts), boolean chains 2 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LogicalPlanDifference.apply (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/LogicalPlanDifference.scala:51— LogicalPlanDifference.apply has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (14 pts), boolean chains 4, loops 2 (3 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CSVHeaderChecker.checkHeaderColumnNames (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVHeaderChecker.scala:67— CSVHeaderChecker.checkHeaderColumnNames has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (17 pts), loops 1 (3 pts), boolean chains 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GenerateUnsafeProjection.writeExpressionsToBuffer (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeProjection.scala:118— GenerateUnsafeProjection.writeExpressionsToBuffer has cognitive complexity 21 (threshold 15). Drivers by points: if/else 10 (12 pts), match/switch 3 (6 pts), boolean chains 3 (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
JsonObjectExpr.checkInputDataTypes (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:2483— JsonObjectExpr.checkInputDataTypes has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (13 pts), match/switch 2 (5 pts), boolean chains 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InterpretedOrdering.compare (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ordering.scala:54— InterpretedOrdering.compare has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (16 pts), match/switch 1 (3 pts), boolean chains 1, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
OptimizeJoinCondition.optimizeCondition (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeJoinCondition.scala:38— OptimizeJoinCondition.optimizeCondition has cognitive complexity 21 (threshold 15). Drivers by points: if/else 7 (11 pts), boolean chains 8, match/switch 1 (2 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AstBuilder.withQueryResultClauses (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:1636— AstBuilder.withQueryResultClauses has cognitive complexity 21 (threshold 15). Drivers by points: if/else 12, boolean chains 9. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
AstBuilder.withAggregationClause (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:2234— AstBuilder.withAggregationClause has cognitive complexity 21 (threshold 15). Drivers by points: if/else 12 (19 pts), boolean chains 2 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
EstimationUtils.getOverlappedRanges (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/EstimationUtils.scala:260— EstimationUtils.getOverlappedRanges has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 7 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JoinEstimation.estimateInnerOuterJoin (cognitive 21) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala:55— JoinEstimation.estimateInnerOuterJoin has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (9 pts), match/switch 4 (9 pts), boolean chains 3 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AdaptiveSparkPlanExec.cancelObsoleteStages (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:502— AdaptiveSparkPlanExec.cancelObsoleteStages has cognitive complexity 21 (threshold 15). Drivers by points: if/else 12 (14 pts), boolean chains 3, error handling 1 (2 pts), match/switch 2 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ParquetToSparkSchemaConverter.convertGroupField (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:402— ParquetToSparkSchemaConverter.convertGroupField has cognitive complexity 21 (threshold 15). Drivers by points: if/else 10 (18 pts), boolean chains 2, match/switch 1 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StructReader.readFromTyped (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala:269— StructReader.readFromTyped has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (16 pts), boolean chains 4, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ViewSyncSchemaToMetaStore.apply (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala:745— ViewSyncSchemaToMetaStore.apply has cognitive complexity 21 (threshold 15). Drivers by points: if/else 4 (7 pts), boolean chains 6, match/switch 2 (5 pts), error handling 1 (3 pts) (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TableCapabilityCheck.apply (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TableCapabilityCheck.scala:39— TableCapabilityCheck.apply has cognitive complexity 21 (threshold 15). Drivers by points: if/else 6 (11 pts), boolean chains 5, match/switch 4 (5 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SortMergeJoinScanner.findNextOuterJoinRows (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala:1269— SortMergeJoinScanner.findNextOuterJoinRows has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (14 pts), loops 1 (4 pts), boolean chains 3 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExtractPythonUDFFromLambda.overArray (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala:584— ExtractPythonUDFFromLambda.overArray has cognitive complexity 21 (threshold 15). Drivers by points: match/switch 5 (13 pts), if/else 5 (8 pts) (nesting depth added 11). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ExtractPythonUDFs.extract (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFs.scala:302— ExtractPythonUDFs.extract has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (16 pts), match/switch 2 (5 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ProgressContext.legacyExtractSourceToNumInputRows (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/ProgressReporter.scala:482— ProgressContext.legacyExtractSourceToNumInputRows has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (12 pts), match/switch 5 (9 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RocksDBFileManager.deleteOldVersions (cognitive 21) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:659— RocksDBFileManager.deleteOldVersions has cognitive complexity 21 (threshold 15). Drivers by points: if/else 15 (16 pts), error handling 2, match/switch 2, boolean chains 1 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ThriftHttpServlet.doPost (cognitive 21) REDACTED:120— ThriftHttpServlet.doPost has cognitive complexity 21 (threshold 15). Drivers by points: if/else 12 (18 pts), boolean chains 2, error handling 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
OrcColumnarBatchReader.initBatch (cognitive 21) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/orc/OrcColumnarBatchReader.java:175— OrcColumnarBatchReader.initBatch has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (19 pts), loops 2 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VariantBuilder.appendWithDeletionImpl (cognitive 21) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:570— VariantBuilder.appendWithDeletionImpl has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (15 pts), boolean chains 4, loops 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CollationFactory.getClosestSuggestionsOnInvalidName (cognitive 21) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:1421— CollationFactory.getClosestSuggestionsOnInvalidName has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (14 pts), loops 3 (4 pts), boolean chains 2, error handling 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UTF8String.getIsValid (cognitive 21) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:469— UTF8String.getIsValid has cognitive complexity 21 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 3 (5 pts), ternaries 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LevelDBIterator.<init> (cognitive 21) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:58— LevelDBIterator.<init> has cognitive complexity 21 (threshold 15). Drivers by points: if/else 13 (19 pts), boolean chains 1, ternaries 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RocksDBIterator.loadNext (cognitive 21) common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java:216— RocksDBIterator.loadNext has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (17 pts), ternaries 1 (3 pts), loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
merge_spark_pr.reconcile_jira_affects_versions (cognitive 21) REDACTED:1379— merge_spark_pr.reconcile_jira_affects_versions has cognitive complexity 21 (threshold 15). Drivers by points: if/else 7 (11 pts), boolean chains 4, error handling 2 (4 pts), loops 1, ternaries 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
merge_spark_pr.prompt_for_components (cognitive 21) REDACTED:2159— merge_spark_pr.prompt_for_components has cognitive complexity 21 (threshold 15). Drivers by points: if/else 5 (12 pts), loops 3 (4 pts), ternaries 1 (4 pts), boolean chains 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
install.install_spark (cognitive 21) REDACTED:128— install.install_spark has cognitive complexity 21 (threshold 15). Drivers by points: if/else 9 (15 pts), error handling 2 (4 pts), loops 1, ternaries 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BooleanOps.astype (cognitive 21) python/pyspark/pandas/data_type_ops/boolean_ops.py:323— BooleanOps.astype has cognitive complexity 21 (threshold 15). Drivers by points: if/else 7 (14 pts), boolean chains 4, ternaries 1 (3 pts) (nesting depth added 9). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
Index.droplevel (cognitive 21) python/pyspark/pandas/indexes/base.py:1326— Index.droplevel has cognitive complexity 21 (threshold 15). Drivers by points: if/else 10 (20 pts), loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Index.difference (cognitive 21) python/pyspark/pandas/indexes/base.py:2031— Index.difference has cognitive complexity 21 (threshold 15). Drivers by points: if/else 11 (15 pts), boolean chains 5, ternaries 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Series._drop (cognitive 21) python/pyspark/pandas/series.py:2720— Series._drop has cognitive complexity 21 (threshold 15). Drivers by points: if/else 10 (15 pts), error handling 1 (3 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectClient._truncate (cognitive 21) python/pyspark/sql/connect/client/core.py:1353— SparkConnectClient._truncate has cognitive complexity 21 (threshold 15). Drivers by points: if/else 8 (20 pts), loops 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
functions._validate_vectorized_udf (cognitive 21) python/pyspark/sql/pandas/functions.py:882— functions._validate_vectorized_udf has cognitive complexity 21 (threshold 15). Drivers by points: if/else 10 (11 pts), boolean chains 7, error handling 1 (2 pts), ternaries 1 (nesting depth added 2). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
typehints.infer_arrow_eval_type (cognitive 21) python/pyspark/sql/pandas/typehints.py:202— typehints.infer_arrow_eval_type has cognitive complexity 21 (threshold 15). Drivers by points: if/else 10 (11 pts), boolean chains 9, loops 1 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
GoldenFileTestMixin.compare_or_generate_golden_matrix (cognitive 21) python/pyspark/testing/goldenutils.py:198— GoldenFileTestMixin.compare_or_generate_golden_matrix has cognitive complexity 21 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 3 (5 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
sql-table-utils.getSqlTableColumns (cognitive 21) sql/core/src/main/resources/org/apache/spark/sql/execution/ui/static/sql-table-utils.js:184— sql-table-utils.getSqlTableColumns has cognitive complexity 21 (threshold 15). Drivers by points: if/else 13, boolean chains 8. Most of this is not in the body itself: 1 of the 21 points is its own statement and the rest belongs to 9 function literals inside it that branch (lines 263, 202, 254, …). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
ClosureCleaner.cleanNonIndyLambdaClosure (cognitive 20) common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:325— ClosureCleaner.cleanNonIndyLambdaClosure has cognitive complexity 20 (threshold 15). Drivers by points: if/else 12 (15 pts), loops 2 (3 pts), boolean chains 2 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ExecutorAllocationManager.removeExecutors (cognitive 20) core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:675— ExecutorAllocationManager.removeExecutors has cognitive complexity 20 (threshold 15). Drivers by points: if/else 15 (19 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
PythonWorkerFactory.startDaemon (cognitive 20) core/src/main/scala/org/apache/spark/api/python/PythonWorkerFactory.scala:338— PythonWorkerFactory.startDaemon has cognitive complexity 20 (threshold 15). Drivers by points: if/else 12 (13 pts), boolean chains 4, error handling 3 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ResourceProfileManager.isSupported (cognitive 20) core/src/main/scala/org/apache/spark/resource/ResourceProfileManager.scala:69— ResourceProfileManager.isSupported has cognitive complexity 20 (threshold 15). Drivers by points: boolean chains 12, if/else 5 (8 pts) (nesting depth added 3). To reduce it, name the conditions: bind each compound test to a well-named local or a small predicate function, so the body reads as a sequence of named decisions rather than a chain of operators.
ResourceUtils.warnOnWastedResources (cognitive 20) core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala:424— ResourceUtils.warnOnWastedResources has cognitive complexity 20 (threshold 15). Drivers by points: if/else 12 (17 pts), boolean chains 3 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExecutorMonitor.onBlockUpdated (cognitive 20) core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala:383— ExecutorMonitor.onBlockUpdated has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 3, match/switch 1 (3 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ChannelShuffleWriter.write (cognitive 20) core/src/main/scala/org/apache/spark/shuffle/local/pipelined/ChannelShuffleWriterReader.scala:136— ChannelShuffleWriter.write has cognitive complexity 20 (threshold 15). Drivers by points: if/else 8 (18 pts), loops 2 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AppStatusListener.updateRDDBlock (cognitive 20) core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:1084— AppStatusListener.updateRDDBlock has cognitive complexity 20 (threshold 15). Drivers by points: if/else 16 (20 pts) (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ShuffleBlockFetcherIterator.collectFetchRequests (cognitive 20) core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala:511— ShuffleBlockFetcherIterator.collectFetchRequests has cognitive complexity 20 (threshold 15). Drivers by points: if/else 6 (12 pts), match/switch 2 (4 pts), boolean chains 3, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
EdgePartition.aggregateMessagesEdgeScan (cognitive 20) graphx/src/main/scala/org/apache/spark/graphx/impl/EdgePartition.scala:383— EdgePartition.aggregateMessagesEdgeScan has cognitive complexity 20 (threshold 15). Drivers by points: if/else 11 (17 pts), boolean chains 2, loops 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LogisticRegression.trainImpl (cognitive 20) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:931— LogisticRegression.trainImpl has cognitive complexity 20 (threshold 15). Drivers by points: if/else 12 (14 pts), boolean chains 4, loops 1, match/switch 1 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
KMeans.runAlgorithmWithWeight (cognitive 20) mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala:257— KMeans.runAlgorithmWithWeight has cognitive complexity 20 (threshold 15). Drivers by points: if/else 10 (16 pts), boolean chains 2, loops 1, match/switch 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveQualify.resolveQualifyCondition (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4466— ResolveQualify.resolveQualifyCondition has cognitive complexity 20 (threshold 15). Drivers by points: match/switch 5 (12 pts), if/else 4 (6 pts), boolean chains 2 (nesting depth added 9). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
CTESubstitution.apply (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CTESubstitution.scala:54— CTESubstitution.apply has cognitive complexity 20 (threshold 15). Drivers by points: if/else 11 (14 pts), match/switch 4 (5 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
CollationTypeCoercion.findCollationContext (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala:305— CollationTypeCoercion.findCollationContext has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (10 pts), match/switch 5 (9 pts), boolean chains 1 (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TypeCoercion.findCommonTypeForBinaryComparison (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:138— TypeCoercion.findCommonTypeForBinaryComparison has cognitive complexity 20 (threshold 15). Drivers by points: if/else 13 (19 pts), match/switch 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TypeCoercion.implicitCast (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala:202— TypeCoercion.implicitCast has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 5, match/switch 1 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SessionCatalog.registerUserDefinedFunction (cognitive 20) REDACTED:2257— SessionCatalog.registerUserDefinedFunction has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (16 pts), boolean chains 4 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UnivocityParser.convert (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala:394— UnivocityParser.convert has cognitive complexity 20 (threshold 15). Drivers by points: if/else 11 (15 pts), error handling 2 (4 pts), loops 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UnBase64.isValidBase64 (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala:3421— UnBase64.isValidBase64 has cognitive complexity 20 (threshold 15). Drivers by points: if/else 6 (12 pts), boolean chains 5, match/switch 1 (2 pts), loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VariantExpressionEvalUtils.buildVariant (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/VariantExpressionEvalUtils.scala:414— VariantExpressionEvalUtils.buildVariant has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (13 pts), loops 3 (6 pts), match/switch 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JacksonParser.convertObject (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:591— JacksonParser.convertObject has cognitive complexity 20 (threshold 15). Drivers by points: error handling 4 (12 pts), if/else 3, boolean chains 2, match/switch 1 (2 pts), loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InlineCTE.buildCTEMap (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InlineCTE.scala:139— InlineCTE.buildCTEMap has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (13 pts), match/switch 3 (7 pts) (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
PullUpProjectAliasThroughWindow.apply (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullUpProjectAliasThroughWindow.scala:78— PullUpProjectAliasThroughWindow.apply has cognitive complexity 20 (threshold 15). Drivers by points: match/switch 6 (14 pts), if/else 4 (5 pts), boolean chains 1 (nesting depth added 9). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
UnwrapCastInBinaryComparison.unwrapCast (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/UnwrapCastInBinaryComparison.scala:112— UnwrapCastInBinaryComparison.unwrapCast has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (9 pts), boolean chains 6, match/switch 3 (5 pts) (nesting depth added 4). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
LikeSimplification.simplifyMultiLike (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:1006— LikeSimplification.simplifyMultiLike has cognitive complexity 20 (threshold 15). Drivers by points: if/else 10 (18 pts), match/switch 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PartitioningUtils.normalizePartitionSpec (cognitive 20) sql/catalyst/src/main/scala/org/apache/spark/sql/util/PartitioningUtils.scala:70— PartitioningUtils.normalizePartitionSpec has cognitive complexity 20 (threshold 15). Drivers by points: match/switch 4 (11 pts), if/else 6, boolean chains 3 (nesting depth added 7). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DataTypeProtoConverter.toConnectProtoTypeDefault (cognitive 20) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/DataTypeProtoConverter.scala:193— DataTypeProtoConverter.toConnectProtoTypeDefault has cognitive complexity 20 (threshold 15). Drivers by points: if/else 8 (14 pts), match/switch 3 (5 pts), boolean chains 1 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ColumnNodeToExpressionConverter.apply (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/classic/columnNodeSupport.scala:47— ColumnNodeToExpressionConverter.apply has cognitive complexity 20 (threshold 15). Drivers by points: match/switch 7 (14 pts), if/else 5 (6 pts) (nesting depth added 8). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkSqlAstBuilder.visitDescribeRelation (cognitive 20) REDACTED:1482— SparkSqlAstBuilder.visitDescribeRelation has cognitive complexity 20 (threshold 15). Drivers by points: if/else 12 (15 pts), match/switch 1 (3 pts), boolean chains 2 (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SparkSqlAstBuilder.visitCreatePipelineDataset (cognitive 20) REDACTED:1631— SparkSqlAstBuilder.visitCreatePipelineDataset has cognitive complexity 20 (threshold 15). Drivers by points: if/else 15 (17 pts), match/switch 1 (3 pts) (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ShufflePartitionsUtil.coalescePartitionsWithSkew (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ShufflePartitionsUtil.scala:128— ShufflePartitionsUtil.coalescePartitionsWithSkew has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (12 pts), loops 2 (4 pts), boolean chains 3, match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CommandUtils.analyzeTable (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:241— CommandUtils.analyzeTable has cognitive complexity 20 (threshold 15). Drivers by points: if/else 11 (18 pts), boolean chains 2 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ShowCreateTableCommand.run (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:1223— ShowCreateTableCommand.run has cognitive complexity 20 (threshold 15). Drivers by points: if/else 11 (20 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JDBCRelation.columnPartition (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala:71— JDBCRelation.columnPartition has cognitive complexity 20 (threshold 15). Drivers by points: if/else 12 (15 pts), boolean chains 4, loops 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
JdbcUtils.getCatalystType (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:211— JdbcUtils.getCatalystType has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 5, match/switch 1 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
OrcFileFormat.buildReaderWithPartitionValues (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:140— OrcFileFormat.buildReaderWithPartitionValues has cognitive complexity 20 (threshold 15). Drivers by points: if/else 13 (17 pts), boolean chains 3 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
GroupPartitionsExec.computeOutputPartitioning (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/GroupPartitionsExec.scala:637— GroupPartitionsExec.computeOutputPartitioning has cognitive complexity 20 (threshold 15). Drivers by points: match/switch 4 (14 pts), boolean chains 3, if/else 3 (nesting depth added 10). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
EnsureRequirements.resolveKeyedPartitioning (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala:1280— EnsureRequirements.resolveKeyedPartitioning has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (10 pts), boolean chains 5, match/switch 3 (5 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SortMergeAsOfJoinScanner.advanceRightTo (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:283— SortMergeAsOfJoinScanner.advanceRightTo has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (14 pts), boolean chains 3, loops 1 (2 pts), match/switch 1 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
TransformWithStateInPySparkStateServer.handleListStateRequest (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:504— TransformWithStateInPySparkStateServer.handleListStateRequest has cognitive complexity 20 (threshold 15). Drivers by points: if/else 14 (19 pts), match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StateStoreHandler.initializeStateStore (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:1665— StateStoreHandler.initializeStateStore has cognitive complexity 20 (threshold 15). Drivers by points: if/else 13 (20 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MicroBatchExecution.executeOneBatch (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/MicroBatchExecution.scala:719— MicroBatchExecution.executeOneBatch has cognitive complexity 20 (threshold 15). Drivers by points: if/else 13 (20 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HDFSBackedStateStoreProvider.readSnapshotFile (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:1014— HDFSBackedStateStoreProvider.readSnapshotFile has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (18 pts), error handling 1, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SqlScriptingExecution.getNextStatement (cognitive 20) sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingExecution.scala:146— SqlScriptingExecution.getNextStatement has cognitive complexity 20 (threshold 15). Drivers by points: if/else 6 (13 pts), boolean chains 6, loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RowSetUtils.toTColumnValue (cognitive 20) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/RowSetUtils.scala:187— RowSetUtils.toTColumnValue has cognitive complexity 20 (threshold 15). Drivers by points: if/else 10 (19 pts), match/switch 1 (nesting depth added 9). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
HiveServer2.startHiveServer2 (cognitive 20) sql/hive-thriftserver/src/main/java/org/apache/hive/service/server/HiveServer2.java:116— HiveServer2.startHiveServer2 has cognitive complexity 20 (threshold 15). Drivers by points: error handling 4 (12 pts), if/else 3 (7 pts), loops 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MutableColumnarRow.get (cognitive 20) sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/MutableColumnarRow.java:197— MutableColumnarRow.get has cognitive complexity 20 (threshold 15). Drivers by points: if/else 20. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
LateMaterializationParquetRecordReader.initializeLateMaterialization (cognitive 20) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/LateMaterializationParquetRecordReader.java:419— LateMaterializationParquetRecordReader.initializeLateMaterialization has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (10 pts), ternaries 4 (6 pts), boolean chains 2, loops 2 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UTF8String.makeValidBytes (cognitive 20) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:390— UTF8String.makeValidBytes has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 3 (5 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UTF8String.subStringIndex (cognitive 20) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1483— UTF8String.subStringIndex has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (15 pts), loops 2 (4 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SslMessageEncoder.encode (cognitive 20) common/network-common/src/main/java/org/apache/spark/network/protocol/SslMessageEncoder.java:54— SslMessageEncoder.encode has cognitive complexity 20 (threshold 15). Drivers by points: if/else 8 (11 pts), ternaries 2 (5 pts), boolean chains 2, error handling 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
OneForOneStreamManager.connectionTerminated (cognitive 20) common/network-common/src/main/java/org/apache/spark/network/server/OneForOneStreamManager.java:134— OneForOneStreamManager.connectionTerminated has cognitive complexity 20 (threshold 15). Drivers by points: if/else 5 (12 pts), loops 2 (4 pts), error handling 1 (3 pts), boolean chains 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RocksDBIterator.<init> (cognitive 20) common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java:50— RocksDBIterator.<init> has cognitive complexity 20 (threshold 15). Drivers by points: if/else 13 (18 pts), boolean chains 1, ternaries 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowGroupedAggIncrementalPartialUDFHandler.run (cognitive 20) python/pyspark/eval_handlers/_arrow.py:466— ArrowGroupedAggIncrementalPartialUDFHandler.run has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (11 pts), loops 4 (8 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
functions._validate_and_transform_single_input (cognitive 20) python/pyspark/ml/functions.py:235— functions._validate_and_transform_single_input has cognitive complexity 20 (threshold 15). Drivers by points: if/else 10 (16 pts), boolean chains 2, ternaries 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.idxmax (cognitive 20) python/pyspark/pandas/frame.py:12606— DataFrame.idxmax has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (12 pts), ternaries 2 (6 pts), loops 1 (2 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (DataFrame.idxmin) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
DataFrame.idxmin (cognitive 20) python/pyspark/pandas/frame.py:12747— DataFrame.idxmin has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (12 pts), ternaries 2 (6 pts), loops 1 (2 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (DataFrame.idxmax) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
Frame.truncate (cognitive 20) python/pyspark/pandas/generic.py:3110— Frame.truncate has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (13 pts), boolean chains 4, ternaries 2 (3 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrameGroupBy.__getitem__ (cognitive 20) python/pyspark/pandas/groupby.py:3954— DataFrameGroupBy.__getitem__ has cognitive complexity 20 (threshold 15). Drivers by points: if/else 6 (11 pts), ternaries 2 (5 pts), loops 1 (3 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Series.__init__ (cognitive 20) python/pyspark/pandas/series.py:427— Series.__init__ has cognitive complexity 20 (threshold 15). Drivers by points: if/else 11 (20 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Series._interpolate (cognitive 20) python/pyspark/pandas/series.py:2327— Series._interpolate has cognitive complexity 20 (threshold 15). Drivers by points: if/else 12 (15 pts), boolean chains 5 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ExternalGroupBy._spill (cognitive 20) python/pyspark/shuffle.py:802— ExternalGroupBy._spill has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (10 pts), loops 4 (10 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArtifactManager._parse_artifacts (cognitive 20) python/pyspark/sql/connect/client/artifact.py:183— ArtifactManager._parse_artifacts has cognitive complexity 20 (threshold 15). Drivers by points: if/else 7 (13 pts), boolean chains 6, error handling 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.dropna (cognitive 20) python/pyspark/sql/connect/dataframe.py:1442— DataFrame.dropna has cognitive complexity 20 (threshold 15). Drivers by points: if/else 10 (17 pts), loops 1 (3 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
local_server._pid_alive (cognitive 20) REDACTED:68— local_server._pid_alive has cognitive complexity 20 (threshold 15). Drivers by points: if/else 5 (10 pts), error handling 5 (7 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ServerPool._reap_retired (cognitive 20) python/pyspark/sql/connect/local_server_pool.py:715— ServerPool._reap_retired has cognitive complexity 20 (threshold 15). Drivers by points: if/else 11 (15 pts), boolean chains 4, ternaries 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
builtin.lit (cognitive 20) python/pyspark/sql/functions/builtin.py:180— builtin.lit has cognitive complexity 20 (threshold 15). Drivers by points: if/else 9 (19 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
(anonymous) (cognitive 20) core/src/main/resources/org/apache/spark/ui/static/environmentpage.js:138— (anonymous) has cognitive complexity 20 (threshold 15). Drivers by points: boolean chains 11, if/else 5 (6 pts), error handling 1 (2 pts), ternaries 1 (nesting depth added 2). Most of this is not in the body itself: 3 of the 20 points are its own statements and the rest belongs to 10 function items inside it that branch (a function literal at line 164, (anonymous)::(anonymous)::createdRow, a function literal at line 186, …). The named helpers are already separate functions; the literals' work cannot be called, reviewed or tested from outside this body. To reduce it, move the named helpers out of the body to the enclosing scope and lift each literal's work into a named function there, so each is measured, reviewed and tested on its own, and reduce whichever one then reads as the largest.
FieldAccessFinder.visitMethod (cognitive 19) common/utils/src/main/scala/org/apache/spark/util/ClosureCleaner.scala:1162— FieldAccessFinder.visitMethod has cognitive complexity 19 (threshold 15). Drivers by points: if/else 5 (9 pts), loops 3 (7 pts), boolean chains 3 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
KafkaDataConsumer.fetchRecord (cognitive 19) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala:619— KafkaDataConsumer.fetchRecord has cognitive complexity 19 (threshold 15). Drivers by points: if/else 13 (19 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
KinesisUtilsPythonHelper.createStream (cognitive 19) connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtilsPythonHelper.scala:33— KinesisUtilsPythonHelper.createStream has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (10 pts), boolean chains 7, match/switch 2 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkContext.createTaskScheduler (cognitive 19) core/src/main/scala/org/apache/spark/SparkContext.scala:3664— SparkContext.createTaskScheduler has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (12 pts), error handling 2 (4 pts), match/switch 2 (3 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TorrentBroadcast.readBlocks (cognitive 19) core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala:189— TorrentBroadcast.readBlocks has cognitive complexity 19 (threshold 15). Drivers by points: if/else 3 (13 pts), match/switch 2 (5 pts), loops 1 (nesting depth added 13). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkSubmit.submit (cognitive 19) core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala:165— SparkSubmit.submit has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (12 pts), error handling 2 (5 pts), boolean chains 2 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Master.startExecutorsOnWorkers (cognitive 19) core/src/main/scala/org/apache/spark/deploy/master/Master.scala:848— Master.startExecutorsOnWorkers has cognitive complexity 19 (threshold 15). Drivers by points: if/else 2 (7 pts), loops 3 (7 pts), match/switch 1 (4 pts), boolean chains 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DoubleRDDFunctions.histogram (cognitive 19) core/src/main/scala/org/apache/spark/rdd/DoubleRDDFunctions.scala:170— DoubleRDDFunctions.histogram has cognitive complexity 19 (threshold 15). Drivers by points: if/else 13 (14 pts), boolean chains 2, match/switch 1 (2 pts), loops 1 (nesting depth added 2). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DAGScheduler.cancelRunningIndependentStages (cognitive 19) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:5263— DAGScheduler.cancelRunningIndependentStages has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (13 pts), error handling 1 (4 pts), boolean chains 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ReplayListenerBus.boundedLines (cognitive 19) core/src/main/scala/org/apache/spark/scheduler/ReplayListenerBus.scala:108— ReplayListenerBus.boundedLines has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (14 pts), loops 2 (3 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ShuffleBlockFetcherIterator.sendRequest (cognitive 19) core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala:264— ShuffleBlockFetcherIterator.sendRequest has cognitive complexity 19 (threshold 15). Drivers by points: if/else 11 (16 pts), match/switch 2, boolean chains 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HttpSecurityFilter.doFilter (cognitive 19) core/src/main/scala/org/apache/spark/ui/HttpSecurityFilter.scala:47— HttpSecurityFilter.doFilter has cognitive complexity 19 (threshold 15). Drivers by points: if/else 11 (17 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HadoopFSUtils.listLeafFiles (cognitive 19) core/src/main/scala/org/apache/spark/util/HadoopFSUtils.scala:218— HadoopFSUtils.listLeafFiles has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (11 pts), error handling 3 (4 pts), match/switch 3, boolean chains 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
BinaryLogisticBlockAggregator.add (cognitive 19) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:83— BinaryLogisticBlockAggregator.add has cognitive complexity 19 (threshold 15). Drivers by points: if/else 13 (17 pts), boolean chains 1, loops 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
HuberBlockAggregator.add (cognitive 19) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala:82— HuberBlockAggregator.add has cognitive complexity 19 (threshold 15). Drivers by points: if/else 11 (17 pts), boolean chains 1, loops 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MFDataGenerator.main (cognitive 19) mllib/src/main/scala/org/apache/spark/mllib/util/MFDataGenerator.scala:55— MFDataGenerator.main has cognitive complexity 19 (threshold 15). Drivers by points: if/else 19. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DataType.nameToType (cognitive 19) sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala:271— DataType.nameToType has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (10 pts), error handling 2 (4 pts), match/switch 2 (3 pts), boolean chains 2 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveOutputRelation.apply (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:4058— ResolveOutputRelation.apply has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (9 pts), match/switch 4 (8 pts), boolean chains 2 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolveTranspiledPythonUDFOptions.apply (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTranspiledPythonUDFOptions.scala:45— ResolveTranspiledPythonUDFOptions.apply has cognitive complexity 19 (threshold 15). Drivers by points: match/switch 3 (10 pts), if/else 4 (7 pts), boolean chains 2 (nesting depth added 10). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ApproxTopKAggregateBuffer.eval (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxTopKAggregates.scala:444— ApproxTopKAggregateBuffer.eval has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (9 pts), match/switch 2 (7 pts), boolean chains 2, loops 1 (nesting depth added 6). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DivModLike.eval (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:701— DivModLike.eval has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (12 pts), match/switch 1 (4 pts), boolean chains 3 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MapZipWith.getKeysWithIndexesBruteForce (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:1710— MapZipWith.getKeysWithIndexesBruteForce has cognitive complexity 19 (threshold 15). Drivers by points: if/else 3 (12 pts), loops 3 (6 pts), boolean chains 1 (nesting depth added 12). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AttributeSeq.matchWithThreeOrLessQualifierParts (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/package.scala:191— AttributeSeq.matchWithThreeOrLessQualifierParts has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (10 pts), match/switch 5 (7 pts), boolean chains 2 (nesting depth added 4). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
PushdownPredicatesAndPruneColumnsForCTEDef.gatherPredicatesAndAttributes (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushdownPredicatesAndPruneColumnsForCTEDef.scala:65— PushdownPredicatesAndPruneColumnsForCTEDef.gatherPredicatesAndAttributes has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (10 pts), match/switch 3 (8 pts), boolean chains 1 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
LikeSimplification.decodeLikePattern (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:935— LikeSimplification.decodeLikePattern has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (16 pts), boolean chains 2, loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
NullPropagation.apply (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:1066— NullPropagation.apply has cognitive complexity 19 (threshold 15). Drivers by points: if/else 5 (10 pts), boolean chains 6, match/switch 2 (3 pts) (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ExtractEquiJoinKeys.unapply (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala:187— ExtractEquiJoinKeys.unapply has cognitive complexity 19 (threshold 15). Drivers by points: boolean chains 9, if/else 4 (5 pts), match/switch 3 (5 pts) (nesting depth added 3). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
JoinWith.typedJoinWith (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/object.scala:849— JoinWith.typedJoinWith has cognitive complexity 19 (threshold 15). Drivers by points: if/else 12 (16 pts), match/switch 1 (2 pts), boolean chains 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
KeyedShuffleSpec.isExpressionCompatible (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala:2361— KeyedShuffleSpec.isExpressionCompatible has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (10 pts), match/switch 3 (5 pts), boolean chains 4 (nesting depth added 4). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
StaxXmlParser.appendXMLCharacterToVariant (cognitive 19) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1407— StaxXmlParser.appendXMLCharacterToVariant has cognitive complexity 19 (threshold 15). Drivers by points: if/else 12 (14 pts), boolean chains 3, error handling 1, match/switch 1 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
Serializer.deserializeMethodArguments (cognitive 19) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/Serializer.scala:140— Serializer.deserializeMethodArguments has cognitive complexity 19 (threshold 15). Drivers by points: match/switch 4 (12 pts), if/else 7 (nesting depth added 8). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ParseSqlResult.collectPlanReferences (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala:328— ParseSqlResult.collectPlanReferences has cognitive complexity 19 (threshold 15). Drivers by points: if/else 10 (12 pts), match/switch 4 (5 pts), boolean chains 2 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
StreamingQueryManager.createQuery (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/classic/StreamingQueryManager.scala:240— StreamingQueryManager.createQuery has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (14 pts), match/switch 3, boolean chains 2 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PushDownLocalSort.pushDown (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/PushDownLocalSort.scala:85— PushDownLocalSort.pushDown has cognitive complexity 19 (threshold 15). Drivers by points: match/switch 4 (9 pts), if/else 6 (8 pts), boolean chains 2 (nesting depth added 7). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ConvertSortMergeJoinToShuffledHashJoin.apply (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ConvertSortMergeJoinToShuffledHashJoin.scala:106— ConvertSortMergeJoinToShuffledHashJoin.apply has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (12 pts), boolean chains 4, match/switch 2 (3 pts) (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LoadDataCommand.run (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:310— LoadDataCommand.run has cognitive complexity 19 (threshold 15). Drivers by points: if/else 13 (17 pts), boolean chains 1, error handling 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
JdbcUtils.makeSetter (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala:503— JdbcUtils.makeSetter has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (10 pts), match/switch 4 (9 pts) (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ParquetFileFormat.buildParquetReader (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:242— ParquetFileFormat.buildParquetReader has cognitive complexity 19 (threshold 15). Drivers by points: if/else 15, boolean chains 4. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
MapReader.readFromTyped (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/SparkShreddingUtils.scala:358— MapReader.readFromTyped has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (13 pts), boolean chains 4, loops 2 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
V2SessionCatalog.createTable0 (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2SessionCatalog.scala:167— V2SessionCatalog.createTable0 has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (15 pts), boolean chains 2, error handling 1, match/switch 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ShuffledHashJoinExec.buildSideOrFullOuterJoinNonUniqueKey (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/ShuffledHashJoinExec.scala:263— ShuffledHashJoinExec.buildSideOrFullOuterJoinNonUniqueKey has cognitive complexity 19 (threshold 15). Drivers by points: if/else 10 (14 pts), boolean chains 3, loops 1 (2 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SQLLastAttemptAccumulator.extractStageRDDScopes (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala:295— SQLLastAttemptAccumulator.extractStageRDDScopes has cognitive complexity 19 (threshold 15). Drivers by points: match/switch 6 (10 pts), if/else 9 (nesting depth added 4). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
InterpreterSession.onInterpreterThread (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/python/InProcessPythonRuntime.scala:192— InterpreterSession.onInterpreterThread has cognitive complexity 19 (threshold 15). Drivers by points: if/else 6 (11 pts), error handling 3 (6 pts), boolean chains 1, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TransformWithStateInPySparkPythonInitialStateRunner.writeNextBatchToArrowStream (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkPythonRunner.scala:154— TransformWithStateInPySparkPythonInitialStateRunner.writeNextBatchToArrowStream has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 3, loops 1 (2 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StatefulStreamlineAggregateExec.doExecute (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/StatefulStreamlineAggregateExec.scala:125— StatefulStreamlineAggregateExec.doExecute has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (9 pts), loops 3 (6 pts), boolean chains 2, match/switch 2 (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
OneSideHashJoiner.storeAndJoinWithOtherSide (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala:783— OneSideHashJoiner.storeAndJoinWithOtherSide has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (11 pts), boolean chains 4, match/switch 4 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
RocksDB.handleMetricsUpdate (cognitive 19) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1288— RocksDB.handleMetricsUpdate has cognitive complexity 19 (threshold 15). Drivers by points: if/else 10 (19 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
OracleDialect.getCatalystType (cognitive 19) REDACTED:155— OracleDialect.getCatalystType has cognitive complexity 19 (threshold 15). Drivers by points: if/else 10 (14 pts), match/switch 2 (3 pts), boolean chains 2 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SqlResource.sqlTable (cognitive 19) sql/core/src/main/scala/org/apache/spark/status/api/v1/sql/SqlResource.scala:89— SqlResource.sqlTable has cognitive complexity 19 (threshold 15). Drivers by points: if/else 15 (16 pts), boolean chains 3 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ReceiverInputDStream.createBlockRDD (cognitive 19) streaming/src/main/scala/org/apache/spark/streaming/dstream/ReceiverInputDStream.scala:94— ReceiverInputDStream.createBlockRDD has cognitive complexity 19 (threshold 15). Drivers by points: if/else 10 (19 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VectorizedRleValuesReader.readBatchInternal (cognitive 19) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:221— VectorizedRleValuesReader.readBatchInternal has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (14 pts), match/switch 1 (3 pts), boolean chains 1, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AmIpFilter.doFilter (cognitive 19) resource-managers/yarn/src/main/java/org/apache/spark/deploy/yarn/AmIpFilter.java:135— AmIpFilter.doFilter has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 1 (3 pts), boolean chains 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Location.append (cognitive 19) core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:956— Location.append has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (11 pts), error handling 1 (4 pts), boolean chains 3, ternaries 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ByteArrayMethods.arrayEquals (cognitive 19) common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java:52— ByteArrayMethods.arrayEquals has cognitive complexity 19 (threshold 15). Drivers by points: if/else 5 (10 pts), loops 3 (5 pts), boolean chains 4 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RemoteBlockPushResolver.reloadFinalizedAppAttemptsShuffleMergeInfo (cognitive 19) common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:1180— RemoteBlockPushResolver.reloadFinalizedAppAttemptsShuffleMergeInfo has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (12 pts), error handling 1 (3 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Title.parse (cognitive 19) REDACTED:2063— Title.parse has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (11 pts), boolean chains 4, loops 2 (3 pts), ternaries 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
_ValidatorSharedReadWrite.meta_estimator_transfer_param_maps_to_java (cognitive 19) python/pyspark/ml/tuning.py:304— _ValidatorSharedReadWrite.meta_estimator_transfer_param_maps_to_java has cognitive complexity 19 (threshold 15). Drivers by points: if/else 4 (11 pts), loops 3 (6 pts), boolean chains 2 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.replace (cognitive 19) python/pyspark/pandas/frame.py:6519— DataFrame.replace has cognitive complexity 19 (threshold 15). Drivers by points: if/else 12 (14 pts), boolean chains 5 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
DataFrame.cov (cognitive 19) python/pyspark/pandas/frame.py:9548— DataFrame.cov has cognitive complexity 19 (threshold 15). Drivers by points: loops 6 (9 pts), if/else 4, ternaries 2 (4 pts), boolean chains 2 (nesting depth added 5). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
DataFrame._reindex_columns (cognitive 19) python/pyspark/pandas/frame.py:10737— DataFrame._reindex_columns has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (12 pts), loops 3 (4 pts), ternaries 2 (3 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DatetimeIndex.__new__ (cognitive 19) python/pyspark/pandas/indexes/datetimes.py:109— DatetimeIndex.__new__ has cognitive complexity 19 (threshold 15). Drivers by points: if/else 12 (19 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TimedeltaIndex.__new__ (cognitive 19) python/pyspark/pandas/indexes/timedelta.py:94— TimedeltaIndex.__new__ has cognitive complexity 19 (threshold 15). Drivers by points: if/else 12 (19 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AtIndexer.__getitem__ (cognitive 19) python/pyspark/pandas/indexing.py:138— AtIndexer.__getitem__ has cognitive complexity 19 (threshold 15). Drivers by points: if/else 10 (15 pts), boolean chains 3, ternaries 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectClient._execute_and_fetch_as_iterator (cognitive 19) python/pyspark/sql/connect/client/core.py:1750— SparkConnectClient._execute_and_fetch_as_iterator has cognitive complexity 19 (threshold 15). Drivers by points: loops 4 (8 pts), error handling 3 (6 pts), if/else 4 (5 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkSession.stop (cognitive 19) python/pyspark/sql/connect/session.py:977— SparkSession.stop has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (11 pts), error handling 3 (6 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkSession._start_connect_server (cognitive 19) python/pyspark/sql/connect/session.py:1270— SparkSession._start_connect_server has cognitive complexity 19 (threshold 15). Drivers by points: if/else 9 (16 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
types._parse_collation_metadata_map (cognitive 19) python/pyspark/sql/types.py:2799— types._parse_collation_metadata_map has cognitive complexity 19 (threshold 15). Drivers by points: if/else 5 (10 pts), boolean chains 3, loops 2 (3 pts), ternaries 1 (3 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UserDefinedFunction.__init__ (cognitive 19) python/pyspark/sql/udf.py:168— UserDefinedFunction.__init__ has cognitive complexity 19 (threshold 15). Drivers by points: if/else 8 (9 pts), boolean chains 4, ternaries 2 (4 pts), error handling 2 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition.
getInnerText (cognitive 19) REDACTED:212— getInnerText has cognitive complexity 19 (threshold 15). Drivers by points: if/else 7 (9 pts), boolean chains 5, loops 1 (3 pts), match/switch 1 (2 pts) (nesting depth added 5). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ExecutorAllocationManager.validateSettings (cognitive 18) core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:213— ExecutorAllocationManager.validateSettings has cognitive complexity 18 (threshold 15). Drivers by points: if/else 11 (12 pts), boolean chains 6 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
MapOutputTrackerMaster.getLocationsWithLargestOutputs (cognitive 18) core/src/main/scala/org/apache/spark/MapOutputTracker.scala:1230— MapOutputTrackerMaster.getLocationsWithLargestOutputs has cognitive complexity 18 (threshold 15). Drivers by points: if/else 5 (15 pts), loops 1 (3 pts) (nesting depth added 12). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
SparkConf.validateSettings (cognitive 18) core/src/main/scala/org/apache/spark/SparkConf.scala:552— SparkConf.validateSettings has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (10 pts), boolean chains 5, match/switch 1 (2 pts), loops 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ReaderInputStream.writeBufferedInputToPythonWorker (cognitive 18) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1500— ReaderInputStream.writeBufferedInputToPythonWorker has cognitive complexity 18 (threshold 15). Drivers by points: boolean chains 6, if/else 3 (6 pts), loops 3 (6 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PipelinedWriterRunnable.run (cognitive 18) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:1572— PipelinedWriterRunnable.run has cognitive complexity 18 (threshold 15). Drivers by points: if/else 5 (10 pts), error handling 4 (5 pts), boolean chains 2, loops 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RPackageUtils.extractRFolder (cognitive 18) core/src/main/scala/org/apache/spark/deploy/RPackageUtils.scala:141— RPackageUtils.extractRFolder has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (17 pts), loops 1 (nesting depth added 11). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Executor.stop (cognitive 18) core/src/main/scala/org/apache/spark/executor/Executor.scala:651— Executor.stop has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (13 pts), error handling 2 (4 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Executor.updateDependencies (cognitive 18) core/src/main/scala/org/apache/spark/executor/Executor.scala:1453— Executor.updateDependencies has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
NettyRpcEnv.askAbortable (cognitive 18) core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala:210— NettyRpcEnv.askAbortable has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (10 pts), match/switch 4 (7 pts), error handling 1 (nesting depth added 4). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
DAGScheduler.removeExecutorAndUnregisterOutputs (cognitive 18) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:4912— DAGScheduler.removeExecutorAndUnregisterOutputs has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (11 pts), boolean chains 4, match/switch 1 (3 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ShuffleBlockPusher.pushUpToMax (cognitive 18) core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala:160— ShuffleBlockPusher.pushUpToMax has cognitive complexity 18 (threshold 15). Drivers by points: if/else 4 (8 pts), loops 3 (6 pts), boolean chains 4 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AppStatusStore.newStageData (cognitive 18) core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:630— AppStatusStore.newStageData has cognitive complexity 18 (threshold 15). Drivers by points: if/else 12 (17 pts), boolean chains 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MemoryStore.evictBlocksToFreeSpace (cognitive 18) core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala:474— MemoryStore.evictBlocksToFreeSpace has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (13 pts), boolean chains 3, loops 1, match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LastAttemptAccumulator.mergeLastAttempt (cognitive 18) core/src/main/scala/org/apache/spark/util/LastAttemptAccumulator.scala:706— LastAttemptAccumulator.mergeLastAttempt has cognitive complexity 18 (threshold 15). Drivers by points: if/else 11 (16 pts), error handling 1, match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkUncaughtExceptionHandler.uncaughtException (cognitive 18) core/src/main/scala/org/apache/spark/util/SparkUncaughtExceptionHandler.scala:52— SparkUncaughtExceptionHandler.uncaughtException has cognitive complexity 18 (threshold 15). Drivers by points: error handling 4 (6 pts), if/else 5 (6 pts), match/switch 1 (3 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BLAS.gemvImpl (cognitive 18) mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala:853— BLAS.gemvImpl has cognitive complexity 18 (threshold 15). Drivers by points: loops 4 (10 pts), if/else 7 (8 pts) (nesting depth added 7). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
LogisticRegression.createOptimizer (cognitive 18) mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:771— LogisticRegression.createOptimizer has cognitive complexity 18 (threshold 15). Drivers by points: if/else 11 (15 pts), boolean chains 3 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GeneralizedLinearRegressionTrainingSummary.toString (cognitive 18) mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala:1621— GeneralizedLinearRegressionTrainingSummary.toString has cognitive complexity 18 (threshold 15). Drivers by points: match/switch 5 (11 pts), if/else 5 (7 pts) (nesting depth added 8). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
BLAS.gemv (cognitive 18) mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala:717— BLAS.gemv has cognitive complexity 18 (threshold 15). Drivers by points: loops 4 (10 pts), if/else 7 (8 pts) (nesting depth added 7). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
YarnAllocator.runAllocatedContainers (cognitive 18) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:755— YarnAllocator.runAllocatedContainers has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (13 pts), error handling 1 (4 pts), loops 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HandleNullInputsForUDF.apply (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:3922— HandleNullInputsForUDF.apply has cognitive complexity 18 (threshold 15). Drivers by points: if/else 5 (10 pts), match/switch 3 (6 pts), boolean chains 2 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
RelationResolution.resolveRelation (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala:162— RelationResolution.resolveRelation has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (11 pts), boolean chains 4, match/switch 1 (2 pts), loops 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ResolveCatalogs.apply (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala:44— ResolveCatalogs.apply has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (15 pts), match/switch 2 (3 pts) (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
StreamingJoinHelper.collectTerms (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/StreamingJoinHelper.scala:213— StreamingJoinHelper.collectTerms has cognitive complexity 18 (threshold 15). Drivers by points: if/else 10 (13 pts), match/switch 3 (5 pts) (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
UnivocityParser.makeConverter (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/UnivocityParser.scala:189— UnivocityParser.makeConverter has cognitive complexity 18 (threshold 15). Drivers by points: if/else 5 (9 pts), match/switch 3 (5 pts), error handling 2 (4 pts) (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
Cast.castToShort (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1233— Cast.castToShort has cognitive complexity 18 (threshold 15). Drivers by points: if/else 10 (15 pts), error handling 1 (2 pts), match/switch 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
Cast.castToByte (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1297— Cast.castToByte has cognitive complexity 18 (threshold 15). Drivers by points: if/else 10 (15 pts), error handling 1 (2 pts), match/switch 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
InternalSequenceBase.eval (cognitive 18) REDACTED:4137— InternalSequenceBase.eval has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (13 pts), boolean chains 3, loops 1 (2 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UpdateFields.fieldSources (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala:1080— UpdateFields.fieldSources has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (11 pts), match/switch 5 (7 pts) (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ZipWith.eval (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/higherOrderFunctions.scala:1829— ZipWith.eval has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 1 (3 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JacksonParser.convertMap (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala:656— JacksonParser.convertMap has cognitive complexity 18 (threshold 15). Drivers by points: error handling 3 (9 pts), if/else 4, match/switch 2 (4 pts), loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JsonInferSchema.compatibleType (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala:383— JsonInferSchema.compatibleType has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (9 pts), loops 3 (6 pts), boolean chains 2, match/switch 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
InjectRuntimeFilter.tryInjectRuntimeFilter (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala:411— InjectRuntimeFilter.tryInjectRuntimeFilter has cognitive complexity 18 (threshold 15). Drivers by points: if/else 4 (11 pts), boolean chains 6, match/switch 1 (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
OptimizeRand.eliminateRand (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/OptimizeRand.scala:52— OptimizeRand.eliminateRand has cognitive complexity 18 (threshold 15). Drivers by points: if/else 13 (17 pts), match/switch 1 (nesting depth added 4). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
OptimizeSubqueries.optimize (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:423— OptimizeSubqueries.optimize has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (13 pts), match/switch 2 (4 pts), boolean chains 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AstBuilder.visitFromClause (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:2096— AstBuilder.visitFromClause has cognitive complexity 18 (threshold 15). Drivers by points: if/else 13 (16 pts), match/switch 1 (2 pts) (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
Project.reconcileColumnType (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala:130— Project.reconcileColumnType has cognitive complexity 18 (threshold 15). Drivers by points: if/else 11 (16 pts), boolean chains 1, match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TreeNode.mapChildren (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:811— TreeNode.mapChildren has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (12 pts), loops 2 (5 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
NumberConverter.encode (cognitive 18) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/NumberConverter.scala:59— NumberConverter.encode has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (16 pts), boolean chains 1, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MLHandler.startHangingHandlerReaper (cognitive 18) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:159— MLHandler.startHangingHandlerReaper has cognitive complexity 18 (threshold 15). Drivers by points: if/else 5 (11 pts), loops 2 (6 pts), error handling 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectPlanner.transformWithRelations (cognitive 18) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:4661— SparkConnectPlanner.transformWithRelations has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (9 pts), match/switch 2 (4 pts), boolean chains 3, error handling 1 (2 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SessionStatsPagedTable.renderMLCacheStatus (cognitive 18) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:529— SessionStatsPagedTable.renderMLCacheStatus has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (16 pts), match/switch 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AvroFileFormat.buildReader (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/avro/AvroFileFormat.scala:89— AvroFileFormat.buildReader has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (10 pts), error handling 1 (3 pts), match/switch 1 (3 pts), boolean chains 2 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BroadcastValueProjector.collectExactValueDomain (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/BroadcastValueProjector.scala:53— BroadcastValueProjector.collectExactValueDomain has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 1, error handling 1, loops 1, match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SQLExecution.withNewExecutionId0 (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:205— SQLExecution.withNewExecutionId0 has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9, error handling 4 (5 pts), boolean chains 2, match/switch 2 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkSqlAstBuilder.visitCreateTable (cognitive 18) REDACTED:596— SparkSqlAstBuilder.visitCreateTable has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (13 pts), match/switch 1 (3 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkSqlAstBuilder.visitCreateView (cognitive 18) REDACTED:772— SparkSqlAstBuilder.visitCreateView has cognitive complexity 18 (threshold 15). Drivers by points: if/else 12 (14 pts), boolean chains 4 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ObjectAggregationIterator.processInputs (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala:153— ObjectAggregationIterator.processInputs has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (9 pts), loops 3 (7 pts), boolean chains 2 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SortBasedAggregator.destructiveIterator (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/ObjectAggregationIterator.scala:255— SortBasedAggregator.destructiveIterator has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (10 pts), boolean chains 4, loops 2 (4 pts) (nesting depth added 3). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
LoadDataCommand.makeQualified (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala:411— LoadDataCommand.makeQualified has cognitive complexity 18 (threshold 15). Drivers by points: if/else 10 (14 pts), boolean chains 2, error handling 1 (2 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CreateViewCommand.run (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/command/views.scala:117— CreateViewCommand.run has cognitive complexity 18 (threshold 15). Drivers by points: if/else 15 (18 pts) (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
JDBCRDD.compute (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala:288— JDBCRDD.compute has cognitive complexity 18 (threshold 15). Drivers by points: error handling 6 (8 pts), if/else 6 (7 pts), boolean chains 2, match/switch 1 (nesting depth added 3). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PushDownUtils.pushFilters (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:56— PushDownUtils.pushFilters has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (12 pts), loops 2 (4 pts), boolean chains 1, match/switch 1 (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
PushDownUtils.replanWithRuntimeFilters (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala:289— PushDownUtils.replanWithRuntimeFilters has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (14 pts), match/switch 2 (4 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PlanExternalUDFs.extractExternalUDFFromAggregate (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/externalUDF/PlanExternalUDFs.scala:113— PlanExternalUDFs.extractExternalUDFFromAggregate has cognitive complexity 18 (threshold 15). Drivers by points: match/switch 3 (10 pts), if/else 5 (8 pts) (nesting depth added 10). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
HashJoin.codegenOuter (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala:531— HashJoin.codegenOuter has cognitive complexity 18 (threshold 15). Drivers by points: if/else 14 (17 pts), match/switch 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
TransformWithStateInPySparkStateServer.serveRequests (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:170— TransformWithStateInPySparkStateServer.serveRequests has cognitive complexity 18 (threshold 15). Drivers by points: error handling 5 (12 pts), if/else 3 (4 pts), boolean chains 1, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TransformWithStateInPySparkStateServer.handleTimerRequest (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkStateServer.scala:278— TransformWithStateInPySparkStateServer.handleTimerRequest has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (15 pts), match/switch 2 (3 pts) (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SymmetricHashJoinStateManagerV4.initializeStateStore (cognitive 18) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:373— SymmetricHashJoinStateManagerV4.initializeStateStore has cognitive complexity 18 (threshold 15). Drivers by points: if/else 11 (18 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HiveClientImpl.runHive (cognitive 18) sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala:912— HiveClientImpl.runHive has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (12 pts), error handling 3 (5 pts), match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HiveAuthFactory.<init> (cognitive 18) sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java:102— HiveAuthFactory.<init> has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (15 pts), error handling 1 (3 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HadoopLineRecordReader.initialize (cognitive 18) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/HadoopLineRecordReader.java:90— HadoopLineRecordReader.initialize has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (15 pts), error handling 2 (3 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ServerConnection.handle (cognitive 18) REDACTED:318— ServerConnection.handle has cognitive complexity 18 (threshold 15). Drivers by points: if/else 10 (15 pts), ternaries 1 (2 pts), error handling 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AwsStsCredentialProvider.init (cognitive 18) connector/credential-aws/src/main/java/org/apache/spark/security/aws/AwsStsCredentialProvider.java:164— AwsStsCredentialProvider.init has cognitive complexity 18 (threshold 15). Drivers by points: if/else 10 (11 pts), boolean chains 5, error handling 1 (2 pts) (nesting depth added 2). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
CollationSpecICU.collationNameToId (cognitive 18) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationFactory.java:889— CollationSpecICU.collationNameToId has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (14 pts), loops 2, match/switch 1 (2 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UTF8String.substring (cognitive 18) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:645— UTF8String.substring has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (8 pts), loops 2 (5 pts), boolean chains 3, ternaries 1 (2 pts) (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
RemoteBlockPushResolver.finalizeShuffleMerge (cognitive 18) common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:762— RemoteBlockPushResolver.finalizeShuffleMerge has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (11 pts), error handling 1 (3 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PushBlockStreamCallback.onData (cognitive 18) common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:1439— PushBlockStreamCallback.onData has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 2, error handling 1 (2 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MessageEncoder.encode (cognitive 18) common/network-common/src/main/java/org/apache/spark/network/protocol/MessageEncoder.java:54— MessageEncoder.encode has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (10 pts), ternaries 2 (5 pts), error handling 1 (2 pts), boolean chains 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowWindowAggIncrementalUDFHandler.run (cognitive 18) python/pyspark/eval_handlers/_arrow.py:609— ArrowWindowAggIncrementalUDFHandler.run has cognitive complexity 18 (threshold 15). Drivers by points: if/else 4 (10 pts), loops 3 (7 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowUDTFWithPartition._process_arrow_batch_by_partitions (cognitive 18) REDACTED:291— ArrowUDTFWithPartition._process_arrow_batch_by_partitions has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (12 pts), loops 3 (5 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
_ValidatorSharedReadWrite.meta_estimator_transfer_param_maps_from_java (cognitive 18) python/pyspark/ml/tuning.py:340— _ValidatorSharedReadWrite.meta_estimator_transfer_param_maps_from_java has cognitive complexity 18 (threshold 15). Drivers by points: if/else 4 (11 pts), loops 3 (6 pts), boolean chains 1 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GroupBy._resolve_grouping_from_diff_dataframes (cognitive 18) python/pyspark/pandas/groupby.py:3798— GroupBy._resolve_grouping_from_diff_dataframes has cognitive complexity 18 (threshold 15). Drivers by points: if/else 8 (15 pts), loops 3 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
plotly.plot_pie (cognitive 18) python/pyspark/pandas/plot/plotly.py:52— plotly.plot_pie has cognitive complexity 18 (threshold 15). Drivers by points: if/else 6 (11 pts), loops 1 (3 pts), ternaries 1 (3 pts), boolean chains 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
plotly.plot_box (cognitive 18) python/pyspark/pandas/plot/plotly.py:149— plotly.plot_box has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (9 pts), ternaries 2 (5 pts), loops 2 (4 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
udf._create_py_udf (cognitive 18) python/pyspark/sql/connect/udf.py:60— udf._create_py_udf has cognitive complexity 18 (threshold 15). Drivers by points: if/else 9 (12 pts), error handling 3 (6 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
functions.vectorized_udf (cognitive 18) python/pyspark/sql/pandas/functions.py:783— functions.vectorized_udf has cognitive complexity 18 (threshold 15). Drivers by points: if/else 11 (13 pts), boolean chains 5 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
transpile._get_function_from_ast (cognitive 18) python/pyspark/sql/transpile.py:1115— transpile._get_function_from_ast has cognitive complexity 18 (threshold 15). Drivers by points: if/else 10 (14 pts), boolean chains 4 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
analyze_udtf._main (cognitive 18) python/pyspark/sql/worker/analyze_udtf.py:100— analyze_udtf._main has cognitive complexity 18 (threshold 15). Drivers by points: boolean chains 6, if/else 4 (5 pts), loops 3, ternaries 2 (3 pts), error handling 1 (nesting depth added 2). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
stagepage.reselectCheckboxesBasedOnTaskTableState (cognitive 18) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:287— stagepage.reselectCheckboxesBasedOnTaskTableState has cognitive complexity 18 (threshold 15). Drivers by points: if/else 7 (11 pts), loops 2 (4 pts), boolean chains 3 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkContext.resumeExecutors (cognitive 17) core/src/main/scala/org/apache/spark/SparkContext.scala:2285— SparkContext.resumeExecutors has cognitive complexity 17 (threshold 15). Drivers by points: if/else 5 (8 pts), error handling 1 (4 pts), match/switch 2 (4 pts), boolean chains 1 (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
RPackageUtils.checkAndBuildRPackage (cognitive 17) core/src/main/scala/org/apache/spark/deploy/RPackageUtils.scala:175— RPackageUtils.checkAndBuildRPackage has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (17 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskSetManager.addPendingTask (cognitive 17) core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:290— TaskSetManager.addPendingTask has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (7 pts), loops 2 (5 pts), match/switch 2 (5 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskSetManager.resourceOffer (cognitive 17) core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:470— TaskSetManager.resourceOffer has cognitive complexity 17 (threshold 15). Drivers by points: if/else 7 (12 pts), boolean chains 5 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExecutorMonitor.onJobEnd (cognitive 17) core/src/main/scala/org/apache/spark/scheduler/dynalloc/ExecutorMonitor.scala:260— ExecutorMonitor.onJobEnd has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (17 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BlockManager.getCurrentBlockStatus (cognitive 17) core/src/main/scala/org/apache/spark/storage/BlockManager.scala:1090— BlockManager.getCurrentBlockStatus has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (13 pts), boolean chains 3, match/switch 1 (nesting depth added 4). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SizeEstimator.getClassInfo (cognitive 17) core/src/main/scala/org/apache/spark/util/SizeEstimator.scala:325— SizeEstimator.getClassInfo has cognitive complexity 17 (threshold 15). Drivers by points: if/else 5 (11 pts), error handling 1 (4 pts), loops 2 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LinearRegression.createOptimizer (cognitive 17) mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala:494— LinearRegression.createOptimizer has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (15 pts), boolean chains 1, match/switch 1 (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ColumnResolutionHelper.resolveLateralColumnAlias (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:354— ColumnResolutionHelper.resolveLateralColumnAlias has cognitive complexity 17 (threshold 15). Drivers by points: match/switch 6 (10 pts), if/else 5 (7 pts) (nesting depth added 6). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ResolverGuard.checkUnresolvedFunction (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ResolverGuard.scala:512— ResolverGuard.checkUnresolvedFunction has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (11 pts), match/switch 2 (6 pts) (nesting depth added 6). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
EquivalentExpressions.alwaysEvaluatedDefinitions (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/EquivalentExpressions.scala:173— EquivalentExpressions.alwaysEvaluatedDefinitions has cognitive complexity 17 (threshold 15). Drivers by points: if/else 5 (11 pts), match/switch 2 (6 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SchemaPruning.getArrayReturningHigherOrderFunctionRootFields (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala:232— SchemaPruning.getArrayReturningHigherOrderFunctionRootFields has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (9 pts), match/switch 3 (7 pts), boolean chains 1 (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
Sequence.eval (cognitive 17) REDACTED:3732— Sequence.eval has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (14 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JsonTableEvaluator.descendInto (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:1123— JsonTableEvaluator.descendInto has cognitive complexity 17 (threshold 15). Drivers by points: if/else 5 (9 pts), loops 2 (4 pts), boolean chains 3, match/switch 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ConvertToCatalyst.convertOperator (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:1111— ConvertToCatalyst.convertOperator has cognitive complexity 17 (threshold 15). Drivers by points: if/else 11 (13 pts), boolean chains 3, match/switch 1 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
CollapseProject.mergeProjectExpressions (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala:1767— CollapseProject.mergeProjectExpressions has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (10 pts), match/switch 3 (4 pts), boolean chains 3 (nesting depth added 2). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
PushdownPredicatesAndPruneColumnsForCTEDef.pushdownPredicatesAndAttributes (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushdownPredicatesAndPruneColumnsForCTEDef.scala:131— PushdownPredicatesAndPruneColumnsForCTEDef.pushdownPredicatesAndAttributes has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (12 pts), match/switch 2 (4 pts), boolean chains 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
RewriteCorrelatedScalarSubquery.evalSubqueryOnZeroTups (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala:793— RewriteCorrelatedScalarSubquery.evalSubqueryOnZeroTups has cognitive complexity 17 (threshold 15). Drivers by points: if/else 7 (11 pts), match/switch 3 (6 pts) (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
HyperLogLogPlusPlusHelper.query (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/HyperLogLogPlusPlusHelper.scala:202— HyperLogLogPlusPlusHelper.query has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (9 pts), boolean chains 4, loops 2 (3 pts), match/switch 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SQLConf.get (cognitive 17) sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:306— SQLConf.get has cognitive complexity 17 (threshold 15). Drivers by points: if/else 11 (17 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectPlanner.handleWriteStreamOperationStart (cognitive 17) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:3688— SparkConnectPlanner.handleWriteStreamOperationStart has cognitive complexity 17 (threshold 15). Drivers by points: if/else 11 (12 pts), match/switch 3 (4 pts), error handling 1 (nesting depth added 2). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkConnectPlanner.handleStreamingQueryCommand (cognitive 17) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:3864— SparkConnectPlanner.handleStreamingQueryCommand has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (13 pts), match/switch 3 (4 pts) (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkConnectAnalyzeHandler.process (cognitive 17) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectAnalyzeHandler.scala:60— SparkConnectAnalyzeHandler.process has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (13 pts), match/switch 2 (3 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ResolveSessionCatalog.getStorageFormatAndProvider (cognitive 17) REDACTED:748— ResolveSessionCatalog.getStorageFormatAndProvider has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (12 pts), match/switch 1 (3 pts), boolean chains 2 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ResolveTranspose.apply (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTranspose.scala:111— ResolveTranspose.apply has cognitive complexity 17 (threshold 15). Drivers by points: if/else 7 (12 pts), match/switch 2 (3 pts), boolean chains 2 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
OptimizeMetadataOnlyQuery.apply (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala:48— OptimizeMetadataOnlyQuery.apply has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (9 pts), match/switch 3 (7 pts), boolean chains 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JoinSelection.checkHintBuildSide (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala:203— JoinSelection.checkHintBuildSide has cognitive complexity 17 (threshold 15). Drivers by points: if/else 7 (16 pts), boolean chains 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TungstenAggregationIterator.next (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/TungstenAggregationIterator.scala:520— TungstenAggregationIterator.next has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (16 pts), boolean chains 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowCachedBatchSerializer.calculateMinMaxFloat (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:810— ArrowCachedBatchSerializer.calculateMinMaxFloat has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (17 pts) (nesting depth added 9). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function. This shape REPEATS in the file: one other method here (ArrowCachedBatchSerializer.calculateMinMaxDouble) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
ArrowCachedBatchSerializer.calculateMinMaxDouble (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:838— ArrowCachedBatchSerializer.calculateMinMaxDouble has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (17 pts) (nesting depth added 9). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function. This shape REPEATS in the file: one other method here (ArrowCachedBatchSerializer.calculateMinMaxFloat) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
ObjectColumnStats.gatherStats (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala:405— ObjectColumnStats.gatherStats has cognitive complexity 17 (threshold 15). Drivers by points: match/switch 4 (11 pts), if/else 6 (nesting depth added 7). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
CreateDataSourceTableAsSelectCommand.run (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/command/createDataSourceTables.scala:153— CreateDataSourceTableAsSelectCommand.run has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (13 pts), boolean chains 2, match/switch 1 (2 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DescribeFunctionCommand.describeSQLFunction (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala:135— DescribeFunctionCommand.describeSQLFunction has cognitive complexity 17 (threshold 15). Drivers by points: if/else 12 (15 pts), match/switch 1 (2 pts) (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
AggregatePushDownUtils.getSchemaForPushedAggregation (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/AggregatePushDownUtils.scala:38— AggregatePushDownUtils.getSchemaForPushedAggregation has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (11 pts), boolean chains 3, match/switch 3 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
DataSource.sourceSchema (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala:251— DataSource.sourceSchema has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (11 pts), boolean chains 5, match/switch 1 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SortMergeFullOuterJoinScanner.advanceNext (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala:1627— SortMergeFullOuterJoinScanner.advanceNext has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (11 pts), boolean chains 6 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
CollectLimitExec.doExecute (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/limit.scala:70— CollectLimitExec.doExecute has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (16 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CoGroupedArrowPythonRunner.newWriter (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/python/CoGroupedArrowPythonRunner.scala:100— CoGroupedArrowPythonRunner.newWriter has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (15 pts), boolean chains 2 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HDFSMetadataLog.verifyBatchIds (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/HDFSMetadataLog.scala:387— HDFSMetadataLog.verifyBatchIds has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (16 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SymmetricHashJoinStateManagerV4.getJoinedRowsAndRemoveMatched (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:520— SymmetricHashJoinStateManagerV4.getJoinedRowsAndRemoveMatched has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (14 pts), boolean chains 1, loops 1, match/switch 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
KeyWithTsToValuesStore.getValuesInRange (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:755— KeyWithTsToValuesStore.getValuesInRange has cognitive complexity 17 (threshold 15). Drivers by points: if/else 11 (13 pts), boolean chains 4 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SymmetricHashJoinStateManagerBase.evictAndReturnByValueCondition (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:1415— SymmetricHashJoinStateManagerBase.evictAndReturnByValueCondition has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (14 pts), boolean chains 2, loops 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AutoSnapshotLoader.loadSnapshot (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/AutoSnapshotLoader.scala:96— AutoSnapshotLoader.loadSnapshot has cognitive complexity 17 (threshold 15). Drivers by points: error handling 3 (7 pts), if/else 5 (6 pts), loops 2 (3 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RocksDB.loadWithoutCheckpointId (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:608— RocksDB.loadWithoutCheckpointId has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (14 pts), boolean chains 2, error handling 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SQLAppStatusListener.aggregateMetrics (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListener.scala:208— SQLAppStatusListener.aggregateMetrics has cognitive complexity 17 (threshold 15). Drivers by points: if/else 11 (14 pts), error handling 1 (2 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
WindowPartitionEvaluator.eval (cognitive 17) sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactory.scala:58— WindowPartitionEvaluator.eval has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8, boolean chains 5, loops 3 (4 pts) (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkExecuteStatementOperation.execute (cognitive 17) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala:213— SparkExecuteStatementOperation.execute has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (12 pts), match/switch 1 (3 pts), boolean chains 1, error handling 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
HiveScriptTransformationExec.createOutputIteratorWithSerde (cognitive 17) sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveScriptTransformationExec.scala:58— HiveScriptTransformationExec.createOutputIteratorWithSerde has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (12 pts), error handling 2 (4 pts), loops 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CheckpointWriteHandler.run (cognitive 17) streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala:223— CheckpointWriteHandler.run has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (12 pts), boolean chains 2, error handling 1 (2 pts), loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ExpressionInfo.<init> (cognitive 17) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java:111— ExpressionInfo.<init> has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (12 pts), boolean chains 5 (nesting depth added 3). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
WkbWriter.calculateSize (cognitive 17) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/geo/WkbWriter.java:81— WkbWriter.calculateSize has cognitive complexity 17 (threshold 15). Drivers by points: loops 5 (10 pts), if/else 7 (nesting depth added 5). To reduce it, break up the iteration: give each loop body a named function, and split a multi-phase loop into one function per phase so no single body carries the whole pipeline.
BytesToBytesMap.safeLookup (cognitive 17) core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:663— BytesToBytesMap.safeLookup has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (16 pts), loops 1 (nesting depth added 10). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
VariantBuilder.appendWithArrayAppendImpl (cognitive 17) common/variant/src/main/java/org/apache/spark/types/variant/VariantBuilder.java:827— VariantBuilder.appendWithArrayAppendImpl has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (12 pts), loops 3, boolean chains 2 (nesting depth added 3). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JavaUtils.deleteRecursivelyUsingJavaIO (cognitive 17) common/utils-java/src/main/java/org/apache/spark/network/util/JavaUtils.java:282— JavaUtils.deleteRecursivelyUsingJavaIO has cognitive complexity 17 (threshold 15). Drivers by points: if/else 5 (7 pts), boolean chains 5, error handling 1 (3 pts), loops 1 (2 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CollationAwareUTF8String.lowercaseMatchLengthFrom (cognitive 17) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:94— CollationAwareUTF8String.lowercaseMatchLengthFrom has cognitive complexity 17 (threshold 15). Drivers by points: if/else 7 (12 pts), boolean chains 3, loops 2 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (CollationAwareUTF8String.lowercaseMatchLengthUntil) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
CollationAwareUTF8String.lowercaseMatchLengthUntil (cognitive 17) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:196— CollationAwareUTF8String.lowercaseMatchLengthUntil has cognitive complexity 17 (threshold 15). Drivers by points: if/else 7 (12 pts), boolean chains 3, loops 2 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body. This shape REPEATS in the file: one other method here (CollationAwareUTF8String.lowercaseMatchLengthFrom) has the same decision points, in the same order, at the same nesting depths — so this is one pattern written twice rather than two separate problems. Splitting this body alone leaves the other exactly as it is. Where these are variations on one operation, the change that clears both is the shared one: lift the common shape into a single routine the variants call, parameterised by whatever genuinely differs between them, and keep in each method only the part that is not shared.
CollationAwareUTF8String.lowercaseTrimLeft (cognitive 17) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1289— CollationAwareUTF8String.lowercaseTrimLeft has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (13 pts), loops 2, boolean chains 1, ternaries 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
UTF8String.levenshteinDistance (cognitive 17) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:2165— UTF8String.levenshteinDistance has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (8 pts), loops 3 (4 pts), ternaries 1 (4 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RemoteBlockPushResolver.registerExecutor (cognitive 17) common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java:876— RemoteBlockPushResolver.registerExecutor has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (14 pts), error handling 1 (2 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GcmEncryptedMessage.transferTo (cognitive 17) common/network-common/src/main/java/org/apache/spark/network/crypto/GcmTransportCipher.java:214— GcmEncryptedMessage.transferTo has cognitive complexity 17 (threshold 15). Drivers by points: if/else 8 (14 pts), error handling 1 (2 pts), loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InstanceList.countingRemoveAllByIndexValues (cognitive 17) common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java:239— InstanceList.countingRemoveAllByIndexValues has cognitive complexity 17 (threshold 15). Drivers by points: loops 3 (7 pts), if/else 4 (6 pts), ternaries 1 (3 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
cloudpickle._find_imported_submodules (cognitive 17) REDACTED:338— cloudpickle._find_imported_submodules has cognitive complexity 17 (threshold 15). Drivers by points: if/else 3 (11 pts), loops 2 (4 pts), boolean chains 2 (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Traceback.from_string (cognitive 17) REDACTED:284— Traceback.from_string has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (11 pts), loops 3 (5 pts), boolean chains 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowBatchedUDFLegacyPandasHandler.run (cognitive 17) python/pyspark/eval_handlers/_pandas.py:767— ArrowBatchedUDFLegacyPandasHandler.run has cognitive complexity 17 (threshold 15). Drivers by points: ternaries 3 (9 pts), if/else 2 (5 pts), loops 2 (3 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
utils.combine_frames (cognitive 17) python/pyspark/pandas/utils.py:118— utils.combine_frames has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (13 pts), boolean chains 2, loops 1 (2 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame.explain (cognitive 17) python/pyspark/sql/classic/dataframe.py:236— DataFrame.explain has cognitive complexity 17 (threshold 15). Drivers by points: boolean chains 8, if/else 5 (7 pts), ternaries 1 (2 pts) (nesting depth added 3). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
WriteOperation.command (cognitive 17) python/pyspark/sql/connect/plan.py:2197— WriteOperation.command has cognitive complexity 17 (threshold 15). Drivers by points: if/else 11 (16 pts), loops 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
WriteOperationV2.command (cognitive 17) python/pyspark/sql/connect/plan.py:2301— WriteOperationV2.command has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (15 pts), loops 2 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowToPandasConversion._convert_array_numpy (cognitive 17) python/pyspark/sql/conversion.py:2075— ArrowToPandasConversion._convert_array_numpy has cognitive complexity 17 (threshold 15). Drivers by points: if/else 12 (16 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowStreamSerializer.dump_stream (cognitive 17) python/pyspark/sql/pandas/serializers.py:78— ArrowStreamSerializer.dump_stream has cognitive complexity 17 (threshold 15). Drivers by points: if/else 9 (15 pts), loops 1, ternaries 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
data_source_pushdown_filters._main (cognitive 17) python/pyspark/sql/worker/data_source_pushdown_filters.py:118— data_source_pushdown_filters._main has cognitive complexity 17 (threshold 15). Drivers by points: if/else 10 (13 pts), loops 2, boolean chains 1, ternaries 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
mlutils.check_params (cognitive 17) python/pyspark/testing/mlutils.py:33— mlutils.check_params has cognitive complexity 17 (threshold 15). Drivers by points: if/else 6 (11 pts), ternaries 1 (4 pts), boolean chains 1, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
pandasutils._assert_pandas_equal (cognitive 17) python/pyspark/testing/pandasutils.py:61— pandasutils._assert_pandas_equal has cognitive complexity 17 (threshold 15). Drivers by points: error handling 3 (6 pts), ternaries 3 (6 pts), boolean chains 3, if/else 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkStringUtils.truncatedString (cognitive 16) common/utils/src/main/scala/org/apache/spark/util/SparkStringUtils.scala:111— SparkStringUtils.truncatedString has cognitive complexity 16 (threshold 15). Drivers by points: if/else 11 (16 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PipelinedWriterTask.awaitExit (cognitive 16) core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala:365— PipelinedWriterTask.awaitExit has cognitive complexity 16 (threshold 15). Drivers by points: error handling 3 (8 pts), if/else 3 (4 pts), match/switch 1 (3 pts), loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Master.scheduleExecutorsOnWorkers (cognitive 16) core/src/main/scala/org/apache/spark/deploy/master/Master.scala:766— Master.scheduleExecutorsOnWorkers has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (9 pts), boolean chains 4, loops 2 (3 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RestSubmissionClient.readResponse (cognitive 16) core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:323— RestSubmissionClient.readResponse has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (9 pts), error handling 4, match/switch 1 (2 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Worker.syncExecutorStateWithMaster (cognitive 16) core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala:811— Worker.syncExecutorStateWithMaster has cognitive complexity 16 (threshold 15). Drivers by points: if/else 3 (8 pts), error handling 1 (5 pts), match/switch 2 (3 pts) (nesting depth added 10). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
DAGScheduler.checkPipelinedStageCreation (cognitive 16) core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala:1593— DAGScheduler.checkPipelinedStageCreation has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (8 pts), match/switch 2 (5 pts), loops 2, boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskSetManager.getAllowedLocalityLevel (cognitive 16) core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:632— TaskSetManager.getAllowedLocalityLevel has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (10 pts), match/switch 2 (3 pts), loops 2, boolean chains 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskSetManager.checkAndSubmitSpeculatableTasks (cognitive 16) core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala:1301— TaskSetManager.checkAndSubmitSpeculatableTasks has cognitive complexity 16 (threshold 15). Drivers by points: if/else 4 (9 pts), boolean chains 6, loops 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AppStatusListener.onJobEnd (cognitive 16) core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:519— AppStatusListener.onJobEnd has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (8 pts), loops 4 (7 pts), match/switch 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
BlockManager.fetchRemoteManagedBuffer (cognitive 16) core/src/main/scala/org/apache/spark/storage/BlockManager.scala:1395— BlockManager.fetchRemoteManagedBuffer has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (12 pts), error handling 1 (2 pts), boolean chains 1, loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ListenerBus.postToAll (cognitive 16) core/src/main/scala/org/apache/spark/util/ListenerBus.scala:102— ListenerBus.postToAll has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (10 pts), error handling 2 (4 pts), boolean chains 1, loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MetricUtils.stringValue (cognitive 16) core/src/main/scala/org/apache/spark/util/MetricUtils.scala:51— MetricUtils.stringValue has cognitive complexity 16 (threshold 15). Drivers by points: if/else 13 (16 pts) (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
Utils.fetchFile (cognitive 16) core/src/main/scala/org/apache/spark/util/Utils.scala:365— Utils.fetchFile has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 2 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Utils.findLocalInetAddress (cognitive 16) core/src/main/scala/org/apache/spark/util/Utils.scala:867— Utils.findLocalInetAddress has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (12 pts), loops 1 (3 pts), boolean chains 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
EdgePartition.innerJoin (cognitive 16) graphx/src/main/scala/org/apache/spark/graphx/impl/EdgePartition.scala:283— EdgePartition.innerJoin has cognitive complexity 16 (threshold 15). Drivers by points: loops 3 (6 pts), boolean chains 5, if/else 2 (5 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LinearSVC.train (cognitive 16) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:184— LinearSVC.train has cognitive complexity 16 (threshold 15). Drivers by points: if/else 12 (13 pts), match/switch 2, boolean chains 1 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
GaussianMixture.trainImpl (cognitive 16) mllib/src/main/scala/org/apache/spark/ml/clustering/GaussianMixture.scala:499— GaussianMixture.trainImpl has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (11 pts), loops 2 (4 pts), boolean chains 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RandomForest.mixedBinSeqOp (cognitive 16) mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala:416— RandomForest.mixedBinSeqOp has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (12 pts), loops 2 (4 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
RandomForest.findSplitsForContinuousFeature (cognitive 16) mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala:1288— RandomForest.findSplitsForContinuousFeature has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (13 pts), loops 1 (3 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GaussianMixture.run (cognitive 16) mllib/src/main/scala/org/apache/spark/mllib/clustering/GaussianMixture.scala:164— GaussianMixture.run has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (10 pts), loops 2 (4 pts), boolean chains 1, match/switch 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
KubernetesClusterManager.createSchedulerBackend (cognitive 16) resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterManager.scala:52— KubernetesClusterManager.createSchedulerBackend has cognitive complexity 16 (threshold 15). Drivers by points: if/else 10 (14 pts), match/switch 1 (2 pts) (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
Client.createConfArchive (cognitive 16) resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:861— Client.createConfArchive has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (11 pts), loops 2 (3 pts), boolean chains 2 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowUtils.fromArrowField (cognitive 16) sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala:622— ArrowUtils.fromArrowField has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (10 pts), boolean chains 3, match/switch 2 (3 pts) (nesting depth added 4). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
LookupFunctions.apply (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:2408— LookupFunctions.apply has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (10 pts), match/switch 2 (5 pts), boolean chains 1 (nesting depth added 7). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ColumnResolutionHelper.resolveExprsAndAddMissingAttrs (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala:50— ColumnResolutionHelper.resolveExprsAndAddMissingAttrs has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (9 pts), match/switch 2 (5 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
FunctionResolution.resolveFunctionCandidate (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala:147— FunctionResolution.resolveFunctionCandidate has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (8 pts), error handling 3 (6 pts), match/switch 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InMemoryCatalog.dropTable (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala:237— InMemoryCatalog.dropTable has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (10 pts), error handling 2 (6 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SelectedField.selectField (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SelectedField.scala:83— SelectedField.selectField has cognitive complexity 16 (threshold 15). Drivers by points: match/switch 6 (11 pts), if/else 4, boolean chains 1 (nesting depth added 5). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
ApproxCountDistinctForIntervals.findHllppIndex (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/ApproxCountDistinctForIntervals.scala:166— ApproxCountDistinctForIntervals.findHllppIndex has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (12 pts), loops 1 (3 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ClassLoaderFileManager.resolveReferencedClasses (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:1692— ClassLoaderFileManager.resolveReferencedClasses has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (11 pts), boolean chains 3, loops 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GenerateUnsafeRowJoiner.create (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala:52— GenerateUnsafeRowJoiner.create has cognitive complexity 16 (threshold 15). Drivers by points: if/else 12 (15 pts), boolean chains 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
UpdateFieldsExpression.eval (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeCreator.scala:813— UpdateFieldsExpression.eval has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (13 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JsonTupleEvaluator.parseRow (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/json/JsonExpressionEvalUtils.scala:430— JsonTupleEvaluator.parseRow has cognitive complexity 16 (threshold 15). Drivers by points: if/else 4 (10 pts), loops 2 (6 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JsonArray.checkInputDataTypes (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:1418— JsonArray.checkInputDataTypes has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (11 pts), boolean chains 3, loops 1 (2 pts) (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
AtLeastNNonNulls.eval (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/nullExpressions.scala:609— AtLeastNNonNulls.eval has cognitive complexity 16 (threshold 15). Drivers by points: if/else 4 (11 pts), match/switch 1 (3 pts), boolean chains 1, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ToCharacterBuilder.build (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/numberFormatExpressions.scala:273— ToCharacterBuilder.build has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (10 pts), match/switch 2 (5 pts), boolean chains 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SchemaOfVariant.schemaOf (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/variant/variantExpressions.scala:2123— SchemaOfVariant.schemaOf has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (8 pts), loops 3 (6 pts), boolean chains 1, match/switch 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
FoldablePropagation.propagateFoldables (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala:1183— FoldablePropagation.propagateFoldables has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (9 pts), match/switch 3 (6 pts), boolean chains 1 (nesting depth added 6). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
AstBuilder.visitAlterTableAlterColumn (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:6871— AstBuilder.visitAlterTableAlterColumn has cognitive complexity 16 (threshold 15). Drivers by points: if/else 12, boolean chains 2, match/switch 1 (2 pts) (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
OperationHelper.collectProjectsAndFilters (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala:55— OperationHelper.collectProjectsAndFilters has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (11 pts), boolean chains 4, match/switch 1 (nesting depth added 4). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
FilterEstimation.evaluateInSet (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala:427— FilterEstimation.evaluateInSet has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (13 pts), boolean chains 2, match/switch 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TreeNode.generateTreeString (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala:1194— TreeNode.generateTreeString has cognitive complexity 16 (threshold 15). Drivers by points: if/else 13 (16 pts) (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
NumberConverter.convert (cognitive 16) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/NumberConverter.scala:139— NumberConverter.convert has cognitive complexity 16 (threshold 15). Drivers by points: if/else 11 (12 pts), boolean chains 4 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
GrpcExceptionConverter.errorsToThrowable (cognitive 16) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/GrpcExceptionConverter.scala:428— GrpcExceptionConverter.errorsToThrowable has cognitive complexity 16 (threshold 15). Drivers by points: if/else 12 (13 pts), boolean chains 2, match/switch 1 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ResponseValidator.verifyResponse (cognitive 16) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/ResponseValidator.scala:63— ResponseValidator.verifyResponse has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (9 pts), boolean chains 3, match/switch 1 (3 pts), error handling 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LiteralValueProtoConverter.getProtoDataTypeDefault (cognitive 16) sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/LiteralValueProtoConverter.scala:544— LiteralValueProtoConverter.getProtoDataTypeDefault has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (14 pts), boolean chains 1, match/switch 1 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkConnectPlanner.transformJvmOrigin (cognitive 16) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:2036— SparkConnectPlanner.transformJvmOrigin has cognitive complexity 16 (threshold 15). Drivers by points: if/else 16. To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
SparkConnectPlanner.transformWindowExpression (cognitive 16) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:2532— SparkConnectPlanner.transformWindowExpression has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (10 pts), match/switch 3 (6 pts) (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SparkConnectPlanner.transformSetOperation (cognitive 16) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala:2603— SparkConnectPlanner.transformSetOperation has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (13 pts), boolean chains 2, match/switch 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectAddArtifactsHandler.onNext (cognitive 16) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectAddArtifactsHandler.scala:62— SparkConnectAddArtifactsHandler.onNext has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (10 pts), match/switch 2 (4 pts), boolean chains 2 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
Dataset.showString (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/classic/Dataset.scala:368— Dataset.showString has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (10 pts), loops 2 (5 pts), boolean chains 1 (nesting depth added 5). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SparkSession.parseParameterizedPlan (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala:658— SparkSession.parseParameterizedPlan has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (12 pts), boolean chains 2, match/switch 1 (2 pts) (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CombineAdjacentAggregation.detachAggregate (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/CombineAdjacentAggregation.scala:117— CombineAdjacentAggregation.detachAggregate has cognitive complexity 16 (threshold 15). Drivers by points: match/switch 4 (8 pts), if/else 6 (7 pts), boolean chains 1 (nesting depth added 5). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
GeneratePredicateHelper.generatePredicateCode (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala:277— GeneratePredicateHelper.generatePredicateCode has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8, boolean chains 5, match/switch 2 (3 pts) (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
ArrowCachedBatchSerializer.drainAndClosePrefetch (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:445— ArrowCachedBatchSerializer.drainAndClosePrefetch has cognitive complexity 16 (threshold 15). Drivers by points: error handling 4 (9 pts), if/else 4 (5 pts), loops 1 (2 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CachedRDDBuilder.buildBuffers (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala:374— CachedRDDBuilder.buildBuffers has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7, boolean chains 4, match/switch 2 (4 pts), error handling 1 (nesting depth added 2). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataSource.resolveRelation (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala:372— DataSource.resolveRelation has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (13 pts), match/switch 2, boolean chains 1 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
VariantInRelation.rewriteType (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PushVariantIntoScan.scala:227— VariantInRelation.rewriteType has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (9 pts), match/switch 3 (7 pts) (nesting depth added 8). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SupportsArchiveFormat.readLocalizedEntries (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:114— SupportsArchiveFormat.readLocalizedEntries has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (7 pts), error handling 3 (5 pts), match/switch 2, boolean chains 1, loops 1 (nesting depth added 3). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InferVariantShreddingSchema.mergeSchema (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/InferVariantShreddingSchema.scala:152— InferVariantShreddingSchema.mergeSchema has cognitive complexity 16 (threshold 15). Drivers by points: if/else 4 (6 pts), loops 3 (6 pts), boolean chains 3, match/switch 1 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkToParquetSchemaConverter.convertFieldDefault (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala:673— SparkToParquetSchemaConverter.convertFieldDefault has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (10 pts), match/switch 2 (4 pts), boolean chains 2 (nesting depth added 5). The drivers above price the dispatch low by construction — a dispatch is charged once however many cases it lists, while each branch inside an arm is charged in full — so most of this count is what the case bodies hold, and the arms are where it can be reduced. To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
EnsureRequirements.resolveChild (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala:123— EnsureRequirements.resolveChild has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (8 pts), match/switch 3 (8 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PythonArrowOutput.newReaderIterator (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonArrowOutput.scala:59— PythonArrowOutput.newReaderIterator has cognitive complexity 16 (threshold 15). Drivers by points: if/else 10 (13 pts), match/switch 1 (2 pts), boolean chains 1 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
UserDefinedPythonTableFunctionAnalyzeRunner.receiveFromPython (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/python/UserDefinedPythonFunction.scala:381— UserDefinedPythonTableFunctionAnalyzeRunner.receiveFromPython has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (9 pts), match/switch 2 (4 pts), loops 3 (nesting depth added 4). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
FlatMapGroupsInPandasWithStateExec.createInputProcessor (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/FlatMapGroupsInPandasWithStateExec.scala:131— FlatMapGroupsInPandasWithStateExec.createInputProcessor has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (11 pts), boolean chains 3, match/switch 1 (2 pts) (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StatFunctions.summary (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/execution/stat/StatFunctions.scala:174— StatFunctions.summary has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (11 pts), boolean chains 2, match/switch 1 (2 pts), error handling 1 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SqlScriptingInterpreter.transformBodyIntoExec (cognitive 16) sql/core/src/main/scala/org/apache/spark/sql/scripting/SqlScriptingInterpreter.scala:70— SqlScriptingInterpreter.transformBodyIntoExec has cognitive complexity 16 (threshold 15). Drivers by points: if/else 13 (15 pts), match/switch 1 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
SparkSQLCLIDriver.processLine (cognitive 16) sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala:586— SparkSQLCLIDriver.processLine has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (14 pts), boolean chains 1, loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HiveExternalCatalog.createTable (cognitive 16) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala:216— HiveExternalCatalog.createTable has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (8 pts), boolean chains 4, error handling 1 (2 pts), match/switch 1 (2 pts) (nesting depth added 3). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
HiveMetastoreCatalog.convertToLogicalRelation (cognitive 16) sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala:202— HiveMetastoreCatalog.convertToLogicalRelation has cognitive complexity 16 (threshold 15). Drivers by points: if/else 12 (16 pts) (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
FlowResolver.attemptResolveFlow (cognitive 16) sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/CoreDataflowNodeProcessor.scala:120— FlowResolver.attemptResolveFlow has cognitive complexity 16 (threshold 15). Drivers by points: match/switch 5 (9 pts), if/else 6 (7 pts) (nesting depth added 5). To reduce it, keep the dispatch but shrink the arms: move each non-trivial case body into its own named function (or onto the value being matched) so the dispatch reads one line per case, and group related cases into a sub-dispatch. Keep every case explicit, and make the behaviour for cases you do not list a deliberate choice rather than an accident.
SparkDefaultUDAFEvaluatorResolver.getEvaluatorClass (cognitive 16) sql/hive/src/main/java/org/apache/hadoop/hive/ql/exec/SparkDefaultUDAFEvaluatorResolver.java:62— SparkDefaultUDAFEvaluatorResolver.getEvaluatorClass has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (11 pts), loops 4 (5 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PlainSaslServer.evaluateResponse (cognitive 16) sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/PlainSaslServer.java:57— PlainSaslServer.evaluateResponse has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (9 pts), boolean chains 3, error handling 3, loops 1 (nesting depth added 1). To reduce it, split the body: this score is breadth rather than depth — many checks laid out side by side rather than nested inside one another, so inverting conditions into early returns has nothing left to flatten. Group the statements between the checks into named steps and move each step into its own function, so the body reads as a short sequence of named stages.
4.run (cognitive 16) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/SessionManager.java:163— 4.run has cognitive complexity 16 (threshold 15). Drivers by points: if/else 3 (7 pts), error handling 1 (4 pts), loops 2 (3 pts), boolean chains 2 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VectorizedRleValuesReader.initFromPage (cognitive 16) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:100— VectorizedRleValuesReader.initFromPage has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (15 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
VectorizedRleValuesReader.readPackedBatch (cognitive 16) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:293— VectorizedRleValuesReader.readPackedBatch has cognitive complexity 16 (threshold 15). Drivers by points: if/else 4 (7 pts), loops 3 (7 pts), boolean chains 2 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ValueCopier.forTypeOrNull (cognitive 16) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ValueCopier.java:66— ValueCopier.forTypeOrNull has cognitive complexity 16 (threshold 15). Drivers by points: if/else 11 (13 pts), boolean chains 3 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition.
BytesToBytesMap.safeLookupWithKeyOperations (cognitive 16) core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:705— BytesToBytesMap.safeLookupWithKeyOperations has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (14 pts), boolean chains 1, loops 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
TaskMemoryManager.spillConsumersForPageAllocation (cognitive 16) core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java:350— TaskMemoryManager.spillConsumersForPageAllocation has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (9 pts), ternaries 1 (3 pts), boolean chains 2, loops 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CollationAwareUTF8String.binaryTrimRight (cognitive 16) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1410— CollationAwareUTF8String.binaryTrimRight has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (9 pts), loops 4 (5 pts), boolean chains 2 (nesting depth added 3). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
CollationAwareUTF8String.lowercaseSubStringIndex (cognitive 16) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1041— CollationAwareUTF8String.lowercaseSubStringIndex has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (11 pts), loops 2 (4 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CollationAwareUTF8String.translate (cognitive 16) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1167— CollationAwareUTF8String.translate has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (13 pts), loops 2 (3 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CollationAwareUTF8String.trimLeft (cognitive 16) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationAwareUTF8String.java:1355— CollationAwareUTF8String.trimLeft has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (12 pts), loops 3 (4 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
HeapMemoryAllocator.allocate (cognitive 16) common/unsafe/src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java:48— HeapMemoryAllocator.allocate has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (13 pts), loops 1 (3 pts) (nesting depth added 10). To reduce it, flatten the nesting: this score is depth rather than breadth — most of its points come from checks stacked inside one another, so the work sits several levels in. Invert each enclosing check into an early exit (a return, or the language's equivalent) so the happy path stays at one level, and where a level cannot be exited early, lift the block it encloses into its own named function.
merge_spark_pr.cherry_pick (cognitive 16) REDACTED:1030— merge_spark_pr.cherry_pick has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (8 pts), error handling 3 (7 pts), loops 1 (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowElementwiseUDFHandler.run (cognitive 16) python/pyspark/eval_handlers/_elementwise.py:240— ArrowElementwiseUDFHandler.run has cognitive complexity 16 (threshold 15). Drivers by points: loops 3 (6 pts), if/else 1 (4 pts), error handling 1 (3 pts), ternaries 1 (3 pts) (nesting depth added 10). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
ArrowTableUDTFArrowHandler._check_return_value (cognitive 16) REDACTED:1184— ArrowTableUDTFArrowHandler._check_return_value has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (13 pts), loops 1 (2 pts), boolean chains 1 (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SummarizerAggState.to_result (cognitive 16) python/pyspark/ml/connect/summarizer.py:50— SummarizerAggState.to_result has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (15 pts), loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
JavaParams._transfer_params_from_java (cognitive 16) python/pyspark/ml/wrapper.py:240— JavaParams._transfer_params_from_java has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (13 pts), boolean chains 2, loops 1 (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
FractionalExtensionOps.astype (cognitive 16) python/pyspark/pandas/data_type_ops/num_ops.py:689— FractionalExtensionOps.astype has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (14 pts), boolean chains 2 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
DataFrame._mark_duplicates (cognitive 16) python/pyspark/pandas/frame.py:5258— DataFrame._mark_duplicates has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (12 pts), ternaries 1 (3 pts), boolean chains 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
GroupBy._spark_groupby (cognitive 16) python/pyspark/pandas/groupby.py:363— GroupBy._spark_groupby has cognitive complexity 16 (threshold 15). Drivers by points: ternaries 3 (7 pts), if/else 3 (6 pts), loops 2 (3 pts) (nesting depth added 8). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
MultiIndex.drop (cognitive 16) python/pyspark/pandas/indexes/multi.py:837— MultiIndex.drop has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (14 pts), loops 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
InternalFrame.restore_index (cognitive 16) python/pyspark/pandas/internal.py:1123— InternalFrame.restore_index has cognitive complexity 16 (threshold 15). Drivers by points: if/else 5 (6 pts), boolean chains 4, ternaries 3 (4 pts), loops 2 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
ExternalMerger.mergeCombiners (cognitive 16) python/pyspark/shuffle.py:326— ExternalMerger.mergeCombiners has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (10 pts), ternaries 2 (4 pts), boolean chains 1, loops 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectClient._analyze (cognitive 16) python/pyspark/sql/connect/client/core.py:1592— SparkConnectClient._analyze has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (13 pts), boolean chains 1, error handling 1, loops 1 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
SparkConnectClient._handle_rpc_error (cognitive 16) python/pyspark/sql/connect/client/core.py:2405— SparkConnectClient._handle_rpc_error has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (14 pts), loops 1 (2 pts) (nesting depth added 9). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
LiteralExpression.__repr__ (cognitive 16) python/pyspark/sql/connect/expressions.py:521— LiteralExpression.__repr__ has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (11 pts), boolean chains 5 (nesting depth added 5). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
PandasToArrowConversion.from_pandas (cognitive 16) python/pyspark/sql/conversion.py:356— PandasToArrowConversion.from_pandas has cognitive complexity 16 (threshold 15). Drivers by points: if/else 7 (9 pts), boolean chains 3, ternaries 2 (3 pts), loops 1 (nesting depth added 3). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
types.from_arrow_type (cognitive 16) python/pyspark/sql/pandas/types.py:388— types.from_arrow_type has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9 (15 pts), boolean chains 1 (nesting depth added 6). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
CatalystTranspiler._safe_category (cognitive 16) python/pyspark/sql/transpile.py:264— CatalystTranspiler._safe_category has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (7 pts), ternaries 3 (6 pts), boolean chains 2, error handling 1 (nesting depth added 4). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
StructType.toInternal (cognitive 16) python/pyspark/sql/types.py:1999— StructType.toInternal has cognitive complexity 16 (threshold 15). Drivers by points: ternaries 3 (9 pts), if/else 6 (7 pts) (nesting depth added 7). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
types._merge_type (cognitive 16) python/pyspark/sql/types.py:3296— types._merge_type has cognitive complexity 16 (threshold 15). Drivers by points: if/else 6 (8 pts), boolean chains 6, loops 1 (2 pts) (nesting depth added 3). To reduce it, split the body into named stages: move each independent step or branch into its own named function so the body reads as a short sequence of named calls rather than one long body.
spark-sql-viz.formatMetricValue (cognitive 16) REDACTED:344— spark-sql-viz.formatMetricValue has cognitive complexity 16 (threshold 15). Drivers by points: if/else 8 (10 pts), boolean chains 5, loops 1 (nesting depth added 2). To reduce it, split the body: most of this score is breadth rather than depth — checks laid out side by side rather than stacked — so group the statements between the checks into named steps and move each step into its own function. Some of it IS depth: where a check sits inside another whose only job is to reach it, merge the two into one condition, and where an else follows a branch that already returns, drop the trailing else and let the rest of the body continue at one level.
spark-sql-viz.setupPlanVizSearch (cognitive 16) REDACTED:850— spark-sql-viz.setupPlanVizSearch has cognitive complexity 16 (threshold 15). Drivers by points: if/else 9, boolean chains 5, ternaries 1 (2 pts) (nesting depth added 1). Most of this is not in the body itself: 3 of the 16 points are its own statements and the rest belongs to 3 function literals inside it that branch (lines 888, 875, 868). The decisions are inside those literals, which nothing outside this body can call, review or test on its own, so splitting the enclosing body is not the move available here. To reduce it, lift the literals' work into a named function or method at the enclosing scope and have each literal call it, then reduce whichever part then reads as the largest.
Change coupling clique: CSVFileFormat.scala, JsonFileFormat.scala, XmlFileFormat.scala sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala— 3 files — `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala`, `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala`, `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala` — all change together with no explicit dependency: a fully-connected co-change clique, not 3 separate couplings. They share one concern (thin parallel siblings over a common abstraction), so extract the shared part into ONE unit and the whole clique's coupling clears at once — you do not need to break each pair individually.
D4 · Code Duplication· Near-duplicate member family (6 members, 27 shared lines) · ×1
Near-duplicate member family (6 members, 27 shared lines) python/pyspark/sql/connect/readwriter.py:210— python/pyspark/sql/connect/readwriter.py:210-263 | python/pyspark/sql/connect/readwriter.py:351-403 | python/pyspark/sql/connect/streaming/readwriter.py:352-394 | python/pyspark/sql/readwriter.py:358-531 | python/pyspark/sql/readwriter.py:814-947 | python/pyspark/sql/streaming/readwriter.py:710-801 — These 6 members are variants of one another: a block of 27 lines reported below appears in every one of them, and the pairwise near-duplicate rows they would otherwise produce are collapsed into this row. Read them as one construct written 6 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 6 times.
D4 · Code Duplication· Near-duplicate member family (4 members, 29 shared lines) · ×1
Near-duplicate member family (4 members, 29 shared lines) core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:30— core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:30-78 | core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala:29-77 | core/src/main/scala/org/apache/spark/deploy/worker/ui/LogPage.scala:64-132 | core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala:46-92 — These 4 members are variants of one another: a block of 29 lines reported below appears in every one of them, and the pairwise near-duplicate rows they would otherwise produce are collapsed into this row. Read them as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
D4 · Code Duplication· Near-duplicate member family (3 members, 21 shared lines) · ×1
Near-duplicate member family (3 members, 21 shared lines) python/pyspark/pandas/frame.py:1546— python/pyspark/pandas/frame.py:1546-1761 | python/pyspark/pandas/groupby.py:4108-4312 | python/pyspark/pandas/groupby.py:4321-4522 — These 3 members are variants of one another: a block of 21 lines reported below appears in every one of them, and the pairwise near-duplicate rows they would otherwise produce are collapsed into this row. Read them as one construct written 3 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 3 times.
Near-duplicate member pair (228 shared lines) python/pyspark/pandas/frame.py:8822— python/pyspark/pandas/frame.py:8822-9156 | REDACTED:3126-3601 — These two members are variants of one another: 228 of their lines are already reported as duplicated blocks below, spread through both bodies rather than gathered into one. Read them as a single construct written twice. The repair is at the members' grain — factor the shared pipeline into one implementation the two call with their differences as parameters or as an injected step, or, where the difference is systematic (sync against async, one transport against another), generate one from the other. Extracting the individual blocks below is not the same fix: it leaves the two bodies in place and the next edit still has to be made twice.
Near-duplicate member pair (67 shared lines) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:92— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:92-207 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:249-372 — These two members are variants of one another: 67 of their lines are already reported as duplicated blocks below, spread through both bodies rather than gathered into one. Read them as a single construct written twice. The repair is at the members' grain — factor the shared pipeline into one implementation the two call with their differences as parameters or as an injected step, or, where the difference is systematic (sync against async, one transport against another), generate one from the other. Extracting the individual blocks below is not the same fix: it leaves the two bodies in place and the next edit still has to be made twice.
Near-duplicate member pair (62 shared lines) core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:541— core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala:541-607 | resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala:152-223 — These two members are variants of one another: 62 of their lines are already reported as duplicated blocks below, spread through both bodies rather than gathered into one. Read them as a single construct written twice. The repair is at the members' grain — factor the shared pipeline into one implementation the two call with their differences as parameters or as an injected step, or, where the difference is systematic (sync against async, one transport against another), generate one from the other. Extracting the individual blocks below is not the same fix: it leaves the two bodies in place and the next edit still has to be made twice.
Near-duplicate member pair (59 shared lines) python/pyspark/pandas/frame.py:12607— python/pyspark/pandas/frame.py:12607-12745 | python/pyspark/pandas/frame.py:12748-12880 — These two members are variants of one another: 59 of their lines are already reported as duplicated blocks below, spread through both bodies rather than gathered into one. Read them as a single construct written twice. The repair is at the members' grain — factor the shared pipeline into one implementation the two call with their differences as parameters or as an injected step, or, where the difference is systematic (sync against async, one transport against another), generate one from the other. Extracting the individual blocks below is not the same fix: it leaves the two bodies in place and the next edit still has to be made twice.
D4 · Code Duplication· Edited copy of a member (23 corresponding lines) · ×1
Edited copy of a member (23 corresponding lines) common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:130— common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:130-152 | common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java:132-154 — These two members are one piece of code written twice and then edited apart: 23 consecutive lines correspond almost exactly, broken only by small local edits. Most of that correspondence is NOT reported as duplicated blocks below — the edits cut it into fragments and only the largest of them clear the block floor, so the rows below understate it. The repair is at the members' grain — factor the shared implementation into one the two call with their differences as parameters or as an injected step, or, where the difference is systematic (an extra return value, one transport against another), generate one from the other. Left alone, the next edit has to be made twice and the two will drift further apart.
D4 · Code Duplication· Edited copy of a member (12 corresponding lines) · ×1
Edited copy of a member (12 corresponding lines) mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:208— mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:208-222 | mllib/src/main/scala/org/apache/spark/mllib/classification/NaiveBayes.scala:252-265 — These two members are one piece of code written twice and then edited apart: 12 consecutive lines correspond almost exactly, broken only by small local edits. Most of that correspondence is NOT reported as duplicated blocks below — the edits cut it into fragments and only the largest of them clear the block floor, so the rows below understate it. The repair is at the members' grain — factor the shared implementation into one the two call with their differences as parameters or as an injected step, or, where the difference is systematic (an extra return value, one transport against another), generate one from the other. Left alone, the next edit has to be made twice and the two will drift further apart.
D4 · Code Duplication· Edited copy of a member (24 corresponding lines) · ×1
Edited copy of a member (24 corresponding lines) common/sketch/src/main/java/org/apache/spark/util/sketch/Platform.java:111— common/sketch/src/main/java/org/apache/spark/util/sketch/Platform.java:111-134 | common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java:264-287 — These two members are one piece of code written twice and then edited apart: 24 consecutive lines correspond almost exactly, broken only by small local edits. Most of that correspondence is NOT reported as duplicated blocks below — the edits cut it into fragments and only the largest of them clear the block floor, so the rows below understate it. The repair is at the members' grain — factor the shared implementation into one the two call with their differences as parameters or as an injected step, or, where the difference is systematic (an extra return value, one transport against another), generate one from the other. Left alone, the next edit has to be made twice and the two will drift further apart.
D4 · Code Duplication· Members sharing a duplicated core (24 members, 50+ identical tokens) · ×1
Members sharing a duplicated core (24 members, 50+ identical tokens) sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:179— sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:179-193 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:199-214 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:220-236 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:242-260 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:281-300 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:322-342 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:365-403 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:427-467 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:492-534 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:560-604 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:632-678 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:707-755 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:205-219 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:221-236 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:238-254 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:256-274 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:276-295 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:297-317 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:319-357 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:359-399 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:401-443 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:445-489 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:491-537 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:540-588 — These 24 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 24 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 24 times.
D4 · Code Duplication· Members sharing a duplicated core (22 members, 50+ identical tokens) · ×1
Members sharing a duplicated core (22 members, 50+ identical tokens) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:273— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:273-284 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:302-314 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:333-346 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:366-380 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:401-416 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:438-454 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:478-495 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:520-538 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:564-583 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:610-630 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:658-679 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:725-747 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:795-818 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:868-892 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:944-969 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1023-1049 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1105-1132 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18203-18213 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18232-18243 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18263-18275 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18296-18309 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18332-18346 — These 22 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 22 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 22 times.
D4 · Code Duplication· Members sharing a duplicated core (10 members, 50+ identical tokens) · ×1
Members sharing a duplicated core (10 members, 50+ identical tokens) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:142— sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:142-158 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:164-176 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:182-194 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:202-216 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:224-239 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:245-257 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:265-282 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:289-303 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:453-468 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:473-491 — These 10 members share a duplicated core: a run of at least 50 identical tokens appears in every one of them. That run is NOT broken out as duplicated-block rows below — it is what admitted this row, and the blocks below cover only the part of it that clears the block floor, so they understate the correspondence. Read the members as one construct written 10 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 10 times.
Duplicated block (63–64 lines × 2) connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:393— connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala:393-456 | connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala:431-493 — before extracting anything, compare `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderAdmin.scala` and `connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReaderConsumer.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 153 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (62 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1110— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1110-1171 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlInputFormat.scala:257-318 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/StaxXmlParser.scala:1110` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (51 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:389— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:389-439 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:536-586 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (45 lines × 2) mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:736— mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:736-780 | mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:892-936 — before extracting anything, compare `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala` and `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala` as WHOLE FILES: this scan already matched 17 separate duplicated blocks between them, totalling at least 331 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala:890` calls `Since` and `mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala:733` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (41–42 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:803— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:803-844 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:1259-1299 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala:803` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17–40 lines × 16) sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:284— sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:284-300 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:325-342 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:369-403 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:432-467 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:498-534 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:567-604 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:640-678 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:716-755 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:279-295 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:300-317 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:323-357 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:364-399 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:407-443 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:452-489 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:499-537 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:549-588 — before extracting anything, compare `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala` and `sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 144 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:284` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (40 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:398— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:398-437 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:1693-1732 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/SymmetricHashJoinStateManager.scala:398` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (29–33 lines × 2) sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala:87— sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala:87-115 | sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUDFs.scala:171-203 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (28–30 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala:814— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala:814-843 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/intervalExpressions.scala:889-916 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (14–29 lines × 18) sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:245— sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:245-258 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:284-297 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:325-338 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:370-398 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:433-461 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:499-527 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:568-596 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:641-669 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:717-745 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:259-272 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:279-292 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:300-313 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:324-352 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:365-393 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:408-436 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:453-481 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:500-528 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:550-578 — before extracting anything, compare `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala` and `sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 144 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:245` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (26–29 lines × 2) connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala:94— connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala:94-122 | connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala:151-176 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `connector/kafka-0-10/src/main/scala/org/apache/spark/streaming/kafka010/ConsumerStrategy.scala:94` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (28 lines × 3) core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:38— core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala:38-65 | core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala:37-64 | core/src/main/scala/org/apache/spark/ui/DriverLogPage.scala:52-79 — before extracting anything, compare `core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala` and `core/src/main/scala/org/apache/spark/deploy/master/ui/LogPage.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 69 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (13–27 lines × 20) sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:222— sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:222-234 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:245-257 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:284-296 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:325-337 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:371-397 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:434-460 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:500-526 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:569-595 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:642-668 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:718-744 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:240-252 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:259-271 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:279-291 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:300-312 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:325-351 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:366-392 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:409-435 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:454-480 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:501-527 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:551-577 — before extracting anything, compare `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala` and `sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 144 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:222` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18–27 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1932— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:1932-1958 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:2619-2636 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (26–27 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowWindowPythonEvaluatorFactory.scala:280— sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowWindowPythonEvaluatorFactory.scala:280-306 | sql/core/src/main/scala/org/apache/spark/sql/execution/window/WindowEvaluatorFactory.scala:77-102 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice.
Duplicated block (12–25 lines × 22) sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:201— sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:201-212 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:222-233 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:245-256 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:284-295 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:325-336 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:372-396 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:435-459 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:501-525 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:570-594 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:643-667 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:719-743 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:223-234 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:240-251 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:259-270 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:279-290 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:300-311 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:326-350 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:367-391 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:410-434 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:455-479 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:502-526 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:552-576 — before extracting anything, compare `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala` and `sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 144 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:201` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (23–24 lines × 3) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala:188— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/csvExpressions.scala:188-211 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:2178-2200 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/xmlExpressions.scala:217-240 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (11–23 lines × 24) sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:181— sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:181-191 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:201-211 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:222-232 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:245-255 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:284-294 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:325-335 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:373-395 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:436-458 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:502-524 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:571-593 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:644-666 | sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:720-742 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:207-217 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:223-233 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:240-250 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:259-269 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:279-289 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:300-310 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:327-349 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:368-390 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:411-433 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:456-478 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:503-525 | sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala:553-575 — before extracting anything, compare `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala` and `sql/connect/common/src/main/scala/org/apache/spark/sql/connect/common/UdfUtils.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 144 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/internal/ToScalaUDF.scala:181` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (21–22 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:152— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:152-173 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:316-336 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:152` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (21 lines × 3) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala:66— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala:66-86 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala:130-150 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala:194-214 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (20–21 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:154— sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:154-174 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/transformwithstate/TransformWithStateExec.scala:171-190 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/python/streaming/TransformWithStateInPySparkExec.scala:154` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19–20 lines × 6) sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:115— sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:115-133 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:165-184 | sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala:111-129 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:89-107 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala:136-155 | sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerSessionPage.scala:91-109 — before extracting anything, compare `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala` and `sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/ui/ThriftServerPage.scala` as WHOLE FILES: this scan already matched 9 separate duplicated blocks between them, totalling at least 118 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala:115` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18–20 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:109— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:109-128 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:305-322 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:109` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:322` calls `Some`, `TypeCheckFailure` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/avroSqlFunctions.scala:128` does not — after which the two agree again for 5 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (16–20 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:6547— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:6547-6566 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:6634-6649 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:6547` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:6633` calls `replaceTableHeader`, `identifierReference` and `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:6545` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (18 lines × 3) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1623— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1623-1640 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1761-1778 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1848-1865 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:1623` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11–17 lines × 3) mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:326— mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala:326-336 | mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala:995-1011 | mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala:323-334 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/classification/LinearSVC.scala` and `mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala` as WHOLE FILES: this scan already matched 6 separate duplicated blocks between them, totalling at least 65 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (16 lines × 4) core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:124— core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:124-139 | core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:153-168 | core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:182-197 | core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:211-226 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala:124` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–16 lines × 4) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:126— sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:126-141 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowEvalPythonEvaluatorFactory.scala:121-136 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonEvaluatorFactory.scala:92-107 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonIncrementalAggregateExec.scala:355-368 — before extracting anything, compare `sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowEvalPythonEvaluatorFactory.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonEvaluatorFactory.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 36 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:126` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11–16 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:900— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:900-910 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:921-936 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala:900` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (14–15 lines × 4) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:86— mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:86-99 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala:83-96 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala:85-98 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregator.scala:91-105 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala` and `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:86` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (13–15 lines × 2) core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala:286— core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala:286-298 | core/src/main/scala/org/apache/spark/storage/BlockManager.scala:962-976 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala:286` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12–15 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:62— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:62-73 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSetVariable.scala:60-74 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveFetchCursor.scala:62` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12–14 lines × 4) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:131— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:131-144 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:502-515 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:601-612 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/bitmapExpressions.scala:699-710 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (14 lines × 4) REDACTED:1574— REDACTED:1574-1587 | REDACTED:1712-1725 | REDACTED:2800-2813 | REDACTED:4523-4536 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:1574` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10–14 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala:271— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala:271-284 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala:463-472 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/collect.scala:271` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13 lines × 9) sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:622— sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:622-634 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:646-658 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:670-682 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:694-706 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:718-730 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:743-755 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:768-780 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:931-943 | sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:980-992 — all 9 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ArrowCachedBatchSerializer.scala:622` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (12–13 lines × 4) sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:464— sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:464-475 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:479-491 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:585-597 | sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:649-661 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:479` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (10–13 lines × 2) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowEvalPythonEvaluatorFactory.scala:95— sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowEvalPythonEvaluatorFactory.scala:95-107 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonEvaluatorFactory.scala:70-79 — before extracting anything, compare `sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowEvalPythonEvaluatorFactory.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonEvaluatorFactory.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 36 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (12 lines × 4) mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:126— mllib/src/main/scala/org/apache/spark/ml/r/AFTSurvivalRegressionWrapper.scala:126-137 | mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala:129-140 | mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala:96-107 | mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala:124-135 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/FMRegressorWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/LinearRegressionWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 40 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (9–11 lines × 2) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:749— sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:749-759 | sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:1124-1132 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:749` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9–10 lines × 5) core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:220— core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:220-228 | core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala:262-270 | streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala:480-488 | streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala:502-510 | streaming/src/main/scala/org/apache/spark/streaming/ui/StreamingPage.scala:524-533 — there are 5 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 5 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (9–10 lines × 4) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:129— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:129-138 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:275-284 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala:151-160 | sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala:204-212 — before extracting anything, compare `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 38 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:129` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9 lines × 17) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:274— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:274-282 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:303-311 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:334-342 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:367-375 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:402-410 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:439-447 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:479-487 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:521-529 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:565-573 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:611-619 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:659-667 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:726-734 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:796-804 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:869-877 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:945-953 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1024-1032 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1106-1114 — all 17 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:274` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8–9 lines × 5) mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:84— mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala:84-91 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala:81-88 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HuberBlockAggregator.scala:83-90 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/LeastSquaresBlockAggregator.scala:89-97 | mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/MultinomialLogisticBlockAggregator.scala:104-111 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/BinaryLogisticBlockAggregator.scala` and `mllib/src/main/scala/org/apache/spark/ml/optim/aggregator/HingeBlockAggregator.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (8 lines × 7) mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:80— mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:80-87 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:72-79 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:77-84 | mllib/src/main/scala/org/apache/spark/ml/r/GaussianMixtureWrapper.scala:75-82 | mllib/src/main/scala/org/apache/spark/ml/r/IsotonicRegressionWrapper.scala:64-71 | mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala:82-89 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:78-85 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/KMeansWrapper.scala` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 39 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `mllib/src/main/scala/org/apache/spark/ml/r/BisectingKMeansWrapper.scala:79` calls `setFeaturesCol` and `mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:71` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (8 lines × 6) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:133— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:133-140 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala:279-286 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala:562-569 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala:155-162 | sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveFileFormat.scala:130-137 | sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFileFormat.scala:207-214 — before extracting anything, compare `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 38 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala:133` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (7 lines × 20) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:306— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:306-312 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:337-343 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:370-376 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:405-411 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:442-448 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:482-488 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:524-530 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:568-574 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:614-620 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:662-668 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:729-735 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:799-805 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:872-878 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:948-954 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1027-1033 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1109-1115 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18235-18241 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18266-18272 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18299-18305 | sql/api/src/main/scala/org/apache/spark/sql/functions.scala:18335-18341 — there are 20 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 20 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (7 lines × 9) sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:575— sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:575-581 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:621-627 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:669-675 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:736-742 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:806-812 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:879-885 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:955-961 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1034-1040 | sql/api/src/main/scala/org/apache/spark/sql/UDFRegistration.scala:1116-1122 — all 9 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (7 lines × 8) core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:423— core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:423-429 | core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:438-444 | core/src/main/scala/org/apache/spark/ui/jobs/AllJobsPage.scala:453-459 | core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:462-468 | core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:477-483 | core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:492-498 | core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:507-513 | core/src/main/scala/org/apache/spark/ui/jobs/JobPage.scala:522-528 — there are 8 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 8 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (7 lines × 6) mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:126— mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeClassifierWrapper.scala:126-132 | mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala:109-115 | mllib/src/main/scala/org/apache/spark/ml/r/GBTClassifierWrapper.scala:133-139 | mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala:117-123 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestClassifierWrapper.scala:136-142 | mllib/src/main/scala/org/apache/spark/ml/r/RandomForestRegressorWrapper.scala:119-125 — before extracting anything, compare `mllib/src/main/scala/org/apache/spark/ml/r/DecisionTreeRegressorWrapper.scala` and `mllib/src/main/scala/org/apache/spark/ml/r/GBTRegressorWrapper.scala` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (4–6 lines × 4) sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:141— sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:141-144 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:170-173 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:238-243 | sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:293-298 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:141` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:169` calls `create`, `getOrElse` and `sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinEvaluatorFactory.scala:141` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (5 lines × 5) sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:127— sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:127-133 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowEvalPythonEvaluatorFactory.scala:122-128 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonEvaluatorFactory.scala:93-99 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonUDTFExec.scala:78-84 | sql/core/src/main/scala/org/apache/spark/sql/execution/python/PythonIncrementalAggregateExec.scala:356-360 — before extracting anything, compare `sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowEvalPythonEvaluatorFactory.scala` and `sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonEvaluatorFactory.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 36 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowAggregatePythonExec.scala:127` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (4–5 lines × 3) core/src/main/scala/org/apache/spark/ui/UIUtils.scala:220— core/src/main/scala/org/apache/spark/ui/UIUtils.scala:220-223 | core/src/main/scala/org/apache/spark/ui/UIUtils.scala:237-241 | core/src/main/scala/org/apache/spark/ui/UIUtils.scala:258-263 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `core/src/main/scala/org/apache/spark/ui/UIUtils.scala:237` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (2–5 lines × 3) mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala:200— mllib/src/main/scala/org/apache/spark/mllib/util/LinearDataGenerator.scala:200-204 | mllib/src/main/scala/org/apache/spark/mllib/util/LogisticRegressionDataGenerator.scala:78-82 | mllib/src/main/scala/org/apache/spark/mllib/util/SVMDataGenerator.scala:49-50 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (43 lines × 2) connector/spark-ganglia-lgpl/src/main/scala/org/apache/spark/metrics/sink/GangliaSink.scala:51— connector/spark-ganglia-lgpl/src/main/scala/org/apache/spark/metrics/sink/GangliaSink.scala:51-93 | core/src/main/scala/org/apache/spark/metrics/sink/GraphiteSink.scala:43-102 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. Note first that the copies are not typed on the same thing: the declarations holding them bind `prop` to `String): Option[String] = Option(property.getProperty(prop)) if (!propertyToOption(GANGLIA_KEY_HOST).isDefined` in one and `String): Option[String] = Option(property.getProperty(prop)) if (propertyToOption(GRAPHITE_KEY_HOST).isEmpty` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (5 lines × 7) sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/PythonStreamingSinkCommitRunner.scala:47— sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/PythonStreamingSinkCommitRunner.scala:47-51 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/UserDefinedPythonDataSource.scala:269-273 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/UserDefinedPythonDataSource.scala:325-329 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/UserDefinedPythonDataSource.scala:510-514 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/UserDefinedPythonDataSource.scala:639-643 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/UserDefinedPythonDataSource.scala:688-692 | sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/python/UserDefinedPythonDataSource.scala:753-757 — there are 7 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 7 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (52–53 lines × 2) common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1847— common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1847-1899 | common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1944-1995 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java:1847` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (10–16 lines × 2) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java:106— sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java:106-121 | sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java:201-210 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaByteArrayReader.java:106` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11–13 lines × 3) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:242— sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:242-252 | sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:352-362 | sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:481-493 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:242` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11 lines × 3) common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:215— common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:215-225 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:253-263 | common/unsafe/src/main/java/org/apache/spark/sql/catalyst/util/CollationSupport.java:292-302 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Each matched range is the entire body of the declaration above it, so the region is already a complete unit: move that whole declaration to the shared location and have each site call it, rather than lifting the lines out of their bodies. Any `return` inside it is the body's own exit and keeps its meaning in the moved unit.
Duplicated block (9 lines × 10) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:150— sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:150-158 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:168-176 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:186-194 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:208-216 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:231-239 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:249-257 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:274-282 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:295-303 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:460-468 | sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:483-491 — all 10 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java:150` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (3–8 lines × 9) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:55— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:55-62 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:65-67 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:139-146 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:149-151 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:209-216 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:219-221 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:274-281 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:325-332 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:410-417 — all 9 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:55` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:282` calls `isNullAt` and `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:68` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (4–7 lines × 3) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:80— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:80-83 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:166-172 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:343-349 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:166` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (2–6 lines × 9) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:68— sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:68-73 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:77-78 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:152-157 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:161-162 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:222-227 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:231-232 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:283-288 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:334-339 | sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:418-421 — all 9 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:68` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:68` calls `isNullAt` and `sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VectorFunctionImplUtils.java:418` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (4–6 lines × 3) sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java:510— sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java:510-513 | sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java:517-522 | sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java:525-530 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `sql/catalyst/src/main/java/org/apache/datasketches/memory/internal/ResourceImpl.java:510` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (128–268 lines × 2) python/pyspark/pandas/frame.py:8822— python/pyspark/pandas/frame.py:8822-8949 | REDACTED:3126-3393 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:8822` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note first that the copies are not typed on the same thing: `Index` names `pyspark.pandas.indexes` in one and `pyspark.pandas.indexes.base` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (4–164 lines × 3) python/pyspark/sql/functions/builtin.py:27807— python/pyspark/sql/functions/builtin.py:27807-27810 | python/pyspark/sql/functions/builtin.py:27824-27987 | python/pyspark/sql/functions/builtin.py:28000-28158 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (163 lines × 2) python/pyspark/sql/functions/builtin.py:28417— python/pyspark/sql/functions/builtin.py:28417-28579 | python/pyspark/sql/functions/builtin.py:28661-28835 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/functions/builtin.py:28417` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (122–131 lines × 2) python/pyspark/pandas/strings.py:1942— python/pyspark/pandas/strings.py:1942-2072 | python/pyspark/pandas/strings.py:2111-2232 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/strings.py:1942` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (118 lines × 2) python/pyspark/sql/functions/builtin.py:29139— python/pyspark/sql/functions/builtin.py:29139-29263 | python/pyspark/sql/functions/builtin.py:29309-29426 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/functions/builtin.py:29139` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (103–114 lines × 2) python/pyspark/sql/protobuf/functions.py:40— python/pyspark/sql/protobuf/functions.py:40-153 | python/pyspark/sql/protobuf/functions.py:174-276 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/protobuf/functions.py:40` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (103 lines × 2) python/pyspark/sql/classic/dataframe.py:1559— python/pyspark/sql/classic/dataframe.py:1559-1661 | python/pyspark/sql/connect/dataframe.py:1525-1627 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own.
Duplicated block (62–101 lines × 3) python/pyspark/sql/functions/builtin.py:24334— python/pyspark/sql/functions/builtin.py:24334-24395 | python/pyspark/sql/functions/builtin.py:24632-24732 | python/pyspark/sql/functions/builtin.py:24797-24892 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/functions/builtin.py:24334` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (99–100 lines × 2) python/pyspark/pandas/frame.py:6314— python/pyspark/pandas/frame.py:6314-6413 | python/pyspark/pandas/series.py:2130-2228 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:6314` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (83–92 lines × 3) python/pyspark/pandas/generic.py:1867— python/pyspark/pandas/generic.py:1867-1954 | python/pyspark/pandas/generic.py:1971-2053 | python/pyspark/pandas/generic.py:2198-2289 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Note first that the copies are not typed on the same thing: the declarations holding them bind `numeric_only` to `bool = False` in one and `bool = None` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (86–88 lines × 2) python/pyspark/pandas/generic.py:1307— python/pyspark/pandas/generic.py:1307-1394 | python/pyspark/pandas/generic.py:1416-1501 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/generic.py:1307` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (66–84 lines × 2) python/pyspark/pandas/indexes/base.py:1425— python/pyspark/pandas/indexes/base.py:1425-1490 | python/pyspark/pandas/indexes/multi.py:745-828 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (76–81 lines × 2) python/pyspark/pandas/frame.py:11443— python/pyspark/pandas/frame.py:11443-11523 | python/pyspark/pandas/frame.py:11574-11649 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (75–76 lines × 2) python/pyspark/pandas/frame.py:5497— python/pyspark/pandas/frame.py:5497-5572 | python/pyspark/pandas/frame.py:5667-5741 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (54–74 lines × 2) python/pyspark/sql/avro/functions.py:37— python/pyspark/sql/avro/functions.py:37-110 | python/pyspark/sql/avro/functions.py:131-184 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note first that the copies are not typed on the same thing: the declarations holding them bind `jsonFormatSchema` to `str` in one and `str = ""` in another, and the duplicated lines use it. The extracted unit therefore needs a parameter type that fits BOTH — their common supertype where they have one, or a new abstraction over them where they do not — and settling that is the step that comes BEFORE the extraction above. Where the two types are deliberately unrelated, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (68 lines × 2) python/pyspark/ml/linalg/__init__.py:720— python/pyspark/ml/linalg/__init__.py:720-787 | python/pyspark/mllib/linalg/__init__.py:827-894 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (61–63 lines × 2) python/pyspark/sql/functions/builtin.py:23473— python/pyspark/sql/functions/builtin.py:23473-23533 | python/pyspark/sql/functions/builtin.py:23961-24023 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (61 lines × 6) python/pyspark/sql/readwriter.py:358— python/pyspark/sql/readwriter.py:358-477 | python/pyspark/sql/readwriter.py:814-878 | python/pyspark/sql/readwriter.py:972-1048 | python/pyspark/sql/streaming/readwriter.py:391-459 | python/pyspark/sql/streaming/readwriter.py:710-772 | python/pyspark/sql/streaming/readwriter.py:826-886 — there are 6 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 6 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/readwriter.py:358` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (50–60 lines × 2) python/pyspark/sql/readwriter.py:1204— python/pyspark/sql/readwriter.py:1204-1263 | python/pyspark/sql/readwriter.py:2398-2447 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/readwriter.py:2398` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (49–58 lines × 2) python/pyspark/sql/readwriter.py:121— python/pyspark/sql/readwriter.py:121-169 | python/pyspark/sql/streaming/readwriter.py:126-183 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on. Note first that the copies are not typed on the same thing: `SparkSession` names `pyspark.sql` in one and `pyspark.sql.session` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (48–56 lines × 2) python/pyspark/ml/linalg/__init__.py:563— python/pyspark/ml/linalg/__init__.py:563-618 | python/pyspark/mllib/linalg/__init__.py:630-677 — before extracting anything, compare `python/pyspark/ml/linalg/__init__.py` and `python/pyspark/mllib/linalg/__init__.py` as WHOLE FILES: this scan already matched 34 separate duplicated blocks between them, totalling at least 641 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (51–54 lines × 2) python/pyspark/sql/worker/create_data_source.py:40— python/pyspark/sql/worker/create_data_source.py:40-90 | python/pyspark/sql/worker/write_into_data_source.py:53-106 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/sql/worker/create_data_source.py:94` calls `read_bool` and `python/pyspark/sql/worker/write_into_data_source.py:109` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (45–53 lines × 2) python/pyspark/pandas/groupby.py:844— python/pyspark/pandas/groupby.py:844-888 | python/pyspark/pandas/groupby.py:1059-1111 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (47 lines × 3) python/pyspark/pandas/generic.py:1229— python/pyspark/pandas/generic.py:1229-1282 | python/pyspark/pandas/generic.py:1522-1568 | python/pyspark/pandas/generic.py:1590-1637 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (46–47 lines × 2) python/pyspark/pandas/series.py:6523— python/pyspark/pandas/series.py:6523-6569 | python/pyspark/pandas/series.py:6599-6644 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (44–46 lines × 2) python/pyspark/core/rdd.py:1234— python/pyspark/core/rdd.py:1234-1277 | python/pyspark/core/rdd.py:1312-1357 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (44–45 lines × 4) python/pyspark/sql/connect/functions/builtin.py:4370— python/pyspark/sql/connect/functions/builtin.py:4370-4413 | python/pyspark/sql/connect/functions/builtin.py:4463-4507 | python/pyspark/sql/functions/builtin.py:28570-28613 | python/pyspark/sql/functions/builtin.py:28826-28869 — there are 4 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 4 sites; resolving a subset leaves the remainder to drift apart. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (40–43 lines × 2) python/pyspark/pandas/groupby.py:4675— python/pyspark/pandas/groupby.py:4675-4714 | python/pyspark/pandas/groupby.py:4757-4799 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/groupby.py:4675` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (39–41 lines × 2) python/pyspark/sql/classic/dataframe.py:442— python/pyspark/sql/classic/dataframe.py:442-482 | python/pyspark/sql/connect/dataframe.py:1134-1172 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own.
Duplicated block (23–36 lines × 2) python/pyspark/pandas/frame.py:1662— python/pyspark/pandas/frame.py:1662-1697 | python/pyspark/pandas/groupby.py:4233-4255 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:1662` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (33–34 lines × 2) python/pyspark/sql/connect/udf.py:158— python/pyspark/sql/connect/udf.py:158-190 | python/pyspark/sql/udf.py:177-210 — before extracting anything, compare `python/pyspark/sql/connect/udf.py` and `python/pyspark/sql/udf.py` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 142 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (30 lines × 4) python/pyspark/sql/connect/readwriter.py:666— python/pyspark/sql/connect/readwriter.py:666-695 | python/pyspark/sql/connect/readwriter.py:712-741 | python/pyspark/sql/readwriter.py:1627-1656 | python/pyspark/sql/readwriter.py:1711-1740 — before extracting anything, compare `python/pyspark/sql/connect/readwriter.py` and `python/pyspark/sql/readwriter.py` as WHOLE FILES: this scan already matched 7 separate duplicated blocks between them, totalling at least 113 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place.
Duplicated block (26–28 lines × 3) python/pyspark/ml/connect/tuning.py:487— python/pyspark/ml/connect/tuning.py:487-514 | python/pyspark/ml/tuning.py:926-952 | python/pyspark/ml/tuning.py:1506-1531 — before extracting anything, compare `python/pyspark/ml/connect/tuning.py` and `python/pyspark/ml/tuning.py` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 44 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (27–28 lines × 2) python/pyspark/sql/classic/dataframe.py:1693— python/pyspark/sql/classic/dataframe.py:1693-1720 | python/pyspark/sql/connect/dataframe.py:1750-1776 — `python/pyspark/sql/classic/dataframe.py` and `python/pyspark/sql/connect/dataframe.py` are one unit implemented once per sibling directory, so they are most likely parallel implementations of one contract rather than a copy of each other — this scan matched 11 separate duplicated blocks between them, totalling at least 328 lines. If both are selected at run time, neither can be retired in favour of the other, and the lines that DIFFER between them are the reason both exist. The move that pays here is to hoist the identical part into a shared location the whole family can reach and give what differs a parameter or a seam, so a change lands once instead of once per sibling; extracting one helper per block leaves every sibling to drift on its own. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/sql/classic/dataframe.py:1721` calls `_to_list`, `cast` and `python/pyspark/sql/connect/dataframe.py:1778` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (13–27 lines × 3) python/pyspark/pandas/frame.py:1674— python/pyspark/pandas/frame.py:1674-1700 | python/pyspark/pandas/groupby.py:4246-4258 | python/pyspark/pandas/groupby.py:4455-4468 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:1674` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/pandas/groupby.py:4259` calls `col` and `python/pyspark/pandas/frame.py:1701` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (25–27 lines × 2) python/pyspark/sql/connect/functions/builtin.py:166— python/pyspark/sql/connect/functions/builtin.py:166-190 | python/pyspark/sql/functions/builtin.py:26594-26620 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere both call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made twice. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (24 lines × 3) python/pyspark/sql/worker/data_source_pushdown_filters.py:119— python/pyspark/sql/worker/data_source_pushdown_filters.py:119-167 | python/pyspark/sql/worker/plan_data_source_read.py:279-319 | python/pyspark/sql/worker/python_streaming_sink_runner.py:43-66 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/worker/data_source_pushdown_filters.py:119` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (23 lines × 3) python/pyspark/sql/connect/streaming/readwriter.py:201— python/pyspark/sql/connect/streaming/readwriter.py:201-223 | python/pyspark/sql/connect/streaming/readwriter.py:372-394 | python/pyspark/sql/connect/streaming/readwriter.py:430-452 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/connect/streaming/readwriter.py:201` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (22 lines × 4) python/pyspark/sql/pandas/typehints.py:63— python/pyspark/sql/pandas/typehints.py:63-84 | python/pyspark/sql/pandas/typehints.py:222-243 | python/pyspark/sql/pandas/typehints.py:415-436 | python/pyspark/sql/pandas/typehints.py:508-529 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/sql/pandas/typehints.py:246` calls `all` and `python/pyspark/sql/pandas/typehints.py:439` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (21–22 lines × 4) python/pyspark/sql/streaming/python_streaming_source_runner.py:227— python/pyspark/sql/streaming/python_streaming_source_runner.py:227-248 | python/pyspark/sql/worker/data_source_pushdown_filters.py:153-173 | python/pyspark/sql/worker/plan_data_source_read.py:305-325 | python/pyspark/sql/worker/python_streaming_sink_runner.py:52-72 — the copies span different directories, so extracting a shared function means choosing where it lives: put it somewhere all 4 call sites can already reach — a location they all depend on today, or a new shared one if there is none — and call it from each site; until then, every change has to be made 4 times.
Duplicated block (16–18 lines × 4) python/pyspark/pandas/accessors.py:412— python/pyspark/pandas/accessors.py:412-427 | python/pyspark/pandas/accessors.py:743-760 | python/pyspark/pandas/frame.py:3319-3334 | python/pyspark/pandas/groupby.py:2162-2177 — there are 4 copies across 3 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 4 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/accessors.py:412` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (16 lines × 12) python/pyspark/sql/connect/readwriter.py:221— python/pyspark/sql/connect/readwriter.py:221-236 | python/pyspark/sql/connect/readwriter.py:370-385 | python/pyspark/sql/connect/readwriter.py:436-451 | python/pyspark/sql/connect/streaming/readwriter.py:198-213 | python/pyspark/sql/connect/streaming/readwriter.py:369-384 | python/pyspark/sql/connect/streaming/readwriter.py:427-442 | python/pyspark/sql/readwriter.py:476-491 | python/pyspark/sql/readwriter.py:885-900 | python/pyspark/sql/readwriter.py:1041-1056 | python/pyspark/sql/streaming/readwriter.py:455-470 | python/pyspark/sql/streaming/readwriter.py:776-791 | python/pyspark/sql/streaming/readwriter.py:879-894 — before extracting anything, compare `python/pyspark/sql/connect/readwriter.py` and `python/pyspark/sql/connect/streaming/readwriter.py` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 71 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/connect/readwriter.py:221` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/sql/connect/readwriter.py:238` calls `isinstance`, `load` and `python/pyspark/sql/connect/readwriter.py:388` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (15 lines × 17) python/pyspark/pandas/testing.py:151— python/pyspark/pandas/testing.py:151-165 | python/pyspark/pandas/testing.py:247-261 | python/pyspark/sql/connect/readwriter.py:221-235 | python/pyspark/sql/connect/readwriter.py:370-384 | python/pyspark/sql/connect/readwriter.py:436-450 | python/pyspark/sql/connect/readwriter.py:890-904 | python/pyspark/sql/connect/streaming/readwriter.py:198-212 | python/pyspark/sql/connect/streaming/readwriter.py:369-383 | python/pyspark/sql/connect/streaming/readwriter.py:427-441 | python/pyspark/sql/readwriter.py:476-490 | python/pyspark/sql/readwriter.py:885-899 | python/pyspark/sql/readwriter.py:1041-1055 | python/pyspark/sql/readwriter.py:2232-2246 | python/pyspark/sql/streaming/listener.py:455-469 | python/pyspark/sql/streaming/readwriter.py:455-469 | python/pyspark/sql/streaming/readwriter.py:776-790 | python/pyspark/sql/streaming/readwriter.py:879-893 — before extracting anything, compare `python/pyspark/sql/connect/readwriter.py` and `python/pyspark/sql/connect/streaming/readwriter.py` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 71 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/testing.py:151` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 14) python/pyspark/sql/connect/readwriter.py:210— python/pyspark/sql/connect/readwriter.py:210-224 | python/pyspark/sql/connect/readwriter.py:351-365 | python/pyspark/sql/connect/readwriter.py:430-444 | python/pyspark/sql/connect/readwriter.py:887-901 | python/pyspark/sql/connect/streaming/readwriter.py:189-203 | python/pyspark/sql/connect/streaming/readwriter.py:352-366 | python/pyspark/sql/connect/streaming/readwriter.py:421-435 | python/pyspark/sql/readwriter.py:464-478 | python/pyspark/sql/readwriter.py:865-879 | python/pyspark/sql/readwriter.py:1035-1049 | python/pyspark/sql/readwriter.py:2229-2243 | python/pyspark/sql/streaming/readwriter.py:446-460 | python/pyspark/sql/streaming/readwriter.py:759-773 | python/pyspark/sql/streaming/readwriter.py:873-887 — before extracting anything, compare `python/pyspark/sql/connect/readwriter.py` and `python/pyspark/sql/connect/streaming/readwriter.py` as WHOLE FILES: this scan already matched 5 separate duplicated blocks between them, totalling at least 71 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. The two sit in different directories, so one cannot simply be deleted in favour of the other while both are reached separately: hoist the shared part into a location both already depend on and have each file call it, and retire whichever file turns out to have no caller of its own left. Extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/connect/readwriter.py:210` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `python/pyspark/sql/readwriter.py:2228` calls `mode` and `python/pyspark/sql/readwriter.py:863` does not — after which the two agree again for 4 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (15 lines × 8) python/pyspark/sql/connect/functions/builtin.py:4388— python/pyspark/sql/connect/functions/builtin.py:4388-4402 | python/pyspark/sql/connect/functions/builtin.py:4482-4496 | python/pyspark/sql/connect/functions/builtin.py:4592-4606 | python/pyspark/sql/connect/functions/builtin.py:4650-4664 | python/pyspark/sql/functions/builtin.py:28588-28602 | python/pyspark/sql/functions/builtin.py:28844-28858 | python/pyspark/sql/functions/builtin.py:29259-29273 | python/pyspark/sql/functions/builtin.py:29422-29436 — there are 8 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 8 sites; resolving a subset leaves the remainder to drift apart. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
Duplicated block (13–15 lines × 4) python/pyspark/sql/streaming/list_state_client.py:48— python/pyspark/sql/streaming/list_state_client.py:48-60 | python/pyspark/sql/streaming/map_state_client.py:58-70 | python/pyspark/sql/streaming/map_state_client.py:106-120 | python/pyspark/sql/streaming/value_state_client.py:44-56 — there are 4 copies across 3 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 4 sites; resolving a subset leaves the remainder to drift apart. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (14–15 lines × 3) python/pyspark/eval_handlers/_pandas.py:346— python/pyspark/eval_handlers/_pandas.py:346-360 | python/pyspark/eval_handlers/_pandas.py:422-435 | python/pyspark/eval_handlers/_pandas.py:509-523 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/eval_handlers/_pandas.py:346` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (8–14 lines × 2) python/pyspark/pandas/frame.py:12677— python/pyspark/pandas/frame.py:12677-12690 | python/pyspark/pandas/frame.py:12818-12825 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (13 lines × 6) python/pyspark/pandas/generic.py:1274— python/pyspark/pandas/generic.py:1274-1286 | python/pyspark/pandas/generic.py:1560-1572 | python/pyspark/pandas/generic.py:1629-1641 | python/pyspark/pandas/generic.py:1946-1958 | python/pyspark/pandas/generic.py:2045-2057 | python/pyspark/pandas/generic.py:2281-2293 — all 6 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/generic.py:1274` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (11–12 lines × 8) python/pyspark/pandas/generic.py:1279— python/pyspark/pandas/generic.py:1279-1289 | python/pyspark/pandas/generic.py:1386-1397 | python/pyspark/pandas/generic.py:1494-1504 | python/pyspark/pandas/generic.py:1565-1575 | python/pyspark/pandas/generic.py:1634-1644 | python/pyspark/pandas/generic.py:1951-1961 | python/pyspark/pandas/generic.py:2050-2060 | python/pyspark/pandas/generic.py:2286-2296 — all 8 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (9–11 lines × 4) python/pyspark/sql/streaming/list_state_client.py:117— python/pyspark/sql/streaming/list_state_client.py:117-126 | python/pyspark/sql/streaming/list_state_client.py:139-149 | python/pyspark/sql/streaming/list_state_client.py:162-171 | python/pyspark/sql/streaming/list_state_client.py:178-186 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/streaming/list_state_client.py:117` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (9–10 lines × 3) python/pyspark/sql/streaming/map_state_client.py:139— python/pyspark/sql/streaming/map_state_client.py:139-148 | python/pyspark/sql/streaming/map_state_client.py:263-271 | python/pyspark/sql/streaming/map_state_client.py:278-286 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/sql/streaming/map_state_client.py:139` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (3–6 lines × 3) python/pyspark/sql/connect/functions/builtin.py:4204— python/pyspark/sql/connect/functions/builtin.py:4204-4206 | python/pyspark/sql/connect/functions/builtin.py:4228-4233 | python/pyspark/sql/connect/functions/builtin.py:4253-4258 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (7 lines × 43) python/pyspark/ml/classification.py:742— python/pyspark/ml/classification.py:742-753 | python/pyspark/ml/classification.py:1805-1817 | python/pyspark/ml/classification.py:2111-2124 | python/pyspark/ml/classification.py:2589-2603 | python/pyspark/ml/classification.py:2980-2990 | python/pyspark/ml/classification.py:3234-3245 | python/pyspark/ml/classification.py:4133-4145 | python/pyspark/ml/clustering.py:422-432 | python/pyspark/ml/clustering.py:800-809 | python/pyspark/ml/clustering.py:1154-1164 | python/pyspark/ml/clustering.py:1718-1728 | python/pyspark/ml/clustering.py:2065-2074 | python/pyspark/ml/feature.py:598-607 | python/pyspark/ml/feature.py:1096-1103 | python/pyspark/ml/feature.py:1439-1447 | python/pyspark/ml/feature.py:1851-1857 | python/pyspark/ml/feature.py:2140-2147 | python/pyspark/ml/feature.py:2588-2594 | python/pyspark/ml/feature.py:2753-2759 | python/pyspark/ml/feature.py:3204-3211 | python/pyspark/ml/feature.py:3935-3942 | python/pyspark/ml/feature.py:4279-4285 | python/pyspark/ml/feature.py:4414-4420 | python/pyspark/ml/feature.py:4674-4681 | python/pyspark/ml/feature.py:4952-4958 | python/pyspark/ml/feature.py:5376-5383 | python/pyspark/ml/feature.py:5589-5595 | python/pyspark/ml/feature.py:5889-5895 | python/pyspark/ml/feature.py:6323-6331 | python/pyspark/ml/feature.py:6558-6564 | python/pyspark/ml/feature.py:6805-6813 | python/pyspark/ml/feature.py:7184-7192 | python/pyspark/ml/feature.py:7677-7686 | python/pyspark/ml/fpm.py:256-263 | python/pyspark/ml/recommendation.py:385-395 | python/pyspark/ml/regression.py:320-331 | python/pyspark/ml/regression.py:870-879 | python/pyspark/ml/regression.py:1130-1142 | python/pyspark/ml/regression.py:1433-1446 | python/pyspark/ml/regression.py:1786-1798 | python/pyspark/ml/regression.py:2186-2197 | python/pyspark/ml/regression.py:2586-2598 | python/pyspark/ml/regression.py:3154-3163 — before extracting anything, compare `python/pyspark/ml/classification.py` and `python/pyspark/ml/regression.py` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `python/pyspark/ml/clustering.py:811` calls `_create_model`, `KMeansModel` and `python/pyspark/ml/classification.py:755` does not — after which the two agree again for 3 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (17 lines × 5) python/pyspark/ml/classification.py:903— python/pyspark/ml/classification.py:903-919 | python/pyspark/ml/classification.py:3371-3387 | python/pyspark/ml/classification.py:4300-4316 | python/pyspark/ml/regression.py:487-504 | python/pyspark/ml/regression.py:2771-2788 — before extracting anything, compare `python/pyspark/ml/classification.py` and `python/pyspark/ml/regression.py` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 42 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. The `return` at the foot of the matched lines is the enclosing body's own terminal exit, not an early one: it moves with them unchanged, and each site calls the extracted unit from the position that `return` occupied — no decision has to be handed back and re-acted on.
Duplicated block (40 lines × 5) python/pyspark/pandas/frame.py:2157— python/pyspark/pandas/frame.py:2157-2229 | python/pyspark/pandas/frame.py:2471-2564 | python/pyspark/pandas/frame.py:2589-2684 | python/pyspark/pandas/frame.py:5991-6067 | python/pyspark/pandas/series.py:1679-1718 — there are 5 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 5 sites; resolving a subset leaves the remainder to drift apart. Read the line range as the matched WINDOW rather than a finished unit: at `python/pyspark/pandas/frame.py:2471` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it. Note first that the copies are not typed on the same thing: `Index` names `pyspark.pandas.indexes` in one and `pyspark.pandas.indexes.base` in another — different types that share a simple name, which is why the text matched. A single extracted unit cannot be given a parameter type that fits both, so unifying those types (or introducing a shared abstraction over them) is the step that has to come BEFORE the extraction above; if they are deliberately separate, the duplication is the price of that separation and the honest resolution is to record the decision rather than to extract.
Duplicated block (57 lines × 3) python/pyspark/pandas/frame.py:2258— python/pyspark/pandas/frame.py:2258-2351 | python/pyspark/pandas/frame.py:2374-2468 | python/pyspark/pandas/series.py:1609-1665 — there are 3 copies across 2 file(s) — more copies than files, so at least one file holds the block twice. Extract it once into a single shared function every call site can reach and call it from all 3 sites; resolving a subset leaves the remainder to drift apart.
Duplicated block (57 lines × 2) python/pyspark/pandas/generic.py:2863— python/pyspark/pandas/generic.py:2863-2923 | python/pyspark/pandas/groupby.py:3421-3477 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. The matched lines also transfer control out of the body holding them, which cannot survive a move into a called unit unchanged: have the extracted unit return that decision and let each site act on it.
D4 · Code Duplication· Near-duplicate member family (4 members, 23 shared lines) · ×1
Near-duplicate member family (4 members, 23 shared lines) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:68— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:68-132 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:66-135 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:67-141 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostedTreesRunner.scala:53-95 — These 4 members are variants of one another: a block of 23 lines reported below appears in every one of them, and the pairwise near-duplicate rows they would otherwise produce are collapsed into this row. Read them as one construct written 4 times. The repair is at the members' grain — factor the shared implementation out once and have all of them call it with their differences as parameters or as an injected step, or, where the difference is systematic, generate them from one template. Extracting the individual blocks below is not the same fix: it leaves every body in place and the next edit still has to be made 4 times.
Near-duplicate member pair (71 shared lines) examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:137— examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:137-240 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:143-248 — These two members are variants of one another: 71 of their lines are already reported as duplicated blocks below, spread through both bodies rather than gathered into one. Read them as a single construct written twice. The repair is at the members' grain — factor the shared pipeline into one implementation the two call with their differences as parameters or as an injected step, or, where the difference is systematic (sync against async, one transport against another), generate one from the other. Extracting the individual blocks below is not the same fix: it leaves the two bodies in place and the next edit still has to be made twice.
D4 · Code Duplication· Edited copy of a member (16 corresponding lines) · ×1
Edited copy of a member (16 corresponding lines) examples/src/main/java/org/apache/spark/examples/ml/JavaFPGrowthExample.java:41— examples/src/main/java/org/apache/spark/examples/ml/JavaFPGrowthExample.java:41-76 | examples/src/main/java/org/apache/spark/examples/ml/JavaWord2VecExample.java:35-71 — These two members are one piece of code written twice and then edited apart: 16 consecutive lines correspond almost exactly, broken only by small local edits. Most of that correspondence is NOT reported as duplicated blocks below — the edits cut it into fragments and only the largest of them clear the block floor, so the rows below understate it. The repair is at the members' grain — factor the shared implementation into one the two call with their differences as parameters or as an injected step, or, where the difference is systematic (an extra return value, one transport against another), generate one from the other. Left alone, the next edit has to be made twice and the two will drift further apart.
D4 · Code Duplication· Edited copy of a member (17 corresponding lines) · ×1
Edited copy of a member (17 corresponding lines) examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:37— examples/src/main/java/org/apache/spark/examples/ml/JavaBinarizerExample.java:37-67 | examples/src/main/java/org/apache/spark/examples/ml/JavaQuantileDiscretizerExample.java:36-72 — These two members are one piece of code written twice and then edited apart: 17 consecutive lines correspond almost exactly, broken only by small local edits. Most of that correspondence is NOT reported as duplicated blocks below — the edits cut it into fragments and only the largest of them clear the block floor, so the rows below understate it. The repair is at the members' grain — factor the shared implementation into one the two call with their differences as parameters or as an injected step, or, where the difference is systematic (an extra return value, one transport against another), generate one from the other. Left alone, the next edit has to be made twice and the two will drift further apart.
Duplicated block (23–25 lines × 3) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeClassificationExample.scala:32— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeClassificationExample.scala:32-54 | examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeClassifierExample.scala:31-55 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestClassifierExample.scala:31-55 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/GradientBoostedTreeClassifierExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/RandomForestClassifierExample.scala` as WHOLE FILES: 82% of the shorter file's lines also appear in the other, so this reads as one file having been copied from the other rather than as a helper waiting to be extracted. The 2 duplicated block(s) this scan matched between them are fragments of that copy, not the extent of it — treat the file pair as the unit. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (22 lines × 3) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:213— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:213-234 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:150-171 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:156-177 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:213` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (19–20 lines × 3) examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala:42— examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala:42-61 | examples/src/main/scala/org/apache/spark/examples/mllib/MultivariateSummarizer.scala:43-62 | examples/src/main/scala/org/apache/spark/examples/mllib/SampledRDDs.scala:39-57 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala:42` it does not close everything it opens, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (17–19 lines × 4) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:97— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:97-113 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:98-116 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:104-122 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala:112-128 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:97` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (18 lines × 5) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:115— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:115-132 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:118-135 | examples/src/main/scala/org/apache/spark/examples/ml/LinearRegressionExample.scala:84-101 | examples/src/main/scala/org/apache/spark/examples/ml/LogisticRegressionExample.scala:92-109 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:124-141 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:115` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (15 lines × 5) examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:87— examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:87-103 | examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala:48-63 | examples/src/main/scala/org/apache/spark/examples/mllib/MovieLensALS.scala:81-97 | examples/src/main/scala/org/apache/spark/examples/mllib/MultivariateSummarizer.scala:49-64 | examples/src/main/scala/org/apache/spark/examples/mllib/SampledRDDs.scala:45-59 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 5 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:87` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just before the matched lines, `examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala:85` calls `required`, `text` and `examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala:47` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (9–15 lines × 3) examples/src/main/scala/org/apache/spark/examples/ml/IndexToStringExample.scala:29— examples/src/main/scala/org/apache/spark/examples/ml/IndexToStringExample.scala:29-42 | examples/src/main/scala/org/apache/spark/examples/ml/OneHotEncoderExample.scala:28-42 | examples/src/main/scala/org/apache/spark/examples/ml/StringIndexerExample.scala:28-36 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (7–11 lines × 5) examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:99— examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:99-107 | examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala:100-110 | examples/src/main/scala/org/apache/spark/examples/ml/RandomForestExample.scala:106-116 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRunner.scala:114-122 | examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala:90-96 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala` and `examples/src/main/scala/org/apache/spark/examples/ml/GBTExample.scala` as WHOLE FILES: this scan already matched 10 separate duplicated blocks between them, totalling at least 139 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/scala/org/apache/spark/examples/ml/DecisionTreeExample.scala:99` it runs out through the closing brace of the declaration holding it and carries on into the declaration that follows — the window is the tail of one member plus the head of the next, so no call can be substituted for those exact lines, and the smallest declaration that contains all of them is the type they sit in. The repeated unit is the member each site sits in: where those members' bodies are the same, move one whole member to the shared location and have the others delegate to it; where the copies are a run of near-identical overloads or wrappers that differ only in their signatures, the repetition IS the run — a one-line delegation has no helper inside it to lift — so generate the run from the set it enumerates, or accept it and keep each member's own documentation with it.
Duplicated block (8–9 lines × 6) examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala:31— examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala:31-39 | examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeRegressionExample.scala:31-39 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingClassificationExample.scala:31-38 | examples/src/main/scala/org/apache/spark/examples/mllib/GradientBoostingRegressionExample.scala:31-38 | examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala:30-37 | examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestRegressionExample.scala:30-37 — before extracting anything, compare `examples/src/main/scala/org/apache/spark/examples/mllib/DecisionTreeClassificationExample.scala` and `examples/src/main/scala/org/apache/spark/examples/mllib/RandomForestClassificationExample.scala` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 32 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (24–27 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:35— examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:35-61 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java:36-62 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestClassifierExample.java:36-59 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 79 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:35` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (13–20 lines × 2) examples/src/main/java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java:267— examples/src/main/java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java:267-279 | examples/src/main/java/org/apache/spark/examples/sql/JavaSparkSQLExample.java:324-343 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/sql/JavaSQLDataSourceExample.java:267` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (18–19 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaFMRegressorExample.java:63— examples/src/main/java/org/apache/spark/examples/ml/JavaFMRegressorExample.java:63-80 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java:65-82 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java:64-82 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 48 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (15–18 lines × 4) examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:34— examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:34-51 | examples/src/main/java/org/apache/spark/examples/ml/JavaFMClassifierExample.java:35-52 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java:35-52 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestClassifierExample.java:35-49 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 79 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (13–18 lines × 2) examples/src/main/java/org/apache/spark/examples/JavaHdfsLR.java:117— examples/src/main/java/org/apache/spark/examples/JavaHdfsLR.java:117-129 | examples/src/main/java/org/apache/spark/examples/JavaPageRank.java:69-86 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from both call sites, so a change lands once.
Duplicated block (16–17 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionSummaryExample.java:32— examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionSummaryExample.java:32-47 | examples/src/main/java/org/apache/spark/examples/ml/JavaLogisticRegressionWithElasticNetExample.java:30-46 | examples/src/main/java/org/apache/spark/examples/ml/JavaMulticlassLogisticRegressionWithElasticNetExample.java:31-47 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once.
Duplicated block (15–16 lines × 3) examples/src/main/java/org/apache/spark/examples/mllib/JavaBinaryClassificationMetricsExample.java:35— examples/src/main/java/org/apache/spark/examples/mllib/JavaBinaryClassificationMetricsExample.java:35-50 | examples/src/main/java/org/apache/spark/examples/mllib/JavaLogisticRegressionWithLBFGSExample.java:40-54 | examples/src/main/java/org/apache/spark/examples/mllib/JavaMulticlassClassificationMetricsExample.java:36-50 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. ★ These copies have DRIFTED, and that is worth reading before extracting anything: just after the matched lines, `examples/src/main/java/org/apache/spark/examples/mllib/JavaBinaryClassificationMetricsExample.java:53` calls `clearThreshold` and `examples/src/main/java/org/apache/spark/examples/mllib/JavaLogisticRegressionWithLBFGSExample.java:57` does not — after which the two agree again for 2 more lines. One of those two behaviours is the intended one and the other is what a copy-paste left behind, so decide which BEFORE unifying them: extracting the shared part will silently settle it, and if the copy that skips the call is the wrong one, that bug is already live.
Duplicated block (12–14 lines × 3) examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:44— examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:44-55 | examples/src/main/java/org/apache/spark/examples/ml/JavaOneHotEncoderExample.java:42-55 | examples/src/main/java/org/apache/spark/examples/ml/JavaStringIndexerExample.java:41-52 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 3 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaIndexToStringExample.java:44` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Duplicated block (10 lines × 6) examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:54— examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java:54-63 | examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeRegressionExample.java:46-55 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java:55-64 | examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeRegressorExample.java:47-56 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestClassifierExample.java:52-61 | examples/src/main/java/org/apache/spark/examples/ml/JavaRandomForestRegressorExample.java:47-56 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/ml/JavaDecisionTreeClassificationExample.java` and `examples/src/main/java/org/apache/spark/examples/ml/JavaGradientBoostedTreeClassifierExample.java` as WHOLE FILES: this scan already matched 4 separate duplicated blocks between them, totalling at least 79 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (6–9 lines × 7) examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java:41— examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeClassificationExample.java:41-48 | examples/src/main/java/org/apache/spark/examples/mllib/JavaDecisionTreeRegressionExample.java:41-48 | examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java:40-48 | examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java:40-47 | examples/src/main/java/org/apache/spark/examples/mllib/JavaNaiveBayesExample.java:34-39 | examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestClassificationExample.java:39-45 | examples/src/main/java/org/apache/spark/examples/mllib/JavaRandomForestRegressionExample.java:39-45 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingClassificationExample.java` and `examples/src/main/java/org/apache/spark/examples/mllib/JavaGradientBoostingRegressionExample.java` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 31 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place.
Duplicated block (2–7 lines × 5) REDACTED:74— REDACTED:74-80 | examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java:90-97 | examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java:132-139 | examples/src/main/java/org/apache/spark/examples/streaming/JavaNetworkWordCount.java:66-72 | examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java:131-132 — before extracting anything, compare `examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKafkaWordCount.java` and `examples/src/main/java/org/apache/spark/examples/streaming/JavaDirectKerberizedKafkaWordCount.java` as WHOLE FILES: this scan already matched 3 separate duplicated blocks between them, totalling at least 34 lines, which is the signature of one file having been copied from the other rather than of a helper waiting to be extracted. If that is what happened, the fix is to keep one copy and have the other call it (or delete it), which resolves this row and its siblings together — extracting one helper per block leaves the fork in place. Read the line range as the matched WINDOW rather than a finished unit: at `REDACTED:74` it runs out through the closing brace of the declaration holding it — the window is that declaration's tail, not a fragment that begins part-way through something, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that. Note that the copies do not run to the end of the range shown: their LAST lines are different code, not the same code under different names — the matched region ends inside that line. Extract the lines above it, and read the last line of each site separately.
Duplicated block (5 lines × 6) examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java:67— examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java:67-72 | examples/src/main/java/org/apache/spark/examples/ml/JavaMaxAbsScalerExample.java:51-55 | examples/src/main/java/org/apache/spark/examples/ml/JavaMinHashLSHExample.java:65-70 | examples/src/main/java/org/apache/spark/examples/ml/JavaMinMaxScalerExample.java:51-55 | examples/src/main/java/org/apache/spark/examples/ml/JavaNormalizerExample.java:50-54 | examples/src/main/java/org/apache/spark/examples/ml/JavaVarianceThresholdSelectorExample.java:57-61 — the copies sit in sibling files of one directory, so a shared home is within easy reach: extract the block into a single shared function the call sites can all reach — a file they already depend on, or a new one alongside them — and call it from all 6 call sites, so a change lands once. Read the line range as the matched WINDOW rather than a finished unit: at `examples/src/main/java/org/apache/spark/examples/ml/JavaBucketedRandomProjectionLSHExample.java:67` it begins part-way through the construct above it, so those exact lines cannot be lifted as they stand — widen the region to the smallest complete statement or declaration that contains it, and extract that.
Coverage not measured — JavaScript/TypeScript suite — Coverage NOT MEASURED: the JavaScript/TypeScript half could not be measured — the jest suite in ui-test/ ran and not one test passed (FAIL tests/utils.test.js), so the coverage would describe the failed run, not the code. Coverage is excluded from the score rather than counted as a near-zero. The named suite step is one the repository's maintainers can perform; once it passes, the real number is measured on the next scan. Alternatively, commit the lcov/Cobertura report your CI produces and it is read without a re-run.
IC1 · Incompleteness & stubs· Empty method body · ×1
Empty method body — does nothing with its inputs sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/ExpressionResolutionValidator.scala:136— `validateLiteral` takes parameters but its body is empty — it accepts inputs and does nothing. Either implement it or remove it.
Type Safety — 0 typed · 34 plain JS — the untyped files are core/src/main/resources/org/apache/spark/ui/static/dataTables.rowsGroup.js, core/src/main/resources/org/apache/spark/ui/static/environmentpage.js, core/src/main/resources/org/apache/spark/ui/static/executorspage.js, core/src/main/resources/org/apache/spark/ui/static/flamegraph.js, core/src/main/resources/org/apache/spark/ui/static/historypage-common.js, REDACTED (+28 more).
R10 · Code Duplication· Duplicated block with local edits (37 matched lines × 2 locations) · ×1
Duplicated block with local edits (37 matched lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:411— core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:411 · REDACTED:192 — the two spans are one implementation copied and then locally edited — 296 tokens are still identical, in the same order in both files, with only local edits between them. The copies have already begun to drift, which is this row's finding: an edit made to one and not the other changes behaviour silently. Diff the two spans first to learn what genuinely differs, then extract the shared core into one module both sites use, passing the differences in as parameters — or, if one copy exists only because the other could not be imported from its context, make one of them the single source the other is generated or re-exported from. If one copy is no longer reachable, delete it rather than letting it shadow the live one.
Duplicated block (33 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/executorspage.js:696— core/src/main/resources/org/apache/spark/ui/static/executorspage.js:696 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:851 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (21 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:871— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:871 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:921 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (18 lines × 4 locations) core/src/main/resources/org/apache/spark/ui/static/executorspage.js:621— core/src/main/resources/org/apache/spark/ui/static/executorspage.js:621 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:639 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:657 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:675 — all 4 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (18 lines × 3 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:524— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:524 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:543 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:561 — all 3 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (17 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/executorspage.js:590— core/src/main/resources/org/apache/spark/ui/static/executorspage.js:590 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:819 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (15 lines × 2 locations) REDACTED:286— REDACTED:286 · REDACTED:301 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (14 lines × 3 locations) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:964— core/src/main/resources/org/apache/spark/ui/static/stagepage.js:964 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:978 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1002 — all 3 copies are in the same file, so extract the block into one function there and call it from every one of those sites — resolving only two of them leaves the rest to drift apart the first time one is edited.
Duplicated block (10 lines × 3 locations) core/src/main/resources/org/apache/spark/ui/static/executorspage.js:234— core/src/main/resources/org/apache/spark/ui/static/executorspage.js:234 · REDACTED:66 · REDACTED:81 — the 3 copies are spread across 2 files, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (9 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/utils.js:114— core/src/main/resources/org/apache/spark/ui/static/utils.js:114 · core/src/main/resources/org/apache/spark/ui/static/utils.js:224 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
R10 · Code Duplication· Duplicated block with local edits (9 matched lines × 2 locations) · ×1
Duplicated block with local edits (9 matched lines × 2 locations) REDACTED:885— REDACTED:885 · REDACTED:1100 — the two spans are one implementation copied and then locally edited — 101 tokens are still identical, in the same order in both spans, with only local edits between them. The copies have already begun to drift, which is this row's finding: an edit made to one and not the other changes behaviour silently. Diff the two spans first to learn what genuinely differs, then extract the shared core into one module both sites use, passing the differences in as parameters — or, if one copy exists only because the other could not be imported from its context, make one of them the single source the other is generated or re-exported from. If one copy is no longer reachable, delete it rather than letting it shadow the live one.
Duplicated block (7 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/executorspage.js:291— core/src/main/resources/org/apache/spark/ui/static/executorspage.js:291 · core/src/main/resources/org/apache/spark/ui/static/executorspage.js:301 — all 2 copies are in the same file, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (7 lines × 3 locations) core/src/main/resources/org/apache/spark/ui/static/executorspage.js:916— core/src/main/resources/org/apache/spark/ui/static/executorspage.js:916 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:384 · core/src/main/resources/org/apache/spark/ui/static/stagepage.js:392 — the 3 copies are spread across 2 files, and the CITED SPAN is not a self-contained block — it runs from inside one construct into the next (the tail of a branch plus the head of the following one, a run of switch arms, the end of a declaration plus the list that follows it) rather than covering a whole unit. So do not lift these lines literally: no call can be substituted for a half-open construct. Extract the enclosing repeated UNIT instead — the whole function, component or branch these lines sit in — and where the repetition IS the construct (a run of switch arms, a stack of near-identical declarations) replace it with one table or registry looked up by key rather than a helper each arm calls. The copies still drift apart the first time only one of them is edited, which is why this is reported.
Duplicated block (6 lines × 2 locations) core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:172— core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:172 · core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:184 — both copies are in the same file, so extract the block into one function there and call it from each site — the copies drift apart the first time only one of them is edited.
Duplicated block (6 lines × 3 locations) core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:176— core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:176 · core/src/main/resources/org/apache/spark/ui/static/streaming-page.js:281 · core/src/main/resources/org/apache/spark/ui/static/structured-streaming-page.js:35 — the 2 copies sit in sibling files in one directory, so check first whether one of them (or an existing module there) already owns this behaviour and the others should call it; otherwise extract it into one module in that directory and have each site call it.
Complex function (anonymous) (cyclomatic 19, cognitive 15) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:678— (anonymous) has cyclomatic complexity 19 and cognitive complexity 15; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive sits below cyclomatic here, so much of the count is breadth — arms side by side rather than stacked — and splitting per arm would leave a function per arm; group the work between the checks into named steps instead. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function updateDetailsPanel (cyclomatic 18, cognitive 19) REDACTED:423— updateDetailsPanel has cyclomatic complexity 18 and cognitive complexity 19; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function getInnerText (cyclomatic 17, cognitive 19) REDACTED:212— getInnerText has cyclomatic complexity 17 and cognitive complexity 19; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function innerSortFunction (cyclomatic 15, cognitive 23) REDACTED:92— innerSortFunction has cyclomatic complexity 15 and cognitive complexity 23; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function (anonymous) (cyclomatic 15, cognitive 18) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:404— (anonymous) has cyclomatic complexity 15 and cognitive complexity 18; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function formatMetricValue (cyclomatic 15, cognitive 16) REDACTED:344— formatMetricValue has cyclomatic complexity 15 and cognitive complexity 16; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function (anonymous) (cyclomatic 15, cognitive 12) REDACTED:1118— (anonymous) has cyclomatic complexity 15 and cognitive complexity 12; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive sits below cyclomatic here, so much of the count is breadth — arms side by side rather than stacked — and splitting per arm would leave a function per arm; group the work between the checks into named steps instead. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function makeSortable (cyclomatic 14, cognitive 22) REDACTED:42— makeSortable has cyclomatic complexity 14 and cognitive complexity 22; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function (anonymous) (cyclomatic 14, cognitive 14) core/src/main/resources/org/apache/spark/ui/static/environmentpage.js:164— (anonymous) has cyclomatic complexity 14 and cognitive complexity 14; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function reselectCheckboxesBasedOnTaskTableState (cyclomatic 12, cognitive 18) core/src/main/resources/org/apache/spark/ui/static/stagepage.js:287— reselectCheckboxesBasedOnTaskTableState has cyclomatic complexity 12 and cognitive complexity 18; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function zoomToNode (cyclomatic 12, cognitive 9) REDACTED:1047— zoomToNode has cyclomatic complexity 12 and cognitive complexity 9; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive sits below cyclomatic here, so much of the count is breadth — arms side by side rather than stacked — and splitting per arm would leave a function per arm; group the work between the checks into named steps instead. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
Complex function (anonymous) (cyclomatic 11, cognitive 12) REDACTED:951— (anonymous) has cyclomatic complexity 11 and cognitive complexity 12; this row is raised above a cyclomatic bar of 10. The two numbers answer different questions and the gap between them is what decides whether to act: cyclomatic counts the independent arms through the body, cognitive counts what it costs to hold them in your head, so nesting and mixed boolean chains raise it while a flat run of independent arms does not. Cognitive is at or above cyclomatic here, so the branching is nested or entangled rather than laid out side by side — extracting each decision into its own named function is the change that pays. Measured by this repository's own parse of the file, so a body assembled at runtime, or generated, is counted as written rather than as it executes.
R3 · Large Files· core/src/main/resources/org/apache/spark/ui/static/stagepage.js · ×1
core/src/main/resources/org/apache/spark/ui/static/stagepage.js: 1077 lines of code core/src/main/resources/org/apache/spark/ui/static/stagepage.js:1— core/src/main/resources/org/apache/spark/ui/static/stagepage.js has 1077 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module.
REDACTED: 1065 lines of code REDACTED:1— REDACTED has 1065 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module.
R3 · Large Files· core/src/main/resources/org/apache/spark/ui/static/executorspage.js · ×1
core/src/main/resources/org/apache/spark/ui/static/executorspage.js: 907 lines of code core/src/main/resources/org/apache/spark/ui/static/executorspage.js:1— core/src/main/resources/org/apache/spark/ui/static/executorspage.js has 907 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module.
R3 · Large Files· core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js · ×1
core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js: 529 lines of code core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js:1— core/src/main/resources/org/apache/spark/ui/static/spark-dag-viz.js has 529 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module.
REDACTED: 459 lines of code REDACTED:1— REDACTED has 459 significant lines (blank lines excluded), over the 400-line bar, and contains logic — a file this long is usually several responsibilities in one module.
Unused dependency 'jquery' — Declared in ui-test/package.json but never imported anywhere in that package or its workspace members — no static import reaches it. Usually that is dead weight and attack surface, but two shapes are indistinguishable from source and are NOT dead: an optional or native peer that another dependency loads dynamically at runtime, and a package a build, docs or test step installs and invokes separately. Confirm which of the three this is before removing it.
Orphaned file core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/AnalysisHelper.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file python/pyspark/streaming/context.py:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file python/pyspark/ml/fpm.py:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/core/src/main/scala/org/apache/spark/sql/columnar/CachedBatchSerializer.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/linearRegression.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file python/pyspark/sql/streaming/stateful_processor.py:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/gen-sql-api-docs.py:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReceiver.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SqlScriptingLogicalPlans.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file connector/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaReporter.java:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/tupleUnionAgg.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/api/src/main/scala/org/apache/spark/sql/streaming/GroupState.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file REDACTED:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file core/src/main/scala/org/apache/spark/SSLOptions.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/JoinEstimation.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file streaming/src/main/scala/org/apache/spark/streaming/Checkpoint.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Orphaned file sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala:1— No one still active in this repository holds living knowledge of this file: everyone who made a meaningful change to it has gone quiet, or what the current team did to it (moves, reformatting, log-level or import edits, renames) never needed understanding it. If it breaks, no one currently understands it. Attach the read to the next change that touches it: have a second person review that change, and leave behind a short comment or test recording what the file is for.
Duplicated predicate common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:145— `!idx.parent().isEmpty() && !idx.value().equals(KVIndex.NATURAL_INDEX_NAME)` appears character-identically in 2 files — common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java, common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java:69— `!index.isChild() || params.parent != null` appears character-identically in 2 files — common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBIterator.java, common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBIterator.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/unsafe/src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java:86— `(memory.pageNumber == MemoryBlock.NO_PAGE_NUMBER) || (memory.pageNumber == MemoryBlock.FREED_IN_TMM_PAGE_NUMBER)` appears character-identically in 2 files — common/unsafe/src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java, common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java:371— `(sizeInBytes == o.sizeInBytes) && ByteArrayMethods.arrayEquals(baseObject, baseOffset, o.baseObject, o.baseOffset, sizeInBytes)` appears character-identically in 2 files — sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java, sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java:516— `Integer.parseInt(blockIdParts[1]) != shuffleId || Integer.parseInt(blockIdParts[2]) != shuffleMergeId` appears character-identically in 2 files — common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java, common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/OpenBlocks.java:59— `Objects.equals(appId, o.appId) && Objects.equals(execId, o.execId) && Arrays.equals(blockIds, o.blockIds)` appears character-identically in 2 files — common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/OpenBlocks.java, common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/RemoveBlocks.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/FinalizeShuffleMerge.java:68— `Objects.equals(appId, o.appId) && appAttemptId == o.appAttemptId && shuffleId == o.shuffleId && shuffleMergeId == o.shuffleMergeId` appears character-identically in 2 files — common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/FinalizeShuffleMerge.java, common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/RemoveShuffleMerge.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/constraints/PrimaryKey.java:74— `Objects.equals(name(), that.name()) && Arrays.equals(columns, that.columns()) && enforced() == that.enforced() && Objects.equals(validationS…` appears character-identically in 2 files — sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/constraints/PrimaryKey.java, sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/constraints/Unique.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/unsafe/src/main/java/org/apache/spark/unsafe/types/BinaryView.java:215— `base instanceof byte[] bytes && offset >= BYTE_ARRAY_OFFSET` appears character-identically in 2 files — common/unsafe/src/main/java/org/apache/spark/unsafe/types/BinaryView.java, common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/network-common/src/main/java/org/apache/spark/network/protocol/EncryptedMessageWithHeader.java:63— `body instanceof InputStream || body instanceof ChunkedStream` appears character-identically in 2 files — common/network-common/src/main/java/org/apache/spark/network/protocol/EncryptedMessageWithHeader.java, common/network-common/src/main/java/org/apache/spark/network/protocol/SslMessageEncoder.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java:446— `currentPage == null || pageCursor + required > currentPage.getBaseOffset() + currentPage.size()` appears character-identically in 2 files — core/src/main/java/org/apache/spark/shuffle/sort/ShuffleExternalSorter.java, core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:194— `dataType instanceof IntegerType || dataType instanceof YearMonthIntervalType` appears character-identically in 2 files — sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java, sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java:196— `dataType instanceof LongType || dataType instanceof DayTimeIntervalType` appears character-identically in 2 files — sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarBatchRow.java, sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarRow.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java:167— `db != null && AppsWithRecoveryDisabled.isRecoveryEnabledForApp(appId)` appears character-identically in 3 files — common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java, common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java, common/network-yarn/src/main/java/org/apache/spark/network/yarn/YarnShuffleService.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java:81— `dt instanceof IntegerType || dt instanceof DateType || dt instanceof YearMonthIntervalType` appears character-identically in 2 files — sql/catalyst/src/main/java/org/apache/spark/sql/vectorized/ColumnarArray.java, sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ValueCopier.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate core/src/main/java/org/apache/spark/security/ServiceCredential.java:82— `expiresAt != null && !now.isBefore(expiresAt)` appears character-identically in 2 files — core/src/main/java/org/apache/spark/security/ServiceCredential.java, core/src/main/java/org/apache/spark/security/UserContext.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java:140— `idx.parent().isEmpty() || idx.value().equals(KVIndex.NATURAL_INDEX_NAME)` appears character-identically in 2 files — common/kvstore/src/main/java/org/apache/spark/util/kvstore/LevelDBTypeInfo.java, common/kvstore/src/main/java/org/apache/spark/util/kvstore/RocksDBTypeInfo.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeArrayWriter.java:173— `input != null && input.changePrecision(precision, scale)` appears character-identically in 2 files — sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeArrayWriter.java, sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/network-common/src/main/java/org/apache/spark/network/crypto/CtrTransportCipher.java:242— `msg instanceof ByteBuf || msg instanceof FileRegion` appears character-identically in 2 files — common/network-common/src/main/java/org/apache/spark/network/crypto/CtrTransportCipher.java, common/network-common/src/main/java/org/apache/spark/network/sasl/SaslEncryption.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java:51— `numRows >= capacity || page == null || page.size() - pageCursor < recordLength` appears character-identically in 2 files — sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/FixedLengthRowBasedKeyValueBatch.java, sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/VariableLengthRowBasedKeyValueBatch.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/api/src/main/java/org/apache/spark/sql/types/Geography.java:69— `obj == null || getClass() != obj.getClass()` appears character-identically in 2 files — sql/api/src/main/java/org/apache/spark/sql/types/Geography.java, sql/api/src/main/java/org/apache/spark/sql/types/Geometry.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/unsafe/src/main/java/org/apache/spark/unsafe/types/BinaryView.java:153— `offset == BYTE_ARRAY_OFFSET && base instanceof byte[] bytes && bytes.length == numBytes` appears character-identically in 2 files — common/unsafe/src/main/java/org/apache/spark/unsafe/types/BinaryView.java, common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ColumnExpressionBase.java:64— `other == null || getClass() != other.getClass()` appears character-identically in 6 files — sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/ColumnExpressionBase.java, sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/constraints/Check.java, sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/constraints/ForeignKey.java, sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/constraints/PrimaryKey.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java:213— `principal.isEmpty() || keyTabFile.isEmpty()` appears character-identically in 2 files — sql/hive-thriftserver/src/main/java/org/apache/hive/service/auth/HiveAuthFactory.java, sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/CLIService.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Duplicated predicate common/network-common/src/main/java/org/apache/spark/network/protocol/RpcRequest.java:74— `requestId == o.requestId && super.equals(o)` appears character-identically in 3 files — common/network-common/src/main/java/org/apache/spark/network/protocol/RpcRequest.java, common/network-common/src/main/java/org/apache/spark/network/protocol/RpcResponse.java, common/network-common/src/main/java/org/apache/spark/network/protocol/UploadStream.java. It is one line, so the duplication detector's token window never sees it; the copies drift when only one is corrected. Give the condition a name and one home.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:314— `debug` at line 314 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:324— `debug` at line 324 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1248— `debug` at line 1248 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1267— `debug` at line 1267 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1286— `debug` at line 1286 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1413— `debug` at line 1413 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1427— `debug` at line 1427 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1445— `debug` at line 1445 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1469— `debug` at line 1469 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1759— `debug` at line 1759 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1777— `debug` at line 1777 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1804— `warning` at line 1804 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.warning("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.warning("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1858— `debug` at line 1858 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1902— `debug` at line 1902 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:1963— `debug` at line 1963 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:2446— `debug` at line 2446 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:2716— `debug` at line 2716 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/sql/connect/client/core.py:2746— `debug` at line 2746 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.debug("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.debug("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/ml/torch/distributor.py:232— `warning` at line 232 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.warning("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.warning("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/ml/torch/distributor.py:637— `info` at line 637 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.info("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.info("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/ml/torch/distributor.py:646— `info` at line 646 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.info("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.info("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/ml/torch/distributor.py:856— `info` at line 856 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.info("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.info("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging python/pyspark/ml/torch/distributor.py:872— `info` at line 872 is handed an f-string as its message, so the logger receives one finished string — the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the record is filtered out. Pass the values beside a constant message instead: `logger.info("order %s moved to %s", order_id, status)` with the standard library's `logging`, which keeps the message and its arguments apart on the record, or `log.info("order moved", order_id=order_id, status=status)` with structlog.
Interpolated log message defeats structured logging sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/LogDivertAppender.java:317— `debug` at line 317 is handed a message assembled at the call — a string concatenated with a value, or `String.format` — through SLF4J or log4j 2, so the logger receives one finished string: the values in it cannot be queried, filtered or aggregated as fields, every occurrence reads as a different message, and the string is built even when the level is filtered out. Pass a constant pattern and the values beside it instead: `log.debug("Order {} moved to {}", orderId, status)`, which keeps the pattern and its arguments apart on the event for a structured encoder (logstash-logback-encoder, log4j 2's JsonTemplateLayout, OpenTelemetry), or `log.atInfo().addKeyValue("order_id", orderId).log("order moved")` with SLF4J 2's fluent API. An exception goes last, as the throwable argument, not into the text.
Off-boarding risk: anonymized user #1 — If anonymized user #1 becomes unavailable, 47 significant file(s) lose their only recent owner: mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala, mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala, mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala, mllib/src/main/scala/org/apache/spark/ml/tree/treeModels.scala, mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala, mllib/src/main/scala/org/apache/spark/ml/feature/VectorIndexer.scala, mllib/src/main/scala/org/apache/spark/ml/evaluation/ClusteringMetrics.scala, mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala (+39 more). Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #2 — If anonymized user #2 becomes unavailable, 32 significant file(s) lose their only recent owner: core/src/main/scala/org/apache/spark/deploy/master/Master.scala, mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala, mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala, core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionServer.scala, core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala, core/src/main/scala/org/apache/spark/deploy/client/StandaloneAppClient.scala, resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala (+24 more). Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #3 — If anonymized user #3 becomes unavailable, 12 significant file(s) lose their only recent owner: core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala, sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/compression/compressionSchemes.scala, core/src/main/scala/org/apache/spark/util/collection/ExternalAppendOnlyMap.scala, streaming/src/main/scala/org/apache/spark/streaming/ui/BatchPage.scala, mllib/src/main/scala/org/apache/spark/ml/tree/impl/DecisionTreeMetadata.scala, sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/STUtils.java, dev/spark_merge_footer.py, sql/api/src/main/scala/org/apache/spark/sql/types/GeometryType.scala (+4 more). Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #4 — If anonymized user #4 becomes unavailable, 11 significant file(s) lose their only recent owner: common/network-common/src/main/java/org/apache/spark/network/TransportContext.java, core/src/main/scala/org/apache/spark/serializer/JavaSerializer.scala, common/network-common/src/main/java/org/apache/spark/network/server/AbstractAuthRpcHandler.java, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTranspiledPythonUDFOptions.scala, common/network-common/src/main/java/org/apache/spark/network/sasl/SaslRpcHandler.java, common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/FetchShuffleBlocks.java, core/src/main/scala/org/apache/spark/deploy/master/ZooKeeperPersistenceEngine.scala, common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/FetchShuffleBlockChunks.java (+3 more). Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #5 — If anonymized user #5 becomes unavailable, 11 significant file(s) lose their only recent owner: common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java, common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/OneForOneBlockFetcher.java, resource-managers/kubernetes/core/src/main/scala/org/apache/spark/shuffle/KubernetesLocalDiskShuffleExecutorComponents.scala, common/sketch/src/main/java/org/apache/spark/util/sketch/CountMinSketch.java, common/kvstore/src/main/java/org/apache/spark/util/kvstore/ArrayWrappers.java, sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/RunTerminationReason.scala, core/src/main/scala/org/apache/spark/scheduler/TaskInfo.scala, resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/ExecutorKubernetesCredentialsFeatureStep.scala (+3 more). Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #6 — If anonymized user #6 becomes unavailable, 8 significant file(s) lose their only recent owner: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/constraints.scala, sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter.java, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CdcNetChangesStatefulProcessor.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateOrdering.scala, sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/DateTimeExpressionUtils.java, sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/RowBasedHashMapGenerator.scala, sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/constraints/Check.java, sql/core/src/main/java/org/apache/spark/sql/execution/joins/JoinHelper.java. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #7 — If anonymized user #7 becomes unavailable, 9 significant file(s) lose their only recent owner: core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleReader.scala, core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleClientHandler.scala, sql/api/src/main/java/org/apache/spark/sql/streaming/Trigger.java, core/src/main/scala/org/apache/spark/shuffle/streaming/MultiShuffleManager.scala, sql/api/src/main/scala/org/apache/spark/sql/execution/streaming/Triggers.scala, core/src/main/scala/org/apache/spark/util/EventLoop.scala, core/src/main/scala/org/apache/spark/shuffle/streaming/StreamingShuffleServerHandler.scala, core/src/main/scala/org/apache/spark/shuffle/streaming/TaskContextAwareLogging.scala (+1 more). Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #8 — If anonymized user #8 becomes unavailable, 6 significant file(s) lose their only recent owner: udf/worker/grpc/src/main/scala/org/apache/spark/udf/worker/grpc/GrpcWorkerSession.scala, udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerDispatcher.scala, sql/core/src/main/scala/org/apache/spark/sql/execution/externalUDF/ExecuteExternalUDFExec.scala, sql/core/src/main/scala/org/apache/spark/sql/execution/externalUDF/PlanExternalUDFs.scala, udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/direct/DirectWorkerProcess.scala, udf/worker/core/src/main/scala/org/apache/spark/udf/worker/core/Termination.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #9 — If anonymized user #9 becomes unavailable, 6 significant file(s) lose their only recent owner: python/pyspark/sql/connect/local_server_pool.py, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NameStreamingSources.scala, sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/StreamingRelation.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/SequentialStreamingUnion.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/NamedStreamingRelation.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/SequentialUnionAnalysis.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #10 — If anonymized user #10 becomes unavailable, 6 significant file(s) lose their only recent owner: python/pyspark/eval_handlers/_base.py, python/pyspark/eval_handlers/verification.py, core/src/main/scala/org/apache/spark/rdd/AsyncRDDActions.scala, python/benchmarks/bench_local_data_to_arrow.py, python/pyspark/eval_handlers/utils.py, python/pyspark/eval_handlers/_typing.py. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #11 — If anonymized user #11 becomes unavailable, 6 significant file(s) lose their only recent owner: sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetCrossReferenceOperation.java, sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetReadState.java, sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/operation/GetPrimaryKeysOperation.java, sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DistributionAndOrderingUtils.scala, common/network-common/src/main/java/org/apache/spark/network/server/StreamManager.java, sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/PartitionReader.java. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #12 — If anonymized user #12 becomes unavailable, 6 significant file(s) lose their only recent owner: python/pyspark/testing/mlutils.py, python/pyspark/pandas/mlflow.py, python/pyspark/statcounter.py, python/pyspark/rddsampler.py, python/pyspark/find_spark_home.py, python/test_coverage/sitecustomize.py. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #13 — If anonymized user #13 becomes unavailable, 5 significant file(s) lose their only recent owner: sql/core/src/main/scala/org/apache/spark/sql/execution/python/ExtractPythonUDFFromLambda.scala, mllib/src/main/scala/org/apache/spark/mllib/tree/model/DecisionTreeModel.scala, mllib/src/main/scala/org/apache/spark/ml/image/ImageSchema.scala, core/src/main/scala/org/apache/spark/ui/JWSFilter.scala, connector/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaRedactionUtil.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #14 — If anonymized user #14 becomes unavailable, 5 significant file(s) lose their only recent owner: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushDownLeftSemiAntiJoin.scala, dev/pr_merge_status.py, sql/catalyst/src/main/scala/org/apache/spark/sql/util/SQLOpenHashSet.scala, dev/next_version_candidates.py, sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DelegatingTable.java. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #15 — If anonymized user #15 becomes unavailable, 5 significant file(s) lose their only recent owner: sql/core/src/main/scala/org/apache/spark/sql/execution/PushDownLocalSort.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PullUpProjectAliasThroughWindow.scala, common/kvstore/src/main/java/org/apache/spark/util/kvstore/KVStore.java, sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantWindowGroupLimits.scala, sql/core/src/main/scala/org/apache/spark/sql/execution/RemoveRedundantSorts.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #16 — If anonymized user #16 becomes unavailable, 4 significant file(s) lose their only recent owner: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/NestedColumnAliasing.scala, sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2VersionMap.scala, sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1ReconciliationStrategy.scala, sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1ForeachBatchHandler.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #17 — If anonymized user #17 becomes unavailable, 3 significant file(s) lose their only recent owner: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/ColumnStats.scala, sql/core/src/main/scala/org/apache/spark/sql/execution/python/ColumnarArrowPythonInput.scala, sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvalPythonEvaluatorFactory.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #18 — If anonymized user #18 becomes unavailable, 3 significant file(s) lose their only recent owner: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SparkPlanGraph.scala, sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala, core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #19 — If anonymized user #19 becomes unavailable, 3 significant file(s) lose their only recent owner: sql/core/src/main/scala/org/apache/spark/sql/execution/BinByExec.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/BinByResolution.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveBinBy.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #20 — If anonymized user #20 becomes unavailable, 3 significant file(s) lose their only recent owner: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/SubstituteParamsParser.scala, sql/core/src/main/scala/org/apache/spark/sql/catalyst/expressions/ParseSql.scala, sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/util/CharVarcharCodegenUtils.java. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #21 — If anonymized user #21 becomes unavailable, 3 significant file(s) lose their only recent owner: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesExecutorBackend.scala, sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/DelegatingCatalogExtension.java, core/src/main/scala/org/apache/spark/security/SocketAuthHelper.scala. Pair on, review, or document these before any departure.
Off-boarding risk: anonymized user #22 — If anonymized user #22 becomes unavailable, 3 significant file(s) lose their only recent owner: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpression.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/internal/connector/SupportsRuntimeCatalystFiltering.scala, sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalogCapability.java. Pair on, review, or document these before any departure.
Disabled test common/unsafe/src/test/scala/org/apache/spark/unsafe/types/UTF8StringPropertyCheckSuite.scala:228— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala:2910— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test core/src/test/scala/org/apache/spark/DriverSuite.scala:33— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala:719— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala:873— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala:2005— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test core/src/test/scala/org/apache/spark/scheduler/OutputCommitCoordinatorSuite.scala:148— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test core/src/test/scala/org/apache/spark/ui/UISuite.scala:78— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test core/src/test/scala/org/apache/spark/ui/UISuite.scala:92— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test core/src/test/scala/org/apache/spark/util/collection/ExternalSorterSuite.scala:102— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/classification/LinearSVCSuite.scala:71— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala:161— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala:55— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala:130— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/regression/LinearRegressionSuite.scala:132— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala:53— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala:92— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/source/image/ImageFileFormatSuite.scala:113— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala:599— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Disabled test mllib/src/test/scala/org/apache/spark/mllib/stat/CorrelationSuite.scala:137— A test is skipped/ignored — coverage that looks present but never runs. Re-enable it or delete it so the green run means something.
Blanket warning suppression connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtilsPythonHelper.scala:32— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression connector/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/SparkAWSCredentials.scala:106— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression core/src/main/scala/org/apache/spark/InternalAccumulator.scala:49— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala:206— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression core/src/main/scala/org/apache/spark/ui/UIUtils.scala:419— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression core/src/main/scala/org/apache/spark/ui/UIWorkloadGenerator.scala:28— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression core/src/main/scala/org/apache/spark/ui/exec/ExecutorThreadDumpPage.scala:99— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression core/src/main/scala/org/apache/spark/ui/storage/StoragePage.scala:91— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression core/src/main/scala/org/apache/spark/util/BlockingThreadPoolExecutorService.scala:25— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression mllib/src/main/scala/org/apache/spark/ml/functions.scala:26— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression mllib/src/main/scala/org/apache/spark/ml/param/params.scala:93— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression mllib/src/main/scala/org/apache/spark/ml/param/shared/sharedParams.scala:27— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression mllib/src/main/scala/org/apache/spark/ml/r/GeneralizedLinearRegressionWrapper.scala:68— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Blanket warning suppression mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala:117— A bare `// scalastyle:off`, `// scalafix:off` or `@nowarn` (or Wartremover's `warts.All`) switches every rule or warning off — hidden debt. Suppress the specific rule on the specific line, with a justification, instead.
Fat interface: HiveSession (26 abstract methods) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSession.java:29— `HiveSession` declares 26 abstract methods: `cancelDelegationToken`, `cancelOperation`, `close`, `closeExpiredOperations`, `closeOperation`, `executeStatement`, `executeStatement`, `executeStatementAsync`, `executeStatementAsync`, `fetchResults`, `getCatalogs`, `getColumns`, `getCrossReference`, `getDelegationToken`, `getFunctions`, `getInfo`, `getMetaStoreClient`, `getNoOperationTime`, `getPrimaryKeys`, `getResultSetMetadata`, `getSchemas`, `getTableTypes`, `getTables`, `getTypeInfo`, `open`, `renewDelegationToken`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. Those 26 declarations carry 24 distinct name(s), so part of the width is overloading rather than separate responsibilities — but 24 operations clears the >15 bar on its own, so the row is about responsibilities, not signatures. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces.
Fat interface: HiveSessionBase (17 abstract methods) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionBase.java:32— `HiveSessionBase` declares 17 abstract methods: `getHiveConf`, `getIpAddress`, `getLastAccessTime`, `getOperationLogSessionDir`, `getPassword`, `getProtocolVersion`, `getSessionHandle`, `getSessionManager`, `getSessionState`, `getUserName`, `getUsername`, `isOperationLogEnabled`, `setIpAddress`, `setOperationLogSessionDir`, `setOperationManager`, `setSessionManager`, `setUserName`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces.
Fat interface: ICLIService (27 abstract methods) sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/ICLIService.java:27— `ICLIService` declares 27 abstract methods: `cancelDelegationToken`, `cancelOperation`, `closeOperation`, `closeSession`, `executeStatement`, `executeStatement`, `executeStatementAsync`, `executeStatementAsync`, `fetchResults`, `fetchResults`, `getCatalogs`, `getColumns`, `getCrossReference`, `getDelegationToken`, `getFunctions`, `getInfo`, `getOperationStatus`, `getPrimaryKeys`, `getQueryId`, `getResultSetMetadata`, `getSchemas`, `getTableTypes`, `getTables`, `getTypeInfo`, `openSession`, `openSessionWithImpersonation`, `renewDelegationToken`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. Those 27 declarations carry 24 distinct name(s), so part of the width is overloading rather than separate responsibilities — but 24 operations clears the >15 bar on its own, so the row is about responsibilities, not signatures. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces.
Fat interface: VectorizedValuesReader (31 abstract methods) sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedValuesReader.java:30— `VectorizedValuesReader` declares 31 abstract methods: `readBinary`, `readBinary`, `readBoolean`, `readBooleans`, `readByte`, `readBytes`, `readDouble`, `readDoubles`, `readFloat`, `readFloats`, `readGeography`, `readGeometry`, `readInteger`, `readIntegers`, `readIntegersWithRebase`, `readLong`, `readLongs`, `readLongsWithRebase`, `readShort`, `readShorts`, `readUnsignedIntegers`, `readUnsignedLongs`, `skipBinary`, `skipBooleans`, `skipBytes`, `skipDoubles`, `skipFixedLenByteArray`, `skipFloats`, `skipIntegers`, `skipLongs`, `skipShorts`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. Those 31 declarations carry 30 distinct name(s), so part of the width is overloading rather than separate responsibilities — but 30 operations clears the >15 bar on its own, so the row is about responsibilities, not signatures. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces.
Fat interface: SpecializedGetters (20 abstract methods) sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/SpecializedGetters.java:31— `SpecializedGetters` declares 20 abstract methods: `get`, `getArray`, `getBinary`, `getBinaryView`, `getBoolean`, `getByte`, `getDecimal`, `getDouble`, `getFloat`, `getInt`, `getInterval`, `getLong`, `getMap`, `getShort`, `getStruct`, `getTimestampLTZNanos`, `getTimestampNTZNanos`, `getUTF8String`, `getVariant`, `isNullAt`. Counted over its ABSTRACT methods, once per signature; `default`, `static` and `private` methods and constants are not counted, because no implementer is forced to write them. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces.
Fat interface: ExternalCatalog (40 abstract members) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/ExternalCatalog.scala:34— `ExternalCatalog` declares 40 abstract members: `createDatabase`, `dropDatabase`, `alterDatabase`, `getDatabase`, `databaseExists`, `listDatabases`, `listDatabases`, `setCurrentDatabase`, `createTable`, `dropTable`, `renameTable`, `alterTable`, `alterTableDataSchema`, `alterTableSchema`, `alterTableStats`, `getTable`, `getTablesByName`, `tableExists`, `listTables`, `listTables`, `listViews`, `loadTable`, `loadPartition`, `loadDynamicPartitions`, `createPartitions`, `dropPartitions`, `renamePartitions`, `alterPartitions`, `getPartition`, `getPartitionOption`, `listPartitionNames`, `listPartitions`, `listPartitionsByFilter`, `createFunction`, `dropFunction`, `alterFunction`, `renameFunction`, `getFunction`, `functionExists`, `listFunctions`. Counted over its ABSTRACT members — a `def` with no body or a `val`/`var` with no initializer, once per signature; concrete members, private and protected members, and members of a companion `object` are not counted, because no implementer is forced to write them. Those 40 declarations carry 38 distinct name(s), so part of the width is overloading rather than separate responsibilities — but 38 operations clears the >15 bar on its own, so the row is about responsibilities, not signatures. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces.
Fat interface: LogicalPlanVisitor[T] (24 abstract members) sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlanVisitor.scala:23— `LogicalPlanVisitor[T]` declares 24 abstract members: `default`, `visitAggregate`, `visitDistinct`, `visitExcept`, `visitExpand`, `visitFilter`, `visitGenerate`, `visitGlobalLimit`, `visitOffset`, `visitIntersect`, `visitJoin`, `visitLocalLimit`, `visitPivot`, `visitProject`, `visitRepartition`, `visitRepartitionByExpr`, `visitRebalancePartitions`, `visitSample`, `visitScriptTransform`, `visitUnion`, `visitWindow`, `visitTail`, `visitSort`, `visitWithCTE`. Counted over its ABSTRACT members — a `def` with no body or a `val`/`var` with no initializer, once per signature; concrete members, private and protected members, and members of a companion `object` are not counted, because no implementer is forced to write them. A wide interface forces every implementer and caller to depend on methods they don't use (the Interface-Segregation 'I' in SOLID). Split it into focused role-interfaces.
Silent fallback default in catch core/src/main/scala/org/apache/spark/api/r/BaseRRunner.scala:281— `getROptions` swallows every exception into `"--vanilla"` — a data error becomes a silent behavior change with no trail. Log the failure or let it propagate. If that constant is the correct answer rather than a stand-in for a value that could not be read, say so in a comment in the catch case or in the Scaladoc, and the row stops firing.
Silent fallback default in catch sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:276— `confHasReplUri` (in `isReplContext`) swallows every exception into `false` — a data error becomes a silent behavior change with no trail. Log the failure or let it propagate. If that constant is the correct answer rather than a stand-in for a value that could not be read, say so in a comment in the catch case or in the Scaladoc, and the row stops firing.
Silent fallback default in catch sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeCompiler.scala:287— `eclInChain` (in `isReplContext`) swallows every exception into `false` — a data error becomes a silent behavior change with no trail. Log the failure or let it propagate. If that constant is the correct answer rather than a stand-in for a value that could not be read, say so in a comment in the catch case or in the Scaladoc, and the row stops firing.
Silent fallback default in catch common/network-common/src/main/java/org/apache/spark/network/util/NettyMemoryMetrics.java:124— `registerArenaMetric` swallows every exception into `return -1` — a data error becomes a silent behavior change with no trail. Log the failure or let it propagate. If that constant is the correct answer rather than a stand-in for a value that could not be read, say so in a comment in the catch or in the Javadoc, and the row stops firing.
Silent fallback default in catch REDACTED:327— `getNodeDetails` swallows every exception into `return {}` — a data error becomes a silent behavior change with no trail. Log the failure or let it raise. If that constant is the correct answer rather than a stand-in for a value that could not be read, say so in a comment on the handler or in the docstring, and the row stops firing.
Silent fallback default on parse failure core/src/main/scala/org/apache/spark/ui/SparkUI.scala:155— `getSparkUser` answers `"<unknown>"` when `NoSuchElementException` is thrown, with no log or throw — a malformed value silently becomes that default. Log the input or fail validation. If that constant is the correct answer rather than a stand-in for a value that could not be read, say so in a comment in the catch case or in the Scaladoc, and the row stops firing.
Silent fallback default on parse failure python/pyspark/util.py:989— `spark_connect_mode` returns `'0'` when `ImportError` is raised, with no log or raise — a malformed value silently becomes that default. Log the input or fail validation. If that constant is the correct answer rather than a stand-in for a value that could not be read, say so in a comment on the handler or in the docstring, and the row stops firing.
D16 · Bus Factor· Further sole-owners (lower concentration) · ×1
Further sole-owners (lower concentration) — 45 other contributor(s) are each the sole owner of a small amount of code below the off-boarding threshold — folded into the bus-factor score and metrics (250 single-owned of 2854 analysed files in total, counted over production source files of roughly 2,400 bytes or more, excluding vendored, generated, example/demo and scaffolding-template trees and test files identified by path convention, largest first; 2854 of the 4632 production source files in this repository met that bar). They are anonymized user #23 (2 file(s)), anonymized user #24 (2 file(s)), anonymized user #25 (2 file(s)), anonymized user #26 (2 file(s)), anonymized user #27 (2 file(s)), anonymized user #28 (2 file(s)) (+39 more) — spread or document their files in the same way, at lower priority than the named off-boarding risks above. The files folded here: core/src/main/scala/org/apache/spark/util/LastAttemptAccumulator.scala, core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala, sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/CLIService.java, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DecimalPrecisionTypeCoercion.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ProjectionOverSchema.scala, connector/kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaTokenUtil.scala, core/src/main/scala/org/apache/spark/status/protobuf/KVStoreProtobufSerializer.scala (+45 more).
No ADRs found — No ADRs found. No recognised ADR directory (`docs/adr/`, `docs/decisions/`, `adr/`, `docs/rfcs/`, an `ADR0001/` folder, or their siblings) exists anywhere in this tree. What was searched, so you can tell an empty log from a search that missed one: every directory under the tree (build output, dependencies and VCS metadata excepted), for a document that is either any non-index page inside a recognised ADR directory, whatever its name and however deeply nested (`docs/adr/use-postgres.md`, `docs/adr/2024/0001-x.md`); or a file anywhere whose name is ADR-shaped (`0001-use-postgres.md`, `adr-012-caching.md`); or, when neither turned anything up, a document carrying the decision-record signature (an "Architecture Decision Record" heading, or Status / Context / Decision / Consequences as section headings). A decision log that clears none of these — unnumbered files outside any recognised directory, without those headings — is not seen by this check and this row is then wrong. If that is your case, say so rather than renaming anything; otherwise, consider recording architectural decisions in `docs/adr/`.
D26 · Project Cohesion· Projects may be oversized for their cohesion · ×1
Projects may be oversized for their cohesion — 10 of 51 project(s) overshoot their size bounds, lowering Project Cohesion to 6.1/10. The most over is `sql/catalyst` (259450 LoC, 3394 module-visible types across 80 directories). Review these for cohesion — draw the boundary inside the module first (group each responsibility into its own package or directory and keep the cross-boundary members non-public), since splitting a published package moves types between packages and breaks consumers.
D28 · Secrets (history)· Rotate the exposed credentials · ×1
D34 · Knowledge Freshness· Further orphaned files (smaller) · ×1
Further orphaned files (smaller) — 538 smaller file(s) also have no living knowledge — folded into the freshness score and metrics rather than raised one row each — most significant first: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/linearRegression.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/tupleUnionAgg.scala, core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala, mllib/src/main/scala/org/apache/spark/mllib/recommendation/ALS.scala, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/AnalysisHelper.scala, python/pyspark/streaming/context.py, python/pyspark/ml/fpm.py, sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveWithCTE.scala (and 530 more) (576 orphaned of 2905 analysed files in total, counted over production source files of roughly 2,400 bytes or more, excluding vendored, generated, example/demo and scaffolding-template trees and test files identified by path convention, largest first; 2905 of the 4632 production source files in this repository met that bar). Attach the read to the next change that touches one of them: have a second person review that change, and leave behind a short comment or test recording what the file is for, so the knowledge comes back at the cost of a change you were making anyway.
IC1 · Incompleteness & stubs· Empty method body · ×1
Empty method body mllib/src/main/scala/org/apache/spark/sql/ml/InternalFunctionRegistration.scala:38— `apply` has an empty body — confirm it's an intentional no-op and not an unfinished method.
No ADRs — No Architecture Decision Records found — no conventional ADR directory, no numbered `NNNN-title` documents in any markup this check reads, and nothing ADR-shaped by content. Design rationale recorded elsewhere (a design-notes tree, a mailing list, pull-request discussion) is not visible to this check and is not re-findable per decision, so a future maintainer cannot ask why one choice was made and get an answer.
Logging is not universal — Only 23/29 runnable modules use logging (modules with no entry point or server are excluded — they are libraries a runnable module hosts). Silent: `connector/credential-aws-integration-tests`, `dev`, `repl`, `sql/connect/client/jvm`, `sql/connect/server` and 1 more.
P4 · Deployment & Rollback· No rollback/health safety · ×1
No rollback/health safety — Deployment automation exists and the service already exposes health probe endpoints (healthz/livez/readyz or a container healthcheck), but no manifest in this repository wires them into readiness/liveness probes, a rolling-update strategy, lifecycle hooks or a migration job — a bad release is harder to detect and reverse. If that wiring lives in a separate ops/Helm repository, document the pointer here so release safety is evidenced.
No changelog — No CHANGELOG/HISTORY/RELEASES file — what shipped when isn't easy to reconstruct for support or audit. (Versioning/tagging makes releases traceable, but a changelog records the what.)
X5 · Nullable reference types· Scala's null is not held to the type system everywhere · ×1
Scala's null is not held to the type system everywhere — No build over this repository's 3346 Scala product file(s) passes `-Yexplicit-nulls`, enables WartRemover's `Null` wart or runs Scalafix's `DisableSyntax.noNulls`, so every reference type admits null and nothing reports a `null` written into the code or returned from a Java API — the first one to reach a method call is a `NullPointerException`. `Option` by convention is a habit no tool here holds the code to.
Outdated: ffi — `ffi` is resolved to 1.17.2 in the committed Gemfile.lock but 1.17.4 is the current release on rubygems.org, and it already satisfies the `"~> 1.15"` requirement declared in docs/Gemfile — so the lockfile is behind this repository's own declared range. This build is therefore missing every fix published since, including any security fix. Run `bundle update ffi` and commit the updated Gemfile.lock.
Outdated: jekyll-redirect-from — `jekyll-redirect-from` is resolved to 0.16.0 in the committed Gemfile.lock but 0.17.0 is the current release on rubygems.org, and it already satisfies the `"~> 0.16"` requirement declared in docs/Gemfile — so the lockfile is behind this repository's own declared range. This build is therefore missing every fix published since, including any security fix. Run `bundle update jekyll-redirect-from` and commit the updated Gemfile.lock.
Appendix B — Reproduction & audit trail
Every external tool invocation behind a deep-scan dimension — the tool, its captured version, the exact command, how many findings it yielded, and a link to the retained raw output. To reproduce any finding: check out the same commit and run the command shown (repo-relative — never an absolute scratch path). The complete raw scanner output is retained verbatim under artifacts/raw/ (indexed in artifacts/raw/index.json); per-invocation exit codes and wall-clock durations are in sidecar.json — kept out of this table so the rendered report stays byte-identical across runs of the same commit.
semgrep: not applicable — No personal data was found crossing a boundary the PII/GDPR ruleset checks — nothing written to a log or console sink, placed in a URL or query string, or persisted to browser storage. That is a clean result for the LEAK surface only: this ruleset detects personal data escaping, it does not inventory the personal data a repository holds, so it is not evidence that this repository has no personal-data surface. The personal-data map (Appendix C) and the C1-C5 compliance cards are what speak to that. semgrep could not parse 48 file(s) — `common/utils/src/main/scala/org/apache/spark/internal/config/ConfigBuilder.scala`, `core/src/main/scala/org/apache/spark/deploy/history/HistoryPage.scala`, `core/src/main/scala/org/apache/spark/deploy/history/HistoryServer.scala`, `core/src/main/scala/org/apache/spark/deploy/history/LogPage.scala`, `core/src/main/scala/org/apache/spark/deploy/master/ui/ApplicationPage.scala`, … (+43 more) — so the PII/GDPR sweep did not cover them at all.
runtime-hardening: not applicable — No Kubernetes/orchestration workloads found in the repository manifests; network egress policy is a cluster-native control that may live at the platform/firewall layer, so there is nothing to assess here.
runtime-hardening: not applicable — No Kubernetes/orchestration workloads found in the repository manifests; seccomp/AppArmor/SELinux confinement is a workload-level control, so there is nothing to assess here.
runtime-hardening: not applicable — No Kubernetes/orchestration workloads found in the repository manifests; runtime threat-detection and admission-control policy are cluster-level controls, so there is nothing to assess here.
Run 01a17762-ac26-7349-862a-a56a76038463 · every finding is also locatable in findings.md, and the complete scoring record (with exit codes + durations) in sidecar.json.
Issues: 211 · Warnings: 4803 · Recommendations: 684 · Info: 2 — Appendix A · all findings · full markdown report.
Generated by Watchdog — deterministic code-health analysis. 26-10-2026 @ 08:45 UTC.
Downloadable artifacts
Machine-readable and reproducible from this commit + frozen rubric — drop them straight into a contract appendix, a CRA dossier, or a downstream SCA / VEX tool.