Skip to content

Unified: Replace swift-syntax with Rust Swift parser - #23140

Draft
paldepind wants to merge 20 commits into
mainfrom
unified/swift-parser-port
Draft

paldepind wants to merge 20 commits into
mainfrom
unified/swift-parser-port

Conversation

@paldepind

@paldepind paldepind commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Implement the native Rust Swift lexer/parser and make it the default extractor backend for Cargo and Bazel.
  • Preserve explicit SwiftSyntax selection for fallback and differential testing.
  • Add oracle-backed parser conformance fixtures, the full comparison corpus, and focused performance tooling.
  • Reduce parser allocation, token-dispatch, and operator-table overhead without changing the sink contract.

Validation

  • Final full parser corpus at f098d1b: 59 projects; 65,658 passed, 0 failed, 8,376 invalid-source skips. Existing skip policy unchanged; no references regenerated.
  • 85 focused parser fixtures pass, including the initialized-binding/newline-closure regression found by CI. The regression source is syntax-valid in SwiftSyntax.
  • Native extractor integration tests and default/explicit-backend Cargo and Bazel tests pass.
  • Current head f098d1b: 47 successful CI checks, one neutral, no failures or pending checks. All three Unified checks pass.

Performance

  • Production backend comparison on 15 pinned DCA projects: 5.00x faster extraction; 1.80x faster complete database creation, geometric means.
  • 60 fresh databases: two repetitions per backend/project, alternating/reversed order, two discarded warmups, two workers, warm source caches. All databases passed health/input-coverage checks with zero diagnostics.
  • Summed project medians: extraction 452.10 to 77.52 seconds; complete creation 560.35 to 192.55 seconds. Shared-workstation variance is substantial; the smallest projects show little end-to-end benefit.
  • Per-project timings and methodology: Unified: Replace swift-syntax with Rust Swift parser #23140 (comment)
  • Separate parser-only internal optimizations measured 1.58x across eight projects; this is distinct from the backend-replacement figures above.

DCA

  • Started after CI passed: github/codeql-dca-main#40671.
  • 53 Unified nightly source targets; database caching disabled, QL-submodule replacement and tuple counting enabled.
  • Compares PR merge base 2e2b727 against f098d1b using the same engine revision. The experiment is running; results are pending.

paldepind and others added 19 commits November 15, 2026 10:41
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 2b06b427-750d-4b6d-8a11-3f23fd8be98f
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 2b06b427-750d-4b6d-8a11-3f23fd8be98f
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 2b06b427-750d-4b6d-8a11-3f23fd8be98f
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 2b06b427-750d-4b6d-8a11-3f23fd8be98f
Stop each corpus project after its first mismatch on valid Swift. Record swift-syntax validity in reference JSON, skip invalid corpus files, and preserve exhaustive fixture and reference-generation runs.\n\nRemove recursive submodule initialization from corpus checkouts.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: 7a84baa3-dfce-406e-ac22-03e2ce08f78e
Add recursive-descent parsing with bottom-up syntax sinks, operator folding, recovery, and oracle-backed fixtures.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Handle key paths, regex and multiline literals, conditional attributes, closures, macros, and modern type syntax with corpus-driven fixtures.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Parse ownership and pack syntax, structured attributes, macros, module selectors, inline arrays, and context-sensitive declarations and closures.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Match the Swift compiler corpus with corrected operator folding, contextual syntax, literals, attributes, and type parsing. Add focused oracle fixtures and optional diagnostic selectors while preserving exhaustive corpus defaults.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Compare raw syntax trees and trivia without constructing desugared AST snapshots or loading desugaring rules.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Add a reusable in-memory parser benchmark and safe hyperfine helpers for direct extraction without database creation.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Reuse child buffers across productions and bypass expression stacks when
there is no operator. Fast-path ASCII lexing and inline cursor predicates
without changing emitted syntax or the public sink contract.

Cache the immutable standard operator table and clone it only when a
source may declare custom infix operators or precedence groups.

Validate against all 59 corpus projects: 65,658 passed, no failures, and
8,376 invalid-source skips. Existing extractor integration tests pass.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Select the Rust backend by default in Cargo and Bazel, retaining explicit swift-syntax opt-in and Rust precedence when both Cargo features are enabled. Forward Cargo options through the pack builder for fallback builds.

Keep mapped AST corpus checks enabled for both backends. Check and regenerate swift-syntax-specific raw snapshots only with swift-syntax, and document backend selection.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Document native Rust defaults and explicit SwiftSyntax fallback flags for Cargo and Bazel.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
Match SwiftSyntax's variable binding disambiguation: braces on the next line after an initializer start a separate closure unless they introduce observing accessors. Keep same-line accessors and uninitialized computed properties unchanged.

Add an oracle-generated parser fixture covering nested shadowing, initialized bindings, same-line accessor recovery, and multiline observing and computed accessors. Reproduce the local-name-binding CodeQL regression with the old native pack, then verify the rebuilt native pack passes the focused test, all 85 parser fixtures, all 59 extractor tests, and all 106 locally available CI-equivalent QL checks.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: f0ffa64f-7391-43d6-9745-52745351435a
@paldepind

Copy link
Copy Markdown
Contributor Author

SwiftSyntax → native Rust: 15-project production benchmark

The fixed CI candidate f098d1b gives 5.00× faster actual extraction and 1.80× faster complete database creation, geometric means across 15 pinned projects from the Unified DCA nightly suite.

Project Extractor speedup Full DB speedup
frzi/swiftui-router 2.72× 1.07×
djyde/WebShell 5.55× 1.06×
vsanthanam/JBird 3.30× 0.98×
prateek147/DVIA-v2 3.55× 1.00×
Alamofire/Alamofire 3.97× 1.24×
dheerajghub/Dovve 5.40× 1.48×
AndrewBennet/ReadingListV1 4.17× 1.79×
utmapp/UTM 6.13× 1.70×
krzysztofzablocki/Sourcery 7.47× 2.13×
JohanDegraeve/xdripswift 7.37× 2.76×
mozilla-mobile/firefox-ios 6.54× 2.64×
soto-project/soto 9.09× 4.76×
ProtonMail/ios-mail 4.16× 1.92×
kickstarter/ios-oss 5.84× 2.98×
manaflow-ai/cmux 4.01× 2.53×

Summed project medians: extraction 452.10 → 77.52 s (5.83×), full database creation 560.35 → 192.55 s (2.91×). These weighted totals are distinct from the equal-project geometric means. Largest project-median extractor RSS fell from 3,646 to 1,990 MiB; every project used less extractor RSS.

Method: Apple M3 Max, two workers; two repetitions per backend/project, alternating backend order and reversing it for repetition two; two discarded warmups; warm source caches but a fresh database for every run. Production extractor timings include parsing, AST mapping/desugaring, TRAP compression and source archival; full creation additionally includes CodeQL/JVM orchestration, import and finalization. This compares parser backends, not just the later parser-internal optimizations.

All 60 measured databases finalized and passed dataset/coverage checks, with zero diagnostics. All 18,026 selected Swift inputs were covered; 72,104 archived-input hash checks passed. These checks establish database health and input coverage, not independent AST equality. All pre-fix measurements were excluded and the full paired trial restarted.

Limitations: shared workstation, only two observations per condition, substantial background-noise variance. Small projects were essentially unchanged end-to-end; JBird and DVIA were slightly slower on the native median. This is a directional result, not a precise universal speedup.

The uncached, tuple-counting, QL-submodule-override DCA was started after CI passed: github/codeql-dca-main#40671. DCA results remain pending.

@paldepind paldepind changed the title Unified: Replace SwiftSyntax with the native Rust Swift parser Unified: Replace swift-syntax with Rust Swift parser Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant