[dynamo] fix error_on_graph_break bug where non-empty checkpoint results in unwanted graph break resumption #166586

williamwen42 · 2025-10-29T23:09:34Z

Stack from ghstack (oldest at bottom):

-> [dynamo] fix error_on_graph_break bug where non-empty checkpoint results in unwanted graph break resumption #166586
[dynamo, nested graph breaks] add disable_nested_graph_breaks decorator/context manager #166477
[dynamo, nested graph breaks] codegen dead nested cells correctly #166476

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @Lucaskabela @mlazos

…lts in unwanted graph break resumption [ghstack-poisoned]

pytorch-bot · 2025-10-29T23:09:38Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166586

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit faa392b with merge base df71b70 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Lucaskabela

Minor though on improving this debuggability

Lucaskabela · 2025-10-30T17:32:19Z

torch/_dynamo/symbolic_convert.py

+                or self.error_on_graph_break
+                or self.is_tracing_resume_prologue
+            ):
+                raise


Should we raise e here for error visibility?

I think re-raise is standard Python practice? It's done below.

williamwen42 · 2025-10-30T17:58:07Z

@pytorchbot merge

pytorchmergebot · 2025-10-30T18:00:02Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-10-30T23:58:30Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

williamwen42 · 2025-10-31T00:06:16Z

@pytorchbot merge

pytorchmergebot · 2025-10-31T00:08:12Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

williamwen42 · 2025-10-31T00:49:00Z

@pytorchbot rebase

pytorchmergebot · 2025-10-31T00:50:38Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

[ghstack-poisoned]

…lts in unwanted graph break resumption ghstack-source-id: 9676e4f Pull Request resolved: #166586

pytorchmergebot · 2025-10-31T00:50:57Z

Successfully rebased gh/williamwen42/322/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/166586)

williamwen42 · 2025-10-31T00:54:53Z

@pytorchbot merge

pytorchmergebot · 2025-10-31T00:55:12Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

pytorchmergebot · 2025-10-31T00:57:09Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…66587) Pull Request resolved: #166587 Approved by: https://github.com/Lucaskabela ghstack dependencies: #166476, #166477, #166586

…lts in unwanted graph break resumption (#166586) Fixes #166589 Pull Request resolved: #166586 Approved by: https://github.com/Lucaskabela ghstack dependencies: #166476, #166477

…66587) Pull Request resolved: #166587 Approved by: https://github.com/Lucaskabela ghstack dependencies: #166476, #166477, #166586

Lucaskabela · 2025-11-04T00:07:55Z

@pytorchbot cherry-pick --onto release/2.9 --fixes "error in new nested graphbreak feature (#166589)" -c fixnewfeature

pytorchbot · 2025-11-04T00:17:07Z

Cherry picking #166586

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x 267d0197bfca0232488d51dd1ff735d619adc2cf returned non-zero exit code 1

Auto-merging test/dynamo/test_decorators.py
Auto-merging torch/_dynamo/symbolic_convert.py
CONFLICT (content): Merge conflict in torch/_dynamo/symbolic_convert.py
error: could not apply 267d0197bfc... [dynamo] fix error_on_graph_break bug where non-empty checkpoint results in unwanted graph break resumption (#166586)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

…lts in unwanted graph break resumption (#166586) Fixes #166589 Pull Request resolved: #166586 Approved by: https://github.com/Lucaskabela ghstack dependencies: #166476, #166477 (cherry picked from commit 267d019)

…lts in unwanted graph break resumption (pytorch#166586) Fixes pytorch#166589 Pull Request resolved: pytorch#166586 Approved by: https://github.com/Lucaskabela ghstack dependencies: pytorch#166476, pytorch#166477

…torch#166587) Pull Request resolved: pytorch#166587 Approved by: https://github.com/Lucaskabela ghstack dependencies: pytorch#166476, pytorch#166477, pytorch#166586

…lts in unwanted graph break resumption (#166925) [dynamo] fix error_on_graph_break bug where non-empty checkpoint results in unwanted graph break resumption (#166586) Fixes #166589 Pull Request resolved: #166586 Approved by: https://github.com/Lucaskabela ghstack dependencies: #166476, #166477 (cherry picked from commit 267d019) Co-authored-by: William Wen <[email protected]>

…lts in unwanted graph break resumption ghstack-source-id: 407b722 Pull Request resolved: pytorch/pytorch#166586

[dynamo] fix error_on_graph_break bug where non-empty checkpoint resu…

22a99c0

…lts in unwanted graph break resumption [ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: dynamo labels Oct 29, 2025

williamwen42 added topic: bug fixes topic category release notes: dynamo labels Oct 29, 2025

Lucaskabela approved these changes Oct 30, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 30, 2025

pytorchmergebot added the merging label Oct 30, 2025

This was referenced Oct 30, 2025

[dynamo] clean up BaseUserFunctionVariable and LocalGeneratorObjectVariable #166673

Closed

[dynamo, nested graph breaks] disallow graph breaks in functorch ops, enable nested graph break tests on test_higher_order_ops.py #166674

Closed

Update

faa392b

[ghstack-poisoned]

pytorchmergebot pushed a commit that referenced this pull request Oct 31, 2025

[dynamo] fix error_on_graph_break bug where non-empty checkpoint resu…

749738e

…lts in unwanted graph break resumption ghstack-source-id: 9676e4f Pull Request resolved: #166586

pytorchmergebot added the Merged label Oct 31, 2025

pytorchmergebot closed this in 267d019 Oct 31, 2025

pytorchmergebot removed the merging label Oct 31, 2025

pytorchmergebot pushed a commit that referenced this pull request Oct 31, 2025

[dynamo] clarify graph break handling/logging in symbolic_convert (#1…

70aeb49

…66587) Pull Request resolved: #166587 Approved by: https://github.com/Lucaskabela ghstack dependencies: #166476, #166477, #166586

BoyuanFeng pushed a commit that referenced this pull request Oct 31, 2025

[dynamo] clarify graph break handling/logging in symbolic_convert (#1…

63fbd7c

…66587) Pull Request resolved: #166587 Approved by: https://github.com/Lucaskabela ghstack dependencies: #166476, #166477, #166586

Lucaskabela added this to the 2.9.1 milestone Nov 4, 2025

This was referenced Nov 4, 2025

[dynamo] fix error_on_graph_break bug where non-empty checkpoint results in unwanted graph break resumption #166925

Merged

[v2.9.1] Release Tracker #166758

Closed

atalman mentioned this pull request Nov 10, 2025

Release 2.9.1 validations checklist and cherry-picks #167476

Open

38 tasks

Khanaksahu pushed a commit to Khanaksahu/pytorch that referenced this pull request Nov 17, 2025

[dynamo] fix error_on_graph_break bug where non-empty checkpoint resu…

ec1e297

…lts in unwanted graph break resumption ghstack-source-id: 407b722 Pull Request resolved: pytorch/pytorch#166586

github-actions bot deleted the gh/williamwen42/322/head branch December 4, 2025 02:18

[dynamo] fix error_on_graph_break bug where non-empty checkpoint results in unwanted graph break resumption #166586

[dynamo] fix error_on_graph_break bug where non-empty checkpoint results in unwanted graph break resumption #166586

Uh oh!

Conversation

williamwen42 commented Oct 29, 2025 • edited by pytorchmergebot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166586

✅ No Failures

Uh oh!

Lucaskabela left a comment

Choose a reason for hiding this comment

Uh oh!

Lucaskabela Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

williamwen42 Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

williamwen42 commented Oct 30, 2025

Uh oh!

pytorchmergebot commented Oct 30, 2025

Merge started

Uh oh!

pytorchmergebot commented Oct 30, 2025

Uh oh!

williamwen42 commented Oct 31, 2025

Uh oh!

pytorchmergebot commented Oct 31, 2025

Merge started

Uh oh!

williamwen42 commented Oct 31, 2025

Uh oh!

pytorchmergebot commented Oct 31, 2025

Uh oh!

pytorchmergebot commented Oct 31, 2025

Uh oh!

williamwen42 commented Oct 31, 2025

Uh oh!

pytorchmergebot commented Oct 31, 2025

Uh oh!

pytorchmergebot commented Oct 31, 2025

Merge started

Uh oh!

Lucaskabela commented Nov 4, 2025

Uh oh!

pytorchbot commented Nov 4, 2025

Cherry picking #166586

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

williamwen42 commented Oct 29, 2025 •

edited by pytorchmergebot

Loading

pytorch-bot bot commented Oct 29, 2025 •

edited

Loading

williamwen42 Oct 30, 2025 •

edited

Loading