fix(seer): Don't run/score Smart Assignment against automatic assignments - #121368
Merged
Conversation
github-actions
Bot
added
the
Scope: Backend
hobzcalvin
force-pushed
the
sa-scoring-human
branch
from
August 5, 2026 21:59
be57e46 to
73a532d
Compare
saponifi3d
approved these changes
Aug 5, 2026
saponifi3d
left a comment
saponifi3d
left a comment
Contributor
There was a problem hiding this comment.
lgtm, i really like how you separated out all the concerns here!
hobzcalvin
added a commit
that referenced
this pull request
Aug 5, 2026
…t ASSIGNED (#121375) We were only pulling issue assignee data if the activity that triggered Smart Assignment was `ActivityType.ASSIGNED`. But really, we can/should pull the issue's current assignee, if any, regardless of why SA is running. (For example, if it's running because of `SEER_RCA_STARTED`, that doesn't mean we shouldn't record the issue's existing assignee from long ago as the ground truth.) Here, we just pull the current assignee if we have one. (Note that #121368 which we're stacked upon ensures that only human-involved assignees are recorded, not automated ones.) We continue falling back on a resolving human if this is a `RESOLUTION_ACTIVITIES` type, since that implies assignee in the absence of one. Other tweak is to skip scoring / updating the `SeerAgentRun` if nothing's actually changing: now that many activities may generate a truthy return for `_ground_truth_updates()`, we should actually confirm there's new information before running the rest of the scoring/update flow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes the biggest current hole in our scoring strategy: we were running Smart Assignment and scoring against "ground truths" that came from automated assignments that the Smart Assignment agent can read:
For example, an ownership rule would trigger auto-assignment of an issue to a user/team. This would trigger a Smart Assignment run, assuming that assignment as ground truth. The SA agent would look at the ownership rule and suggest the user (or user from the team), then we would check if the suggestion matched the assignment -> we got it correct! This isn't a fair test.
So we collect the set of
ActivityIntegrations that indicate automated assignment, and a) don't run Smart Assignment on these assignments, and b) don't consider these assignments as sources of ground truth during scoring.