Skip to content

feat(smart-assignment): Increase daily caps, sample non-Seer triggers at 10% - #121680

Open
hobzcalvin wants to merge 1 commit into
masterfrom
sa-sampling
Open

feat(smart-assignment): Increase daily caps, sample non-Seer triggers at 10%#121680
hobzcalvin wants to merge 1 commit into
masterfrom
sa-sampling

Conversation

@hobzcalvin

@hobzcalvin hobzcalvin commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

(I'm not sure how our feat(thingy) nomenclature is defined, but I think smart-assignment deserves its own?!)

In preparation for rolling Smart Assignment out to all ~2400 Seer customers (from the current ~400), I did some math to balance daily spend and statistical significance of our scored runs. Basically, we always want to run on SEER_* activities that need a suggested assignee (and thus aren't likely to give us a ground truth to score against), but for SET_RESOLVED_* and ASSIGNED activities (which give us a guaranteed ground truth but are only used for internal scoring) we can run them about 10% of the time and still get the data we need.

This will cost us about $100/day at current costs per run ($0.10), but the global cap increase will set the max spend per day at about $200. An increase in the per-org cap also just ensures we're preventing one customer breaking SA for everyone else, but we should never hit that.

I'll be adding datadog monitors for when we hit the daily caps--turns out we WERE hitting them, before #121368 significantly reduced the number of (useless) SA runs. This change will further reduce our number of daily runs by only running on 10% of resolved/assigned activities, though we'll counteract that by 6x-ing the number of enabled orgs.

Resolves ISWF-3269

@hobzcalvin
hobzcalvin requested review from a team as code owners August 10, 2026 22:57
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Aug 10, 2026
@linear-code

linear-code Bot commented Aug 10, 2026

Copy link
Copy Markdown

ISWF-3269

saponifi3d
saponifi3d approved these changes Aug 10, 2026
View reviewed changes

@saponifi3d saponifi3d left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - the getsentry tests might be failing because of a busted master, can try updating.

if that doesn't work, then something in this pr is likely effecting the unit tests there and we'll need to update getsentry as well.

@github-actions

github-actions Bot commented Aug 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Backend Test Failures

Failures on 9a13177 in this run:

tests/sentry/replays/endpoints/test_organization_replay_count.py::OrganizationReplayCountEndpointTest::test_eap_spans_transaction — log 
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/replays/endpoints/test_organization_replay_count.py:796: in test_eap_spans_transaction
    assert response.data == {"/api/foo/": 2}
E   AssertionError: assert defaultdict(<...api/foo/': 1}) == {'/api/foo/': 2}
E     
E     Differing items:
E     �[0m{�[33m'�[39;49;00m�[33m/api/foo/�[39;49;00m�[33m'�[39;49;00m: �[94m1�[39;49;00m}�[90m�[39;49;00m != �[0m{�[33m'�[39;49;00m�[33m/api/foo/�[39;49;00m�[33m'�[39;49;00m: �[94m2�[39;49;00m}�[90m�[39;49;00m
E     
E     Full diff:
E     �[0m�[91m- {�[39;49;00m�[90m�[39;49;00m
E     �[92m+ defaultdict(<class 'int'>, {�[39;49;00m�[90m�[39;49;00m
E     �[91m-     '/api/foo/': 2,�[39;49;00m�[90m�[39;49;00m
E     ?                  ^�[90m�[39;49;00m
E     �[92m+     '/api/foo/': 1,�[39;49;00m�[90m�[39;49;00m
E     ?                  ^�[90m�[39;49;00m
E     �[91m- }�[39;49;00m�[90m�[39;49;00m
E     �[92m+ })�[39;49;00m�[90m�[39;49;00m
tests/sentry/seer/anomaly_detection/test_store_data.py::AnomalyDetectionStoreDataTest::test_anomaly_detection_fetch_historical_data_eap_spans — log 
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/seer/anomaly_detection/test_store_data.py:177: in test_anomaly_detection_fetch_historical_data_eap_spans
    assert {"time": int(expected_ts_1), "count(span.duration)": 1} in result.data.get(
E   AssertionError: assert {'count(span.duration)': 1, 'time': 1785803400} in [{'time': 1783557000}, {'time': 1783558800}, {'time': 1783560600}, {'time': 1783562400}, {'time': 1783564200}, {'time': 1783566000}, ...]
E    +  where [{'time': 1783557000}, {'time': 1783558800}, {'time': 1783560600}, {'time': 1783562400}, {'time': 1783564200}, {'time': 1783566000}, ...] = <built-in method get of dict object at 0x7f66fec29400>('data')
E    +    where <built-in method get of dict object at 0x7f66fec29400> = {'data': [{'time': 1783557000}, {'time': 1783558800}, {'time': 1783560600}, {'time': 1783562400}, {'time': 1783564200}..., {'time': 1785974400}, {'time': 1785976200}, {'time': 1785978000}], confidence=[], sampling_rate=[], sample_count=[])}.get
E    +      where {'data': [{'time': 1783557000}, {'time': 1783558800}, {'time': 1783560600}, {'time': 1783562400}, {'time': 1783564200}..., {'time': 1785974400}, {'time': 1785976200}, {'time': 1785978000}], confidence=[], sampling_rate=[], sample_count=[])} = SnubaTSResult(data={'data': [{'time': 1783557000}, {'time': 1783558800}, {'time': 1783560600}, {'time': 1783562400}, {...'bytes_scanned': 0}}, start=datetime.datetime(2026, 7, 9, 0, 30), end=datetime.datetime(2026, 8, 6, 1, 0), rollup=1800).data

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@saponifi3d saponifi3d saponifi3d approved these changes
@shayna-ch shayna-ch Awaiting requested review from shayna-ch

Assignees

No one assigned

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@hobzcalvin @saponifi3d