fix: reduce false positives for non-ASCII (Chinese) usernames + add CSDN site - #2876
Merged
Merged
Conversation
Closes soxoj#2665. Every site results entry flows through make_site_result (checking.py:788), which initializes results_site = {} and then unconditionally populates it (site/username/keywords/parsing_enabled/url_main/cookies up front, checker at the end, plus status or url_user+future on every branch). It has no return path that yields an empty dict. check_site_for_username wraps that result and likewise never returns a falsy entry, and maigret.py does not construct site-result dicts by hand. So the four downstream 'if not dictionary: continue' guards (all tagged '# TODO: fix no site data issue') were dead code — the entries they guarded against cannot occur. Removed: - maigret/maigret.py:101 (extract_ids_from_results) - maigret/report.py:151 — 'if not dictionary or dictionary.get("is_similar")' trimmed to just the is_similar check (real logic) - maigret/report.py:467 (extended-report builder) - maigret/report.py:609 (generate_txt_report) - maigret/report.py:627 — 'if not site_result or not site_result.get("status")' trimmed to just the status check (real logic) - maigret/report.py:687 (generate_json_report) plus the four '# TODO: fix no site data issue' comments. No behavior change for well-formed inputs; downstream .get() calls are safe because entries are always populated dicts. 292 tests pass.
- soxoj#2633: When searching non-ASCII usernames, skip presence detection if the response body doesn't contain the username at all, preventing false positives on sites that return generic error pages. - soxoj#2634: Add CSDN (blog.csdn.net) — major Chinese dev blog with 40M+ users, alexa rank soxoj#45 in China.
Owner
|
Thank you so much for this PR — it's a really thoughtful contribution! Two small asks before merge: (1) CSDN needs |
…tive fix Addresses PR soxoj#2876 review feedback from soxoj: - Add tls_fingerprint to CSDN protection (curl_cffi required) - Add 2 test cases for soxoj#2633 non-ASCII username false positive fix: - non-ASCII username not in response → no false CLAIMED - non-ASCII username in response → normal CLAIMED behavior
Contributor
Author
|
Addressed both review points:
All 55 tests pass (53 existing + 2 new). |
Owner
|
Amazing, thanks! Great contribution! 🥰 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes to improve Chinese/Chinese username support in maigret:
1. False-positive fix for non-ASCII usernames (#2633)
When searching a non-ASCII (e.g. Chinese) username, many sites return a generic 200 page without any user-specific content. The existing
messagecheck type defaultsis_presense_detected = Truewhen nopresenseStrsare defined, causing false CLAIMED results.Fix: If the username contains any non-ASCII character AND the response body does not contain the username string at all, skip presence detection. This is a conservative heuristic: it only triggers when we can prove the page has no relation to the searched user.
2. Add CSDN (blog.csdn.net) — Chinese dev blog (#2634)
CSDN is the largest Chinese developer blog platform with 40M+ registered users and Alexa rank #45 in China. Added with
checkType: messageusing thedata-username="attribute as presence marker, tagged["cn", "coding", "blog"].Testing
test_checking.py: 53/53 passed ✅data-username="csdnnews", nonexistent returns 521 (WAF, handled by error detection → UNKNOWN)Closes #2633
Addresses #2634