Page MenuHomePhabricator
Create Task
Maniphest T343020

Converting MediaWiki Metrics to StatsLib
Open, Needs TriagePublic
Actions

Description

See More

Migration Status Dashboard

This task aims to convert MediaWiki statistics end-to-end to a Prometheus-capable metrics library and port/migrate all existing metrics to this new implementation.

The hypothesis driving this project is that by aggregating all MediaWiki statistics in a single modern system, we can improve the developer workflow and assist with troubleshooting by reducing tool fragmentation. Success will be measured by ensuring that all metrics produced by MediaWiki are exported to Prometheus and supported by the observability team.

This project was initiated in follow up to an RFC (T249164: RFC: Better interface for generating metrics in MediaWiki) and a desire to sunset our previous generation metrics collection stack (Graphite - Wikitech && T228380: Tech debt: sunsetting of Graphite).

The transition to Prometheus metrics from Graphite offers several advantages, including the ability to tag and label data, adding multidimensionality to these metrics, and allowing us to stay on a currently supported system and decommission older tech.

Related to: T240685: MediaWiki Prometheus support

Details

Show related patches Customize query in gerrit

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
OpenNoneT343020 Converting MediaWiki Metrics to StatsLib
InvalidcolewhiteT343021 Deploy prometheus-statsd-exporter to Test Env
ResolvedcolewhiteT343022 Configure MediaWiki to use Stats lib in Test Env
ResolvedherronT343023 Deploy StatsD Exporter to production
ResolvedcolewhiteT343024 Configure MediaWiki to use new StatsLib in production
ResolvedJoeT343025 Identify path forward for k8s deployment of prometheus-statsd-exporter
ResolvedherronT343026 Configure Prometheus to scrape MW metrics from statsd-exporter
DuplicateNoneT343028 Audit & convert stats for mediawiki core
DuplicateNoneT343029 Audit & convert stats for mediawiki extensions
DuplicateNoneT343045 Audit & convert stats for mediawiki modules
ResolvedherronT350591 Audit legacy mediawiki stats used in production dashboards
ResolvedfgiunchediT350597 Audit and prioritize metrics for conversion to statslib that are used for graphite-based alerting
ResolvedcolewhiteT350638 Track MediaWiki stats usage
ResolvedKrinkleT355837 Add Prometheus support to statsd.js via mw.track()
ResolvedDAlangi_WMFT356062 Migrate ObjectCache in MW core to StatsLib
ResolvedDAlangi_WMFT356810 Migrate MediaWiki.timing.editResponseTime.* to statslib
ResolveddcausseT359033 EPIC: Convert CirrusSearch metrics to statslib
DuplicateDAlangi_WMFT360288 Migrate LoginSignupSpecialPage to use StatsLib
ResolvedDAlangi_WMFT360290 Migrate DeferredUpdates to use StatsLib
DuplicateNoneT360457 Migrate CentralAuth to use StatsLib
DuplicateDAlangi_WMFT360458 Migrate CentralAuth to use StatsLib
ResolvedcolewhiteT368740 Simplify asserting expected stats in unit / integration tests
ResolvedcolewhiteT377506 Migrate MediaWiki.extension.UrlShortener.* to statslib
OpenNoneT382549 (mw.track) Migrate MediaWiki.edit_recovery.* to Prometheus
ResolvedaaronT392016 Migrate remaining rdbms code to StatsLib
· · ·

Event Timeline

lmata created this task.Jul 28 2023, 5:45 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 28 2023, 5:45 PM
lmata added a subscriber: colewhite.Jul 28 2023, 5:45 PM
lmata added a project: Observability-Metrics.Jul 28 2023, 5:56 PM
Joe changed the status of subtask T343025: Identify path forward for k8s deployment of prometheus-statsd-exporter from Open to In Progress.Sep 21 2023, 4:04 PM
lmata edited projects, added SRE Observability (FY2023/2024-Q2); removed SRE Observability (FY2023/2024-Q1).Oct 9 2023, 4:28 PM
lmata moved this task from Inbox to Up next on the SRE Observability (FY2023/2024-Q2) board.
Joe closed subtask T343025: Identify path forward for k8s deployment of prometheus-statsd-exporter as Resolved.Oct 25 2023, 6:33 AM
colewhite closed subtask T343021: Deploy prometheus-statsd-exporter to Test Env as Invalid.Oct 27 2023, 11:05 PM
herron closed subtask T343023: Deploy StatsD Exporter to production as Resolved.Oct 31 2023, 2:18 PM
herron closed subtask T343026: Configure Prometheus to scrape MW metrics from statsd-exporter as Resolved.
colewhite closed subtask T343022: Configure MediaWiki to use Stats lib in Test Env as Resolved.Dec 14 2023, 12:42 AM
DAlangi_WMF subscribed.Jan 10 2024, 1:11 PM
lmata closed subtask T343024: Configure MediaWiki to use new StatsLib in production as Resolved.Jan 12 2024, 5:41 PM
lmata moved this task from Up next to Epics In Progress on the SRE Observability (FY2023/2024-Q2) board.Jan 12 2024, 5:46 PM
lmata edited projects, added SRE Observability (FY2023/2024-Q3); removed SRE Observability (FY2023/2024-Q2).
colewhite added a subtask: T350638: Track MediaWiki stats usage.Jan 16 2024, 11:00 PM
colewhite closed subtask T350638: Track MediaWiki stats usage as Resolved.
colewhite updated the task description. (Show Details)
herron closed subtask T350591: Audit legacy mediawiki stats used in production dashboards as Resolved.Jan 17 2024, 8:47 PM
colewhite added a subtask: T355837: Add Prometheus support to statsd.js via mw.track().Jan 25 2024, 12:30 AM
DAlangi_WMF changed the status of subtask T356062: Migrate ObjectCache in MW core to StatsLib from Open to In Progress.Feb 19 2024, 1:37 PM
DAlangi_WMF added a subtask: T356810: Migrate MediaWiki.timing.editResponseTime.* to statslib.Feb 27 2024, 10:43 AM
bd808 updated the task description. (Show Details)Feb 27 2024, 5:38 PM
dcausse added a subtask: T359033: EPIC: Convert CirrusSearch metrics to statslib.Mar 4 2024, 10:05 AM
DAlangi_WMF closed subtask T356062: Migrate ObjectCache in MW core to StatsLib as Resolved.Mar 8 2024, 6:38 PM
DAlangi_WMF added a subtask: T360288: Migrate LoginSignupSpecialPage to use StatsLib.Mar 18 2024, 9:11 AM
DAlangi_WMF changed the status of subtask T360288: Migrate LoginSignupSpecialPage to use StatsLib from Open to In Progress.
DAlangi_WMF changed the status of subtask T360290: Migrate DeferredUpdates to use StatsLib from Open to In Progress.Mar 18 2024, 9:29 AM
DAlangi_WMF changed the status of subtask T360458: Migrate CentralAuth to use StatsLib from Open to In Progress.Mar 19 2024, 6:32 PM
colewhite edited projects, added SRE Observability (FY2023/2024-Q4); removed SRE Observability (FY2023/2024-Q3).Apr 3 2024, 10:47 PM
colewhite moved this task from Inbox to Epics In Progress on the SRE Observability (FY2023/2024-Q4) board.Apr 3 2024, 11:09 PM
lmata mentioned this in T240685: MediaWiki Prometheus support.Apr 4 2024, 8:31 PM
DAlangi_WMF closed subtask T360290: Migrate DeferredUpdates to use StatsLib as Resolved.Apr 15 2024, 10:32 AM
colewhite added a subtask: T363585: (mw.track) Migrate MediaWiki.wikibase.client.databridge to statslib.Apr 26 2024, 3:43 PM
colewhite added a subtask: T363586: Migrate MediaWiki.*.wikibase.client.scribunto.* to statslib.
colewhite removed a subtask: T363585: (mw.track) Migrate MediaWiki.wikibase.client.databridge to statslib.
colewhite removed a subtask: T363586: Migrate MediaWiki.*.wikibase.client.scribunto.* to statslib.
lmata mentioned this in T228380: Tech debt: sunsetting of Graphite.May 20 2024, 3:54 PM
Lucas_Werkmeister_WMDE mentioned this in T326607: Future of liuggio/statsd-php-client?.Jun 24 2024, 9:56 AM
colewhite edited projects, added SRE Observability (FY2024/2025-Q1); removed SRE Observability (FY2023/2024-Q4).Jul 2 2024, 5:55 PM
Michael subscribed.Jul 11 2024, 1:43 PM
Nemo_bis awarded a token.Jul 12 2024, 6:56 AM
Nemo_bis subscribed.Jul 13 2024, 6:58 PM
fgiunchedi closed subtask T350597: Audit and prioritize metrics for conversion to statslib that are used for graphite-based alerting as Resolved.Sep 10 2024, 10:51 AM
lmata moved this task from FY2024/2025-Q1 to FY2024/2025-Q2 on the SRE Observability board.Oct 3 2024, 2:12 AM
lmata edited projects, added SRE Observability (FY2024/2025-Q2); removed SRE Observability (FY2024/2025-Q1).
colewhite added a subtask: T377506: Migrate MediaWiki.extension.UrlShortener.* to statslib.Oct 17 2024, 8:21 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q2) board.Nov 5 2024, 5:07 PM
colewhite closed subtask T377506: Migrate MediaWiki.extension.UrlShortener.* to statslib as Resolved.Nov 8 2024, 11:07 PM
gerritbot added a comment.Nov 22 2024, 1:43 AM

Change #1094089 had a related patch set uploaded (by Aaron Schulz; author: Aaron Schulz):

[mediawiki/core@master] rdbms: convert LBFactory/LoadBalancer to using StatsFactory

https://gerrit.wikimedia.org/r/1094089

gerritbot added a project: Patch-For-Review.Nov 22 2024, 1:43 AM
andrea.denisse awarded a token.Nov 26 2024, 5:51 PM
Krinkle closed subtask T355837: Add Prometheus support to statsd.js via mw.track() as Resolved.Dec 20 2024, 12:42 AM
lmata edited projects, added SRE Observability (FY2024/2025-Q3); removed SRE Observability (FY2024/2025-Q2).Jan 14 2025, 3:56 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q3) board.
colewhite closed subtask T368740: Simplify asserting expected stats in unit / integration tests as Resolved.Feb 5 2025, 10:04 PM
dcausse closed subtask T359033: EPIC: Convert CirrusSearch metrics to statslib as Resolved.Feb 11 2025, 2:36 PM
gerritbot added a comment.Mar 7 2025, 11:41 PM

Change #1125545 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/Chart@master] add Prometheus-compatible counters to render(Start|End) metrics

https://gerrit.wikimedia.org/r/1125545

gerritbot added a comment.Mar 11 2025, 1:55 AM

Change #1125545 merged by jenkins-bot:

[mediawiki/extensions/Chart@master] add Prometheus-compatible counters to render(Start|End) metrics

https://gerrit.wikimedia.org/r/1125545

colewhite mentioned this in rECRT21964e09780d: add Prometheus-compatible counters to render(Start|End) metrics.Mar 11 2025, 1:55 AM
ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.21; 2025-03-18).Mar 11 2025, 2:00 AM
gerritbot added a comment.Mar 25 2025, 10:41 PM

Change #1131107 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/WikimediaEvents@master] track mw.js.deprecate metrics in Prometheus

https://gerrit.wikimedia.org/r/1131107

gerritbot added a comment.Mar 27 2025, 8:11 PM

Change #1131107 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] track mw.js.deprecate metrics in Prometheus

https://gerrit.wikimedia.org/r/1131107

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.23; 2025-04-01); removed MW-1.44-notes (1.44.0-wmf.21; 2025-03-18).Mar 27 2025, 9:01 PM
lmata edited projects, added SRE Observability (FY2024/2025-Q4); removed SRE Observability (FY2024/2025-Q3).Apr 1 2025, 12:04 PM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2024/2025-Q4) board.
Krinkle mentioned this in T383563: mw.track: support for histogram metrics.Apr 10 2025, 2:48 PM
gerritbot added a comment.Apr 25 2025, 10:37 PM

Change #1139134 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/VisualEditor@master] Add prometheus-compatible mw.track calls

https://gerrit.wikimedia.org/r/1139134

gerritbot added a comment.Apr 25 2025, 10:39 PM

Change #1139135 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/WikimediaEvents@master] searchSli: Add prometheus-compatible mw.track calls

https://gerrit.wikimedia.org/r/1139135

gerritbot added a comment.Apr 25 2025, 10:40 PM

Change #1139136 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/skins/Vector@master] Add prometheus-compatible mw.track call

https://gerrit.wikimedia.org/r/1139136

gerritbot added a comment.Apr 25 2025, 10:41 PM

Change #1139137 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/MediaSearch@master] Add prometheus-compatible mw.track call

https://gerrit.wikimedia.org/r/1139137

gerritbot added a comment.Apr 25 2025, 10:42 PM

Change #1139138 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/IPInfo@master] Add prometheus-compatible mw.track calls

https://gerrit.wikimedia.org/r/1139138

gerritbot added a comment.Apr 25 2025, 10:43 PM

Change #1139139 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/SearchVue@master] Add prometheus-compatible mw.track call

https://gerrit.wikimedia.org/r/1139139

gerritbot added a comment.Apr 28 2025, 8:11 AM

Change #1139135 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] searchSli: Add prometheus-compatible mw.track calls

https://gerrit.wikimedia.org/r/1139135

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.27; 2025-04-29); removed MW-1.44-notes (1.44.0-wmf.23; 2025-04-01).Apr 28 2025, 9:00 AM
gerritbot added a comment.Apr 28 2025, 11:24 AM

Change #1139138 merged by jenkins-bot:

[mediawiki/extensions/IPInfo@master] Add prometheus-compatible mw.track calls

https://gerrit.wikimedia.org/r/1139138

gerritbot added a comment.Tue, Apr 29, 6:31 PM

Change #1139917 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/IPInfo@master] add missing required 'mediawiki_' prefix

https://gerrit.wikimedia.org/r/1139917

gerritbot added a comment.Tue, Apr 29, 6:32 PM

Change #1139918 had a related patch set uploaded (by Cwhite; author: Cwhite):

[mediawiki/extensions/WikimediaEvents@master] add missing required 'mediawiki_' prefix

https://gerrit.wikimedia.org/r/1139918

gerritbot added a comment.Tue, Apr 29, 8:39 PM

Change #1139136 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Add prometheus-compatible mw.track call

https://gerrit.wikimedia.org/r/1139136

ReleaseTaggerBot edited projects, added MW-1.44-notes (1.44.0-wmf.28; 2025-05-06); removed MW-1.44-notes (1.44.0-wmf.27; 2025-04-29).Tue, Apr 29, 9:00 PM
gerritbot added a comment.Tue, Apr 29, 9:59 PM

Change #1139918 merged by jenkins-bot:

[mediawiki/extensions/WikimediaEvents@master] add missing required 'mediawiki_' prefix

https://gerrit.wikimedia.org/r/1139918

gerritbot added a comment.Wed, Apr 30, 9:25 PM

Change #1139917 merged by jenkins-bot:

[mediawiki/extensions/IPInfo@master] add missing required 'mediawiki_' prefix

https://gerrit.wikimedia.org/r/1139917

Sgs mentioned this in T393243: "Invalid stat name WikimediaEvents_Search_AutocompleteResults_seconds" Console error when typing in Search box.Tue, May 6, 2:23 PM
Mvolz added a project: Editing-team (Kanban Board).Thu, May 8, 9:28 AM
Mvolz moved this task from Incoming to Code Review on the Editing-team (Kanban Board) board.
VPuffetMichel added a project: Essential-Work.Thu, May 8, 1:58 PM
HCoplin-WMF closed subtask T392016: Migrate remaining rdbms code to StatsLib as Resolved.Fri, May 9, 6:10 PM
Krinkle mentioned this in T394956: Restore support for gauge metrics from MediaWiki PHP (post-Prometheus migration).Wed, May 21, 9:45 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits