In T228380#10841868, @KSarabia-WMF wrote:

See More

Is there an example I can follow of how to do this? The documentation has an example for PHP but not for client side.
I'm wondering if this is all I have to do or if I am missing something else.

Calling this one done!

Dashboard looks migrated thanks to @andrea.denisse!

I'm going to call this resolved as we have a few methods available now.

Checking back on this, it seems the situation hasn't improved much.

I imported and modified a dashboard to help us see what's going on. It shows that up until a couple hours ago, we were still at 46k unique timeseries.

The "generated" field is is now calculated from meta.dt - age and stored.

Now that graphite is fully deprecated, I don't think we need this library any more.

In T228380#10775566, @fgiunchedi wrote:

Yes in pushgateway you have a "grouping key" say for example job=foo and then can replace all metrics and their labels pushed under that grouping key.

Would statsd-exporter TTL help in this case to avoid metrics lingering around ?

In T391532#10768362, @Lucas_Werkmeister_WMDE wrote:

The new query GUI version got merged and deployed, and (in a new browser profile) I can see requests being sent to e.g. https://www.wikidata.org/beacon/stats?wikibase_queryService_ui_app_init_total:1|c, but so far the metrics aren’t showing up in Thanos or Grafana…

In T228380#10774463, @Krinkle wrote:

In T388540#10774449, @Krinkle wrote:

Is there a best practice for how to query these correctly such that when multiple are found, the correct/most recent is returned for any given interval point?

The fields explosion event that prompted this task substantially slowed ingest across all topics. Given that, I do not think splitting indexes on the OpenSearch side would have isolated the problem logs from the rest of the log streams.

Another option is partitioning the data into more indexes to reduce index size.

codesearch mw.track counters
codesearch mw.track timers

The feature has landed. Thanks, all!

Update: we saw immediate ingest improvement after removing the out_request and outRequest fields which were generated by mobileapps. We should watch for more inexplicable dips in throughput in the coming days if this was an incomplete mitigation.

In T228380#10745825, @KSarabia-WMF wrote:

Hi all, quick question as we address this Graphite migration work:
Can someone confirm whether the following metric families have already been instrumented and are available in Prometheus?

PagePreviewsApiResponse.*
PagePreviewsPreviewShow.*
These power several Web Team dashboard widgets (e.g., API response time, TTP, preview count, etc.), and we're trying to determine if this data is already flowing into Prometheus, or if we need to reinstrument it ourselves.

Appreciate any guidance—thanks so much!

cc: @ssingh @Jdrewniak

In T348796#10745299, @cscott wrote:

Is this task stalled? Parsoid is having trouble migrating its byte-count metrics to prometheus so this is a blocker for us at this time. I note that @Michael's blocker seems to have been solved by adding bucket support to the JavaScript side in mw.track, but we really need PHP-side support.

@fgiunchedi noted

Affected dashboards: https://grafana.wikimedia.org/d/K6DEOo5Ik/grafana-graphite-datasource-utilization

The final step in the migration is to update the dashboards using these metrics to use the Prometheus metrics provided by the Thanos datasource. I found these dashboards still referencing Graphite for this data:

• Wikidata Quality Constraints (WBQC) https://grafana.wikimedia.org/d/000000344
• Ladsgroup-test https://grafana.wikimedia.org/d/000000378

A couple points for awareness as you plan the project:

We have alerting now and we know a simple restart of statsv brings it back. Optimistically closing.

In T390596#10705375, @ecarg wrote:

I'm wondering if there are fields that SRE is additionally dropping before Logstash emission

We do a lot of filtering on the legacy pipeline for fields that cause problems. All filtering we do there is manual and changes regularly.

We've rolled out a logstash filter to check for name KafkaSSE and to cast the assignments field into a string. This can be undone when it is no longer needed.

Checking back today, istio-ingressgateway is logging about 1000/logs/sec these days.

I think it's because the object is the second argument rather than the measurement.

Log volume is down quite a bit since raising the production-ratelimit log level filter. Will check back in on this in a few days to re-evaluate in case there are other applications doing the same thing.

First suspect is kubernetes.container_name:"production-ratelimit" generating debug logs at around 1500/sec.

Dashboards migrated

From around the same time:

Mar 19 11:45:21 webperf1003 python3[3604232]: Process Process-1:
Mar 19 11:45:21 webperf1003 python3[3604232]: Traceback (most recent call last):
Mar 19 11:45:21 webperf1003 python3[3604232]:   File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Mar 19 11:45:21 webperf1003 python3[3604232]:     self.run()
Mar 19 11:45:21 webperf1003 python3[3604232]:   File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
Mar 19 11:45:21 webperf1003 python3[3604232]:     self._target(*self._args, **self._kwargs)
Mar 19 11:45:21 webperf1003 python3[3604232]:   File "/srv/deployment/statsv/statsv/statsv.py", line 268, in process_queue
Mar 19 11:45:21 webperf1003 python3[3604232]:     emit(sock, statsd_addr, statsd_message)
Mar 19 11:45:21 webperf1003 python3[3604232]:   File "/srv/deployment/statsv/statsv/statsv.py", line 195, in emit
Mar 19 11:45:21 webperf1003 python3[3604232]:     sock.sendto(payload.encode('utf-8'), addr)
Mar 19 11:45:21 webperf1003 python3[3604232]: socket.gaierror: [Errno -2] Name or service not known

It seems the statsv process wedged itself. After restarting the process, metrics are now flowing again.

Config deployed!

Dashboards migrated

Dashboards migrated

Dashboards migrated.

Linking this task here in case it helps with the investigation: T385058: logstash.rb uses deprecated Socket.gethostbyname

Restbase metrics are now being ingested by Prometheus.

In T359239#10580976, @andrea.denisse wrote:

Hi @colewhite,

Should we remove the metrics from MWTrack?

Reference: MWTrack usage in ContentTranslation

Let me know your thoughts.

The change is deployed. I'm not sure there is anything left to do, but optimistically resolving.

The rest of the metrics on the dashboard appear to be generated by Airflow.

In T381607#10531933, @Sgs wrote:

I'm trying to build a panel ( Echo notifications ) that allows to filter by wiki and "user types" and I'm failing to make a query that would work also for the "All" option. I've customized the all option to be of value .* and used the regex operator =~