openvibers/openvibe.host Commits

alerts: page when a GPU instance cannot be given back (#303)

OP OpenVibers committed 7 days ago · 27959e5810 · 1 file, +9 −0

OpenVibe.AI's GPU provisioner rents a marketplace GPU, writes its row before the
rental and gives the instance back when it is idle, when its reservation runs out
or when it dies. A destroy the marketplace refuses leaves the instance running and
still being paid for, so the provisioner counts the tries and exposes them as
ai_gpu_destroy_failures. Three tries in a row is the alert: the money runs on until
somebody gives the instance back.

The same way the provider balance alert was added (a gauge in AI, a rule here).

Changes

ddiff --git a/deploy/prometheus/openvibe-rules.yml b/deploy/prometheus/openvibe-rules.yml
iindex 0427d33..68b0c1f 100644
---- a/deploy/prometheus/openvibe-rules.yml
++++ b/deploy/prometheus/openvibe-rules.yml
@@@ -36,6 +36,15 @@ groups:
          labels: { severity: page }
          annotations:
            summary: 'OpenVibe.AI provider {{ $labels.provider }} balance is ${{ printf "%.2f" $value }} (below $2): top it up before AI stops answering'
++      # OpenVibe.AI's GPU provisioner (AI server/gpu/): a marketplace instance that could not be given back
++      # three times in a row is still running and still being paid for. This pages: the money runs on until
++      # somebody gives it back, and the owner set a monthly cap that the instance is inside of.
++      - alert: OpenVibeAiGpuInstanceNotDestroyed
++        expr: ai_gpu_destroy_failures >= 3
++        for: 10m
++        labels: { severity: page }
++        annotations:
++          summary: 'OpenVibe.AI could not give a {{ $labels.marketplace }} GPU instance back ({{ $value }} tries in a row): it is still running and still being paid for'
        - alert: HighErrorRate
          expr: |
            sum by (service) (rate(http_requests_total{status=~"5.."}[10m]))