Nhungamiro yehunyanzvi

ML Monitoring naPrometheus naGrafana

Prometheus inogona kuunganidza nhamba yenguva-yakatevedzana metrics yakafumurwa nemhando masevhisi, ukuwo Grafana achiona iwo metrics mumadhibhodhi uye zviziviso.

Teerera kugwaro iriAI-yakarondedzerwa · ~3 min

Pa peji ino3 min verenga
  1. Pfupiso
  2. Kudzika Kwakadzika
  3. Kuitwa muNyika yeChokwadi
  4. Ramangwana reML Monitoring naPrometheus naGrafana
  5. Kuzvishandisa
  6. Ramba Uchiongorora
  7. Mibvunzo inowanzo bvunzwa

Pfupiso

Inobatsira ML-sevhisi masaini anosanganisira chiyero chekukumbira, latency, zvikanganiso, kushandisa zviwanikwa uye nekunyatsosarudzwa kufanotaura pfupiso, nekutarisa kune label cardinality uye kuvanzika.

Kudzika Kwakadzika

Prometheus inzira yekutarisa inounganidza nhamba yenguva-yakatevedzana data, kazhinji nekukwenya HTTP metrics endpoint yakafumurwa nesevhisi. Sevha yekufungidzira inogona kutumira zviverengero zvezvikumbiro uye kutadza, geji dzehudzamu hwemutsetse kana zviwanikwa, uye histograms ye latency uye saizi yekukumbira. Malebula anobvumira kuunganidzwa nemaitiro akasungirirwa senge endpoint, status code kana model version. Iyo yekuburitsa fomati uye metric semantics inofanirwa kuenderana saka mibvunzo inoreva zvinotarisirwa nevashandisi. Grafana inogona kubvunza Prometheus uye kuratidza nguva-yakatevedzana mapaneru, matafura uye chenjedzo mumadhibhodhi. Inobatsira ML sevhisi dashboard inosanganisa hutano hwesevhisi ine modhi-inoenderana masaini: chiyero chekukumbira, p50/p95 latency, kukanganisa kodhi, saturation, batch size, GPU memory uye coarse kufanotaura zvipfupiso. Sevhisi uye masaini masaini anopindura mibvunzo yakasiyana. A hutano latency chati hairatidze kuti kufanotaura kunobatsira; kufanotaura kudonha hazvirevi hazvo kuti sevhisi yadzikira. Label cardinality inoda kutarisirwa. Mabhii akadai seyemushandisi ID, mbishi kukurumidza, ID yekukumbira kana isina kuvharwa chinhu ID anogona kugadzira yakakura nhamba yenguva yakatevedzana, kuwedzera ndangariro uye kubvunza mutengo. Sensitive values ​​haifanire kuiswa mumetrics labels. Shandisa matanda kana zviwanikwa zvine zvidzoreso zvekusvika uye sampling yemamiriro epamusoro-soro, uye metrics ezviverengero zvakaunganidzwa uye kugovera. Histograms inotsigira kuunganidzwa kwezviitiko zvese, nepo mutengi-parutivi pfupiso quantiles inogona kusaungana nenzira imwechete. Tsanangura chenjedzo pamusoro pezvinangwa zvinogona kuitwa sevhisi uye sanganisira hwindo renguva kudzivirira kuita ruzha rupfupi. Chengetedza chenjedzo kune varidzi uye vaedze. Yambiro yemhando yemhando inowanzoenderana nemazino anonokerwa kana mametrics eproxy, uye inofanira kudzorwa zvakaparadzana nemaaramu ekukwira. Dashboards inoda mayuniti akajeka, modhi shanduro, mamakisi ekutumira uye zvikumbaridzo zvakanyorwa. Prometheus naGrafana vanopa kuunganidza uye kuona zvikamu; havangotsananguri zvinorehwa ML metrics kana kutsanangura kuti sei fungidziro yachinja. Zvekuvanzika, kuchengetwa uye mitemo yekuwana inoshanda kune yese chiratidzo chinoburitswa nesevhisi.

Kuitwa muNyika yeChokwadi

Iyo modhi API inotumira kunze kwekukumbira counter uye latency histograms nekuguma uye chimiro kirasi. Prometheus anokwenya metrics endpoint, uye Grafana inoratidza mareti uye latency percentiles.

Chikwata chinoteedzera mashandisirwo endangariro yeGPU uye kudzika kwemutsetse padivi pemodhi yekukumbira latency kuti isiyanise saturation yekushandisa kubva kunononoka preprocessing.

Dashboard inoratidza kufanotaura-zvibodzwa kugoverwa neyakaganhurwa modhi-vhezheni label, asi inodzivirira maID ID semetric label nekuti imwe neimwe yakakosha kukosha inogadzira nhevedzano yenguva.

Yambiro inodzima kana mwero yemhosho kana kuti latency ichipfuura chinangwa chesevhisi kwenguva yakatsanangurwa, nepo mushumo wakasiyana unoongorora mhando yemhando kana mavara asvika.

Ramangwana reML Monitoring naPrometheus naGrafana

Monitoring stacks inogona kuvandudza kana ML sevhisi zvikwata zvinobvumirana pane diki yakagovaniswa seti yechikumbiro, latency, sosi uye kufanotaura-kugovera metrics nehuridzi hwakajeka. Madhibhodhi anofanirwa kumaka kuburitswa kwemhando uye kuparadzanisa hutano hwekushanda kubva pakunonoka kuongororwa kwemhando. Ongorora vhoriyamu yekuzivisa uye manyepo kuti zvikwata zvepa-runhare zvigone kupindura zvinobudirira. Hunhu hwepamusoro-soro uye hune hunyoro ndehwemumatanda anodzorwa kana zvitevedzwa panzvimbo pezvinyorwa zvemetric zvakakura. Zvirinani zvirinani zvinoita kuti shanduko yehurongwa ionekwe, nepo kududzira-kwakanangana nedomasi kuchiri kuda modhi uye mamiriro echigadzirwa.

Kuzvishandisa

Nei zvichikosha

  • Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
  • Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
  • Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.

Chenjera

  • Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
  • Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
  • Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.

Matanho anotevera

  1. Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
  2. Benchmark pasi pechokwadi mutoro uye data mamiriro.
  3. Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
  4. Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.

Ramba Uchiongorora

Tsamba yemahara

Tora iyo yemazuva ese AI pfupiso

Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.

Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.

Zviedze iwe pachako

Take the ML Monitoring with Prometheus and Grafana quiz

Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.

Tanga mibvunzo

Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo

Mibvunzo inowanzo bvunzwa

Chii chinonzi ML Monitoring nePrometheus naGrafana?

Prometheus inogona kuunganidza nhamba yenguva-yakatevedzana metrics yakafumurwa nemhando masevhisi, ukuwo Grafana achiona iwo metrics mumadhibhodhi uye zviziviso. Inobatsira ML-sevhisi masaini anosanganisira chiyero chekukumbira, latency, zvikanganiso, kushandisa zviwanikwa uye nekunyatsosarudzwa kufanotaura pfupiso, nekutarisa kune label cardinality uye kuvanzika.

Ko Prometheus anowanzo kuunganidza sei metrics kubva kune modhi sevhisi?

Prometheus anowanzo kukwenya sevhisi yekupedzisira inoburitsa metrics mune inotsigirwa fomati.

Ndeipi metric mhando inokodzera kuwanda kwekukumbira kuverenga?

Macounter anoteedzera monotonically anowedzera mahotareti, ane reset anobatwa sezvo process inotangazve.

Sei uchidzivirira mushandisi ID sePrometheus label?

Imwe neimwe yakasarudzika label seti inogadzira nhevedzano, saka epamusoro-kadhiinality mavara anogona kuremedza kuchengetedza uye mibvunzo.

Chii chinogona kupa latency histogram?

Histograms inorekodha zvakacherechedzwa mumabhaketi uye inotsigira aggregate latency kuongororwa.

Chii chinofanira kuratidzwa neGrafana dashboard yebasa rekufungidzira?

Masaini ekushanda uye emhando pamwe chete anobatsira vashandisi kusiyanisa hutano hwekushandira kubva pakushanduka kwemaitiro.