HejuruUbuyobozi bukurikira
LLM Isuzuma Ibikorwa: Promptfoo na DeepEval
Tekiniki
Ubuyobozi bwa tekiniki
LLM eval dataset ni igenamigambi ryinjiza ryuzuye, buri kimwe cyahujwe nigisubizo cya zahabu, ibimenyetso bifatika cyangwa rubricike yo gutondekanya amanota, ko ukoresha ibyifuzo byawe kugirango upime ubuziranenge no gufata ibyasubiye inyuma.
Umva iki gitaboAI-byavuzwe · ~3 min
Nibyingenzi kuko guhinduka kubibazo, icyitegererezo cyangwa kugarura bishobora kunoza imyitwarire imwe mugihe uciye indi. Hatariho ikizamini gihamye, amakipe arangiza gusuzuma impinduka ukoresheje ingero nke zatoranijwe.
Start with real usage. Logs, support tickets, search queries and user interviews show what people actually ask, including the messy phrasing that invented examples miss. Before launch, have domain experts write inputs. You can add synthetic cases generated by an LLM, as long as a person reviews them and removes unrealistic or duplicate items. Next, define what correct means for each case. Some tasks have one right answer, such as classification, extraction or a factual lookup. For these, store a golden answer that can be compared exactly or after light normalization. For open-ended tasks, write a rubric of specific, checkable criteria, such as 'mentions the refund deadline' or 'does not recommend a competitor'. Vague criteria like 'is helpful' lead to inconsistent grading, whether the grader is a person or a judge model. Then cover the range of real inputs on purpose. Tag cases by intent, difficulty, language and user type, and check that the important groups are represented. Add edge cases and adversarial inputs: - ambiguous questions - requests with missing information - very long inputs - out-of-scope requests that should be declined - prompt-injection attempts Keep a regression slice made of past failures. Size the set to the decision it has to support. A few dozen well-chosen cases can reveal large problems early. Detecting small differences reliably takes more. With 100 pass/fail cases and a pass rate near 80 percent, the margin of error is roughly plus or minus 8 percentage points at 95 percent confidence. A two-point improvement is indistinguishable from noise at that size. Report results for each slice, not only an overall average. A common mistake is to treat the dataset as finished. Products and users change, so refresh it from new logs. Keep a held-out portion that you never tune against, so your prompts do not overfit to the test.
Itsinda ryunganira ryohereza amatike 300 nyayo hamwe namakuru yakuweho, akandika ibirango bikwiye kuri buri, kandi agatanga amanota yose mashya kubarwanya.
Kubwincamake yinama aho amagambo menshi arikose, itsinda ryandika rubric: ikubiyemo ibyemezo bitatu byingenzi, ntisaba ibirego bidashyigikiwe, iguma munsi yamagambo 150.
Itsinda ryongeramo imanza: ibyinjijwe ubusa, ikibazo mu cyesipanyoli, inyandiko ikubiyemo kugerageza-guterwa vuba, hamwe no gusaba bidashoboka umufasha agomba kwanga.
Iyo umukoresha atanze igisubizo kibi mubikorwa, urubanza rujya muri dataset hamwe nigisubizo nyacyo, bityo gukosora bikageragezwa kuva icyo gihe.
Igikoresho cyo gusuzuma cyakuze vuba. Gufungura-isoko yimikorere hamwe na platform yakiriwe noneho icunga imibare, ikoresha amanota kandi ikurikirana ibisubizo mugihe. Gukora amakuru yubukorikori birashoboka ko azakomeza gutera imbere, bigatuma imanza zidasanzwe zihendutse gutwikira. Ntishobora gusimbuza umukoresha nyawo inyongeramusaruro nka ankeri kubyingenzi. Mugihe porogaramu ziba abakozi benshi, eval igenda irushaho gushyiramo intambwe-ntambwe yimirimo ikurikirana kubisubizo byanyuma hamwe nintambwe zafashwe munzira. Indero yibanze ihinduka gahoro gahoro: gusobanura intsinzi neza, gutwikira inyongeramusaruro nyayo, koresha imanza zihagije kugirango wizere imibare, kandi uhindure buri gihe umusaruro.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
LLM eval dataset ni igenamigambi ryinjiza ryuzuye, buri kimwe cyahujwe nigisubizo cya zahabu, ibimenyetso bifatika cyangwa rubricike yo gutondekanya amanota, ko ukoresha ibyifuzo byawe kugirango upime ubuziranenge no gufata ibyasubiye inyuma. Nibyingenzi kuko guhinduka kubibazo, icyitegererezo cyangwa kugarura bishobora kunoza imyitwarire imwe mugihe uciye indi. Hatariho ikizamini gihamye, amakipe arangiza gusuzuma impinduka ukoresheje ingero nke zatoranijwe.
Imikoreshereze nyayo ifata ibyo abantu babaza mubyukuri, harimo interuro mbi. Imanza ya sintetike ninyongera yasubiwemo, ntabwo ari umusimbura.
Gufungura-kurangiza ibisubizo nkincamake ntibishobora guhuzwa neza. Ibipimo byihariye, bisuzumwa reka reka abanyeshuri babacire urubanza buri gihe.
Ibipimo byiza birasobanutse kandi birashobora kugenzurwa. Ibipimo bidasobanutse nka 'bifasha' biganisha ku manota adahuye n'abantu hamwe n'abacamanza b'icyitegererezo.
Kuri 95 ku ijana icyizere margin ni amanota 8 kuri ubu bunini. Itandukaniro rito hagati ya verisiyo ntishobora kwizerwa nta manza nyinshi.
Ongeraho buri kintu cyananiranye kuri dataset bivuze ko impinduka zanyuma zidashobora kongera gucecekesha ikosa rimwe.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
LLM Isuzuma Ibikorwa: Promptfoo na DeepEval
Tekiniki