HejuruUbuyobozi bukurikira
Dual-LLM Icyitegererezo Kurwanya Gutera Byihuse
Tekiniki
Ubuyobozi bwa tekiniki
Guhindura umucamanza wa LLM bisobanura gusuzuma uburyo ibyemezo byayo bigereranywa nu manza zabantu mu manza imwe n’ibizamini.
Umva iki gitaboAI-byavuzwe · ~3 min
Imibare y’amasezerano irashobora gufasha kuranga umucamanza wihariye, ariko amasezerano menshi yonyine ntagaragaza ko rubric ifite agaciro cyangwa ko umucamanza afite ukuri.
Using an LLM as a judge is attractive because it scales far beyond what human review can cover, but a judge that has never been checked against human judgment is essentially untested. Calibration starts by collecting a set of examples that have already been rated by humans - ideally by more than one rater, so inter-human agreement provides context for interpreting judge-human agreement - and then running the LLM judge on those same examples using its intended rubric and prompt. The core comparison typically uses Cohen's kappa, a statistic that measures agreement between two raters while correcting for the agreement expected by pure chance, which matters because raw percent agreement can look deceptively high when one category is very common. A confusion matrix, which cross-tabulates human labels against judge labels for each category, then shows exactly where disagreement is concentrated - for example, a judge might agree with humans well on clear passes and clear failures but disagree heavily in a middle 'borderline' category. That pattern has several possible causes, including unclear rubric boundaries, judge-specific errors, or disagreement among human raters. Inspect examples before deciding whether to revise the rubric, add examples, adjust the judge, or adjudicate the reference labels. A common misconception is that a single round of calibration is sufficient forever; judges can drift as the underlying model, prompt, or the distribution of inputs being judged changes over time, so recalibration on fresh human-labeled samples on a regular cadence is standard practice. Another misconception is that perfect agreement is the bar to hit; human-human agreement is a useful reference for the task’s subjectivity, not a universal target or strict ceiling for judge performance.
Itsinda ryubaka umucamanza wa LLM kugirango ritondere ibisubizo byunganira abakiriya nk '' ifasha 'cyangwa' bidafasha 'ikusanya ingero 200 zanditswemo abantu, ikayobora umucamanza ku ngero zimwe, kandi ikabara kappa ya Cohen kugirango igenzure amasezerano birenze amahirwe.
Injeniyeri asuzuma matrisa yitiranya ibintu maze avumbura umucamanza atondekanya buri gihe ibisubizo byemewe ku mipaka nk '' byiza, 'agaragaza rubricike idasobanura neza imipaka iri hagati yibi byiciro byombi.
Isosiyete ishimangira icyifuzo cy’umucamanza yongeraho ibisubizo bibiri by’ingero kuri buri rwego nyuma yo kubona ko kutumvikana n’abacamanza byibanze ku manota yo hagati aho gutsindira neza cyangwa gutsindwa.
Itsinda ryongeye gukora igenzura rya kalibrasi buri gihembwe ku cyitegererezo gishya cy’amakuru yanditseho abantu, kuva umucamanza wemeranijwe neza n’abantu mu mezi atandatu ashize yatangiye kugenda nyuma y’ivugurura ryihuse ridafite aho rihuriye n’icyitegererezo.
Abacamanza ba LLM bakoreshwa mugusuzuma byinshi byikora, ariko imikorere yabo irashobora guhinduka hamwe na verisiyo yicyitegererezo, ibisobanuro, uburyo bwo gusubiza, hamwe no kuvanga imirimo. Ongera usubiremo iyo kimwe muri ibyo byinjira gihindutse kandi ukomeze gufata abantu. Raporo yamasezerano hamwe nudashidikanya nicyiciro-urwego rwamakosa; ntugasobanure igipimo gihuye nkukuri keretse ibirango byerekanwe hamwe nigishushanyo mbonera cyerekana ibyo bisobanuro. Isubiramo ryabantu rikomeje kuba ingenzi kuburanishwa cyangwa ingaruka zikomeye. Ongera usuzume nyuma ya dataset cyangwa rubric ivugurura kuri buri gusohora kwingenzi.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Guhindura umucamanza wa LLM bisobanura gusuzuma uburyo ibyemezo byayo bigereranywa nu manza zabantu mu manza imwe n’ibizamini. Imibare y’amasezerano irashobora gufasha kuranga umucamanza wihariye, ariko amasezerano menshi yonyine ntagaragaza ko rubric ifite agaciro cyangwa ko umucamanza afite ukuri.
Amasezerano ntarengwa ashobora kugaragara hejuru kubera ko icyiciro kimwe cyiganje; kappa ihindura kubyo.
Matrix yo kwitiranya ibintu isenya amasezerano no kutumvikana ukurikije ibyiciro, byerekana aho umucamanza atandukana nabantu.
Kudahuza kwibanda birashobora kwerekana imbibi zidasobanutse, imyitwarire yihariye yumucamanza, cyangwa ibirango byabantu bidahuye; genzura imanza mbere yo guhitamo gukosorwa.
Icyitegererezo gishya, cyihuta, rubric, cyangwa imirimo ivanze irashobora guhindura imyitwarire yumucamanza, bityo kalibrasi igomba gusubirwamo mugihe ibintu bihindutse.
Amasezerano apima guhuza munsi ya rubrici yatoranijwe hamwe na label yerekana; ntabwo ishyiraho ko rubric ifata ibyagenewe kubaka cyangwa ko ibirango byayo aribyo.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Dual-LLM Icyitegererezo Kurwanya Gutera Byihuse
Tekiniki