InoteveraGaidhi rinotevera
Kudzosa Metrics: Recall@K, MRR uye nDCG
Tekinoroji
Nhungamiro yehunyanzvi
Kudzosa magwaro evabereki, kunonziwo kutsvagisa kudiki-kusvika-kukuru, kunotsvaga pazvimedu zvidiki, zvemavara chaizvo asi inopa modhi yemutauro chikamu chikuru kana gwaro chikamu chimwe nechimwe chakabva.
Teerera kugwaro iriAI-yakarondedzerwa · ~3 min
Izvo zvine basa nekuti inobvisa yakajairwa chunking kutengeserana-kure: madiki chunks anoenderana nemibvunzo nenzira kwayo, uye mubereki mukuru anopa iyo modhi yakakwana mamiriro ekupindura nemazvo.
Fixed-size chunking inomanikidza kutengeserana. Zvidiki zvidiki, taura mitsara mishoma, inoburitsa zvakapinza: iyo vector inomiririra imwe pfungwa, saka mubvunzo nezve iyo pfungwa inofanana nayo zvakanyanya. Asi chunk yemitsara miviri haiwanzo kuve nemamiriro akakwana kuti modhi ipindure zvakanaka; inogona kusiya tsananguro yendima nhatu kare kana kusarudzika mumutsara unotevera. Makuru machunks anotakura iwo mameseji, asi mabatirwo awo anodzima mazano akati wandei pamwechete, saka kudzoreredza kunoita kushoma uye ndima yakakodzera inogona kuburitswa nechunk inongori pamusoro-musoro. Kutora kudiki-ku-kukuru kunotsemura mabasa maviri. Iyo sisitimu inonongedza matinji evana ekutsvaga, uye mwana wega wega anochengeta chinongedzo kumubereki mukuru: chikamu, peji, kana gwaro rose. Panguva yekubvunza inowana vana vakanaka, yobva yatarisa kumusoro uye inodzosera vabereki vavo. Kana vana vakati wandei vachigovana mubereki mumwe, mubereki anotumirwa kamwe chete, izvo zvinobvisawo zvakapetwa. LangChain's ParentDocumentRetriever inoshandisa izvi nechitoro chevana uye chitoro chemagwaro chakasiyana chevabereki. LlamaIndex inopa mapatani ane hukama: mutsara-hwindo kudzoreredza, iyo inodzosa mutsara unofananidzwa pamwe nenhamba yakatarwa yemitsara yakavakidzana, uye otomatiki-kubatanidza kudzoreredza, iyo inotsiva akawanda akadzoserwa machunks eshizha neakagovaniswa mubereki node kana zvakakwana zvawo zvakaenzana. Maitiro anowanzo kurova yakagadziriswa-saizi chunking pamagwaro akarongwa senge manyoro, zvibvumirano, marongero uye mabhuku ezvinyorwa, uko zvinoreva zvinoenderana nechikamu chakatenderedza. Zvinobatsira zvishoma kana magwaro atove mapfupi, kana kana vabereki vakakura zvekuti vashoma vavo vanofashukira hwindo remukati kana kuviga mhinduro mumagwaro asina basa. Imwe pfungwa isiriyo ndeyekuti uku kuri kungoshandisa machunks makuru. Haisi: iyo yekudzoreredza unit uye chizvarwa yuniti yakasiana nemaune saizi, saka unoramba uchinyatsoenderana pasina kufa nenzara muenzaniso wemamiriro.
Iyo yemukati IT inobatsira bot inonongedza nhanho yega yega yekugadzirisa senge yakaparadzana mwana chunk, asi kana nhanho ichienderana inodzosera maitiro ese, saka modhi haiudze vashandisi kuti vaite nhanho 4 pasina nhanho 1 kusvika 3.
Chishandiso chekutsvagisa zviri pamutemo chinoenderana nechikamu chimwe chete chenguva dzezviziviso zvekugumisa, chobva chapfuudza modhi yacho chikamu chekondirakiti yakazara, kusanganisira tsananguro nezvimwe zvinoshandura zvinorehwa nendima.
Mubatsiri wekosi yeyunivhesiti anotsvaga mitsara-chikamu chechunks kubva pane zvinyorwa zvehurukuro uye anodzosera peji yakatenderedza, saka mhinduro nezve fomula inosanganisirawo mamiriro ainoshanda pasi pawo.
Mubatsiri weinishuwarenzi anotora machisi akati wandei kubva mugwaro repolicy rimwe, ovaisa nemubereki, uye anotumira chikamu ichocho chepolicy kamwe chete pane zvimedu zvishanu zvinopindirana.
Diki-kusvika-hombe iri kuita yekusarudzika pane hunyengeri, uye makuru masisitimu atofumura hierarchical kana hwindo rekudzoreredza semagadzirirwo sarudzo. Mahwindo makuru emamiriro ezvinhu anoderedza chirango chekutumira vabereki vakakura, izvo zvinoita kuti pateni idhure kutora. Mibvunzo yakavhurika inoramba iri pamusoro pekusarudza miganhu yevabereki otomatiki, semuenzaniso kushandisa ongororo yedhizaini kana chikamu chakadzidzwa pane kuvimba nemisoro. Iri kuwedzera kusanganiswa nevanogadzirisa uye nemaitiro-ekumisikidza maitiro ayo anoedza kupa madiki chunks kuziva zvakavatenderedza panguva ye indexing. Kuedza pamagwaro ako pachako, kwete mabhenji akajairwa, inoramba iri nzira yakavimbika yekusarudza saizi yemwana nemubereki.
Tsamba yemahara
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Kudzosa magwaro evabereki, kunonziwo kutsvagisa kudiki-kusvika-kukuru, kunotsvaga pazvimedu zvidiki, zvemavara chaizvo asi inopa modhi yemutauro chikamu chikuru kana gwaro chikamu chimwe nechimwe chakabva. Izvo zvine basa nekuti inobvisa yakajairwa chunking kutengeserana-kure: madiki chunks anoenderana nemibvunzo nenzira kwayo, uye mubereki mukuru anopa iyo modhi yakakwana mamiriro ekupindura nemazvo.
Zvimedu zvidiki zvinopa zvakanangana zvekumisikidza asi zvakanyanyisa mamiriro; zvimedu zvakakura zvinopa mamiriro asi zvakadzika zvakadzika. Zvidiki-kusvika-zvikuru zvinotsvaga zvidiki uye zvinodzoka zvakakura kuti uwane zvese.
Vana vanoiswa indexed kuti vanyatsoenderana; vabereki vanongotariswa mushure memutambo wemwana.
Kuronga nemubereki kunoreva kuti chikamu chinosanganisirwa kamwe chete, chinochengetedza tokeni uye chinobvisa zvakapetwa zvidimbu.
Vana vakaiswa muchitoro chevector kutsvaga, nepo vabereki vakazara vakagara muchitoro chemagwaro uye vanotorwa neID.
Mutsara-hwindo rekudzoreredza rinoenderana pamwero wemutsara wobva wawedzera kusvika pafafitera remitsara yakatenderedza kugovera mamiriro.
Ramba uchidzidza
Mamwe madhairekitori akasarudzirwa nyaya iyi
InoteveraGaidhi rinotevera
Kudzosa Metrics: Recall@K, MRR uye nDCG
Tekinoroji