Nhungamiro yehunyanzvi

RAG yeMultimodal

Multimodal RAG ndeyekudzoreredza-yakawedzerwa chizvarwa icho chinogona kutsvaga nekushandisa mifananidzo, machati, matafura uye mapeji akaongororwa, kwete kungoita mavara akajeka chunks.

Teerera kugwaro iriAI-yakarondedzerwa · ~3 min

Pa peji ino3 min verenga
  1. Pfupiso
  2. Kudzika Kwakadzika
  3. Kuitwa muNyika yeChokwadi
  4. Ramangwana reMultimodal RAG
  5. Kuzvishandisa
  6. Ramba Uchiongorora
  7. Mibvunzo inowanzo bvunzwa

Pfupiso

Inoshanda nekunyora zvinyorwa zvinoonekwa mumameseji, kudzvanya mifananidzo uye zvinyorwa munzvimbo yakagovaniswa, kana kudzoreredza peji rese zvidzitiro zvine mhando dzechiratidzo. Izvo zvine basa nekuti yakawanda-chaiyo-yenyika ruzivo inogara mumaPDF, masiraidhi uye madhayagiramu anonyora-chete mapaipi anoputika kana kufuratira.

Kudzika Kwakadzika

Ruzivo rwesangano rwakawanda haruna kuchena. Inogara mumasiraidhi decks, zvibvumirano zvakaongororwa, madhirowa einjiniya, matafura emari uye machati mukati mePDF. Mameseji-chete pombi anowanzo mhanyisa OCR kana PDF parser, inorasa marongero uye chunks zvinobuda. Matafura anova mitsara yakasanganiswa, machati echati anonyangarika, uye dhayagiramu haibatsire chero chinhu. Multimodal RAG inochengeta ruzivo rwekuona ruchidzoserwa. Pane nzira nhatu dzakafara. Yekutanga inoshandura zvese kuita zvinyorwa: bvisa matafura muMarkdown kana HTML, shandisa mufananidzo wemutauro wechiratidzo kunyora tsananguro yemifananidzo nemachati, uye indengedze iwo ane akajairwa zvinyorwa zvakamisikidzwa. Iyo iri nyore uye inokodzera hurongwa huripo, asi chero chinhu chinopotswa necaption chakarasika. Yechipiri inomisikidza mifananidzo uye mameseji munzvimbo yakagovaniswa vector, sezvinoita CLIP-maitiro mamodheru, saka mubvunzo wemavara unogona kutora mufananidzo kana nhamba zvakananga. Izvi zvinokodzera mifananidzo-inorema kuunganidzwa senge zvigadzirwa zvemakatalogi, asi zvakajairika kumisikidzwa kwemifananidzo hakuna kusimba pakuverenga zvinyorwa zvakakora mukati memagwaro. Yechitatu inobata peji yega yega sepikicha. ColPali, yakaburitswa muna 2024, inomisikidza mifananidzo yemapeji nemhando yemutauro wechiratidzo uye inoaenzanisa nemibvunzo uchishandisa nguva yadarika, achienzanisa akawanda emibvunzo-tokeni mavheta ane akawanda emifananidzo-chigamba mavheji muchimiro cheColBERT. Iyo inosvetuka OCR uye dhizaini parsing zvachose, uye yakaita nesimba paViDoRe gwaro-rekudzosa bhenji yakaunzwa padivi payo. Kutora ihafu yebasa. Jenareta inofanirawo kuona zviri mukati, saka mhinduro dzinowanzobva kumuenzaniso unokwanisa kuona wakapihwa mufananidzo wepeji kana chimiro chakadimburwa, kwete chinyorwa chete. Imwe pfungwa isiriyo ndeyekuti multimodal RAG inoreva kusiya mameseji mapaipi. Mukuita, masisitimu akasanganiswa anobatanidza mavara akapatsanurwa, kuburitsa tafura uye kudzoreredza kwepeji-mufananidzo zvakajairika, nekuti imwe neimwe nzira inobata kukundikana kwevamwe.

Kuitwa muNyika yeChokwadi

Mubatsiri anogadzirisa wemugadziri anotora dhayagiramu yakaputika yepombi modhi uye anoiratidza kune inokwanisa kuona modhi, iyo inozivisa gasket rakakodzera kubva kune zvinyorwa zvedhirowa.

Muongorori wezvemari anobvunza nezve mari yekota nechikamu, uye sisitimu inotora mufananidzo wepeji ine bar chart pane kuvimba neOCR mavara akadonhedza kukosha kwechati.

Mutengesi anoita kuti vatengesi vatsvage katarogu ine mavara ekuti "green ceramic table rambi" uye anoshandisa zvinomisikidzwa nemifananidzo-mavara kudzorera zvinoenderana nemifananidzo yechigadzirwa.

Hofisi yekurekodha inoratidzira makumi emakumi emakumi emafomu akaongororwa semapeji skrini, saka mabharani anogona kuwana magwaro ane hurombo-mhando scanner inoburitsa isingavimbike OCR mavara.

Ramangwana reMultimodal RAG

Mamodheru emutauro wechiratidzo anoramba achinatsiridza pakuverenga magwaro zvakananga, izvo zvinosimbisa nyaya yekudzosa mufananidzo wepeji uye nekusvetuka nhanho dzisina kusimba dzekupfuura. Multi-vector kuchengetedza mutengo unoramba uri chipingamupinyi chinoshanda, uye vector dhatabhesi vanga vachiwedzera rutsigiro rwekunonoka-kupindirana kutsvaga kugadzirisa. Vhidhiyo neodhiyo kudzoreredza zvinoshandisa pfungwa dzakafanana, uchishandisa mafuremu kana zvinyorwa semayuniti ekutora, asi midziyo haina kukura. Ongororo zvakare inononoka mavara RAG; mabhenji eruzhinji akadai serubatsiro rweViDoRe, asi masangano acharamba achida bvunzo seti dzakavakwa kubva kune avo ega akaongororwa mafomu, machati nemasiraidhi kuti azive nzira inovaitira.

Kuzvishandisa

Nei zvichikosha

  • Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
  • Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
  • Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.

Chenjera

  • Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
  • Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
  • Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.

Matanho anotevera

  1. Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
  2. Benchmark pasi pechokwadi mutoro uye data mamiriro.
  3. Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
  4. Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.

Ramba Uchiongorora

Tsamba yemahara

Tora iyo yemazuva ese AI pfupiso

Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.

Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.

Zviedze iwe pachako

Take the Multimodal RAG quiz

Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.

Tanga mibvunzo

Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo

Mibvunzo inowanzo bvunzwa

Chii chinonzi Multimodal RAG?

Multimodal RAG ndeyekudzoreredza-yakawedzerwa chizvarwa icho chinogona kutsvaga nekushandisa mifananidzo, machati, matafura uye mapeji akaongororwa, kwete kungoita mavara akajeka chunks. Inoshanda nekunyora zvinyorwa zvinoonekwa mumameseji, kudzvanya mifananidzo uye zvinyorwa munzvimbo yakagovaniswa, kana kudzoreredza peji rese zvidzitiro zvine mhando dzechiratidzo. Izvo zvine basa nekuti yakawanda-chaiyo-yenyika ruzivo inogara mumaPDF, masiraidhi uye madhayagiramu anonyora-chete mapaipi anoputika kana kufuratira.

Chii chinowanzoitika chisina kunaka kana pombi-chete yeRAG pombi inogadzira PDF izere nematafura nemachati?

Kuisa kune mavara akajeka kunorasa marongerwo, saka chimiro chetafura chinotsemuka uye ruzivo rwekuona rwunonyangarika.

Ndeipi kushaya simba kukuru kwecaption-everything-mu-text approach?

Kudzoreredza kunogona chete kuwana izvo zvinogadzirwa tsananguro; zvakasiiwa hazvionekwe.

Chii chinoita CLIP-maitiro modhi inoita kuti igoneke mune multimodal RAG?

Nzvimbo yakagovaniswa inoreva kuti chinyorwa chebvunzo vector chinogona kufananidzwa zvakananga nemavheji emifananidzo.

Chii chinosiyanisa maitiro eColPali?

ColPali inobata iyo peji peji pachayo seyekudzoreredza unit, ichidzivirira danho rekupaza zvachose.

Mukunonoka kusangana, mubvunzo unopihwa sei pane peji?

Kudyidzana kwanonoka kunochengeta akawanda mavheji pamativi ese uye kuunganidza iwo akanakisa match, sezvinoita ColBERT yemavara.