InoteveraGaidhi rinotevera
Position Interpolation yeContext Extension
Mutauro AI
Nhungamiro yehunyanzvi
Positional Interpolation yeLong Context
Positional Interpolation (PI) inzira yakapfava, ine pesvedzero inotambanudza hwindo reTransformer remukati nekudzvanya ma indices ezvinzvimbo muhuwandu iyo modhi inotoziva.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.
Kudzika Kwakadzika
Yakaunzwa ne Meta vaongorori (Chen et al.) muna 2023, Positional Interpolation inobata chokwadi chekuti mamodheru ane RoPE anokundikana zvakanyanya kana achiwedzera kune zvinzvimbo kupfuura kudzidziswa. Muono wacho unopokana: pane kukumbira modhi kuti ibate mahombe ezvinzvimbo zvayasati yamboona, PI inokamura maindisi enzvimbo anouya nechikero kuitira kureba kwe, toti, 8K mepu kudzoka muyekutanga 2K renji. Nekuti iyo modhi yakadzidziswa pane iyo siyana, iyo inotenderera inogara mu-kugovera. Mushure mekungoita chiuru-yekugadzirisa matanho, modhi yeLLaMA yakawedzera nzira iyi yakabatwa kusvika kumakumi matatu nemaviri mamiriro. Iro bepa rakaratidza kuwedzeredza kunogona kukwidza kutarisisa zvibodzwa kune hukuru hukuru, nepo kududzira kuchiita kuti irambe yakasungwa uye yakagadzikana, ndosaka kududzira kunoshanda zvakanyanya zvirinani pane kuwedzera.
Kuitwa muNyika yeChokwadi
Kuwedzera 2K-context LLaMA modhi yekubata 8K-32K tokens ine angangoita chiuru-yekugadzirisa matanho.
Kugadzirisa iyo iripo yekukurukura modhi yekupfupisa-refu-gwara pasina kudzidziswa kubva pakutanga
Kushanda seyambiro yekutanga iyo NTK-inoziva kuyera uye YaRN inovandudza pairi
Kugonesa kodhi-yakareba kodhi kana yemutemo-gwaro kuongororwa pamhando dzakadzidziswa pakutanga nemahwindo mapfupi
Ramangwana rePositional Interpolation for Long Context
Positional Interpolation yakave hwaro hwemafungu ekuteedzera, kusanganisira NTK-inoziva kuyera uye YaRN, iyo inodudzira zvakanyanya kusarudza kuchengetedza ruzivo rwenzvimbo. Iyo nzira ndeye nzira dzinoda kushoma kana kusanyatso gadzirisa uye kuenda kubheka-refu-mamiriro ekubata mupretraining. PI inoramba iri yakakosha yekutanga uye inowanzobatanidzwa neazvino frequency-aware zvirongwa zvekusvika 128K-pamwe nemamiriro ekunze windows zvinobudirira.
Kuzvishandisa
Nei zvichikosha
- Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
- Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
- Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Chenjera
- Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
- Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
- Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Matanho anotevera
- Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
- Benchmark pasi pechokwadi mutoro uye data mamiriro.
- Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
- Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the Positional Interpolation for Long Context quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
What is Positional Interpolation for Long Context?
Positional Interpolation (PI) inzira yakapfava, ine pesvedzero inotambanudza hwindo reTransformer remukati nekudzvanya ma indices ezvinzvimbo muhuwandu iyo modhi inotoziva. Panzvimbo pokuwedzera kune zvisingaoneki, inopindirana mukati mevakadzidziswa, ichingoda kurongeka kwenguva pfupi.
Ndeipi pfungwa huru kumashure kwePositional Interpolation?
PI inokamura maindisi enzvimbo nechikero zvekuti mepu yakarebesa inoteedzana kudzokera muchikamu chakadzidziswa, ichichengeta kutenderera mukugovera.
Sei naive extrapolation kune nzvimbo refu ichitadza?
Iyo PI bepa yakaratidza zvisingaonekwe zvinzvimbo zvakakura zvinogona kuburitsa kutarisisa zvibodzwa zvehukuru kupfuura kudzidziswa, kudzikamisa softmax.
Zvingangoita kuti ingani-ikugadzirisa zvakanaka iyo yekutanga PI basa yaida kuwedzera LLaMA kusvika ku32K?
Pepanhau rakashuma kuti nhanho dzinokwana chiuru dzekugadzirisa zvakanaka dzaive dzakakwana kuwedzera mamiriro kusvika kumatokeni makumi matatu nemaviri.
Kana iwe ukawedzera mamiriro nechinhu s, PI inoshandura sei chinzvimbo m?
PI inodzoreredza chinzvimbo m kusvika m/s, ichidzvanya hutsva hukuru muyero yepakutanga yakadzidziswa.
PI yakapesvedzera sei nzira dzakazotevera seYaRN?
PI yakamisa kududzira senzira yakachengeteka; NTK-inoziva kuyera uye YaRN yakakwenenzvera nekududzira ma frequency zvakanyanya kusarudza.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi