Dhexgalka Mawqifka ah ee Mawduuca Dheer
Dhexgalka Mawqifka ah (PI) waa farsamo fudud oo saamayn leh oo fidisa daaqadda macnaha guud ee Transformer iyadoo ku tuujinaysa tusmooyin boos cusub oo kala duwan oo moodelku hore u yaqaanay.
Dulmar
Instead of extrapolating to unseen positions, it interpolates within trained ones, requiring only brief fine-tuning.
quusid qoto dheer
Waxaa soo bandhigay Meta cilmi-baarayaal (Chen et al.) 2023, Dhexdhexaadinta Mawqifku waxay wax ka qabanaysaa xaqiiqda ah in moodooyinka leh RoPE ay ku dhacaan masiibo marka loo eego boosaska ka baxsan tababarka. Aragtidu waa mid ka soo horjeeda: halkii la waydiisan lahaa moodeelka si uu u xakameeyo qiyamka booska weyn ee uusan waligiis arkin, PI waxay u qaybisaa tusmooyinka booska soo galaya qodob miisaan ah si dhererka bartilmaameedka ah, dheh, khariidadaha 8K dib ugu soo celinta asalka 2K. Sababtoo ah moodelku waxa lagu tababaray xadkaas, wareegyadu waxay ahaanayaan qaybinta. Kaliya 1,000 tilaabo hagaajin ah ka dib, qaabka LLAMA ayaa sidan u fidiyay ilaa 32K macnaha guud. Warqaddu waxay muujisay ka-saariddu waxay qarxin kartaa dhibcaha dareenka qiyamka weyn, halka isdhexgalka ay ka dhigayso mid xaddidan oo xasilloon, taas oo ah sababta isdhexgalka u shaqeeyo si ka fiican marka loo eego extrapolation.
Aragtida Farsamada
PI waxay dib u cabbiraysaa booska m ilaa m/s halka s ay tahay qodobka kordhinta (tusaale, dherer cusub oo loo qaybiyay dhererka asalka ah). RoPE-da tani waxay si wax ku ool ah u yaraynaysaa tallabada wareegta ee u dhaxaysa jagooyinka ku xiga, xidhidhaynta jagooyin badan oo xagal ah oo tababaran. Fikradda ku xidhan warqaddu waxay muujinaysaa dhibcooyinka fiiro gaar ah loo leeyahay in si fiican loo xakameeyo, halka ka saarista naive ay soo saari karto buundooyinka amarada ka weyn wax kasta oo lagu arko tababarka, xasiloonida softmax.
Saamaynta Istiraatijiyadeed
Qiimaha iyo miisaaniyada
Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
Go'aamo cad
Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
Xakamaynta tayada
Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Mustaqbalka isdhexgalka booska ee macnaha dheer
Dhexgalka Mawqifka waxa uu noqday aasaaska mowjad dabagal ah, oo ay ku jiraan NTK-aware scaling iyo YaRN, kuwaas oo si aad ah u dhexgala si loo ilaaliyo tafatirka gudaha. Ujeedadu waxay ku wajahan tahay hababka u baahan hagaajin yar ama aan lahayn hagaajinta iyo dubista habsocodka muddada dheer ee tababarka hore. PI ayaa weli ah asaas qiimo leh waxaana inta badan lagu daraa qorshayaal ogaanaya soo noqnoqda oo cusub si loo gaaro 128K iyo daaqadaha macnaha guud si hufan.
Dhaqangelinta Adduunka-dhabta ah
Kordhinta qaabka 2K-context LLAMA si loo maamulo 8K-32K calaamado leh ilaa 1,000 tillaabo hagaajin ah
La qabsiga qaabka wada sheekaysiga ee jira ee soo koobida dukumeentiga dheer iyada oo aan dib looga tabobarn meel eber ah
U adeegida sida aasaaska fikradeed ee NTK-ka warhaynta cabirka iyo YaRN ay horumariyaan
Awood u yeelashada koodka macnaha dheer ama falanqaynta dukumeenti sharci ee moodooyinka markii hore lagu tababaray daaqado gaagaaban
Khatarta & Dariiqyada Ilaalada
Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Qorshe Hawleedka Dhaqangelinta
Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Positional Interpolation for Long Context quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Hagaha xiga
Dhexdhexaadinta Booska ee Kordhinta Macnaha guud
Su'aalaha soo noqnoqda
What is Positional Interpolation for Long Context?
Dhexgalka Mawqifka ah (PI) waa farsamo fudud oo saamayn leh oo fidisa daaqadda macnaha guud ee Transformer iyadoo ku tuujinaysa tusmooyin boos cusub oo kala duwan oo moodelku hore u yaqaanay. Halkii laga soo saari lahaa boosaska aan la arki karin, waxay isku xirtaa kuwa tababaran, waxayna u baahan tahay hagaajin kooban oo kooban.
Waa maxay fikradda asaasiga ah ee ka dambeysa Interpolation Positional?
PI waxay u qaybisaa tusmooyinka booska iyada oo loo eegayo miisaan sidaas oo kale ah taxane dheer oo khariidad dib ugu celinaya booska la tababaray, iyada oo la ilaalinayo wareegyada qaybinta.
Waa maxay sababta ka-baxsanaanta naive-ka ee jagooyinka dhaadheer u guul-darreysato?
Waraaqda PI waxay muujisay jagooyin waaweyn oo aan la arki karin waxay soo saari karaan dhibcaha dareenka amarrada ka weyn tababarka, xasilloonida softmax.
Qiyaastii intee in le'eg hagaajinta hagaajinta ayaa shaqada PI ee asalka ahi u baahnayd in la kordhiyo LLAMA ilaa 32K?
Wargeysku wuxuu sheegay in ku dhawaad 1,000 tillaabo hagaajin ah ay ku filan yihiin in la kordhiyo macnaha guud ilaa 32K calaamado.
Haddii aad macnaha guud ku kordhiso qodob s, sidee buu PI u beddelaa boos m?
PI waxay cabbiraysaa booska m ilaa m/s, iyadoo ku cadaadinaysa kala duwanaanshaha cusub ee kala duwan ee tababarka asalka ah.
Sidee buu PI u saameeyay hababka dambe sida YaRN?
PI waxay dejisay isdhexgalka sidii habka badbaadada leh; NTK-ka warhaynta iyo YaRN ayaa sifeeyay iyaga oo is-dhex-galiyay soo noqnoqoshada si aad u xulan.