Incamake
Icyerekezo cyumurongo gisimbuza quadratic softmax kwitondera muri Transformers hamwe namayeri yimibare ipima umurongo hamwe nuburebure bwikurikiranya. Abakora nuburyo bwingenzi bugereranya softmax ukoresheje intungamubiri zidasanzwe, gukora urutonde rurerure cyane kubara.
Imirongo Itondekanya hamwe na Kernels ikora ni tekinoroji yo kubaka tekiniki igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo.
Kwibira cyane
Icyerekezo gisanzwe gihindura amanota hagati ya buri kimenyetso, gitwara igihe hamwe nibuka bikura hamwe na kare yuburebure (O (n ^ 2)). Kwitonda kumurongo byongeye kubara kuburyo igiciro gikura gusa umurongo (O (n)). Igitekerezo cyingenzi: softmax kwitondera ni softmax (QK ^ T) V, ariko niba usimbuye softmax hamwe nikarita yerekana ikarita phi, ubona phi (Q) (phi (K) ^ T V). Kuberako kugwiza matrix bifitanye isano, ubara phi (K) ^ T V ubanza (mato mato mato mato mato), ukirinda amanota manini n-by-n amanota yose. Abakora, guhera kuri Google muri 2020, bituma ibi bigereranywa kwizerwa rya softmax yukuri ukoresheje FAVOR + (Byihuta Byihutirwa Binyuze muburyo bwiza bwa Orthogonal Random ibiranga), ushushanya ibipimo bitunguranye bituma igereranya ryintangiriro ritabogamye kandi rihamye.
Ubushishozi
UMUKUNZI WA FAVOR + agereranya softmax kernel exp (q.k) akoresheje ibintu byiza bidasanzwe: ishushanya ibibazo nurufunguzo binyuze muri progaramu ya Gaussiya yapfunditswe mu buryo bworoshye, byemeza uburemere butari bwiza kandi birinda ihungabana ry’imibare yabateganijwe mbere. Gukoresha imiterere ya orthogonal idasanzwe igabanya itandukaniro. Icy'ingenzi, matrike ya n-by-n ntabwo yigeze iba iy'umubiri, bityo kwibuka bigabanuka kuva kuri quadratic kugera kumurongo, bigatuma urutonde rwibihumbi mirongo.
Kumenya Imirongo Itondekanya hamwe nudukoryo
Icyerekezo cyumurongo gisimbuza quadratic softmax kwitondera muri Transformers hamwe namayeri yimibare ipima umurongo hamwe nuburebure bwikurikiranya. Abakora nuburyo bwingenzi bugereranya softmax ukoresheje intungamubiri zidasanzwe, gukora urutonde rurerure cyane kubara. Imirongo Itondekanya hamwe na Kernels ikora ni tekinoroji yo kubaka tekiniki igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo. Kugirango wubake byimbitse, fata umurongo witondere hamwe na Kernels ikora nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mu myitozo, amakipe akomeye akoresha umurongo witondere hamwe na Kernels ikora neza yubaka imyubakire, amakuru, hamwe nibikorwa remezo birwanya kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa-Isi
Gutunganya genomic cyangwa proteine ndende aho kwitabwaho kwadratic byuzuye byananiza ububiko bwa GPU
Urwego-urwego rwincamake kuri raporo ndende cyane nta gucogora, ukoresheje Imiterere-yuburyo bwumugongo
Uburyo bwiza-buringaniye bwamajwi cyangwa ibihe-byerekana uburyo aho bikurikirana intambwe ibihumbi mirongo
Kugabanya ibiciro byimyanzuro mugihe kirekire-biganiro biganirwaho mugusimbuza ibice bimwe bya softmax hamwe numurongo-wo kwitondera ibintu
Uburyo bwo Gushyira mu bikorwa
Imirongo Itondekanya hamwe nudukoryo twimyitozo
Gutunganya genomic cyangwa proteine ndende aho kwitabwaho kwadratic byuzuye byananiza ububiko bwa GPU.
Gutunganya urutonde rurerure rwa genomic cyangwa proteine aho kwitabwaho kwuzuye kwari kunaniza ububiko bwa GPU Amakipe ubusanzwe abona ibisubizo byiza mugihe asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro nibiciro byigihe.
Imirongo Itondekanya hamwe nudukoryo twimyitozo
Urwego-urwego rwincamake kuri raporo ndende cyane nta gucogora, ukoresheje Imiterere-yuburyo bwumugongo.
Urwego-rwerekana incamake kuri raporo ndende cyane nta gutitira, ukoresheje uburyo bwa Performer-style-backbone Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Imirongo Itondekanya hamwe nudukoryo twimyitozo
Uburyo bwiza-buringaniye bwamajwi cyangwa ibihe-byerekana uburyo aho bikurikirana intambwe ibihumbi mirongo.
Uburyo bwiza-burebure bwamajwi cyangwa ibihe-byerekana urugero aho ibihe bigenda bikurikirana intambwe ibihumbi mirongo Amakipe akunze kubona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo byimbere, kandi akurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe.
Imirongo Itondekanya hamwe nudukoryo twimyitozo
Kugabanya ibiciro byimyanzuro mugihe kirekire-biganiro biganirwaho mugusimbuza ibice bya softmax hamwe numurongo-wo kwitondera ibintu.
Kugabanya ikiguzi cyo gufata umwanzuro muburyo burebure bwo kuganira mugusimbuza ibice bimwe bya softmax hamwe numurongo-wo kwitondera umurongo Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.