TensorRT hamwe na moteri yerekana
TensorRT nububiko bwibitabo bwa NVIDIA bukusanya imiyoboro yimyitozo yatojwe mumoteri nziza cyane ikora vuba cyane kuri NVIDIA GPUs.
Incamake
It matters because the same model can run 2-6x quicker and cheaper at inference time without changing what it predicts.
Kwibira cyane
Moteri yumwanzuro ifata moderi yatojwe ikongera ikayandika kugirango ikorwe byihuse kubikoresho bigenewe. TensorRT ibikora kuri NVIDIA GPUs binyuze munzira nyinshi. Ikora layer fusion, guhuza ibikorwa nka convolution, kubogama-kongeramo, na ReLU mumurongo umwe wa GPU kugirango ugabanye traffic traffic. Ikoresha kalibrasi yuzuye, ikamanuka kuri FP32 ikagera kuri FP16 cyangwa INT8 (na FP8 kuri Hopper) mugihe ubitse neza. Ikoresha kernel auto-tuning, igereranya ibikorwa byinshi bya buri cyiciro kuri GPU yawe neza kandi igatora byihuse. Igisubizo ni dosiye ikurikirana 'moteri' ihujwe nuburyo bumwe bwa GPU. TensorRT-LLM yagura ibi hamwe na paje ya KV-cache, mu ndege, hamwe na tensor parallelism kuri moderi nini y'ururimi.
Ubushishozi
Umuvuduko munini uturuka mumayeri abiri. Kernel fusion ikuraho ingendo-shuri kugirango igabanye GPU yibuke kwisi yose mugukomeza ibisubizo hagati mubitabo byihuse hamwe nibisangiwe. Quantisation kuri INT8 ipakira indangagaciro enye aho FP32 imwe yicaye, inshuro enye zinjiza arithmetic yinjiza kuri censor cores, ariko ikenera kalibrasi ya dataset kugirango ibare ibintu byapimye kuri tensor kugirango igabanuka ryimibare idasenya ukuri. Moteri ifite ibyuma byihariye kuko auto-tuning iteka mubitereko byiza kuri GPU yibanze hamwe nuburyo bwo kwibuka.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Kazoza ka TensorRT na Moteri Yerekana
Moteri zifatika zigenda zigana neza (FP8, FP4, hamwe na gahunda zivanze) hamwe na LLM yihariye nkibishushanyo mbonera hamwe na KV-cache yerekana ubwenge. TensorRT-LLM hamwe nabanywanyi nka vLLM bahurira kuri prefill / decode itandukanijwe kandi ikomeza. Witegereze gukusanya hamwe (Torch-TensorRT, ONNX), kubara mu buryo bwikora hamwe na kalibrasi nkeya, hamwe n'inkunga nini yo kuvanga-impuguke zigenda zikora nka moderi nini bihendutse bihinduka intambara yo hagati.
Gushyira mu bikorwa Isi
Guhindura icyitegererezo cya YOLO kuri moteri ya TensorRT INT8 kuburyo ikora mugihe nyacyo kuri NVIDIA Jetson muri robot cyangwa kamera yubwenge
Gukorera moderi ya Llama cyangwa Mistral hamwe na TensorRT-LLM ukoresheje indege yo mu ndege kugirango ugabanye ibimenyetso-isegonda kuri H100 GPUs mumugongo wa chatbot
Kunoza imvugo-imenyekanisha imvugo hamwe na FP16 itomoye kugirango ugabanye gutinda kwandikirwa muri serivise nzima
Gukusanya urwego-rwerekana ibyifuzo kuri moteri ya TensorRT yahujwe kugirango ikemure amamiriyoni yibisabwa kumasegonda ku giciro gito cya GPU
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the TensorRT and Inference Engines quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Gukoresha AI
Ibibazo bikunze kubazwa
What is TensorRT and Inference Engines?
TensorRT nububiko bwibitabo bwa NVIDIA bukusanya imiyoboro yimyitozo yatojwe mumoteri nziza cyane ikora vuba cyane kuri NVIDIA GPUs. Nibyingenzi kuko moderi imwe irashobora gukora 2-6x byihuse kandi bihendutse mugihe cyimyanzuro idahinduye ibyo iteganya.
Niyihe ntego yibanze ya moteri yerekana nka TensorRT?
Moteri zifatika zifata icyitegererezo kimaze gutozwa hanyuma ukongera kukandika kumuvuduko mwinshi kubikoresho byihariye; ntibatoza icyitegererezo.
Nigute guhuza ibice byihutisha gufata umwanzuro?
Fusion ikomatanya ibikorwa nka conv, kubogama, no gukora muri kernel imwe kugirango ibisubizo bigufi bigume mububiko bwihuse aho kwandikwa inyuma kugirango buhoro buhoro kwibuka kwisi.
Kuki kubara INT8 mubisanzwe bisaba kalibrasi ya dataset?
Calibration ikoresha amakuru ahagarariye binyuze murugero kugirango hamenyekane ibipimo bya tensor, bityo intera ntarengwa ya INT8 ibika agaciro gakwirakwizwa.
Kuki moteri ya TensorRT yakozwe muri rusange yihariye imwe mubwubatsi bwa GPU?
Auto-tuning ibipimo ngenderwaho ku ntego ya GPU igahitamo icyiza, bigatuma moteri ihujwe nuburanga bwububiko.
Ni ibihe bintu biranga TensorRT-LLM bifasha cyane cyane gutanga imvugo nini y'ururimi neza?
TensorRT-LLM yongeraho LLM yihariye yo gutezimbere nko mu ndege yogeza hamwe na paje ya KV-cache kugirango yinjize byinshi mubikorwa byakazi-byakazi.