Samfuran TF-IDF da Jakar Kalmomi
Bag-of-words yana juyar da rubutu zuwa kirga kalma ba tare da kula da tsari ba, kuma TF-IDF tana ƙididdige waɗannan ƙididdiga sosai, keɓaɓɓun kalmomi suna da mahimmanci fiye da na gama-gari.
Dubawa
Together they were the workhorses of search and text classification before deep learning.
Zurfafa nutsewa
Misalin jakar kalmomi (BoW) yana wakiltar daftarin aiki azaman ƙidayar kalma, watsar da nahawu da tsarin kalma: 'kare ya ciji mutumin' da 'mutumin ya ciji kare' suna kama da juna. Wannan sauƙi yana aiki da mamaki don ayyuka da yawa. TF-IDF tana tace BoW ta hanyar sake yin nauyi. Mitar Kalma (TF) tana auna sau nawa kalma ke bayyana a cikin takarda, yayin da Inverse Document Frequency (IDF) ke rage nauyin kalmomin da ke bayyana a cikin takardu da yawa. Ƙirƙirar su yana ba da ƙima mai yawa ga kalmomin da suke akai-akai a cikin takarda ɗaya amma ba kasafai a cikin tarin ba, kamar keɓancewar jigo, yayin da kalmomin gama gari irin su 'da' ke samun kusan-sifili nauyi. TF-IDF vectors ikon binciken mahimmin mahimmin mahimmin kalmomi da ciyar da rafukan gargajiya kamar Naive Bayes da SVMs.
Fahimtar Fasaha
IDF yawanci ana lissafta shi azaman log(N/df), inda N shine jimlar adadin takardu kuma df shine adadin takaddun da ke ɗauke da kalmar, don haka kalma a kowace takarda tana haifar da IDF kusa da sifili. Makin TF-IDF na ƙarshe shine TF wanda aka ninka ta IDF. Abubuwan daftarin aiki galibi ana daidaita su da L2 kuma idan aka kwatanta su da kamanni na cosine, wanda ke auna kusurwa tsakanin vectors kuma yayi watsi da bambance-bambancen tsayin daftarin aiki.
Dabarun Tasiri
Gudu da sikelin
Gudun aikin harshe na iya tafiya da sauri ba tare da sadaukar da daidaito ba.
Shiga ku isa
Yana faɗaɗa damar shiga cikin harsuna da salon sadarwa.
Shawarwari masu haske
Ƙungiyoyi za su iya ciyar da ƙarin lokaci akan hukunci yayin da aiki da kai ke sarrafa maimaitawa.
Makomar TF-IDF da Samfuran Jakunkuna
Matsakaicin jijiyoyi masu yawa da samfuran canji yanzu suna ɗaukar tsari na kalma da ma'ana cewa BoW da TF-IDF ba za su iya ba, don haka ƙira mai zurfi sun mamaye NLP na yanke-yanke. Duk da haka TF-IDF ya kasance mai sauri, mai fassara, tushe mai ƙarancin albarkatu wanda ke da wahala a doke shi don neman mahimmin kalmomi, kuma har yanzu yana ƙarfafa tsarin dawo da matasan inda aka haɗa maƙiyan TF-IDF/BM25 tare da manyan abubuwan da aka haɗa don haɓaka bincike da haɓaka-ƙarnuka.
Aiwatar da Gaskiyar Duniya
Injunan bincike ta TF-IDF ko BM25 magajinsa akan tambaya
Tace spam ta amfani da fasalulluka-na-kalmomi da aka ciyar a cikin Naive Bayes classifier
Cire mahimman kalmomi ko alamomi daga labarin ta ɗaukar mafi girman sharuddan TF-IDF
Yana ba da shawarar labaran labarai iri ɗaya ta hanyar kwatanta TF-IDF vectors tare da kamancen cosine
Hatsari & Tsare-tsare
Abubuwan da aka ruɗe suna iya shigar da rahotanni cikin nutsuwa, kwararar tallafi, ko abubuwan bincike.
Hankali na gaggawa na iya ƙirƙirar sakamako mara daidaituwa a cikin buƙatun iri ɗaya.
Za a iya fallasa bayanan rubutu mai ma'ana idan ikon samun dama yana da rauni.
Taswirar Hanya
Ƙayyade tsarin fitarwa, sautin, da ma'auni masu inganci kafin fitowa.
Amsa a ƙasa tare da amintattun tushe a duk lokacin da daidaito ya shafi mahimmanci.
Ajiye wurin binciken ɗan adam don abubuwan da ake samu masu girma.
Bibiyar tsarin gazawar kuma sake horar da tsokaci ko tafiyar aiki akai-akai.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the TF-IDF and Bag-of-Words Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Haɗin Kalma
Tambayoyin da ake yawan yi
What is TF-IDF and Bag-of-Words Models?
Bag-of-words yana juyar da rubutu zuwa kirga kalma ba tare da kula da tsari ba, kuma TF-IDF tana ƙididdige waɗannan ƙididdiga sosai, keɓaɓɓun kalmomi suna da mahimmanci fiye da na gama-gari. Tare sun kasance dawakan aiki na bincike da rarraba rubutu kafin zurfafa ilmantarwa.
Wane mahimmin bayani ke watsi da samfurin jakunkuna?
Bag-of-word yana kiyaye adadin kalmomi amma yana watsar da tsarin kalma da tsarin nahawu.
Menene sashin IDF na TF-IDF yake yi?
Mitar Takardun Rubuce-rubucen yana rage nauyin kalmomin da ke bayyana a cikin takardu da yawa, don haka kalmomin gama gari kamar 'da' suna ba da gudummawa kaɗan.
Kalmar da ta bayyana a kowace takarda a cikin tarin tana samun ƙimar IDF kusa da me?
Tare da IDF = log (N/df), idan df yayi daidai N rabon shine 1 kuma log (1) shine 0, yana ba da kalmar kusan babu nauyi.
Yaya ake ƙididdige makin TF-IDF na wani lokaci?
Nauyin TF-IDF shine kalmar mitar da aka ninka ta mitar daftarin aiki.
Wanne ma'aunin kamanni ne aka saba amfani da shi don kwatanta abubuwan daftarin aiki na TF-IDF?
Kwatankwacin Cosine yana auna kusurwa tsakanin vectors kuma daidaitaccen misali ne don kwatanta TF-IDF ba tare da la'akari da tsayin daftarin aiki ba.