摘要:漢語分詞系統是該漢語分詞系統的封裝版安裝步驟官方文檔的漢語分詞示例歡迎科研人員技術工程師企事業單位與個人參與平臺的建設工作。
NLPIR/ICTCLAS 漢語分詞系統(http://ictclas.nlpir.org)
PyNLPIR 是該漢語分詞系統的 python 封裝版(http://pynlpir.readthedocs.io...)
安裝步驟:
① pip install pynlpir
② pynlpir update
官方文檔的漢語分詞示例:
import pynlpir pynlpir.open() str = "歡迎科研人員、技術工程師、企事業單位與個人參與 NLPIR 平臺的建設工作。" result = pynlpir.segment(str) print(result) # output: [("歡迎", "verb"), ("科研", "noun"), ("人員", "noun"), ("、", "punctuation mark"), ("技術", "noun"), ("工程師", "noun"), ("、", "punctuation mark"), ("企事業", "noun"), ("單位", "noun"), ("與", "conjunction"), ("個人", "noun"), ("參與", "verb"), ("NLPIR", "noun"), ("平臺", "noun"), ("的", "particle"), ("建設", "verb"), ("工作", "verb"), ("。", "punctuation mark")]
可能遇到的問題:
① raise RuntimeError("NLPIR function "NLPIR_Init" failed.")
解決方案:
訪問 https://github.com/NLPIR-team... 倉庫,
下載 license 例如 NLPIR-ICTCLAS 分詞系統授權中的 NLPIR.user 文件,
替換路徑 path_to_local_python/Lib/site-packages/pynlpir/Data 下的同名文件以更新授權。
中文停用詞表:
["啊","阿","哎","哎呀","哎喲","唉","俺","俺們","按","按照","吧","吧噠","把","罷了","被","本","本著","比","比方","比如","鄙人","彼","彼此","邊","別","別的","別說","并","并且","不比","不成","不單","不但","不獨","不管","不光","不過","不僅","不拘","不論","不怕","不然","不如","不特","不惟","不問","不只","朝","朝著","趁","趁著","乘","沖","除","除此之外","除非","除了","此","此間","此外","從","從而","打","待","但","但是","當","當著","到","得","的","的話","等","等等","地","第","叮咚","對","對于","多","多少","而","而況","而且","而是","而外","而言","而已","爾后","反過來","反過來說","反之","非但","非徒","否則","嘎","嘎登","該","趕","個","各","各個","各位","各種","各自","給","根據","跟","故","故此","固然","關于","管","歸","果然","果真","過","哈","哈哈","呵","和","何","何處","何況","何時","嘿","哼","哼唷","呼哧","乎","嘩","還是","還有","換句話說","換言之","或","或是","或者","極了","及","及其","及至","即","即便","即或","即令","即若","即使","幾","幾時","己","既","既然","既是","繼而","加之","假如","假若","假使","鑒于","將","較","較之","叫","接著","結果","借","緊接著","進而","盡","盡管","經","經過","就","就是","就是說","據","具體地說","具體說來","開始","開外","靠","咳","可","可見","可是","可以","況且","啦","來","來著","離","例如","哩","連","連同","兩者","了","臨","另","另外","另一方面","論","嘛","嗎","慢說","漫說","冒","么","每","每當","們","莫若","某","某個","某些","拿","哪","哪邊","哪兒","哪個","哪里","哪年","哪怕","哪天","哪些","哪樣","那","那邊","那兒","那個","那會兒","那里","那么","那么些","那么樣","那時","那些","那樣","乃","乃至","呢","能","你","你們","您","寧","寧可","寧肯","寧愿","哦","嘔","啪達","旁人","呸","憑","憑借","其","其次","其二","其他","其它","其一","其余","其中","起","起見","豈但","恰恰相反","前后","前者","且","然而","然后","然則","讓","人家","任","任何","任憑","如","如此","如果","如何","如其","如若","如上所述","若","若非","若是","啥","上下","尚且","設若","設使","甚而","甚么","甚至","省得","時候","什么","什么樣","使得","是","是的","首先","誰","誰知","順","順著","似的","雖","雖然","雖說","雖則","隨","隨著","所","所以","他","他們","他人","它","它們","她","她們","倘","倘或","倘然","倘若","倘使","騰","替","通過","同","同時","哇","萬一","往","望","為","為何","為了","為什么","為著","喂","嗡嗡","我","我們","嗚","嗚呼","烏乎","無論","無寧","毋寧","嘻","嚇","相對而言","像","向","向著","噓","呀","焉","沿","沿著","要","要不","要不然","要不是","要么","要是","也","也罷","也好","一","一般","一旦","一方面","一來","一切","一樣","一則","依","依照","矣","以","以便","以及","以免","以至","以至于","以致","抑或","因","因此","因而","因為","喲","用","由","由此可見","由于","有","有的","有關","有些","又","于","于是","于是乎","與","與此同時","與否","與其","越是","云云","哉","再說","再者","在","在下","咱","咱們","則","怎","怎么","怎么辦","怎么樣","怎樣","咋","照","照著","者","這","這邊","這兒","這個","這會兒","這就是說","這里","這么","這么點兒","這么些","這么樣","這時","這些","這樣","正如","吱","之","之類","之所以","之一","只是","只限","只要","只有","至","至于","諸位","著","著呢","自","自從","自個兒","自各兒","自己","自家","自身","綜上所述","總的來看","總的來說","總的說來","總而言之","總之","縱","縱令","縱然","縱使","遵照","作為","兮","呃","唄","咚","咦","喏","啐","喔唷","嗬","嗯","噯","啊哈","啊呀","啊喲","挨次","挨個","挨家挨戶","挨門挨戶","挨門逐戶","挨著","按理","按期","按時","按說","暗地里","暗中","暗自","昂然","八成","白白","半","梆","保管","保險","飽","背地里","背靠背","倍感","倍加","本人","本身","甭","比起","比如說","比照","畢竟","必","必定","必將","必須","便","別人","并非","并肩","并沒","并沒有","并排","并無","勃然","不","不必","不常","不大","不得","不得不","不得了","不得已","不迭","不定","不對","不妨","不管怎樣","不會","不僅僅","不僅僅是","不經意","不可開交","不可抗拒","不力","不了","不料","不滿","不免","不能不","不起","不巧","不然的話","不日","不少","不勝","不時","不是","不同","不能","不要","不外","不外乎","不下","不限","不消","不已","不亦樂乎","不由得","不再","不擇手段","不怎么","不曾","不知不覺","不止","不止一次","不至于","才","才能","策略地","差不多","差一點","常","常常","常言道","常言說","常言說得好","長此下去","長話短說","長期以來","長線","敞開兒","徹夜","陳年","趁便","趁機","趁熱","趁勢","趁早","成年","成年累月","成心","乘機","乘勝","乘勢","乘隙","乘虛","誠然","遲早","充分","充其極","充其量","抽冷子","臭","初","出","出來","出去","除此","除此而外","除此以外","除開","除去","除卻","除外","處處","川流不息","傳","傳說","傳聞","串行","純","純粹","此后","此中","次第","匆匆","從不","從此","從此以后","從古到今","從古至今","從今以后","從寬","從來","從輕","從速","從頭","從未","從無到有","從小","從新","從嚴","從優","從早到晚","從中","從重","湊巧","粗","存心","達旦","打從","打開天窗說亮話","大","大不了","大大","大抵","大都","大多","大凡","大概","大家","大舉","大略","大面兒上","大事","大體","大體上","大約","大張旗鼓","大致","呆呆地","帶","殆","待到","單","單純","單單","但愿","彈指之間","當場","當兒","當即","當口兒","當然","當庭","當頭","當下","當真","當中","倒不如","倒不如說","倒是","到處","到底","到了兒","到目前為止","到頭","到頭來","得起","得天獨厚","的確","等到","叮當","頂多","定","動不動","動輒","陡然","都","獨","獨自","斷然","頓時","多次","多多","多多少少","多多益善","多虧","多年來","多年前","而后","而論","而又","爾等","二話不說","二話沒說","反倒","反倒是","反而","反手","反之亦然","反之則","方","方才","方能","放量","非常","非得","分期","分期分批","分頭","奮勇","憤然","風雨無阻","逢","弗","甫","嘎嘎","該當","概","趕快","趕早不趕晚","敢","敢情","敢于","剛","剛才","剛好","剛巧","高低","格外","隔日","隔夜","個人","各式","更","更加","更進一步","更為","公然","共","共總","夠瞧的","姑且","古來","故而","故意","固","怪","怪不得","慣常","光","光是","歸根到底","歸根結底","過于","毫不","毫無","毫無保留地","毫無例外","好在","何必","何嘗","何妨","何苦","何樂而不為","何須","何止","很","很多","很少","轟然","后來","呼啦","忽地","忽然","互","互相","嘩啦","話說","還","恍然","會","豁然","活","伙同","或多或少","或許","基本","基本上","基于","極","極大","極度","極端","極力","極其","極為","急匆匆","即將","即刻","即是說","幾度","幾番","幾乎","幾經","既...又","繼之","加上","加以","間或","簡而言之","簡言之","簡直","見","將才","將近","將要","交口","較比","較為","接連不斷","接下來","皆可","截然","截至","藉以","借此","借以","屆時","僅","僅僅","謹","進來","進去","近","近幾年來","近來","近年來","盡管如此","盡可能","盡快","盡量","盡然","盡如人意","盡心竭力","盡心盡力","盡早","精光","經常","竟","竟然","究竟","就此","就地","就算","居然","局外","舉凡","據稱","據此","據實","據說","據我所知","據悉","具體來說","決不","決非","絕","絕不","絕頂","絕對","絕非","均","喀","看","看來","看起來","看上去","看樣子","可好","可能","恐怕","快","快要","來不及","來得及","來講","來看","攔腰","牢牢","老","老大","老老實實","老是","累次","累年","理當","理該","理應","歷","立","立地","立刻","立馬","立時","聯袂","連連","連日","連日來","連聲","連袂","臨到","另方面","另行","另一個","路經","屢","屢次","屢次三番","屢屢","縷縷","率爾","率然","略","略加","略微","略為","論說","馬上","蠻","滿","沒","沒有","每逢","每每","每時每刻","猛然","猛然間","莫","莫不","莫非","莫如","默默地","默然","吶","那末","奈","難道","難得","難怪","難說","內","年復一年","凝神","偶而","偶爾","怕","砰","碰巧","譬如","偏偏","乒","平素","頗","迫于","撲通","其后","其實","奇","齊","起初","起來","起首","起頭","起先","豈","豈非","豈止","迄","恰逢","恰好","恰恰","恰巧","恰如","恰似","千","萬","千萬","千萬千萬","切","切不可","切莫","切切","切勿","竊","親口","親身","親手","親眼","親自","頃","頃刻","頃刻間","頃刻之間","請勿","窮年累月","取道","去","權時","全都","全力","全年","全然","全身心","然","人人","仍","仍舊","仍然","日復一日","日見","日漸","日益","日臻","如常","如此等等","如次","如今","如期","如前所述","如上","如下","汝","三番兩次","三番五次","三天兩頭","瑟瑟","沙沙","上","上來","上去","一.","一一","一下","一個","一些","一何","一則通過","一天","一定","一時","一次","一片","一番","一直","一致","一起","一轉眼","一邊","一面","上升","上述","上面","下","下列","下去","下來","下面","不一","不久","不變","不可","不夠","不盡","不盡然","不敢","不斷","不若","不足","與其說","專門","且不說","且說","嚴格","嚴重","個別","中小","中間","豐富","為主","為什麼","為止","為此","主張","主要","舉行","乃至于","之前","之后","之後","也就是說","也是","了解","爭取","二來","云爾","些","亦","產生","人","人們","什麼","今","今后","今天","今年","今後","介于","從事","他是","他的","代替","以上","以下","以為","以前","以后","以外","以後","以故","以期","以來","任務","企圖","偉大","似乎","但凡","何以","余外","你是","你的","使","使用","依據","依靠","便于","促進","保持","做到","儻然","兒","允許","元/噸","先不先","先后","先後","先生","全體","全部","全面","共同","具體","具有","兼之","再","再其次","再則","再有","再次","再者說","決定","準備","凡","凡是","出于","出現","分別","則甚","別處","別是","別管","前此","前進","前面","加入","加強","十分","即如","卻","卻不","原來","又及","及時","雙方","反應","反映","取得","受到","變成","另悉","只","只當","只怕","只消","叫做","召開","各人","各地","各級","合理","同一","同樣","后","后者","后面","向使","周圍","呵呵","咧","唯有","啷當","嘍","嗡","嘿嘿","因了","因著","在于","堅決","堅持","處在","處理","復雜","多么","多數","大力","大多數","大批","大量","失去","她是","她的","好","好的","好象","如同","如是","始而","存在","孰料","孰知","它們的","它是","它的","安全","完全","完成","實現","實際","宣布","容易","密切","對應","對待","對方","對比","小","少數","爾","爾爾","尤其","就是了","就要","屬于","左右","巨大","鞏固","已","已矣","已經","巴","巴巴","幫助","并不","并不是","廣大","廣泛","應當","應用","應該","庶乎","庶幾","開展","引起","強烈","強調","歸齊","當前","當地","當時","形成","徹底","彼時","往往","後來","後面","得了","得出","得到","心里","必然","必要","怎奈","怎麼","總是","總結","您們","您是","惟其","意思","愿意","成為","我是","我的","或則","或曰","戰斗","所在","所幸","所有","所謂","擴大","掌握","接著","數/","整個","方便","方面","無","無法","既往","明顯","明確","是不是","是以","是否","顯然","顯著","普通","普遍","曾","曾經","替代","最","最后","最大","最好","最後","最近","最高","有利","有力","有及","有所","有效","有時","有點","有的是","有著","有著","末##末","本地","來自","來說","構成","某某","根本","歡迎","歟","正值","正在","正巧","正常","正是","此地","此處","此時","此次","每個","每天","每年","比及","比較","沒奈何","注意","深入","清楚","滿足","然後","特別是","特殊","特點","猶且","猶自","現代","現在","甚且","甚或","甚至于","用來","由是","由此","目前","直到","直接","相似","相信","相反","相同","相對","相應","相當","相等","看出","看到","看看","看見","真是","真正","眨眼","矣乎","矣哉","知道","確定","種","積極","移動","突出","突然","立即","竟而","第二","類如","練習","組成","結合","繼后","繼續","維持","考慮","聯系","能否","能夠","自后","自打","至今","至若","致","般的","良好","若夫","若果","范圍","莫不然","獲得","行為","行動","表明","表示","要求","規定","覺得","譬喻","認為","認真","認識","許多","設或","誠如","說明","說來","說說","諸","諸如","誰人","誰料","賊死","賴以","距","轉動","轉變","轉貼","達到","迅速","過去","過來","運用","還要","這一來","這次","這點","這種","這般","這麼","進入","進步","進行","適應","適當","適用","逐步","逐漸","通常","造成","遇到","遭到","遵循","避免","那般","那麼","部分","采取","里面","重大","重新","重要","針對","問題","防止","附近","限制","隨后","隨時","隨著","難道說","集中","需要","非特","非獨","高興","若果 "]
文章版權歸作者所有,未經允許請勿轉載,若此文章存在違規行為,您可以聯系管理員刪除。
轉載請注明本文地址:http://specialneedsforspecialkids.com/yun/44383.html
摘要:表示學習和深度學習的興起是密切相關。自然語言處理中的深度學習在自然語言的表示學習中提及深度學習這是因為深度學習首要的用處就是進行自然語言的表示。圖是深度學習在自然語言理解中應用描述。 本文根據達觀數據特聘專家復旦大學黃萱菁教授在達觀數據舉辦的長三角人工智能應用創新張江峰會上的演講整理而成,達觀數據副總裁魏芳博士統稿 一、概念 1 什么是自然語言和自然語言理解? 自然語言是指漢語、英語、...
閱讀 881·2023-04-25 19:17
閱讀 2179·2021-09-10 11:26
閱讀 1898·2019-08-30 15:54
閱讀 3411·2019-08-30 15:53
閱讀 2681·2019-08-30 11:20
閱讀 3392·2019-08-29 15:12
閱讀 1230·2019-08-29 13:16
閱讀 2384·2019-08-26 12:19