Unsupervised text tokenizer for Neural Network-based text generation.
-
Updated
Mar 1, 2026 - C++
#
word-segmentation
Here are 152 public repositories matching this topic...
Language: All
Filter by language
Sort: Most stars
Sort options
Bai Du NLP:Fen Ci ,Ci Xing Biao Zhu ,Ming Ming Shi Ti Shi Bie ,Ci Zhong Yao Xing
python java named-entity-recognition lexical-analysis chinese-nlp word-segmentation part-of-speech-tagger chinese-word-segmentation
-
Updated
May 25, 2021 - C++
SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm
spellcheck fuzzy-search fuzzy-matching edit-distance levenshtein levenshtein-distance spelling spell-check chinese-text-segmentation word-segmentation approximate-string-matching spelling-correction damerau-levenshtein text-segmentation chinese-word-segmentation symspell
-
Updated
Jan 20, 2026 - C#
Thai natural language processing in Python
python natural-language-processing thai-language thai computational-linguistics text-processing soundex nlp-library word-segmentation thai-nlp hacktoberfest thai-nlp-library thai-soundex hacktoberfest-accepted
-
Updated
Mar 3, 2026 - Python
Unsupervised text tokenizer focused on computational efficiency
-
Updated
Mar 29, 2024 - C++
Python port of SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm
python spellcheck fuzzy-search fuzzy-matching edit-distance levenshtein levenshtein-distance spelling spell-check chinese-text-segmentation word-segmentation approximate-string-matching spelling-correction damerau-levenshtein text-segmentation chinese-word-segmentation symspell
-
Updated
Feb 13, 2026 - Python
CKIP Transformers
-
Updated
Apr 21, 2023 - Python
Kiwi(jineunghyeong hangugeo hyeongtaeso bunseoggi)
nlp cpp morphology korean word-segmentation morphological-analysis korean-text-processing korean-tokenizer korean-nlp
-
Updated
Feb 23, 2026 - C++
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization
-
Updated
Jun 2, 2025 - Python
A Vietnamese natural language processing toolkit (NAACL 2018)
java nlp natural-language-processing parsing vietnamese python3 named-entity-recognition ner word-segmentation pos-tagging dependency-parsing pos-tagger vietnamese-nlp sentence-segmentation vietnamese-tokenizer vncorenlp word-segmenter rdrsegmenter vnmarmot
-
Updated
Feb 12, 2023 - Java
BERT for Multitask Learning
nlp text-classification transformer named-entity-recognition pretrained-models part-of-speech ner word-segmentation bert cws encoder-decoder multi-task-learning multitask-learning
-
Updated
Apr 12, 2023 - Jupyter Notebook
AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models
nlp natural-language-processing crf pytorch information-extraction named-entity-recognition chinese-nlp ner word-segmentation bert sequence-labeling relation-extraction natural-language-understanding entity-typing token-classification multi-modal-ner
-
Updated
Nov 15, 2023 - Python
A Japanese tokenizer based on recurrent neural networks
nlp natural-language-processing japanese tokenizer nlp-library word-segmentation dynet pos-tagging sequence-labeling
-
Updated
Feb 12, 2026 - Python
Juman++ (a Morphological Analyzer Toolkit)
nlp japanese tokenizer cjk word-segmentation pos-tagging part-of-speech-tagger morphological-analysis pos-tagger morphological-analyser juman
-
Updated
Oct 3, 2023 - C++
Cantonese Linguistics and NLP
python nlp natural-language-processing linguistics cantonese computational-linguistics word-segmentation jyutping pycantonese stop-words part-of-speech-tagging
-
Updated
May 23, 2024 - Python
Python API for Kiwi
-
Updated
Dec 25, 2025 - Python
Zhong Wen Wen Ben Fen Lei , Xu Lie Biao Zhu Gong Ju Bao (pytorch),Zhi Chi Zhong Wen Chang Wen Ben , Duan Wen Ben De Duo Lei , Duo Biao Qian Fen Lei Ren Wu ,Zhi Chi Zhong Wen Ming Ming Shi Ti Shi Bie , Ci Xing Biao Zhu , Fen Ci , Chou Qu Shi Wen Ben Zhai Yao Deng Xu Lie Biao Zhu Ren Wu . Chinese text classification and sequence labeling toolkit, supports multi class and multi label classification, text similsrity, text summary and NER.
text-classification transformers python3 pytorch named-entity-recognition chinese-text-segmentation pretrained-models word-segmentation bert pos-tagging sequence-labeling chinese-text-classification
-
Updated
Jul 18, 2024 - Python
This repository is archived! The maintained MeCab can be found https://github.com/shogo82148/mecab
-
Updated
Oct 15, 2024 - C++
A PyTorch implementation of the BI-LSTM-CRF model.
nlp crf pytorch ner word-segmentation pos-tagging sequence-labeling bi-lstm-crf bilstm crf-model lstm-crf bilstm-crf sequence-tagging
-
Updated
Oct 30, 2024 - Python
MONPA Wang Pai Shi Yi Ge Ti Gong Zheng Ti Zhong Wen Duan Ci , Ci Xing Biao Zhu Yi Ji Ming Ming Shi Ti Bian Shi De Duo Ren Wu Mo Xing
nlp named-entity-recognition pos ner word-segmentation albert bert pos-tagging chinese-word-segmentation
-
Updated
Feb 20, 2025 - Python
Improve this page
Add a description, image, and links to the word-segmentation topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the word-segmentation topic, visit your repo's landing page and select "manage topics."