lexical-analysis

Star

A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.

Here are 805 public repositories matching this topic...

Language: All

Filter by language

All 805 C 196 Python 117 C++ 110 Java 103 Yacc 39 C# 29 Go 24 JavaScript 24 Lex 21 Jupyter Notebook 18

Sort: Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

DoctorWkt / acwj

Star 12.8k

A Compiler Writing Journey

c parsing compiler lexical-analysis

Updated Sep 24, 2025
C

baidu / lac

Star 4k

Bai Du NLP:Fen Ci ,Ci Xing Biao Zhu ,Ming Ming Shi Ti Shi Bie ,Ci Zhong Yao Xing

python java named-entity-recognition lexical-analysis chinese-nlp word-segmentation part-of-speech-tagger chinese-word-segmentation

Updated May 25, 2021
C++

SeanLee97 / xmnlp

Star 1.3k

xmnlp:Ti Gong Zhong Wen Fen Ci , Ci Xing Biao Zhu , Ming Ming Ti Shi Bie ,Qing Gan Fen Xi ,Wen Ben Jiu Cuo ,Wen Ben Zhuan Pin Yin ,Wen Ben Zhai Yao ,Pian Pang Bu Shou ,Ju Zi Biao Zheng Ji Wen Ben Xiang Si Du Ji Suan Deng Gong Neng

nlp sentiment-analysis pinyin lexical-analysis segmentation spell-checker ner radical postagging sentence-similarity sentence-embeddings

Updated Nov 12, 2022
Python

katef / libfsm

Sponsor

Star 978

DFA regular expression library & friends

automata regexes fsm compiler regex regexp lex finite-state-machine lexer lexical-analysis compilers lexing compiler-principles lexer-generator nfa regex-validator finite-state-automata finite-state-machines dfa

Updated Feb 10, 2026
C

DmitryRyumin / INTERSPEECH-2023-24-Papers

Star 687

INTERSPEECH 2023-2024 Papers: A complete collection of influential and exciting research papers from the INTERSPEECH 2023-24 conference. Explore the latest advances in speech and language processing. Code included. Star the repository to support the advancement of speech technology!

signal-processing machine-translation transmission language-modeling speech-synthesis speech-recognition lexical-analysis acoustic prosody asr speech-analysis adaptation linguistic-analysis speech-production audio-signals interspeech self-supervised-learning speech-technology interspeech2023 interspeech2024

Updated Dec 25, 2024

qinwf / jiebaR

Star 362

Chinese text segmentation with R. RYu Yan Zhong Wen Fen Ci (Wen Dang Yi Geng Xin :https://qinwenfeng.com/jiebaR/ )

nlp chinese lexical-analysis cppjieba jieba chinese-text-segmentation

Updated Jul 13, 2020
C++

WGrape / lexer

Star 345

A lexical analyzer based on DFA that is built using JS and supports multi-language extensions / Yi Ge Ji Yu DFADe Zhi Chi Duo Yu Yan Kuo Zhan De JSBan Kai Yuan Ci Fa Fen Xi Qi

javascript lexer lexical-analysis dfa lexical-analyzer

Updated Dec 4, 2025
JavaScript

ghaiklor / llvm-kaleidoscope

Star 268

LLVM Tutorial: Kaleidoscope (Implementing a Language with LLVM)

language llvm lexer lexical-analysis kaleidoscope llvm-tutorial llvm-ir

Updated Dec 29, 2022
C++

gecko0307 / dlib

Star 226

Allocators, I/O streams, math, geometry, image and audio processing for D

networking containers linear-algebra dlang image-processing json-parser computational-geometry threading lexical-analysis xml-parser utility-library memory-allocation audio-processing

Updated Jan 1, 2026
D

drifter1 / compiler

Star 188

This is an implementation of a compiler for a simple C-like language that uses the C tools Flex and Bison.

c parser bison flex compiler mips assembly mips-assembly lexer lexical-analysis symbol-table syntax-analysis compiler-design machine-code semantic-analysis

Updated Feb 26, 2026
C

SwiftStudies / OysterKit

Star 177

OysterKit is a framework that provides a native Swift scanning, lexical analysis, and parsing capabilities. In addition it provides a language that can be used to rapidly define the rules used by OysterKit called STLR

swift language decoder lexical-analysis parsing-capabilities

Updated Aug 15, 2019
Swift

AnjaneyaTripathi / c-compiler

Star 143

A compiler that accepts any valid program written in C. It is made using Lex and Yacc. Returns a symbol table, parse tree, annotated syntax tree and intermediate code.