Keyword: tokenizer

unbash
Released
2 weeks ago
Version
4.0.1
Fast 0-deps bash parser written in TypeScript
ast bash lexer parser posix scanner +2
@projectwallace/css-parser
Released
4d ago
Version
0.16.2
High-performance CSS lexer and parser, optimized for CSS inspection and analysis
ast css lexer parser tokenizer
garu-ko
Released
5d ago
Version
0.9.6
Ultra-lightweight Korean morphological analyzer for the web (1MB model, WASM, F1 93.7% NIKL MP)
korean morphological-analysis korean-nlp pos-tagging tokenizer korean-tokenizer +10
@mineproness/tokenizer
Released
2d ago
Version
1.1.1
The @mineproness/tokenizer make Easier to Split or make Tokens IDs without any Dependencies. I Write own Code for Raw Javascript.
AI LLM TOKENS IDS TOKENIZER
yume-dsl-rich-text
Released
21h ago
Version
1.5.1
Single-pass recursive rich-text DSL parser without regex, with pluggable tag handlers. Markdown alternative.
rich-text dsl parser custom-markup single-pass recursive +6
@regmisatyam/retex
Released
3d ago
Version
0.3.0
ReTeX — a modern, high-performance LaTeX-inspired markup language and compiler for resumes, research papers, and developer portfolios.
resume cv latex markup compiler parser +7
@promptwise-dev/core
Released
5d ago
Version
1.0.1
The shared PromptWise engine: heuristic prompt rewriting, context-flood detection, global memory, and persona inference. Zero dependencies; runs in Node and the browser.
prompt prompt-engineering llm tokens tokenizer optimization +4
nlp-js-tools-french
Released
4 years ago
Version
1.0.9
POS Tagger and lemmatizer for javascript
nlp tokenizer tokenize pos tagger Part-Of-Speech +7
tokenfit
Released
6d ago
Version
0.1.1
Fit text into LLM token budgets — estimate, trim, and pack prompts with zero dependencies.
llm tokens tokenizer prompt context-window openai +6
@altopelago/aeon-lexer
Released
18h ago
Version
0.9.6
Tokenize AEON source text for parsers, editors, and tooling.
aeon aeonite lexer tokenizer tokens syntax +1
lindera-nodejs-win32-x64-msvc
Released
2 months ago
Version
3.0.7
Node.js bindings for Lindera morphological analysis engine
morphological analysis tokenizer japanese korean chinese +2
garu-minisearch-tokenizer
Released
5d ago
Version
0.4.1
Korean tokenizer for MiniSearch, powered by garu-ko (1MB browser-native Korean morphological analyzer)
minisearch tokenizer korean search wasm morphological-analysis +1
garu-orama-tokenizer
Released
5d ago
Version
0.4.1
Korean tokenizer for Orama search, powered by garu-ko (1MB browser-native Korean morphological analyzer)
orama tokenizer korean search wasm morphological-analysis +1
lindera-nodejs-linux-arm64-gnu
Released
2 months ago
Version
3.0.7
Node.js bindings for Lindera morphological analysis engine
morphological analysis tokenizer japanese korean chinese +2
lindera-nodejs-darwin-x64
Released
6d ago
Version
4.0.0
Node.js bindings for Lindera morphological analysis engine
morphological analysis tokenizer japanese korean chinese +2
creditcards-tokenizer
Released
1h ago
Version
1.2.902
Wix Restaurants credit-cards tokenizer
credit cards tokenizer tokenization pci
lexer-next
Released
3 years ago
Version
4.1.1
lexer for recursive descent parsers
lexer tokenizer parse parser
expression-engine
Released
2 years ago
Version
1.11.1
An expression tokenizer, parser and evaluator.
expression tokenizer parser evaluator
@applitools/css-tree
Released
20h ago
Version
1.2.0
A temporary custom copy of https://github.com/csstree/csstree
css ast tokenizer parser walker lexer +4
konverter
Released
20h ago
Version
0.1.1
Wandelt Tabellen (Excel/CSV/JSON) ins tokensparendste Format fürs LLM — lokal & modellbewusst. CLI + Browser-Tool.
llm tokens tokenizer csv json xlsx +5
qb-json-strict
Released
19h ago
Version
1.0.3
Opt-in RFC 8259 number/string content validation layer over qb-json-next
json validate rfc8259 conformance tokenizer qb +1
@hdnax/sqlingo.js
Released
2d ago
Version
0.4.2
SQL parser, transpiler, optimizer, and engine for JavaScript/TypeScript - 33+ dialects (Postgres, MySQL, BigQuery, Snowflake, DuckDB, etc). Port of Python SQLGlot.
sql tokenizer parser transpiler optimizer sqlingo +14
string-punctuation-tokenizer
Released
2d ago
Version
2.2.0
Small library that provides functions to tokenize a string into an array of words with or without punctuation
string punctuation tokenizer
@msm-core/cst
Released
2 weeks ago
Version
0.3.2
Contextual Semantic Tokenizer — zero-dependency, multilingual, reversible
tokenizer semantic nlp arabic english hdc +1
underthesea-wasm
Released
4d ago
Version
0.1.1
Vietnamese word segmentation in JS/TS — an unofficial in-process WebAssembly port of underthesea's word_tokenize. Exact parity, no Python.
vietnamese nlp word-segmentation tokenizer word-tokenize underthesea +2