code_editor_tokenizer.hpp File Reference

Lightweight per-language tokenizers for CodeEditor syntax highlighting. More...

#include <cstddef>
#include <string_view>
#include <vector>
#include <pulp/view/code_editor.hpp>

Go to the source code of this file.

Classes
struct	pulp::view::Token
	One token span — half-open [start, start+length) into the line. More...

Namespaces
namespace	pulp
namespace	pulp::view
	Hover-cursor tracking for platform window hosts.

Enumerations
enum class	pulp::view::TokenClass {   pulp::view::text , pulp::view::keyword , pulp::view::type , pulp::view::identifier ,   pulp::view::number , pulp::view::string , pulp::view::comment , pulp::view::preprocessor ,   pulp::view::punctuation , pulp::view::heading , pulp::view::link }
	Token classification for tokenize_line. More...

Functions
std::vector< Token >	pulp::view::tokenize_line (std::string_view line, CodeLanguage language)
	Tokenize a single line of source text for language.
std::vector< Token >	pulp::view::tokenize_cpp_line (std::string_view line)
std::vector< Token >	pulp::view::tokenize_javascript_line (std::string_view line)
std::vector< Token >	pulp::view::tokenize_python_line (std::string_view line)
std::vector< Token >	pulp::view::tokenize_json_line (std::string_view line)
std::vector< Token >	pulp::view::tokenize_lua_line (std::string_view line)
std::vector< Token >	pulp::view::tokenize_markdown_line (std::string_view line)
bool	pulp::view::is_keyword (std::string_view name, CodeLanguage language)
	True when name is one of the keywords the tokenizer recognizes for language.

Detailed Description

Lightweight per-language tokenizers for CodeEditor syntax highlighting.

Each call to tokenize_line(line, language) walks a single source line and returns a flat list of Token spans the editor can color. The tokenizers are deliberately tiny — they are NOT full parsers, just enough state to color keywords, strings, comments, numbers, and preprocessor directives the way a typical IDE skim does. Multi-line constructs (block comments, multi-line strings) are reported on the first line only; the editor paints each line independently in the existing native fallback paint path.

All tokenizers are stateless, allocation-free per token, and run in O(n) over the input characters — safe to call on every paint without caching. Headless-friendly: no platform / canvas dependencies.

License-lineage note: the tokenizer keyword lists are derived from the official language references (ISO C++, ECMAScript, Python, CommonMark, ECMA-404 JSON, Lua manual) — no copy from the reference framework's tokeniser sources.