Skip to content

Custom FTS5 tokenizers, functions - #421

Merged
ncruces merged 11 commits into
mainfrom
fts5
Aug 31, 2026
Merged

Custom FTS5 tokenizers, functions#421
ncruces merged 11 commits into
mainfrom
fts5

Conversation

@ncruces

@ncruces ncruces commented Aug 25, 2026

Copy link
Copy Markdown
Owner

@ncruces
ncruces changed the base branch from main to mmap August 25, 2026 16:28
@ncruces
ncruces changed the base branch from mmap to main August 25, 2026 22:30
@ncruces
ncruces changed the base branch from main to mmap August 25, 2026 22:33
@ncruces
ncruces changed the base branch from mmap to main August 25, 2026 22:38
@ncruces
ncruces marked this pull request as ready for review August 25, 2026 22:58
NyaaaWhatsUpDoc
NyaaaWhatsUpDoc approved these changes Aug 26, 2026
View reviewed changes

@NyaaaWhatsUpDoc NyaaaWhatsUpDoc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this all looks great to me 👍

NyaaaWhatsUpDoc
NyaaaWhatsUpDoc reviewed Aug 26, 2026
View reviewed changes
Comment thread ext/fts5/fts5.go
@ncruces ncruces changed the title Custom FTS5 tokenizers. Custom FTS5 tokenizers Aug 26, 2026
@ncruces ncruces changed the title Custom FTS5 tokenizers Custom FTS5 tokenizers, functions Aug 27, 2026
@ncruces ncruces mentioned this pull request Aug 27, 2026
Closed
@ncruces ncruces linked an issue Aug 27, 2026 that may be closed by this pull request
Wrap the FTS5 module extension API #54
Open
@ncruces ncruces mentioned this pull request Aug 27, 2026
Open
@ncruces

ncruces commented Aug 28, 2026

Copy link
Copy Markdown
Owner Author

@pgaskin I added minimal tests for everything you requested (so, at least the happy path should work), but would still appreciate your look into this.

@ncruces

ncruces commented Aug 28, 2026

Copy link
Copy Markdown
Owner Author

I still want to see if I can "hide" the exposed internal APIs from godoc, but in theory that's it. This should be ready to ship.

I'd also like to do a release, since there are a few nice unreleased things (a memory configuration fix, mmap_size, JSON v2, UUID in core).

@ncruces

ncruces commented Aug 28, 2026

Copy link
Copy Markdown
Owner Author

I was able to hide almost everything. This turned out to be a huge refactoring PR, but seems good to go.

@NyaaaWhatsUpDoc

NyaaaWhatsUpDoc commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

i'll take a proper look over it again tomorrow, but on brief look the new wrapping context that gets passed to the auxilliary function hooks looks like a really nice, clean API!

@ncruces

ncruces commented Aug 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner Author

Thanks. No invention there, that's exactly how the API works to begin with. 😀

The most weird thing is how iterators were done, which maps to Go iterators, which only really works because it's either one int or 2.

@ncruces
ncruces merged commit bb9d775 into main Aug 31, 2026
47 of 53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@NyaaaWhatsUpDoc NyaaaWhatsUpDoc NyaaaWhatsUpDoc approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Wrap the FTS5 module extension API

2 participants

@ncruces @NyaaaWhatsUpDoc