Vendor custom SQLite build with tamerye enabled

oysteikt commented

2026-04-06 02:04:37 +02:00

Owner

Needed for #23

See usage: https://ytyaru.hatenablog.com/entry/2021/03/09/000000

Needed for #23 See usage: https://ytyaru.hatenablog.com/entry/2021/03/09/000000

oysteikt added the packaging nix labels 2026-04-06 02:04:37 +02:00

oysteikt added this to the Kanban project 2026-04-06 02:04:38 +02:00

oysteikt commented

2026-04-06 02:06:07 +02:00

Author

Owner

Note that we'll need to crossbuild it for android and bundle it with mugiten as well.

It would be nice if this could be done in a way so that interactively running flutter run in the devshell just works

We can use pkgs.pkgsCross.aarch64-android.sqlite

Note that we'll need to crossbuild it for android and bundle it with mugiten as well. It would be nice if this could be done in a way so that interactively running `flutter run` in the devshell just works We can use `pkgs.pkgsCross.aarch64-android.sqlite`

oysteikt added a new dependency 2026-04-06 02:06:50 +02:00

#23 word search: fuzzy kana search

oysteikt commented

2026-04-06 05:23:34 +02:00

Author

Owner

Turns out that the builtin ICU won't do it, as mentioned in ytyaru's article LIKE does not work across the kana boundary.

Consider building https://hg.sr.ht/~cwt/fts5-icu-tokenizer

See also chapter 6 about static linkage: https://sqlite.org/loadext.html

Turns out that the builtin ICU won't do it, as mentioned in ytyaru's article `LIKE` does not work across the kana boundary. Consider building https://hg.sr.ht/~cwt/fts5-icu-tokenizer See also chapter 6 about static linkage: https://sqlite.org/loadext.html

oysteikt moved this to Working in Kanban on 2026-04-06 05:31:08 +02:00

oysteikt commented

2026-05-05 18:59:26 +02:00

Author

Owner

I built https://hg.sr.ht/~cwt/fts5-icu-tokenizer and did some interactive testing, but it seemingly only helps with word splitting - not kana transliteration 🫤

oysteikt referenced this issue from mugiten/tamerye

2026-05-07 11:42:46 +02:00

Add a collation that equates hira/katakana #6

oysteikt changed title from ~~Vendor custom SQLite build with ICU extension enabled~~ to Vendor custom SQLite build with tamerye enabled

2026-05-22 04:03:31 +02:00

oysteikt moved this to Next Version in Kanban on 2026-05-31 19:03:27 +02:00

oysteikt moved this to High pri in Kanban on 2026-06-03 09:30:05 +02:00

oysteikt commented

2026-06-03 18:17:43 +02:00

Author

Owner

It doesn't seem like it will be any easy to vendor a fully custom built static sqlite with tamerye embedded until android cross compilation is fixed in nixpkgs. However, until then, we could use the natively compiled counterpart for building jadb, and use a dynamically loaded extension in the app. How bout that?

oysteikt moved this to Working in Kanban on 2026-06-03 19:12:18 +02:00

oysteikt added reference build-with-tamerye

2026-06-03 19:12:23 +02:00

oysteikt commented

2026-06-03 19:49:45 +02:00

Author

Owner

See also: https://github.com/tekartik/sqflite/issues/1149

oysteikt commented

2026-06-04 18:01:29 +02:00

Author

Owner

The relevant branch for jadb just works now, hijacking this issue for figuring out how to embed a custom sqlite into mugiten.

There seems to be a split between mobile platforms + mac vs everything else, where the previous have 1st grade support and prebuilt binaries that will be baked into the app by sqflite itself, while the other ones depend on the sqflite_common_ffi package. The ffi package earlier used something called sqlite3_flutter_libs, but after dart 3.10.0 we use the sqlite3 package instead.

The sqlite3 is an early adopter of dart hooks, which lets you customize the build. However, it seems like the options available assume that you want to either use the hook to compile a new version from an amalgamation, or find a system version of sqlite3 via dlopen, dynamic linking or static linking. I don't think the two last options are an alternative with APK an java bytecode, but maybe we can somehow teach dlopen a full path to a sqlite.so packed out from the assets?

For now, it might also be possible to do a custom compile with the flags required to enable extension loading, and just bundle the extension in assets and pack it out before opening the sqlite database, so we can instantly load the extension.

https://github.com/tekartik/sqflite/blob/master/sqflite_common_ffi/doc/using_ffi_instead_of_sqflite.md
https://github.com/tekartik/sqflite/blob/master/sqflite/doc/qa.md#cross-platform-support
https://github.com/simolus3/sqlite3.dart
https://github.com/simolus3/sqlite3.dart/tree/main/sqlite3
https://github.com/simolus3/sqlite3.dart/blob/main/sqlite3/doc/hook.md

The relevant branch for jadb just works now, hijacking this issue for figuring out how to embed a custom sqlite into mugiten. There seems to be a split between mobile platforms + mac vs everything else, where the previous have 1st grade support and prebuilt binaries that will be baked into the app by `sqflite` itself, while the other ones depend on the `sqflite_common_ffi` package. The ffi package earlier used something called `sqlite3_flutter_libs`, but after dart 3.10.0 we use the `sqlite3` package instead. The sqlite3 is an early adopter of dart hooks, which lets you customize the build. However, it seems like the options available assume that you want to either use the hook to compile a new version from an amalgamation, or find a system version of sqlite3 via dlopen, dynamic linking or static linking. I don't think the two last options are an alternative with APK an java bytecode, but maybe we can somehow teach dlopen a full path to a sqlite.so packed out from the assets? For now, it might also be possible to do a custom compile with the flags required to enable extension loading, and just bundle the extension in assets and pack it out before opening the sqlite database, so we can instantly load the extension. https://github.com/tekartik/sqflite/blob/master/sqflite_common_ffi/doc/using_ffi_instead_of_sqflite.md https://github.com/tekartik/sqflite/blob/master/sqflite/doc/qa.md#cross-platform-support https://github.com/simolus3/sqlite3.dart https://github.com/simolus3/sqlite3.dart/tree/main/sqlite3 https://github.com/simolus3/sqlite3.dart/blob/main/sqlite3/doc/hook.md

oysteikt commented

2026-06-04 18:28:11 +02:00

Author

Owner

https://github.com/simolus3/sqlite3.dart/blob/main/sqlite3/doc/hook.md#default-binaries claims that the binaries are compiled without setting SQLITE_OMIT_LOAD_EXTENSION, maybe loading the extension already from assets already just works?

https://github.com/simolus3/sqlite3.dart/blob/main/sqlite3/doc/hook.md#default-binaries claims that the binaries are compiled without setting `SQLITE_OMIT_LOAD_EXTENSION`, maybe loading the extension already from assets already just works?

oysteikt commented

2026-06-08 17:26:31 +02:00

Author

Owner

https://github.com/sqliteai/sqlite-extensions-guide

.aar (Android Archive) is an Android-specific package format that bundles compiled code, resources, and native libraries (.so files) for distribution via Maven Central or JitPack. When you add an .aar dependency to your Android project, the native .so libraries are automatically extracted to the device's nativeLibraryDir at installation time, where they can be loaded as SQLite extensions. This is the standard way to distribute SQLite extensions for Android apps.

👀

https://github.com/sqliteai/sqlite-extensions-guide > **.aar** (Android Archive) is an Android-specific package format that bundles compiled code, resources, and native libraries (`.so` files) for distribution via Maven Central or JitPack. When you add an `.aar` dependency to your Android project, the native `.so` libraries are automatically extracted to the device's `nativeLibraryDir` at installation time, where they can be loaded as SQLite extensions. This is the standard way to distribute SQLite extensions for Android apps. 👀

oysteikt commented

2026-06-08 17:30:55 +02:00

Author

Owner

https://github.com/simolus3/sqlite3.dart/blob/main/sqlite3/example/custom_extension/README.md