WIP

This was earlier used to compensate for a double nesting bug. This has
been fixed in the latest version of the xml package.

.gitea/workflows/build-and-test.yml

@@ -4,7 +4,7 @@ on:
   pull_request:
   push:
 jobs:
-  evals:
+  build:
     runs-on: debian-latest
     steps:
     - uses: actions/checkout@v6
@@ -69,3 +69,6 @@ jobs:
         ssh-key: ${{ secrets.OYSTEIKT_GITEA_WEBDOCS_SSH_KEY }}
         host: microbel.pvv.ntnu.no
         known-hosts: "microbel.pvv.ntnu.no ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEq0yasKP0mH6PI6ypmuzPzMnbHELo9k+YB5yW534aKudKZS65YsHJKQ9vapOtmegrn5MQbCCgrshf+/XwZcjbM="
+
+    - name: Run benchmarks
+      run: nix develop .# --command dart run benchmark_harness:bench --flavor jit

README.md

@@ -18,26 +18,4 @@ Note that while the license for the code is MIT, the data has various licenses.
 | **Tanos JLPT levels:** | https://www.tanos.co.uk/jlpt/ |
 | **Kangxi Radicals:**   | https://ctext.org/kangxi-zidian |
 
-## Implementation details
-
-### Word search
-
-The word search procedure is currently split into 3 parts:
-
-1. **Entry ID query**:
-
-Use a complex query with various scoring factors to try to get list of
-database ids pointing at dictionary entries, sorted by how likely we think this
-word is the word that the caller is looking for. The output here is a `List<int>`
-
-2. **Data Query**:
-
-Takes the entry id list from the last search, and performs all queries needed to retrieve
-all the dictionary data for those IDs. The result is a struct with a bunch of flattened lists
-with data for all the dictionary entries. These lists are sorted by the order that the ids
-were provided.
-
-3. **Regrouping**:
-
-Takes the flattened data, and regroups the items into structs with a more "hierarchical" structure.
-All data tagged with the same ID will end up in the same struct. Returns a list of these structs.
+See [docs/overview.md](./docs/overview.md) for notes and implementation details.

benchmark/benchmark.dart

@@ -0,0 +1,5 @@
+import './search/word_search.dart';
+
+Future<void> main() async {
+  await WordSearchBenchmark.main();
+}

benchmark/search/word_search.dart

@@ -0,0 +1,45 @@
+import 'package:benchmark_harness/benchmark_harness.dart';
+import 'package:jadb/search.dart';
+import 'package:sqflite_common/sqlite_api.dart';
+
+import '../../test/search/setup_database_connection.dart';
+
+class WordSearchBenchmark extends AsyncBenchmarkBase {
+  Database? connection;
+
+  static final List<String> searchTerms = [
+    'kana',
+    'kanji',
+    'kawaii',
+    'sushi',
+    'ramen',
+  ];
+
+  WordSearchBenchmark() : super('WordSearchBenchmark');
+
+  static Future<void> main() async {
+    print('Running WordSearchBenchmark...');
+    await WordSearchBenchmark().report();
+    print('Finished WordSearchBenchmark');
+  }
+
+  @override
+  Future<void> setup() async {
+    connection = await setupDatabaseConnection();
+  }
+
+  @override
+  Future<void> run() async {
+    for (final term in searchTerms) {
+      await connection!.jadbSearchWord(term);
+    }
+  }
+
+  @override
+  Future<void> teardown() async {
+    await connection?.close();
+  }
+
+  // @override
+  // Future<void> exercise() => run();
+}

docs/lemmatizer.md

@@ -0,0 +1,13 @@
+# Lemmatizer
+
+The lemmatizer is still quite experimental, but will play a more important role in the project in the future.
+
+It is a manual implementation of a [Finite State Transducer](https://en.wikipedia.org/wiki/Morphological_dictionary#Finite_State_Transducers) for morphological parsing. The FST is used to recursively remove affixes from a word until it (hopefully) deconjugates into its dictionary form. This iterative deconjugation tree will then be combined with queries into the dictionary data to determine if the deconjugation leads to a real known word.
+
+Each separate rule is a separate static object declared in `lib/util/lemmatizer/rules`.
+
+There is a cli subcommand for testing the tool interactively, you can run
+
+```bash
+dart run jadb lemmatize -w '食べさせられない'
+```

docs/overview.md

@@ -0,0 +1,27 @@
+# Overview
+
+This is the documentation for `jadb`. Since I'm currently the only one working on it, the documentation is more or less just notes to myself, to ensure I remember how and why I implemented certain features in a certain way a few months down the road. This is not a comprehensive and formal documentation for downstream use, neither for developers nor end-users.
+
+- [Word Search](./word-search.md)
+- [Lemmatizer](./lemmatizer.md)
+
+## Project structure
+
+- `lib/_data_ingestion` contains all the code for reading data sources, transforming them and compiling them into an SQLite database. This is for the most part isolated from the rest of the codebase, and should not be depended on by any code used for querying the database.
+- `lib/cli` contains code for cli tooling (e.g. argument parsing, subcommand handling, etc.)
+- `lib/const_data` contains database data that is small enough to warrant being hardcoded as dart constants.
+- `lib/models` contains all the code for representing the database schema as Dart classes, and for converting between those classes and the actual database.
+- `lib/search` contains all the code for searching the database.
+- `lib/util/lemmatizer` contains the code for lemmatization, which will be used by the search code in the future.
+- `migrations` contains raw SQL files for creating the database schema.
+
+## SQLite naming conventions
+
+> [!WARNING]
+> All of these conventions are actually not enforced yet, it will be fixed at some point.
+
+- Indices are prefixed with `IDX__`
+- Crossref tables are prefixed with `XREF__`
+- Trigger names are prefixed with `TRG__`
+- Views are prefixed with `VW__`
+- All data sources should have a `<datasource>_Version` table, which contains a single row with the version of the data source used to generate the database.

docs/word-search.md

@@ -0,0 +1,21 @@
+# Word search
+
+The word search procedure is currently split into 3 parts:
+
+1. **Entry ID query**:
+
+Use a complex query with various scoring factors to try to get list of
+database ids pointing at dictionary entries, sorted by how likely we think this
+word is the word that the caller is looking for. The output here is a `List<int>`
+
+2. **Data Query**:
+
+Takes the entry id list from the last search, and performs all queries needed to retrieve
+all the dictionary data for those IDs. The result is a struct with a bunch of flattened lists
+with data for all the dictionary entries. These lists are sorted by the order that the ids
+were provided.
+
+3. **Regrouping**:
+
+Takes the flattened data, and regroups the items into structs with a more "hierarchical" structure.
+All data tagged with the same ID will end up in the same struct. Returns a list of these structs.

flake.lock

@@ -3,7 +3,7 @@
     "jmdict-src": {
       "flake": false,
       "locked": {
-        "narHash": "sha256-lh46uougUzBrRhhwa7cOb32j5Jt9/RjBUhlVjwVzsII=",
+        "narHash": "sha256-eOc3a/AYNRFF3w6lWhyf0Sh92xeXS7+9Qvn0tvvH6Ys=",
         "type": "file",
         "url": "http://ftp.edrdg.org/pub/Nihongo/JMdict_e.gz"
       },
@@ -15,7 +15,7 @@
     "jmdict-with-examples-src": {
       "flake": false,
       "locked": {
-        "narHash": "sha256-5oS2xDyetbuSM6ax3LUjYA3N60x+D3Hg41HEXGFMqLQ=",
+        "narHash": "sha256-nx+WMkscWvA/XImKM7NESYVmICwSgXWOO1KPXasHY94=",
         "type": "file",
         "url": "http://ftp.edrdg.org/pub/Nihongo/JMdict_e_examp.gz"
       },
@@ -27,7 +27,7 @@
     "kanjidic2-src": {
       "flake": false,
       "locked": {
-        "narHash": "sha256-orSeQqSxhn9TtX3anYtbiMEm7nFkuomGnIKoVIUR2CM=",
+        "narHash": "sha256-2T/cAS/kZmVMURStgHVhz524+J9+v5onKs8eEYf2fY0=",
         "type": "file",
         "url": "https://www.edrdg.org/kanjidic/kanjidic2.xml.gz"
       },
@@ -36,29 +36,13 @@
         "url": "https://www.edrdg.org/kanjidic/kanjidic2.xml.gz"
       }
     },
-    "kanjivg-src": {
-      "flake": false,
-      "locked": {
-        "lastModified": 1772352482,
-        "narHash": "sha256-8EG3Y1daI2B24NELQwU+eXl/7OmWnW/RXMAQSRVLzWw=",
-        "ref": "refs/heads/master",
-        "rev": "0b4309cf6d74799b0e4b72940d8267fbe73f72d0",
-        "revCount": 2212,
-        "type": "git",
-        "url": "https://git.pvv.ntnu.no/mugiten/kanjivg.git"
-      },
-      "original": {
-        "type": "git",
-        "url": "https://git.pvv.ntnu.no/mugiten/kanjivg.git"
-      }
-    },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1771848320,
-        "narHash": "sha256-0MAd+0mun3K/Ns8JATeHT1sX28faLII5hVLq0L3BdZU=",
+        "lastModified": 1774386573,
+        "narHash": "sha256-4hAV26quOxdC6iyG7kYaZcM3VOskcPUrdCQd/nx8obc=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "2fc6539b481e1d2569f25f8799236694180c0993",
+        "rev": "46db2e09e1d3f113a13c0d7b81e2f221c63b8ce9",
         "type": "github"
       },
       "original": {
@@ -84,7 +68,6 @@
         "jmdict-src": "jmdict-src",
         "jmdict-with-examples-src": "jmdict-with-examples-src",
         "kanjidic2-src": "kanjidic2-src",
-        "kanjivg-src": "kanjivg-src",
         "nixpkgs": "nixpkgs",
         "radkfile-src": "radkfile-src"
       }

flake.nix

@@ -24,11 +24,6 @@
       url = "https://www.edrdg.org/kanjidic/kanjidic2.xml.gz";
       flake = false;
     };
-
-    kanjivg-src = {
-      url = "git+https://git.pvv.ntnu.no/mugiten/kanjivg.git";
-      flake = false;
-    };
   };
 
   outputs = {
@@ -37,8 +32,7 @@
     jmdict-src,
     jmdict-with-examples-src,
     radkfile-src,
-    kanjidic2-src,
-    kanjivg-src,
+    kanjidic2-src
   }: let
     inherit (nixpkgs) lib;
     systems = [
@@ -49,7 +43,12 @@
       "armv7l-linux"
     ];
 
-    forAllSystems = f: lib.genAttrs systems (system: f system nixpkgs.legacyPackages.${system});
+    forAllSystems = f: lib.genAttrs systems (system: let
+      pkgs = import nixpkgs {
+        inherit system;
+        overlays = [ self.overlays.sqlite-icu-ext ];
+      };
+    in f system pkgs);
   in {
     apps = forAllSystems (system: pkgs: {
       default = {
@@ -83,15 +82,12 @@
 
     devShells = forAllSystems (system: pkgs: {
       default = pkgs.mkShell {
-        buildInputs = with pkgs; [
+        packages = with pkgs; [
           dart
           gnumake
           lcov
-          sqlite-analyzer
-          sqlite-interactive
-          sqlite-web
-          # sqlint
-          sqlfluff
+          sqldiff
+          sqlite-interactive-icu-ext
         ];
         env = {
           LIBSQLITE_PATH = "${pkgs.sqlite.out}/lib/libsqlite3.so";
@@ -99,8 +95,34 @@
           LD_LIBRARY_PATH = lib.makeLibraryPath [ pkgs.sqlite ];
         };
       };
+
+      sqlite-debugging = pkgs.mkShell {
+        packages = with pkgs; [
+          sqlite-interactive-icu-ext
+          sqlite-analyzer
+          sqlite-web
+          sqlint
+          sqlfluff
+        ];
+      };
     });
 
+    overlays.sqlite-icu-ext = final: prev: let
+      overrideArgs = prev': {
+        configureFlags = prev'.configureFlags ++ [
+          "--with-icu-config=${lib.getExe' prev.icu.dev "icu-config"}"
+          "--enable-icu-collations"
+        ];
+
+        buildInputs = prev'.buildInputs ++ [
+          prev.icu
+        ];
+      };
+    in {
+      sqlite-icu-ext = prev.sqlite.overrideAttrs overrideArgs;
+      sqlite-interactive-icu-ext = prev.sqlite-interactive.overrideAttrs overrideArgs;
+    };
+
     packages = let
       edrdgMetadata = {
         license = [{
@@ -134,6 +156,8 @@
         ln -s ${src} $out
       '';
 
+      inherit (pkgs) sqlite-icu-ext sqlite-interactive-icu-ext;
+
       jmdict = pkgs.callPackage ./nix/jmdict.nix {
         inherit jmdict-src jmdict-with-examples-src edrdgMetadata;
       };
@@ -148,17 +172,20 @@
 
       database-tool = pkgs.callPackage ./nix/database_tool.nix {
         inherit src;
+        sqlite = pkgs.sqlite-icu-ext;
       };
 
       database = pkgs.callPackage ./nix/database.nix {
         inherit (self.packages.${system}) database-tool jmdict radkfile kanjidic2;
         inherit src;
+        sqlite = pkgs.sqlite-icu-ext;
       };
 
       database-wal = pkgs.callPackage ./nix/database.nix {
         inherit (self.packages.${system}) database-tool jmdict radkfile kanjidic2;
         inherit src;
         wal = true;
+        sqlite = pkgs.sqlite-icu-ext;
       };
 
       docs = pkgs.callPackage ./nix/docs.nix {

lib/_data_ingestion/jmdict/seed_data.dart

@@ -5,13 +5,23 @@ import 'package:jadb/_data_ingestion/jmdict/objects.dart';
 import 'package:jadb/table_names/jmdict.dart';
 import 'package:sqflite_common/sqlite_api.dart';
 
+/// A wrapper for the result of resolving an xref, which includes the resolved entry and a flag
+/// indicating whether the xref was ambiguous (i.e. could refer to multiple entries).
 class ResolvedXref {
   Entry entry;
   bool ambiguous;
+  int? senseOrderNum;
 
-  ResolvedXref(this.entry, this.ambiguous);
+  ResolvedXref(this.entry, this.ambiguous, senseOrderNum);
 }
 
+/// Resolves an xref (pair of kanji, optionally reading, and optionally sense number) to an a specific
+/// JMdict entry, if possible.
+///
+/// If the xref is ambiguous (i.e. it could refer to multiple entries), the
+/// first entry is returned, and the returned value is marked as ambiguous.
+///
+/// If the xref cannot be resolved to any entry at all, an exception is thrown.
 ResolvedXref resolveXref(
   SplayTreeMap<String, Set<Entry>> entriesByKanji,
   SplayTreeMap<String, Set<Entry>> entriesByReading,
@@ -65,9 +75,10 @@ ResolvedXref resolveXref(
       'kanjiRef: ${xref.kanjiRef}, readingRef: ${xref.readingRef}, '
       'senseOrderNum: ${xref.senseOrderNum}',
     );
-    return ResolvedXref(candidateEntries.first, true);
+
+    return ResolvedXref(candidateEntries.first, true, xref.senseOrderNum);
   } else {
-    return ResolvedXref(candidateEntries.first, false);
+    return ResolvedXref(candidateEntries.first, false, xref.senseOrderNum);
   }
 }
 
@@ -152,14 +163,14 @@ Future<void> seedJMDictData(List<Entry> entries, Database db) async {
         b.insert(JMdictTableNames.senseRestrictedToKanji, {
           'entryId': e.entryId,
           'senseId': s.senseId,
-          'kanji': rk,
+          'kanjiOrderNum': e.kanji.indexWhere((k) => k.reading == rk) + 1,
         });
       }
       for (final rr in s.restrictedToReading) {
         b.insert(JMdictTableNames.senseRestrictedToReading, {
           'entryId': e.entryId,
           'senseId': s.senseId,
-          'reading': rr,
+          'readingOrderNum': e.readings.indexWhere((r) => r.reading == rr) + 1,
         });
       }
       for (final ls in s.languageSource) {
@@ -181,24 +192,17 @@ Future<void> seedJMDictData(List<Entry> entries, Database db) async {
 
   print('  [JMdict] Building xref trees');
   final SplayTreeMap<String, Set<Entry>> entriesByKanji = SplayTreeMap();
+  final SplayTreeMap<String, Set<Entry>> entriesByReading = SplayTreeMap();
 
   for (final entry in entries) {
     for (final kanji in entry.kanji) {
-      if (entriesByKanji.containsKey(kanji.reading)) {
-        entriesByKanji.update(kanji.reading, (list) => list..add(entry));
-      } else {
-        entriesByKanji.putIfAbsent(kanji.reading, () => {entry});
-      }
+      entriesByKanji.putIfAbsent(kanji.reading, () => {});
+      entriesByKanji.update(kanji.reading, (set) => set..add(entry));
     }
-  }
-  final SplayTreeMap<String, Set<Entry>> entriesByReading = SplayTreeMap();
-  for (final entry in entries) {
+
     for (final reading in entry.readings) {
-      if (entriesByReading.containsKey(reading.reading)) {
-        entriesByReading.update(reading.reading, (list) => list..add(entry));
-      } else {
-        entriesByReading.putIfAbsent(reading.reading, () => {entry});
-      }
+      entriesByReading.putIfAbsent(reading.reading, () => {});
+      entriesByReading.update(reading.reading, (set) => set..add(entry));
     }
   }
 
@@ -207,6 +211,7 @@ Future<void> seedJMDictData(List<Entry> entries, Database db) async {
 
   for (final e in entries) {
     for (final s in e.senses) {
+      final seenSeeAlsoXrefs = <int>{};
       for (final xref in s.seeAlso) {
         final resolvedEntry = resolveXref(
           entriesByKanji,
@@ -214,16 +219,24 @@ Future<void> seedJMDictData(List<Entry> entries, Database db) async {
           xref,
         );
 
+        if (seenSeeAlsoXrefs.contains(resolvedEntry.entry.entryId)) {
+          print(
+            'WARNING: Skipping duplicate seeAlso xref from sense ${s.senseId} to entry ${resolvedEntry.entry.entryId}\n'
+            '  (kanjiRef: ${xref.kanjiRef}, readingRef: ${xref.readingRef}, senseOrderNum: ${xref.senseOrderNum})',
+          );
+          continue;
+        }
+        seenSeeAlsoXrefs.add(resolvedEntry.entry.entryId);
+
         b.insert(JMdictTableNames.senseSeeAlso, {
           'senseId': s.senseId,
           'xrefEntryId': resolvedEntry.entry.entryId,
-          'seeAlsoKanji': xref.kanjiRef,
-          'seeAlsoReading': xref.readingRef,
-          'seeAlsoSense': xref.senseOrderNum,
+          'xrefSenseOrderNum': resolvedEntry.senseOrderNum,
           'ambiguous': resolvedEntry.ambiguous,
         });
       }
 
+      final seenAntonymXrefs = <int>{};
       for (final ant in s.antonyms) {
         final resolvedEntry = resolveXref(
           entriesByKanji,
@@ -231,12 +244,18 @@ Future<void> seedJMDictData(List<Entry> entries, Database db) async {
           ant,
         );
 
+        if (seenAntonymXrefs.contains(resolvedEntry.entry.entryId)) {
+          print(
+            'WARNING: Skipping duplicate antonym xref from sense ${s.senseId} to entry ${resolvedEntry.entry.entryId}\n'
+            '  (kanjiRef: ${ant.kanjiRef}, readingRef: ${ant.readingRef}, senseOrderNum: ${ant.senseOrderNum})',
+          );
+          continue;
+        }
+        seenAntonymXrefs.add(resolvedEntry.entry.entryId);
+
         b.insert(JMdictTableNames.senseAntonyms, {
           'senseId': s.senseId,
           'xrefEntryId': resolvedEntry.entry.entryId,
-          'antonymKanji': ant.kanjiRef,
-          'antonymReading': ant.readingRef,
-          'antonymSense': ant.senseOrderNum,
           'ambiguous': resolvedEntry.ambiguous,
         });
       }

lib/_data_ingestion/kanjivg/objects.dart

@@ -1,92 +0,0 @@
-import 'package:jadb/_data_ingestion/sql_writable.dart';
-
-/// Enum set in the kvg:position attribute, used by `<g>` elements in the KanjiVG SVG files.
-enum KanjiPathGroupPosition {
-  bottom,
-  kamae,
-  kamaec,
-  left,
-  middle,
-  nyo,
-  nyoc,
-  right,
-  tare,
-  tarec,
-  top,
-}
-
-/// Contents of a \<g> element in the KanjiVG SVG files.
-class KanjiPathGroupTreeNode extends SQLWritable {
-  final String id;
-  final List<KanjiPathGroupTreeNode> children;
-  final String? element;
-  final String? original;
-  final KanjiPathGroupPosition? position;
-  final String? radical;
-  final int? part;
-
-  KanjiPathGroupTreeNode({
-    required this.id,
-    this.children = const [],
-    this.element,
-    this.original,
-    this.position,
-    this.radical,
-    this.part,
-  });
-
-  @override
-  Map<String, Object?> get sqlValue => {
-    'id': id,
-    'element': element,
-    'original': original,
-    'position': position?.name,
-    'radical': radical,
-    'part': part,
-  };
-}
-
-/// Contents of a `<text>` element in the StrokeNumber's group in the KanjiVG SVG files
-class KanjiStrokeNumber extends SQLWritable {
-  final int num;
-  final double x;
-  final double y;
-
-  KanjiStrokeNumber(this.num, this.x, this.y);
-
-  @override
-  Map<String, Object?> get sqlValue => {'num': num, 'x': x, 'y': y};
-}
-
-/// Contents of a `<path>` element in the KanjiVG SVG files
-class KanjiVGPath extends SQLWritable {
-  final String id;
-  final String type;
-  final String svgPath;
-
-  KanjiVGPath({required this.id, required this.type, required this.svgPath});
-
-  @override
-  Map<String, Object?> get sqlValue => {
-    'id': id,
-    'type': type,
-    'svgPath': svgPath,
-  };
-}
-
-class KanjiVGItem extends SQLWritable {
-  final String character;
-  final List<KanjiVGPath> paths;
-  final List<KanjiStrokeNumber> strokeNumbers;
-  final List<KanjiPathGroupTreeNode> pathGroups;
-
-  KanjiVGItem({
-    required this.character,
-    required this.paths,
-    required this.strokeNumbers,
-    required this.pathGroups,
-  });
-
-  @override
-  Map<String, Object?> get sqlValue => {'character': character};
-}

lib/_data_ingestion/kanjivg/seed_data.dart

@@ -1,7 +0,0 @@
-import 'package:sqflite_common/sqflite.dart';
-
-Future<void> seedKanjiVGData(Iterable<String> xmlContents, Database db) async {
-  final b = db.batch();
-
-  await b.commit(noResult: true);
-}

lib/_data_ingestion/open_local_db.dart

@@ -27,6 +27,7 @@ Future<Database> openLocalDb({
           await db.execute('PRAGMA journal_mode = WAL');
         }
         await db.execute('PRAGMA foreign_keys = ON');
+        await db.execute("SELECT icu_load_collation('ja_JP', 'japanese')");
       },
       readOnly: !readWrite,
     ),

lib/_data_ingestion/tanos-jlpt/csv_parser.dart

@@ -10,11 +10,12 @@ Future<List<JLPTRankedWord>> parseJLPTRankedWords(
 ) async {
   final List<JLPTRankedWord> result = [];
 
-  final codec = CsvCodec(
+  final codec = Csv(
     fieldDelimiter: ',',
     lineDelimiter: '\n',
     quoteMode: QuoteMode.strings,
     escapeCharacter: '\\',
+    parseHeaders: false,
   );
 
   for (final entry in files.entries) {
@@ -29,7 +30,6 @@ Future<List<JLPTRankedWord>> parseJLPTRankedWords(
         .openRead()
         .transform(utf8.decoder)
         .transform(codec.decoder)
-        .flatten()
         .map((row) {
           if (row.length != 3) {
             throw Exception('Invalid line in $jlptLevel: $row');

lib/models/verify_tables.dart

@@ -1,6 +1,5 @@
 import 'package:jadb/table_names/jmdict.dart';
 import 'package:jadb/table_names/kanjidic.dart';
-import 'package:jadb/table_names/kanjivg.dart';
 import 'package:jadb/table_names/radkfile.dart';
 import 'package:jadb/table_names/tanos_jlpt.dart';
 import 'package:sqflite_common/sqlite_api.dart';
@@ -22,7 +21,6 @@ Future<void> verifyTablesWithDbConnection(DatabaseExecutor db) async {
     ...KANJIDICTableNames.allTables,
     ...RADKFILETableNames.allTables,
     ...TanosJLPTTableNames.allTables,
-    ...KanjiVGTableNames.allTables,
   };
 
   final missingTables = expectedTables.difference(tables);

lib/models/word_search/word_search_result.dart

@@ -1,5 +1,6 @@
 import 'package:jadb/models/common/jlpt_level.dart';
 import 'package:jadb/models/jmdict/jmdict_kanji_info.dart';
+import 'package:jadb/models/jmdict/jmdict_misc.dart';
 import 'package:jadb/models/jmdict/jmdict_reading_info.dart';
 import 'package:jadb/models/word_search/word_search_match_span.dart';
 import 'package:jadb/models/word_search/word_search_ruby.dart';
@@ -45,6 +46,13 @@ class WordSearchResult {
   /// the original searchword.
   List<WordSearchMatchSpan>? matchSpans;
 
+  /// Whether the first item in [japanese] contains kanji that likely is rare.
+  bool get hasUnusualKanji =>
+      (japanese.first.furigana != null &&
+          kanjiInfo[japanese.first.base] == JMdictKanjiInfo.rK) ||
+      senses.where((sense) => sense.misc.contains(JMdictMisc.onlyKana)).length >
+          (senses.length / 2);
+
   /// All contents of [japanese], transliterated to romaji
   List<String> get romaji => japanese
       .map((word) => transliterateKanaToLatin(word.furigana ?? word.base))

lib/table_names/kanjivg.dart

@@ -1,9 +0,0 @@
-abstract class KanjiVGTableNames {
-  static const String version = 'KanjiVG_Version';
-  static const String entry = 'KanjiVG_Entry';
-  static const String path = 'KanjiVG_Path';
-  static const String strokeNumber = 'KanjiVG_StrokeNumber';
-  static const String pathGroup = 'KanjiVG_PathGroup';
-
-  static Set<String> get allTables => {version, entry, path, strokeNumber, pathGroup};
-}

migrations/0001_JMDict.sql

@@ -1,3 +1,5 @@
+SELECT icu_load_collation('ja_JP', 'japanese');
+
 CREATE TABLE "JMdict_Version" (
   "version" VARCHAR(10) PRIMARY KEY NOT NULL,
   "date" DATE NOT NULL,
@@ -55,13 +57,13 @@ CREATE TABLE "JMdict_KanjiElement" (
   "elementId" INTEGER PRIMARY KEY,
   "entryId" INTEGER NOT NULL REFERENCES "JMdict_Entry"("entryId"),
   "orderNum" INTEGER NOT NULL,
-  "reading" TEXT NOT NULL,
+  "reading" TEXT NOT NULL COLLATE japanese,
   "news" INTEGER CHECK ("news" BETWEEN 1 AND 2),
   "ichi" INTEGER CHECK ("ichi" BETWEEN 1 AND 2),
   "spec" INTEGER CHECK ("spec" BETWEEN 1 AND 2),
   "gai" INTEGER CHECK ("gai" BETWEEN 1 AND 2),
   "nf" INTEGER CHECK ("nf" BETWEEN 1 AND 48),
-  UNIQUE("entryId", "reading"),
+  -- UNIQUE("entryId", "reading"),
   UNIQUE("entryId", "orderNum")
 ) WITHOUT ROWID;
 
@@ -80,14 +82,14 @@ CREATE TABLE "JMdict_ReadingElement" (
   "elementId" INTEGER PRIMARY KEY,
   "entryId" INTEGER NOT NULL REFERENCES "JMdict_Entry"("entryId"),
   "orderNum" INTEGER NOT NULL,
-  "reading" TEXT NOT NULL,
+  "reading" TEXT NOT NULL COLLATE japanese,
   "readingDoesNotMatchKanji" BOOLEAN NOT NULL DEFAULT FALSE,
   "news" INTEGER CHECK ("news" BETWEEN 1 AND 2),
   "ichi" INTEGER CHECK ("ichi" BETWEEN 1 AND 2),
   "spec" INTEGER CHECK ("spec" BETWEEN 1 AND 2),
   "gai" INTEGER CHECK ("gai" BETWEEN 1 AND 2),
   "nf" INTEGER CHECK ("nf" BETWEEN 1 AND 48),
-  UNIQUE("entryId", "reading"),
+  -- UNIQUE("entryId", "reading"),
   UNIQUE("entryId", "orderNum")
 ) WITHOUT ROWID;
 
@@ -120,17 +122,17 @@ CREATE INDEX "JMdict_Sense_byEntryId_byOrderNum" ON "JMdict_Sense"("entryId", "o
 CREATE TABLE "JMdict_SenseRestrictedToKanji" (
   "entryId" INTEGER NOT NULL,
   "senseId" INTEGER NOT NULL REFERENCES "JMdict_Sense"("senseId"),
-  "kanji" TEXT NOT NULL,
-  FOREIGN KEY ("entryId", "kanji") REFERENCES "JMdict_KanjiElement"("entryId", "reading"),
-  PRIMARY KEY ("entryId", "senseId", "kanji")
+  "kanjiOrderNum" INTEGER NOT NULL CHECK ("kanjiOrderNum" > 0),
+  FOREIGN KEY ("entryId", "kanjiOrderNum") REFERENCES "JMdict_KanjiElement"("entryId", "orderNum"),
+  PRIMARY KEY ("entryId", "senseId", "kanjiOrderNum")
 ) WITHOUT ROWID;
 
 CREATE TABLE "JMdict_SenseRestrictedToReading" (
   "entryId" INTEGER NOT NULL,
   "senseId" INTEGER NOT NULL REFERENCES "JMdict_Sense"("senseId"),
-  "reading" TEXT NOT NULL,
-  FOREIGN KEY ("entryId", "reading") REFERENCES "JMdict_ReadingElement"("entryId", "reading"),
-  PRIMARY KEY ("entryId", "senseId", "reading")
+  "readingOrderNum" INTEGER NOT NULL CHECK ("readingOrderNum" > 0),
+  FOREIGN KEY ("entryId", "readingOrderNum") REFERENCES "JMdict_ReadingElement"("entryId", "orderNum"),
+  PRIMARY KEY ("entryId", "senseId", "readingOrderNum")
 ) WITHOUT ROWID;
 
 -- In order to add xrefs, you will need to have added the entry to xref to.
@@ -145,32 +147,23 @@ CREATE TABLE "JMdict_SenseRestrictedToReading" (
 
 CREATE TABLE "JMdict_SenseSeeAlso" (
   "senseId" INTEGER NOT NULL REFERENCES "JMdict_Sense"("senseId"),
-  "xrefEntryId" INTEGER NOT NULL,
-  "seeAlsoReading" TEXT,
-  "seeAlsoKanji" TEXT,
-  "seeAlsoSense" INTEGER,
+  "xrefEntryId" INTEGER NOT NULL REFERENCES "JMdict_Entry"("entryId"),
+  -- Sometimes the cross reference is to a specific sense
+  "xrefSenseOrderNum" INTEGER,
   -- For some entries, the cross reference is ambiguous. This means that while the ingestion
   -- has determined some xrefEntryId, it is not guaranteed to be the correct one.
   "ambiguous" BOOLEAN NOT NULL DEFAULT FALSE,
-  FOREIGN KEY ("xrefEntryId", "seeAlsoKanji") REFERENCES "JMdict_KanjiElement"("entryId", "reading"),
-  FOREIGN KEY ("xrefEntryId", "seeAlsoReading") REFERENCES "JMdict_ReadingElement"("entryId", "reading"),
-  FOREIGN KEY ("xrefEntryId", "seeAlsoSense") REFERENCES "JMdict_Sense"("entryId", "orderNum"),
-  UNIQUE("senseId", "xrefEntryId", "seeAlsoReading", "seeAlsoKanji", "seeAlsoSense")
+  FOREIGN KEY ("xrefEntryId", "xrefSenseOrderNum") REFERENCES "JMdict_Sense"("entryId", "orderNum"),
+  UNIQUE("senseId", "xrefEntryId", "xrefSenseOrderNum")
 );
 
 CREATE TABLE "JMdict_SenseAntonym" (
   "senseId" INTEGER NOT NULL REFERENCES "JMdict_Sense"("senseId"),
-  "xrefEntryId" INTEGER NOT NULL,
-  "antonymReading" TEXT,
-  "antonymKanji" TEXT,
-  "antonymSense" INTEGER,
+  "xrefEntryId" INTEGER NOT NULL REFERENCES "JMdict_Entry"("entryId"),
   -- For some entries, the cross reference is ambiguous. This means that while the ingestion
   -- has determined some xrefEntryId, it is not guaranteed to be the correct one.
   "ambiguous" BOOLEAN NOT NULL DEFAULT FALSE,
-  FOREIGN KEY ("xrefEntryId", "antonymKanji") REFERENCES "JMdict_KanjiElement"("entryId", "reading"),
-  FOREIGN KEY ("xrefEntryId", "antonymReading") REFERENCES "JMdict_ReadingElement"("entryId", "reading"),
-  FOREIGN KEY ("xrefEntryId", "antonymSense") REFERENCES "JMdict_Sense"("entryId", "orderNum"),
-  UNIQUE("senseId", "xrefEntryId", "antonymReading", "antonymKanji", "antonymSense")
+  UNIQUE("senseId", "xrefEntryId")
 );
 
 -- These cross references are going to be mostly accessed from a sense

migrations/0011_KanjiVG.sql

@@ -1,45 +0,0 @@
-CREATE TABLE "KanjiVG_Version" (
-  "version" VARCHAR(10) PRIMARY KEY NOT NULL,
-  "date" DATE NOT NULL,
-  "hash" VARCHAR(64) NOT NULL
-) WITHOUT ROWID;
-
-CREATE TRIGGER "KanjiVG_Version_SingleRow"
-BEFORE INSERT ON "KanjiVG_Version"
-WHEN (SELECT COUNT(*) FROM "KanjiVG_Version") >= 1
-BEGIN
-    SELECT RAISE(FAIL, 'Only one row allowed in KanjiVG_Version');
-END;
-
-CREATE TABLE "KanjiVG_Entry" (
-  "character" CHAR(1) PRIMARY KEY NOT NULL
-) WITHOUT ROWID;
-
-CREATE TABLE "KanjiVG_StrokeNumber" (
-  "character" CHAR(1) NOT NULL REFERENCES "KanjiVG_Entry"("character"),
-  "strokeNum" INTEGER NOT NULL,
-  "x" REAL NOT NULL,
-  "y" REAL NOT NULL,
-  PRIMARY KEY ("character", "strokeNum")
-) WITHOUT ROWID;
-
-CREATE TABLE "KanjiVG_Path" (
-  "character" CHAR(1) NOT NULL REFERENCES "KanjiVG_Entry"("character"),
-  "pathId" TEXT NOT NULL,
-  "type" VARCHAR(10) NOT NULL,
-  "svgPath" TEXT NOT NULL,
-  PRIMARY KEY ("character", "pathId")
-) WITHOUT ROWID;
-
-CREATE TABLE "KanjiVG_PathGroup" (
-  "character" CHAR(1) NOT NULL REFERENCES "KanjiVG_Entry"("character"),
-  "groupId" TEXT NOT NULL,
-  "parentGroupId" TEXT REFERENCES "KanjiVG_PathGroup"("groupId"),
-  "element" TEXT,
-  "original" TEXT,
-  "position" VARCHAR(10),
-  "radical" TEXT,
-  "part" INTEGER,
-  PRIMARY KEY ("character", "groupId"),
-  CHECK ("position" IN ('bottom', 'kamae', 'kamaec', 'left', 'middle', 'nyo', 'nyoc', 'right', 'tare', 'tarec', 'top') OR "position" IS NULL)
-) WITHOUT ROWID;

nix/database_tool.nix

@@ -1,6 +1,7 @@
 {
   src,
   buildDartApplication,
+  sqlite,
 }:
 buildDartApplication {
   pname = "jadb-database-tool";
@@ -9,6 +10,9 @@ buildDartApplication {
 
   dartEntryPoints."bin/jadb" = "bin/jadb.dart";
 
+  # NOTE: here we are overriding the implicitly added runtimeDependency from the package fixup in pub2nix.
+  runtimeDependencies = [ sqlite ];
+
   # NOTE: the default dart hooks are using `dart compile`, which is not able to call the
   #       new dart build hooks required to use package:sqlite3 >= 3.0.0. So we override
   #       these phases to use `dart build` instead.

pubspec.lock

@@ -5,18 +5,18 @@ packages:
     dependency: transitive
     description:
       name: _fe_analyzer_shared
-      sha256: "3b19a47f6ea7c2632760777c78174f47f6aec1e05f0cd611380d4593b8af1dbc"
+      sha256: "8d718c5c58904f9937290fd5dbf2d6a0e02456867706bfb6cd7b81d394e738d5"
       url: "https://pub.dev"
     source: hosted
-    version: "96.0.0"
+    version: "98.0.0"
   analyzer:
     dependency: transitive
     description:
       name: analyzer
-      sha256: "0c516bc4ad36a1a75759e54d5047cb9d15cded4459df01aa35a0b5ec7db2c2a0"
+      sha256: "6141ad5d092d1e1d13929c0504658bbeccc1703505830d7c26e859908f5efc88"
       url: "https://pub.dev"
     source: hosted
-    version: "10.2.0"
+    version: "12.0.0"
   args:
     dependency: "direct main"
     description:
@@ -29,10 +29,18 @@ packages:
     dependency: transitive
     description:
       name: async
-      sha256: "758e6d74e971c3e5aceb4110bfd6698efc7f501675bcfe0c775459a8140750eb"
+      sha256: e2eb0491ba5ddb6177742d2da23904574082139b07c1e33b8503b9f46f3e1a37
       url: "https://pub.dev"
     source: hosted
-    version: "2.13.0"
+    version: "2.13.1"
+  benchmark_harness:
+    dependency: "direct dev"
+    description:
+      name: benchmark_harness
+      sha256: a2d3c4c83cac0126bf38e41eaf7bd9ed4f6635f1ee1a0cbc6f79fa9736c62cbd
+      url: "https://pub.dev"
+    source: hosted
+    version: "2.4.0"
   boolean_selector:
     dependency: transitive
     description:
@@ -93,10 +101,10 @@ packages:
     dependency: "direct main"
     description:
       name: csv
-      sha256: bef2950f7a753eb82f894a2eabc3072e73cf21c17096296a5a992797e50b1d0d
+      sha256: "2e0a52fb729f2faacd19c9c0c954ff450bba37aa8ab999410309e2342e7013a2"
       url: "https://pub.dev"
     source: hosted
-    version: "7.1.0"
+    version: "8.0.0"
   equatable:
     dependency: "direct main"
     description:
@@ -141,10 +149,10 @@ packages:
     dependency: transitive
     description:
       name: hooks
-      sha256: "7a08a0d684cb3b8fb604b78455d5d352f502b68079f7b80b831c62220ab0a4f6"
+      sha256: e79ed1e8e1929bc6ecb6ec85f0cb519c887aa5b423705ded0d0f2d9226def388
       url: "https://pub.dev"
     source: hosted
-    version: "1.0.1"
+    version: "1.0.2"
   http_multi_server:
     dependency: transitive
     description:
@@ -189,18 +197,18 @@ packages:
     dependency: transitive
     description:
       name: matcher
-      sha256: "12956d0ad8390bbcc63ca2e1469c0619946ccb52809807067a7020d57e647aa6"
+      sha256: dc0b7dc7651697ea4ff3e69ef44b0407ea32c487a39fff6a4004fa585e901861
       url: "https://pub.dev"
     source: hosted
-    version: "0.12.18"
+    version: "0.12.19"
   meta:
     dependency: transitive
     description:
       name: meta
-      sha256: "9f29b9bcc8ee287b1a31e0d01be0eae99a930dbffdaecf04b3f3d82a969f296f"
+      sha256: df0c643f44ad098eb37988027a8e2b2b5a031fd3977f06bbfd3a76637e8df739
       url: "https://pub.dev"
     source: hosted
-    version: "1.18.1"
+    version: "1.18.2"
   mime:
     dependency: transitive
     description:
@@ -213,10 +221,10 @@ packages:
     dependency: transitive
     description:
       name: native_toolchain_c
-      sha256: "89e83885ba09da5fdf2cdacc8002a712ca238c28b7f717910b34bcd27b0d03ac"
+      sha256: "6ba77bb18063eebe9de401f5e6437e95e1438af0a87a3a39084fbd37c90df572"
       url: "https://pub.dev"
     source: hosted
-    version: "0.17.4"
+    version: "0.17.6"
   node_preamble:
     dependency: transitive
     description:
@@ -341,10 +349,10 @@ packages:
     dependency: "direct main"
     description:
       name: sqlite3
-      sha256: b7cf6b37667f6a921281797d2499ffc60fb878b161058d422064f0ddc78f6aa6
+      sha256: caa693ad15a587a2b4fde093b728131a1827903872171089dedb16f7665d3a91
       url: "https://pub.dev"
     source: hosted
-    version: "3.1.6"
+    version: "3.2.0"
   stack_trace:
     dependency: transitive
     description:
@@ -389,26 +397,26 @@ packages:
     dependency: "direct dev"
     description:
       name: test
-      sha256: "54c516bbb7cee2754d327ad4fca637f78abfc3cbcc5ace83b3eda117e42cd71a"
+      sha256: "8d9ceddbab833f180fbefed08afa76d7c03513dfdba87ffcec2718b02bbcbf20"
       url: "https://pub.dev"
     source: hosted
-    version: "1.29.0"
+    version: "1.31.0"
   test_api:
     dependency: transitive
     description:
       name: test_api
-      sha256: "93167629bfc610f71560ab9312acdda4959de4df6fac7492c89ff0d3886f6636"
+      sha256: "949a932224383300f01be9221c39180316445ecb8e7547f70a41a35bf421fb9e"
       url: "https://pub.dev"
     source: hosted
-    version: "0.7.9"
+    version: "0.7.11"
   test_core:
     dependency: transitive
     description:
       name: test_core
-      sha256: "394f07d21f0f2255ec9e3989f21e54d3c7dc0e6e9dbce160e5a9c1a6be0e2943"
+      sha256: "1991d4cfe85d5043241acac92962c3977c8d2f2add1ee73130c7b286417d1d34"
       url: "https://pub.dev"
     source: hosted
-    version: "0.6.15"
+    version: "0.6.17"
   typed_data:
     dependency: transitive
     description:

pubspec.yaml

@@ -9,7 +9,7 @@ environment:
 dependencies:
   args: ^2.7.0
   collection: ^1.19.0
-  csv: ^7.1.0
+  csv: ^8.0.0
   equatable: ^2.0.0
   path: ^1.9.1
   sqflite_common: ^2.5.0
@@ -18,6 +18,7 @@ dependencies:
   xml: ^6.5.0
 
 dev_dependencies:
+  benchmark_harness: ^2.4.0
   lints: ^6.0.0
   test: ^1.25.15