`cursor_pos` is read as UTF-16 code units, but Jupyter messaging 5.2 specifies Unicode codepoints

[jicoopdev]

Summary

complete_request.cursor_pos is passed through to JavaKernel.complete(String code, int at) and
used directly as a Java String index — i.e. as a UTF-16 code unit offset. The Jupyter
messaging spec requires cursor_pos to be a Unicode codepoint offset as of protocol 5.2, and
JJava advertises protocol_version 5.3. Any client that sends spec-compliant offsets therefore
gets completions computed at the wrong position as soon as a non-BMP character (emoji, some CJK
extension characters, mathematical alphanumerics) appears anywhere before the cursor — off by one
per astral character. The cursor_start / cursor_end in the reply are wrong in the same way.

Environment

• JJava 1.0-a7 (kernel banner: Java 21.0.12+8-LTS :: JJava 1.0-a7 :: Protocol v5.3,
  implementation_version: 1.0-a7, protocol_version: 5.3)
• Also verified still present on main @ db31f2c
• JDK 21.0.12+8 (Temurin), Linux x86_64 — not JDK-dependent
• Driven over the plain Jupyter ZMQ protocol via jupyter_client

What the spec says

From the Jupyter messaging spec, complete_request / inspect_request:

Changed in version 5.2: Due to a widespread bug in many frontends, cursor_pos in versions
prior to 5.2 is ambiguous in the presence of "astral-plane" characters. In 5.2, cursor_pos
must be the actual encoding-independent offset in unicode codepoints.

Reproduction

One code string, containing a single emoji before the cursor, sent twice with different
cursor_pos conventions. The prefix is chosen so that being one position short changes the
answer: Math.ma completes to max( alone, while Math.m also matches min( and the
multiply* family.

code = "\"😀\"; Math.ma"      i.e.  "😀"; Math.ma
       13 UTF-16 code units, 12 codepoints; cursor at end of text

cursor_pos sent	convention	matches returned	cursor_start / cursor_end
12	codepoints (spec-compliant)	max(, min(, multiplyExact(, multiplyFull(, multiplyHigh(	11 / 12
13	UTF-16 code units	max(	11 / 13

The spec-compliant request completes Math.m — one position behind the actual cursor — and
returns five candidates instead of one. Only the UTF-16 offset produces completions for the text
that is really before the cursor.

An ASCII-only control where the two conventions coincide behaves correctly, confirming the
emoji is the whole difference:

code = "String s; Math.ma"   (17 units == 17 codepoints), cursor_pos = 17
   -> matches: ["max("], cursor_start 15, cursor_end 17     (correct)

The reply positions are affected too. In the UTF-16 row above, ma begins at UTF-16 index 11 but
at codepoint index 10, and the cursor is at codepoint 12. A spec-compliant client should have
received cursor_start: 10, cursor_end: 12; it received 11 / 13. A client that honours those
values as codepoints replaces the wrong range.

Mechanism

request.getCursorPos() is handed to complete() unconverted in
BaseKernel.handleCompleteRequest (BaseKernel.java:452 at 1.0-a7, :451 on main), and
JavaKernel.complete uses it as a String index throughout — line numbers at 1.0-a7, with
main's in parentheses:

// jjava-kernel/src/main/java/org/dflib/jjava/kernel/JavaKernel.java
int lineStart = code.lastIndexOf('\n', at - 1) + 1;   // :285  (main :301)
String line = code.substring(lineStart, at);          // :286  (main :302)
...
    .completionSuggestions(code, at, replaceStart);    // :311  (main :327)

and the same untouched integers travel back out as cursor_start / cursor_end:

return new ReplacementOptions(options, lineStart, at);         // :305  (main :321)
return new ReplacementOptions(options, replaceStart[0], at);   // :327  (main :343)

JavaKernel.inspect(String code, int at, boolean extraDetail) (:234, main :250) has the
same issue — it calls code.charAt(at + 1) and
sourceCodeAnalysis().documentation(code, at + 1, true) — so inspect_request.cursor_pos is
affected identically. I verified this one by reading the source rather than on the wire.

Impact

Limited to code containing non-BMP characters before the cursor — in practice, emoji or
astral-plane characters in string literals or comments. Where it does apply, completion and
inspection silently operate on the wrong position, and completion-accept can replace the wrong
range of text. It is also a straightforward protocol-conformance gap for a kernel advertising
protocol 5.3.

Note this cuts both ways for frontends: a client that already compensates by sending UTF-16
offsets works correctly against JJava today and would break if this is fixed, so the fix is worth
a line in the release notes.

Suggested fix

Convert at the protocol boundary, in both directions:

int at = code.offsetByCodePoints(0, cursorPos);          // inbound
int cursorStart = code.codePointCount(0, sourceStart);   // outbound

Doing it once in BaseKernel.handleCompleteRequest / handleInspectRequest fixes every
BaseKernel subclass at the cost of changing what the complete / inspect overrides receive;
doing it inside JavaKernel keeps that contract but leaves the same trap for other
implementations. Either way, the ASCII fast path is a single code.length() == codePointCount
check if the conversion cost matters.