Tracking issue for non-ASCII identifiers (feature "non_ascii_idents")

[DemiMarie]

Non-ASCII identifiers are currently feature gated. Handling of them should be fixed and the feature gate removed.

[steveklabnik]

/cc @rust-lang/lang

[pnkfelix]

nominating

[nrc]

cc @SimonSapin

Apparently we implement this: http://www.unicode.org/reports/tr31/ or something like it.

I would like to see this stabilised, but it will take some work to persuade ourselves that we are doing the right thing.

[SimonSapin]

I have no idea what the right thing is here. In addition to Unicode recommendations, we might want to look at what other languages actually do, and what related bug reports or criticism they get. Or was this already done when the feature was first introduced?

[petrochenkov]

@SimonSapin
C and C++ use http://unicode.org/reports/tr31/#Alternative_Identifier_Syntax (with some minor restrictions) and I haven't seen any complaints about it on isocpp forums or issue lists :)
Overview of the problem: http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1518.htm
Implementation in Clang: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Lex/UnicodeCharSets.h?view=markup
cc #4928

There's also a problem with normalization of identifiers and mapping unicode mod names to the filesystem names (on OS X, IIRC), but I can't find the relevant link here it is: #2253. (In the worst case non-inline mods and extern crates can be forced to be ASCII)

[pnkfelix]

Yes #2253 is the big issue I know of that makes me worry about premature stabilization of non-unicode identifiers.

(The discussion there is more broad and arguably could be forked off into two threads; e.g. we could take one normalization path for identifiers and another for string literal contents.)

[pnkfelix]

we may want to migrate This discussion to the RFCS repo, e.g. at rust-lang/rfcs#802

[bstrie]

I agree that this is a feature that deserves to be put through the RFC process.

[aturon]

I've repurposed this issue to track stabilization (or deprecation, etc) of the non_ascii_idents feature gate.

[nikomatsakis]

After discussion in the lang team meeting, we decided that yes, an RFC would be the proper way forward here. We need something that collects the solutions from other languages, analyzes their pros/cons, and suggests the appropriate choice for Rust. This is controversial and complex enough that it should be brought to the community at large -- especially as many of us hacking on Rust on a daily basis don't have a lot of experience with non-ASCII anyhow.