mwsherman

While this article is about perf — and trading off semantic precision by design — there is a Unicode standard for sentence boundaries, may be interesting: https://www.unicode.org/reports/tr29/#Sentence_Boundaries

I implemented the sentence boundaries, but also thought that the notion of a “phrase” might be useful for such applications: https://github.com/clipperhouse/uax29/tree/master/phrases

mwsherman

7 ay öncediscuss

Shameless plug, if one wishes to track down allocations in Go, an allocations explorer for VS Code: https://marketplace.visualstudio.com/items?itemName=Clipperh...

mwsherman

7 ay öncediscuss

I’ve found C#’s frozen dictionary to be useful: https://learn.microsoft.com/en-us/dotnet/api/system.collecti...

It’s optimized for fast reads in exchange for expensive creation.

mwsherman

9 ay öncediscuss

That’s true, converting in either direction will typically allocate. Which it must, semantically.

One can use unsafe for a zero-copy conversion, but now you are breaking the semantics: a string becomes mutable, because its underlying bytes are mutable.

Or! One can often handle strings and bytes interchangeably with generics: https://github.com/clipperhouse/stringish

mwsherman

9 ay öncediscuss

There is mention of how len() is bytes, not “characters”. A further subtlety: a rune (codepoint) is still not necessarily a “character” in terms of what is displayed for users — that would be a “grapheme”.

A grapheme can be multiple codepoints, with modifiers, joiners, etc.

This is true in all languages, it’s a Unicode thing, not a Go thing. Shameless plug, here is a grapheme tokenizer for Go: https://github.com/clipperhouse/uax29/tree/master/graphemes

mwsherman

9 ay öncediscuss

In Go, string effectively serves as a read-only slice, if we are talking about bytes.

ReadOnlySpan<T> in C# is great! In my opinion, Go essentially designed in “span” from the start.

mwsherman

9 ay öncediscuss

A Go allocations explorer for VS Code. Extension: https://marketplace.visualstudio.com/items?itemName=Clipperh...

Source: https://github.com/clipperhouse/go-allocations-vsix

mwsherman

9 ay öncediscuss

Shameless plug, you may wish to do Lucene-style tokenizing using the Unicode standard: https://github.com/clipperhouse/uax29/tree/master/words

mwsherman

9 ay öncediscuss

I move between Go and C#. I wrote a zero-allocation package in Go [1] and then ported to C# — and the allocations exploded!

I had forgotten, or perhaps never realized, that substrings in C# allocate. The solution was Spans.

Notably, it caused me to realize that Go had “spans” designed in from the start.

[1] https://github.com/clipperhouse/uax29

Submissions

Making Unicode things fast in Go

Show HN: Go Allocations Explorer for VS Code

Allocations Are a Dependency

I deal with copyright claims [video]

comments