Append code points to strings

bgrommes

Problem

I'm casting about for a way to quickly build a string from code points without allocating a 1 character string to append to an existing string, over and over. It occurs to me that you might be able to do this more performantly for us.

Proposed Solution

A FromChars(Array(Of Integer)) As String method for String that would take an array of code points and add them in sequence to the string.

Since code points are the same for all UTF encodings (and for ASCII), it doesn't matter what encoding originally produced the code points.

Alternatives Considered

I suppose you could convert code points to strings yourself but it's clunky (though it could be hidden in a utility method). Given an array of code points:

var chars() As String
chars.Reserve(codePoints.Count)

For c As Integer = 0 To codePoints.LastIndex
    # I'm assuming For is a bit faster than For Each
    chars.Append(Chr(codePoints[c]))
Next

var result As String = chars.Join("")

But whatever this benchmarks out as, I wonder if it could be beat by a native ability, potentially by a considerable margin, if say you could convert the code points to a char[] buffer and then just new up a string from that.

Who Would This Help?

You can probably tell that this is a follow on to my suggestion to add an ability to read / write files at the character level. If I'm reading character by character, I would likely produce an array of code points that it would be nice to convert to a string in one go.

Garry

Thanks for the suggestion. I’ve implemented this for the next release as a shared String method:

String.FromCodePoints(codePoints() As Integer) As String

This is intended as the bulk equivalent of Chr(codePoint). For example:

Var codePoints() As Integer = [72, 101, 108, 108, 111]
Print(String.FromCodePoints(codePoints)) # Hello

Var symbols() As Integer = [65, 9829, 8364, 128640]
Print(String.FromCodePoints(symbols)) # A♥€🚀

I chose a construction method rather than an append method because Objo strings are immutable, so repeated appending would still allocate new strings. String.FromCodePoints() lets the runtime validate the input, compute the final string size, allocate once, and then fill the string directly.

The method accepts Unicode scalar values in the same valid range as Chr():

0 through 0x10FFFF
excluding surrogate code points 0xD800 through 0xDFFF

Invalid values raise a runtime error rather than being silently replaced.

One important detail: code points above 0xFFFF are stored internally as surrogate pairs, so they still have a Length of 2 in the current string model, matching the existing Chr() behaviour.

This is now documented and internally I've covered it with tests for ASCII, BMP characters, supplementary-plane code points, empty arrays, null characters, and invalid code points.

https://feedback.objo.dev/feature/474

bgrommes

Garry Thanks, Garry, this is perfect ... I actually meant a construction method. I had tried to edit the title of the post but don't have permission to do that.