Encoding helpers and BOM detection
Separate from the binary radix encodings (Base16-Base85, in Bodu.Text.Encoding), the Bodu.Core package ships, in the Bodu.Text namespace, a set of helpers for working with the BCL Encoding itself: zero-ceremony string↔byte[] conversion, allocation-conscious (pooled / owned-memory) surfaces, byte-order-mark (BOM / preamble) handling, UTF classification, fallback configuration, and chunked transcoding. These are the everyday utilities that make System.Text.Encoding pleasant to use without hand-rolling preamble logic or ArrayPool<byte> plumbing.
Three types make up the surface:
- StringEncodingExtensions - extension methods on
string. - EncodingExtensions - extension methods on
System.Text.Encoding. - EncodingDetection - static BOM-sniffing.
Converting strings to bytes
StringEncodingExtensions removes the encoding.GetBytes(...) boilerplate and adds UTF-8 fast paths and preamble-aware variants:
using Bodu.Text;
byte[] utf8 = "héllo".ToUtf8Bytes(); // UTF-8, no BOM
byte[] utf16 = "héllo".ToBytes(System.Text.Encoding.Unicode);
byte[] withBom = "héllo".ToBytesWithPreamble(System.Text.Encoding.Unicode); // BOM prepended
int byteCount = "héllo".GetUtf8ByteCount(); // size without allocating
For hot paths, encode straight into a caller-supplied span or an IBufferWriter<byte>, or rent the buffer from the pool:
using System.Buffers;
using Bodu.Buffers;
using Bodu.Text;
Span<byte> destination = stackalloc byte[64];
int written = "héllo".EncodeUtf8To(destination);
if ("héllo".TryEncodeUtf8To(destination, out int n)) { /* … */ }
var bufferWriter = new ArrayBufferWriter<byte>();
"héllo".WriteUtf8To(bufferWriter); // append to a pipeline
using PooledBufferBuilder<byte> pooled = "héllo".GetUtf8BytesPooled(); // ArrayPool-backed (Bodu.Buffers)
Byte-order marks (preambles)
EncodingExtensions centralizes BOM handling so you never slice preamble bytes by hand:
using Bodu.Text;
System.Text.Encoding enc = System.Text.Encoding.UTF8;
byte[] bytes = [0xEF, 0xBB, 0xBF, (byte)'h', (byte)'i']; // a UTF-8 BOM, then "hi"
bool hasBom = enc.HasPreamble();
bool startsBom = enc.StartsWithPreamble(bytes);
ReadOnlySpan<byte> body = enc.StripPreamble(bytes); // bytes minus any leading BOM
string text = enc.GetStringSkippingPreamble(bytes); // decode, ignoring a leading BOM
To detect the encoding from the bytes, use EncodingDetection:
using Bodu.Text;
byte[] bytes = File.ReadAllBytes("input.txt");
System.Text.Encoding chosen =
EncodingDetection.TryDetectByPreamble(bytes, out System.Text.Encoding? detected)
? detected
: System.Text.Encoding.UTF8; // sensible default when there is no BOM
Classifying an encoding
using Bodu.Text;
System.Text.Encoding enc = System.Text.Encoding.UTF8;
enc.IsUtf8(); // true
enc.IsAscii();
enc.IsAnyUtf(); // UTF-8 / UTF-16 / UTF-32, either endianness
enc.IsUtf16LittleEndian();
enc.IsUtf32BigEndian();
Fallback configuration
Switch an encoding between throwing on invalid data and replacing it, without mutating a shared instance:
using Bodu.Text;
System.Text.Encoding strict = System.Text.Encoding.UTF8.WithExceptionFallbacks(); // throws on malformed input
System.Text.Encoding lenient = System.Text.Encoding.UTF8.WithReplacementFallbacks(); // substitutes U+FFFD
Allocation-conscious transcoding
For large or streamed payloads, the owned-memory and chunked surfaces avoid intermediate arrays:
using System.Buffers;
using Bodu.Buffers;
using Bodu.Text;
System.Text.Encoding enc = System.Text.Encoding.UTF8;
string text = "héllo, wörld";
using IMemoryOwner<byte> owner = enc.GetBytesOwner(text); // caller disposes
using PooledBufferBuilder<char> chars = enc.GetCharsPooled(owner.Memory.Span);
// Incremental transcoding with explicit backpressure: EncodeChunk extends the stateful Encoder.
System.Text.Encoder encoder = enc.GetEncoder();
Span<byte> byteDestination = stackalloc byte[8];
OperationStatus status = encoder.EncodeChunk(text, byteDestination, flush: true,
out int charsRead, out int bytesWritten);
EncodeChunk (on Encoder) and DecodeChunk (on Decoder) return an OperationStatus: Done when the whole chunk was converted, or DestinationTooSmall when output remains, so a caller can drive a pull-based loop and grow the destination as needed. Invalid data is never reported as a status: the encoder's or decoder's fallback handles it, throwing under the exception fallback.
API summary
| Type | Highlights |
|---|---|
| StringEncodingExtensions | ToUtf8Bytes / ToBytes / ToBytesWithPreamble, EncodeUtf8To / TryEncodeUtf8To, WriteUtf8To, GetUtf8ByteCount, pooled variants. |
| EncodingExtensions | HasPreamble / StripPreamble / StartsWithPreamble / GetStringSkippingPreamble, IsUtf8 / IsAscii / IsAnyUtf / endianness checks, WithExceptionFallbacks / WithReplacementFallbacks, owned / pooled buffers, EncodeChunk / DecodeChunk. |
| EncodingDetection | TryDetectByPreamble - identify an encoding from a leading BOM. |
Where to go next
- Bodu.Core guides - the namespace map of the package this namespace ships in.
- Bodu.Text introduction - the namespace's surfaces and when to use each.
- Bodu.Text API reference - full namespace overview.
- The IBinaryEncoding interface - the binary-to-text contract of the separate
Bodu.Text.Encodingpackage (Base16 to Base85), for bytes to a printable alphabet rather than to characters.