Unicode and Strings by chromatic: a summary

Perl strings can contain text, but text and the bytes used to store it are not the same thing. Understanding this difference helps you work with characters from different languages.

1. Characters and Bytes

Consider the word:

my $word = "café";

It contains four characters: c, a, f, and é.

But how many bytes does it occupy?

That depends on the encoding. In UTF-8, the character é occupies two bytes. The complete word therefore occupies five bytes.

An encoding describes how characters are represented as bytes.

Unicode assigns a unique number, called a code point, to each character. For example:

UTF-8 is a popular way of representing these Unicode characters as bytes.

2. Strings in Perl

Perl can work with both text and binary data. A Perl scalar can contain a string of Unicode characters or a sequence of byte values.

use utf8;
use feature 'say';

my $text  = "café";
my $bytes = "caf\xC3\xA9";

say length($text);   # 4
say length($bytes);  # 5

The first string contains four Unicode characters. The second contains five byte-valued characters.

The two variables represent the same UTF-8 data, but Perl interprets them differently.

3. Decoding and Encoding

When Perl reads data from a file or another external source, it often receives bytes. To work with these bytes as text, Perl must interpret them using the correct encoding.

This conversion is called decoding.

use Encode qw(decode);

my $bytes = "caf\xC3\xA9";
my $text  = decode('UTF-8', $bytes);

say length($text);   # 4

The opposite conversion is called encoding. It converts text into bytes, for example when you want to write UTF-8 data to a file.

use Encode qw(encode);

my $text  = "café";
my $bytes = encode('UTF-8', $text);

Remember:

• decode() converts bytes into text.
• encode() converts text into bytes.

4. Working with UTF-8 Files

You do not always have to use decode() and encode() yourself. Perl can handle these conversions automatically through filehandle layers.

For example, when reading a UTF-8 file:

open my $fh, '<:encoding(UTF-8)', 'input.txt'
    or die "Cannot open file: $!";

while (my $line = <$fh>) {
    print $line;
}

close $fh;

The :encoding(UTF-8) layer tells Perl that the file contains UTF-8 text. Perl automatically decodes the incoming bytes when reading.

For writing UTF-8 text, use:

open my $fh, '>:encoding(UTF-8)', 'output.txt'
    or die "Cannot open file: $!";

print {$fh} "café";

close $fh;

Perl automatically encodes the text as UTF-8 when writing.

An important rule

There are two ways to decode incoming data:

1. Explicit decoding:

use Encode qw(decode);

my $text = decode('UTF-8', $bytes);

2. Automatic decoding through a filehandle:

open my $fh, '<:encoding(UTF-8)', 'input.txt'
    or die $!;

my $text = <$fh>;

Both approaches convert UTF-8 bytes into Perl text. The second is usually more convenient when reading text files.

Remember:

• decode() explicitly converts bytes into text.
• :encoding(UTF-8) automatically decodes input and encodes output through a filehandle.

There is no need to call decode() again when the filehandle already performs the conversion.

5. What Does use utf8 Do?

You may have noticed this statement in Perl programs:

use utf8;

It tells Perl that your source code uses UTF-8. This allows you to write Unicode characters directly in strings:

use utf8;

my $word = "café";

However, use utf8 does not configure file input or output. It concerns the Perl source code itself.

6. A Common Mistake

Be careful when combining Unicode text and byte strings.

use utf8;

my $text  = "café";
my $bytes = "caf\xC3\xA9";

my $result = $text . $bytes;

The second variable contains UTF-8 bytes, but Perl does not automatically know that they represent UTF-8 text. An implicit conversion can produce unexpected characters, such as é instead of é.

The solution is to decode the bytes first:

use utf8;
use Encode qw(decode);

my $text  = "café";
my $bytes = "caf\xC3\xA9";

my $decoded = decode('UTF-8', $bytes);

my $result = $text . $decoded;

Now both variables contain Unicode text, and they can be combined correctly.

7. Remember These Rules

1. Characters are not bytes. A character can occupy one or more bytes.
2. Unicode identifies characters. UTF-8 represents those characters as bytes.
3. Decode incoming bytes. Convert them into text before processing.
4. Encode outgoing text. Convert it into bytes when writing or transmitting it.
5. Use use utf8 for your source code. It does not configure file input or output.
6. Use filehandle layers when possible. They can perform decoding and encoding automatically.

The most important principle is simple:

Work with text as Unicode inside your program. Decode incoming data and encode outgoing data using the appropriate encoding.

Source: chromatic, Modern Perl, Chapter 1, "The Perl Philosophy," section "Unicode and Strings"