Perl
----

RPMs
     perl

Ubuntu packages
     perl, libintl-perl

File extension
     pl, PL, pm, perl, cgi

String syntax

       * "abc"

       * 'abc'

       * qq (abc)

       * q (abc)

       * qr /abc/

       * qx (/bin/date)

       * /pattern match/

       * ?pattern match?

       * s/substitution/operators/

       * $tied_hash{"message"}

       * $tied_hash_reference->{"message"}

       * etc., issue the command man perlsyn for details

gettext shorthand
     __ (double underscore)

gettext/ngettext functions
     gettext, dgettext, dcgettext, ngettext, dngettext, dcngettext, pgettext, dpgettext, dcpgettext, npgettext, dnpgettext, dcnpgettext

textdomain
     textdomain function

bindtextdomain
     bindtextdomain function

bind_textdomain_codeset
     bind_textdomain_codeset function

setlocale
     Use setlocale (LC_ALL, "");

Prerequisite
     use POSIX;

     use Locale::TextDomain; (included in the package libintl-perl which is available on the Comprehensive Perl Archive Network CPAN, https://www.cpan.org/).

Use or emulate GNU gettext
     platform dependent: gettext_pp emulates, gettext_xs uses GNU gettext

Extractor
     xgettext -k__ -k\$__ -k%__ -k__x -k__n:1,2 -k__nx:1,2 -k__xn:1,2 -kN__ -kN__n:1,2 -k__p:1c,2 -k__np:1c,2,3 -kN__p:1c,2 -kN__np:1c,2,3

Formatting with positions
     Both kinds of format strings support formatting with positions.

     printf "%2\$d %1\$d", ... (requires Perl 5.8.0 or newer)

     __expand("[new] replaces [old]", old => $oldvalue, new => $newvalue)

Portability
     The libintl-perl package is platform independent but is not part of the Perl core.  The programmer is responsible for providing a dummy implementation of the required functions if the package is not installed on the target system.

po-mode marking
     ---

Documentation
     Included in libintl-perl, available on CPAN (https://www.cpan.org/).

An example is available in the examples directory: hello-perl.

The xgettext parser backend for Perl differs significantly from the parser backends for other programming languages, just as Perl itself differs significantly from other programming languages.  The Perl parser backend offers many more string marking facilities than the other backends but it also has some Perl specific limitations, the worst probably being its imperfectness.

General Problems Parsing Perl Code
..................................

It is often heard that only Perl can parse Perl.  This is not true. Perl cannot be parsed at all, it can only be executed. Perl has various built-in ambiguities that can only be resolved at runtime.

The following example may illustrate one common problem:

     print gettext "Hello World!";

Although this example looks like a bullet-proof case of a function invocation, it is not:

     open gettext, ">testfile" or die;
     print gettext "Hello world!"

In this context, the string gettext looks more like a file handle.  But not necessarily:

     use Locale::Messages qw (:libintl_h);
     open gettext ">testfile" or die;
     print gettext "Hello world!";

Now, the file is probably syntactically incorrect, provided that the module Locale::Messages found first in the Perl include path exports a function gettext.  But what if the module Locale::Messages really looks like this?

     use vars qw (*gettext);

     1;

In this case, the string gettext will be interpreted as a file handle again, and the above example will create a file testfile and write the string ``Hello world!'' into it.  Even advanced control flow analysis will not really help:

     if (0.5 < rand) {
        eval "use Sane";
     } else {
        eval "use InSane";
     }
     print gettext "Hello world!";

If the module Sane exports a function gettext that does what we expect, and the module InSane opens a file for writing and associates the handle gettext with this output stream, we are clueless again about what will happen at runtime.  It is completely unpredictable.  The truth is that Perl has so many ways to fill its symbol table at runtime that it is impossible to interpret a particular piece of code without executing it.

Of course, xgettext will not execute your Perl sources while scanning for translatable strings, but rather use heuristics in order to guess what you meant.

Another problem is the ambiguity of the slash and the question mark. Their interpretation depends on the context:

     # A pattern match.
     print "OK\n" if /foobar/;

     # A division.
     print 1 / 2;

     # Another pattern match.
     print "OK\n" if ?foobar?;

     # Conditional.
     print $x ? "foo" : "bar";

The slash may either act as the division operator or introduce a pattern match, whereas the question mark may act as the ternary conditional operator or as a pattern match, too.  Other programming languages like awk present similar problems, but the consequences of a misinterpretation are particularly nasty with Perl sources.  In awk for instance, a statement can never exceed one line and the parser can recover from a parsing error at the next newline and interpret the rest of the input stream correctly.  Perl is different, as a pattern match is terminated by the next appearance of the delimiter (the slash or the question mark) in the input stream, regardless of the semantic context.  If a slash is really a division sign but mis-interpreted as a pattern match, the rest of the input file is most probably parsed incorrectly.

There are certain cases, where the ambiguity cannot be resolved at all:

     $x = wantarray ? 1 : 0;

The Perl built-in function wantarray does not accept any arguments. The Perl parser therefore knows that the question mark does not start a regular expression but is the ternary conditional operator.

     sub wantarrays {}
     $x = wantarrays ? 1 : 0;

Now the situation is different.  The function wantarrays takes a variable number of arguments (like any non-prototyped Perl function). The question mark is now the delimiter of a pattern match, and hence the piece of code does not compile.

     sub wantarrays() {}
     $x = wantarrays ? 1 : 0;

Now the function is prototyped, Perl knows that it does not accept any arguments, and the question mark is therefore interpreted as the ternaray operator again.  But that unfortunately outsmarts xgettext.

The Perl parser in xgettext cannot know whether a function has a prototype and what that prototype would look like.  It therefore makes an educated guess.  If a function is known to be a Perl built-in and this function does not accept any arguments, a following question mark or slash is treated as an operator, otherwise as the delimiter of a following regular expression.  The Perl built-ins that do not accept arguments are wantarray, fork, time, times, getlogin, getppid, getpwent, getgrent, gethostent, getnetent, getprotoent, getservent, setpwent, setgrent, endpwent, endgrent, endhostent, endnetent, endprotoent, and endservent.

If you find that xgettext fails to extract strings from portions of your sources, you should therefore look out for slashes and/or question marks preceding these sections.  You may have come across a bug in xgettext's Perl parser (and of course you should report that bug).  In the meantime you should consider to reformulate your code in a manner less challenging to xgettext.

In particular, if the parser is too dumb to see that a function does not accept arguments, use parentheses:

     $x = somefunc() ? 1 : 0;
     $y = (somefunc) ? 1 : 0;

In fact the Perl parser itself has similar problems and warns you about such constructs.

Which keywords will xgettext look for?
......................................

Unless you instruct xgettext otherwise by invoking it with one of the options --keyword or -k, it will recognize the following keywords in your Perl sources:

  * gettext

  * dgettext:2

     The second argument will be extracted.

  * dcgettext:2

     The second argument will be extracted.

  * ngettext:1,2

     The first (singular) and the second (plural) argument will be extracted.

  * dngettext:2,3

     The second (singular) and the third (plural) argument will be extracted.

  * dcngettext:2,3

     The second (singular) and the third (plural) argument will be extracted.

  * pgettext:1c,2

     The first (message context) and the second argument will be extracted.

  * dpgettext:2c,3

     The second (message context) and the third argument will be extracted.

  * dcpgettext:2c,3

     The second (message context) and the third argument will be extracted.

  * npgettext:1c,2,3

     The first (message context), second (singular), and third (plural) argument will be extracted.

  * dnpgettext:2c,3,4

     The second (message context), third (singular), and fourth (plural) argument will be extracted.

  * dcnpgettext:2c,3,4

     The second (message context), third (singular), and fourth (plural) argument will be extracted.

  * gettext_noop

  * %gettext

     The keys of lookups into the hash %gettext will be extracted.

  * $gettext

     The keys of lookups into the hash reference $gettext will be extracted.

How to Extract Hash Keys
........................

Translating messages at runtime is normally performed by looking up the original string in the translation database and returning the translated version.  The ``natural'' Perl implementation is a hash lookup, and, of course, xgettext supports such practice.

     print __"Hello world!";
     print $__{"Hello world!"};
     print $__->{"Hello world!"};
     print $$__{"Hello world!"};

The above four lines all do the same thing.  The Perl module Locale::TextDomain exports by default a hash %__ that is tied to the function __().  It also exports a reference $__ to %__.

If an argument to the xgettext option --keyword, resp. -k starts with a percent sign, the rest of the keyword is interpreted as the name of a hash.  If it starts with a dollar sign, the rest of the keyword is interpreted as a reference to a hash.

Note that you can omit the quotation marks (single or double) around the hash key (almost) whenever Perl itself allows it:

     print $gettext{Error};

The exact rule is: You can omit the surrounding quotes, when the hash key is a valid C (!) identifier, i.e. when it starts with an underscore or an ASCII letter and is followed by an arbitrary number of underscores, ASCII letters or digits.  Other Unicode characters are not allowed, regardless of the use utf8 pragma.

What are Strings And Quote-like Expressions?
............................................

Perl offers a plethora of different string constructs.  Those that can be used either as arguments to functions or inside braces for hash lookups are generally supported by xgettext.

  * double-quoted strings

          print gettext "Hello World!";

  * single-quoted strings

          print gettext 'Hello World!';

  * the operator qq

          print gettext qq |Hello World!|;
          print gettext qq <E-mail: <guido\@imperia.net>>;

     The operator qq is fully supported.  You can use arbitrary delimiters, including the four bracketing delimiters (round, angle, square, curly) that nest.

  * the operator q

          print gettext q |Hello World!|;
          print gettext q <E-mail: <guido@imperia.net>>;

     The operator q is fully supported.  You can use arbitrary delimiters, including the four bracketing delimiters (round, angle, square, curly) that nest.

  * the operator qx

          print gettext qx ;LANGUAGE=C /bin/date;
          print gettext qx [/usr/bin/ls | grep '^[A-Z]*'];

     The operator qx is fully supported.  You can use arbitrary delimiters, including the four bracketing delimiters (round, angle, square, curly) that nest.

     The example is actually a useless use of gettext.  It will invoke the gettext function on the output of the command specified with the qx operator.  The feature was included in order to make the interface consistent (the parser will extract all strings and quote-like expressions).

  * here documents

          print gettext <<'EOF';
          program not found in $PATH
          EOF

          print ngettext <<EOF, <<"EOF";
          one file deleted
          EOF
          several files deleted
          EOF

     Here-documents are recognized.  If the delimiter is enclosed in single quotes, the string is not interpolated.  If it is enclosed in double quotes or has no quotes at all, the string is interpolated.

     Delimiters that start with a digit are not supported!

Invalid Uses Of String Interpolation
....................................

Perl is capable of interpolating variables into strings.  This offers some nice features in localized programs but can also lead to problems.

A common error is a construct like the following:

     print gettext "This is the program $0!\n";

Perl will interpolate at runtime the value of the variable $0 into the argument of the gettext() function.  Hence, this argument is not a string constant but a variable argument ($0 is a global variable that holds the name of the Perl script being executed).  The interpolation is performed by Perl before the string argument is passed to gettext() and will therefore depend on the name of the script which can only be determined at runtime. Consequently, it is almost impossible that a translation can be looked up at runtime (except if, by accident, the interpolated string is found in the message catalog).

The xgettext program will therefore terminate parsing with a fatal error if it encounters a variable inside of an extracted string.  In general, this will happen for all kinds of string interpolations that cannot be safely performed at compile time.  If you absolutely know what you are doing, you can always circumvent this behavior:

     my $know_what_i_am_doing = "This is program $0!\n";
     print gettext $know_what_i_am_doing;

Since the parser only recognizes strings and quote-like expressions, but not variables or other terms, the above construct will be accepted.  You will have to find another way, however, to let your original string make it into your message catalog.

If invoked with the option --extract-all, resp. -a, variable interpolation will be accepted.  Rationale: You will generally use this option in order to prepare your sources for internationalization.

Please see the manual page man perlop for details of strings and quote-like expressions that are subject to interpolation and those that are not.  Safe interpolations (that will not lead to a fatal error) are:

  * the escape sequences \t (tab, HT, TAB), \n (newline, NL), \r (return, CR), \f (form feed, FF), \b (backspace, BS), \a (alarm, bell, BEL), and \e (escape, ESC).

  * octal chars, like \033 Note that octal escapes in the range of 400-777 are translated into a UTF-8 representation, regardless of the presence of the use utf8 pragma.

  * hex chars, like \x1b

  * wide hex chars, like \x{263a} Note that this escape is translated into a UTF-8 representation, regardless of the presence of the use utf8 pragma.

  * control chars, like \c[ (CTRL-[)

  * named Unicode chars, like \N{LATIN CAPITAL LETTER C WITH CEDILLA} Note that this escape is translated into a UTF-8 representation, regardless of the presence of the use utf8 pragma.

The following escapes are considered partially safe:

  * \l lowercase next char

  * \u uppercase next char

  * \L lowercase till \E

  * \U uppercase till \E

  * \E end case modification

  * \Q quote non-word characters till \E

These escapes are only considered safe if the string consists of ASCII characters only.  Translation of characters outside the range defined by ASCII is locale-dependent and can actually only be performed at runtime; xgettext doesn't do these locale-dependent translations at extraction time.

Except for the modifier \Q, these translations, albeit valid, are generally useless and only obfuscate your sources.  If a translation can be safely performed at compile time you can just as well write what you mean.

Valid Uses Of String Interpolation
..................................

Perl is often used to generate sources for other programming languages or arbitrary file formats.  Web applications that output HTML code make a prominent example for such usage.

You will often come across situations where you want to intersperse code written in the target (programming) language with translatable messages, like in the following HTML example:

     print gettext <<EOF;
     <h1>My Homepage</h1>
     <script language="JavaScript"><!--
     for (i = 0; i < 100; ++i) {
         alert ("Thank you so much for visiting my homepage!");
     }
     //--></script>
     EOF

The parser will extract the entire here document, and it will appear entirely in the resulting PO file, including the JavaScript snippet embedded in the HTML code.  If you exaggerate with constructs like the above, you will run the risk that the translators of your package will look out for a less challenging project.  You should consider an alternative expression here:

     print <<EOF;
     <h1>$gettext{"My Homepage"}</h1>
     <script language="JavaScript"><!--
     for (i = 0; i < 100; ++i) {
         alert ("$gettext{'Thank you so much for visiting my homepage!'}");
     }
     //--></script>
     EOF

Only the translatable portions of the code will be extracted here, and the resulting PO file will begrudgingly improve in terms of readability.

You can interpolate hash lookups in all strings or quote-like expressions that are subject to interpolation (see the manual page man perlop for details).  Double interpolation is invalid, however:

     # TRANSLATORS: Replace "the earth" with the name of your planet.
     print gettext qq{Welcome to $gettext->{"the earth"}};

The qq-quoted string is recognized as an argument to xgettext in the first place, and checked for invalid variable interpolation.  The dollar sign of hash-dereferencing will therefore terminate the parser with an ``invalid interpolation'' error.

It is valid to interpolate hash lookups in regular expressions:

     if ($var =~ /$gettext{"the earth"}/) {
        print gettext "Match!\n";
     }
     s/$gettext{"U. S. A."}/$gettext{"U. S. A."} $gettext{"(dial +0)"}/g;

When To Use Parentheses
.......................

In Perl, parentheses around function arguments are mostly optional. xgettext will always assume that all recognized keywords (except for hashes and hash references) are names of properly prototyped functions, and will (hopefully) only require parentheses where Perl itself requires them.  All constructs in the following example are therefore ok to use:

     print gettext ("Hello World!\n");
     print gettext "Hello World!\n";
     print dgettext ($package => "Hello World!\n");
     print dgettext $package, "Hello World!\n";

     # The "fat comma" => turns the left-hand side argument into a
     # single-quoted string!
     print dgettext smellovision => "Hello World!\n";

     # The following assignment only works with prototyped functions.
     # Otherwise, the functions will act as "greedy" list operators and
     # eat up all following arguments.
     my $anonymous_hash = {
        planet => gettext "earth",
        cakes => ngettext "one cake", "several cakes", $n,
        still => $works,
     };
     # The same without fat comma:
     my $other_hash = {
        'planet', gettext "earth",
        'cakes', ngettext "one cake", "several cakes", $n,
        'still', $works,
     };

     # Parentheses are only significant for the first argument.
     print dngettext 'package', ("one cake", "several cakes", $n), $discarded;

How To Grok with Long Lines
...........................

The necessity of long messages can often lead to a cumbersome or unreadable coding style.  Perl has several options that may prevent you from writing unreadable code, and xgettext does its best to do likewise.  This is where the dot operator (the string concatenation operator) may come in handy:

     print gettext ("This is a very long"
                    . " message that is still"
                    . " readable, because"
                    . " it is split into"
                    . " multiple lines.\n");

Perl is smart enough to concatenate these constant string fragments into one long string at compile time, and so is xgettext.  You will only find one long message in the resulting POT file.

Note that the future Perl 6 will probably use the underscore (_) as the string concatenation operator, and the dot (.) for dereferencing.  This new syntax is not yet supported by xgettext.

If embedded newline characters are not an issue, or even desired, you may also insert newline characters inside quoted strings wherever you feel like it:

     print gettext ("<em>In HTML output
     embedded newlines are generally no
     problem, since adjacent whitespace
     is always rendered into a single
     space character.</em>");

You may also consider to use here documents:

     print gettext <<EOF;
     <em>In HTML output
     embedded newlines are generally no
     problem, since adjacent whitespace
     is always rendered into a single
     space character.</em>
     EOF

Please do not forget that the line breaks are real, i.e. they translate into newline characters that will consequently show up in the resulting POT file.

Bugs, Pitfalls, And Things That Do Not Work
...........................................

The foregoing sections should have proven that xgettext is quite smart in extracting translatable strings from Perl sources.  Yet, some more or less exotic constructs that could be expected to work, actually do not work.

One of the more relevant limitations can be found in the implementation of variable interpolation inside quoted strings.  Only simple hash lookups can be used there:

     print <<EOF;
     $gettext{"The dot operator"
               . " does not work"
               . "here!"}
     Likewise, you cannot @{[ gettext ("interpolate function calls") ]}
     inside quoted strings or quote-like expressions.
     EOF

This is valid Perl code and will actually trigger invocations of the gettext function at runtime.  Yet, the Perl parser in xgettext will fail to recognize the strings.  A less obvious example can be found in the interpolation of regular expressions:

     s/<!--START_OF_WEEK-->/gettext ("Sunday")/e;

The modifier e will cause the substitution to be interpreted as an evaluable statement.  Consequently, at runtime the function gettext() is called, but again, the parser fails to extract the string ``Sunday''.  Use a temporary variable as a simple workaround if you really happen to need this feature:

     my $sunday = gettext "Sunday";
     s/<!--START_OF_WEEK-->/$sunday/;

Hash slices would also be handy but are not recognized:

     my @weekdays = @gettext{'Sunday', 'Monday', 'Tuesday', 'Wednesday',
                             'Thursday', 'Friday', 'Saturday'};
     # Or even:
     @weekdays = @gettext{qw (Sunday Monday Tuesday Wednesday Thursday
                              Friday Saturday) };

This is perfectly valid usage of the tied hash %gettext but the strings are not recognized and therefore will not be extracted.

Another caveat of the current version is its rudimentary support for non-ASCII characters in identifiers.  You may encounter serious problems if you use identifiers with characters outside the range of 'A'-'Z', 'a'-'z', '0'-'9' and the underscore '_'.

Maybe some of these missing features will be implemented in future versions, but since you can always make do without them at minimal effort, these todos have very low priority.

A nasty problem are brace format strings that already contain braces as part of the normal text, for example the usage strings typically encountered in programs:

     die "usage: $0 {OPTIONS} FILENAME...\n";

If you want to internationalize this code with Perl brace format strings, you will run into a problem:

     die __x ("usage: {program} {OPTIONS} FILENAME...\n", program => $0);

Whereas {program} is a placeholder, {OPTIONS} is not and should probably be translated. Yet, there is no way to teach the Perl parser in xgettext to recognize the first one, and leave the other one alone.

There are two possible work-arounds for this problem.  If you are sure that your program will run under Perl 5.8.0 or newer (these Perl versions handle positional parameters in printf()) or if you are sure that the translator will not have to reorder the arguments in her translation -- for example if you have only one brace placeholder in your string, or if it describes a syntax, like in this one --, you can mark the string as no-perl-brace-format and use printf():

     # xgettext: no-perl-brace-format
     die sprintf ("usage: %s {OPTIONS} FILENAME...\n", $0);

If you want to use the more portable Perl brace format, you will have to do put placeholders in place of the literal braces:

     die __x ("usage: {program} {[}OPTIONS{]} FILENAME...\n",
              program => $0, '[' => '{', ']' => '}');

Perl brace format strings know no escaping mechanism.  No matter how this escaping mechanism looked like, it would either give the programmer a hard time, make translating Perl brace format strings heavy-going, or result in a performance penalty at runtime, when the format directives get executed.  Most of the time you will happily get along with printf() for this special case.