Delimiter

A delimiter is a sequence of one or more characters used to specify the boundary between separate, independent regions in plain text or other data stream. An example of a delimiter is the comma character, which acts as a field delimiter in a sequence of comma-separated values.

Delimiters represent one of various means to specify boundaries in a data stream. There are alternate means as well. Declarative notation, for example, is an alternate method that uses a length field at the start of a data stream to specify the number of characters that the data stream contains.

This article emphasizes the use of delimiters in computing. For more general treatment of delimiters in written human languages, see interword separation.

Overview

• Field and Record delimiters; and
• Bracket delimiters.

Field and record delimiters

For example, the (CSV) file format uses a comma as the delimiter between fields, and a end-of-line indicator as the delimiter between records. For instance: fname,lname,age,salary nancy,davolio,33,$30000 erin,borakova,28,$25250 tony,raphael,35,$28700 specifies a simple flat file database table using the (CSV) file format.

Conventions

Computing platforms historically use certain delimiters by convention. The following tables depict just a few examples for comparison.

Programming languages (See also, Comparison of programming languages (syntax)).

   !

   >-

   ! Pascal

   | singlequote

   >-

   ! C

   | doublequote, singlequote

   >-

Field and Record delimiters (See also, ASCII, Control character).

   !

   >-

   ! Unix (and Mac OS X)

   | Tab

   >-

   ! Windows

   | Tab

   >-

   ! Classic Mac OS

   | Tab

   >-

   ! Unicode

   | UNIT SEPARATOR
Position 31 (U+001F)

Bracket delimiters

Bracket delimiters (also block delimiters, region delimiters, balanced delimiters) mark both the start and end of a region of text. They are used in almost all programming languages, including Wikicode.

Common examples of bracket delimiters include:

Delimiter collision

In some contexts, a malicious user or attacker may seek to exploit this problem intentionally. Consequently, delimiter collision can be the source of security vulnerabilities and exploits. Malicious users can take advantage of delimiter collision in languages such as SQL and HTML to deploy such well-known attacks as SQL injection and Cross-site scripting, respectively.

Solutions

Because delimiter collision is a very common problem, various methods for avoiding it have been invented.

Escape character

• text can be rendered unreadable when littered with numerous escape characters;
• they require a mechanism to 'escape the escapes' when not intended as escape characters; and
• although easy to type, they can be cryptic to someone unfamiliar with the language.

Escape sequence

print "Nancy said x34Hello World!x34 to the crowd."; ### use x34

produces the same output as:

print "Nancy said "Hello World!" to the crowd."; ### use escape char

One drawback of escape sequences, when used by people, is the need to memorize the codes that represent individual characters (see also: character entity reference, numeric character reference).

Dual quoting delimiters

print 'Nancy said "Hello World!" to the crowd.';

produces the desired output without requiring escapes. This approach, however, only works when the string does not contain both types of quotation marks.

Multiple quoting delimiters

For example in Perl:

print qq^Nancy doesn't want to say "Hello World!" anymore.^; print qq@Nancy doesn't want to say "Hello World!" anymore.@; print qq§Nancy doesn't want to say "Hello World!" anymore.§;

all produce the desired output through use of the quotelike operator, which allows characters to act as delimiters. Although this method is more flexible, few languages support it. Perl and Ruby are two that do.

Content boundary

This is usually done by specifying a random sequence of characters followed by an identifying mark such as a UUID, a timestamp, or some other distinguishing mark. (See e.g., MIME, Here documents).

Whitespace or indentation

Regular expression syntax

In specifying a regular expression, alternate delimiters may also be used to simplify the syntax for match and substitution operations in Perl.

For example, a simple match operation may be specified in perl with the following syntax:

$string1 = 'Nancy said "Hello World!" to the crowd.'; # specify a target string print $string1 =~ m/[aeiou]+/; # match one or more vowels

The syntax is flexible enough to specify match operations with alternate delimiters, making it easy to avoid delimiter collision:

$string1 = 'Nancy said "httq://Hello/World.htm" is not a valid address.'; # target string

print $string1 =~ m@httq://@; # match using alternate regular expression delimiter print $string1 =~ m{httq://}; # same as previous, but different delimiter print $string1 =~ m!httq://!; # same as previous, but different delimiter

ASCII armor

This technique is used, for example, in Microsoft's ASP.NET web development technology, and is closely associated with the "VIEWSTATE" component of that system.

Example

The first code fragment shows a simple HTML tag in which the VIEWSTATE value contains characters that are incompatible with the delimiters of the HTML tag itself:

This first code fragment is not well-formed, and would therefore not work properly in a "real world" deployed system.

In contrast, the second code fragment shows the same HTML tag, except this time incompatible characters in the VIEWSTATE value are removed through the application of base64 encoding:

This prevents delimiter collision and ensures that incompatible characters will not appear inside the HTML code, regardless of what characters appear in the original (decoded) text.

Notes and references

See also

• Delimiter-separated values
• String literal
• CamelCase (used in WikiWikiWeb as an alternate method of link creation that does not require delimiters to indicate links)
• Federal Standard 1037C (contains a simple definition for "delimiter")
• Naming collision
• Sigil