A delimiter is a sequence of one or more characters used to specify the boundary between separate, independent regions in plain text or other data stream. An example of a delimiter is the comma character, which acts as a field delimiter in a sequence of comma-separated values.
Delimiters represent one of various means to specify boundaries in a data stream. There are alternate means as well. Declarative notation, for example, is an alternate method that uses a length field at the start of a data stream to specify the number of characters that the data stream contains.
This article emphasizes the use of delimiters in computing. For more general treatment of delimiters in written human languages, see interword separation.
Delimiters can be broken down into:
Field and Record delimiters; and
Field and record delimiters
Field delimiters separate data fields. Record delimiters separate groups of fields.
Bracket delimiters (also block delimiters, region delimiters, balanced delimiters) mark both the start and end of a region of text. They are used in almost all programming languages, including Wikicode.
Common examples of bracket delimiters include:
( and )
Parentheses. The Lisp programming language syntax is cited as recognizable primarily from its use of parentheses.
used in some web templates to specify language boundaries. These are also called template delimiters.
Delimiter collision is a problem that occurs when an author or programmer introduces delimiters into text without actually intending them to be interpreted as boundaries between separate regions. In the case of Comma-separated values files, for example, this can occur whenever an author attempts to include a comma as part of a field value (e.g., salary = "$30,000"). In the case of XML, for example, this can occur whenever an author attempts to specify an angle bracket character.
In some contexts, a malicious user or attacker may seek to exploit this problem intentionally. Consequently, delimiter collision can be the source of security vulnerabilities and exploits. Malicious users can take advantage of delimiter collision in languages such as SQL and HTML to deploy such well-known attacks as SQL injection and Cross-site scripting, respectively.
Because delimiter collision is a very common problem, various methods for avoiding it have been invented.
One method for avoiding delimiter collision is to use escape characters. From a language design standpoint, these are adequate, but they have drawbacks:
text can be rendered unreadable when littered with numerous escape characters;
they require a mechanism to 'escape the escapes' when not intended as escape characters; and
although easy to type, they can be cryptic to someone unfamiliar with the language.
Escape sequences are similar to escape characters, except they usually consist of some kind of mnemonic instead of just a single character. One use is in string literals that include a doublequote (") character. For example in Perl, the code:
In contrast to escape sequences and escape characters, dual delimiters
provide yet another way to avoid delimiter collision. Some languages, for example, allow the use of either a singlequote (') or a doublequote (") to specify a string literal. For example in Perl:
produces the desired output without requiring escapes. This approach, however, only works when the string does not contain both types of quotation marks.
Multiple quoting delimiters
In contrast to dual delimiters, multiple delimiters are even more flexible for avoiding delimiter collision.
all produce the desired output through use of the quotelike operator, which allows characters to act as delimiters. Although this method is more flexible, few languages support it. Perl and Ruby are two that do.
A content boundary is a special type of delimiter that is specifically designed to resist delimiter collision. It works by allowing the author to specify a long sequence of characters that is guaranteed to always indicate a boundary between parts in a multi-part message, with no other possible interpretation.
This is usually done by specifying a random sequence of characters followed by an identifying mark such as a UUID, a timestamp, or some other distinguishing mark. (See e.g., MIME, Here documents).
In specifying a regular expression, alternate delimiters may also be used to simplify the syntax for match and substitution operations in Perl.
For example, a simple match operation may be specified in perl with the following syntax:
The syntax is flexible enough to specify match operations with alternate delimiters, making it easy to avoid delimiter collision:
Although principally used as a mechanism for text encoding of binary data,
"ASCII armoring" is a programming and systems administration technique
that also helps to avoid delimiter collision in some circumstances. This technique is contrasted from the other approaches described above because it is more complicated, and therefore not suitable for small applications and simple data storage formats. The technique employs a special encoding scheme or hash function, such as base64, to ensure that delimiter characters do not appear in transmitted data.
This technique is used, for example, in Microsoft's ASP.NET web development technology, and is closely associated with the "VIEWSTATE" component of that system.
The following simplified example demonstrates how this technique works in practice.
The first code fragment shows a simple HTML tag in which the VIEWSTATE value contains characters that are incompatible with the delimiters of the HTML tag itself:
This first code fragment is not well-formed, and would therefore not work properly in a "real world" deployed system.
In contrast, the second code fragment shows the same HTML tag, except this time incompatible characters in the VIEWSTATE value are removed through the application of base64 encoding:
This prevents delimiter collision and ensures that incompatible characters will not appear inside the HTML code, regardless of what characters appear in the original (decoded) text.