Mercurial > ~dholland > hg > ag > index.cgi

comparison doc/misc/html/examples/mpp/token.html @ 0:13d2b8934445

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Import AnaGram (near-)release tree into Mercurial.
author David A. Holland
date Sat, 22 Dec 2007 17:52:45 -0500
parents
children
comparison
equal deleted inserted replaced
-1:000000000000 0:13d2b8934445
1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
2 <HTML>
3 <HEAD>
4 <TITLE> Token Definitions - Macro preprocessor and C Parser </TITLE>
5 </HEAD>
6
7
8 <BODY BGCOLOR="#ffffff" BACKGROUND="tilbl6h.gif"
9 TEXT="#000000" LINK="#0033CC"
10 VLINK="#CC0033" ALINK="#CC0099">
11
12 <P>
13 <IMG ALIGN="right" SRC="../../images/agrsl6c.gif" ALT="AnaGram"
14 WIDTH=124 HEIGHT=30 >
15 <BR CLEAR="all">
16 Back to :
17 <A HREF="../../index.html">Index</A> |
18 <A HREF="index.html">Macro preprocessor overview</A>
19 <P>
20 <IMG ALIGN="bottom" SRC="../../images/rbline6j.gif" ALT="----------------------"
21 WIDTH=1010 HEIGHT=2 >
22 <P>
23
24 <H1> Token Definitions - Macro preprocessor and C Parser </H1>
25 <IMG ALIGN="bottom" SRC="../../images/rbline6j.gif" ALT="----------------------"
26 WIDTH=1010 HEIGHT=2 >
27 <P>
28 <BR>
29
30 <H2>Introduction</H2>
31 <P>
32
33 The token module, <tt>token.cpp</tt>, and its header file,
34 <tt>token.h</tt>, provide
35 detailed support to the macro preprocessor for manipulating and
36 transmitting tokens between processes.
37 <P>
38 <BR>
39
40 <H2> Type and Class Definitions </H2>
41 <H4> <tt>token</tt>, <tt>token_id</tt> </H4>
42
43 The macro preprocessor includes a token scanner module,
44 <tt>ts.syn</tt>, which identifies legitimate C tokens in the input
45 stream and then uses a string dictionary, <tt>td</tt>, also known as
46 the "token dictionary", to keep track of the particular
47 sequence of characters which make up each token. The actual
48 unit of data which is passed from function to function is
49 also called a token and is a pair consisting of the
50 dictionary index and a type field. An <tt>enum</tt> statement in
51 <tt>token.h</tt> provides definitions for the various
52 types. The <tt>enum</tt> type is called <tt>token_id</tt>.
53 <P>
54 There are several things to note about the type definition.
55 First, all single character punctuation tokens are given an
56 ID corresponding to their ASCII character code. Multi-
57 character punctuation tokens are given codes in the range
58 normally occupied by the capital letters. Reserved words,
59 numeric constants, and alphabetic tokens are given
60 identifiers above 128 so they cannot be confused with ASCII
61 characters.
62 <P>
63 The syntax files which accept
64 token input (<tt>ex.syn</tt>, <tt>mas.syn</tt>,
65 <tt>jrc.syn</tt>) have matching <tt>enum</tt> statements. Because
66 punctuation characters are identified by their ASCII codes,
67 these parsers can simply use the character representation
68 for such tokens.
69 <P>
70
71 <H4> <tt>token_sink</tt> </H4>
72 A token sink is an abstract class much like a
73 <A HREF="../../oldclasslib/charsink.html">
74 character_sink </A>. Derived classes, however, comprise the
75 <tt>token_accumulator</tt> class, the <tt>token_translator
76 class</tt>, the
77 <tt>c_parser</tt> class, and the
78 <tt>expression_evaluator</tt> class. The
79 macro substitution parser, <tt>mas</tt>, could also have been
80 configured as a <tt>token_sink</tt>; however, the point of this
81 particular example is to show a number of different ways of
82 interfacing parsers, so a different technique was used for
83 mas.
84 <P>
85
86 <H4> <tt>token_accumulator</tt> </H4>
87 The <tt>token_accumulator</tt> class is derived from the
88 <tt>token_sink</tt>
89 class. It is identical in function to the
90 <tt>string_accumulator</tt>
91 class except that it uses strings of tokens instead of
92 strings of characters. It uses the <tt>END_OF_FILE</tt> token as a
93 string terminator.
94 <P>
95
96 <H4> <tt>token_translator</tt> </H4>
97 The <tt>token_translator</tt> class is similar to the
98 <tt>output_file</tt>
99 class, except that it takes a pointer to a character sink as
100 an argument to the constructor. This allows the
101 <tt>token_translator</tt> to be used to output a token sequence to a
102 file, or to translate a token sequence into a string in a
103 <tt>string_accumulator</tt>.
104 <P>
105 The <tt>space_required</tt> flag is used to insert blank characters
106 where necessary to separate alphanumeric tokens from each
107 other.
108 <P>
109 <BR>
110
111 <IMG ALIGN="bottom" SRC="../../images/rbline6j.gif" ALT="----------------------"
112 WIDTH=1010 HEIGHT=2 >
113 <P>
114 <IMG ALIGN="right" SRC="../../images/pslrb6d.gif" ALT="Parsifal Software"
115 WIDTH=181 HEIGHT=25>
116 <BR CLEAR="right">
117
118 <P>
119 Back to :
120 <A HREF="../../index.html">Index</A> |
121 <A HREF="index.html">Macro preprocessor overview</A>
122 <P>
123
124 <ADDRESS><FONT SIZE="-1">
125 AnaGram parser generator - examples<BR>
126 Token Definitions - Macro preprocessor and C Parser <BR>
127 Copyright &copy; 1993-1999, Parsifal Software. <BR>
128 All Rights Reserved.<BR>
129 </FONT></ADDRESS>
130
131 </BODY>
132 </HTML>
133