Using Lexical-Analyzer Generators

Summary: In today's lab, you will explore the standard Unix lexical analyzer generator, lex.

Collaboration: Feel free to work on this lab in pairs or trios.

Turning It In: Save your answers in a plain text file and submit it using the ECA.

Grading: I expect that you will gain more from doing this lab than from me grading this lab. I will simply scan through your answers to see if you had any particularly valuable insights.

Background: lex (or flex, in its GNU implementation) is one of the standard lexical analyzer generators. You can use lex to build lexers and also to build programs that react in other ways to regular expressions they encounter.

You can get more information on flex from the man page. Please don't print the man page; it's over sixty pages long. I'll keep a copy at the front of the room.

Structure of a Lex File:

c-code-prefixed-by-tabs-and-named-regular-expressions
%%
pattern1	c-code-to-execute-upon-matching-pattern
pattern2	c-code-to-execute-upon-matching-pattern
...
patternn	c-code-to-execute-upon-matching-pattern
%%
more-c-code

Sample Lex File:

%%
" "		{ }
\n              { printf("Howdy!\n"); }
begin           { printf("Keyword-Begin\n"); }
[a-z][a-z0-9_]* { printf("Look, an identifier: '%s'\n",yytext); }
%%
int yywrap() { return 1; }

The yywrap function gets called by the lexer whenever it hits the end of file. If yywrap returns a nonzero value, the lexer stops.

Using Lex Files:

Suppose we saved this as example.yy (yes, .yy is the normal suffix for lex files). To build an executable from it, you would do the following:

% flex example.yy
	# Generates file lex.yy.c
% gcc -lfl lex.yy.c
	# Generates file a.out
% a.out < file

Problems:

1. Figure out what the example file does with things not described in the regular expressions.

2. Use flex to separate input into

• (a) White space (grouped together) and (b) non-whitespace.
• (a) Sequences of a's and (b) everything else
• (a) Unparameterized XHTML tags and (b) everything else
• (a) XHTML tags with parameters and (b) everything else
• (a) C comments and (b) everything else
• (a) Pascal comments and (b) everything else
• (a) 4 keywords (of your choice), (b) identifiers, and (c) everything else

3. Look at the lex.yy.c and see if you can figure out where and how the finite automaton is coded. You might want to try variants of the lex program to see what difference it makes.