Lexical analysis - Compiler Design

Jeena Thomas, Asst Professor, CSE, SJCET Palai
1

» The scanning/lexical analysis phase of a compiler
performs the task of reading the source program
as a file of characters and dividing up into
tokens.
» Usually implemented as subroutine or co-routine
of parser.
» Front end of compiler.
2

» Each token is a sequence of characters that
represents a unit of information in the source
program.
3

» Keywords which are fixed string of letters .eg: “if”,
“while”.
» Identifiers which are user-defined strings
composed of letters and numbers.
» Special symbols like arithmetic symbols.
4

» Scanners perform pattern matching process.
» The techniques used to implement lexical analyzers
can also be applied to other areas such as query
languages and information retrieval systems.
» Since pattern directed programming is widely
useful, pattern action language called Lex for
specifying lexical analyzers.
» In lex , patterns are specified by regular
expressions, and a compiler for lex can generate an
efficient finite-automaton recognizer for the
regular expression.
5

» A software tool that automates the
construction of lexical analyzers allows people
with different backgrounds to use pattern
matching in their own areas.
» Jarvis[1976] Lexical analyzer generator to create
a program that recognizes imperfections in
printed circuit boards.
» The circuits are digitally scanned and
converted into “strings” of line segments at
different angles.
» The “lexical analyzer” looked for patterns
corresponding to imperfections in the string of
line segments. 6

» It can utilize the best-known pattern-matching
algorithms and thereby create efficient lexical
analyzers for people who are not experts in
pattern-matching techniques.
7

Winter2007
SE
G2
10
1
Ch
ap
ter
8
8
» Lexical analyzer is the first phase of a compiler.
» Its main task is to read input characters and produce as
output a sequence of tokens that parser uses for syntax
analysis.

9

Type Examples
ID foo n_14 last
NUM 73 00 517 082
REAL 66.1 .5 10. 1e67 5.5e-10
IF if
COMMA ,
NOTEQ !=
LPAREN (
RPAREN )

Type Examples
comment /* ignored */
preprocessor directive #include <foo.h>
#define NUMS 5, 6
macro NUMS
whitespace t n b

» Separation of the input source code into tokens.
» Stripping out the unnecessary white spaces from
the source code.
» Removing the comments from the source text.
» Keeping track of line numbers while scanning the
new line characters. These line numbers are used
by the error handler to print the error messages.
» Preprocessing of macros.
12

» There are several reasons for separating the
analysis phase of compiling into lexical analysis
and parsing:
» It leads to simpler design of the parser as the
unnecessary tokens can be eliminated by scanner.
» Efficiency of the process of compilation is
improved. The lexical analysis phase is most time
consuming phase in compilation. Using specialized
buffering to improve the speed of compilation.
» Portability of the compiler is enhanced as the
specialized symbols and characters(language and
machine specific) are isolated during this phase. 13

» Connected with lexical analysis are three important
terms with similar meaning.
» Lexeme
» Token
» Patterns
14

» A token is a pair consisting of a token name and
an optional attribute value. Token name:
Keywords, operators, identifiers, constants, literal strings,
punctuation symbols(such as commas,semicolons)
» A lexeme is a sequence of characters in the
source program that matches the pattern for a
token and is identified by the lexical analyzer as
an instance of that token. E.g.Relation
{<.<=,>,>=,==,<>}
15

» A pattern is a description of the form that
the lexemes of token may take.
» It gives an informal or formal description of
a token.
» Eg: identifier
» 2 purposes
» Gives a precise description/ specification of
tokens.
» Used to automatically generate a lexical
analyzer
16

17
» const pi = 3.1416;
» The substring pi is a lexeme for the token
“identifier.”

» x=x*(acc+123)
18

19

» 1.) let us consider a statement “fi(a==f)”. Here “fi”
is a misspelled keyword. This error is not
detected in lexical analysis as “fi” is taken as an
identifier. This error is then detected in other
phases of compilation.
» 2.) in case the lexical analyzer is not able to
continue with the process of compilation, it
resorts to panic mode of error recovery.
• Deleting the successive characters from the
remaining input until a token is detected.
• Deleting extraneous characters.
20

• Inserting missing characters
• Replacing an incorrect character by a correct
character.
• Transposing two adjacent characters
21

» Is the strategy generally followed by the
lexical analyzer to correct the errors in the
lexemes.
» It is nothing but the minimum number of the
corrections to be made to convert an invalid
lexeme to a valid lexeme.
» But it is not generally used in practice
because it is too costly to implement.
22

23

» Scanners are special pattern matching
processors.
» For representing patterns of strings of
characters, Regular Expressions(RE) are used.
» A regular expression (r) is defined by set of
strings that matches it.
» This set is called as the language generated by
the regular expression and is represented as
L(r).
» The set of symbols in the language is called the
alphabet of the language is represented as ∑. 24

» An alphabet is a finite set of symbols.
» Example
» A set of alphabetic characters is represented as
L={A,…,Z,a,…,z} and set of digits is represented as
D={0,1,…,9}.
» LUD is a language.
» Strings over LUD- Begin,Max1, max1, 123, €…
25

26

» Intersection
» L∩M={ s|s is in L and S is in M}
» Exponentiation
» Li
=L Li-1
27

» Choice among alternates
» Concatenation
» Repetition
28

» Indicated by metacharacter ‘|’(vertical bar)
» r|s
» R.E that matches any string that is matched either
by r or s.
» L(r|s)= L(r) U L(s)
29

» Consider L(r)={a},
» L(s)={b},
» L(t)= {€},
» L(u)= {}.
» What do the following R.E represent?
» (i) r|s
» (ii) r|t
» (iii) r|u
30

» rs
» It matches any string that is a concatenation of 2
strings, the first of which matches r and second
of which matches s.
» L(rs) = L(r) L(s)
31

» Consider L(r) ={a}, L(s)={b}, L(t) ={€}, L(u)={ },
L(v)={c}. What do following R.E represent?
» (i) rs
» (ii) rt
» (iii) ru
» (iv) (r|s)v
32

» Also called Kleene closure
» Represents any finite concatenation of strings
each matches strings from L(r).
» r*
» Let S={a}, then L(a*)={€, a, aa, aaa,…}
» S*={€}USUSSUSSSU….=
»
33

» Consider L(r)= {a},
» L(s)={b}.
» What do the following R.E represent?
» (i) r*
» (ii) (rs) *
» (iii) (r|s)*
» (iv) (r|ss)*
34

» Repetition --------------(highest)
» Concatenation left associative
» Choice-------------------(lowest)
35

» One or more instances: (r)+
» Zero of one instances: r?
» Zero or more instances: r*
36

37

38

» Regular Expression(RE): represents pattern of
string of characters.
» Language (L(r)): set of strings
» Alphabet(∑): set of symbols
» Meta character: is a special character (not a part
of ∑) used in R.E eg: *, | etc
» Basic R.E: R.E consisting of only one character
» Regular language: language defined by RE
39

Lexical analysis - Compiler Design

In this document

More Related Content

What's hot

Viewers also liked

Similar to Lexical analysis - Compiler Design

More from Muhammed Afsal Villan

Recently uploaded

Lexical analysis - Compiler Design