jflex tutorial

60 %
40 %
Information about jflex tutorial

Published on November 27, 2007

Author: Charlo

Source: authorstream.com

JFlex:  JFlex Basically, a lexer is a Finite-State “Transducer” plus bells and whistles Arbitrary Java code can be associated with actions state transitions Specify the “transducer” in a .flex file; JFlex compiles it into a .java file By default, JFlex gives you convenience methods to access results of state transitions A simple task:  A simple task Charniak’s statistical parser takes input sentences delimited by <s>...</s> Suppose we want to take a Reader over such input and get back a Tokenizer over the tokens, which returns Word objects, plus a special end-of-sentence character garbage garbage garbage <s>Stocks skyrocketed on news that investigation of Cheney ’s energy taskforce was dropped . </s>more garbage edu.stanford.nlp.process.AbstractTokenizer:  edu.stanford.nlp.process.AbstractTokenizer ... /** * Internally fetches the next token. * * @return the next token in the token * stream, or null if none exists. */ protected abstract Object getNext(); ... Lexical Rules:  Lexical Rules Basically you’re specifying a finite-state automaton* with actions associated with state transitions *though not strictly limited by FSA expressivity Schematic .flex file:  Schematic .flex file {user code} %% {options and declarations} %% {lexical rules} Lexical Rules (schematic):  Lexical Rules (schematic) <YYINITIAL> { {BeginSentence} { yybegin{SENTENCE}; return yylex(); } {WhiteSpace} { /* ignore */ return yylex();} . { /* ignore */ return yylex();} } <SENTENCE> { {EndSentence} { yybegin{YYINITIAL}; return SENTENCE_BOUNDARY; } {Token} { return new Word(yytext()); } {Space} { /* ignore */ return yylex(); } } Lexical Rules (detail):  Lexical Rules (detail) <YYINITIAL> { {BeginSentence} / .* { yybegin(SENTENCE); return yylex();} ... } <SENTENCE> { {EndSentence} / .* { yybegin(YYINITIAL); return SENTENCE_BOUNDARY;} {Token} { return new Word(yytext()); } ... } Options and declarations: States and Macros:  Options and declarations: States and Macros Macros can be used to define other macros Order of macro definition is irrelevant %state SENTENCE SentenceLetter = s BeginSentence = <{SentenceLetter}> EndSentence = <\/{SentenceLetter}> WhiteSpace = [ \t\r\n\f] Token = [^ \t\r\n\f]+ Other options and declarations:  Other options and declarations %class CharniakTokenizer %implements Tokenizer %extends AbstractTokenizer %unicode %type Object %eofval{ return null; %eofval} Options & declarations: class-internal code (1):  Options & declarations: class-internal code (1) %{ static final Word SENTENCE_BOUNDARY = new Word("SENTENCE_BOUNDARY"); public Object getNext() { try { Object o = yylex(); return o; } catch(IOException e) { return null; } } ... %} Options & declarations: class-internal code (2):  Options & declarations: class-internal code (2) %{ ... public static void main(String[] args) throws IOException { Reader r = new FileReader(args[0]); Tokenizer t = new CharniakTokenizer(r); while(t.hasNext()) { System.out.println(t.next()); } } %} User Code inserted directly into the file:  User Code inserted directly into the file package rog; import java.util.*; import java.io.*; import edu.stanford.nlp.ling.Word; import edu.stanford.nlp.process.*; /** A lexer for Charniak input sentences * @author Roger Levy */ Beyond FSA expressivity:  Beyond FSA expressivity %class ParenCounter %{ private int numParens = 0; %} ... %% ... <YYINITIAL> { \( { numParens++; return yytext(); } \) { if(numParens == 0) throw new RuntimeException( "error – too many close parens!"); else { numParens--; return yytext(); } } }

Add a comment

Related presentations

Related pages

JFlex - JFlex The Fast Scanner Generator for Java

This is the homepage of JFlex. JFlex is a lexer/scanner generator for Java, written in Java, with emphasis on Unicode support, speed and platform independence
Read more

Tutorial JFlex - YouTube

Jflex tutorial. Jflex tutorial. Skip navigation Upload. Sign in. Search. ... Tutorial JFLEX en Java Con NetBeans , Analizador Léxico - Duration ...
Read more

JFlex User's Manual

JFlex User’s Manual Version 1.6.1, March 16, 2015 ... JFlex is a lexical analyser generator for Java1 written in Java. It is also a rewrite of the very
Read more

JFlex oder JLex | tutorials.de - User helfen Usern

Hallo Zusammen, hat sich jemand von euch mit JFlex oder JLex schon auseinander gesetzt? Kennt einer vielleicht eine Internetseite auf Deutsch oder...
Read more

Tutorial JFLEX en Java Con NetBeans , Analizador Léxico ...

Tutorial JFLEX en Java Con NetBeans , Analizador Léxico Alonso Hernandez. ... Standard YouTube License; Show more Show less. Loading ...
Read more

I/O and Parsing Tutorial 22-02-13 - University of Birmingham

Structure of tutorial 1.Example program to access and write to an XML file 2.Example usage of JFlex
Read more

JFlex Tutorials

JFlex Tutorials
Read more

Flex tutorial - TutorialsPoint

Flex tutorial - Learn Adobe Flex programming in simple and easy steps starting with Flex Environment Setup, Web Application Creation, Deploy and Debug ...
Read more

FLEX Tutorial - University of California, Riverside

FLEX Tutorial Lan Gao [What is a Scanner?] [How to use FLEX?] What is a Scanner? The scanner performs lexical analysis of a certain program (in our case ...
Read more

Lexical Analysis With Flex, for Flex 2.5.37

Lexical Analysis With Flex, for Flex 2.5.37. Lexical Analysis With Flex, for Flex 2.5.37. ... The manual includes both tutorial and reference sections.
Read more