ó
|
Y\¬Qc @ sE d Z d d l Z d d l m Z m Z d e j f d YZ d S( sý Convert graminit.[ch] spit out by pgen to Python code.
|
|
Pgen is the Python parser generator. It is useful to quickly create a
|
parser from a grammar file in Python's grammar notation. But I don't
|
want my parsers to be written in C (yet), so I'm translating the
|
parsing tables to Python data structures and writing a Python parse
|
engine.
|
|
Note that the token numbers are constants determined by the standard
|
Python tokenizer. The standard token module defines these numbers and
|
their names (the names are not used much). The token numbers are
|
hardcoded into the Python tokenizer and into pgen. A Python
|
implementation of the Python tokenizer is also available, in the
|
standard tokenize module.
|
|
On the other hand, symbol numbers (representing the grammar's
|
non-terminals) are assigned by pgen based on the actual grammar
|
input.
|
|
Note: this module is pretty much obsolete; the pgen module generates
|
equivalent grammar tables directly from the Grammar.txt input file
|
without having to invoke the Python pgen C program.
|
|
iÿÿÿÿN( t grammart tokent Converterc B s2 e Z d Z d Z d Z d Z d Z RS( s2 Grammar subclass that reads classic pgen output files.
|
|
The run() method reads the tables as produced by the pgen parser
|
generator, typically contained in two C files, graminit.h and
|
graminit.c. The other methods are for internal use only.
|
|
See the base class for more documentation.
|
|
c C s( | j | | j | | j d S( s< Load the grammar tables from the text files written by pgen.N( t parse_graminit_ht parse_graminit_ct
|
finish_off( t selft
|
graminit_ht
|
graminit_c( ( sZ /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/lib2to3/pgen2/conv.pyt run/ s c C s y t | } Wn# t k
|
r5 } d | | f GHt SXi | _ i | _ d } xº | D]² } | d 7} t j d | } | r¥ | j r¥ d | | | j f GHqU | j \ } } t | } | | j k sØ t
|
| | j k sí t
|
| | j | <| | j | <qU Wt S( sð Parse the .h file written by pgen. (Internal)
|
|
This file is a sequence of #define statements defining the
|
nonterminals of the grammar as numbers. We build two tables
|
mapping the numbers to names and back.
|
|
s Can't open %s: %si i s ^#define\s+(\w+)\s+(\d+)$s %s(%s): can't parse %s( t opent IOErrort Falset symbol2numbert number2symbolt ret matcht stript groupst intt AssertionErrort True( R t filenamet ft errt linenot linet mot symbolt number( ( sZ /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/lib2to3/pgen2/conv.pyR 5 s*
|
c ! C sÅ y t | } Wn# t k
|
r5 } d | | f GHt SXd } | d | j } } | d k sq t | | f | d | j } } | d k s¦ t | | f | d | j } } i } g } xØ| j d r£xC| j d r t j d | } | st | | f t t | j
|
\ } }
|
} g } x t | D]u } | d | j } } t j d | } | st | | f t t | j
|
\ } } | j | | f qHW| d | j } } | d k söt | | f | | | |
|
f <| d | j } } qÞ Wt j d
|
| } | sKt | | f t t | j
|
\ } } | t | k st | | f g } x´ t | D]¦ } | d | j } } t j d | } | sät | | f t t | j
|
\ } } }
|
| | |
|
f } | t | k s6t | | f | j | qW| j | | d | j } } | d k st | | f | d | j } } qÌ W| | _ i } t j d | } | sÝt | | f t | j d } xöt | D]è} | d | j } } t j d | } | sFt | | f | j d } t t | j d d d d \ } } } } | j | | k s§t | | f | j | | k sÌt | | f | d k sêt | | f | | } | t | k st | | f | d | j } } t j d | } | sYt | | f i } t | j d } x` t | D]R \ } } t | } x7 t d D]) } | d | >@r¦d | | d | <q¦q¦WqW| | f | | <qÿW| d | j } } | d k s t | | f | | _ g } | d | j } } t j d | } | spt | | f t | j d } x§ t | D] } | d | j } } t j d | } | sÙt | | f | j
|
\ } } t | } | d k r d } n t | } | j | | f qW| d | j } } | d k sdt | | f | | _ | d | j } } | d k s¢t | | f | d | j } } t j d | } | sãt | | f t | j d } | t | j k st | d | j } } | d k sHt | | f | d | j } } t j d | } | st | | f t | j d } | t | j k sÅt | | f | d | j } } t j d | } | s t | | f t | j d } | | j k s< t | | f | | _ | d | j } } | d k sz t | | f y | d | j } } Wn t k
|
r¨ n Xd sÁ t | | f d S( sù Parse the .c file written by pgen. (Internal)
|
|
The file looks as follows. The first two lines are always this:
|
|
#include "pgenheaders.h"
|
#include "grammar.h"
|
|
After that come four blocks:
|
|
1) one or more state definitions
|
2) a table defining dfas
|
3) a table defining labels
|
4) a struct defining the grammar
|
|
A state definition has the following form:
|
- one or more arc arrays, each of the form:
|
static arc arcs_<n>_<m>[<k>] = {
|
{<i>, <j>},
|
...
|
};
|
- followed by a state array, of the form:
|
static state states_<s>[<t>] = {
|
{<k>, arcs_<n>_<m>},
|
...
|
};
|
|
s Can't open %s: %si i s #include "pgenheaders.h"
|
s #include "grammar.h"
|
s static arc s) static arc arcs_(\d+)_(\d+)\[(\d+)\] = {$s \s+{(\d+), (\d+)},$s };
|
s' static state states_(\d+)\[(\d+)\] = {$s \s+{(\d+), arcs_(\d+)_(\d+)},$s static dfa dfas\[(\d+)\] = {$s0 \s+{(\d+), "(\w+)", (\d+), (\d+), states_(\d+),$i i i i s \s+("(?:\\\d\d\d)*")},$i s! static label labels\[(\d+)\] = {$s \s+{(\d+), (0|"\w+")},$t 0s grammar _PyParser_Grammar = {
|
s
|
\s+(\d+),$s dfas,
|
s \s+{(\d+), labels},$s \s+(\d+)$N( R
|
R R t nextR t
|
startswithR R t mapR R t ranget appendt lent statest groupR R t evalt enumeratet ordt dfast Nonet labelst startt StopIteration(! R R R R R R t allarcsR% R t nt mt kt arcst _t it jt st tt stateR* t ndfasR R t xt yt zt firstt rawbitsett ct byteR, t nlabelsR- ( ( sZ /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/lib2to3/pgen2/conv.pyR T sö $ $ -%%
|
$ '! c C s i | _ i | _ xj t | j D]Y \ } \ } } | t j k r_ | d k r_ | | j | <q" | d k r" | | j | <q" q" Wd S( s1 Create additional useful structures. (Internal).N( t keywordst tokensR( R, R t NAMER+ ( R t ilabelt typet value( ( sZ /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/lib2to3/pgen2/conv.pyR ù s " ( t __name__t
|
__module__t __doc__R R R R ( ( ( sZ /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/lib2to3/pgen2/conv.pyR $ s
|
¥( RK R t pgen2R R t GrammarR ( ( ( sZ /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/lib2to3/pgen2/conv.pyt <module> s
|
