ó
|
L\¬Qc @ s d Z d d l Z d d l m Z d d l m Z m Z m Z m Z m Z m
|
Z
|
m Z m Z m Z m Z m Z m Z m Z m Z d d l m Z y d d l m Z Wn! e k
|
rÇ d d l m Z n Xd d d d
|
d d d d d d d d d d d d d d d d g Z d d" d YZ d e f d YZ e d e d e f d YZ e d e d d# d YZ d d$ d YZ y e Wn e k
|
rªe Z n Xd d% d! YZ d S(& s+
|
csv.py - read/write/investigate CSV files
|
iÿÿÿÿN( t reduce( t Errort __version__t writert readert register_dialectt unregister_dialectt get_dialectt list_dialectst field_size_limitt QUOTE_MINIMALt QUOTE_ALLt QUOTE_NONNUMERICt
|
QUOTE_NONEt __doc__( t Dialect( t StringIOR
|
R R R R R R t excelt excel_tabR R R R R R t SnifferR R t
|
DictReadert
|
DictWriterc B sV e Z d Z d Z e Z d Z d Z d Z d Z
|
d Z d Z d Z d Z d Z RS( sÇ Describe an Excel dialect.
|
|
This must be subclassed (see csv.excel). Valid attributes are:
|
delimiter, quotechar, escapechar, doublequote, skipinitialspace,
|
lineterminator, quoting.
|
|
t c C s) | j t k r t | _ n | j d S( N( t __class__R t Truet _validt _validate( t self( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyt __init__- s c C s: y t | Wn% t k
|
r5 } t t | n Xd S( N( t _Dialectt TypeErrorR t str( R t e( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR 2 s N( t __name__t
|
__module__R t _namet FalseR t Nonet delimitert quotechart
|
escapechart doublequotet skipinitialspacet lineterminatort quotingR R ( ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR s c B s2 e Z d Z d Z d Z e Z e Z d Z e
|
Z RS( s; Describe the usual properties of Excel-generated CSV files.t ,t "s
|
( R! R" R R& R' R R) R$ R* R+ R
|
R, ( ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR 9 s c B s e Z d Z d Z RS( sE Describe the usual properties of Excel-generated TAB-delimited files.s ( R! R" R R& ( ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR C s s excel-tabc B sP e Z d d d d d Z d Z e d Z e j d Z d Z RS( R c O sI | | _ | | _ | | _ t | | | | | _ | | _ d | _ d S( Ni ( t _fieldnamest restkeyt restvalR t dialectt line_num( R t ft
|
fieldnamesR0 R1 R2 t argst kwds( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR J s c C s | S( N( ( R ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyt __iter__S s c C sR | j d k r< y | j j | _ Wq< t k
|
r8 q< Xn | j j | _ | j S( N( R/ R% R t nextt StopIterationR3 ( R ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR5 V s c C s | | _ d S( N( R/ ( R t value( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR5 ` s c C sä | j d k r | j n | j j } | j j | _ x | g k rX | j j } q: Wt t | j | } t | j } t | } | | k r¬ | | | | j <n4 | | k rà x% | j | D] } | j | | <qÆ Wn | S( Ni ( R3 R5 R R9 t dictt zipt lenR0 R1 ( R t rowt dt lft lrt key( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR9 d s
|
N( R! R" R% R R8 t propertyR5 t setterR9 ( ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR I s
|
c B s> e Z d d d d Z d Z d Z d Z d Z RS( R t raiseR c O sY | | _ | | _ | j d k r4 t d | n | | _ t | | | | | _ d S( NRF t ignores- extrasaction (%s) must be 'raise' or 'ignore'( RF s ignore( R5 R1 t lowert
|
ValueErrort extrasactionR ( R R4 R5 R1 RJ R2 R6 R7 ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR | s
|
c C s, t t | j | j } | j | d S( N( R< R= R5 t writerow( R t header( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyt writeheader s c C s
| j d k r\ g | D] } | | j k r | ^ q } | r\ t d d j | q\ n g | j D] } | j | | j ^ qf S( NRF s( dict contains fields not in fieldnames: s , ( RJ R5 RI t joint getR1 ( R t rowdictt kt wrong_fieldsRC ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyt _dict_to_list s (c C s | j j | j | S( N( R RK RS ( R RP ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyRK s c C s= g } x$ | D] } | j | j | q W| j j | S( N( t appendRS R t writerows( R t rowdictst rowsRP ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyRU s ( R! R" R RM RS RK RU ( ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR { s
|
c B s> e Z d Z d Z d d Z d Z d Z d Z RS( se
|
"Sniffs" the format of a CSV file (i.e. delimiter, quotechar)
|
Returns a Dialect object.
|
c C s d d d d d g | _ d S( NR- s t ;t t :( t preferred( R ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR § s c C s | j | | \ } } } } | s? | j | | \ } } n | sQ t d n d t f d Y} | | _ | | _ | p d | _ | | _ | S( sI
|
Returns a dialect (or None) corresponding to the sample
|
s Could not determine delimiterR2 c B s e Z d Z d Z e Z RS( t sniffeds
|
( R! R" R# R+ R
|
R, ( ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR2 º s R. ( t _guess_quote_and_delimitert _guess_delimiterR R R) R& R' R* ( R t samplet
|
delimitersR' R) R& R* R2 ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyt sniff¬ s c C s< g } xC d D]; } t j | t j t j B } | j | } | r Pq q W| sb d t d d f Si } i } d } x | D]ø } | j d d }
|
| |
|
} | r¿ | j | d d | | <n y | j d d }
|
| |
|
} Wn t k
|
rô q{ n X| r0| d k s| | k r0| j | d d | | <n y | j d
|
d }
|
Wn t k
|
r[q{ n X| |
|
r{ | d 7} q{ q{ Wt
|
| d | j } | rÛt
|
| d | j } | | | k } | d k rçd } qçn d } d } t j d i | d 6| d 6t j } | j | r&t } n t } | | | | f S( s
|
Looks for text enclosed between two identical quotes
|
(the probable quotechar) which are preceded and followed
|
by the same character (the probable delimiter).
|
For example:
|
,'some text',
|
The quote with the most wins, same with the delimiter.
|
If there is no quotechar the delimiter can't be determined
|
this way.
|
sF (?P<delim>[^\w
|
"'])(?P<space> ?)(?P<quote>["']).*?(?P=quote)(?P=delim)sC (?:^|
|
)(?P<quote>["']).*?(?P=quote)(?P<delim>[^\w
|
"'])(?P<space> ?)sD (?P<delim>>[^\w
|
"'])(?P<space> ?)(?P<quote>["']).*?(?P=quote)(?:$|
|
)s* (?:^|
|
)(?P<quote>["']).*?(?P=quote)(?:$|
|
)R i t quotei t delimt spacec S s | | | | k r | p | S( N( ( t at bt quotes( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyt <lambda>ø s c S s | | | | k r | p | S( N( ( Re Rf t delims( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyRh ü s s
|
s] ((%(delim)s)|^)\W*%(quote)s[^%(delim)s\n]*%(quote)s[^%(delim)s\n]*%(quote)s\W*((%(delim)s)|$)( sF (?P<delim>[^\w
|
"'])(?P<space> ?)(?P<quote>["']).*?(?P=quote)(?P=delim)sC (?:^|
|
)(?P<quote>["']).*?(?P=quote)(?P<delim>[^\w
|
"'])(?P<space> ?)sD (?P<delim>>[^\w
|
"'])(?P<space> ?)(?P<quote>["']).*?(?P=quote)(?:$|
|
)s* (?:^|
|
)(?P<quote>["']).*?(?P=quote)(?:$|
|
)N( t ret compilet DOTALLt MULTILINEt findallR$ R% t
|
groupindexRO t KeyErrorR t keyst searchR ( R t dataR` t matchest restrt regexpRg Ri t spacest mt nRC R' Rc R* t dq_regexpR) ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR] É s`
|
|
|
c C s t d | j d } g t d D] } t | ^ q% } t d t | } d } i } i } i } d t | t | }
|
} x|
|
t | k r| d 7} xk | |
|
| !D]\ } xS | D]K } | j | i } | j | } | j | d d | | <| | | <qÀ Wq³ WxÎ | j D]À } | | j
|
} t | d k rb| d d d k rbq n t | d k rÒt d | | | <| j | | | | d | | d t d | d f | | <q | d | | <q W| j
|
} t | | } d } d } x t | d k r| | k rxp | D]h \ } } | d d k r4| d d k r4| d | | k r| d k s| | k r| | | <qq4q4W| d
|
8} qWt | d k r| j d } | d j | | d j d | k } | | f S| }
|
| | 7} q W| s"d St | d k rxZ | j D]L } | | j k r>| d j | | d j d | k } | | f Sq>Wn g | j
|
D] \ } } | | f ^ q} | j | d d } | d j | | d j d | k } | | f S( s½
|
The delimiter /should/ occur the same number of times on
|
each row. However, due to malformed data, it may not. We don't want
|
an all or nothing approach, so we allow for small variations in this
|
number.
|
1) build a table of the frequency of each character on every line.
|
2) build a table of frequencies of this frequency (meta-frequency?),
|
e.g. 'x occurred 5 times in 10 rows, 6 times in 1000 rows,
|
7 times in 2 rows'
|
3) use the mode of the meta-frequency to determine the /expected/
|
frequency for that character
|
4) find out how often the character actually meets that goal
|
5) the character that best meets its goal is the delimiter
|
For performance reasons, the data is evaluated in chunks, so it can
|
try and evaluate the smallest portion of the data possible, evaluating
|
additional chunks as necessary.
|
s
|
i i
|
i i c S s | d | d k r | p | S( Ni ( ( Re Rf ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyRh D s c S s d | d | d f S( Ni i ( ( Re Rf ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyRh J s g ð?gÍÌÌÌÌÌì?g{®Gáz?s %c R iÿÿÿÿN( R i ( t filterR% t splitt ranget chrt minR> RO t countRq t itemsR t removet floatR[ t sort( R Rs R` t ct asciit chunkLengtht iterationt charFrequencyt modesRi t startt endt linet chart metaFrequencyt freqR t modeListt totalt consistencyt thresholdRQ t vRc R* R@ ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR^ sx %
|
& !
|
+
|
c C s t t | | j | } | j } t | } i } x t | D] } d | | <qI Wd } xý | D]õ } | d k r Pn | d 7} t | | k r¢ qj n xº | j D]¬ } xW t t t
|
t g D]3 }
|
y |
|
| | PWqÈ t t f k
|
rú qÈ XqÈ Wt | | }
|
|
|
t k r$t }
|
n |
|
| | k r¯ | | d k rQ|
|
| | <q[| | =q¯ q¯ Wqj Wd } x¢ | j D] \ } } t | t d k rÊt | | | k r½| d 7} q
|
| d 8} qvy | | | Wn! t t f k
|
rÿ| d 7} qvX| d 8} qvW| d k S( Ni i i ( R R Ra R9 R> R} R% Rq t intt longR t complexRI t OverflowErrorR t typeR ( R R_ t rdrRL t columnst columnTypest it checkedR? t colt thisTypet hasHeadert colType( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyt
|
has_header~ sN
|
|
N( R! R" R R R% Ra R] R^ R¤ ( ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyR ¢ s L i( ( ( ( ( R Rj t functoolsR t _csvR R R R R R R R R R
|
R R R R R t cStringIOR t ImportErrort __all__R R R R R t NameErrorR R ( ( ( sK /tmp/ndk-User/buildhost/install/prebuilt/darwin-x86_64/lib/python2.7/csv.pyt <module> s2 ^ 2"
|
|
