Recent Changes

Friday, February 2

  1. msg Matching empty lines verbatim message posted Matching empty lines verbatim Hi, I'm writing a lexer/parser for Visual TrueType Talk, a simple command language. Input looks li…
    Matching empty lines verbatim
    Hi,
    I'm writing a lexer/parser for Visual TrueType Talk, a simple command language. Input looks like:
    /* VTTTalk glyph 2, char 0x41 (A) */
    /* GUI generated July 2015 */
     
    /* Y direction */
    ResYAnchor(2,8)
    ResYAnchor(4,2)
    YIPAnchor(2,1,4)
    YInterpolate(1,16,9,13,4)
    YDelta(16,1/4@16,1/2@18)
    YDelta(9,1/4@16,5/8@18)
    YDelta(13,3/8@16,1@18)
    YShift(1,0)
    ResYLink(1,17,72)
    YShift(17,8)
    ResYAnchor(7,8)
     
    /* X direction */
    XInterpolate(18,3,2,4,16,9,5,7,8,6,19)
    Align(2,1,17,16)
    Align(7,0,8,9)
     
    Smooth()

    My grammar so far:
    comment = cStyleComment
     
    point = pyparsing_common.integer
    ppem = pyparsing_common.integer
    range_of_ppems = ppem + Suppress("..") + ppem
    # range_of_ppems.setParseAction(lambda token: range(token[0], token[1] + 1))
     
    fraction = pyparsing_common.signed_integer + Suppress(
        "/") + pyparsing_common.integer
    fraction.setParseAction(lambda token: fractions.Fraction(token[0], token[1]))
    distance = pyparsing_common.signed_integer ^ fraction
    distance_ppems = delimitedList(ppem ^ range_of_ppems, delim=";")
    distance_at_ppems = distance + Suppress("@") + Group(distance_ppems)
    # distance_at_ppems.setParseAction(
    #     lambda token: Distance(token[0], token[1].asList()))
     
    angleFlag = oneOf("/ Ø")("angleFlag")
    controlFlag = oneOf("<< >> <> >< ||")("controlFlag")
    minimumDistanceFlag = oneOf("< > <= >=") ^ (
        Literal(">=") + Group(distance
                              | Suppress("(") + distance + Suppress(",") + ppem +
                              Suppress(",") + distance + Suppress(")")))
    # minimumDistanceFlag.setParseAction(
    #     lambda token: MinimumDistanceFlag(token[0], token[1].asList() if token.get(1) else None)
    # )
     
    command_name = Word(alphas + "_")
    command_argument = point ^ distance ^ distance_at_ppems ^ minimumDistanceFlag
    command = command_name + Optional(angleFlag ^ controlFlag) + nestedExpr(
        content=delimitedList(command_argument))
    # command.setParseAction(construct_command)
     
    p = OneOrMore(comment | command)

    I want to preserve empty lines as they are for roundtrip testing. Matching "EOL = ~StringEnd() + LineEnd()" matches each "\n", but not empty lines as such. I want the final token list to look like "['/* VTTTalk glyph 2, char 0x41 (A) */', '/* GUI generated July 2015 */', '', '/* Y direction */', <my object>, ...]". How?
    3:08 am

Thursday, January 25

  1. 6:34 am
  2. msg associativity in operatorPrecedence message posted associativity in operatorPrecedence I am trying to understand how associativity of infix operators is handled in operatorPrecedence. Th…
    associativity in operatorPrecedence
    I am trying to understand how associativity of infix operators is handled in operatorPrecedence. The following code

    from pyparsing import *
     
    variable = Word(alphas+'_', alphanums+'_')
    parser1 = operatorPrecedence(variable, [('+', 2, opAssoc.RIGHT)])
    parser2 = operatorPrecedence(variable, [('+', 2, opAssoc.LEFT)])
     
    parsed1 = parser1.parseString('x + y + z', parseAll=True)
    print(parsed1)
    parsed2 = parser2.parseString('x + y + z', parseAll=True)
    print(parsed2)

    prints

    [['x', '+', ['y', '+', 'z']]]
    [['x', '+', 'y', '+', 'z']]

    I would have expected this:

    [['x', '+', ['y', '+', 'z']]]
    [[['x', '+', 'y'], '+', 'z']]

    I.e. parser2 seems to flatten the expression rather than making it left-associative. Is this a bug or a feature?
    6:24 am

Wednesday, January 3

  1. msg parseAction - action is not called message posted parseAction - action is not called from pruga.elm import ast   from pyparsing import (alphanums, a…
    parseAction - action is not called
    from pruga.elm import ast
     
    from pyparsing import (alphanums,
                            alphas,
                            ParserElement,
                            Keyword,
                            Suppress,
                            Word,
                            delimitedList)
     
    ParserElement.enablePackrat()
     
    class Node:
     
        def __init__(self, s, loc, tokens):
            print('init NODE WITH  TOKENS', tokens, f'"{s}"')
     
     
    IMPORT = Suppress(Keyword('import'))
     
    ModuleName = Word(alphas.upper(), alphanums + '_')('name')
     
    QualifiedModuleName = delimitedList(ModuleName, delim='.')
     
    # QualifiedModuleName has not defined any parseActons
    ImportStatement = IMPORT + QualifiedModuleName('module') 
     
    # I define parseAction now, but this action will not be used
    QualifiedModuleName.setParseAction(Node)
     
    source = 'import X.Y.Z'
    ast = ImportStatement.parseString(source)
    print('ImportStatement defined before QualifiedModuleName.setParseAction(Node)')
    print(ast)
     
    # identically definition ImportStatement, but is defined after QualifiedModuleName.setParseAction(Node)
    ImportStatement = IMPORT + QualifiedModuleName('module') 
     
    ast = ImportStatement.parseString(source)
    print('The same ImportStatement defined after QualifiedModuleName.setParseAction(Node)')
     
    print(ast)
     

    output

    ImportStatement defined before QualifiedModuleName.setParseAction(Node)
    ['X', 'Y', 'Z']
    init NODE WITH TOKENS ['X', 'Y', 'Z'] "import X.Y.Z"
    The same ImportStatement defined after QualifiedModuleName.setParseAction(Node)
    [<main.Node object at 0x7f136483c080>]
    1:54 am

Friday, November 17

  1. 8:42 am

Sunday, October 1

  1. 3:00 pm

Sunday, September 3

  1. msg += operator and user function for math equation message posted += operator and user function for math equation ah, i did not notice the tokens become a list if i use Group function. After removing the Group and…
    += operator and user function for math equation
    ah, i did not notice the tokens become a list if i use Group function. After removing the Group and use code like this, i got
    def assignVar(t):
        global vars_
        vars_=defaultdict(float)
        print('t0 is {}'.format(t[0]))
        vars_[t[0]]=t[-1]
    The var on lhs is updated with rhs value. If there is any better way, please let me know.
    Another question is on the funccall. I would need to map many user function to python function. In the excelExpr.py example the funcCall is actually an aggregation using MatchFirst of various functions. This seems a bit tedious as we need to a new line each time we have a new function and then append the new function name in the funcCall line. Is there easier way such as just using a dictionary?
    Sorry for all these many questions but i am glad to be able to make some progress little by little.
    6:12 am
  2. msg += operator and user function for math equation message posted += operator and user function for math equation Thanks for the clarification. Would you mind to give a simple example on how to assign the token va…
    += operator and user function for math equation
    Thanks for the clarification. Would you mind to give a simple example on how to assign the token value on rhs to variable on lhs? I tried something like this:
    assignmentExpr = Group(varname('lhs') + EQ('operator') + expr('rhs'))('assignment').addParseAction(lambda t: t[0] = t[-1])
    but i got an error
    SyntaxError: lambda cannot contain assignment

    Also, what is the difference between addParseAction and setParseAction?
    4:59 am

Saturday, September 2

  1. msg += operator and user function for math equation message posted += operator and user function for math equation The parsed data returned by pyparsing implements all the methods of a list so that it can be treate…
    += operator and user function for math equation
    The parsed data returned by pyparsing implements all the methods of a list so that it can be treated as a list (len, for i in result, result[0], etc.). If you *must* have a list (for JSON serialization, for example), you can use result.asList() and the parsed data will be converted to a nested list. Note that this will strip any defined results names, like" fname" or "argslist".

    augAssignOp won't really fit into into the infixNotation model because '+=' isn't really an operator. You can't write "x * (a += 14)". The '+=' can only occur as part of an enhanced assignment statement. So you should really implement '+=' as another form of assignment:

    EQ = Literal('=')
    PLUS_EQ = Literal('+=')
    assignmentExpr = varname('lhs') + EQ('operator') + arithExpr('rhs')
    augmentedAssignmentExpr = varname('lhs') + PLUS_EQ('operator') + arithExpr('rhs')

    and then have your overall parser match for "assignment | augmentedAssignment".

    statement_parser = assignmentExpr | augmentedAssignemtExpr

    You could attach a parse action to statement_parser that would look at the variable name in "lhs", the value in "rhs", and the assignment or augmented assignment based on whether "operator" was "=" or "+=".
    8:47 am
  2. msg += operator and user function for math equation message posted += operator and user function for math equation For my work, there is no exponential operation, and this makes things easier. But on the other hand…
    += operator and user function for math equation
    For my work, there is no exponential operation, and this makes things easier. But on the other hand, the user defined function is exactly like a python function and can take argument such as list or keyword argument (name = value). i plan to directly write those functions in python, so i just need to call those functions with parsed argument list.
    I have read more examples such as arith.py and excelExpr.py and discussion in this forum. And it seems infixNotation helps to simplify the definition of operators. But i am puzzled whether i should treat the brackets as an operator and if so how to do that.
    I have tried a bit with new style of code like below by making list a kind of parseelement instead of an operator
    from pyparsing import *
     
    compOp = oneOf("< > >= <=")
    multOp = oneOf("* /")
    addOp = oneOf("+ -")
    augassignOp = Literal('+=')
     
    EQ,LPAR,RPAR,LBRA,RBRA = map(Suppress, '=()[]')
     
    varname = Regex(r'[a-zA-Z_][a-zA-Z0-9_]*')
    fname = Regex(r'[a-zA-Z][a-zA-Z0-9_]*')
    decimalnumber = Regex(r'[+-]?\d+(\.\d*)?')
     
    expr = Forward()
    listmaker = Group(LBRA + delimitedList(expr) + RBRA)('list')
    funccall = Group(fname('fname') + LPAR + Group(delimitedList(expr))("arglist") + RPAR)
    operand = decimalnumber | listmaker | funccall | varname
    expr <<= infixNotation(operand,
        [
        (multOp, 2, opAssoc.LEFT),
        (addOp, 2, opAssoc.LEFT),
        (compOp, 2, opAssoc.LEFT),
    #    (augassignOp, 2, opAssoc.LEFT),
        ])
     
    l = ['cond(1,select(4,[2, 3]))']
     
    for item in l:
        source = item
        expr.runTests(source)

    The parsing seems to work as I got below result:
    cond(1,select(4,[2, 3]))
    [['cond', ['1', ['select', ['4', ['2', '3']]]]]]
    [0]:
      ['cond', ['1', ['select', ['4', ['2', '3']]]]]
      - arglist: ['1', ['select', ['4', ['2', '3']]]]
        [0]:
          1
        [1]:
          ['select', ['4', ['2', '3']]]
          - arglist: ['4', ['2', '3']]
            - list: ['2', '3']
          - fname: 'select'
      - fname: 'cond'

    But i am puzzled how to pass the argument back to the function as a list. Also i have not figured out how to handle the keyword argument.

    Also, i am still stuck at the augassignOp because i cannot use a lambda expression like l"ambda a, b: a = a + b" as shown in the airth.py example.

    Would appreciate any feedback. Thanks.
    6:28 am

More