Until recently, Slang’s parser really sucked. It was a quick hack implemented with Scala’s parser combinator library. Nothing really wrong about that in particular, but there was a gaping hole in the grammar: no operator precedence. So to get an expression like
a + b * c to mean
a + (b * c) I had to add the parentheses myself. In fact, there were even more problems — some things that should have been left-associative were right-associative. This resulted in very hairy test code, with lots of parentheses everywhere.
Although I think parsers are cool, I am actually not very good at writing one for a complex grammar. I feel that I just know too little about the theory behind them or how to put it to practical use. I’ve used parser combinators before and think they are probably the easiest way for newbies like me to implement parsers, so that’s what I used. The use of symbolic names in the library might be scary the first time, but actually I think parsing is one of the few contexts where use of lots of symbols and extremeley concise code is desirable. It allows one to put a lot of code on a few lines, and when you are looking at or writing a parser, you want to see many productions of the grammar at the same time to understand what is going on. At least I do.
For Slang, I implemented something minimal that could parse the language. I had no idea how to solve operator precedence well with parser combinators, and I didn’t want to spend a lot of time studying parsers, because the next compiler phases seemed more interesting at first. But getting the parser right is important for actually using the language because it’s the first thing that processes the code and reports errors. A parser that only kind of works can be very annoying.
Thankfully Miles Sabin suggested that I should look into mixfix operator parsers, and I did. I don’t know exactly where the word mixfix comes from, so I’m assuming it means mixed fixity — operators can be prefix, infix, postfix or closed. Here are some samples:
- prefix :
- infix :
a + b
- closed :
Of course, most languages have operators with all of these fixities. The term mixfix actually refers to something more flexible than that — a mixfix operator can be seen as a sequence of alternating name parts and “holes in the expression”. A hole is where the operator’s arguments go.
_ + _has two holes and one name part
+(and is infix)
if _ then _ else _has three name parts
elseand three holes (and is prefix)
In the mixfix viewpoint, many syntactic constructs might be seen as operators that can have precedence in relation to others, and this concept of many name parts can make it easier to let users define their own operators in a more flexible way than just a single prefix or infix word (as is allowed by Scala). I think this would be a really nice way of creating internal DSL-s. In Slang, like in Scala, most operators are really methods. Slang doesn’t allow user-defined fixity or precedence for methods yet (or even multiple argument lists), but I may add this feature one day.
There are existing languages that support mixfix operators, such as Agda, Maude and BitC. To my knowledge, all these languages assign numeric precedence values to operators, and no language currently uses the exact scheme we will look at, although it was proposed for Agda.
Mixfix operators can be implemented in many ways, but one of the first things I found was the paper Parsing Mixfix Operators by Anders Danielsson and Ulf Norell that was a great help to me. I was able to implement the grammar scheme described in that paper on top of Scala’s parser combinators and patch that into Slang’s existing parser with minimal changes to existing productions. The characteristics of the grammar scheme described in Danielsson’s paper seemed like a good enough fit for what I wanted for Slang:
- operator name parts and holes alternate — there can’t be two subsequent name parts or two subsequent holes
if _ then _ else _is ok,
if _ _ else _is not
- operator precedence is described as a directed acyclic graph (DAG), not as a total or partial ordering. You only have to describe the precedence relations where they make sense (more about this in the next post)
a directed edge
'+' -> '*'means “
*binds tighter than
- operator precedence is not transitive
'=' -> '+'and
'&' -> '='does not mean “
+binds tighter than
- prefix operators are treated as right-associative
!!a = !(!(a))
- postfix operators are treated as left-associative
n!! = ((n)!)!
- left-associative and right-associative operators of the same precedence can’t appear next to each other
+:is a right-associative
a + b +: cwould not be allowed
- parses are precedence correct
- implementation using left-recursion is possible, for example when using Scala’s Packrat parsers
There weren’t any restrictions I couldn’t live with (in fact, we could relax some of the above requirements and the scheme would still work for some grammars), so I decided to implement this grammar scheme for Slang, pretty much as described in the paper. Although I didn’t really grok all of the Agda code samples, the principles were easily understandable. I implemented it as a separate library (available on GitHub) that builds on top of the existing Scala parser combinator library. It might even be somewhat usable in it’s current state, but needs improvement.
In the next post we’ll look at how to define a grammar for an arithmetic and boolean algebra language using mixfix operators. In the third part, we will actually implement the parser for the grammar, look at performance issues and whether we can solve them with packrat parsers.