2013-08-12-capnproto-0.2-no-more-haskell.md (8253B)
1 --- 2 layout: post 3 title: "Cap'n Proto v0.2: Compiler rewritten Haskell -> C++" 4 author: kentonv 5 --- 6 7 Today I am releasing version 0.2 of Cap'n Proto. The most notable change: the compiler / code 8 generator, which was previously written in Haskell, has been rewritten in C++11. There are a few 9 other changes as well, but before I talk about those, let me try to calm the angry mob that is 10 not doubt reaching for their pitchforks as we speak. There are a few reasons for this change, 11 some practical, some ideological. I'll start with the practical. 12 13 **The practical: Supporting dynamic languages** 14 15 Say you are trying to implement Cap'n Proto in an interpreted language like Python. One of the big 16 draws of such a language is that you can edit your code and then run it without an intervening 17 compile step, allowing you to iterate faster. But if the Python Cap'n Proto implementation worked 18 like the C++ one (or like Protobufs), you lose some of that: whenever you change your Cap'n Proto 19 schema files, you must run a command to regenerate the Python code from them. That sucks. 20 21 What you really want to do is parse the schemas at start-up -- the same time that the Python code 22 itself is parsed. But writing a proper schema parser is harder than it looks; you really should 23 reuse the existing implementation. If it is written in Haskell, that's going to be problematic. 24 You either need to invoke the schema parser as a sub-process or you need to call Haskell code from 25 Python via an FFI. Either approach is going to be a huge hack with lots of problems, not the least 26 of which is having a runtime dependency on an entire platform that your end users may not otherwise 27 want. 28 29 But with the schema parser written in C++, things become much simpler. Python code calls into 30 C/C++ all the time. Everyone already has the necessary libraries installed. There's no need to 31 generate code, even; the parsed schema can be fed into the Cap'n Proto C++ runtime's dynamic API, 32 and Python bindings can trivially be implemented on top of that in just a few hundred lines of 33 code. Everyone wins. 34 35 **The ideological: I'm an object-oriented programmer** 36 37 I really wanted to like Haskell. I used to be a strong proponent of functional programming, and 38 I actually once wrote a complete web server and CMS in a purely-functional toy language of my own 39 creation. I love strong static typing, and I find a lot of the constructs in Haskell really 40 powerful and beautiful. Even monads. _Especially_ monads. 41 42 But when it comes down to it, I am an object-oriented programmer, and Haskell is not an 43 object-oriented language. Yes, you can do object-oriented style if you want to, just like you 44 can do objects in C. But it's just too painful. I want to write `object.methodName`, not 45 `ModuleName.objectTypeMethodName object`. I want to be able to write lots of small classes that 46 encapsulate complex functionality in simple interfaces -- _without_ having to place each one in 47 a whole separate module and ending up with thousands of source files. I want to be able to build 48 a list of objects of varying types that implement the same interface without having to re-invent 49 virtual tables every time I do it (type classes don't quite solve the problem). 50 51 And as it turns out, even aside from the lack of object-orientation, I don't actually like 52 functional programming as much as I thought. Yes, writing my parser was super-easy (my first 53 commit message was 54 "[Day 1: Learn Haskell, write a parser](https://github.com/kentonv/capnproto/commit/6bb49ca775501a9b2c7306992fd0de53c5ee4e95)"). 55 But everything beyond that seemed to require increasing amounts of brain bending. For instance, to 56 actually encode a Cap'n Proto message, I couldn't just allocate a buffer of zeros and then go 57 through each field and set its value. Instead, I had to compute all the field values first, sort 58 them by position, then concatenate the results. 59 60 Of course, I'm sure it's the case that if I spent years writing Haskell code, I'd eventually become 61 as proficient with it as I am with C++. Perhaps I could un-learn object-oriented style and learn 62 something else that works just as well or better. Basically, though, I decided that this was 63 going to take a lot longer than it at first appeared, and that this wasn't a good use of my 64 limited resources. So, I'm cutting my losses. 65 66 I still think Haskell is a very interesting language, and if works for you, by all means, use it. 67 I would love to see someone write at actual Cap'n Proto runtime implementation in Haskell. But 68 the compiler is now C++. 69 70 **Parser Combinators in C++** 71 72 A side effect (so to speak) of the compiler rewrite is that Cap'n Proto's companion utility 73 library, KJ, now includes a parser combinator framework based on C++11 templates and lambdas. 74 Here's a sample: 75 76 {% highlight c++ %} 77 // Construct a parser that parses a number. 78 auto number = transform( 79 sequence( 80 oneOrMore(charRange('0', '9')), 81 optional(sequence( 82 exactChar<'.'>(), 83 many(charRange('0', '9'))))), 84 [](Array<char> whole, Maybe<Array<char>> maybeFraction) 85 -> Number* { 86 KJ_IF_MAYBE(fraction, maybeFraction) { 87 return new RealNumber(whole, *fraction); 88 } else { 89 return new WholeNumber(whole); 90 } 91 }); 92 {% endhighlight %} 93 94 An interesting fact about the above code is that constructing the parser itself does not allocate 95 anything on the heap. The variable `number` in this case ends up being one 96-byte flat object, 96 most of which is composed of tables for character matching. The whole thing could even be 97 declared `constexpr`... if the C++ standard allowed empty-capture lambdas to be `constexpr`, which 98 unfortunately it doesn't (yet). 99 100 Unfortunately, KJ is largely undocumented at the moment, since people who just want to use 101 Cap'n Proto generally don't need to know about it. 102 103 **Other New Features** 104 105 There are a couple other notable changes in this release, aside from the compiler: 106 107 * Cygwin has been added as a supported platform, meaning you can now use Cap'n Proto on Windows. 108 I am considering supporting MinGW as well. Unfortunately, MSVC is unlikely to be supported any 109 time soon as its C++11 support is 110 [woefully lacking](http://blogs.msdn.com/b/somasegar/archive/2013/06/28/cpp-conformance-roadmap.aspx). 111 112 * The new compiler binary -- now called `capnp` rather than `capnpc` -- is more of a multi-tool. 113 It includes the ability to decode binary messages to text as a debugging aid. Type 114 `capnp help decode` for more information. 115 116 * The new [Orphan]({{ site.baseurl }}/cxx.html#orphans) class lets you detach objects from a 117 message tree and re-attach them elsewhere. 118 119 * Various contributors have declared their intentions to implement 120 [Ruby](https://github.com/cstrahan/capnp-ruby), 121 [Rust](https://github.com/dwrensha/capnproto-rust), C#, Java, Erlang, and Delphi bindings. These 122 are still works in progress, but exciting nonetheless! 123 124 **Backwards-compatibility Note** 125 126 Cap'n Proto v0.2 contains an obscure wire format incompatibility with v0.1. If you are using 127 unions containing multiple primitive-type fields of varying sizes, it's possible that the new 128 compiler will position those fields differently. A work-around to get back to the old layout 129 exists; if you believe you could be affected, please [send me](mailto:temporal@gmail.com) your 130 schema and I'll tell you what to do. [Gory details.](https://groups.google.com/d/msg/capnproto/NIYbD0haP38/pH5LildInwIJ) 131 132 **Road Map** 133 134 v0.3 will come in a couple weeks and will include several new features and clean-ups that can now 135 be implemented more easily given the new compiler. This will also hopefully be the first release 136 that officially supports a language other than C++. 137 138 The following release, v0.4, will hopefully be the first release implementing RPC. 139 140 _PS. If you are wondering, compared to the Haskell version, the new compiler is about 50% more 141 lines of code and about 4x faster. The speed increase should be taken with a grain of salt, 142 though, as my Haskell code did all kinds of horribly slow things. The code size is, I think, not 143 bad, considering that Haskell specializes in concision -- but, again, I'm sure a Haskell expert 144 could have written shorter code._