roadmap.md (7394B)
1 --- 2 layout: page 3 title: Road Map 4 --- 5 6 # Road Map 7 8 This is a list of big ideas we'd like to implement in Cap'n Proto. We don't know in what order 9 these will actually happen; as always, real work is driven by real-world needs. 10 11 ## Language Features 12 13 * **Inline lists:** Lets you define a field composed of a fixed number of elements of the same 14 type, and have those elements be stored directly within the struct rather than as a separate 15 object. Useful mainly to avoid the need to validate list lengths when the length should always 16 be the same. Also saves a pointer's worth of space. 17 * **Type aliases:** Ability to define a type which is just an alias of some other type, and 18 have it show up as e.g. a `typedef` in languages that support that. (The current `using` 19 keyword is intended only for local use and does not affect code generation.) 20 * **Doc comments:** Harvest doc comments from schema files and use them to generate doc comments 21 on generated code. Also make them available in the compiled schema so that a documentation 22 generator could use them. 23 * **Encapsulated types:** This will allow you to create a hand-written wrapper around a 24 type which will be automatically injected into the generated code, so that you can provide a 25 nicer interface which encapsulates the type's inner state. 26 * **Maps:** Based on encapsulated and parameterized types. 27 28 ## RPC Protocol Features 29 30 * **Dynamic schema transmission:** Allow e.g. Python applications to obtain schemas directly from 31 the RPC server so that they need not have a local copy. Great for interactive debugging. 32 * **Three-way introductions (level 3 RPC):** Allow RPC interactions between more than two parties, 33 with new connections formed automatically as needed. 34 * **Bulk and Realtime**: Add features that make it easier to design Cap'n Proto APIs for bulk 35 data transfers (with flow control) and realtime communications (where it's better to drop 36 messages than to deliver them late). 37 * **UDP transport**: Cap'n Proto RPC could benefit from implementing a UDP transport, in order 38 to achieve zero-round-trip three-party introductions and to implement "realtime" APIs (see 39 "bulk and realtime", above). 40 * **Encrypted transport**: Cap'n Proto RPC should support an encrypted transport which uses 41 capability-based authorization (not PKI), can accomplish zero-round-trip three-party 42 introductions (via a pre-shared key from the introducer) and based on modern crypto. TLS is 43 not designed for this, but we don't want to invent new crypto; we intend to build on 44 [libsodium](https://github.com/jedisct1/libsodium) and the 45 [Noise Protocol Framework](http://noiseprotocol.org/) as much as possible. 46 47 ## C++ Cap'n Proto API Features 48 49 * **Plain Old C Structs:** The code generator should also generate a POCS type corresponding 50 to each struct type. The POCS type would use traditional memory allocation, thus would not 51 support zero-copy, but would support a more traditional and easy-to-use C++ API, including 52 the ability to mutate the object over time without convoluted memory management. POCS types 53 could be extracted from and inserted into messages with a single copy, allowing them to be 54 used easily in non-performance-critical code. 55 * **Multi-threading:** It should be made easy to assign different Cap'n Proto RPC objects 56 to different threads and have them be able to safely call each other. Each thread would still 57 have an anyschronous event loop serving the objects that belong to it. 58 * **Shared memory RPC:** Zero-copy inter-process communication. 59 * **JSON codec customization:** Extend the JSON library to support customizing the JSON 60 representation using annotations. For example, a field could be given a different name in 61 JSON than it is in Cap'n Proto. The goal of these features would be to allow any reasonable 62 pre-existing JSON schema to be representable as a Cap'n Proto type definition, so that 63 servers implementing JSON APIs can use Cap'n Proto exclusively on the server side. 64 * **LZ4 integration:** Integrate LZ4 compression into the API to further reduce bandwidth needs 65 with minimal CPU overhead. 66 * **Annotations API:** For each annotation definition, generate code which assists in extracting 67 that annotation from schema objects in a type-safe way. 68 69 ## C++ KJ API Features 70 71 KJ is a framework library that is bundled with Cap'n Proto, but is broadly applicable to C++ 72 applications even if they don't use Cap'n Proto serialization. 73 74 * **Fiber-based concurrency:** The C++ runtime's event loop concurrency model will be augmented 75 with support for fibers, which are like threads except that context switches happen only at 76 well-defined points (thus avoiding the need for mutex locking). Fibers essentially provide 77 syntax sugar on top of the event loop model. 78 * **TLS bindings:** Write bindings for e.g. OpenSSL to make it easy to integrate with the KJ 79 I/O framework, Cap'n Proto RPC, and the KJ HTTP library. 80 * **Modern crypto bindings:** A thin wrapper around 81 [libsodium](https://github.com/jedisct1/libsodium) with a nice C++ API, e.g. representing 82 keys using fixed-size, trivially-copyable classes. 83 * **Event loop integrations:** We should provide off-the-shelf integrations with popular event 84 loop libraries, such as libuv, libev, libevent, boost::asio, and others, so that it's easier 85 to use Cap'n Proto RPC in applications that already use another event framework. 86 87 ## Storage 88 89 * **ORM interface:** Define a standard interface for capabilities that represent remotely-stored 90 objects, with get, put, publish, and subscribe methods. Ideally, parameterize this interface 91 on the stored type. 92 * **mmap-friendly mutable storage format:** Define a standard storage format that is friendly 93 to mmap-based use while allowing modification. (With the current serialization format, mmap 94 is only useful for read-only structures.) Possibly based on the ORM interface, updates only 95 possible at the granularity of a whole ORM entry. 96 97 ## Tools 98 99 * **Schema compatibility checker:** Add a `capnp` command which, given two schemas, verifies 100 that the latter is a compatible upgrade from the former. This could be used as a git hook 101 to prevent submission of schema changes that would break wire compatibility. 102 * **RPC debugger:** Add a `capnp` command which sends an RPC from the command line and prints 103 the result. Useful for debugging RPC servers. 104 105 ## Quality Assurance 106 107 These things absolutely must happen before any 1.0 release. Note that it's not yet decided when 108 a 1.0 release would happen nor how many 0.x releases might precede it. 109 110 * **Expand test coverage:** There are lots of tests now, but some important scenarios, such as 111 handling invalid of invalid input, need better testing. 112 * **Performance review:** Performance is already very good compared to competitors, but at some 113 point we need to break out the profiler and really hone down on the details. 114 * **Security review:** We need a careful security review to make sure malicious input cannot 115 crash an application or corrupt memory. 116 117 ## Infrastructure 118 119 Note: These are very large projects. 120 121 * **JSON-HTTP proxy:** Develop a web server which can expose a Cap'n Proto RPC backend as a 122 JSON-over-HTTP protocol. 123 * **Database:** A fast storage database based on Cap'n Proto which implements the ORM interface 124 on top of the mmap storage format.