capnproto/doc/roadmap.md

layout	title
page	Road Map

Road Map

This is a list of big ideas we'd like to implement in Cap'n Proto. We don't know in what order these will actually happen; as always, real work is driven by real-world needs.

Language Features

• Inline lists: Lets you define a field composed of a fixed number of elements of the same type, and have those elements be stored directly within the struct rather than as a separate object. Useful mainly to avoid the need to validate list lengths when the length should always be the same. Also saves a pointer's worth of space.
• Type aliases: Ability to define a type which is just an alias of some other type, and have it show up as e.g. a typedef in languages that support that. (The current using keyword is intended only for local use and does not affect code generation.)
• Doc comments: Harvest doc comments from schema files and use them to generate doc comments on generated code. Also make them available in the compiled schema so that a documentation generator could use them.
• Encapsulated types: This will allow you to create a hand-written wrapper around a type which will be automatically injected into the generated code, so that you can provide a nicer interface which encapsulates the type's inner state.
• Maps: Based on encapsulated and parameterized types.

RPC Protocol Features

• Dynamic schema transmission: Allow e.g. Python applications to obtain schemas directly from the RPC server so that they need not have a local copy. Great for interactive debugging.
• Three-way introductions (level 3 RPC): Allow RPC interactions between more than two parties, with new connections formed automatically as needed.
• Bulk and Realtime: Add features that make it easier to design Cap'n Proto APIs for bulk data transfers (with flow control) and realtime communications (where it's better to drop messages than to deliver them late).
• UDP transport: Cap'n Proto RPC could benefit from implementing a UDP transport, in order to achieve zero-round-trip three-party introductions and to implement "realtime" APIs (see "bulk and realtime", above).
• Encrypted transport: Cap'n Proto RPC should support an encrypted transport which uses capability-based authorization (not PKI), can accomplish zero-round-trip three-party introductions (via a pre-shared key from the introducer) and based on modern crypto. TLS is not designed for this, but we don't want to invent new crypto; we intend to build on libsodium and the Noise Protocol Framework as much as possible.

C++ Cap'n Proto API Features

• Plain Old C Structs: The code generator should also generate a POCS type corresponding to each struct type. The POCS type would use traditional memory allocation, thus would not support zero-copy, but would support a more traditional and easy-to-use C++ API, including the ability to mutate the object over time without convoluted memory management. POCS types could be extracted from and inserted into messages with a single copy, allowing them to be used easily in non-performance-critical code.
• Multi-threading: It should be made easy to assign different Cap'n Proto RPC objects to different threads and have them be able to safely call each other. Each thread would still have an anyschronous event loop serving the objects that belong to it.
• Shared memory RPC: Zero-copy inter-process communication.
• JSON codec customization: Extend the JSON library to support customizing the JSON representation using annotations. For example, a field could be given a different name in JSON than it is in Cap'n Proto. The goal of these features would be to allow any reasonable pre-existing JSON schema to be representable as a Cap'n Proto type definition, so that servers implementing JSON APIs can use Cap'n Proto exclusively on the server side.
• LZ4 integration: Integrate LZ4 compression into the API to further reduce bandwidth needs with minimal CPU overhead.
• Annotations API: For each annotation definition, generate code which assists in extracting that annotation from schema objects in a type-safe way.

C++ KJ API Features

KJ is a framework library that is bundled with Cap'n Proto, but is broadly applicable to C++ applications even if they don't use Cap'n Proto serialization.

• Fiber-based concurrency: The C++ runtime's event loop concurrency model will be augmented with support for fibers, which are like threads except that context switches happen only at well-defined points (thus avoiding the need for mutex locking). Fibers essentially provide syntax sugar on top of the event loop model.
• TLS bindings: Write bindings for e.g. OpenSSL to make it easy to integrate with the KJ I/O framework, Cap'n Proto RPC, and the KJ HTTP library.
• Modern crypto bindings: A thin wrapper around libsodium with a nice C++ API, e.g. representing keys using fixed-size, trivially-copyable classes.
• Event loop integrations: We should provide off-the-shelf integrations with popular event loop libraries, such as libuv, libev, libevent, boost::asio, and others, so that it's easier to use Cap'n Proto RPC in applications that already use another event framework.

Storage

• ORM interface: Define a standard interface for capabilities that represent remotely-stored objects, with get, put, publish, and subscribe methods. Ideally, parameterize this interface on the stored type.
• mmap-friendly mutable storage format: Define a standard storage format that is friendly to mmap-based use while allowing modification. (With the current serialization format, mmap is only useful for read-only structures.) Possibly based on the ORM interface, updates only possible at the granularity of a whole ORM entry.

Tools

• Schema compatibility checker: Add a capnp command which, given two schemas, verifies that the latter is a compatible upgrade from the former. This could be used as a git hook to prevent submission of schema changes that would break wire compatibility.
• RPC debugger: Add a capnp command which sends an RPC from the command line and prints the result. Useful for debugging RPC servers.

Quality Assurance

These things absolutely must happen before any 1.0 release. Note that it's not yet decided when a 1.0 release would happen nor how many 0.x releases might precede it.

• Expand test coverage: There are lots of tests now, but some important scenarios, such as handling invalid of invalid input, need better testing.
• Performance review: Performance is already very good compared to competitors, but at some point we need to break out the profiler and really hone down on the details.
• Security review: We need a careful security review to make sure malicious input cannot crash an application or corrupt memory.

Infrastructure

Note: These are very large projects.

• JSON-HTTP proxy: Develop a web server which can expose a Cap'n Proto RPC backend as a JSON-over-HTTP protocol.
• Database: A fast storage database based on Cap'n Proto which implements the ORM interface on top of the mmap storage format.