Decoupling the development time experience of Flutter from the Dart programming language.
Hydro-SDK is a project with one large, ambitious goal. "Become React Native for Flutter".
It aims to do that by:
- Decoupling the API surface of Flutter from the Dart programming language.
- Decoupling the development time experience of Flutter from the Dart programming language.
- Providing first-class support for over-the-air distribution of code.
- Providing an ecosystem of packages from
pub.dev, automatically projected to supported languages and published to other package systems.
In this article I want to delve into some of the gritty details of how Hydro-SDK compiles, manages and hot-reloads TypeScript code.
ts2hc is a command line program distributed as part of each Hydro-SDK release.
ts2hc's job is to turn TypeScript code into bytecode and debug symbols. Users generally don't interact with it directly but rather indirectly through commands like
hydroc build and
ts2hc will lower
counter-app/ota/lib/counterApp.ts, and all of its dependencies into individual Lua modules. These Lua modules are then bundled, with the result looking something like the following:
The lowered and bundled Lua still somewhat resembles the input TypeScript. TypeScript ES6 modules are wrapped into Lua immediately invoked function expressions (IIFE), assigned string keys in the
package.preload map and their
exports made available by
Lua lacks builtin object-oriented programming (OOP) facilities (whether prototypal or otherwise). TypeScript language features which don't quite map one-to-one with Lua are shimmed using
__TS_* functions made available through the
lualib_bundle module (which
ts2hc injects during bundling). Above, the
CounterApp class is lowered into a series of calls to
__TS__ClassExtends, followed by placing its declared methods on its
The Lua bundle output by
ts2hc will eventually be turned into bytecode by the PUC-RIO Lua 5.2 compiler, distributed under the name
luac52 by Hydro-SDK. The
build method on the
CounterApp class above would compile into something like the following:
Lua bytecode is outside the scope of this article, though is mentioned here for completeness.
In addition to lowering,
ts2hc also undertakes an analysis pass of each output Lua module to discover its functions.
From the above example,
ts2hc will record the following functions:
corresponding to the constructor and
build method declarations of the original
CounterApp class. These names are obviously captured from the original declaration names. For cases like our example (where clear function names where given), this works well enough.
ts2hc cannot rely on the programmer giving it clear and comprehensible function names however.
ts2hc takes the original symbol names from the output Lua module and performs name mangling. The intent of name mangling is to uniquely identify a given function, no matter where or how it was declared.
ts2hc's name mangling approach is heavily inspired by the IA-64 Itanium C++ ABI as well as Rust's name mangling approach. Whereas those two approaches rely heavily on type-information to produce their mangled names, Lua is a dynamic and un-typed language and offers no such convenience.
ts2hc first considers the names and declaration order of function arguments resulting in the following:
That is not quite enough information to uniquely identify a function however.
ts2hc further considers the hash of the TypeScript filename, and a disambiguation index suffix to resolve mangled name conflicts by declaration order resulting in the following:
This form works perfectly for functions that are named by the programmer like class methods or free functions. Consider however if the
build method on the
CounterApp class were written to have an anonymous closure:
For anonymous closures,
ts2hc simply names them "anonymousclosure". In order to uniquely identify anonymous closures (or any nested function declarations), a [dominator analysis](https://en.wikipedia.org/wiki/Dominator(graph_theory)) with respect to the declaration order of every function is performed. The dominance frontier of the root function (in our case,
CounterApp.build) forms a directed acyclic graph. A walk along the transitive reduction of the dominance frontier from the root function to a given child defines the ordering of the mangled names that child needs to include in order to be unique.
For the anonymous closure in
CounterApp.build above, this yields the following:
This form encodes enough information to uniquely identify a function.
ts2hc will join each functions mangled name with information like the line/column numbers in the original TypeScript file, the line/column numbers in the Lua module the original TypeScript file was lowered into, the line/column numbers the function ended up in in the final Lua bundle and other information into a single debug symbol. These debug symbols are what power function maps, provide readable stack traces as well as hot-reload.
Common Flutter Runtime (CFR) is a blanket term given to the Lua 5.2 virtual machine, binding system and other libraries at the core of Hydro-SDK's runtime environment. Users generally don't interact with the CFR directly, but rather through widgets like
ts2hc and CFR are at the very core of the developer experience and runtime system of Hydro-SDK. They work together to support goal 2 above by providing an analog to Flutter's killer development-time feature; hot-reload.
In Flutter, hot-reload is provided by Dart VM. Dart VM's hot-reload is based on a few pillars:
- The program behaves as if method lookup happens at every call
- The "atoms" of reload are methods. Methods are never mutated. Changes to a method declaration create a new method, mutating the class or library's method dictionary. The old method may still exist if it has been captured by a closure or stack frame.
- Closures capture their function when they are created. A closure always has the same function before and after a change, and all invocations of a given closure run the same function.
State is Retained
- Hot reload does not reset fields, neither the fields of instances nor those of classes or libraries
CFR's hot-reload is inspired by (and largely abides by) the same pillars. CFR diverges from Dart VM however on the "Immutable Methods" pillar. In CFR, closures (and their scopes) are refreshed prior to every invocation. This means that old functions can never be invoked after a hot-reload no matter if they were captured by a closure. The only exception to this is if an old function is a stack frame. CFR uses the mangled name of debug symbols provided to it by
ts2hc to uniquely address functions, allowing it to perform just-in-time method lookups and late-binding in a manner similar to Dart VM.
build method of the
MyHomePageState class from the Counter-App showcase:
Making simple additions or deletions like changing the string "You have pushed the button this many times" to something else or adding more
Text widgets results in successful hot-reloads and no change in app state.
Consider a change to the original
build method to the following:
The above change will result in an error like the following:
This error is the result of
Colors.blue.swatch being uninitialized. Recall the example Lua module output above. Imported symbols are assigned to the value of calls to
build method now closes over symbols that are uninitialized (the newly imported symbols). This is unfortunately an artifact of how TypeScript modules are represented when lowered. The result is that referencing newly imported symbols in a hot-reloaded function will usually trigger an exception.
Hot-reload in Hydro-SDK is implemented purely in terms of Lua. Support for hot-reloading other programming languages (like Haxe and C#) in Hydro-SDK should not suffer from this same limitation (though will probably come with their own challenges and limitations).