The Art of Science

If you’ve taken a look through reia-lang.org you may have noticed several references to Reia using High Performance Erlang, otherwise known as HiPE.  This is a Just-In-Time (JIT) compiler for Erlang which actually produces native code.  JIT is a long sought after feature in virtual machines as it allows the typical "virtual CPU" module of VMs to be bypassed entirely with program instructions running directly on the physical CPU.  In other words: speed!

Well, those statements were more about planned support for HiPE, even if they made it seem it already existed.  Well, no longer: Reia will now automatically take advantage of HiPE if support is present in your Erlang VM.

Erlang is typically regarded as a slow language, but that sort of description always needs to be taken with a grain of salt: in what regard is it slow, and compared to what?  Looking at the Computer Language Benchmarks Game, Erlang manages to beat out the popular dynamic scripting languages by a considerable margin, including Perl, PHP, Python, and Ruby.  Compared to these languages Erlang seems relatively fast.

The way Reia is presently implemented it usurps a number of the performance enhancements the Erlang compiler can normally achieve, and also saddles the runtime with a number of slow features such as a method dispatch system.  Reia will never be as fast as Erlang, but that said, Reia can already leverage the performance enhancements a JIT compiler brings to the table.

One of the complaints I’ve heard about so many languages is an inability to do certain things from the interactive interpreter which you can do otherwise in content loaded from files.  Erlang does not allow module declarations from the interactive interpreter.  And while it’s not entirely impossible, it’s indicative of a wide gulf between the language as evaluated on the fly and a language which is fundamentally designed to be compiled.

I greatly enjoyed how Ruby allows you do to virtually anything from the interactive interpreter that you otherwise could in normal program code, including defining functions as well as declaring modules and classes.  I sought to do the same in Reia, my dynamic scripting language for Erlang’s VM.  Reia now allows on-the-fly module declarations from anywhere, including both scripts and the interactive interpreter.  For example, the latest version on github allows you to do declare a module from the interactive interperer like this:

  >> module Foo
  ..   def bar
  ..     42
  .. 
  => ~ok
  >> Foo.bar()
  => 42 

This declares a Reia module named ‘Foo’ and defines one function under it, which returns 42.  After being declared, the ‘Foo’ module is compiled to BEAM bytecode and loaded directly into Erlang’s code server.  The "~ok" you see returned (which is Reia syntax for an atom called ‘ok’) is actually coming from the Smerl library by Yariv Sadan, and indicates that the module was successfully compiled and loaded into the code server.  Smerl allows for simple metaprogramming in Erlang by allowing you to build "meta modules" on-the-fly which then outputs all the Erlang forms needed to pass to compile:forms in order to buld a BEAM (or HiPE) module.

When Reia evaluates module declarations, it first compiles the Reia code to Erlang forms then passes them to Smerl to build modules, which invokes the Erlang compiler and loads the resulting bytecode into the Erlang code server.  The end result: on-the-fly module declaration.  You can call methods from compiled Reia modules just as you would any Erlang function, just keeping in mind that the module name is capitalized.  From eshell, you could call ‘Foo’:bar() to invoke the same function.

This allows a number of things which Erlang presently does not, and that doesn’t just mean declaring modules from the interactive interpreter.  Reia allows multiple module declarations from the same file, and furthermore provides a toplevel scope which exists outside of modules and is evaluated as the file is being loaded.  This means you can declare modules in a file then invoke functions in them immediately afterward, just like the above example from the interactive interpreter.

There’s a number of limits at present.  All module declarations need to be done from toplevel scope, i.e. you can’t conditionally declare a module.  You also can’t place statements besides defs inside a module declaration, so you can’t conditionally declare functions/methods either.  However, that said, module declaration is no longer limited to compiled code in Reia.  You can now do it anywhere.

There’s several possible optimizations down the road.  The compiler could be optimized to:

1. Automatically produce HiPE versions of the modules if HiPE is available
2. Keep a cache of generated BEAM/HiPE bytecode and only recompile if the original source file changes
3. Precompile multiple modules in the same source file, employing cross-module optimizations