What Makes Ruby Tick

When I first started working with computers, one thing really amazed me. You can literally print out the code for something like the Morris worm (a malicious bit of code that wreaked all sorts of havoc) and it was quite harmless. But, when fed to the computer in the “proper way” it suddenly turned into something quite different. How does this work with ruby? How does your code go from being “just text” into something that can connect to an HTTP server or fire off some SQL queries?

What Ruby are We Talking About?

Here comes the shocker: there’s actually more than one ruby. In fact, we don’t know how many rubies there are, really. We all know just the one, you know, the one you can get from apt-get or rvm or something. But, in reality, it is only one of the implementations of the Ruby programming language. Ruby (the language) is a specification which compilers/interpreters are built to implement. The Perl community has come up with a nice convention for this. “Perl” (with the uppercase “P”) denotes the language itself, where “perl” denotes the implementation of the language. So, saying pass perl the -f flag is okay, but, saying that with “Perl” doesn’t make any sense. In fact, there is the Rubinius implementation of the Ruby programming language which offers some pretty neat upsides in comparision to the standard implementation. Anyway, we don’t really know how many rubies there are. Maybe someone is making a new ruby right now, in some garage or garden shed somewhere. Henceforth, whenever we talk about something related to the specific implementation of Ruby, we always must specify what implementation we are talking about.

The Virtual Machine

An essential portion of the mainline ruby interpreter is the virtual machine. Long story short: A virtual machine is simply a sort of virtual computer that follows some preset commands. Regular computers have something called “assembly language” which are short commands which they know how to execute (these commands directly translate into binary code, which computers can crunch on happily). A virtual machine is some software that runs on a computer and emulates this sort of thing, except for the fact that whoever wrote that software gets to decide what sort of “short commands” this machine will understand. These short commands are called instructions. These instructions are usually fundamental things that can’t be derivied from other things (a bit like axioms in math/geometry). For example, the Java Virtual Machine has seperate instructions for addition, storing in memory, and type conversion. A bunch of these instructions put together into something cohesive (i.e. something the virtual machine can run) is called bytecode. Now, how does this virtual machine come into play with Ruby? Well, when we type in “ruby hello.rb”, hello.rb isn’t run immediately, instead, it gets converted into bytecode. Then, this bytecode is interpreted and the program is run.

Why So Indirect?

Why do we need this whole virtual machine thing in the middle anyway? Why can’t we just run the code “directly”? First of all, it is much easier to “reason” with a small instruction set than with all of Ruby. So, when the bytecode is actually run, some of it can be optimized. Second of all, writing a direct interpreter leads to all sorts of insanity in actually “converting” the Ruby code into something that can be run immediately. Lastly, the bytecode is then made completely independent of the operating system and underlying hardware. As a result, wherever you are able to implement the virtual machine, Ruby will be able to run. In fact, the Java Virtual Machine has been a hotbed of programming language development. JRuby, JPython, Clojure, Scala and all sorts of other exciting languages have taken advantage of the portability of the Java Virtual Machine and built languages that can be turned into JVM bytecode.

The Ruby Virtual Machine

Ruby presents a very interesting case to look at when considering virtual machines, because it has recently gone under a significant change. Matz’s Ruby (or, MRI) now uses something called YARV (Yet Another Ruby VM) as its underlying virtual machine. The move to YARV happened in the 1.9 release because the old virtual machine had some optimization problems. By swapping out the underlying virtual machine, Ruby is quickly losing its image of being “too slow for any real use”. It is pretty commendable that the Ruby core team was able to seperate out the virtual machine within one release cycle. That speaks not just of the team’s work on the problem, but their ability to keep things separated. I still get flashbacks from the times spent with the perl codebase. So far, we’ve come to grips with the fact that MRI takes your ruby code, turns it into bytecode for YARV, and then YARV executes that bytecode. But, how does this conversion into bytecode actually happen? Well, there’s two parts to it.

Tokenizer

Compiler/interpreter construction is a very interesting, but sometimes very theoretical, portion of computer science. I’m going to try and keep the theory (and, therefore, rigor) to a minimum to provide a nice overview of what is happening. MRI takes a look at your Ruby code and categorizes it into chunks. These chunks are called tokens and the portion that does this is called the “lexer”. Considering Ruby code in this example would be much too complex, so let’s try and relate this to something like a “calculator language”. A line in this language might look like this (without quotes): (4 + 5)/2 The lexer will take this and turn it into: OPENPAREN NUMBER OPERATOR NUMBER CLOSEPAREN OPERATOR NUMBER That doesn’t look like much of a change to us, but we’ve just shifted the problem domain completely. We’ve taken the input (just a string of characters and symbols) to something that gives us an idea of what type of input we are handling. To the lexer, both 496 and 46 constitute a NUMBER, giving the general structure of the string. Now, what do we do with this newly accquired information?

Parsing

The parser basically take the tokens obtained from the lexer and turns them into bytecode. Let’s continue on with our example of the calculator language. The parser will recognize that “NUMBER OPERATOR NUMBER” is a pattern which leads to some specific bytecode (i.e. addition or division) and it can take these tokens and translate them into the bytecode which runs on the underlying virtual machine. MRI does something quite similar (although far more complicated) where it looks for specific patterns in the tokens and then turns them into bytecode. Now, Ruby is a large language, so it can’t possibly be this simple. Unfortunately, it isn’t. There are several more constructs that come under and between the enormous domes of lexing and parsing (e.g. AST trees). However, the general idea is fairly simple – the lexer figures out tokens, the parser looks at patterns within these tokens to churn out the bytecode.

Conclusion

Hopefully you’ve enjoyed the super quick, all-over-the-place look at how Ruby actually looks at your code before magically spitting out the output. Compiler construction is an awesome subject and one that combines theory with practice (which is something I find very interesting) – this guide should have given you a very rough overview of how stuff is built. Thanks for reading!

Frequently Asked Questions about Ruby Tick