How to compile code in the browser with WebAssembly
Browsers have become powerful beasts. First used to share research papers at CERN, the browser can now run Google Earth, play Unity 3D games, and even design buildings in AutoCAD.
With this kind of power, could a browser compile and run your code too? Ridiculous. Surely that couldn’t work…
Then again, why not? There’s no way I could ignore such a fascinating challenge. After four months of punching the keys and poring over documentation, I’ve finally created my answer: Hackpad.
Update August, 2021: Go Wasm is now named Hackpad!
Hackpad is a Go development environment with the essentials to write and run code entirely within the browser, using the power of WebAssembly (Wasm). It’s also completely open source. Hackpad is made up of three core WebAssembly components: An “operating system,” an editor, and a shell.
In this article, I’ll show you what Hackpad is, how it works, and what lies ahead.
Write code with Hackpad
Hackpad makes it possible to both write and run Go programs using the actual Go compiler. In other words: Write code, go build, then run the program. This is quite different from familiar sites like Go Playground because the code actually runs in the browser — even after you disconnect from the internet.
On the demo site Hackpad boots the OS, places a Go install onto the virtual file system, starts the code editor, and surfaces common tools like terminals and compiler controls. I’ve layered together three key programs to make this work, and I’ll dig into each of these in the next section. First, a quick tour of the IDE.
In the editor you can write Go code in multiple tabs, import external libraries, reformat the code, and run your program.
To run custom Go CLI commands or other programs, just open up a terminal. To pass input to a program or save its output, use a file redirect like ./playground > output.txt or connect outputs to inputs with | pipes.
You can also build and install custom Wasm programs anywhere on the virtual file system. In the above screenshot, the count-lines program was compiled and saved in /bin. From here, it can be run just like any other built-in command.
How it works
Hackpad is composed of three core programs: The “operating system,” the editor, and one or more shells. All three of these are sandwiched together with the browser, where the operating system is a translator to access virtual files and processes.
At a high level, the operating system (OS) intercepts system calls between Go programs and the browser to provide a virtual file system and processes. Since the browser doesn’t have any notion of files or processes, this OS is an essential building block to run Go programs. It isn’t really a true OS, but it does provide important abstractions normally unavailable in browsers.
We’ll also need a Go compiler and standard library to write our programs. To kick us off, a Go installation is automatically downloaded onto the virtual file system at startup.
The editor and shell programs sit atop the OS just like a compiled program from the editor. The editor provides file tabs, controls, and terminals to write and run your Go code. A shell is launched in each terminal tab so you can run custom go commands or your own programs, like a real terminal.
“Operating system”
Operating systems run programs on all kinds of different hardware. They present programs with a facade of resources to interact with, but they usually don’t allow direct access. If a program opens a file and writes some data, then the OS returns a file handle and dutifully writes the data to a hardware disk. These kinds of important actions are often called system calls, or “syscalls.”
Similarly, Hackpad’s operating system makes it possible to run Go programs by intercepting system calls and manipulating virtual resources instead of real ones. Since Go programs targeting Wasm rely on global JavaScript functions to perform system calls, the OS component replaces them with custom code. Browsers don’t actually provide files or processes, so these replacement functions create virtual versions and then hand them back to Go.
Processes in a browser present an interesting problem. To the seasoned web developer, it’s obvious there’s no such thing as a “JavaScript process,” and there’s definitely no way to run two programs at exactly the same time. To combat this limitation, I used context switching to jump between processes as if they ran on a single processor core. Context switching allows the OS to swap out process-specific file tables and environment variables whenever a new process takes priority. Tricky business for sure, but now JavaScript’s built-in task scheduler can take it from here.
Virtual file system
A short aside on the virtual file system, since it was especially difficult to get right. A modern-day file system (FS) includes loads of edge-cases and features. File permissions, native pipes, and file locks are all required to work perfectly for go build to run without hitting a fatal error. Ugh.
To save some time, I avoided writing my own in-memory file system from scratch and picked up Afero. Afero provided a good starting point and defined a powerful FS abstraction. From there, I created several custom file systems: A mountable FS, a streaming gzip FS, and an experimental IndexedDB FS. Unfortunately, even with a strong foundation, bugs popped up everywhere.
It turns out the Go CLI is quite unhappy when the operating system beneath it isn’t real. Shocking, I know.
It took a little over a month to nail down every file system bug. Tracking down bugs was especially difficult because most of them crashed the OS or printed cryptic error messages. I hope browser vendors and the Wasm community can spend more time on the debugging experience, as this was a huge pain point.
Editor
The editor fills a more conventional web app role. It sets up the web page with code editor tabs, controls, a build console, and terminal tabs.
The editor program runs atop the OS, starting new processes like any normal Go program would. Today, the editor tabs run CodeMirror and keep their connected files in sync with the file system.
The terminals start shell processes, then connect input and output to xterm.js for a more familiar terminal experience.
Shell
The shells inside each terminal tab are an interactive way to run go commands or any other Wasm program on the file system. There were a few shells written in Go out there, but none that I tried actually compiled to Wasm and worked out of the box. Luckily, the shell was a fun learning process all its own. I had to learn all about terminal escape codes to support command-line input editing, then get fancy with process file redirects and environment variables.
And that’s it! I love combining new libraries in Hackpad to test my crazy ideas. I can’t wait to see what folks will do with it.
What lies ahead
WebAssembly certainly has its ups and downs right now, but it also has huge potential just waiting to be realized.
For the upsides, the community has been busy. They’re putting up promising proposals and curating all sorts of standards to push the envelope. In particular, I’m keeping my eye on Wasm Threads and the evolving WASI standard as the next game-changers in this space.
Unfortunately, there are some downsides. One of my biggest gripes is the lack of debugger support for Wasm. While there is a debugger in the browser, it can only step through the assembly instructions — not the source code. Since I’m not fluent in the Wasm text format, diagnosing and fixing issues in my source language is super frustrating.
As for Hackpad, there’s a big exciting future ready for the taking. If the community shows interest, I’d love to add features like mobile device support (lower memory usage), file saves across sessions, or maybe native execution of WASI programs! If you want a piece of the action, give us a shout on Github.
