Console #29
Lunatics, Lisp, and Opa
lunatic
Lunatic is an actor system for Rust/WebAssembly.
language: Rust, stars: 420, watchers: 15, forks: 7, issues: 3
last commit: November 27, 2020, first commit: August 26, 2020
awesome-lisp-companies
Awesome Lisp Companies is a curated list for companies that use Lisp extensively in their stack.
stars: 177, watchers: 22, forks: 17, issues: 1
last commit: November 24, 2020, first commit: December 29, 2017
https://twitter.com/azzamsyawqi
browsh
Browsh is a fully interactive, real-time, and modern text-based browser rendered to TTYs and browsers.
language: Javascript, stars: 12807, watchers: 207, forks: 368, issues: 157
last commit: October 31, 2020, first commit: April 22, 2016
web-ext
web-ext is a command line tool to help build, run, and test web extensions.
language: Javascript, stars:1433, watchers:52, forks: 272, issues:157
last commit: November 20, 2020, first commit: February 01, 2016
opa
The Open Policy Agent (OPA) is an open source, general-purpose policy engine that enables unified, context-aware policy enforcement across the entire stack.
language: Go, stars: 4269, watchers: 102, forks: 500, issues: 222
last commit: November 25, 2020, first commit: December 28, 2015
https://twitter.com/openpolicyagent
An Interview With Bernard Kolobara of Lunatic
What is your background?
I was born in Bosnia but, because of the war in the 90s, I spent a big part of my childhood as a refugee in Croatia and Germany. From an early age, I was fascinated by computers, even though I didn’t have access to them. These gray boxes with a black screen and text scrolling over them just seemed magical to me. I would consume all content related to them, watch all the computer segments on tv and annoy local shop owners by standing there reading through all the computer magazines but never buying them.
Naturally, this led me to pick up programming once I got my first computer. I started with the BASIC programming language, but during my time at the University of Zagreb I tried to learn anything I could get my hands on C, Java, Python, Ruby, Clojure, you name it. Just being exposed to so many languages and programming paradigms made me a much better developer. It is good to master a few tools, but you also need to experience everything else that is out there before settling.
During my life, I moved a lot around Europe and worked for different companies. I lived in Munich (Germany), Geneva (Switzerland), and Lisbon (Portugal). From my previous jobs, the one that left the biggest impact on me was CERN. I worked there on data acquisition monitoring tools for the Atlas experiment. Working on such an ambitious project and experiencing first hand the scale of it was fulfilling.
Currently, I live in Vienna (Austria) and work on Lunatic full time.
What was it like growing up as a refugee?
I was a kid, so I did kids’ stuff. My parents did a great job in allowing me to have a more or less normal childhood with the resources they had. Except for the constant moving and trying to fit in, I don’t remember it being particularly rough.
What's an opinion you have that most people don't agree with?
Apparently, trusting in science and believing the experts is not something most people nowadays do.
If you could dictate that everyone in the world should read one book, what would it be?
The Willpower Instinct by Kelly McGonigal. It may look like a typical self-help book, but it contains a lot of deep research and simple tricks that helped me stay more productive.
If I gave you $10 million to invest in one thing right now, where would you put it?
It’s always a good idea to maintain a well-diversified portfolio of stocks, bonds, and some cryptocurrency. But if I could put it in only one thing I would use it to hire a great team to help me out with Lunatic.
What would the 1-year roadmap for this hypothetical team of Lunatic developers be?
There are a few features I would right away start working on. One of the big ones would be adding support for other languages. Currently, there is only first-class support for Rust, but any language that can compile to WebAssembly could be supported. There is a lot of development going on across different languages to add WebAssembly support and Lunatic directly benefits from it. Another feature I can’t wait to get out is support for running Lunatic in a distributed cluster.
Of course, testing and building a bug-free runtime would always stay a priority. But with these two I believe that Lunatic would be ready for mass adoption.
Why was Lunatic started?
Last year I played around with the idea of building a browser-based massive multiplayer game. It’s common knowledge in the game development community that you should not be too ambitious with your first project. However, I didn’t want to spend my time building simple Tetris clones. In the end, I decided that I will build a set of tools that allow me to build multiplayer games with the complexity of single-player games so that even without any previous experience building games I could take on a more challenging first project.
One of the hardest problems to solve in multiplayer games is how to synchronize state between different game clients, so everyone has the same view of the world. I came up with the idea to just collect all the input on the server, make all the calculations, and send a diff of the world changes to the clients. In this case, every client would see the same world state and I could just focus on the server-side. This limits the development to a simple game loop and I could fully focus on the gameplay instead of synchronization. A big inspiration for this architecture was Elixir’s library Phoenix LiveView. They showcased an example of rendering website updates at 60fps using multiple clients without any issues. Because of this, I picked Elixir as the language to implement my game framework in. However, I soon started running into performance problems with Elixir. Erlang, the platform Elixir runs on, is optimised for networking, but not for doing a lot of computation-intensive tasks. Since games consist of mostly math, this presented right away a big obstacle. As I started exploring other alternatives to Erlang/Elixir it became clear that no true competitor existed.
Erlang seemed so far ahead in some aspects. Just take a look at hot code reloading. Erlang was developed originally to run on top of phone switches and you can’t quit a phone call between two parties just because you decided to update your application. So they developed this elegant way of updating the code without even breaking a tcp connection. The complete isolation of Erlang processes is another feature I couldn’t find in any other programming language or framework, except for much heavier alternatives like Docker.
This motivated me to start working on my alternative. Around the same time, WebAssembly was becoming more popular and I decided to integrate it into Lunatic. Browser vendors invested a lot of money and effort to make WebAssembly secure and lightweight. Finally, there was an alternative that I could use for process isolation with characteristics similar to Erlang's, but usable from other languages. At this point, all the puzzle pieces fell in place for Lunatic’s architecture.
What was the game idea?
It was a combination of GTA 2 and Hacknet, as multiplayer. The world you play in would be completely persistent, allowing people to write some code on a computer that is executed while they are offline. This was one of the main reasons I needed strong sandboxing. The missions would require some computer knowledge and sometimes coordination between multiple players. However, the main goal was to create a sandboxed environment for people to explore and modify.
Players go through great trouble to construct amazing worlds in Minecraft. I was wondering what they would be creating if their tool was code instead of blocks.
Are there any overarching goals of Lunatic that drive design or implementation?
I would love for Lunatic to be easily scalable and eliminate the whole monolith vs microservices discussion. You should be able to build an application, run it on your development machine, and then move it to a cluster of 100 machines without any modification to your code. As Lunatic’s design is based on processes that communicate with each other through messages, if those are sent locally on the same machine or exchanged over a network between different machines, it should not make a difference to your application. You should still get a simple development experience, but can also later scale if you need to.
What tradeoffs have been made in Lunatic as a consequence of these goals?
We need to avoid global state at all cost. If we want to be able to move a process from one machine to another this process needs to contain his whole state and can’t share any resources with others.
What is the most challenging problem that’s been solved in Lunatic, so far?
Lunatic is implemented in Rust and uses a work-stealing async executor to handle massive concurrency. The async story in Rust is really good, but one issue I ran into right away was how Rust implements the zero cost async abstraction. The Rust compiler will take your async code and construct a “perfect” state machine. Perfect because you should not be able to do it more efficiently by hand. For this to work, Rust needs to see your whole application, but because Lunatic uses WebAssembly that generates JIT code, Rust doesn’t know how this code will look and can’t generate the state machine. To solve this, I developed a library called async-wormhole, which uses a separate stack to execute the JIT generated code. Using some ideas pioneered by the generator library libfringe, async-wormhole has almost no overhead except a bit of additional space for the stack. It allows you to switch between processes in sub-nanosecond time, making it equivalent to a function call. It’s at the core of Lunatic’s design.
How do you eventually intend to monetize Lunatic?
Lunatic is open source. I think that it would be really hard nowadays to get developers to use proprietary runtimes for their code and make their success dependent on your survival. However, even if I’m not able to maintain the project anymore in the future, hopefully, there will be an engaged community continuing the development.
I intend to eventually develop a rich set of tools around Lunatic, for deployment and distribution, that I will hopefully be able to monetize. This would allow me to continue dedicating my time to Lunatic.
What is the best way for a new developer to contribute to Lunatic?
If you would like to add a new feature or have some other ideas, I would suggest you just open a GitHub issue. From there we can discuss the best way to approach it and I can even guide you with the implementation. The project is young and there is a lot to be done. I’m looking forward to people joining in.
Where do you see software development in-general heading next?
I don’t think that there is one specific direction that software development is headed. I believe that the ecosystem is just going to get more diverse with many different approaches to different problems, which is a good thing, we need to keep innovating and trying different things.
Do you have any suggestions for someone trying to make their first contribution to an open-source project?
Try to work on something that you use or need. In this case, you will have a much better understanding of the problem and possible solutions. Most of my contributions are bugs in libraries that I discovered while using them or features I needed.
This Week’s Question
If Console had a revenue share for inviting friends, would you participate?
Last Week’s Responses
Which type of projects would you like to see more in Console?
Like what you saw here? Why not share it?
Or, better yet, share Console!
Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.