Console #95 -- Serenity OS, OpenDrop, and Benthos

An Interview with Ashley of Benthos

Mar 6

Sponsorship

LEX - Real Estate Investing For All

Interested in investing in commercial real estate but not sure where to start?

LEX has created a new way for you to invest in real estate.

LEX turns individual buildings into public stocks via IPO so you can invest, trade, and manage your own portfolio of high-quality commercial real estate.

Any US investor can open a LEX account, browse opportunities in various asset classes such as multifamily and office buildings, and buy shares of individual buildings.

LEX opens up direct and tax advantaged ownership in an asset class that has previously been inaccessible to most investors.

Get started today and explore LEX’s live assets in New York City and upcoming IPO in Seattle.

Projects

Serenity OS

Serenity is a graphical Unix-like OS for x86, with 90s aesthetics.

language: C++, stars: 18210, watchers: 310, forks: 1960, issues: 562
last commit: March 03, 2022, first commit: October 10, 2018

social: twitter.com/awesomekling
repo: github.com/SerenityOS/serenity

OpenDrop is a command-line tool that allows sharing of files between devices directly over Wi-Fi. Its unique feature is that it is protocol-compatible with Apple AirDrop which allows sharing files with Apple devices running iOS and macOS.

language: Python, stars: 6686, watchers: 73, forks: 192, issues: 18
last commit: April 29, 2021, first commit: May 16, 2019

social: github.com/schmittner
repo: github.com/seemoo-lab/opendrop

benthos

Benthos is a high-performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations, and filters on payloads.

language: Go, stars: 4049, watchers: 91, forks: 387, issues: 197
last commit: February 23, 2022, first commit: February 10, 2021

social: twitter.com/jeffail
repo: github.com/Jeffail/benthos

Already subscribed? Why not share Console with the best engineer you know?

Share Console

An Interview With Ashley of Benthos

Hey Ashley! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?

I'm based in the UK and I was lucky in my junior years to find a hip start-up called DataSift near to where I lived. They had some awesome engineers to learn from, building bleeding edge data engineering tech for both batch and streaming. I've continued working in that space at various organizations for about ten years now.
For the first three years of my career, I was mostly working with C++11. I enjoyed the speed but hated the tooling. I've never worked on a C++ project where the build system didn't frustrate me. Then, about seven years ago, I was given opportunities to work in Go and I gradually phased C++ out of my life, and now I work pretty much exclusively in Go (and a bit of Rust).

Who or what are your biggest influences as a developer?

The fact that with software the feedback loop between an idea, implementation, and failure is almost frictionless. The only limiting factor was how much hurt my brain could handle. The more times I ran that loop the more often it misfired into something that worked and I could feel good about, and at that point, I felt other engineering disciplines were too sluggish and full of obstacles to get into.
I love that aspect of software more than anything else. It's what empowers us to work on an almost even playing field with huge enterprises and get the same level of satisfaction from a finished project in a month that could take another discipline years.

How do you separate good project ideas from bad ones?

If I don't find the idea exciting then it's bad because I won't finish it, otherwise, I find out by building it. As long as I'm having fun and learning then in my mind, it's a good project, even if it ends up nowhere (the majority of my ideas).
That philosophy doesn't scale when you need to present an idea to investors upfront, but one of my favorite aspects of open source is that the barrier to entry is so minimal that you can test drive several project ideas in the time it takes a traditional entrepreneur to put a deck together and iron their clothes.

Why was Benthos started?

It was a fairly common case of having a conceptual tool in my mind that I believed would solve a lot of problems at my job, but couldn't get permission to build it (rightfully, I think). Instead, I decided to do the experimental work as an open-source hobby project. I didn't originally think I'd finish it, I was mostly using it as a sort of showcase of all the modern features I thought a good production data streaming service should have.
After about a year of playing around, I had something stable that I felt was ready to try out, and by surprise, we still had those same problems at work. It would take another year or so to get it deployed as a trial, and from that point onwards I'm pretty sure a plot of its growth in the platform would look exponential. Eventually, other organizations took interest and since then it's been a steady growth of both the project and my ability to live off it.

What were some of the problems you were experiencing at your job?

I'd summarise it as operational complexity. In the world of stream processing there's a vast number of things that can go wrong, and because of the amount of traffic running through the systems it can take a lot of time and manual effort to recover in those events. First you need to resolve the immediate cause of the fault (networking issues, disk failures, crashes, etc) and then you need to account for all the pending traffic that built up since the problem occurred.

Most of that risk and recovery effort can actually be avoided by building applications that have minimal (or zero) state, solid transaction-based delivery guarantees, and responsive back-pressure. Not only does that reduce the number of things that can fail, but rather than manually chasing the lost data during the outage you can actually just rely on the services to come back online and catch up automatically.

However, building applications that behave this way is a significant effort and streaming pipelines are often made up of lots of unique services each responsible for solving specific problems. What I did is build a generic, configuration driven service that could solve the vast majority of our streaming problems in a stateless way, whilst also honouring the transaction-based delivery guarantees that I mentioned. It didn't solve every streaming problem that we had, but it massively consolidated our architecture and allowed us to then focus on more interesting problems.

You know a lot about streaming! How did you learn so much about streaming architectures? Any resources or tips you can share with the readers?

For me it was all from practical experience as I got to work in an environment that encouraged experimentation. We were able to really push systems to their limits and see where the cracks form, which was a bit of a crash course in all the things that can go wrong and how to prevent them.

There's a growing library of material describing how to get value out of streaming systems and the justifications for building them, Confluent in particular offers a huge amount of resources, which is where I'd recommend people start. However, if you want to build up a solid grasp on the operations and resiliency side I'd recommend seeking out some practical experience as the documentation hasn't caught up quite yet, just make sure you're not far from the people responsible for running and maintaining these systems.

Are there any overarching goals of Benthos that drive design or implementation?

Statelessness is a key architectural feature of Benthos. Data streaming services are almost always operating on a scale where horizontal distribution is a necessity, and services with distributed state have a huge operational cost, both in initial setup and especially when things go wrong.
The core philosophy of Benthos is to eliminate that cost by solving data streaming problems in a stateless way, where the complex stateful problems are solved by leveraging stateful services you probably already have like databases, caches, etc.

If so, what trade-offs have been made in Benthos as a consequence of these goals?

There are some advanced stream processing use cases that Benthos just doesn't solve and likely never will. Knowing when to stop takes some discipline as there's always the temptation to add just a little bit of distributed state in order to solve something cool.

What is the best way for a new developer to contribute to Benthos?

Play with it, we have plenty of issues to dig into (https://github.com/Jeffail/benthos/issues) but the fastest way to contribute is to use it for something you find interesting, identify some documentation, fields, or customization that you think would be useful or would've made things easier, and add them. Good documentation is critical in a project like this, so new examples, use cases, and the like are always sought after.

P.S: We are now on Twitter as @ConsoleWeekly. Follow and tag us in your tweets!

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at console.substack@gmail.com or mention us @ConsoleWeekly!

Console