Console

Noise, Diffsitter, and Scylla

Sponsorship

If you, or someone you know, is interested in sponsoring the newsletter, please reach out at console.substack@gmail.com


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Why not spread the word by forwarding Console to your friends and colleagues?


Projects

scylla

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB.

language: C++, stars: 7107, watchers: 345, forks: 832, issues: 1772

last commit: August 29, 2021, first commit: August 9, 2014

https://twitter.com/ScyllaDB

noise

noise is an opinionated, easy-to-use P2P network stack for decentralized applications, and cryptographic protocols written in Go.

language: Go, stars: 1643, watchers: 74, forks: 188, issues: 22

last commit: March 30, 2020, first commit: June 13, 2018

https://twitter.com/PerlinNetwork

diffsitter

Diffsitter is a tree-sitter based AST difftool to get meaningful semantic diffs.

language: Rust, stars: 881, watchers: 9, forks: 15, issues: 12

last commit: September 13, 2021, first commit: June 26, 2020

https://afnan.io/


🎉 Shout out to Dragos for becoming a Gold Tier Patreon subscriber this week! Thank you for your support Dragos! 🎉

Console is powered by donations. We use your donations to grow the newsletter via advertisement. If you’d like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation 😊

Donate Today


An Interview With Botond Dénes of Scylla

Hey Botond! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like?

I’m from a small village in Northern Transylvania, Romania. I was fascinated by computers as a child, although I mostly just played games on them. I was introduced to programming in high-school and liked it so much that I decided I want to become a programmer.

I started my career as a web-programmer but I switched to C++ as soon as I could. I stuck to it ever since (with a minor Python de-tour).

I like to work on Linux, do backend work, and to be exposed to a wide variety of challenges, so ScyllaDB is a perfect fit for me.

What is one app on your phone that you can’t live without that you think others should know about?

I mostly use my phone for phone stuff: communicating (calls, text, chat, email). Lately I started listening to YouTube videos while doing things. I find this a very useful way to turn something dull like mowing the lawn into an opportunity to learn something or just be entertained.

Have you looked into any podcasts for this?

Not yet, so far I have only subscribed to podcast versions of some YouTube channels that I already listen to. I have YouTube Premium, so I can listen to videos in the background, and hence I didn't have the motivation to look for podcasts yet.

If you could teach every 12 year old in the world one thing, what would it be and why?

Two things come to mind that I think are sorely missing from general education: how to handle money and the local equivalent of the highway code. Even if you never drive a car, knowing how the roads “work” makes traveling that much safer for you.

What are you currently learning?

C++. I feel like even after 10 years of using it there is still much to learn about it. Especially with the new standard release cadence, which we usually adopt as soon as mainstream compilers start supporting it.

What have you been listening to lately?

I have a broad range of interests: history, cars, economy, science, space, gaming, etc. I try to give everything a slice of my limited listening/free time.

Are there any overarching goals of Scylla that drive design or implementation?

Make efficient use of modern multi-core CPUs and fast SSDs. This is what dictated the thread-per-core architecture of Scylla as well as the exclusive use of asynchronous I/O and subsequently cooperative multitasking. All of this is provided by the open-source framework Scylla is built on: seastar.

Another important goal is self-tuning, to this end Scylla employs classic control theory to self-tune things like the scheduling and I/O priority of background tasks vs. foreground tasks.

What trade-offs have been made in Scylla as a consequence of these goals?

While I don’t feel we had to make any big trade-offs, for sure we have given up comfort when opting to write Scylla in C++ using seastar’s very awkward continuation API. C++ just gained native coroutine support, so things are getting considerably better on this front.

Are there any competitors or projects similar to Scylla? If so, what were they lacking that made you consider building something new?

Our main competitor is Apache Cassandra. Scylla was in fact started as a rewrite of Apache Cassandra in C++, with the goal to provide a vastly faster and more efficient version of it, which can also scale vertically to huge machines, as well as requiring little to no tuning cycles.

At what point did Scylla stop becoming a rewrite of Cassandra and start to become its own project? I'm guessing there was a point in time when a decision was made to start providing different behaviors than Cassandra supports, so I'm curious about that point in time and what that behavior was. Or, would you still consider Scylla a rewrite?

Although started as a rewrite, providing a faster and more efficient Apache Cassandra was gradually shifted out as a goal as we approached feature parity and we started aiming higher. We developed Alternator, a DynamoDB Compatible API to provide a more performant and lower cost alternative to DynamoDB users, we developed our own CDC API because we think the one Apache Cassandra has is cumbersome to use and we are working on making cluster operations strongly consistent by making use of RAFT, just to mention a few examples.

What was the most surprising thing you learned while working on Scylla?

Just how slow it can be to compile complex C++ code. I think when looking at compile times, Scylla is very competitive with notoriously slow to compile monsters like Chromium and Firefox. This is “thanks” to our heavy use of templates and inlining to maximise performance.

What is your typical approach to debugging issues filed in the Scylla repo?

Usually somebody first triages the issue, to make sure it is a legit bug indeed. Then it is picked up by the team-lead of the relevant team and it is assigned to an engineer. But the process is not written in stone. Sometimes engineers jump on issues that are found in the code they wrote on their own initiative.

How do you balance your work on open-source with your day job and other responsibilities?

Working on the Scylla open-source project is my day job, which is a dream come true for me. I occasionally work on proprietary projects as well (like Scylla-enterprise) but the vast majority of my time is spent on Scylla open-source.

What is the best way for a new developer to contribute to Scylla?

You can look for issues on the github issue tracker labeled with the label n00b. Be sure to check out CONTRIBUTING.md first and of course familiarizing yourself with the basics doesn’t hurt either.

What motivates you to continue contributing to Scylla?

Databases are in an ever greater demand thanks to the ever increasing amount and value of data. It is great to work on something that is such a critical piece of the infrastructure of almost every company. I feel like I’m doing something meaningful. Also contributing to ScyllaDB is simply a lot of fun for me, I love the kind of challenges I’m facing, I’m never bored.

Console

Elvish, Not Only Fans, and Public APIs

Sponsorship

What is Streamr? 

The Streamr Network is being built to provide a decentralized real-time messaging protocol for Web 3.0. The Network is a decentralized, topic-based publish-subscribe messaging system. When building on Streamr, dApp developers no longer have to rely on centralized services for message transport. The network serves as a seamless plug-and-play solution that connects to the data economy. 

Participate in the Streamr Brubeck Testnets and earn rewards

Over 4000 nodes are participating in the Streamr Network globally in its current testnet phase, Brubeck, and you can get involved too! Contribute your idle bandwidth and run a node in the Streamr Brubeck Testnets to earn rewards. There are 2M DATA tokens up for grabs!

If you want to participate and earn rewards in DATA, visit this link to learn how to get started. Streamr Brubeck Testnet 2 is going live on Thursday, September the 16th. There will be a new version of the Streamr Broker node software available ahead of Testnet 2 so to stay up to date with the latest developments on the Streamr Network Brubeck Testnets, follow Streamr on Twitter and Discord.


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Refer 10 friends to Console and we’ll donate $100 to an open-source project of your choice!

Share Console


Projects

public-apis

public-apis is a collective list of free APIs for use in software and web development.

stars: 156157, watchers: 3473, forks: 17715, issues: 42

last commit: September 11, 2021, first commit: March 20, 2016

elvish

Elvish is an expressive programming language and a versatile interactive shell, combined into one seamless package.

language: Go, stars: 4458, watchers: 114, forks: 260, issues: 258

last commit: September 11, 2021, first commit: June 14, 2013

https://elv.sh/

not-only-fans

not-only-fans is an open source, self-hosted digital content subscription platform like “onlyfans.com” with cryptocurrency payment.

language: PHP, stars: 189, watchers: 4, forks: 24, issues: 0

last commit: August 28, 2021, first commit: August 25, 2021

https://twitter.com/easychen


🎉 Shout out to Gwendolyn for becoming a Gold Tier Patreon subscriber this week! Thank you for your support Gwendolyn! 🎉

Console is powered by donations. We use your funds to grow the newsletter via advertisement. If you’d like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation 😊

Donate Today


An Interview With Easy Chen of Not Only Fans

Hey Easy! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like?

我之前在新浪公司(它运营着微博)做过工程师和产品经理,后来回重庆(一个相当赛博朋克的中国城市)做独立开发和开源。我在初中的时候就接触到了BASIC,后来学了C语言,并一心想用它编写游戏。但到大学以后遇到互联网,就「移情别恋」到Web开发了。我最熟悉的语言是 PHP 和 JavaScript,喜欢的框架是 React 和 Laravel。Vue3,Go 和 Rust 我觉得也很不错,已经在学习清单上了。

I previously worked as an engineer and product manager at Sina (which runs weibo.com) before returning to Chongqing (a rather cyberpunk Chinese city 

 ) as an independent developer. I was introduced to BASIC in middle school, then learned C and had my heart set on coding games in it. But when I encountered the Internet in college, I "moved on" to web development. The languages I'm most familiar with are PHP and JavaScript, and the frameworks I like are React and Laravel. Vue3, Go and Rust, which I think are great, are on my list.

Why was Not Only Fans started?

最初是因为大概一两年前中国的内容平台修改版权协议引发创作者不满,我觉得他们应该有一个自己架设的、用于和自己粉丝交流的网站。Mastodon 是一个很不错的项目,但没有专栏和投稿功能,另外在本地化和支付方面也不是很适合中国用户。于是我决定自己用业余时间写一个更符合需求的平台,但后来发现工作量比我想象的要大,一直断断续续的写着。直到最近读到了Only Fans的报道,觉得类似的事情还会不断出现,于是赶工完成了一个早期版本,并开源了出来。

About a year or two ago, China's content platforms revise copyright agreements, sparking discontent among creators. I thought they should have an own website they could set up to communicate with their followers.

Mastodon was a great project, but it didn't have a column or submission feature, and it wasn't well suited for Chinese users in terms of localization and payment. So I decided to write a platform more in line with the needs in my spare time, but then I realized that the workload was bigger than I thought and I kept writing it on and off. Until recently I read about Only Fans and thought something like this would keep coming up, so I rushed to finish an early version with cryptocurrency payments and open sourced it.

Where did the name for Not Only Fans come from?

Not Only Fans 的名字受启发于 Not Hotdog,只是为了好玩。作为一个Self-Hosted的平台,它的名字并没有太多机会面对最终用户,因为搭建者会换成自己的名称和品牌。

The name Not Only Fans was inspired by Not Hotdog ( 

 ), just for fun. As a self-hosted platform, its name doesn't have much chance to face the end user, as the builder will change to its own name and brand after hosting.

Are there any overarching goals of Not Only Fans that drive design or implementation?

NOF关注的是所有创作者。平台的崛起,一方面为个人创作者提供了流量,另一方面又拿走了控制权。NOF并不想也无法取代平台,它只是想提供另一种选择,以及至少为创作者和平台之间可能的博弈增加一枚砝码。长远来讲,我希望NOF在它自己的领域最终能成为类似WordPress的产品。坦白说,我并不清楚它所面向的需求是否足够强烈,因为Self-hosted和加密货币都是不低的门槛。

NOF is concerned with all creators. The rise of big platforms, on the one hand, provides traffic for individual creators, and on the other hand, takes away the control. NOF doesn't want to and can't replace platforms, it just wants to provide an alternative, and at least one more weight to the possible game between creators and platforms. In the long run, I hope that NOF will eventually become a WordPress-like product in its own domain. Frankly, I'm not sure if the demand it's geared towards is strong enough, as self-hosted and cryptocurrency are both not a low bar.

What was the most surprising thing you learned while working on Not Only Fans?

大概是我这两天发现的 —— 我们的用户很可能不需要智能合约分账,而需要一个简单的支持更多加密货币的支付界面。

Probably what I discovered in the last couple of days - our users probably don't need smart contracts, but rather a simple payment interface that supports more cryptocurrencies.

Is Not Only Fans intended to eventually be monetized if it isn’t monetized already?

我希望NOF能像WordPress一样保持开源和免费,但也许可以通过提供 Cloud hosting 来获取一部分收益支持项目发展。我暂时只能以 side project 的方式来维护 NOF,如果能有收益的话,我就可以分出更多精力用在它上边。

I hope NOF will remain open source and free like WordPress, but maybe I can get some revenue to support the project by providing cloud hosting. I can only maintain NOF as a side project for now, but if I can get some revenue, I can spare more time for it.

Are there any other projects besides Not Only Fans that you’re working on?

我现在的主要精力用在开发 windmark.pro 上,它是一个用 markdown 和 tailwind 「写」视频的工具。我偶尔也做一点设计,比如我刚刚开源了HTTP状态码T恤( https://github.com/easychen/http-t-shirts )。

I'm working on windmark.pro recently, it's a tool for "writing" videos in markdown and tailwind. I also occasionally do a little design work, like the HTTP status code t-shirt I just open sourced (https://github.com/easychen/http-t-shirts).

What is one question you would like to ask another open-source developer that I didn’t ask you?

我想问问开源项目的开发者们是如何面向全球宣传产品和寻找合作伙伴的,这个问题让身在中国的我很困扰。因此也非常感谢 console 提供和大家交流的机会。

I'd like to ask developers of open source projects how they promote their products and find partners globally, a question that bothers me in China. And so thanks a lot to Console for the opportunity to talk to you all.

I've been curious about this as well myself.  I would like to get more Chinese readers for the newsletter, but for some reason that's been more difficult for me than I would have expected. I don't think the language barrier fully explains this phenomenon.  But, also, as far as I can tell, Asia and the West tend to hang out in different places on the Internet.  For example, I recently checked out v2ex, which I'd never heard of before.  Anyway, because of this, I have no idea where to go looking for more Chinese readers, and I'm sure you face the same problem with your software.

It's true that habits are different, newsletters are less popular among Chinese developers, tech portals and WeChat are more popular. So Chinese developers usually post technical articles to multiple sites after writing them, and even have apps and browser plugins developed for this purpose (like wechatsync and artipub) The Console interview is quite interesting, maybe it could be posted to v2ex.

Console

Kind, Clockwise, and Drago

Sponsorship

Lustre

Find Amazon overwhelming? This AI-powered shopping tool called Lustre does extensive product research for you—from reviews to prices—so you can always get the best product for your money. Check out Lustre here. It’s 100% free!


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Refer 10 friends to Console and we’ll donate $100 to an open-source project of your choice!

Share Console


Projects

AttackSurfaceAnalyzer

Attack Surface Analyzer is a Microsoft developed open source security tool that analyzes the attack surface of a target system and reports on potential security vulnerabilities introduced during the installation of software or system misconfiguration.

language: C#, stars: 2107, watchers: 65, forks: 220, issues: 36

last commit: August 04, 2021, first commit: February 15, 2019

https://twitter.com/OpenAtMicrosoft

Kind

Kind is a minimal, efficient and practical proof and programming language.

language: Kind, stars: 1950, watchers: 70, forks: 79, issues: 23

last commit: September 03, 2021, first commit: February 19, 2021

http://uwu.tech/

drago

Drago allows you to securely connect anything with WireGuard and manage all your networks from a single place.

language: Go, stars: 687, watchers: 24, forks: 26, issues: 21

last commit: August 18, 2021, first commit: January 08, 2020

https://www.seashell.sh/

clockwise

Clockwise is a meeting cost calculator designed to encourage more efficient meetings.

language: Go, stars: 434, watchers: 11, forks: 6, issues: 7

last commit: August 08, 2021, first commit: July 27, 2021

https://www.linkedin.com/in/alexander-perlman-a396a654/


Console is powered by donations. We use your funds to grow the newsletter via advertisement. If you’d like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation 😊

Donate Today


An Interview With Rígille Menezes of Kind

Hey Rígille! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, and what languages or frameworks do you like?

I’m a math student from Brazil. I learned python when I was young but I only really invested in learning programming after getting discouraged by the prospects of academic career. I got a job as a C++ developer in a startup that was building a trading platform. Most of my general programming experience comes from my year there. Now I’m working at uwu tech. My favorite languages are C and Kind.

What about an academic career discouraged you?

Academic work seems very restrictive for me. Industry has its restrictions too, but I see them as rooted in the demands of the public. Those demands act as a standard and objective criteria of success. Academia on the other hand feels more disconnected and subjective.

How did you find out about uwu tech and what made you decide to join?

I found out about uwu tech because of a friend of mine, Vitor Chiarelli, who was from my class. He told me it was a company devoted to developing applications with that new proof language. Looking back it was such a coincidence. I had become interested in formal methods after reading “How to write a 21st century proof” by Leslie Lamport not long before that. At the time, I was addicted to the “Software Foundations” series, but I hadn’t told him about that. Sometimes things just work out.

What is one app on your phone that you can’t live without that you think others should know about?

MessagEase, for sure. It's a keyboard app. Qwerty wasn't designed to be so small and because of that people are slow to type and make mistakes. The most popular solution, kind of crazy if you think about it, is to make a program monitor and guess what you meant to type. Why not design a keyboard that doesn't need to be big? That's MessagEase. The transition took me one week years ago. Now I can type fast, correctly, without looking, and even use vim on my phone easily :)

If you could teach every 12 year old in the world one thing, what would it be and why?

Lojban of course!

Just kidding. I wouldn’t really make every 12 year old learn something. No single teaching could fit everyone’s contexts and preferences, even more so considering that learning something makes you incur an opportunity cost that might be too big. That said, non-violent communication is something I could teach that’s most likely to be useful in general.

What are you currently learning?

I'm mostly learning and researching about compilers on the side, working through Andrew Appel's books. I think Kind could one day be one of the fastest languages out there but it's necessary to target a lower level language to one day seize this potential. My reasoning is that more optimizations become possible when the compiler knows more about the behavior of the program. Most compilers have to guess what the program is about, but not Kind's. The user could tell us anything he wants about his program using the type system.

Also some mathematically inclined people showed up in the community and they want to implement real numbers in Kind. I'm researching existing implementations to give them guidance.

Do you have an order you recommend people read Andrew’s books in?

To learn about compilers I recommend starting with “Modern compiler implementation in ML”, there are also C and Java versions to choose from. “Compiling with continuations” talks about a way of representing control flow that’s especially useful for functional language compilation. To learn about formal methods I recommend the “Software foundations” series and then “Program logics for certified compilers” if you’re feeling brave.

How do you separate good project ideas from bad ones?

The best way is to try them out and see how it goes. Try to realize as soon as possible if it's bad, if it is, drop it. I'm still getting used to this, and still impressed by how easily Victor throws away bad projects. Everything looked very different one year ago, now it's much better.

I also believe you can find out which projects are bad faster if you first develop and reason about a specification be it formal or informal. What's the state of the system? How does the state change? What are the invariants? Those questions direct you to the most important aspects of your project, even if it has nothing to do with software development.

Who, or what was the biggest inspiration for Kind?

Well I can't speak for Victor here, but Kind is only possible because Aaron Stump and Peng Fu developed the theory of self types in 2014.
The link to the original paper is this one. I must admit I have a hard time reading it. Even though I learned how to program in some dependently typed languages I don’t have a type theory background and it isn’t easy to search for the meaning of a notation. What works surprisingly well is to start at the examples and go back from there.

Are there any overarching goals of Kind that drive design or implementation?

Yes there are exactly two goals: keep the implementation simple and make developers more productive. All other aspects originate from these goals.

It's hard to be as expressive as a mathematician would like and keep a simple implementation. The things you can define using set theory are wild. Maybe too wild. You might have heard Banach and Tarski proved you can divide a sphere in disjoint parts and rebuild them as two identical spheres without ever stretching anything. And they really proved it. The argument is sound and the logic is consistent as far as we know. But if I presented to you a point of the sphere, could you calculate whether it goes on the first sphere or on the second sphere? You can't! There's no such algorithm. So is this even real? I don't know… Anyway we would like to make Kind's logic more expressive to better serve e.g. mathematicians but we don't know how to do that yet.

Also to make the language easy to use we allowed programmers to write infinite loops. That means circular arguments are also accepted by default. Checking that your argument isn't circular and that your program terminates will be optional.

What is the most challenging problem that’s been solved in Kind, so far?

I wasn't around when it was done. But from a purely technical perspective the type checker for the core of the language was the most challenging problem. It's just 700 lines of Javascript but it took many months of research, trial, and error to get there.

From a more human perspective the biggest challenge was to package all this in a way that’s useful and friendly for everyone. I’ve been leaning more of the technical and theoretical aspects here because I find it all really interesting. But you don’t need to know any of this to use Kind. Indeed at uwu tech, about 6 months ago, 5 people with little or no programming experience were hired. After 3 months they were already building a multiplayer game with rollback netcode, reactive user interface, and calculating state from a stream of events.

I could teach Kind to my non-programmer friends and it would be one of the fastest ways to build these kinds of web apps, that’s beautiful. We still need better documentation though.

What is your typical approach to debugging issues filed in the Kind repo?

First I try to reproduce the error with the smallest example possible and then debug using logs. Since functions are pure by default you can narrow down the undesired behavior looking at what parameters were different when something went wrong.

How is Kind intended to eventually be monetized?

A programming language is hard to monetize. What you can do is to build a product with that language and monetize that. I mentioned uwu tech, it’s a company created by Victor — who’s the one that actually started Kind much earlier — that’s building products using Kind. We’re currently building games and doing some cryptocurrency-related work.

There’s also an idea floating around to create a trustless market for theorems and programs. Imagine you could write a unit test, attach some money to a smart contract and get a program. You wouldn’t need to review the program for correctness because it would be proven to pass your test in all cases. We have most components needed for that, we just don’t know how to split the reward among many contributors in a way that incentivizes everyone to share their code freely. A winner-take-all model would incentivize people to withhold their partial results.

Can you say more about the cryptocurrency project?

Unfortunately I can’t give more details about the kind of cryptocurrency work we’re doing yet. But it shouldn’t be long before it becomes public.

How do you balance your work on open-source with your day job and other responsibilities?

I’m currently working on Kind full-time :D

I also have to worry about college and taking care of my kitty, but there’s plenty of time for that if I stick to a routine.

What is the best way for a new developer to contribute to Kind?

Chat with us at telegram, ask questions and share your ideas :)

Console

Zero, ClickHouse, and Ocean

Sponsorship

If you, or someone you know, is interested in sponsoring the newsletter, please reach out at console.substack@gmail.com


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Refer 10 friends to Console and we’ll donate $100 to your favorite open-source project!

Share Console


Projects

ClickHouse

ClickHouse is an open-source column-oriented database management system that allows generating analytical data reports in real time. Open-sourced by Yandex.

language: C++, stars: 18831, watchers: 663, forks: 3667, issues: 1634

last commit: August 27, 2021, first commit: Dec 1, 2008

https://twitter.com/ClickHouseDB

ocean

Ocean is a work in progress C compiler from scratch.

language: C, stars: 146, watchers: 4, forks: 2, issues: 3

last commit: August 25, 2021, first commit: June 29, 2021

https://r1chard.nl/

zero

Zero is an open source tool which makes it quick and easy for startup technical founders and developers to build everything they need to launch and grow high-quality SaaS applications faster and more cost-effectively.

language: Go, stars: 83, watchers: 15, forks: 20, issues: 29

last commit: August 25, 2021, first commit: August 21, 2019

https://twitter.com/commitdev


Have a project you’d like to see featured in Console? Drop us a line at console.substack@gmail.com


An Interview With Bill Monkman of Zero

Hey Bill! Thanks for joining us! Let’s start with your background. Where have you worked in the past, how did you learn how to program, and what languages or frameworks do you like?

I taught myself to code in high school, mostly making DOS games in C++, and got my first real job writing Perl for a small domain name registrar in 2000. I realized pretty early that I liked the startup life, and since then I’ve worked mostly at startups of varying sizes, including writing auth systems for games like America’s Army: Operations and Halo, building SaaS apps in PHP for shipping companies, and spending a big chunk of time at Hootsuite. I started there as one of the first employees and over 8 years learned a lot by building and leading the creation of much of the backend and infrastructure that allowed us to scale to tens of millions of users. I needed a new challenge after that time and joined Bench Accounting where I helped improve their architecture and set them up to be able to scale. In 2019, some old colleagues and I created commit.dev to try to give startup engineers more agency over their careers.

I mostly work on the backend or infrastructure and have spent a lot of time focusing on distributed systems and building and scaling startups.

Who or what are your biggest influences as a developer?

Anyone who is passionate about writing code, solving problems, being creative, and learning. I’ve been lucky enough to meet and have some great chats with people like Armon Dadgar and Kelsey Hightower who I think are great role models in the world of open source and DevOps.

What's an opinion you have that most people don't agree with?

I fundamentally dislike most of the trend of “serverless” applications in the industry. I think it has its place and can be very useful as glue in cloud platforms but has a lot of shortcomings that I often see glossed over or misunderstood.

What is one app on your phone that you can’t live without that you think others should know about?

1Password

If you could teach every 12 year old in the world one thing, what would it be and why?

Critical thinking. A desire to dig deeper and try to understand things more fully is a common trait between a lot of people I respect most.

If I gave you $10 million to invest in one thing right now, where would you put it?

Probably something boring and safe.

What are you currently learning?

Rust, archery

What have you been listening to lately?

Phoebe Bridgers, Bo Burnham - Inside, the In The Heights soundtrack, Penelope Scott, loving the new Big Red Machine singles that have been coming out.

Why was Zero started?

I’ve worked at startups for a long time and have seen the same foundational code being written over and over again - infrastructure, user management/auth, logging/metrics, CI/CD, notifications, etc. and during our work with Commit, we have been exposed to a ton more early-stage startups facing these same challenges. Over the course of helping them, we built up a big repository of code to streamline all these things and built a CLI and supporting tools to make it really easy to reuse.

Where did the name for Zero come from?

We were thinking of it as all the things that happen before commit number one on your product - all the initial setup and stuff. So it became Commit Zero, and then just Zero.

Who, or what was the biggest inspiration for Zero?

Not having to write a bunch of code yourself before you can get to the point where you’re actually building out your application!

Are there any overarching goals of Zero that drive design or implementation?

For each feature we are adding, we weigh a bunch of different tech before we decide what to add into the Zero ecosystem 

  • Quality: Is it a great tool that will let a project start small but scale big? 

  • Simplicity: Will we be able to fully automate its setup in a way that will be easy to operate and maintain?

  • Price: Ideally open source, but ideally as low-cost as possible

If so, what trade-offs have been made in Zero as a consequence of these goals?

One example might be user management/authentication - I like using Auth0 and I think they’ve done a lot of things right, but I have a hard time recommending it to people if their product has the potential to grow to many users (for example, something with a freemium model) because of the cost. We ended up integrating Ory Kratos and Oathkeeper instead despite some added complexity, though also leaving the door open to use just part of the Ory stack and integrating with external tools like Auth0.

What is the most challenging problem that’s been solved in Zero, so far?

In the binary itself, probably the module system where we can pull in various modules from GitHub or other sources, read a definition file in them that specifies what parameters are required and how to prompt the user for them, and everything required to render out a project and create real-world systems out of them. For example, rendering the AWS infrastructure module, checking it into your GitHub, and then applying all the terraform inside to actually create your infrastructure in your AWS account.

In the generated code/ecosystem, probably just all the “glue” stuff we have had to write to automate everything in a way where it all works together, and effectively testing for regressions (which we’re still working on.)

Are there any competitors or projects similar to Zero? If so, what were they lacking that made you consider building something new?

There are some things like Heroku or Render that are near the same space, but they are trying to get you on to their platform and locked in, and we’ve seen a lot of issues with people using Heroku. It’s fine for a tiny project and makes things easy to get started, but simply doesn’t scale past a certain point, at which point you’re in for a huge migration headache. As far as I know, there’s not anything that is doing what we’re doing, probably because people realized like we did that there’s no way to make money from it since it’s fully open source and creating resources for you inside your cloud provider account.

What is your typical approach to debugging issues filed in the Zero repo?

We eat our own dog food with Zero by using it for all of our other software endeavours within Commit - we have a number of systems running in production that were fully built with Zero and that has been a great way to get feedback, debug issues, and develop new features.

What is the release process like for Zero?

Zero is actually a whole constellation of repositories and applications, including the CLI binary which eases the setup process, but also all the various modules that build your frontend, backend, infrastructure, static site, etc. all in various languages.

As much as possible we have CI pipelines set up with tests, linting and validation, changelog management, vulnerability scanning, documentation generation, etc. set up that helps make it easier on us when dealing with a bunch of different repos.

From the point of view of a user of Zero, the release process for your generated system is a pretty standard CI pipeline using either GitHub Actions or CircleCI that will kick off tests,  build, then deploy to staging for every merge to the main branch, then a manual approval step before deploying the same artifact to production.

Is Zero intended to eventually be monetized if it isn’t monetized already?

No, as I mentioned above there’s not really any way to make money with the design we are going for since we are creating everything within your accounts.

We do see a tremendous benefit to Commit, though, by creating a tool that helps startups, helping startup engineers build and grow their skills, and building a thriving community around Zero.

How do you balance your work on open-source with your day job and other responsibilities?

Commit is dedicated to the idea of open source, and we open up as much as possible of not just our code but also our processes and learnings and try to always work out loud. There are some great examples of this on Commit’s blog.

Do you think any of your projects do more harm than good?

Commit helps give startup-minded engineers new ways to grow their careers, Zero helps them build things better and faster, and build their skill set while they are at it. As far as I’m concerned it’s certainly a net positive for our profession. I can’t say for sure that people aren’t using it to build things that do more harm than good, but I certainly hope they aren’t! Be cool, people!

What is the best way for a new developer to contribute to Zero?

Check out the repos or the ZenHub board we use to track everything, look for the “good first issue” labels, join our community Slack. We have the CLI tool in Go, backend services in Go and Node, a frontend in React and plenty of DevOps stuff in the AWS stack module, so there’s something for almost everyone!

If you plan to continue developing Zero, where do you see the project heading next?

So far there are a handful of startups who have used Zero to set up their production systems but right now we’re looking to onboard a bunch more and gather feedback, then fix any issues and build features based on that feedback.

What motivates you to continue contributing to Zero?

It’s solving a lot of problems that I see startups constantly facing, and doing it using tools I have personally used and seen be successful in not just starting a new project but also scaling that project up to a huge degree. 

It’s also an incredibly powerful tool to help developers grow their skill sets by giving them solid, stable foundations to build upon. One great example of this is Kubernetes - I think if set up properly it can be an amazing tool and ecosystem to build SaaS apps upon and there are a ton of developers out there who want to get more experience with it but just don’t know where to start or are wary of the potential complexity, learning curve, etc. but with Zero we are able to quickly give them a well-organized cluster, defined fully in code, with examples of how to use it, deploy to it and all sorts of information and learning resources to help them become experts.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

“Chop wood, carry water.” - Find a project you like, look at their backlog, join their community chat, and just pick something nice and small to get used to their process and get introduced to their community. Don’t worry about working on some huge, impactful feature. Start small, get involved, and focus on learning.

Console

Airyx, Darts, and Topfew

Sponsorship

Linx

Build APIs and integrations faster. Designed for developers, Linx is a general-purpose low-code platform with similar power and flexibility as hand-coding. Try Linx today!


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Refer 10 friends to Console and we’ll donate $100 to an open-source project of your choice!

Share Console


Projects

darts

Darts is a python library for easy manipulation and forecasting of time series.

language: Python, stars: 2267, watchers: 38, forks: 171, issues: 23

last commit: August 19, 2021, first commit: September 13, 2018

https://twitter.com/unit8co

airyx

Airyx is a BSD-based OS project that aims to provide an experience like and some compatibility with macOS.

language: Objective-C, stars: 1042, watchers: 37, forks: 23, issues: 24

last commit: August 18, 2021, first commit: January 31, 2021

https://twitter.com/airyxos

snaprecovery

Snaprecovery allows you to recover old Snaps that have “disappeared” from Snapchat.

language: Shell, stars: 369, watchers: 4, forks: 14, issues: 1

last commit: August 18, 2021, first commit: January 02, 2021

https://twitter.com/sidheart

topfew

Topfew finds the fields (or combinations of fields) which appear most often in a stream of records.

language: Go, stars: 72, watchers: 4, forks: 2, issues: 0

last commit: May 25, 2021, first commit: May 19, 2020

https://twitter.com/timbray


An Interview With Zoe Knox of Airyx

Hey Zoe! Thanks for joining us! Let’s start with your background. Where have you worked in the past, how did you learn how to program, and what languages or frameworks do you like?

I grew up at a time when personal computers were just becoming a big thing and I was always fascinated by them! I was also lucky enough to have parents who supported that interest. I taught myself to program in BASIC, then assembly language when I was about 12, and it just went from there… Since then I have worked at Bell Northern Research, Corel, e-smith, Mitel, helped run a small ISP during the shift to broadband, worked for myself and a handful of startups, and am now head of engineering at The OpenNMS Group.

I mostly program in C/C++, Objective-C/Objective-C++, Java, Javascript, shell scripts and occasionally assembly language. I’ve dabbled in other languages (perl, Python, LISP, Rust, Go) but am not very proficient in any of them.

I've always thought it would be interesting to start an ISP.  What was that experience like?  I would imagine it's highly competitive and very capital intensive.

Super competitive. I started there as a network engineer/Unix system admin and became sort of the unofficial manager. I got frustrated because the owner wasn't really interested in growth and I had to push them hard to adopt new technologies and keep up with times. It was a good learning experience though.

What sort of things did you learn?

I learned a lot about the physical connections and how the internet works - circuits and channels and framing and link-level stuff, and about routing protocols and all the plumbing that keeps it running. Computer and network security from the trenches. People management, conflict resolution, influencing. And yeah, lots of general business stuff which served me well later on!

What’s your most controversial programming opinion?

That optimization still matters and Moore’s Law isn’t an excuse for bloat. This probably isn’t as controversial anymore, but programmers for a long time seemed to ignore how much memory and CPU their software needed and just threw hardware at the problem. The number of times I’ve heard statements like “disk is cheap” or “RAM is cheap” is amazing. Now that it’s way harder to pack more and more transistors into a chip and as we’ve moved more into cloud environments where you pay for CPU and RAM usage, I think people are catching on to this idea again.

Why was Airyx started?

It really got started as a “because I can” project after a conversation. Apple had just announced the shift away from Intel architecture, and we were mostly joking about giving Intel MacBooks a longer life by making a compatible OS. But more seriously, I believe in open source and wanted a more transparent alternative to macOS that I could run on arbitrary x86 hardware - one that feels and looks as good, that can run at least some of my Mac applications, and that isn’t trying to make my laptop into a tablet.

Where did the name for Airyx come from?

It was originally called Helium but there are a lot of “Helium” projects and I wanted to avoid confusion. So I wanted a name that had a similar sense of lightness and airyness, of being easy or simple. And it had to be short and memorable. Of course I also had to find something where the domain name was available! After a lot of searching I landed on airyx.

What is the most challenging problem that’s been solved in Airyx, so far (code links to any particularly interesting sections are welcomed)?

One of the more challenging tasks was adapting the Cocoa frameworks from Cocotron to use the GNUstep Objective-C runtime, libobjc2. They were written to use a built-in objc runtime without a lot of modern features. A lot of stepping through stack frames and object files went into figuring out the differences in things like reference counting, implementing exception handlers and support for Tiny Strings. 

https://github.com/mszoek/airyx/commit/e8490a8ac7f6d2974dd29ca77f96d70338e99c04

https://github.com/mszoek/airyx/commit/3218c92deb59d9ea0529f8f77841e77b3b9d6b79

https://github.com/mszoek/airyx/commit/035dafbe49a885d7715b1cf104ffaa92fe1dc11c

I think my linker magic to support ELF Frameworks is also pretty cool. Can I highlight that? :)

https://github.com/mszoek/airyx/commit/2d0acbfc958cd00a16a113401da176bc8dbb43af

Are there any competitors or projects similar to Airyx? If so, what were they lacking that made you consider building something new?

Yeah, absolutely! I don’t think of them as competitors so much as different directions. It’s very rare that one project meets everyone’s needs. Here are a few:

PureDarwin is a usable system based on Apple’s open-source code. It’s very cool and could be used as the basis of Airyx, but I chose FreeBSD for wider hardware support, more ready-to-run tools, and Linux compatibility.

GNUstep implements the NeXTStep APIs which were the basis of Cocoa and has added a lot since, but they’ve stayed very true to the NeXT look and feel and have built essentially a portable overlay onto a Unix-like system. Airyx is a full OS that integrates Cocoa natively and follows more of a modern aesthetic, so they’re fairly different. I probably could have built Airyx from FreeBSD + GNUstep and it would have been a massive fork. That felt like the wrong decision.

helloSystem does a great job of making a simple, usable, and fast system that feels like a Mac. We share a lot of code and ideas but do differ in some key areas.

Darling is an emulation layer for Linux that can execute Darwin binaries. It basically creates a VM environment where you can run stuff that thinks it is on Darwin. I’m very interested in this project for later when we attack binary compatibility! It differs from Airyx in that I see this sort of like Windows Subsystem for Linux - it’s a component of an OS that provides a compatibility layer, rather than the OS itself.

What are the key areas of differentiation between Airyx and helloSystem?

I think mainly that helloSystem is very much built on FreeBSD using the standard package set (so everything is in /usr/local) and that it intends to be very simple and easy to use by following some Mac paradigms but is otherwise FreeBSD. They're interested in some pieces of Airyx but we don't always agree on implementation details. A recent example is how to leverage LaunchServices.framework: https://github.com/mszoek/airyx/issues/37

Meanwhile my goal with Airyx is to build something elegant and simple to use, following a recent macOS aesthetic (more or less), but also to be compatible with it at a source level, eventually a binary level, to have the same user experience in terms of key bindings, consistency of keyboard shortcuts and menus throughout applications, to replace or supplement rc.d with launchd and launchctl and so on. Airyx leaves /usr/local empty for user additions and that was important to me. It uses a modified FreeBSD base OS with a patched compiler/linker and will likely also have kernel patches in the future to support Mach system calls, Mach-O object files, and possibly file systems. I don't think I could achieve the goals of Airyx without breaking a bit with the standard FreeBSD approach, although I know this will be controversial in some circles.

So it means Airyx and helloSystem can share code and collaborate but the two systems will likely always be distinct.

What is the release process like for Airyx?

Slow! It’s mostly just been me working on it until very recently. The first commit was Jan 30 2021. The first PoC was released around April, I think, and the current 0.2.2 PoC was released May 22. I’m aiming to have the substantially updated 0.3.0 build ready for September. You can check out the “Release notes” and Screenshots pages at https://airyx.org for a preview.

What is the best way for a new developer to contribute to Airyx?

Take a look at the GitHub issues list and see if there’s something you can fix. Join the IRC channel (#airyx on Libera) or hit me up on Twitter (@airyxOS) to talk about other ideas. There’s a lot to do and the roadmap is mostly in my head.

I really need developers who know Objective-C and C++ right now, especially folks who know Qt5 and can help with AppKit and UIKit. So if you’re reading this, please get in touch!

Are there any other projects besides Airyx that you’re working on?

I’m very lucky to work for a company making open-source software, so yes! My day job is managing the development of OpenNMS, a fully open-source enterprise network monitoring platform. My role there is different so I’ve been able to get multiple perspectives on OSS development, from the developer view, the community view, and also from the business view of running a successful business around open source. It’s extremely valuable. Also, we’re hiring!

Do you have any other project ideas that you haven’t started?

Oh my god, so many! I wish I had the time and resources to start and manage them all. 

Care to elaborate?

Sure... one idea was to start building regional grassroots mesh networks to provide ubiquitous free wifi to everyone and remove the central control of internet providers.

Another was a site to rate healthcare providers on their service to specific groups like LGBT+ or racialized communities and then allow people to search it to find the closest or best provider for a certain specialty for people like them.

How about a regional book registry and loan service? You publish what you have, others do the same, and people can browse the selection, reserve and check out books from each other.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Just go for it! I’ve learned so much from open-source code, and have met some really cool people. And some jerks - not everyone is great. Be prepared to be challenged and try to leave your ego at home, but also recognize when comments cross the line. If a project feels toxic, pick another one.

Pick a project that speaks to you personally - you’re doing this for free in your spare time so you need to believe in it. Look for the low-hanging fruit in the issues list. They’re often tagged as “good first issue” or something like that, or ask in the project chat/mailing list/whatever how to get going. And there are other ways to contribute as well - documentation, translation, artwork, testing, forum moderation, support… find something you can do and just do it!

Loading more posts…