Console 56

FastAPI, TiddlyWiki, and unfork

Sponsorships

Getro: The best way to give and get introductions

Getro is a B2B SaaS platform that helps independent professional networks — including venture capital funds, accelerators, membership communities, associations, governments, and universities make better introductions for their members, measure the outcomes of their intros, and in some cases, make money doing it! 

Getro is looking for a Lead Backend Engineer, Full Stack Engineers, and a Product Manager to join their team. 

👩‍💻 Remote first company - we’re all remote, since before COVID: high level of flexibility, high level of ownership

❤️ Culture is extremely important - we're driven by our values of compassionate candor, curiosity, committed ownership, inclusivity, openness, and gratitude

🌱 Seed stage company with $1.7M ARR, 17 employees across 7 countries. We are a good fit if you want to influence the direction of the product / company in a very direct way

⏰ Working day is based on EST-CET timezone

🗣️ Working language is English


FastAPI

FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.

language: Python, stars: 31774, watchers: 505, forks: 2224, issues: 891

last commit: May 26, 2021, first commit: December 05, 2018

https://twitter.com/tiangolo

TiddlyWiki5

TiddlyWiki is a non-linear personal web notebook that anyone can use and keep forever, independently of any corporation.

language: JavaScript, stars: 6029, watchers: 250, forks: 984, issues: 1006

last commit: June 04, 2021, first commit: November 22, 2011

https://twitter.com/Jermolene

unfork

unfork(2) is the inverse of fork(2). sort of.

language: C++, stars: 1290, watchers: 22, forks: 35, issues: 0

last commit: May 31, 2021, first commit: October 29, 2019

https://twitter.com/whitequark


An Interview With Sebastián Ramírez of FastAPI

Hey Sebastian! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?

I'm from Colombia. 🇨🇴 I was homeschooled all my life, so I don't have any official degree. I started with sloppy HTML, CSS, and some JavaScript when I was around 15. I kept studying online, reading documentation sites and standards. At some point I took many online courses. That's how I got into Python.

I started working in a startup in Colombia, from there I ended up in the Middle East, and then I joined a spin-off company there in Dubai. After that I moved to Berlin, Germany, to join Explosion, and I worked there until very recently. Now, I quit to dedicate a high percentage of my time to FastAPI, Typer, and my other open source projects while doing some consulting.

I like modern Python for most of the things and TypeScript for some others. I'm a fan of Docker and containers. I've had the chance to try and use many databases and similar tools, and I like to understand how they work and what to use for each use case.

Any favorite databases in particular?

I like them for different things. PostgreSQL for simple systems that benefit from relations. SQLite for small local things. MongoDB has great docs and a nice query language for documents. ElasticSearch has very good performance and features for text and analytics. Redis works great for simple things that fit in memory and/or require raw performance, for example, job queues. And so on with a bunch of others with different use cases... 😅

What was it like quitting a stable job to work on open-source full-time?  How did you know it was the right decision at the time?

I had no idea if it would work or not, but I realized I was not having enough time for all the work needed in open-source, and I would have to change my work structure to be able to handle that.

At the same time, there have been several organizations interested in FastAPI and me in different ways, which gave me a hint that maybe I could make it work. And if it didn't work, in the worst case where I couldn't put as much time into open-source as I want to, I could accept a job offer.

I also have a huge advantage in that my wife has a stable job, and we don't have children.

So in general, the risk was not really that bad.

Who or what are your biggest influences as a developer?

A lot of people and things, I'm sure I'm gonna miss many so I'll just mention a couple that come to mind quickly:

  • Andrew Ng, who "taught" me Machine Learning on the first versions of Coursera and encouraged me to build things to help others

  • Tom Christie, taught me by example how to handle a big and healthy open source community, and to go ahead and solve and build what is needed

What's an opinion you have that most people don't agree with?

Some people agree with this, but governments (issuing visas) and big corporations still don't seem to agree that much.

I think that years of work experience or university degrees are not what define skill set levels.

What’s your most controversial programming opinion?

Not necessarily controversial, but I feel very strongly in favor of a list of features in programming languages, libraries, and tools:

  • Autocompletion

  • Type checks, the squiggly red lines

  • Simplicity

  • Deduplication, the less times writing the same thing in several places, the better, hopefully only once

...and a couple of others.

But in some cases, some people, tools, and technologies have other priorities on top and are okay with sacrificing these for theirs. But it's not really that controversial.

What is one app on your phone that you can’t live without that you think others should know about?

KeePassXC, an open-source password manager for desktop, and KeePass2Android, the mobile version.

If you could dictate that everyone in the world should read one book, what would it be?

It's not necessarily a book, but I would encourage everyone to learn at least the basics of computers and programming.

If you could teach every 12 year old in the world one thing, what would it be and why?

To study on their own. To be curious and follow the path of whatever seems would solve the problems they care about and that would help their communities and society. I think that could help a lot with the world's problems, and it worked more or less for me. And that depends a lot less on where you are born, who your parents are, how much money they (or you) have, where (and if) did you go to University, etc.

If I gave you $10 million to invest in one thing right now, where would you put it?

Into building ways to teach programming and technology to kids around the world, for free or with very low cost. Encouraging them to solve their local problems, help their communities, and society. I'm not sure about the "business model", but I think that could work in some way.

Are you familiar with freeCodeCamp?  Any opinion on them?

I think it's great, but I haven't taken courses there myself (yet), I wish it existed when I was starting. That and other platforms like Khan Academy solve a big part of the problem. The missing piece is probably making these platforms more mainstream. Maybe figuring out ways to make the whole path until people build a real-life product simpler.

What are you currently learning?

Many things!

Recently, several internal parts of how Python works underneath around the type annotations.

Anything in particular?

Particularly PEP 563: https://www.python.org/dev/peps/pep-0563/

What it implies, and how it would have affected some tools if it hadn't been rolled back.

All the context is not really explained anywhere yet, I should probably write a blog post about it.

There's some more information here: https://github.com/samuelcolvin/pydantic/issues/2678 and in a couple of discussions in the Python mailing lists.

I made a simple short summary of what would have been affected here: https://twitter.com/tiangolo/status/1384512019756376064

In short, everything turned out well, even though it involved some painful extra effort from the Python Steering Council.

But coming back to the question, I ended up learning a lot about how Python internally treats these type annotations that I love and use so much. 🤓

What resources do you use to stay up to date on software engineering?

I follow some people on Twitter, and I'm subscribed to several newsletters about technology, Python, and others.

What have you been listening to lately?

A weird combination of music, metal and rock on one side, and electronic music on the other, dubstep, liquid drum and bass, etc.

How do you separate good project ideas from bad ones?

I avoid building new things and new projects a lot. I avoided building FastAPI, Typer, and other tools for quite a while.

First I try to (almost) exhaustively search for a tool that already does what I want, and that solves the problem I care about.

When I don't find something, and no one builds it, I end up building it.

I also try to avoid creating packages that abstract too much, when it would have been equivalently easy to implement directly. As then developers would have to learn the new package instead of learning how things actually work underneath.

Why was FastAPI started?

To solve several problems, or features I wanted to have that benefited my work and the work of the developers I was leading.

It was the best combination I could come up, from a lot of ideas from previous frameworks and tools.

There's a lot more details in the docs about Alternatives, Inspiration and Comparisons: https://fastapi.tiangolo.com/alternatives/

Who, or what was the biggest inspiration for FastAPI?

Several tools inspired several features, I think I couldn't attribute it to a single inspiration.

But if there's one single thing, it would be modern type annotations (also called type hints) in Python. That's what allows autocompletion and type checks in editors. And FastAPI uses them for everything, to do data validation, serialization, documentation, etc. It's all built around these type annotations.

Are there any overarching goals of FastAPI that drive design or implementation?

On the specifics of the design itself, I think that list above of the features I care about are very important.

And on the philosophical side, one of my main personal objectives is to help others be more efficient, build things, and solve problems they care about and that help their communities and society.

If so, what trade-offs have been made in FastAPI  as a consequence of these goals?

I have traded a lot of extra effort on the maintenance side (more of my effort) in exchange for better developer experience. For example, I avoid using **kwargs as much as possible, so people can get autocompletion and type checks, and be more efficient, instead of having to go to the docs all the time.

What is the most challenging problem that’s been solved in FastAPI, so far?

Pydantic itself exchanges ease of use for final developers with internal complexity. So, it's very easy to use, but its internals are more difficult to understand. Adding features to it and learning how it worked, and how to re-use its internals for FastAPI was particularly challenging.

The other thing is the dependency injection system in FastAPI, supporting dependencies with yield. It's designed to be very easy to use for developers, but the internals to achieve that, while supporting async and normal def functions combined in arbitrary "trees" of dependencies was a bit mind-bending when building it. Fortunately the results were great and it was totally worth it.

Do you have any links to code involving the support for yield?

Here are the docs for the dependencies with yield: https://fastapi.tiangolo.com/tutorial/dependencies/dependencies-with-yield/

The technical details (unnecessary for almost everyone) are that it detects if a function is normal or async and then makes it a context manager or an async context manager. Then it uses an AsyncExitStack from the standard library to evaluate those dependencies and run their teardown after sending a response, and after handling any necessary background tasks.

Are there any competitors or projects similar to FastAPI? If so, what were they lacking that made you consider building something new?

I prefer to call them inspirations, because I took ideas from most of them and combined them together. There's a lot more details on Alternatives, Inspiration and Comparisons: https://fastapi.tiangolo.com/alternatives/

What was the most surprising thing you learned while working on FastAPI?

That it wasn't crazy, my ideas were indeed useful, and it was worth building it. I didn't expect that.

What is your typical approach to debugging issues filed in the FastAPI repo?

Most of the issues are questions. Or bugs in the developer's code. Very few are actual issues in FastAPI.

But that takes a lot of time and effort. Fortunately, the awesome FastAPI Experts help a lot with that: https://fastapi.tiangolo.com/fastapi-people/#most-active-users-last-month

What is the release process like for FastAPI?

I tend to release often, when there are new features or changes. I prefer to release several times a day if the features or changes are sufficiently large. That way, if for some reason, in a strange case the last changes broke something for a particular use case (which is not common, but could happen), people could still benefit from the other recent features, fixes, etc.

Is FastAPI intended to eventually be monetized if it isn’t monetized already?

I want to keep FastAPI fully open source and free, with the best documentation and features possible, and helping as many people as I can.

I recently quit my job to dedicate a high percentage of my time to FastAPI, Typer, and the other open source projects, as I wasn't having enough time for them.

I'm currently structuring things to make it sustainable.

For now I'm not building a company, but I'll do some limited consulting for some teams, while dedicating a big chunk of my time to open source.

What is your main source of revenue?

For now it will be these consultancy projects and sponsors. At some point, I would like to figure out a product or service that would be useful for corporates and companies with budget or investments, complementing FastAPI. While keeping FastAPI itself fully free and open source, as that's my main objective.

How do you balance your work on open-source with your day job and other responsibilities?

Up until recently, I quit my job, but for the past 9 months or so, it was mainly my free time and weekends, so that wasn't sustainable.

But now that my focus is on these open source projects it will be easier to handle. And for example, the arrangement with one of the main organizations I'm now working with (I'll announce it soon) involves a high percentage of my working time dedicated to open source.

Do you think any of your projects do more harm than good?

I hope not. My objective is to help people help others.

What is the best way for a new developer to contribute to FastAPI?

It's actually to help others with their issues, which makes them official FastAPI Experts. But there are many other ways to help too: https://fastapi.tiangolo.com/help-fastapi/

If you plan to continue developing FastAPI, where do you see the project heading next?

I have lots of plans for FastAPI, Typer, and others. And even for new projects that will complement them. There's a long list of things I want to work on.

What motivates you to continue contributing to FastAPI?

When I see that I'm helping people build high quality products faster, more efficiently, and solve problems, that motivates me a lot. Especially when I see small teams, solo developers, or teams in developing countries like mine building things quickly and provide value to their communities, that's very exciting for me.

Where do you see software development in-general heading next?

I'm not very good at predicting things except from what I could make happen. So I can imagine software development getting more and more convenient and enjoyable, with more and more people being able to use it to solve all types of problems.

Where do you see open-source heading next?

I think open source is in a strange position in that it results in very high quality products, it's used and depended on a lot, and provides a lot of value. But it's generally not very sustainable. It's not easy to find wealthy open source projects.

That's something we have to work on with companies. Everyone would win, companies would save many expensive developer hours implementing workarounds for open source projects if the original project already supported those features or had those bugs fixed. It would be much faster and efficient to have the maintainer do that than internal developer employees work around that.

I think there's a disconnect between the person that controls and approves the budget in organizations and the open source maintainers. There's something we are failing to understand what to offer that would make it simpler for them.

Maybe it's a support license, meaning that the project will keep maintained just as it is, but with paid certainty for the company.

That is an interesting insight.  I would say there is even a disconnect between the person that controls the budget and developers using the code, not just the maintainers.  Most developers understand how valuable the code is that they're using, but the CFO for example, might not if they’re not technical.

Exactly, I think developers are not against sponsoring open-source in some way, in most cases, they are the ones that make it happen. But only a few realize they could do that. And the CFO probably has a hard time understanding how is it that something that is already "working" and already "free" would need to be maintained and/or paid for in some way.

I guess software looks like a static thing that can be bought and used, like a car or a laptop, but it tends to be an ongoing evolving thing, that has new errors, has to solve errors generated by dependencies, has new features, can have better docs, etc. And that, and the fact that this evolving nature is beneficial for the companies and products, is probably not obvious or intuitive for people in charge of budgets.

That's probably what we, developers, have to help shape by talking to them and making it easier to understand. Or by talking to maintainers and helping them present their request for resources in a way that is acceptable for managers.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Start learning a project you like, read the docs. You will find typos, it's almost certain. That's the first and easiest contribution to make. I wrote a bit about it here: https://dev.to/tiangolo/how-to-start-contributing-to-open-source-1jmg


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Refer 10 friends to Console and we’ll donate $100 to an open-source project of your choice!

Console 55

CloseTab, Hyperbeam, and rpg-cli

Sponsorships

If you, or someone you know, is interested in sponsoring the newsletter, please reach out at console.substack@gmail.com


awesome-privacy

Awesome Privacy is a curated list of services and alternatives that respect your privacy.

stars: 2379, watchers: 58, forks: 101, issues: 45

last commit: May 29, 2021, first commit: December 07, 2020

rpg-cli

rpg-cli is a bare-bones JRPG-inspired terminal game written in Rust. It can work as an alternative to cd where you randomly encounter enemies as you change directories.

language: Rust, stars: 722, watchers: 7, forks: 15, issues: 6

last commit: May 29, 2021, first commit: April 23, 2021

https://twitter.com/folano_

hyperbeam

Hyperbeam is a 1–1 end-to-end encrypted internet pipe powered by Hyperswarm and Noise.

language: JavaScript, stars: 362, watchers: 6, forks: 19, issues: 7

last commit: December 14, 2020, first commit: December 12, 2020

https://twitter.com/mafintosh

CloseTab

CloseTab is a bookmark with a snooze button. Bookmark, buffer and complete your reading list.

language: Vue, stars: 28, watchers: 3, forks: 3, issues: 0

last commit: May 18, 2021, first commit: May 19, 2019

https://twitter.com/vettijoe


An Interview With Joe Lewis of CloseTab

Hey Joe! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?

I’m from Chennai, India. I always had this strange excitement – breaking electronic stuff, peeking inside and putting them together and it ended up being a costly affair. The moment I realized how cheap and equally exciting it was to do the same with computer programs, I knew it was going to stick with me for the rest of my life. This was way back in my high school days. 

I started my career at a startup building a document platform for lawyers back in 2013. Then I moved to Zoho where we are building a collaborative word processor that runs in the browser

I wrote my first non-trivial piece of software in Python. So, it’s always in my favorites list. But over the years I’ve come to prefer languages with stronger guarantees for large codebases. I write Java & Javascript at work and started learning / dabbling with Kotlin & Rust recently.

Who or what are your biggest influences as a developer? 

Thanks for asking. My biggest influence as a developer is Yuvi Panda. We shared the same high school but we didn't interact much while there. His old blog ended up pulling me into exciting corners of programming and pushed me into this weird space: Hacker News.

Hacker News back in those days was like a dream land for me, where suddenly I could find amazing tech content and share ideas with interestingly like-minded people. 

These two certainly had a huge impact on my early thinking.

What’s your most controversial programming opinion?

The front-end ecosystem isn’t really as bad as it’s projected these days. Freedom of choice in picking tools is not a bug. It’s a feature.

If you could teach every 12 year old in the world one thing, what would it be and why?

I would make them aware of the two types of thinking. And persuade them to focus on stuff that makes them do “Type II thinking”.

What are you currently learning?

Dart/Flutter & Kotlin/Compose for an interesting project I’m working on. 

Can you give us a sneak peak into what the idea is?

Here's the fundamental idea:

I recently got to know that there isn't a good enough rich text editing library for iOS/Android yet. Most apps throw in a native rich text view, and save them as HTML to make them render-able across platforms.  HTML isn't a real good data model to store rich text for many different reasons. For instance, it's impossible to write a real-time collaborative editor using HTML as the data model. 

Markdown is a saner alternative. But there's no way to represent complex rich text schema in Markdown. 

I'm writing a library to render rich text editing widgets across all platforms. Embedding apps can choose to keep it simple or to extend with sophisticated elements like @mentions. Imagine ProseMirror but cross-platform. It should come with out of the box support for real-time collaboration. 

It's in a very early phase and might take months to be worthy of sneak peeks :)

What resources do you use to stay up to date on software engineering? 

I subscribe to people who do interesting things that I don’t normally get to do. Their breadcrumbs lets me get the overall picture of where we are all heading as the industry. 

Any people in particular you suggest others should follow?

Definitely! @b0rk (Julia Evans) and @dan_abramov (Dan Abramov)

How do you separate good project ideas from bad ones? 

I usually validate it against my own itch. If the project scratches my own itch, then I immediately start working on it. Read Me Later is one such idea. Of course, this doesn’t scale. So for commercial ideas I resort to the Google test :)

Google the idea and see if there's a product in that space. If it exists, the market exists. Study the competing product’s roadmap & pricing strategy. If the new idea offers a better proposal than those competitors, it’s not a bad one after all.  

Why was CloseTab started?

I needed a bookmarking system that makes me come back to links that I wanted to read later. Most bookmarking services today are dumb buckets, where we dump things and forget. Most of them suck at helping me complete my ever growing reading list. That’s why I started ReadMeLater. 

Who, or what was the biggest inspiration for CloseTab?

Those 50+ browser tabs that I regularly end up with :) I didn’t want them to dump them in my abyss of bookmarks. I wanted to close them confidently, knowing they’ll come back to me later!

Are there any competitors or projects similar to CloseTab? If so, what were they lacking that made you consider building something new?

Pinboard. Pinboard at least has a concept of read-later. But it didn’t do anything much to solve the actual problem, which is – we forget stuff once we bookmark it.  

How is CloseTab intended to eventually be monetized?

For now the cost of keeping the hosted service running is negligible. This is because the database is mostly meta about bookmarks and textual content. But the roadmap includes automatic archival of bookmarks, Pocket syncing, and a bunch of other stuff that will require a small yearly premium. This revenue will in turn fuel the growing number of free users, making it a sustainable service. 

What is the best way for a new developer to contribute to CloseTab?

The entire codebase is Javascript – both the front-end and the server. That itself should make it easy enough for devs to start contributing. 

Porting the existing browser extensions to Safari & Edge should be an easy & useful direction for new devs. In fact, the Firefox extension itself was a contribution from the community.

If you plan to continue developing CloseTab, where do you see the project heading next? 

I’m seeing it as an anki style feedback system that helps internet readers productively manage their reading list. 

What motivates you to continue contributing to CloseTab? 

I use it myself :)

Where do you see software development in-general heading next? 

Software development is a broad topic to have a meaningful answer. I’ll split it up.

User Interface Programming:

UI programming is swiftly moving towards a functional style of programming, taking state management concepts from FP. We’ll quickly see Qt and other GUI frameworks either pick up this trend or die trying.

Server Programming:

In the start of the decade we saw huge rewrites of softwares written in legacy Java/C++ into dynamic languages like Python/Ruby/NodeJs. By the end of the decade though, we saw the race turning back to strictly typed technologies like Go, Rust & the likes. I think the trend is here to stay. The industry realized that fast prototyping and making quality software don’t go hand in hand.

The future of server programming are technologies that cut infrastructure costs and developer productivity (maintaining large codebases) both of which mandate stricter typing and more first class primitives to represent business concepts.  

Domain Driven Stack:

We were lucky enough to experience the tech era where things were developed individually and fit into a broader ecosystem. The software stack right now i.e the database, operating system, and the CPU architecture are all different layers, sold by different companies. We are swiftly moving into consolidation where these layers are now built and sold by single companies/organizations, advocating more security and efficiency. 

This trend is what I see when Apple introduces new instructions for their M1 CPUs specifically designed for Swift. Or when Google makes their own processors for their AI frameworks. We might soon end up  with the industry merging these layers and making little room for innovations in the bottom layers of the software stack for hobbyist programmers. 


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Refer 10 friends to Console and we’ll donate $100 to an open-source project of your choice!

Console 54

Sherlock, Send, and DataFusion

Sponsorships

A·Team: A collective for top independent developers + product builders

A·Team is a stealth, invite-only collective of top independent builders; mostly UX/UI designers, software developers, and product folks. We match you to high-paying work with vetted clients, generally $110-$190/hr. We're a bit different from other platforms though in that we're quite small & selective, have a community at the core, and try to meet everyone 1:1 to better understand what they're looking for. We also match companies with teams rather than individuals, which means a) higher hourly rates and b) it's easy to work alongside friends and other insanely talented builders. We launched ~14 months ago, and have paid more than $4 million to A·Team members since.

We're staying small, but looking for a few more independent builders; if you mention "console" under who referred you, we'll fast-track you in. Here's the link to join (takes <90 seconds).


sherlock

Sherlock is a tool to hunt down social media accounts by username across social networks.

language: Python, stars: 25218, watchers: 861, forks: 2567, issues: 152

last commit: January 15, 2021, first commit: December 24, 2018

https://twitter.com/sidheart

send

Send is a a fork of Mozilla’s Firefox Send, which was simple, private, file sharing.

language: JavaScript, stars: 1147, watchers: 13, forks: 42, issues: 7

last commit: May 19, 2021, first commit: May 24, 2017

https://twitter.com/likecaffeinated

arrow-datafusion

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

language: Rust, stars: 437, watchers: 37, forks: 51, issues: 207

last commit: May 23, 2021, first commit: February 05, 2016

https://twitter.com/TheASF

readmelater

Read Me Later is a bookmark with a snooze button. Bookmark, buffer and complete your reading list.

language: Vue, stars: 28, watchers: 3, forks: 3, issues: 0

last commit: May 18, 2021, first commit: May 19, 2019

https://twitter.com/vettijoe


An Interview With Andy Grove of Ballista

Hey Andy! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?

I’m originally from the UK and started learning how to code when I was around 13 years old. I knew right away that this is what I wanted to do as a career, so I finished secondary school when I was 16 and went straight into a career as a junior programmer for a financial services company in London. My first job was working with databases (dBase III and Clipper). I eventually moved onto C++, and then Java, and have primarily been working with the JVM for the past 20 years. Around 5 years ago, I decided that I needed to start learning some new skills and this is when I began to invest in learning Rust. I was keen to get back to a compiled language and I was intrigued by Rust’s unique approach to memory management.

Who or what are your biggest influences as a developer?

Apache projects: Hadoop, Spark, Catalyst, Arrow, and the people behind those projects.

What’s your most controversial programming opinion?

I’m not often controversial so I’ll take this opportunity to give you two controversial views: 1) Hand-written parsers are better than using parser generators! Since discovering the Pratt parser paper, I have been a huge fan of this approach. 2) When coding in Rust it is often fine to avoid explicit lifetimes and just store data on the heap using Box. You’re still better off than using a garbage-collected language. Optimization can come later, as you get more comfortable with the language.

What are you currently learning?

I’m not learning anything specific right now, but I am continuing to learn about query engines and distributed computing, both in my day job, and with my involvement with Apache Arrow. I’m getting the sense that I could continue learning about these topics for the rest of my career.

How do you separate good project ideas from bad ones?

That’s a tough one. What is a “good” project idea? Is it the one you learn the most from? Is it the one that gets the most GitHub stars? Also, it can be very subjective. I once built a tool called FireStorm/DAO which was a simple tool for reverse-engineering database schemas and then generating Java code based on the schema. Some people loved the tool and some people hated it. That was definitely a good learning experience for me. There are different audiences with different needs and opinions. These days, I tend to work on side projects just to explore ideas or learn a particular skill. They are usually very short-lived. Ballista was an unusual one because it gained a lot of traction early on and grew into something larger.

Why was Ballista started?

Partly for the same bad reasons that many projects start. I wanted to learn something new and I had a new favorite programming language and needed a real project to build. Also, for some well thought out reasons (hopefully) in that a natively-compiled systems level language like Rust has a lot of benefits over a garbage-collected language for large-scale data processing. I was particularly excited for the performance of compiled code and the efficient memory usage. Early experiments demonstrated that Ballista would use an order of magnitude less memory than Spark in some cases. The lack of GC also means that performance is consistent.

Who, or what was the biggest inspiration for Ballista?

Apache Spark is by far the biggest inspiration for Ballista. Apache Spark is a brilliant platform but I always felt that it was implemented in the wrong language. However, rather than being a direct port or rewrite of Spark, Ballista is a reimagining of Spark based on the current state of the art, which has moved on a bit since Spark was started in 2009.

Are there any overarching goals of Ballista  that drive design or implementation?

Yes, language-neutrality.

What trade-offs have been made in Ballista  as a consequence of these goals?

If this had been a pure Rust solution then there would have been no need to write serde code for serializing query plans but because of the language-neutral architecture we have had to write lots of tedious code to translate between Rust structs and Google Protobuf format.

What is the best way for a new developer to contribute to Ballista?

Ballista is now part of the Apache Arrow DataFusion repository, and we have a number of GitHub issues tagged with “good first issue”, so I would recommend finding an issue that looks interesting and then start asking questions there, or on the Arrow mailing list.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

My advice would be to start small and pick a trivial issue to work on, even a documentation issue. This is a good way to get to know the tools and processes and start to interact with the other developers that are working on the project.


Like what you saw here? Why not share it!

Share

Or, better yet, share Console!

Share Console

Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.

Console 53

Hotdogs, Humbug, and Delta

Sponsorships

Streamr Network

The Streamr Network is an open-source, massively scalable pub/sub network for transporting encrypted data points to thousands of subscribers. You can use the network in your own applications via the SDKs or by directly calling the API. If you're an app builder, the Streamr Data Fund exists to support projects that further the vision of a decentralized data economy. All proposals are welcome at https://streamr.network/fund. Learn more about the Streamr Project at streamr.network.


delta

delta is a viewer for git and diff output.

language: Rust, stars: 8964, watchers: 43, forks: 139, issues: 104

last commit: May 10, 2021, first commit: June 23, 201

https://twitter.com/dandavison7

thdwb

The Hot Dog Web Browser is a web browser with its own layout and rendering engine, parsers, and UI toolkit, made from scratch entirely in Go.

language: Go, stars: 841, watchers: 17, forks: 39, issues: 4

last commit: May 10, 2021, first commit: July 29, 2019

https://twitter.com/danfragoso

humbug

Humbug helps you understand what keeps users coming back to your developer tool, as well as any friction they experience.

language: Python, stars: 12, watchers: 4, forks: 5, issues: 11

last commit: April 30, 2021, first commit: February 24, 2021

https://twitter.com/Bugout_dev


An Interview With Neeraj Kashyap of Bugout.dev

Hey Neeraj! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?

One of my early programs drew geometrical figures for my math homework (hundreds of problems) that I was too lazy to draw by hand. This was a QBasic program, which I learned by reading the Absolute Beginner’s Guide to Qbasic, and reading the source code of Gorillas.

I have had two great loves in programming:
Common Lisp - Was introduced to Lisp in an AI class in college, and immediately fell in love. I spent months worth of late nights writing a computer algebra system in Lisp.
Python - Was introduced to Python when I was doing some number theoretic experiments as part of my PhD work, in 2012. Fell in love with the expressiveness of the language, and have been obsessed ever since.

I started my life as a mathematician and initially had no desire to do anything but teach mathematics and do mathematical research. That changed when my grandmother was diagnosed with progressive supranuclear palsy. Watching her progression with the disease made me realize that I wanted to build things that were of actual use to people - people like her, and like my mother who was her primary caregiver.

I spent two years in Japan as a postdoctoral researcher, developing an algorithm to help diagnose neurodegenerative disorders earlier than they are normally caught.

Progress is slow in academia, so I took a chance and moved to the Bay Area to start a company, which didn’t work out. My journey since then has taken me to a couple of startups, and to Google, where I worked on Tensorflow and Cloud AI.

I am now working on Bugout. We are building tools which help creators of developer tools (libraries, command line tools, and APIs) understand who their users are, and how they are using their code.

Who or what are your biggest influences as a developer?

About 10 years ago, I took a course on recursion theory with Larry Moss, a logician at Indiana University. That course really made me fall in love with computing.

In that course, we would write programs against a register machine in a language called 1# in which the only symbols were “1” and “#”. One week, our assignment was to write a universal program - basically a 1# interpreter. Completing that project gave me the confidence to tackle other difficult problems in computer science.

What's an opinion you have that most people don't agree with?

Libraries and frameworks should be open source. Server code should not be open source.

If you run a server, that server’s users are trusting you with their data and with data about how they use your server. This is a serious responsibility. Besides other security measures, you should also rely on obfuscation. To whatever extent possible, limit the number of people who know how your server is implemented.

What’s your most controversial programming opinion?

bash is an amazing programming language. It gives you direct access to the wisdom of the ancients. You will be very well rewarded for writing at least one serious bash script a week.

What is one app on your phone that you can’t live without that you think others should know about?

tiktok :)

If you could dictate that everyone in the world should read one book, what would it be?

I would never dictate something like this. In fact, I have a counter recommendation - go out there and get your hands on some real garbage. I mean books that no one has ever recommended to anybody else in the history of humanity. Books that you know from the start that you are going to disagree with and vehemently. Books that nobody has even taken the trouble to read before. Read them.

If you are caught in an echo chamber, this is the surest way to break out of it.

You can find books like this at your public library and at library book sales. You can also find them on Amazon - just search for books that cost less than a dollar.

If you could teach every 12 year old in the world one thing, what would it be and why?

Don’t do your homework. It is a complete waste of your time. Create something instead.

If I gave you $10 million to invest in one thing right now, where would you put it?

I would set aside $8,000,000 in a trust for my family - children, wife, parents, sister, in-laws. Just to get rid of all financial worries.

I would invest the remaining $2,000,000 into Bugout.

What would you spend it on at Bugout?

We are currently hiring fairly technical people to join our marketing and sales teams. 1/4 of the resources would go there.

For us, developer relations and customer success are closely tied together. Improving our on-boarding and our documentation is becoming increasingly important. As is communicating the value of Bugout to the world, through blog posts, videos, etc. Our customer success team needs to be highly technical, as well - we intend to hire engineers into these roles. I would put 1/3 of the money there.

The rest would go towards hiring engineers, paying for infrastructure, and other assorted costs of running a business.

What are you currently learning?

Learning about different blockchain implementations.

Any blockchains in particular that you’re interested in?

I really like Ethereum. Polkadot is pretty interesting.

Don't understand Monero in detail yet, but it seems worth spending some time on.

What resources do you use to stay up to date on software engineering?

Curiosity. :)

I read Hackers News a lot, and also various subreddits for programming topics I’m interested in (e.g. r/programming, r/machinelearning).

What have you been listening to lately?

The last thing I listened to was the album Blade of the Ronin by Cannibal Ox.

How do you separate good project ideas from bad ones?

Are you still excited about an idea a week after you had it? If so, it’s a good idea. If not, move on with your life.

Why was Bugout started?

My co-founder (Sophia) and I really admire people who make things. Bugout embodies everything we think will be useful to the next generation of programmers.

How did you and Sophia meet?

We met at a healthcare startup called doc.ai. I was managing the data science team there and Sophia was head of communications.

Are there any overarching goals of Bugout that drive design or implementation?

Our immediate focus is on helping people who maintain libraries, APIs, or command line tools understand their users better.

They use Bugout to learn who their users are, how they use their software, and what kind of issues they experience. This knowledge comes in the form of usage metrics and crash reports. Our reporting libraries are freely available at https://github.com/bugout-dev/humbug.

Two key principles that drive our design and implementation:
1. Providing a magical user experience requires deep understanding of your users - something you can only get from the client side.
2. Privacy is paramount. Bugout does not transmit any data from any client without the user’s explicit consent.

What trade-offs have been made in Bugout as a consequence of these goals?

Not a technical trade-off, but a product trade-off.

Our end user’s privacy is our highest priority. We have added a mechanism that anyone can activate to disable all Bugout reporting in any tool that they use. They simply have to set the environment variable BUGGER_OFF=true and no Bugout reports will be sent back from their system for any tool that they use.

The trade-off here is that this short circuit makes Bugout a little bit less useful to customers since we report data about fewer users. But it is the only ethical choice.

What is the most challenging problem that’s been solved in Bugout, so far?

As part of our machine learning work, we need static analyzers that do more than understand code. We need analyzers that understand how code changes over time.

Since we couldn’t find such a project, we had to build our own - Locust. It produces a JSON object describing the changes to a code base over time, with time measured in git commits.

Interesting section: For the most part, we are doing standard abstract syntax tree traversals. The interesting part is the function which superimposes git diffs onto the results of these traversals (link).

We will be launching features in the coming month or two which use Locust data to drive code modifications.

Are there any competitors or projects similar to Bugout? If so, what were they lacking that made you consider building something new?

Our biggest competitor is Sentry.

If you want to use a tool like Sentry to collect usage or crash reports from your users’ machines, you have to implement all privacy-related functionality yourself - processing and storing user consent, serving GDPR requests for data access or data deletion.

Not only is it hard to get this right, but it is also a distraction from just building your product. We built Bugout to solve this pain.

What is your typical approach to debugging issues filed in Bugout repos?

We prefer that our users report issues to us on our community Slack. We are still at a scale where that’s more effective than working with issues on GitHub.

It’s easier for users to report problems or make requests because things are more casual on Slack.

It’s also better because we can interact in real time. Conversations on GitHub issues are asynchronous and it can sometimes take days or weeks to resolve simple matters.

Regardless of whether an issue is filed on GitHub or mentioned to us on Slack, our approach is similar. First we identify if it’s a bug, feature request, or a misunderstanding.

We try to resolve bugs within a day of being filed.

For feature requests or misunderstandings, we first talk to the reporter to really understand their use case.

If it’s something that Bugout is already capable of, we explain it to the reporter and also improve our documentation.

If it’s something new, we prioritize internally, tell the reporter when we think we will have bandwidth to work on it, and invite them to collaborate with us on it.

What is the release process like for Bugout?

We release several times a day in the repos that are seeing active development. All releases take place automatically upon merging into the main branches of our repositories.

Our APIs run purely on VMs on AWS. We use just plain old SSH to deploy. Here is an example of one of our deployment scripts.

Actually, we are pretty opinionated about deployments. I wrote a blog post about it at the end of last year: https://blog.bugout.dev/2020/12/03/break-out-of-the-deployment-death-loop/

How is Bugout monetized?

Our customers use the Bugout API to collect reports from their clients. We charge them per 1000 reports they post to our API.

What is the best way for a new developer to contribute to Bugout?

We have a few open source projects: https://github.com/bugout-dev

If any of them seem interesting to you, join us on the Bugout slack and ping me - my username is @zomglings. We can talk about what you would like to achieve, how that fits into the project in question, and I can onboard you onto the code base.

After that it’s Slack conversations, GitHub issues, and pull requests all the way.

If you plan to continue developing Bugout, where do you see the project heading next?

Our dream is that, when a bug report comes into a repo, Bugout can analyze what the issue is, modify the code base to fix the bug, and submit a pull request. This would give any programmer working on the issue a good place to start (and hopefully finish).

In the next couple of months we will start releasing some of our code modification features. We want to make it easy even for non-technical people to manage Bugout reporting in projects that they participate in.

We are also working on tighter integrations into developer environments. We just launched our VSCode extension in which programmers can hover over exceptions to see how many of their users have experienced that error recently, and even see the individual stack traces. This is something we will be expanding on.

Where do you see open-source heading next?

Regular contributors get the short end of the stick when it comes to sharing in the rewards for building a great project. This has led to a polarization in open source projects - projects that don’t have corporate backing tend to have very few regular contributors.

It takes time to build something really useful. A good team gets there faster. I hope that we can find more equitable ways of rewarding open source contributors so that we see more such teams.

There is a lot of potential for this kind of equity on public blockchains like Ethereum. You can tell from transactions how frequently particular methods were called on a contract. You could hold a percentage of the funds that were sent to the contract as a pool from which to pay out developers who had contributed to the contract’s development. The payout could be proportional to the amount of money spent on methods they contributed to.

A big problem is the problem of quantification. Is a one line bugfix worth more than a documentation change? How much more? How do we encode this difference in value into a program?

If last year’s Hacktoberfest drama is anything to go by, we will be contending with this issue very soon. This will be huge for open source, and for all software development.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

First, make sure your contribution is welcome.

The last thing you likely want is to spend your time building something only to have it summarily rejected because it doesn’t align with the project maintainers’ vision for the project. For most projects, it should be easy to get hold of the relevant maintainer and run your idea by them first - on a GitHub issue or over email or over Slack/Discord.

Next, make it easy for a reviewer to approve your change.

Open source reviewers are a busy lot. The more time they have to spend understanding and testing your change, the less likely it is to get merged. Document everything that you have done. If you made any decisions that weren’t obvious, explain your rationale. At the very least, run the existing tests and make sure they pass. Lint your code according to the project’s style guide (if one exists). If possible, add new tests for your change. Such a change would be a pleasure to review.

Finally, a piece of advice from a writing class I once took: When someone critiques your work, shut up and fucking listen!


Like what you saw here? Why not share it!

Share

Or, better yet, share Console!

Share Console

Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.

Console 52

Icecream, Boltstream, and Opal

Sponsorships

The Daily Upside

The Daily Upside is a business newsletter that covers the most important stories in business in a style that’s engaging, insightful, and fun. Written by a former investment banker, The Daily Upside delivers quality insights and surfaces unique stories you won’t read elsewhere. Sign up here for free.


opal

Opal is policy and data administration, distribution, and real-time updates on top of Open Policy Agent.

language: Python, stars: 242, watchers: 12, forks: 5, issues: 10

last commit: May 05, 2021, first commit: November 09, 2020

boltstream

Boltstream is a live video streaming website and backend.

language: Python, stars: 1628, watchers: 43, forks: 97, issues: 1

last commit: December 10, 2020, first commit: December 07, 2020

https://twitter.com/benwilber

icecream

Never use print() to debug again.

language: Python, stars: 4645, watchers: 46, forks: 88, issues: 12

last commit: May 04, 2021, first commit: February 13, 2018


An Interview With Or Weis of Authorizon

Hey Or! Let’s start with your background, where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?

I've been working with computers from the age of 5. Got my first major leap as an officer in an IDF intl. Corps unit called 8200.
Worked at a startup called Intigua (creating containers before containers were a thing). Founded a startup called Reactful, worked as a VP of R&D in Netline (providing cyber defence solutions to governments and like minded agencies worldwide), most recently before my current role I was the founder / CEO of Rookout, the market leader in production debugging.
Now I’m leading Authorizon and OPAL.ac with Asaf Cohen.

Do you have any insight as to what it was like to make the transition from VP to CEO?

At Netline I was working very closely with the business side - so I was familiar with the challenges on the other side of the table I was crossing into. 

In general I think one of the most challenging aspects of being a CEO is telescoping - i.e. the constant shift of focus from the overall big picture to the most minute (sometimes technical) details.  I think aside from recognizing it - it just requires a lot of practice to get right.

How did you and Asaf meet?

Asaf was a student of mine in the intelligence corps.

Who or what are your biggest influences as a developer?

  • Guido Van Russom and Python 

  • Biology and nature (see point about recommended book)

What's an opinion you have that most people don't agree with?

Capitalism and free-markets are a great idea; but the version we are working with today fails in some critical areas; and can actually be fixed easily. [Key point to mention - trickle down economics- it’s just a lie - it doesn’t work - instead we should apply what I call pour down economics, where we should proactively incentivise people to reinvest their money and take risks.]

What’s your most controversial programming opinion?

Test-Driven-Design is a really bad idea

What sort of approach do you prefer if not TDD?

I prefer design driven design - i.e. be an engineer and think through what you're building - where there's a need for more tests- do more tests; more manual experimentation... do that - more trial and error, more automation... These are all tools in your arsenal - you should use them all accordingly - If you need a prefixed answer for everything - you probably would be better off doing something else.   That said I'm not against tests.

What is one app on your phone that you can’t live without that you think others should know about?

Audible / Smart-AudioBookPlayer - I devour so much content through them.

If you could dictate that everyone in the world should read one book, what would it be?

The Selfish Gene by Richard Dawkins

If you could teach every 12 year old on the planet one thing, what would it be, and why?

Critical thinking with evolution, recursion, and emergent phenomena.  These are built in mechanics of our world that control and affect so many aspects of our lives from the economy and politics, through startups, to how your immune system works and baci biology - and somehow most people never learn about these.

If I gave you $10 million to invest in one thing right now, where would you put it?

S&P 500; putting all your money in one thing is a really bad investment strategy, so it’s better to invest in the market as a whole. Otherwise in authorizon of course...

What are you currently learning?

FastAPI and Typer by Tiangolo ; really great open-source frameworks for modern Python applications.

What resources do you use to stay up to date on software engineering?

Friends and building new things - forcing myself to encounter new challenges.

How do you separate good project ideas from bad ones?

Feedback-loops and testing.

Why was Authorizon started?

At my last company we ended up rebuilding authorization 5 times. Research taught us basically all companies are building authorization on their own from scratch - and worse endup rebuilding it every 6 months on average….

Who, or what was the biggest inspiration for Authorizon?

  • Companies like Stripe, Twilio, and Auth0.

  • Eugenio Pace (the founder of Auth0) in particular is very inspiring.

Are there any overarching goals of Authorizon that drive design or implementation?

Knowing our space, it was critical for us to build Authorizon on an open-source foundation. So we invested in taking a lot of our solution into open-source with OPAL.ac 

What trade-offs have been made in Authorizon  as a consequence of these goals?

Simply put - less features, and delay in offering the product as SaaS in favour of launching open-source first.

What is the most challenging problem that’s been solved in Authorizon and OPAL.ac, so far?

Real time communications in an efficient, lightweight, scalable fashion.
Before we created OPAL.ac as source code we created two underlying open-source packages 

What was the most surprising thing you learned while working on Authorizon and OPAL.ac ?

How quick a community can start to bloom- less than a week into launching OPAL.ac we already had people joining our Slack, asking to have Zoom call with us, and contributing code.

What is your typical approach to debugging issues filed in the OPAL.ac repo?

Depends on the issue, but we’ll either try to replicate locally, or on the cloud - and debug with Rookout. Most issues so far were more feature requests than bugs. And the bugs were mostly around communications- where our tests actually shine.

What is the release process like for Authorizon?

We’re very Gitops and CI/CD driven.
Pull Request -> Test validation -> CR -> integration / regression test -> Publish

Is OPAL.ac / Authorizon intended to eventually be monetized if it isn’t monetized already?

OPAL.ac is pure open-source - Authorizon - which provides Fullstack authorization as a service using OPAL.ac ;  will be available as a SaaS service with paid tiers.

How do you balance your work on open-source with your day job and other responsibilities?

They are one and the same- Open-source for us is a way to build a company that’s geared for developers.

Do you think any of your projects do more harm than good?

Hell no.

What is the best way for a new developer to contribute to OPAL.ac?

Look at the Issues on Github, ask questions on Github or Slack, and open a pull-request when you’re ready ;-)

If you plan to continue developing Authorizon, where do you see the project heading next?

SaaS and adding more policy-engines (e.g. Google Zanzibar support)

Where do you see software development in-general heading next?

Everything Mesh - the difference between edge, cloud, on-prem, my-code, your code, 3rd-party code, will become super blurred.

Where do you see open-source heading next?

Becoming the de-facto way large solutions bridge architectures

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

It all starts with people - it's about the community not the code (at least not just the code) - reach out:

What is one question you would like to ask another open-source developer that I didn’t ask you?

One of the biggest challenges of building open-source is getting people to hear about it. What are the special hacks you use to get there?


Like what you saw here? Why not share it!

Share

Or, better yet, share Console!

Share Console

Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.

Loading more posts…