Console 32
Console is looking for sponsors! If you, or someone you know, is interested in sponsoring the newsletter, please reach out at console.substack@gmail.com
torrent
language: Go, stars: 3685, watchers: 134, forks: 440, issues: 33
last commit: December 10, 2020, first commit: June 20, 2012
mundane
Mundane is a Rust cryptography library that is “difficult” to misuse, open sourced by Google.
language: Rust, stars: 529, watchers: 16, forks: 23, issues: 9
last commit: March 27, 2020, first commit: September 27, 2018
PerformanceOverhaulCyberpunk
language: C++, stars: 1931, watchers: 68, forks: 51, issues: 28
last commit: December 12, 2020, first commit: December 17, 2020
VerbalExpressions
VerbalExpressions is a set of libraries in various languages for verbally constructing regular expressions. The stats provided below are for the most popular Javascript repo.
stars: 11752, watchers: 287, forks: 520, issues: 22
last commit: December 18, 2020, first commit: July 19, 2013
https://twitter.com/luotojesse
ihp
IHP is a modern “batteries-included” Web Framework, built on top of Haskell and Nix.
language: Haskell, stars: 1571, watchers: 40, forks: 77, issues: 87
first commit: December 19, 2020, last commit: December 03, 2017
https://twitter.com/_marcscholten
Help Wanted
James is looking for a front-end engineer to improve the twtxt front-end, you can contact him at prologic@shortcircuit.net.au
If you’re interested in posting a help wanted ad for your project, send an email to console.substack@gmail.com
An Interview With Matt Joiner of torrent
What is your background?
I travelled a lot as a child, but most memorably lived in rural Australia. I learned to program by mucking around with QBasic on some games that came with the family’s first computer. In early high-school I had some classes that exposed us to building websites with HTML, and made some simple mods for popular games like Starcraft and Total Annihilation. I was exposed to Pascal and Visual Basic later in high school through some friends that were a lot more competent at the time. In first-year university, I remember reading a Java book from cover to cover in the first few weeks and that really kickstarted my programming efforts, writing applet games and Diablo tools.
I got my first programming job as support at a webhosting company, where I taught myself Perl. I learned C at university and landed a job at a defence research company doing C++ and some Python. From there I went on to work at several start-ups, and Google, picking up Go, and some data-crunching technologies along the way.
I really like languages/frameworks that are simple at the core, and let you build what you need on top without assumptions. I found C++ needlessly complex. C was my first love, and Python later opened my eyes to a lot more concepts, I remember learning a lot just reading the docs and standard library code, particularly around its threading and concurrency implementations. I primarily use Go due to the simple language, useful runtime, and implicit green threading. However I make frequent forays into other technologies, like Elm, Haskell, and Rust.
What's an opinion you have that most people don't agree with?
People work too hard, and worse, think that working hard and not smart still deserves recognition.
What’s your most controversial programming opinion?
I think Go’s error handling is really bad. The language isn’t complex enough to express what is needed to discriminate error reasoning reliably, and in general the best information you can get is in the stack trace from a panic. Furthermore, if that’s the route you do decide to take, it’s still too verbose (if err != nil…, careful checking of outstanding resources, how defers will operate in this situation at every return/panic point, etc.). When it’s possible to implement support for these things in a library instead of special casing things in the language, that’s promising.
Is there a language you feel has better error handling than Go? What would you say makes it better than Go?
The error handling support that Python provides is very good (albeit complex, but compare it with C++ where I think exceptions are basically unusable), with the ability to create an exception hierarchy, throw exceptions from other exceptions, stack traces, context managers etc. Haskell also has excellent error handling, there was a recent article covering a few elements of this.
In general, I think error handling and exceptional circumstances should not detract from the readability of the expected flow for successful execution (Joe Armstrong talks about this, search for mentions of “specification” in https://erlang.org/download/armstrong_thesis_2003.pdf). Go, unfortunately, makes these all explicit and very messy. Go’s context and recent errors package improvements are good because they standardize things, but they’re still messy (having to add Context everywhere, with conflation between request scoping and cancellation).
What is one app on your phone that you can’t live without that you think others should know about?
I use an offline GPS program called Maverick. I always know where I am, and where I’ve been. Unfortunately, the developer hasn’t updated it in a very long time.
If you could dictate that everyone in the world should read one book, what would it be?
Lord of the Rings. I can’t think of a specific non-fiction book that everyone should read, and Tolkien’s work is the definitive modern fantasy and mythology. Otherwise, it might be something on concurrency, statistics, and information theory and how they manifest in the real world.
If I gave you $10 million to invest in one thing right now, where would you put it?
Probably electric vehicles. I’d love to say programming language research, but software tooling doesn’t seem to attract money easily.
What are you currently learning?
Erlang. I’m always on the lookout for new approaches to concurrency. I looked at Elixir first on the recommendation of a friend but found it borrowed too much from Erlang and I was missing context.
I recently tried out trio in Python, and was quite impressed, but found it difficult to return to Python’s dynamic typing.
What resources do you use to stay up to date on software engineering?
I subscribe to things that catch my interest, and unsubscribe when they get tedious. Two newsletters that I still receive years later and have remained relevant are Porter.io, and StatusCode.
Why was anacrolix/torrent started?
My interest in concurrency and (video) streaming led me to try out concurrency ideas in Python by implementing parts of BitTorrent. It turned out to be a rabbit-hole that I didn’t return from.
Are there any overarching goals of anacrolix/torrent that drive design or implementation?
anacrolix/torrent was designed from the start to support streaming data directly from the network. Put another way, it should be possible to read directly from the BitTorrent network as though you were reading from a local file.
What trade-offs have been made in anacrolix/torrent as a consequence of these goals?
Streaming from the network on demand requires a different approach to the algorithms that prioritize peers and pieces that you find in traditional BitTorrent clients. anacrolix/torrent typically favours leaving peers connected (it will keep useful connections alive unless the slot is needed is underperforming), and can overlap requests to reduce read latency.
What is the most challenging problem that’s been solved in anacrolix/torrent, so far?
Some more interesting parts of anacrolix/torrent are the storage interfaces that allow storing, serving and caching data on all kinds of backends, like sqlite3, memory mapped files, Amazon S3, in-memory, as well as arranging data by piece as opposed to by file.
The DHT implementation has evolved considerably, concurrency considerations there are very complex, involving rate limiting against the OS and network capabilities, trying to preempt connection tracking slots in common NAT implementations, and prioritizing those resources against the need to avoid starving simultaneous announcements and network traversals. I’ve had numerous iterations of the concurrency in that repo, the current one makes use of an STM implementation in Go, which has some interesting performance considerations.
Recently a lot of changes have been made to anacrolix/torrent to handle scaling up to very large numbers (100k+) of simultaneous torrents. In particular, at a very large scale there are resource considerations to be made regarding tracker and DHT announcing, shared-secret stream decryption, and storage management. These are ongoing developments.
What is your typical approach to debugging issues filed in the anacrolix/torrent repo?
Generally I’m able to make use of the fact that I’m across the entire stack and can pick the point in the code to jump into and start testing hypotheses. I’ll confirm that the reported use-case matches my suspicions, and that there’s no XY problem present (So that I don’t solve something that wasn’t even the real issue. Additionally, there might be another way to frame an issue that would allow for a solution that won’t force the implementation to make bad compromises).
What is the release process like for anacrolix/torrent?
Generally if development slows down for a period, and I’m happy with the stability of master and haven’t seen any issues for a while, I’ll tag a minor release. I’ll try not to do this more than once a month unless there’s a good reason.
How is anacrolix/torrent currently monetized?
I am currently paid to consult on my own work for a larger project. Additionally, I have been funded for some major features on GitHub Sponsors and IssueHunt to be added to anacrolix/torrent. I am always open to more funding to accelerate feature development and performance improvements of which there is still plenty of runway.
How do you balance your work on open-source with your day job and other responsibilities?
I’m very fortunate in my current day job in that there’s considerable overlap with my open-source projects. We’re making extensive use of anacrolix/{torrent,dht,confluence}. I only work part-time, which gives me downtime for recreation, side projects, and other hobbies.
What is the best way for a new developer to contribute to torrent?
Find a use-case you have, and give it a try. If you have questions about what’s possible, feel free to ask on GitHub or in Gitter as many other people have. I do maintain a specific label where I’m looking for more input, but you can jump in on any issue too.
Where do you see anacrolix/torrent heading next?
The project is gradually crossing off everything on the laundry-list (probably out of date) of BitTorrent features seen elsewhere. Recently it gained WebSeed and Webtorrent/WebRTC support. I expect soon, BitTorrent 2 support will be on the cards.
Where do you see software development in-general heading next?
Unfortunately software development is rapidly increasing in complexity. Siloes are forming in technologies but this is further fracturing things and harming interoperability. I see the situation worsening for the next few decades before some real advances are made in the underlying concepts in software development (we might even become an actual engineering discipline).
Where do you see open-source heading next?
I think open-source serves several functions. It allows people to publicly experiment with ideas of all kinds and get feedback. It acts as a back-stop, where new work must do something better to stand out. When open-source advances, it’s a sign that the field is advancing. In the future I hope that open-source continues to evolve to be a marketplace of ideas, funding and interoperability.
Do you think any of your projects do more harm than good?
Some of my projects are not great. Often those ones are attempts to work around rough edges in the tools I’m using. I bundled a lot of utility code into anacrolix/missinggo because the pain of creating a repo for every unrelated package was too high. It’s since become a maintenance nightmare, there are parts of it referenced in various major versions due to my early adoption of Go modules. I created argument parsing and logging packages due to the clunky and unwieldy syntax of the popular solutions in Go, but they are turning out to be non-trivial.
Do you think there is too much noise in the open source world?
There are a lot of projects that get started, and quickly get buried under boilerplate and tedium and the author loses interest before solving any interesting problems. There are some good comments on HN regarding risk-first or front-loaded risk development that I think are appropriate here.
How do you separate good projects from bad ones?
I wish there were a way to quickly pick good solutions from bad ones. This is particularly relevant when you’re new to an ecosystem, like when learning a new language. Often the criteria for development differ significantly to what general users might use. So-called “awesome” pages are often skewed due to opinion, not comprehensive, too comprehensive, or out of date. The social ranking mechanisms like GitHub stars, while flawed, are often the only way to distinguish projects. I’m often attracted to a small, clean package for something I need, only to find later that it’s incomplete, the author lost interest, or it makes bad assumptions that aren’t worth dealing with.
Do you have any suggestions for someone trying to make their first contribution to an open-source project?
Find something you care about, and make a careful, calculated attempt to improve it. Keep things small to start, build trust and gauge your audience's receptiveness. Don’t be intimidated, and don’t be afraid to try new things.
This Week’s Question
Of the following chat services, which do you prefer?
Last Week’s Responses
If Console had a subscriber only chatroom with the interviewees, would you join?
Like what you saw here? Why not share it?
Or, better yet, share Console!
Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.