Console
CyberChef, Continuous Reforestation, and Git Split Diffs
Sponsorship
The Reshape
Stay up to date with AI 🤖 & Data Science 📊 by reading top-notch articles. We are the first to spot hot news (data proven!). At the same time, we scour the Internet to find the most overlooked publications. Get a package of both in a concise form delivered straight to your inbox for free 📬.
Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.
Already subscribed? Refer 10 friends to Console and we’ll donate $100 to an open-source project of your choice and send you a Console sticker pack!
Projects
CyberChef
CyberChef is a simple, intuitive web app for carrying out all manner of “cyber” operations within a web browser open-sourced by GCHQ.
language: JavaScript, stars: 12446, watchers: 315, forks: 1631, issues: 251
last commit: March 26, 2021, first commit: November 28, 2016
git-split-diffs
git-split-diffs is GitHub style split diffs with syntax highlighting in your terminal.
language: TypeScript, stars: 1923, watchers: 8, forks: 18, issues: 4
last commit: July 07, 2021, first commit: April 10, 2021
https://twitter.com/banga_shrey
continuous-reforestation
A GitHub Action for planting trees within your development workflow using the Reforestation as a Service (RaaS) API developed by DigitalHumani.
language: Python, stars: 130, watchers: 7, forks: 3, issues: 4
last commit: March 27, 2021, first commit: December 09, 2019
https://twitter.com/protontypes
An Interview With Jesse Smith of DistroWatch
Hey Jesse! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?
This question covers a lot of ground and I'll try to keep my answer from rambling on too long. In the past I've worked in a variety of locations and positions. I've been a janitor, convenience store clerk, support desk IT guy, server admin. I've been a roaming IT guy that goes into different businesses all the time to try to fill in or fix issues. I've also done product and music reviews for a few publications.
Location-wise, I'm from eastern Canada. I grew up in a rural environment in a very scenic region of Nova Scotia. I was lucky, as a kid, to have relatively early access to computers, both at school and at home. I learned to program by accidentally typing "list" instead of "run" one day while trying to run a BASIC program and was fascinated by the gibberish that appeared on the screen. Being a curious lad of about nine I started making minor adjustments to the code on the screen to see how it would impact how the program would run. For the first six or seven years I was mostly self taught.
Later, I was able to learn Pascal in high school and then took a more in-depth set of programming, sysadmin, and database courses in college. While I was there I learned UNIX and started running Linux at home.
As for which languages and frameworks I enjoy, I'm pretty flexible and happy to use whichever tool seems most convenient at the time. I mostly use C/C++ for local applications and JavaScript or PHP when working on websites. While I picked up COBOL, Assembly, Java, Delphi, Python, and a handful of other languages over the years I've rarely ever had a real-world opportunity to use them.
Typically the level of code I work with doesn't lend itself to using frameworks much, but I have learned to appreciate Qt for desktop programming. The "signals and slots" approach takes a little while to get used to, but I appreciate how clean and powerful Qt can be.
What sort of music were you reviewing?
I mostly reviewed metal music. I used to write for We Love Metal (they've since shut down). I'd often get assigned the more obscure or unusual bands, stuff the other reviewers weren't interested in. So I ended up listening to symphony metal, some instrumental-only groups. I really enjoyed Lullacry, Benedictum, and Kevin M Thomas.
I also reviewed some pop and rock albums. I reviewed Sarah Jackson Holman early on, Yusif, and Snakecharmer.
One of my favourite outcomes from a review was I was the first person to review an up and coming artist named Natasha Mira Todd, probably about ten years ago. She's gone on to bigger and better things, but at the time she was entirely self-published and working out of her home. She and her family were so thrilled at the review - just getting more exposure I suppose - her mother sent me a thank-you for the article. It was really sweet of them.
Who or what are your biggest influences as a developer?
The "who" part of this question is tricky. I haven't had many mentors, guides, or people I looked up to in my journey as a developer. I have an uncle who provided me with some programming books and answered questions for me when I was younger and I appreciated that. In a fun twist, his son (my cousin) interviewed me about what it's like to be a developer for one of his college courses last year. So the relationship came full circle.
When I was in college I had two instructors in particular who were a big influence. My Pascal and Assembly language professor taught me a lot about efficient code and squeezing the most out of the hardware. My UNIX professor certainly put me on the path of running Linux and BSD. His approach to things like security - balancing pragmatism versus idealism - had a strong influence on my work as a system administrator.
As for "what" is my biggest influence... It's probably my situation, the context of what I'm working on or my perceived need at the time. Early on in my developer journey I was often working with some tight constraints - whether it was time, computing resources, or tools. As a result I have tended to take the approach which gives me the best result for the least amount of resources/time/patching.
Two of my earlier projects were prime examples. One had to run on DOS machines, on processors of any speed, and be easy to upgrade by non-techie people in remote areas without a network connection. My first open source project had some uncomfortable requirements of being graphical, cross-platform, it had to be super easy to use and install, and I didn't have access to the target platforms for testing purposes. I ended up writing the whole thing in structured JavaScript to create what we'd probably call a web-app these days, but at the time (late 2002) it was just a fancy web page. I checked earlier this year and, 18 years later, the thing still runs flawlessly on modern browsers, which I feel is a good sign for something I slapped together fresh out of college.
What's an opinion you have that most people don't agree with?
Attack Of The Clones was my favourite Star Wars prequel, does that count?
In technology circles, I think the opinion (or maybe it's better to call it an outlook) I have that many people don't seem to share is finding a practical balance. Oftentimes people get caught up in their quest for something: the perfect design, unbreakable security, using the best programming language, the distro with the most up to date software, or one ultimate distro that can do everything. Call me either practical or lazy, but I'm inclined to see computers as tools. Very interesting tools, to be sure, but ultimately just tools. So while I have preferences, I'll try to find the best tool for the job in front of me.
For example, I prefer open source software, but I'll use non-free firmware if it gets my wireless network running. I use some basic security approaches to lock down my machines, but I'm more interested in being able to detect and recover from an intrusion or theft than I am in preventing one. I have a directory full of scripts on my workstation for accomplishing or automating all sorts of little tasks and they're written in a mix of bash, tcsh, awk, and PHP. I don't have a strong preference (some might say loyalty) to any one language or approach, I'll use whatever seems to best fit the job at the moment.
What’s your most controversial programming opinion?
Well, I just mentioned using and liking both PHP and JavaScript, so those might qualify. Honestly, I think both of those languages (and COBOL for that matter) have an undeserved poor reputation because they're beginner-friendly and that results in a lot of awkward, poor code being written in them. But the languages themselves I find quite straightforward and consistent to work with and I've never had a serious problem with them.
Another outlook I have is that I regard C++ as a perfectly good language, as long as you ignore most of its features. I like writing C++ as it seemed to be originally intended: C with classes. I think too many programmers see the many things a language _can_ do and dive in, not stopping to think how unreadable their code is likely to become. I'm getting off track here, but my unpopular opinion is that more programmers should focus on writing source code that is readable and usable by other programmers rather than writing the coolest or most powerful code they can.
If you could dictate that everyone in the world should read one book, what would it be?
One of my favourite books, which I happened to just finish reading again, is A Wizard Of Earthsea by Ursula K LeGuin. It explores a young man wrestling with pride, fear, and his demons (inner and outer) in a fantasy setting. The book has a simple flow and language to it, making it quite accessible, while sharing perspectives on calmness, self-knowledge, the importance of friendship, and the dangers of ego-centric pride. Plus there's a cool fight between a wizard and a dragon, so that's pretty exciting.
If you had to suggest 1 person developers should follow, who would it be?
I feel I don't have a good answer for this as I tend not to focus much on any one developer's work or opinions. So I'm going to cheat and recommend two whose work I respect. Jim Hall, the creator of freeDOS (@FreeDOS_Project on Twitter) is an outstanding and super friendly guy who always seems to have positive and insightful things to say; and Jonas Termansen (@sortiecat on Twitter) is doing some really cool work through Sortix. Even if you don't have any interest in these open source operating systems, I'd recommend listening to their views on clean design and standards.
If you could teach every 12 year old in the world one thing, what would it be and why?
You have a lot of tough questions. One thing? Someone once asked me if I thought honesty or compassion was more important. Then pointed out that it's almost never an either/or choice. You can be honest and compassionate. You can disagree without being cruel. You can speak the truth without being a jerk and be kind without lying.
Too often I hear people respond to having their behaviour called mean by saying they're "just being honest" or justify lying by saying they "don't want to hurt the other person's feelings." Good communicators can be kind and honest, firm and compassionate. I wish more people knew that.
What are you currently learning?
Right now I'm working on some projects which are testing the limits of my knowledge (and patience) with Shopify and Wordpress. Both solid platforms which these projects are trying to push in unexpected directions.
Care to get into specifics?
With the Shopify situation we're looking at expanding the "shop" to include more blog-like and community-oriented options. Shopify can be expanded this way, but it's not really designed for it and the available plugins are mostly commercial. So we're looking at ways to extend the social, search, and blog options without spending too much money.
The Wordpress account is a different beast. Wordpress is pleasantly extendable, that's all well and good. But my client wants to merge the Wordpress user accounts with an existing database of customers so they can interact with the new Wordpress website and their old shop seamlessly.
How do you separate good project ideas from bad ones?
There are a few things which come to mind. One of the big questions is whether the project's scope is too big. People who come to me with projects they want to implement which will revolutionize something or be used by millions of people probably aren't realistic. Another consideration is whether the project is sustainable long-term. I could write software which will calculate income taxes or insurance rates, but they'd need to be updated and tested against new rules every year. Does the person or team involved have the time to do that consistently on a reasonable time table?
Finally, I try to take on projects I personally find interesting and useful. As much as possible I want to be engaged. Often I hear new developers asking, "I'm looking for a project. What should I work on?" To me this is like a painter asking what they should paint or a writer asking what the topic of their first novel should be. Find what motivates you, what you want to share, what you want to make better.
Who, or what was the biggest inspiration for DistroWatch?
I don't want to speak for Ladislav, but I think the inspiration was just trying to make sense of the expanding Linux ecosystem and the lack of consistency between the various distributions. Each project has its own release schedule, package versions, package names, set of features, filesystem organization, etc. It was difficult to make an informed comparison between any two Linux distributions. DistroWatch strives to gather up the data and present it in a consistent format, making an "apples to apples" comparison easier.
Are there any overarching goals of DistroWatch that drive design or implementation?
Our goal has always been to try to provide useful information to our readers. Whether that's through reviews, offering up a glossary of terms, making it easier to find documentation, answering questions, making download links easier to find, or helping them compare features between distributions. We want to gather and share information in a way that makes sense to people.
Whether we implement a new feature or provide a new information page often depends on how useful we think it will be to our readership. Is it something people will find useful or will it be one more button or clutter that gets in the way?
What trade-offs have been made in DistroWatch as a consequence of these goals?
I think the trade-offs we tend to see the most are in the Search page and in how information gets added to DistroWatch. We get a lot of suggestions for search features people want to see or tags people want to have added to distributions in our database. Many of these don't get added because, well, the site has been active for 20 years. If just one person in that 20 years wants us to track a specific package or make it possible to search for how many people are on a development team, then it might not be worth looking into. But if a dozen people all ask us to make it possible to search for a specific init implementation then we tend to take notice and add it.
With regards to how information gets added to DistroWatch, earlier on in my involvement with the website I wanted to make it more open, more wiki-like. However, it quickly became apparent through some tests that we'd end up getting a lot of spam and a lot of inaccurate information that way. It would take longer for us to verify and curate data from volunteers than it would to just track and manage the information ourselves. This means it's somewhat less convenient for people to get new information or projects added to DistroWatch, but it means what has been added has been verified.
What is the most challenging problem that’s been solved in DistroWatch, so far?
Earlier I mentioned that each distribution organizes itself differently. Linux distributions use different package managers, use different names when packaging the same software, or place information in different locations. It is both our mission and greatest challenge to sort through all that diversity (some might say chaos) and try to extract the information to organize it into a set format. This makes it easier for visitors to our site to search for and compare packages and features between projects.
The tricky part is learning how a distribution organizes its data and then trying to automate extracting the information. I have dozens of scripts for cracking open a project's install media (ISO file), finding key bits of information and then sorting it into tables. Almost every distro needs its own script, or tweak to an existing script, to gather up all the data we organize and share. I guess it's fair to say there isn't one specific bit of code that stands out, it's more the process of regularly writing and tweaking scripts to harvest the diverse information from hundreds of distributions.
Are there any competitors or projects similar to DistroWatch?
As far as I know, there weren't any similar products or websites around when DistroWatch was created. I think that's why it gained so much attention. I don't think anyone was doing anything similar, at least at the time, so DistroWatch was providing an unusual, maybe unique, service. The closest project to it I can think of was FreshMeat.net which tracked open source releases and version information for specific programs. However, I don't know of anything else which did the same for distributions.
These days there still really isn't anything truly similar. There are some websites like Repology which track repository information for various projects, and others which review distributions or announce new product releases. Still others provide torrents for new distributions. But in each case they are basically each focusing on one small part of what we do. DistroWatch covers a lot of bases - linking to hardware vendors, new releases, package information, tutorials, an overview of key features, searches for key components, providing command line examples for dozens of popular programs, etc. I don't know of anyone else that does this.
From time to time someone will come along and decide they want to duplicate what we do, with a slightly different approach, and try to make The Next DistroWatch. However, they usually don't last beyond the planning stage because people greatly underestimate the amount of time it takes to run a website like this. Even once you've got all the website code written, which could take weeks or months, once all the infrastructure is in place then it can take an hour to process and publish a single new release. We usually do about two of those a day, on average. Plus writing articles, answering questions, and adding new projects to our database. We can get a few dozen to a few hundred e-mails and notifications a day, almost every day. People interested in making a DistroWatch-like website usually aren't prepared for that kind of time commitment. As a result we tend not to see many websites trying to do the same thing.
What was the most surprising thing you learned while working on DistroWatch?
One thing which regularly stuns me is both how resilient and how fragile open source software can be. The open nature of Linux distributions, and their components, means anyone can improve them, anyone can fork a dormant project to keep it running, anyone can fix a security bug. Anyone can come along and make a better mousetrap to improve the ecosystem.
At the same time, some critical, key pieces of infrastructure are barely maintained or abandoned. It's surprising to me how often it is revealed that some important piece of software, used on millions of computers, is maintained by one person in their spare time, or even effectively discontinued. When I stepped in to help maintain SysV init, I don't think anyone had worked on it in nearly five years. When OpenSSL caused a panic over the state of its code, it was basically a part-time project for a few developers. Yet both projects had been running on millions of machines. So much of the open source landscape is like that - amazing diversity, flexibility and power mixed with barely maintained keystones.
Where do you see the project heading next?
My ideas for the site have shifted slightly over the years. I used to be interested in more open contributions, more contributors, more interviews, more features! These days I'm more interested in small, polishing adjustments. Clearing up bugs, keeping things running smoothly, getting processes into place that make the day to day tasks run better. I'm more interested in providing curated, accurate information rather than having more information of unverified quality.
That was the key insight with this newsletter as well. Plenty of open-source newsletters out there are just rounding up a bunch of projects, but not many are highly curating them.
I agree. I think the "Bazaar" approach is great for giving projects exposure, but it's not great at helping people find specific projects or reliable information. Most of the "top 10 alternative applications" or "top 10 distro" lists I see often include projects that either aren't really open source or aren't maintained anymore.
Are there any other projects besides DistroWatch that you’re working on?
Several! I also maintain a handful of open source projects, including SysV init, the Bftpd file server, LimitCPU (a fork of cpulimit), the Dungeons and Dragons Character Generator, Sopwith SDL, the cross-platform port of doas which is used by FreeBSD and several Linux distributions, and I'm thinking of returning to the 2-D Atomic Tanks game which I maintained from 2005 to 2009. People can keep up with my work and new releases on my Patreon page.
Do you have any other project ideas that you haven’t started?
A few hundred. Most of them though are not realistic due to cost or time constraints. I sometimes toy with some code, or look at projects other people have started with similar ideas - open source competitors to Facebook or Twitter, for example, income tax calculators, I used to love coding for a MUD called Age Of Legacy and that siren song often calls to me. I'd love to make a multiple-player, networked version of Star Trek 25th Anniversary, a peer-to-peer open source, security-first chat program, a lightweight on-demand media player for the Raspberry Pi, a modern port of Legends of Valour, ... However, time is precious and limited. I do my best to balance my open source work with my home life and other hobbies. Most of these will probably never go beyond the design board unless someone hires me to work on them.
Where do you see software development heading next?
Probably more abstraction and more AI-style contributions. I think we're about due for another step up in terms of higher level languages or the way code is written. Something that makes it easier to snap together modules of code or ways to automate more of the code creation. For my generation Python and Visual BASIC were sort of a step up the modular, scripted abstraction ladder for computer languages. I suspect we're due for another step up where the developer gives some guidelines to the development environment and it uses an AI-like parser to try to figure out what the developer wants, then writes some basic code and module framework the developer can tweak and fill in as needed. We are close to that now with tools like Qt Creator, and I'm thinking the next step will be for the development environment to automate more of the "grunt work" in the coding process.
Along those lines, what are your thoughts on the no code movement?
I have mixed feelings about it. I like the idea of reducing the barrier to entry for people who want to create. Anything that encourages people to solve problems they see and makes finding solutions easier is good, in my opinion.
However, I also think there is a trade-off to this. More people creating programs without deeper knowledge likely means more bugs, more issues, more security holes. Unless, of course, we're very careful with how the tools people will be using are safe.
As an example, I occasionally get called in to fix projects, usually websites, which were created by "no code" tools. What You See Is What You Get editors, that sort of thing. When the site owners want to expand and discover the initial website isn't up to the challenge, they need someone who can code to come in and re-write or fix things.
So I think "no code" tools and approaches have their place, especially for addressing simple solutions. But I think there will always be a place for people like me who work more behind the scenes, a level down from the "no code" folks.
Where do you see open-source heading next?
I think more big companies are seeing the value of not only using open source, but being active open source contributors. Microsoft not only supports Linux on its cloud platforms, but runs it on Windows via the WSL and contributes to the Linux kernel. Netflix gives back financially and through development hours to FreeBSD. Google builds Chrome from the open source Chromium. I suspect more and more companies will do most of their development in the open, but save a little special customization or optimization for their final build. It makes good business sense to do most of the work in the open where the developer community can help and then sell a polished version to consumers.
Do you have any suggestions for someone trying to make their first contribution to an open-source project?
Two things, I suppose. One is to find a project which interests you. Find the bug you want fixed or the feature you want to see implemented. Then dive in. Pick something that you are motivated to work on, the result is very rewarding.
The other suggestion is to talk with the people who run the project you want to work on. Some of them don't want help, some want a specific patch style, some may like your idea but feel it is out of scope for their project. Talk with the other developers first before you do a lot of work which may not be accepted. They can either guide your efforts to areas that will be helpful or help teach you what you need to know about their project.