Console #131 - Interview with Jesse of Distro Watch
Featuring Mono Sans, Podman, DevOps Roadmap, and cozo
Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!
Mona Sans is a new variable font from GitHub. Variable fonts enable different variations of a typeface to be incorporated into one single file.
Podman Desktop is a graphical tool for developing on containers and Kubernetes.
👨🔬 DevOps Roadmap
Step by step guide on how to become a DevOps engineer, with links to relevant learning resources.
stars: 3190, issues: 1, last commit: 7 days
cozo is a general-purpose, transactional, relational database that uses Datalog for query, is embeddable, and focuses on graph data and algorithms.
language: Rust, stars: 1497, issues: 4, last commit: 4 days
Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!
🎤 Interview With Jesse Smith of DistroWatch
Hey Jesse! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?
I’m from eastern Canada, a quiet, beautiful, rural corner of the world. I originally learned to program by accident when, one day I was using a Commodore 64 at school and typed “list” instead of “run” after loading a program into memory. This resulted in the instructions of the BASIC program spilling onto the screen and, being a curious young lad, I started modifying some of the lines to see if it would cause the program to work differently. It did! After that I started learning by trial and error, modifying copies of small programs and games and learning what the key words meant as I went along.
Some family members and friends were kind enough to introduce me to programming books and reference guides after that and I was well and truly down the rabbit hole and in love with the experience! I went to college to learn programming, database management, and UNIX.
I was, honestly, terrible at learning UNIX and decided to start using something similar at home so I could immerse myself in it. One of my fellow students introduced me to Linux and I was quickly hooked on it.
Sadly, I graduated around the time the dot-com bubble burst and the Y2K bug was no longer a threat, so there weren’t many programming or admin jobs available in my area. I did a series of odd jobs for a few years, working on open source projects, tinkering with Linux and BSD servers, and eventually decided to work freelance. I basically just turned my hobby into my main job.
I’ve worked in a handful of offices and for a few companies, often doing admin or tech support work. But the bulk of my career is in writing about open source for sites like DistroWatch, hosting IT infrastructure, and helping people create websites. It’s a lot of fun!
Who or what are your biggest influences as a developer?
You’re probably looking for a hero figure or an idol, or some guiding drive? Honestly almost all of my motivation comes from one of two things: either I have a need for a tool, or I see something cool that I want to be involved in. Most of my development efforts come from wanting to make my life easier or make my job more efficient. Or I find myself thinking, “You know what I’d love to play - an attack submarine simulator!” Or I find myself looking for a way to download and extract information from new Linux distro ISOs and several hours later I’m knee deep in a weird mess of scripts that are tangled together to gather all the data I want.
What's an opinion you have that most people don't agree with?
I have a few, I suppose. One of the big ones I run into a lot is I’m not interested in how old a tool or piece of software is, I’m only interested in how well it works for me. I regularly run into people - developers, users, sysadmins - who are enthusiastic about a new admin tool, software language, or development style. They’re so intent on “This is the new industry standard, we must do it this way with this tool!” Which is great, if it makes their lives better, but often these shifts are more disruptive than helpful - especially if they are an early adopter. Other people are very intent on version numbers and insist on always having the latest version of their kernel or office suite or compiler. Which, again, is great if it’s actually filling a need. However, I usually find people aren’t upgrading for any reason other than wanting to be on the cutting edge, not because the new version improves their situation.
The reverse is also true. Some people stand by older tools and software and insist that because these tools have been around longer they must be better. They often have an outlook that change is bad. While I agree with these folks that change for the sake of change isn’t helpful, I also believe we should be open to new approaches if they make our lives better.
Basically, my view is I’m okay with change if it improves my life. But I don’t think we should blindly adopt new ways of doing things without evaluating them. Nor do I think we should stay stuck in the past just because older tools have worked well enough up to this point.
What’s your most controversial programming opinion?
There are some languages I didn’t really have a use for and decided not to pursue further because they were more effort to learn than I was getting out of them, but none I strongly disliked.
Any languages in particular?
Perl comes to mind. I used it a little around the year 2000 for some server-side website code, but rarely ran into Perl (professionally or personally) in other situations. However, almost every web-based project I worked after that used PHP so I put my time and effort into learning PHP.
What is one app on your phone that you can’t live without that you think others should know about?
Hmm, I tend not to use phone apps often. I switch phone operating systems every few devices so I’m as likely to be running a Blackberry as an /e/ OS or UBports device. Which means I’m not married to any one application or set of apps. I use generic SMS and a calendar app just about every day and that is about it. I suppose the only specific app I really use consistently that isn’t generic is KDE Connect. When I’m running /e/ OS (or Google Android in the past) I really liked being able to remotely connect my phone to my computer to share files or control my media player.
If you could dictate that everyone in the world should read one book, what would it be?
That is a hard one. I love so many books, often for wildly different reasons and in different genres. The book I probably had the most fun reading was Half Asleep In Frog Pajamas by Tom Robbins. It’s unusual in that it is written in the second person. I enjoy all of Robbins’ work. However, if I had to pick a book for someone else it would probably be something from Ursula K LeGuin. Either Left Hand Of Darkness, which won multiple awards back in the 1960s and is a science fiction work that explores a world without gender; or her novel The Dispossessed which is a sci-fi story that follows a man traveling between two worlds, one which is entirely communist and one which is entirely capitalist. The story spends a lot of time showing both the good and bad parts of each system and, I think, explores how any good idea can cause problems when taken to dogmatic extremes.
If I gave you $10 million to invest in one thing right now, where would you put it?
Honestly, ten million dollars where I live would be enough for me to buy a house, pay off the debts of my close friends/family, and retire comfortably. Maybe set up funds to help friends with their art careers or educational funds for their kids. That would be my first instinct.
But if you’re thinking of something more worldly, then I’d really like to set up some small scale, community programs. Things like community gardens and mesh networks. I want to feed and connect people.
What are you currently learning?
I’ve been playing with remote tools a lot, learning weird ways I can use things like dd, rsync, and OpenSSH. From a programming point of view I’ve been spending a lot of my efforts lately porting tools I use between platforms. Each OS has its own quirks and I find it fascinating finding out how to deal with these differences.
How do you separate good project ideas from bad ones?
I guess there are a few tests I run through my head. First, has someone else already done this or started it? I tend to maintain more open source projects than I start because lots of people want to create new things, but few like to maintain anything. I’m the other way around. Give me a project that is 95% done and ask me to keep it running, fixing the compiler warnings and porting it to new platforms and I’m a happy guy. Anyway, the first thing I do if I have a project idea is look up whether someone else has already started the work and can I just modify it to suit my needs.
The second thing I do is try to get a sense of how long it will take. I usually only start projects I am fairly certain I can finish on my own in a reasonable time frame. A lot of eager developers dive into a project and open source it, hoping others will do a lot of the work. (The ‘build it and they will come’ expectation.) But usually other people don’t contribute to your project unless it’s already useful to them. So I think it’s a mistake to rely on others to help finish a project.
Third, and perhaps finally, I want to know if the project will be useful. Does it make my life easier, will I make money from it, will it teach me something new? I want to get something out of the experience.
When and why did you decide to start porting tools across Linux platforms?
I have almost always found myself running multiple operating systems. Windows and Linux in college, Linux and FreeBSD professionally, having friends who were working on something with me who ran Windows or macOS while I was running Linux at home. As a result I’ve almost always strived to use cross platform tools and port programs I’m using between the systems I’m running. I guess I started doing this around 2002 and it became a habit.
First, I have to admit a mistake on my part. After learning about the nq program, I downloaded it and gave it a try on my Linux-powered laptop. I really liked it and so checked for a FreeBSD port and didn’t find one. Maybe because “nq” is such a short, random-seeming name, or maybe I was just momentarily stupid. Anyway, I didn’t find a port and so downloaded the source to a FreeBSD box I had available and ported the nq software. In this case it was pretty easy as the author of nq did a great job keeping most aspects of the program portable. I then submitted my porting work to the FreeBSD Ports project.
A few days later I learned nq had been ported to FreeBSD already and it either didn’t come up or I didn’t see it when I searched FreshPorts for it. At first I figured my efforts were wasted, but then I realized I’d taken a slightly different approach. I’d merged in some pull requests for documentation updates, added some changes that would allow nq to run on other platforms like GNU Hurd, and so on. So I feel some people may be able to benefit from my work, even if the FreeBSD community didn’t need it. I’ve left my port up on GitHub in case someone finds it useful and I plan to continue making little tweaks and improvements to it.
Anyway, this gives a small glimpse at my usual process: check for an existing port or similar port, download the source code, and try to get it running. Sometimes this leads to finding dependencies which also need to be ported, or a reliance on system functions which are not present. In which case those functions need to be patched in or ported too.
How do you keep the port up to date with the GitHub repo?
FreeBSD has a really robust framework that handles most of the effort. Really all I need to do, in most situations (including nq) is subscribe to the GitHub repository’s news feed for new releases. Typically I’ll get a notice when a new version is published and then all I need to do is update the version information in FreeBSD’s Makefile and the FreeBSD Ports system handles the rest. There is a great guide on how to do this in the FreeBSD Porter’s Handbook . I can bump the version number, confirm the latest version builds and runs, and submit the change to be included by a Ports committer. Once someone on the Ports team signs off on it, a FreeBSD binary package is built automatically.
How does this then go from being a binary package to available via the typical "apt-get" style tools?
Basically, with most projects I've worked on, and I think it's the same with FreeBSD, the process is mostly automated. Once a new version or patch is submitted to the build servers, the project's build system automatically builds the package for every supported CPU architecture. Then successfully built packages are copied (synchronized) to the main repository server. Other repository mirrors may then grab copies of the new packages, making them available to end-users through tools like apt-get, pkg, or pacman. When a package fails to build usually the developer responsible for the package/port will get a notification so they know to fix the issue.
The exact process varies a bit from one operating system or distribution to another, but this is generally how I've seen it work.
Why did you decide to port doas?
This was one of those cases where I saw a new tool being introduced (on OpenBSD, this time) and thought “I’d really like to use that.” The doas program is smaller than alternatives like sudo and the configuration file is so easy to read. The entire doas configuration might just read “permit jesse as root” and that is all I need. The simplicity and minimalism really appealed to me as sudo offers more functionality and complexity than I want.
Anyway, because doas was so small, and someone else had already done a little porting work to get it to compile on NetBSD, I was able to get started migrating it to FreeBSD and Linux fairly easily.
Are there any overarching goals of doas that drive design or implementation?
I’d say the overall goals are to stay small, secure, and easy to configure. Alternatives tend to have a lot of feature-creep, always adding in things people want. But doas strives to just provide what the tool really needs, across multiple operating systems - it’s highly portable.
Some people have e-mailed me asking for support for one thing or another, like visible password feedback or Touch ID support. But one little feature here and another there and it adds up after a while. Unless it’s really needed new stuff doesn’t get added to doas, it is supposed to be the lean cousin to sudo.
What trade-offs have been made in doas as a consequence of these goals?
I suppose the trade off is that in staying minimal and portable, many people aren’t interested in using it. Some people really like special sudo features like sudoedit, password feedback, or third-party GUI configuration tools. Since doas stays minimal, many people will stick to the more feature-rich sudo utility.
From a development point of view I think the only negative so far has been how much compatibility code needed to be added to get doas running on multiple platforms. The doas utility runs on NetBSD, FreeBSD, GNU/Linux, illumos and macOS, apart from the original OpenBSD build. Since the original was made to leverage OpenBSD-specific features, a few of us have had to add missing functionality and work-arounds to doas. I’d prefer to have those functions built into the base OS rather than clutter up the doas code repository, but most cross-platform projects end up with a pile of miscellaneous functions after a while that are missing from one operating system or another.
What is the most challenging problem that was solved when porting doas?
As I recall, porting doas to NetBSD and FreeBSD was fairly straightforward, but Linux is just different enough there were a handful of quirks that took a while to iron out. Mostly dealing with how Linux would treat input piped from other commands and the output of doas, such as the prompt. There were a few places in the doas code early on that basically said, “If we are on Linux, do things this way. On any other platform, do it the other way.”
How do you balance your work on open-source with your day job and other responsibilities?
Luckily, my day job mostly involves working with open source. If I’m testing out a new version of a Linux distribution or trying out a neat ZFS feature, or getting doas running on FreeBSD, then I am doing my day job.
However, in all seriousness, it’s mostly a timing thing. From 9am to 6pm I’m generally wearing my “day job” hat. Then in the evening, if I have the time and motivation, I work on side projects or things that interest me that don’t have work value.
I’m very fortunate that at this point in my life not only is my hobby my job, but I don’t have many responsibilities. I have a lot of interests, but relatively few obligations so I’m able to explore what interests me most of the time.
Do you think any of your projects do more harm than good?
Wow, that is an interesting question. I like to think that none of my current projects cause any harm. And, ideally, are useful to at least a few people. I don’t think my projects like doas, SysV init, cpulimit, or the Dungeons and Dragons 3rd Edition Character Generator have hurt anyone.
D&D Character Generator: dnd3rd.sourceforge.net
I will say that, in the past, I did a few jobs in western Canada in the oil & gas industry and there were times when I questioned the ethics of making their systems run smoothly. At first it didn’t really rattle around in my mind much, I was just focused on keeping the computers running, but around the time the film An Inconvenient Truth came out higher up members of an organization I worked for got involved in climate change denial efforts. That didn’t sit well with me and I started looking for alternative places to work shortly after that.
It certainly takes some backbone to do something like this. Hats off!
Thanks. Although my next job interview was with a lawyer's office, so I'm not sure if I was sliding up or down the moral ladder at the time.
In all seriousness, there were other warning signs I should leave and I was struggling with health issues at the time. Eventually multiple factors took hold and I moved on. It was difficult, but I felt better about things once I shifted toward other work.
I want to say though that the people I worked with at those oil & gas & energy companies were top notch. I have nothing but love and respect for my previous co-workers in the IT (and other departments) at those companies. They were good people trying to do good work. The overall direction of the companies were the issue, not the amazing people I got to work along side. That made leaving harder.
There is a certain amount of "scale" that matters when assessing whether something is morally wrong. For example, an individual person can be morally good, in a small group morally neutral, as a country morally bad. I personally tend to focus on the lower scales in my life since they seem the most within my control. I think that might be why your story is so interesting. You recognized that you personally weren't doing anything wrong, but that the larger organization was and decided to opt-out. It could have been very easy for you to think "Well, I'm keeping these servers running and I'm doing a great job of it, what they decide to do with them is their problem, not mine".
That was a tricky call for me too because, on the whole, I tend to focus on what is immediately in front of me. What I am working on right now, how this impacts myself and the people immediately around me at the moment. Professionally, I'm focused on fixing this bug, running this script, adding this one feature. Personally, I'm focused on what am I doing today, what am I eating next, what is my to-do list for this afternoon? I tend not to expand my focus to the long-term very often.
When people ask me job interview questions like "Where do you see yourself in five years?" I don't have an answer. I don't think about things that way. I'm interested in today, in improving my situation right now, in doing what I can right now with what I have on hand. When my "bubble" gets larger than that it fills up with too many things, too many distractions. It unsettles me, to be honest, trying to keep track of more than that.
All of which to say, it was unusual for me to look at the big picture, work-wise. I think you're right, engineers and programmers tend to focus on making a better tool today, not what someone else might do with the tool tomorrow. A person working to build a better flying drone often won't stop to ponder whether their company is going to use it in search & rescue operations or military operations over seas.
But there have been a few places I've worked, not many (thankfully), where the company's stance or the stance of the higher ups bothered me enough to stick my head out of my bubble and ask what was going on; whether I was comfortable being a part of that machine. It's not always an easy thing to do, shifting from "Am I doing something wrong?" to "Are we doing something wrong?" Still, I think it's important to make that shift from time to time and try to answer it honestly. No one is perfect, no organization is perfect, but I think we should strive, as much as possible, for ethical behaviour in ourselves and the groups we support.
What are some examples of managerial behaviour that triggered “sticking your head out of the bubble”?
There were a few flags at a few companies I worked at. And I don't feel I should go into matching specific issues at specific companies. But I can share, in a general sense, some actions that have caused me to pause and take notice at various places I've worked, not just those in the energy sector, but in general. At one place some of the higher ups were pushing anti-climate change propaganda. At one place workers were told not to fix things when customers reported the product was breaking; instead we were told to leave things broken and try to up-sell the business's customers on newer products, even if they couldn't use them. At one business workers were strongly encouraged to invest a percentage of their income back into the company as a retirement plan, while the stock was falling.
I don't think I lasted at any of those places a full year. Partly because their ethics didn't line up with mine, but also because the climate of the company didn't match my style. We just were good fits for each other. On the other hand I've worked for and with some great people. I got to do server work for We Love Metal (the former Canadian metal magazine) who were lovely. I think I can say now that I once got to do some work for iXsystems, an open source company that sponsors a lot of open source projects and related businesses and everyone there went above and beyond to make me feel welcome. I got to do some website backend for the amazing Barnaby Graphic Design team and that was rewarding. I've been writing for DistroWatch for about ten years now and the owner and I just click really well together.
What I'm saying is I've been really lucky, my career has been mostly highlights with just a few potholes in the road.
If a new developer wanted to get started porting Linux tools how would you suggest they get started?
My three pieces of advice would be to…
Pick a tool they already use and are familiar with how it works from the end-user’s point of view. Preferably pick something small, even if it has already been ported, pick something small for practise.
Port it to a system you are familiar with, preferably one you have coded on. It’s going to be doubly hard to port something if you are learning how to compile software on the platform at the same time
Look at the examples of others. Find similar tools and look at their ports - the Makefile and patches. This will give you a better idea of how to proceed.
Where do you see open-source heading next?
I suspect open source is going to increasingly try to provide alternatives to large company products, especially on-line, centrally controlled services. In the recent past we’ve seen open source, sometimes distributed, tools come along to offer alternatives to Facebook, Twitter, iCloud, Android/iOS, and YouTube. I believe we will see more efforts along those lines. I think we’re likely to see more of these sorts of efforts to replace popular tools from giant software companies with smaller, more privacy-focused alternatives.
I suspect, perhaps hope is the better word, that we are likely to see more open source alternatives to things like Whatsapp, Spotify, and Facebook in the coming decade.
Do you have any suggestions for someone trying to make their first contribution to an open-source project?
My advice here is similar to my advice when it comes to porting. Start with a piece of software you already use. Identify something you want it to do or a bug you want fixed. Bonus points if it is a smaller project so you can get a better feel for how the code works as a whole.
Then contact the project and ask if they want your contribution and/or are willing to help. Some projects are very picky about what they merge into their code and how it is done. Talk to someone on the project before you start writing code or you may end up doing it all over again, or the patch might sit unnoticed in a mailing list thread. Engage with the project you want to help, then do the work.
Finally, remember contributing to new projects isn’t easy. You need to get accustomed to the way they do things, maybe debate the merits of your efforts, you might end up changing your code to match their style guide, or simply being ignored the first few times. Try to stay positive. This is a learning experience as much as a coding contribution so try to stay open minded and friendly. Contributing code is like dating or applying for a job, it can take a few times to match up with the right opportunity, and it’s worth it when the right match is made.
What is one question you would like to ask another open-source developer that I didn’t ask you?
What is the biggest quirk or “gotcha” in your favourite language? Every coding language has some weird, messed up quirk to it. Like PHP using zero to mean false in some situations, but the number zero in others, so the line ‘if (strpos(“Hi”, “H”) )’ returns zero which will mean false, even though the string search function will find the letter H in the word “Hi”. To properly check PHP requires the triple operator: ‘if (strpos(“Hi”, “H”) !== FALSE)’. This is maddening the first time a new developer encounters it. Or how in C the line ‘if (a = 0)’ doesn’t run because ‘a’ gets assigned the value zero, which is false. Instead we need to use ‘if (a == 0)’ which takes a while to get used to coming from some other languages.
Developers learn all sorts of these quirks in their preferred coding languages and I like to absorb their wisdom so I can avoid the same costly (and frustrating) mistakes.
I’d also be interested in hearing from other developers what their favourite distribution/copyright license is and why.
Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!