Console 43

LibrePhotos, PoliceSettlements, and CompreFace

Sponsorships

If you, or someone you know, is interested in sponsoring the newsletter, please reach out at console.substack@gmail.com


CompreFace

CompreFace is a free and open-source facial recognition system.

language: Python, stars: 900, watchers: 37, forks: 47, issues: 11

last commit: March 02, 2021, first commit: September 16, 2019

https://twitter.com/exadel

police-settlements

Police-settlements is a FiveThirtyEight/The Marshall Project effort to collect comprehensive data on police misconduct settlements from 2010–19.

language: R, stars: 107, watchers: 10, forks: 9, issues: 0

last commit: February 22, 2021, first commit: February 22, 2021

https://twitter.com/datadhrumil

librephotos

LibrePhotos is a self-hosted alternative to Google Photos.

language: Python, stars: 2114, watchers: 49, forks: 76, issues: 86

last commit: February 27, 2021, first commit: May 23, 2017

glci

glci allows you to test your Gitlab CI Pipelines changes locally using Docker.

language: JavaScript, stars: 430, watchers: 9, forks: 14, issues: 7

last commit: March 02, 2021, first commit: February 17, 2021

https://twitter.com/_damnhotuser


Help Wanted

If you’re interested in posting a help wanted ad for your project to thousands of open-source developers, send an email to console.substack@gmail.com


An Interview With Sergii of CompreFace

Hey Sergii! Let’s start with your background. Where have you worked in the past?

I’ve been working for Exadel for more than 8 years where I started as Junior Developer and am now Tech Lead/AI Practice Lead. 

You’re working on CompreFace at Exadel, why was CompreFace started?

We just formed the Artificial Intelligence and Data Science Community at Exadel and wanted to try out some interesting technologies. At the same time, Exadel was solving a problem that the company has a lot of experience with (AI) but couldn’t show it because most of our projects are under NDA. So, we were trying to find a project that was:

  1. Interesting for developers

  2. Easy to understand and show to potential clients

  3. Would add value to Exadel as a whole 

This wasn’t an easy task. We created a list of ideas and after several weeks of debates, we landed on the idea of a face recognition solution. Soon after, Exadel created the AI Practice.  One of the functions of which is to show the world Exadel’s AI solutions. We decided that the best way to show CompreFace is to open-source our facial recognition system.

Are there any overarching goals of CompreFace that drive design or implementation, and if so, what trade-offs have been made in CompreFace as a consequence of these goals?

The main goal of CompreFace is to make face recognition more accessible to a wide variety of developers. This means that CompeFace should be easy to use by non-machine learning developers and should not depend on the programming language chosen. In other words, CompreFace should work with “add face” and “recognize face” concepts. To achieve this goal CompreFace needed to:

  1. Store embeddings (the results from the neural network) in a database

  2. Provide a REST API for all face recognition operations

The choice here was to put everything in Docker containers, as it would not only allow us to meet these requirements but also to easily add more functionality and make it extendable and scalable. 

The consequence here is that even though Docker is very popular, there are some developers that don’t have it installed yet, so they will have to install Docker first. Of course, for those who already have Docker installed, CompreFace can be installed in one command.

What is the most challenging problem that’s been solved in CompreFace, so far?

At the beginning of the project, there was a period when there was no Python developer on our team, so we decided to make the Python part of the application as simple as possible and moved everything we could to a Java server. We decided to work with the database only from the Java server, but that meant we had to actually recognize faces from embeddings in Java as well.

CompreFace uses euclidean distance for face recognition, and there are tons of examples using different libraries of how to calculate it in Python. But after research, I realized that there are no examples and there is no good choice for linear algebra in Java. After many failures, I managed to implement it using the ND4J library (part of Deeplearning4j), so it has the same performance as the fastest Python implementations. The implementation can be found here.

What was the most surprising thing you learned while working on CompreFace?

Probably that face recognition is not a solved problem. Every few months there are new algorithms that come about that are better than the previous. 

For example:

Most of them are presented at major Computer Vision conferences, like CVPR and ECCV.

With things developing so quickly in this space, how do you stay up to date?

To stay on top of the trends I recommend checking the paperswithcode website, they collect all new research papers and group them by tasks.

What is the release process like for CompreFace?

We automate releases using Jenkins, so probably the release process from a development perspective is not so interesting. We just take the last docker image that went through regression testing and push it to our DockerHub.

But the release process includes not only delivering a product, but also documentation and marketing. I’m a software developer and am not as familiar with marketing stuff, but I realize that without it users won’t find our face recognition system and won’t use it. I hope this next information will be useful for developers who are thinking of starting a new project on GitHub.

Before each release, I:

  1. Prepare release notes to post in the GitHub repository

  2. Prepare a blog post for our blog

  3. Prepare posts for social media (like LinkedIn and Facebook)

  4. Prepare posts for external resources like Reddit and Hacker News

  5. Prepare an email to subscribed users.

Then on the release date, I post everything simultaneously and wait for feedback.

How do you intend on monetizing CompreFace?

At this point, our plan is to use a typical open-source monetization strategy, which is a free product with paid support and services. The offered services include integration, customization, new feature development, training, etc. To reach us you can use the Exadel contact form or email CompreFace.

What is the best way for a new developer to contribute to CompreFace?

First of all, they could contact me personally and just ask, I believe I can find a suitable task for everyone :)

There are different directions where contributors (not only developers) can help:

  1. Just use CompreFace and report ideas and bugs on GitHub

  2. Share their knowledge and experience via posting guides and articles, or just improve our documentation

  3. Create SDKs for their favorite programming language

  4. Integrate CompreFace support to other platforms like Home Assistant or DreamFactory

  5. Contribute code

  6. If you are a machine learning developer and eager to research something, we have interesting topics to dive into, just contact us, I’ll share them with you 

  7. And last, but not least, you can just give a star to our free facial recognition system on GitHub

Where do you see CompreFace heading next?

Right now we are finishing the development of new cool CompreFace features:

  1. Face detection and face verification services

  2. Plugins - age, gender, landmarks, calculator

  3. Adding scalability

  4. Support of InsightFace library and face recognition models

  5. GPU support

  6. And more 

There will be more information after the next release. 

We also received lots of feedback from users, so the next CompreFace improvements could include:

  1. Adding SDKs for the most popular languages (like Python, JS, etc)

  2. Support of ARM processors

  3. Manage face collection from UI

  4. Video support 

  5. Liveness detection (this definitely requires long research)

Any idea on the priority of the SDK support?

Yeah, the biggest priority is Python and Javascript, but it depends more on contributor availability then on our desires :)

Where do you see software development in-general heading next?

Software developers don’t like routine work, so they always try to automate it. This is also what CompreFace is about - every face recognition project requires developers to implement certain features that CompreFace already has. So, CompreFace helps simplify the work of developers. The same will be with other routine tasks, some cool solutions are coming to help automate them.

Where do you see open-source heading next?

More companies realize they have great technologies they created for themselves and it will be a win-win if they share them with everyone. This is why Exadel started to open-source its solution and there are already plenty of repositories in the Exadel GitHub account. I believe we will have even more solutions from Exadel and other companies.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

I suggest choosing a less popular and mature project, but a project that you are passionate about. Why?  

  1. You will have more joy from contributing to it

  2. It’s easier to find out how to contribute, a project owner is always very happy to find contributors

  3. You will be proud of yourself if you know that the part of the success of the project is your work

And, lastly, don’t hesitate to contact the team and ask how you can help. There are likely plenty of ways to help that you haven’t even thought about.


Like what you saw here? Tell your friends about Console and win free swag!

Refer Friends

Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.

Console 42

Unleash, nq, and Blocklists

Console Referral Program

Drop your friend’s email in this Google Form, and be entered to win an Amazon gift card when they open their first Console email!


Sponsorships

Circles for Zoom

Circles is a new way to experience Zoom.  Circles turns each participant into a movable circle on your screen to put you back in control of your desktop.  With Circles, you can now easily take notes and multi-task during your Zoom meetings.  Circles is free to use, and you'll love the flexibility and control you get over your meetings. Get Circles today!


unleash

Unleash is an open-source feature toggle service.

language: JavaScript, stars: 2996, watchers: 58, forks: 240, issues: 21

last commit: February 26, 2021, first commit: September 29, 2014

https://twitter.com/unleash_hosted

nq

nq is a Unix command line queue utility which allows the creation of very lightweight job queue systems that require no setup, maintenance, supervision, or any long-running processes.

language: C, stars: 2410, watchers: 49, forks: 51, issues: 3

last commit: February 04, 2021, first commit: July 31, 2015

https://twitter.com/leahneukirchen

blocklists

Blocklists is a shared list of problem domains people may want to block with hosts files.

language: Digital Command Language, stars: 4241, watchers: 163, forks: 326, issues: 13

last commit: February 18, 2021, first commit: May 27, 2016

https://twitter.com/jmdugan

clubhouse-py

clubhouse-py is a clubhouse API written in Python. Standalone client included.

language: Python, stars: 1345, watchers: 59, forks: 193, issues: 0

last commit: February 25, 2021, first commit: February 19, 2021

https://twitter.com/stereotype32


Help Wanted

If you’re interested in posting a help wanted ad for your project to thousands of open-source developers, send an email to console.substack@gmail.com


An Interview With Ivar of Unleash

Hey Ivar! Let’s start with your background. Where are you from, where have you worked in the past, how did you learn how to program, etc?

I am a Norwegian software engineer and have been in the industry for 11 years now. I started my career as a consultant working in the banking & insurance industry. The first few years I tried to learn as much as I could from my peers and advance my technical skills. At some point I got tired of all the bureaucracy working in this sector, and wanted to work with a product where the teams were following agile practices, experimented a lot and moved fast. To fulfill these desires I joined FINN.no the leading Online Classified Marketplace in Norway. Here I took on multiple roles over the years, ranging from Developer, Lead Developer to Architect and eventually Chief Enterprise Architect. 

What do you feel were the biggest differences between these roles?

I enjoyed both roles. As a developer I was able to focus more, and run fast. As an architect you have a much broader picture, and work more on longer term direction. I actually enjoyed both roles, but being architect for almost 5 years I feel it is good to mostly code again. And I am fortunate to have been able to transform my little side-project in to my full time job.

What are your biggest influences as a developer?

I think a combination of the Agile and DevOps movements has influenced me the most as a software developer. At FINN.no I was part of the transformation where we went from four releases per year to more than 2000 releases to production every week. The first step of this journey was to automate all our infrastructure and our deployment pipelines. It almost felt like we had implemented everything in the DevOps handbook. You get tremendous power when you combine Agile practices with the capabilities of shipping code in small batches to production all the time. This reduces your lead time and exponentially grows the number of experiments you are able to run. 

What’s your most controversial programming opinion?

Just ship it. People tend to hold on to their code for too long in my opinion, and are scared to let it loose in the wild. The faster you ship your code the faster you are able to learn. Iterate, improve, and ship again. 

What is one app on your phone that you can’t live without that you think others should know about?

The browser. I think too many  things try to be apps, when they could just do with an optimized mobile webpage instead. 

Which browser are you using?

Mostly Chrome. But I have started playing with Brave, as I like their take on privacy.

If you could dictate that everyone in the world should read one book, what would it be?

If you work in the software industry I feel it is mandatory to read The Phoenix Project”.

If I gave you $10 million to invest in one thing right now, where would you put it?

I would put 50% in my company, building Unleash. The rest I would spread across promising startups in Africa. Africa is up and coming and they have a lot of interesting startup initiatives and the market potential is huge! 

What are you currently learning?

I recently started playing around with BigQuery. Learning how I can use it to analyze hundreds of terabytes of request data is a fun night activity when the kids are at sleep. 

What resources do you use to stay up to date on software engineering?

I follow various blogs, Reddit, and Hacker News, in addition, I Google a lot. Recently Google Discover also has been providing me with good recommendations on things to read up on. 

How do you separate good project ideas from bad ones?

I collect feedback from my team, peers, friends, users. If there is a new crazy idea for Unleash I sometimes create an issue and see if anyone responds. If there are no reactions, not even an emoji, it’s a sign; either it is a fantastic idea nobody understands or more likely a bad idea.  

Who, or what was the biggest inspiration for Unleash?

Because I wanted to go faster. We had already automated our deployment pipelines all the way to production. We were doing weekly sprints, but still we were not in a position where new code was moved to production. I felt this caused multiple problems for us; we were postponing the learning opportunities.  In addition, we kept the code at separate branches, which reduced communication and increased the cost of merging the code back to the main branch. 

It became clear to me that feature toggles were the tool we were missing in our toolkit. This would essentially allow us to detach the process of moving code into production from releasing it to customers.

I actually like another project called “Togglez”. It was a nice framework for feature toggling in Java. But I wanted something a bit simpler, that could scale across all our microservices and scale to more than 10k req/s. We also needed to support both Java and node.js and wanted a service across all our microservices. 

What trade-offs have been made in Unleash as a consequence of its overarching design goals?

There are actually three overarching goals in the architecture of Unleash. And they are actually interconnected. 

  • Performance - Unleash should be super-fast. You do not want your application to synchronize reaching out to another service to check whether a feature should be enabled or disabled for a specific user request. Thus all the important logic is implemented in the SDKs. The service itself is “just a small CRUD” service with a fancy Admin UI on top. 

  • Resilience - Networks are unreliable and you don’t want your app to stop working just because you're not able to talk to the Unleash API. If the Unleash API becomes unavailable for a short amount of time, the cache in the SDK will minimize the effect. The client will not be able to get updates when the API is unavailable, but the SDK will keep running with the last known state. Sane defaults and local backups also allow the client application to start without the Unleash API being available (even though our hosted service has proven to be extremely stable over the years). 

  • Privacy - In today's world where user data is scattered all over the place. In Unleash we have decided that user data is not automatically shared from the Unleash SDK to the Service itself. Instead we take a different approach where you define your rules on the server side, we synchronize the rules back to the client and evaluation happens on a per user request basis, locally in the customers application. In order to support Single Page applications you do not want to push your entire configuration to the browser. This is the reason why we have developed the Unleash-Proxy which sits between the Unleash API end user and only exposes the evaluated feature toggle for a specific user. 

I have to ask, what is "extremely stable"?  How much downtime have you had so far?

The registered downtime over 1.5 years of hosting it for customers is 10 minutes. I think this is more luck, and the fact that AWS is really stable.

What is the most challenging problem that’s been solved in Unleash, so far?

There are a lot of small challenges we have tackled. We are asking the users to put the SDK into their business critical applications and thus it is important that the SDK is performant and does not bring along a lot of dependencies. Especially in Java, transitive dependencies used to be a problem with the shared classpath. This has led us to only bring along two dependencies: slf4j-api to support logging, and gson to deal with json parsing. Aside from those we are only using core Java modules. This leads to a bit of nasty code to perform a simple HTTP get call, but it does the job. 

What was the most surprising thing you learned while working on Unleash?

I learned more from our first few paying customers than I did from open source users over four years. 

What is your typical approach to debugging issues filed in the Unleash repo?

Pretty standard. I start debugging the case, and if it makes sense I try to write a test that triggers that exact problem.

What is the release process like for Unleash?

We use Semver and try to release a new version of Unleash every week, if it makes sense. 

How is Unleash currently monetized?

Unleash has been around as an open source product since 2015. In 2019, we expanded it into an open-core model and offer Unleash Enterprise, with additional functionality, either as a hosted service or something you can self-host in your own data center. 

How do you balance your work on open-source with your day job and other responsibilities?

I am lucky enough to be working full-time on Unleash. I would say that I probably use about 80% on the open-source version at the moment. Given that the Enterprise version also benefits from the open-source I would say this is time well spent. 

What is the best way for a new developer to contribute to Unleash?

Come join us. We welcome help of all kinds, and even try to engage both via pull-request, or more interactively in our Slack community. I have even engaged in VC with contributors to help land a PR. 

If you plan to continue developing Unleash, where do you see the project heading next?

There is so much more to do! First I want Unleash to become the de-facto open-source platform for feature management and experimentation. A key part of this is to help the team using Unleash understand whether an improvement of a feature is better than the previous version. There is also room for automation around these decisions. Why should a developer have to look at the metrics dashboard and decide whether to increase exposure to a new feature or turn it off if it is not functioning? 

Where do you see software development in-general heading next?

I don’t think I am the person that can give you an answer to where the software industry will move in general. But I strongly believe that increased awareness of users' privacy will affect how we design systems in the future. GDPR, CCPA etc is just the beginning.  Why should central services be allowed to hold all this data about people of the free world? There are a lot of interesting ideas around this topic today, solutions emerging around the blockchain and the Solid project

Where do you see open-source heading next?

I think open-source is up and coming and there is a lot of momentum going on. 

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Just do it. It’s often not that hard but make sure the project is still active and that requests are responded to. If you plan to send a pull request, it can be beneficial to open an issue request first, to verify that your planned contribution is in line with the goal of the project and that the maintainers welcome external contributions.  


Like what you saw here? Why not share it?

Share

Or, better yet, share Console!

Share Console

Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.

Console 41

PCP, Red Star, and Endgame

Console Referral Program

Be entered to win an Amazon gift card when your friend opens their first Console email!


Sponsorships

MontyCloud

MontyCloud DAY2 helps AWS customers automate their CloudOps. Cloud architects and IT Managers from fortune 500 ISV to a fortune 500 luxury goods retailer use DAY2 to manage their AWS operations across multiple accounts and regions. DAY2 helps you:

  1. Deploy & Provision: Set up highly compliant, secure, operations-ready AWS Cloud accounts and regions in just a few clicks.

  2. Secure & Govern with Bots: Gain continuous visibility as well as compliance and security assessments. Enforce preventive guardrails across your organization.

  3. Automate CloudOps: Seamlessly integrate with your existing workflows, enable routine cloud operations and self-service tasks in clicks instead of code.

You can get started with MontyCloud DAY2 for FREE!. Connect your AWS account to instantly discover all your resources, organize them into their applications context and get an instant assessment of your security and compliance posture.

Get started today!


cloudquery

Cloudquery transforms your cloud infrastructure into a SQL or Graph database for easy monitoring, governance, and security.

language: Go, stars: 1164, watchers: 37, forks: 57, issues: 8

last commit: February 13, 2021, first commit: November 18, 2020

https://twitter.com/yevgenypats

redstar-tools

redstar-tools includes several binaries from and tools for the North Korean Red Star OS.

language: Shell, stars: 192, watchers: 18, forks: 24, issues: 4

last commit: September 14, 2016, first commit: November 27, 2015

https://twitter.com/_takeshix

endgame

endgame is an AWS pentesting tool that lets you use one-liner commands to backdoor an AWS account’s resources with a rogue AWS account — or share the resources with the entire internet. This is a fork of the original repo, which was taken down by Salesforce.

language: Python, stars: 89, watchers: 2, forks: 116, issues: 0

last commit: February 16, 2021, first commit: February 07, 2021

https://twitter.com/kmcquade3

pcp

pcp is a command line peer-to-peer data transfer tool based on libp2p.

language: Go, stars: 416, watchers: 11, forks: 7, issues: 2

last commit: February 20, 2021, first commit: January 18, 2021

https://dtrautwein.eu


An Interview With Yevgeny of Cloudquery

Hey Yevgeny! let’s start with your background, where have you worked in the past, where are you from and what languages or frameworks do you like? 

I’m based at Tel-Aviv. I’ve been a software engineer and an entrepreneur with a focus on devtools and cyber-security for more than 10 years.  I’m a Gopher :) 

What’s your most controversial programming opinion?

An idea in programming is never controversial as long as you can clearly explain it :)

What is one app on your phone that you can’t live without that you think others should know about?

I guess it’s a mix - messaging (signal, telegram, whatsapp, slack) and email.

If you could dictate that everyone in the world should read one book, what would it be?

Too many good books in different areas. I love to read books that have some relation to my day-to-day work/life. If I’m building a startup I might read more business books, if I’m on a long vacation I might read adventure books, etc…

If you’re looking for a business/startup related book I’d suggest “The Lean Startup” by Eric Ries.

What resources do you use to stay up to date on software engineering?

There is no particular place but I usually work on engineering related projects (right now it is cloudquery, for example) and I just google and research relevant stuff while working.

How do you separate good project ideas from bad ones?

Usually by the traction it gets once you release it. If people use it, then it’s a good idea if not then not so much - but, it also depends on what the purpose of the project was.

Why was cloudquery started? 

Cloudquery started as an open-source framework for security, compliance and visibility of cloud infrastructure and SaaS applications. Think about osquery, but for the cloud: agentless and written in Go (no C/C++) . The cloud infrastructure security market is flooded with expensive enterprise products but actually very few high-quality open-source products/frameworks.

What are some examples of expensive enterprise products currently in the space?

Classic stuff from Palo-Alto Network, Checkpoint, and also bunch of startups. 

What is your typical approach to debugging issues filed in the cloudquery repo? 

Pretty standard - run locally and try to reproduce.

What is the release process like for cloudquery?

KISS (keep-it-simple-stupid). Using Goreleaser.

How is cloudquery eventually intended to be monetized?

We plan to have a SaaS version with various tiers. Think terraform <-> terraform cloud. 

How do you balance your work on open-source with your day job and other responsibilities?

Luckily my day job and my work on the open-source project is the same :)

What is the best way for a new developer to contribute to cloudquery?

  • Just by running/trying cloudquery and reporting any bugs or missing features you will do a great service to the project.

  • To sum-up in this order: 1) bug reports 2) feature requests 3) code-contribution

If you plan to continue developing cloudquery, where do you see the project heading next?

I see the project evolving and getting more coverage for different cloud-providers and resources. Essentially becoming the go-to open-source framework for cloud infrastructure security, compliance, and visibility. 

Where do you see software development in-general heading next?

Less run-time (virtual-machine) languages (Python/Java) and more languages like Go/Rust. 

Why do you think that is?  The inefficiencies of a virtual machine?

I think because it made sense in the past but today with power of Go there is no need to install on a host a whole virtual-machine/runtime only to be able to run a program. This makes writing, shipping, and running software, stable, quick, and efficient.

Do you think containerization has something to do with why it doesn't make sense anymore, or is it something else?

Containerisation definitely makes sense but when you run things locally as a developer there is nothing more convenient than one single binary. Also performance is a big thing as well - you can still containerize Go applications but you will get probably much better performance then JVM/Python/etc... I don't say Python/Java don’t have their place - for example in data-science, scripting Python is a great fit. But for backend/production or local applications Go is becoming a go-to language.

Where do you see open-source heading next?

I see open-source becoming king in the cloud-security infrastructure market. 

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Look at previous PRs and see what is accepted and what is not as it varies vastly between projects.


Like what you saw here? Why not share it?

Share

Or, better yet, share Console!

Share Console

Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.

Console 40

1 second, torrent-net, and Devilbox

Sponsorships

The Daily Upside

The Daily Upside is a business newsletter that covers the most important stories in business in a style that’s engaging, insightful, and fun. Written by a former investment banker, The Daily Upside delivers quality insights and surfaces unique stories you won’t read elsewhere.

It’s completely free, and you are guaranteed to learn something new every day. 


devilbox

Devilbox is a modern Docker LAMP stack and MEAN stack for local development.

language: PHP, stars: 3278, watchers: 110, forks: 436, issues: 43

last commit: December 21, 2020, first commit: October 09, 2016

https://twitter.com/everythingcli

github1s

github1s opens a GitHub repo in VS Code in 1 second. Go to https://github1s.com/microsoft/vscode to try an example out.

language: TypeScript, stars: 10571, watchers: 66, forks: 241, issues: 37

last commit: February 13, 2021, first commit: June 16, 2019

torrent-net

torrent-net was a distributed search engine using BitTorrent and SQLite.

language: C, stars: 845, watchers: 27, forks: 35, issues: 3

last commit: April 24, 2017, first commit: April 14, 2017

https://twitter.com/lmatteis

life-dashboard

life-dashboard is a low power, heads up display for every day life running on a Kindle.

language: Rust, stars: 672, watchers: 12, forks: 16, issues: 6

last commit: October 10, 2020, first commit: August 24, 2018

https://www.instagram.com/davidhampgonsalves/


Help Wanted

If you’re interested in posting a help wanted ad for your project to thousands of open-source developers, send an email to console.substack@gmail.com


An Interview With Cytopia of Devilbox

What is your background?

The very first touch points I had with code actually came from gaming. Back in those days games were shipped on CDs and in order to play them, you had to have the CD inserted in the drive at all times, even if the game was fully installed. If you lent CDs to your friends and still wanted to play them, you somehow had to trick the games. This was usually done by opening the exe file with a disassembler, finding the error message and changing the “jump if equal”  (JE) command into a ”jump if not equal” command (JNE) - or vice versa. This made the game start even if the CD was not inserted. This magic had a big impact on me. Being able to manipulate the computer to your liking by simply knowing how stuff works was a big deal.

From there I found my way into simple 2D game development: C++ on Visual Studio 6.0 and DirectDraw. I had read all the “learn C++ in 24 hours/days” tutorials and I was hyped. I continued my way to 3D games with OpenGL and eventually ended up with C and nasty pointers.

At some point I was introduced to web pages and found selfhtml.org - the largest resource during that time. Then everything continued naturally into backend development and I landed my first freelance jobs aside school. This path had been stable for a couple of years and I went on learning about Linux systems and servers - loved it.

I was really enjoying programming and also server administration, but during that time, jobs only allowed you to do either one of them and I felt very unsatisfied as both disciplines had become a great passion. I was switching back and forth between them until finally DevOps hit the market. This could not have been any better, I was finally able to combine both disciplines on a professional level.

Currently, I have a very similar feeling with DevOps and the area of security, which I’ve been digging into for a couple of years now. But, it turns out the industry is merciful and there is a thing called DevSecOps as well.

What's an opinion you have that most people don't agree with?

I myself am a big fan of automated tests and I implement them like crazy. Most people I know want their tests to finish as quickly as possible (<2min) and optimize them for time. Which, of course, is valid, especially in short living deployments, such as on Kubernetes, where you want your service fix to go live right away. For me, time of a test is not a meaningful metric - at least not for the kinds of projects I am working on at GitHub. I’d rather have thorough tests, even if they take many hours. This is also the part that eats up most of my time on the projects and, in fact, usually the tests take up the majority of code in the repository. I just want to be able to release with confidence and provide stable software (as stable as it can get, of course).

Devilbox for instance, runs more than 24 hours on Travis CI for each pull request (the actual run time is around 9 hours) and it is running all of the tests against most combinations of available services and their respective versions. Since the introduction of “credits” on Travis I usually already run out of them at the very beginning of each month, so I had to port many CI tests to GitHub Actions.

CI is also some kind of playground to me.  From time to time I take a couple of weekends to try out all kinds of ideas while neglecting the actual development. github.com/devilbox/vhost-gen for instance, which is a cli tool to generate virtual host templates for Apache or Nginx, is just fuzzing 15 minutes for each Python version during a test. It is basically throwing all kinds of random arguments to the tool for that duration and tries to produce an error (github.com/devilbox/vhost-gen/blob/master/.ci/fuzzy.sh). I was quite surprised how many times my tests have failed and how error prone my code actually was. This method has really helped to make it rock-solid.

The downside of all this, of course, is that I also must maintain all the tests and they do eventually break over time.

That fuzzing is interesting, do you have any failed test case examples?

What is one app on your phone that you can’t live without that you think others should know about?

I don’t have any particular killer apps, as I don’t use my phone too frequently - it’s pretty old and is mainly used for some mobility apps (car / bicycle sharing), hangboard training, alarm to get me going in the morning and pagerduty to keep me awake at night. I’m also still a fan of good old sms from time to time.

Hangboard? You must be a rock climber?

Not rock climbing, but just bouldering. I started a year ago and do it for fun. I like it a lot and the training/bouldering is very holistic, demands all kinds of muscles, core-strength, flexibility and most of all creativity :-)

If you could dictate that everyone in the world should read one book, what would it be?

Clean Code: A Handbook of Agile Software Craftsmanship - by Uncle Bob

If I gave you $10 million to invest in one thing right now, where would you put it?

I have a very strong “Enter” keypress and my Lenovo keyboard breaks about once a year and needs to be replaced (one can verify this by reading the comments here: youtube.com/watch?v=lN10hgl_Ts8). With that money at hand, I would ask Lenovo to design a stronger keyboard.
As I don’t have that money: In case someone from Lenovo is reading this, please make it more stable:

What are you currently learning?

As I am currently working in the field of DevOps/SRE I am mostly focusing on AWS, Terraform, Kubernetes and all kinds of automation around this area. Aside from that I’m catching up a lot with security, particularly with web application security and the whole ecosystem of tools. If I happen to not find a particular tool that fulfills my need, I usually dive all in and write it myself (mostly with Python). For instance I’ve written pwncat, because netcat was lacking some functionality I wanted for various hacking challenges: pwncat.org/ Incidentally this then turned into a full-blown project of its own. The most important things I took out of it were: threading/locking, networking, unix shell deep-dive and of course a lot of new Python skills.

What resources do you use to stay up to date on software engineering?

I usually stay sharp on reddit, twitter, hacker news and slashdot as well as a few slack/discord channels regarding DevOps, Python and security.

How do you separate good project ideas from bad ones?

I just don’t. I have so many ideas of things I could write and my computer is stuffed endlessly with unfinished projects. It is not necessarily a time waster to me, as I do learn a lot along the way, even if that project will never go live or is never used again. This is basically a method of teaching myself new stuff, which might not be the best, but somehow it does work for me.

Why was Devilbox started?

A few years back, I was working for an agency and had to work on a broad variety of different PHP projects every day. Some were just about quick and minor fixes and others were about adding full-blown features. As we were also hosting most of the projects ourselves, but had specific customer requirements (e.g.: apache vs nginx) or even totally different PHP major versions (legacy projects vs. greenfield) I was simply trying to make my everyday life easier by somehow automating this. By that time, Docker was gaining popularity and I saw the opportunity to combine learning this technology and solving my massive configuration issue locally. That’s when the very first basic version of Devilbox was born.
Oct 9th, 2016: github.com/cytopia/devilbox/commit/be1620e077b2edc087fbfc16a20ad4435bd25b67

Are there any overarching goals of Devilbox that drive design or implementation?

With Devilbox I’ve always tried to integrate as many different versions as possible, even older ones to achieve backwards compatibility with ancient projects. Over the years it went down as far as PHP 5.2.

One day however, I’ve received a feature request for PHP 1: github.com/cytopia/devilbox/issues/564
As stupid as this issue might have been, I wanted to see if it is possible. It took a whole weekend, but it was indeed possible and I’ve dockerized PHP 1: github.com/devilbox/docker-php-1.0
This version is somehow totally different from all PHP versions and after a little digging, the most near-PHP version I could find was PHP 1.99, which I’ve then also dockerized: github.com/devilbox/docker-php-1.99s
The latter is fully working, but neither of these two projects made it into the official Devilbox project. The initial issue however is still pinned to remind me of this fun challenge.

What is the most challenging problem that’s been solved in Devilbox, so far?

The most challenging part I was facing, once I initially started Devilbox, was file and directory permission synchronization between containers and the host system.
Usually services, such as Nginx or Apache run their master process as root and their child processes as some user with whatever uid/gid (example: 100:100).

Now let’s imagine your local project directory on your host system is mounted into the web server docker container and you do a file upload. The webserver will store the uploaded file with the owner/group it is running, which is most likely different to your local uid/gid. If you want to remove the file locally, you will get a permission denied error and thus forcing you to use sudo or root.

One possibility to fix this is to ensure the web server child processes run with the same uid/gid as your local host user/group. The way to go during that time was to specify the uid/gid of the web server within the Dockerfile. So, in order to set this to your local uid/gid, you had to adjust the Dockerfile (or use build args) and rebuild it to match your local setup.
This however is not very practical as I wanted to ship pre-build docker images that everybody could use without rebuilding them. To do so, it had to be achieved during Docker run-time and I was creating a rather complex entrypoint script for the web server and php images that made this dynamic.
Under the hood most Devilbox images take env variables for uid and gid and will re-create the user with these attributes. Additionally a lot of already existing files and directories within the container need to be adjusted to not lock out the services itself.
For the PHP images the entrypoint script dealing with this situation can be seen here: github.com/devilbox/docker-php-fpm/blob/master/Dockerfiles/base/data/docker-entrypoint.d/101-uid-gid.sh
Another example for the nginx image is here: github.com/devilbox/docker-nginx-mainline/blob/master/data/docker-entrypoint.d/01-uid-gid.sh
At first this seemed easier than expected, but I was facing many errors and a lot of edge cases reported by various users. I was quite busy fixing numerous reported issues on this feature. So, to get this as stable as possible I’ve created regression tests for each and every one of the reported issues and let them run during each PR. After some time, the feature has become stable and I can now change or refactor the code with confidence.
If you want to read more about the concept of synchronizing permissions in general, I’ve documented this in more detail here:
github.com/devilbox/docker-php-fpm#unsynchronized-permissions

What is your typical approach to debugging issues filed in the Devilbox repo?

If this is about debugging issues and Linux, it is very straightforward. I have endless CI checks that will give me a first hint in case something fails and I’m also able to replicate it locally 1:1, as all checks are dockerized to ensure what I do locally is done on CI in the exact same way.
When it comes to Windows and MacOS, it is more complicated as I don’t have those systems or licenses at hand and rely on other people running Devilbox on those.

What is the release process like for Devilbox?

I’m a big fan of semver and stick to it as close as it gets. Most of the time I release bug fixes when any of the CI pipelines fail - and they fail a lot for various projects (see the Devilbox organization for involved projects: github.com/devilbox/).
Even just bug fix releases are time-consuming as you want to have regression tests for them to save time in the future.
Feature requests take even longer and I need to make sure that they are fully backwards compatible and that every new feature has been documented properly in readthedocs: devilbox.readthedocs.io/.
For anything that requires manual steps after a new release, I have an updating document that describes each step in detail: github.com/cytopia/devilbox/blob/master/UPDATING.md

Is Devilbox intended to eventually be monetized if it isn’t monetized already?

I don’t have any plans to monetize it. That would most likely discourage new people from using it and I personally find it a great tool, especially for “young” developers. Besides, it is based completely on other free open-source software. I also use it myself - not as often anymore as a few years back, but still enough to keep it up, and I wouldn’t like to put the pressure on me to act on requests people have paid for. It is still a hobby project and I enjoy working on it.
From time to time I do receive some requests from people offering to pay to implement feature X or Y, but that’s not something I want or can keep up, as I still have a full-time job aside.
Nonetheless, I do have a GitHub sponsor button attached to that project and would love if people donated to it.

If you plan to continue developing Devilbox, where do you see the project heading next?

I have endless ideas, but unfortunately time is a limiting factor. The most urgent things I will have to do is to minimize code and make everything smaller, while keeping all edge-cases covered. Docker images have grown quite a lot and need a redesign, or, more specifically, I will have to create more flavors allowing for small, medium, and big images that a user can pick from.
Another urgent major feature I have to tackle is the performance on MacOS, which is still an issue in 2021. This is not a problem with Devilbox per se, but rather with MacOS’s implementation of Docker. There are various solutions out there to overcome this, but they require some heavy lifting on my end (github.com/cytopia/devilbox/issues/105).
Also the AutoDNS feature is something that requires a major rewrite, which I’m currently working on. Once in place, it will allow me to remove all kinds of port-forwards from the PHP container, which will play nicely with minimizing code (github.com/cytopia/devilbox/issues/248).
Another feature that is currently being worked on is the separation of the Devilbox intranet from the main project. This would allow the community to create different frontends or even submit changes more easily. The frontend part has already been done by GitHub user hurrtz and is available in the develop branch here: github.com/devilbox/web-ui/tree/develop. What is left to do on my end is to create the API endpoints in the Devilbox repository itself.


Like what you saw here? Why not share it?

Share

Or, better yet, share Console!

Share Console

Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.

Console 39

watchpoints, trees, and doas

Sponsorships

The Browser

Every day, The Browser hand-picks five weird and wonderful pieces of writing for you to enjoy: try our latest free edition with selections on art, smallpox, the Dunning-Kruger effect and more. Console readers can get 20% off your first year by using the code CONSOLE20 at checkout.


tree

tree is a library for working with nested data structures in Python, open-sourced by DeepMind.

language: Python, stars: 538, watchers: 12, forks: 21, issues: 13

last commit: October 08, 2020, first commit: June 20, 2019

https://twitter.com/superbobry

doas

This repo is a port of OpenBSD’s doas which runs on FreeBSD, Linux, NetBSD, and illumos.

language: C, stars: 311, watchers: 11, forks: 24, issues: 4

last commit: February 04, 2021, first commit: June 22, 2016

watchpoints

watchpoints is an easy-to-use, intuitive variable/object monitor tool for Python that behaves similar to watchpoints in gdb.

language: Python, stars: 183, watchers: 3, forks: 4, issues: 2

last commit: February 03, 2021, first commit: December 05, 2020

https://twitter.com/viztracer

developer-roadmap

developer-roadmap contains a roadmap to becoming a web developer in 2021.

stars: 147525, watchers: 7668, forks: 21708, issues: 53

last commit: February 02, 2021, first commit: March 15, 2017

https://twitter.com/kamranahmedse


Help Wanted

If you’re interested in posting a help wanted ad for your project to thousands of open-source developers, send an email to console.substack@gmail.com


An Interview With Jesse Smith of DistroWatch

Hey Jesse! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?

My background - I’m from eastern Canada, a quiet, beautiful, rural corner of the world. I originally learned to program by accident when, one day I was using a Commodore 64 at school and typed “list” instead of “run” after loading a program into memory. This resulted in the instructions of the BASIC program spilling onto the screen and, being a curious young lad, I started modifying some of the lines to see if it would cause the program to work differently. It did! After that I started learning by trial and error, modifying copies of small programs and games and learning what the key words meant as I went along.

Some family members and friends were kind enough to introduce me to programming books and reference guides after that and I was well and truly down the rabbit hole and in love with the experience! I went to college to learn programming, database management, and UNIX.

I was, honestly, terrible at learning UNIX and decided to start using something similar at home so I could immerse myself in it. One of my fellow students introduced me to Linux and I was quickly hooked on it.

Sadly, I graduated around the time the dot-com bubble burst and the Y2K bug was no longer a threat, so there weren’t many programming or admin jobs available in my area. I did a series of odd jobs for a few years, working on open source projects, tinkering with Linux and BSD servers, and eventually decided to work freelance. I basically just turned my hobby into my main job.

I’ve worked in a handful of offices and for a few companies, often doing admin or tech support work. But the bulk of my career is in writing about open source for sites like DistroWatch, hosting IT infrastructure, and helping people create websites. It’s a lot of fun!

Who or what are your biggest influences as a developer?

You’re probably looking for a hero figure or an idol, or some guiding drive? Honestly almost all of my motivation comes from one of two things: either I have a need for a tool, or I see something cool that I want to be involved in. Most of my development efforts come from wanting to make my life easier or make my job more efficient. Or I find myself thinking, “You know what I’d love to play - an attack submarine simulator!” Or I find myself looking for a way to download and extract information from new Linux distro ISOs and several hours later I’m knee deep in a weird mess of scripts that are tangled together to gather all the data I want.

What's an opinion you have that most people don't agree with?

I have a few, I suppose. One of the big ones I run into a lot is I’m not interested in how old a tool or piece of software is, I’m only interested in how well it works for me. I regularly run into people - developers, users, sysadmins - who are enthusiastic about a new admin tool, software language, or development style. They’re so intent on “This is the new industry standard, we must do it this way with this tool!” Which is great, if it makes their lives better, but often these shifts are more disruptive than helpful - especially if they are an early adopter. Other people are very intent on version numbers and insist on always having the latest version of their kernel or office suite or compiler. Which, again, is great if it’s actually filling a need. However, I usually find people aren’t upgrading for any reason other than wanting to be on the cutting edge, not because the new version improves their situation.

The reverse is also true. Some people stand by older tools and software and insist that because these tools have been around longer they must be better. They often have an outlook that change is bad. While I agree with these folks that change for the sake of change isn’t helpful, I also believe we should be open to new approaches if they make our lives better. 

Basically, my view is I’m okay with change if it improves my life. But I don’t think we should blindly adopt new ways of doing things without evaluating them. Nor do I think we should stay stuck in the past just because older tools have worked well enough up to this point.

What’s your most controversial programming opinion?

I don’t have any languages I really dislike. I like COBOL, I like JavaScript, and I like PHP. I’m not saying any of these are without fault, but I like their quirks and their power. I think each of these, especially the scripting languages, get a bad reputation largely because they are accessible and a lot of newcomers write insecure or messy code in them. But I really like them. I also like C, Pascal, and Assembly. I’ve rarely run into a coding language I didn’t like.

There are some languages I didn’t really have a use for and decided not to pursue further because they were more effort to learn than I was getting out of them, but none I strongly disliked. 

Any languages in particular?

Perl comes to mind. I used it a little around the year 2000 for some server-side website code, but rarely ran into Perl (professionally or personally) in other situations. However, almost every web-based project I worked after that used PHP so I put my time and effort into learning PHP.

What is one app on your phone that you can’t live without that you think others should know about?

Hmm, I tend not to use phone apps often. I switch phone operating systems every few devices so I’m as likely to be running a Blackberry as an /e/ OS or UBports device. Which means I’m not married to any one application or set of apps. I use generic SMS and a calendar app just about every day and that is about it. I suppose the only specific app I really use consistently that isn’t generic is KDE Connect. When I’m running /e/ OS (or Google Android in the past) I really liked being able to remotely connect my phone to my computer to share files or control my media player.

If you could dictate that everyone in the world should read one book, what would it be?

That is a hard one. I love so many books, often for wildly different reasons and in different genres. The book I probably had the most fun reading was Half Asleep In Frog Pajamas by Tom Robbins. It’s unusual in that it is written in the second person. I enjoy all of Robbins’ work. However, if I had to pick a book for someone else it would probably be something from Ursula K LeGuin. Either Left Hand Of Darkness, which won multiple awards back in the 1960s and is a science fiction work that explores a world without gender; or her novel The Dispossessed which is a sci-fi story that follows a man traveling between two worlds, one which is entirely communist and one which is entirely capitalist. The story spends a lot of time showing both the good and bad parts of each system and, I think, explores how any good idea can cause problems when taken to dogmatic extremes.

If I gave you $10 million to invest in one thing right now, where would you put it?

Honestly, ten million dollars where I live would be enough for me to buy a house, pay off the debts of my close friends/family, and retire comfortably. Maybe set up funds to help friends with their art careers or educational funds for their kids. That would be my first instinct.

But if you’re thinking of something more worldly, then I’d really like to set up some small scale, community programs. Things like community gardens and mesh networks. I want to feed and connect people.

What are you currently learning?

I’ve been playing with remote tools a lot, learning weird ways I can use things like dd, rsync, and OpenSSH. From a programming point of view I’ve been spending a lot of my efforts lately porting tools I use between platforms. Each OS has its own quirks and I find it fascinating finding out how to deal with these differences.

How do you separate good project ideas from bad ones?

I guess there are a few tests I run through my head. First, has someone else already done this or started it? I  tend to maintain more open source projects than I start because lots of people want to create new things, but few like to maintain anything. I’m the other way around. Give me a project that is 95% done and ask me to keep it running, fixing the compiler warnings and porting it to new platforms and I’m a happy guy. Anyway, the first thing I do if I have a project idea is look up whether someone else has already started the work and can I just modify it to suit my needs.

The second thing I do is try to get a sense of how long it will take. I usually only start projects I am fairly certain I can finish on my own in a reasonable time frame. A lot of eager developers dive into a project and open source it, hoping others will do a lot of the work. (The ‘build it and they will come’ expectation.) But usually other people don’t contribute to your project unless it’s already useful to them. So I think it’s a mistake to rely on others to help finish a project.

Third, and perhaps finally, I want to know if the project will be useful. Does it make my life easier, will I make money from it, will it teach me something new? I want to get something out of the experience.

When and why did you decide to start porting tools across Linux platforms?

I have almost always found myself running multiple operating systems. Windows and Linux in college, Linux and FreeBSD professionally, having friends who were working on something with me who ran Windows or macOS while I was running Linux at home. As a result I’ve almost always strived to use cross platform tools and port programs I’m using between the systems I’m running. I guess I started doing this around 2002 and it became a habit.

You mentioned on Reddit that you ported nq to FreeBSD. What is the typical process like to take a GitHub repo, port it to FreeBSD, and make it available via the official repositories for an OS?

First, I have to admit a mistake on my part. After learning about the nq program, I downloaded it and gave it a try on my Linux-powered laptop. I really liked it and so checked for a FreeBSD port and didn’t find one. Maybe because “nq” is such a short, random-seeming name, or maybe I was just momentarily stupid. Anyway, I didn’t find a port and so downloaded the source to a FreeBSD box I had available and ported the nq software. In this case it was pretty easy as the author of nq did a great job keeping most aspects of the program portable. I then submitted my porting work to the FreeBSD Ports project.

A few days later I learned nq had been ported to FreeBSD already and it either didn’t come up or I didn’t see it when I searched FreshPorts for it. At first I figured my efforts were wasted, but then I realized I’d taken a slightly different approach. I’d merged in some pull requests for documentation updates, added some changes that would allow nq to run on other platforms like GNU Hurd, and so on. So I feel some people may be able to benefit from my work, even if the FreeBSD community didn’t need it. I’ve left my port up on GitHub in case someone finds it useful and I plan to continue making little tweaks and improvements to it.

Anyway, this gives a small glimpse at my usual process: check for an existing port or similar port, download the source code, and try to get it running. Sometimes this leads to finding dependencies which also need to be ported, or a reliance on system functions which are not present. In which case those functions need to be patched in or ported too.

How do you keep the port up to date with the GitHub repo?

FreeBSD has a really robust framework that handles most of the effort. Really all I need to do, in most situations (including nq) is subscribe to the GitHub repository’s news feed for new releases. Typically I’ll get a notice when a new version is published and then all I need to do is update the version information in FreeBSD’s Makefile and the FreeBSD Ports system handles the rest. There is a great guide on how to do this in the FreeBSD Porter’s Handbook . I can bump the version number, confirm the latest version builds and runs, and submit the change to be included by a Ports committer. Once someone on the Ports team signs off on it, a FreeBSD binary package is built automatically.

How does this then go from being a binary package to available via the typical "apt-get" style tools?

Basically, with most projects I've worked on, and I think it's the same with FreeBSD, the process is mostly automated. Once a new version or patch is submitted to the build servers, the project's build system automatically builds the package for every supported CPU architecture. Then successfully built packages are copied (synchronized) to the main repository server. Other repository mirrors may then grab copies of the new packages, making them available to end-users through tools like apt-get, pkg, or pacman. When a package fails to build usually the developer responsible for the package/port will get a notification so they know to fix the issue.

The exact process varies a bit from one operating system or distribution to another, but this is generally how I've seen it work.

Why did you decide to port doas?

This was one of those cases where I saw a new tool being introduced (on OpenBSD, this time) and thought “I’d really like to use that.” The doas program is smaller than alternatives like sudo and the configuration file is so easy to read.  The entire doas configuration might just read “permit jesse as root” and that is all I need. The simplicity and minimalism really appealed to me as sudo offers more functionality and complexity than I want.

Anyway, because doas was so small, and someone else had already done a little porting work to get it to compile on NetBSD, I was able to get started migrating it to FreeBSD and Linux fairly easily.

Are there any overarching goals of doas that drive design or implementation?

I’d say the overall goals are to stay small, secure, and easy to configure. Alternatives tend to have a lot of feature-creep, always adding in things people want. But doas strives to just provide what the tool really needs, across multiple operating systems - it’s highly portable.

Some people have e-mailed me asking for support for one thing or another, like visible password feedback or Touch ID support. But one little feature here and another there and it adds up after a while. Unless it’s really needed new stuff doesn’t get added to doas, it is supposed to be the lean cousin to sudo.

What trade-offs have been made in doas as a consequence of these goals?

I suppose the trade off is that in staying minimal and portable, many people aren’t interested in using it. Some people really like special sudo features like sudoedit, password feedback, or  third-party GUI configuration tools. Since doas stays minimal, many people will stick to the more feature-rich sudo utility.

From a development point of view I think the only negative so far has been how much compatibility code needed to be added to get doas running on multiple platforms. The doas utility runs on NetBSD, FreeBSD, GNU/Linux, illumos and macOS, apart from the original OpenBSD build. Since the original was made to leverage OpenBSD-specific features, a few of us have had to add missing functionality and work-arounds to doas. I’d prefer to have those functions built into the base OS rather than clutter up the doas code repository, but most cross-platform projects end up with a pile of miscellaneous functions after a while that are missing from one operating system or another.

What is the most challenging problem that was solved when porting doas?

As I recall, porting doas to NetBSD and FreeBSD was fairly straightforward, but Linux is just different enough there were a handful of quirks that took a while to iron out. Mostly dealing with how Linux would treat input piped from other commands and the output of doas, such as the prompt. There were a few places in the doas code early on that basically said, “If we are on Linux, do things this way. On any other platform, do it the other way.”

How do you balance your work on open-source with your day job and other responsibilities?

Luckily, my day job mostly involves working with open source. If I’m testing out a new version of a Linux distribution or trying out a neat ZFS feature, or getting doas running on FreeBSD, then I am doing my day job.

However, in all seriousness, it’s mostly a timing thing. From 9am to 6pm I’m generally wearing my “day job” hat. Then in the evening, if I have the time and motivation, I work on side projects or things that interest me that don’t have work value.

I’m very fortunate that at this point in my life not only is my hobby my job, but I don’t have many responsibilities. I have a lot of interests, but relatively few obligations so I’m able to explore what interests me most of the time.

Do you think any of your projects do more harm than good?

Wow, that is an interesting question. I like to think that none of my current projects cause any harm. And, ideally, are useful to at least a few people. I don’t think my projects like doas, SysV init, cpulimit, or the Dungeons and Dragons 3rd Edition Character Generator have hurt anyone.

I will say that, in the past, I did a few jobs in western Canada in the oil & gas industry and there were times when I questioned the ethics of making their systems run smoothly. At first it didn’t really rattle around in my mind much, I was just focused on keeping the computers running, but around the time the film An Inconvenient Truth came out higher up members of an organization I worked for got involved in climate change denial efforts. That didn’t sit well with me and I started looking for alternative places to work shortly after that.

It certainly takes some backbone to do something like this. Hats off!

Thanks. Although my next job interview was with a lawyer's office, so I'm not sure if I was sliding up or down the moral ladder at the time.
In all seriousness, there were other warning signs I should leave and I was struggling with health issues at the time. Eventually multiple factors took hold and I moved on. It was difficult, but I felt better about things once I shifted toward other work.
I want to say though that the people I worked with at those oil & gas & energy companies were top notch. I have nothing but love and respect for my previous co-workers in the IT (and other departments) at those companies. They were good people trying to do good work. The overall direction of the companies were the issue, not the amazing people I got to work along side. That made leaving harder.

There is a certain amount of "scale" that matters when assessing whether something is morally wrong.  For example, an individual person can be morally good, in a small group morally neutral, as a country morally bad.  I personally tend to focus on the lower scales in my life since they seem the most within my control.  I think that might be why your story is so interesting.  You recognized that you personally weren't doing anything wrong, but that the larger organization was and decided to opt-out.  It could have been very easy for you to think "Well, I'm keeping these servers running and I'm doing a great job of it, what they decide to do with them is their problem, not mine".

That was a tricky call for me too because, on the whole, I tend to focus on what is immediately in front of me. What I am working on right now, how this impacts myself and the people immediately around me at the moment. Professionally, I'm focused on fixing this bug, running this script, adding this one feature. Personally, I'm focused on what am I doing today, what am I eating next, what is my to-do list for this afternoon? I tend not to expand my focus to the long-term very often.
When people ask me job interview questions like "Where do you see yourself in five years?" I don't have an answer. I don't think about things that way. I'm interested in today, in improving my situation right now, in doing what I can right now with what I have on hand. When my "bubble" gets larger than that it fills up with too many things, too many distractions. It unsettles me, to be honest, trying to keep track of more than that.
All of which to say, it was unusual for me to look at the big picture, work-wise. I think you're right, engineers and programmers tend to focus on making a better tool today, not what someone else might do with the tool tomorrow. A person working to build a better flying drone often won't stop to ponder whether their company is going to use it in search & rescue operations or military operations over seas.
But there have been a few places I've worked, not many (thankfully), where the company's stance or the stance of the higher ups bothered me enough to stick my head out of my bubble and ask what was going on; whether I was comfortable being a part of that machine. It's not always an easy thing to do, shifting from "Am I doing something wrong?" to "Are we doing something wrong?" Still, I think it's important to make that shift from time to time and try to answer it honestly. No one is perfect, no organization is perfect, but I think we should strive, as much as possible, for ethical behaviour in ourselves and the groups we support.

What are some examples of managerial behaviour that triggered “sticking your head out of the bubble”?

There were a few flags at a few companies I worked at. And I don't feel I should go into matching specific issues at specific companies. But I can share, in a general sense, some actions that have caused me to pause and take notice at various places I've worked, not just those in the energy sector, but in general. At one place some of the higher ups were pushing anti-climate change propaganda. At one place workers were told not to fix things when customers reported the product was breaking; instead we were told to leave things broken and try to up-sell the business's customers on newer products, even if they couldn't use them. At one business workers were strongly encouraged to invest a percentage of their income back into the company as a retirement plan, while the stock was falling.
I don't think I lasted at any of those places a full year. Partly because their ethics didn't line up with mine, but also because the climate of the company didn't match my style. We just were good fits for each other. On the other hand I've worked for and with some great people. I got to do server work for We Love Metal (the former Canadian metal magazine) who were lovely. I think I can say now that I once got to do some work for iXsystems, an open source company that sponsors a lot of open source projects and related businesses and everyone there went above and beyond to make me feel welcome. I got to do some website backend for the amazing Barnaby Graphic Design team and that was rewarding. I've been writing for DistroWatch for about ten years now and the owner and I just click really well together.
What I'm saying is I've been really lucky, my career has been mostly highlights with just a few potholes in the road.

If a new developer wanted to get started porting Linux tools how would you suggest they get started?

My three pieces of advice would be to…

  1. Pick a tool they already use and are familiar with how it works from the end-user’s point of view. Preferably pick something small, even if it has already been ported, pick something small for practise.

  2. Port it to a system you are familiar with, preferably one you have coded on. It’s going to be doubly hard to port something if you are learning how to compile software on the platform at the same time

  3. Look at the examples of others. Find similar tools and look at their ports - the Makefile and patches. This will give you a better idea of how to proceed.

Where do you see open-source heading next?

I suspect open source is going to increasingly try to provide alternatives to large company products, especially on-line, centrally controlled services. In the recent past we’ve seen open source, sometimes distributed, tools come along to offer alternatives to Facebook, Twitter, iCloud, Android/iOS, and YouTube. I believe we will see more efforts along those lines. I think we’re likely to see more of these sorts of efforts to replace popular tools from giant software companies with smaller, more privacy-focused alternatives.

I suspect, perhaps hope is the better word, that we are likely to see more open source alternatives to things like Whatsapp, Spotify, and Facebook in the coming decade.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

My advice here is similar to my advice when it comes to porting. Start with a piece of software you already use. Identify something you want it to do or a bug you want fixed. Bonus points if it is a smaller project so you can get a better feel for how the code works as a whole.

Then contact the project and ask if they want your contribution and/or are willing to help. Some projects are very picky about what they merge into their code and how it is done. Talk to someone on the project before you start writing code or you may end up doing it all over again, or the patch might sit unnoticed in a mailing list thread. Engage with the project you want to help, then do the work.

Finally, remember contributing to new projects isn’t easy. You need to get accustomed to the way they do things, maybe debate the merits of your efforts, you might end up changing your code to match their style guide, or simply being ignored the first few times. Try to stay positive. This is a learning experience as much as a coding contribution so try to stay open minded and friendly. Contributing code is like dating or applying for a job, it can take a few times to match up with the right opportunity, and it’s worth it when the right match is made.

What is one question you would like to ask another open-source developer that I didn’t ask you?

What is the biggest quirk or “gotcha” in your favourite language? Every coding language has some weird, messed up quirk to it. Like PHP using zero to mean false in some situations, but the number zero in others, so the line ‘if (strpos(“Hi”, “H”) )’ returns zero which will mean false, even though the string search function will find the letter H in the word “Hi”. To properly check PHP requires the triple operator: ‘if (strpos(“Hi”, “H”) !== FALSE)’. This is maddening the first time a new developer encounters it. Or how in C the line ‘if (a = 0)’ doesn’t run because ‘a’ gets assigned the value zero, which is false. Instead we need to use ‘if (a == 0)’ which takes a while to get used to coming from some other languages.

Developers learn all sorts of these quirks in their preferred coding languages and I like to absorb their wisdom so I can avoid the same costly (and frustrating) mistakes.

I’d also be interested in hearing from other developers what their favourite distribution/copyright license is and why.


Like what you saw here? Why not share it?

Share

Or, better yet, share Console!

Share Console

Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.

Loading more posts…