Console

Ciphey, Kratos, and ArchiveBox

Sponsorship

Ladder

Having trouble using Linkedin to get career advice and find job opportunities? Ladder is a community-driven career development platform where you have direct access to industry leaders who want to see you succeed. Come join my AMA so you can ask me anything you’d like about how I got to where I am. Just click here and sign up!


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Why not spread the word by forwarding Console to the best engineer you know?


Jobs

Want to be paid for contributing to open-source? Check out the continually updated Console job board!

We’ve got some great open-source jobs from Canonical, SUSE, Brave, and many others. But, we’re looking for more!

If you’re an employer looking to get your job in front of thousands of the best engineers on the planet, you can request to post your job here.


Projects

kratos

Kratos is a Go framework for microservices.

language: Go, stars: 15053, watchers: 406, forks: 3014, issues: 36

last commit: October 15, 2021, first commit: January 10, 2019

social: https://go-kratos.dev/

ArchiveBox

ArchiveBox is open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more…

language: Python, stars: 11943, watchers: 152, forks: 656, issues: 111

last commit: August 11, 2021, first commit: May 05, 2017

social: https://twitter.com/ArchiveBoxApp

Ciphey

Ciphey is a fully automated decryption/decoding/cracking tool, using natural language processing & artificial intelligence, along with some common sense.

language: Python, stars: 8492, watchers: 187, forks: 524, issues: 60

last commit: October 06, 2021, first commit: July 18, 2019

social: https://twitter.com/bee_sec_san

MagInkCal

MagInkCal is an E-Ink Magic Calendar that automatically syncs to Google Calendar and runs off a battery powered Raspberry Pi Zero.

language: Python, stars: 2310, watchers: 34, forks: 60, issues: 3

last commit: October 06, 2021, first commit: September 24, 2021


Console is powered by donations. We use your donations to grow the newsletter readership via advertisement. If you’d like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation 😊

Donate To Console


An Interview With Bee of Ciphey

Hey Bee! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc?

Computer science at the University of Liverpool, I started coding when I was 10 (or 11) in the summer before secondary school in the UK. My first project was actually Ciphey! But back then it was significantly worse.

I use Python and Rust primarily, I love Rust a lot and I've been using it more recently.

What's an opinion you have that most people don't agree with?

Languages, frameworks, all suck. The absolute must important thing is to write simple code. Deciding on React vs Vue is silly. Why sit there deciding when you can just go and build?

What is one app on your phone that you can’t live without that you think others should know about?

I am a very, very big fan of You Need a Budget (YNAB)! I love tracking every penny I spend and optimising my finances :)

If you could dictate that everyone in the world should read one book, what would it be?

The Alchemist by Paulo Coelho, without spoiling it it is truly a great book about achieving your wildest dreams.

If you had to suggest 1 person developers should follow, who would it be?

Hannah!

We've been friends for a few years and she always inspires me. She uses her tech skills for ✨ good ✨ and does a lot of great work. She created a startup to help dementia sufferers, she spends time helping teach other devs and more.
Every time I think about doing something in tech, I think to myself "Is this good? Would Hannah do it?"
I think there's something noble, inspiring and amazing about someone who spends her time choosing to help people instead of chasing the money.
Definitely a lil nugget to follow.

If you could teach every 12 year old in the world one thing, what would it be and why?

You do not have to love or show respect to your parents. It is your choice to make.

If I gave you $10 million to invest in one thing right now, where would you put it?

Sadly, I can't talk about a specific company because I might get done for manipulating the market

However, I am very bullish on tech for good. Even if it doesn't make the greatest return at least my money is doing good for the world.

I saw Hannah's vrcalm app from your previous answer.  Are there other "tech for good" start-ups similar to Hannah's that you would recommend people check out?

  • https://monzo.com/ for banking 

  • https://withplum.com/ for investing / saving

  • https://www.wearewild.com/ for sustainable deoderant

  • https://zevvle.com/ for mobile phone networks

  • https://honestmobile.co.uk/ for sustainable & eco-friendly mobile phone networks

I use all of them (apart from phone networks, I am stuck in a contract but I'll be going with Honest after it. Also I don't use Plum anymore, I prefer to invest manually) and I loved them all :) 

What are you currently learning?

I'm currently learning Rust (still!) and I've been reading a lot of finance stuff lately. Mostly around the 2008 market crash or cryptocurrency tokenomics.

Tokenomics generally?  Or are you looking into specific project tokenomics? 

Generally! Although I am looking into becoming a Solana developer. They use Rust, a language I know quite well! 
Charles Hoskinson said about language design:

  • Ethereum - Too abstract, every week they are trying to fix a new bug or exploit

  • Unknown crypto (?) - Not abstract enough, you need to be a maths major and expert programmer to use it well.

I think Rust solves both of these problems:

  • Security, so you don't need to spend every week solving exploits

  • Semi between readable and close to mathematical formulae. You can read the code just fine and get up to scratch in a couple of weeks.

I like that part a lot :) 

What have you been listening to lately?

Bad Blood The Final Chapter by John Carryrou, weekly updates on Lizzie Holmes' court proceedings.

How do you separate good project ideas from bad ones?

There is no bad ones, there is only bad implementations. I wrote about this here! https://skerritt.blog/make-popular-open-source-projects/

This is fantastic.  I interview a lot of open-source stuff and these tips in your post almost always make the difference between a project getting a lot of traction and getting ignored, even if the software is of the same quality (sometimes even better sadly).

Absolutely, my friend invented a netcat competitor that was super good. Way better than anything out there right now. But, he writes so esoterically you would have never realised it was a networking tool! Sadly he deleted the repo so I can't share :( But it was truly great while it lasted.

Why was Ciphey started?

Al Sweigart of "Automate the Boring Stuff" fame once wrote a book on crypto in Python https://inventwithpython.com/hacking/ 

I read the book and thought "how hard is it to make a single program that does all of this?" and that's how I did it!!

Where did the name for ciphey come from?

Basically I took Cipher and added a Y to it.

This is a mistake because I should have called it Cyphey, the English word for Cipher. I am on a quest to use English spellings in all of my code (not American). This is quite funny around Hacktoberfest when people submit PRs to "fix grammar issues" and I have to inform them, hey, you American, there's other spellings ya know???? (Reject Americanism, Embrace Tradition)

Are there any overarching goals of ciphey that drive design or implementation?

I want to get to a stage where OffSec is so annoyed by it, they ban it completely from OSCP. Some of my other tools have been banned from the exam :)

What is your typical approach to debugging issues filed in the ciphey repo?

I normally cry my eyes out, eat ice cream and beg God for forgiveness. 

Python debugging sucks, at least in Rust I have actual debugging tools!

How do you balance your work on open-source with your day job and other responsibilities?

I work in the UK so that means we don't have those weird 120 hour work weeks that most people seem to have. That means that my day looks like:

  • Wake up at 7

  • Workout, meditate, read, enjoy the sunrise

  • Work 10am to 5pm

  • 5pm onwards I'll do open source if I feel like it

Really not that big of a deal to be honest 😅

Sometimes I'll work past 5pm if I need to, but to be honest 99% of stuff I work on isn't absolutely urgent and can wait until the next day so what's the point 🤷

Also, I work at home from the same desk / room I use for relaxing so if my work time slips into my relax time I am worried my work/life balance will suffer.

Sometimes I work earlier (7am or so) if I am very bored and too injured / achey to workout, but that's because I am bored. However! I will never work past 5pm unless it is absolutely essential. That's me time and I can't take that away from me :) 

Do you think any of your projects do more harm than good?

Do you think the creator of the hammer gets upset that people use her tools to kill people? No! My tools do not harm people, I suppose you could use them to cause harm. But the majority of people like them so 🤷

And if they do cause harm, then good ? I mean, the only harm would be to a computer? Destroy all computers, let's go back to farming full time.

If you plan to continue developing ciphey, where do you see the project heading next?

I am rewriting in Rust! :) <3

Do you have any other project ideas that you haven’t started?

Kinda, if I come up with a project idea that I don't have time to work on I'll let one of my friends know so they can get the fame xoxo

You wouldn't mind giving a few to your new friends, the Console readers, would you? 😆

I think the best thing to do is to think. For example, port scanning.
People think you need to scan every port to know if it's open or not.
What if you already knew what was open before scanning? What if you knew what was open before even making a connection to the server?
Now, dear Console reader -- you might think "this is impossible, don't be silly!" and you'd be right. But, it's not 100% impossible.
You can get pretty close to knowing what ports are open before you scan. 
People look at these "impossible" ideas and think they can never come up with a solution. You don't need a 100% solution. You only need to get 30% of the way there. Everyone else thought it was impossible until you showed "hey, we can go further than this".
With name-that-hash, people thought it was impossible to tell the difference between NTLM and MD4. But it's not. There are many ways people know the difference:

  • Crack it

  • Context

It's just they never thought of applying these to code. Name-that-hash provides you with context. NTLM appears on Windows, MD4 mostly on Linux. Now you know the difference.
It's still an impossible problem, but we are 50% of the way to a solution. And that 50% is way further than anyone else has gotten.

Think about it. Do you really need to scan every port to know it's open? You do not. 

I'll leave this as an exercise to the reader, the knowledge you'll gain from questioning what's impossible and what isn't, and from realising that you only need to get 30% of the way to a solution is far greater than any project I can give you 😊

Where do you see software development heading next?

Hopefully automated enough that I can become a full time cow hugger ('farmer') and work 1 - 2 hours / week and earn a 6 figure salary.

Where do you see open-source heading next?

Hacktoberfest is soon which means every open source project on the planet will be absolutely spammed by everyone all the time for the smallest of fixes and we will spend all of our time begging GitHub to fix their process.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

  • Work on open source (I mentor people) http://discord.skerritt.blog

What are the funniest GitHub issues you’ve received while working on ciphey?

Probably the person who commented on a GitHub issue that ciphey should decrypt RSA and complained that we didn't.

Essentially we asked them:

"Let me get this straight. You want us, university students, to break the prime factorisation problem and use that to easily break RSA. And then publish that, for free, on GitHub?"

We got so many "how do you install Python?" comments we literally made a wiki page for it. 😂

We had a similar issue on Hackernews whereby someone was very confused that Ciphey can "break every single encryption known to all of humankind" and was amazed that we had published it on GitHub 😅

I swear those folks never ever click the links before commenting!

Console

Tile38, Blitsort, and Air

Sponsorship

If you, or someone you know, is interested in sponsoring the newsletter, please reach out at console.substack@gmail.com


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Why not spread the word by forwarding Console to the best engineer you know?


Jobs

Want to be paid for contributing to open-source? Check out the Console job board!

We’ve got some great open-source jobs from Canonical, SUSE, Brave, and many others. But, we’re looking for more!

If you’re an employer looking to get your job in front of thousands of the best engineers on the planet, you can request to post your job here.


Projects

tile38

Tile38 is an open source, in-memory geolocation data store, spatial index, and realtime geofence. It supports a variety of object types including lat/lon points, bounding boxes, XYZ tiles, Geohashes, and GeoJSON.

language: Go, stars: 7720, watchers: 204, forks: 474, issues: 114

last commit: October 02, 2021, first commit: March 04, 2016

social: https://tile38.com/

air

air is live reload for Go apps.

language: Go, stars: 4724, watchers: 41, forks: 317, issues: 68

last commit: July 11, 2021, first commit: October 12, 2017

replace-jquery

replace-jquery automatically finds jQuery methods from existing projects and generates vanilla js alternatives.

language: JavaScript, stars: 773, watchers: 17, forks: 25, issues: 4

last commit: September 20, 2021, first commit: August 24, 2021

social: https://twitter.com/sachinchoolur

blitsort

Blitsort is an in-place stable adaptive rotate merge sort based on quadsort.

language: C, stars: 285, watchers: 8, forks: 8, issues: 1

last commit: July 11, 2021, first commit: July 05, 2021


Console is powered by donations. We use your donations to grow the newsletter readership via advertisement. If you’d like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation 😊

Donate To Console


An Interview With Joshua Baker of Tile38

Hey Joshua! Thanks for joining us. Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like?

I'm a retired programmer currently living in Tempe, Arizona. As a child I grew up in Alaska. I taught myself to program at a young age. A friend in junior high let me mess around on his dad's old IBM PC. I would learn from the manual that came with the computer, and I made programs in Basic. I skipped college and jumped into doing programming professionally. I helped local small businesses with their websites and databases. I later started a design and development agency with a friend, which morphed into an iPhone app company. We made an app called Golfshot GPS. It was a big hit with golfers and was even featured in an Apple commercial.

I sold the business a few years later.

I now write open source software in my free time. My favorite languages are Go and C.

Wow! What was growing up in Alaska like?

I had lots of interesting experiences. My dad worked for an organization that provided resources and logistics to remote villages. He was sent out to these villages to help with town management, and me and my brother bounced around with him. The smallest and most remote place I remember was Nikolai. We lived a stones' throw from the Kuskokwim river. Fishing trips, mushing, snowmachines. So many adventures. The downside is that we moved a lot so it was hard to keep connections with childhood friends.

Why did you choose Tempe?

My wife and I chose Tempe when we started Shotzoom, the company that made Golfshot GPS. Tempe is a business friendly town in the Phoenix metro valley. It also has the Arizona State campus in the downtown area, which is the university that my wife was attending at the time.

Is Golfshot GPS the reason you were able to retire?

More or less. I wasn't initially thinking about retirement after I sold the business. I goofed around with a couple other business ideas, but at the same time I started dabbling in open source. Right away I found joy in pure coding without being bogged down by the responsibility of maintaining a business and customer expectations. So much so that open source has taken over my coding life ever since.

Since you're retired what takes up the majority of your day now? Tile38?

Tile38 is still a passion for sure, but it doesn't take up the majority of my time. During the pandemic, my wife and I bought a little land in northern Arizona to build on. A getaway from the scorching Phoenix heat. We are in the process of building an off-grid geodesic dome. Learning to build is taking most of my time.

I've heard of people building and living in these! They're fascinating. That's amazing that you're building one on your own. Do you have any resources you're using to do this?

There's no way I could do it on my own. My wife is doing much of the work with me. We also have friends and family that are pitching in.

We watched a bunch of videos on YouTube about off grid building and geodesic domes. We decided on getting a dome kit by Pacific Domes. They're well established and provide a ton of support.

What's been the most surprising thing you've learned about the process so far?

We're still very early in on the process, but county permitting is a bit perplexing. Every county is different, but ours needs a septic system permitted and installed before we break ground. Even though we plan on using composting toilets.

What's been the biggest challenge?

Right now it's just finding the time to get up north and doing the work. The property is very rural, it's about a four hour drive from our Tempe home.

Aside from dome building, what are you currently learning?

I'm spending a lot of time learning about more accurate ways to do geospatial calculations. I'm experimenting with ellipsoidal geodesics and such. I recently ported GeographicLib Go, which is a pretty sweet library that provides nanometer accuracy.

Why was Tile38 started?

A few years ago I was working on a little urban agriculture product. It had some automation features and I thought it would be helpful to have specialized software for managing the geospatial and geofencing. That software became Tile38 and it's been my biggest time suck ever since.

Are there any overarching goals of Tile38 that drive design or implementation?

My main goal right now is to keep things fun. I have no major milestones or deadlines. I'm generally driven by whatever tickles my fancy.

What trade-offs have been made in Tile38 as a consequence of these goals?

I occasionally get people trying to push me to add stuff to Tile38. I don't mind when it's an awesome feature that will help everyone. But often it's something they only need for their company, or something that's just not a good fit. It's never fun to say no, but I can't make everyone happy.

What motivates you to continue contributing to Tile38?

The people mostly. I like to hear about their positive experiences. It's just so rad to think that some dev on the other side of the planet is able to solve a hard problem because of a little code I shared on the internet.

Is Tile38 intended to eventually be monetized if it isn’t monetized already?

Tile38 is not currently being monetized. I've explored doing so in the past but the business models around open source database software are not that interesting to me. If I do choose to monetize in the future, it will probably be a modest geofencing service built on Tile38. Something simple and affordable.

If you plan to continue developing Tile38, where do you see the project heading next?

I will continue building better, faster geospatial operations. And possibly the geofencing service that I mentioned.

Console

Wirestrustee, SQLModel, and prettymaps

Sponsorship

Kasm Workspaces - The Container Streaming Platform

Linux desktops and applications running in remote, disposable Docker containers delivered straight to your browser.  Simplify secure remote access or protect the endpoint via zero-trust browser isolation.  Deploy on-prem or in the cloud. Try it now (no installs, or signups)


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Why not spread the word by forwarding Console to the best engineer you know?


Jobs

Want to be paid for contributing to open-source? Introducing…

the Console job board!

We already have some great open-source jobs from Canonical, SUSE, Brave, and many others. But, we’re looking for more!

If you’re an employer looking to get your job in front of thousands of the best engineers on the planet, you can request to post your job here.


Projects

prettymaps

prettymaps is a small set of Python functions to draw pretty maps from OpenStreetMap data. Based on osmnx, matplotlib and shapely libraries.

language: Jupyter Notebook, stars: 6728, watchers: 62, forks: 250, issues: 25,

last commit: October 01, 2021, first commit: March 05, 2021

social: https://twitter.com/marceloprates_

sqlmodel

SQLModel is a library for interacting with SQL databases from Python code, with Python objects.

language: Python, stars: 5314, watchers: 91, forks: 176, issues: 96

last commit: August 25, 2021, first commit: August 24, 2021

social: https://twitter.com/tiangolo

wiretrustee

Wirestrustee is a WireGuard-based mesh network that connects your devices into a single private network.

language: Go, stars: 1414, watchers: 28, forks: 38, issues: 20

last commit: September 27, 2021, first commit: May 01, 2021

social: https://wiretrustee.com/


Console is powered by donations. We use your donations to grow the newsletter readership via advertisement. If you’d like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation 😊

Donate To Console


An Interview With Mikhail Bragin of Wiretrustee

What is your background (where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like, etc)?

I'm from Russia.

I have a BA in Information Systems and Technologies that I received in Moscow State University of Transport.

I have 10+ years of software engineering (mostly in Java) and management experience.

The funny fact is that until the 4th year of university I didn't really like to program and I didn't even imagine I would be a software engineer. However, once I had to solve a particular problem (we started an online store with my friends) I loved programming because I discovered the power I could possess :) 

I worked with the vehicle monitoring systems back in Russia (kinda IoT but back then it wasn't a mainstream :))

In 2012 I moved to Brazil to join PSafe - Brazilian cyber security startup headquartered in Rio De Janeiro. I worked on Lockbox - a Latin American Dropbox. The key advantage was higher upload / download speeds compared to Dropbox (due to local data hosting).

After this project I continued at PSafe and led a data analytics team for Psafe's most popular mobile antivirus app (100M+ installs on Android market).

In total, I spent 3+ years in Brazil and moved to Germany in late 2015 to join Searchmetrics.

At Searchmetrics I started as a Senior Java Developer in the search data team responsible for a number of distributed services and data processing pipelines.

I quickly grew to tech lead, then to team lead and finally business unit lead responsible for the core data layer of the company mostly doing management work (budget, planning, product, vision, etc).

In March 2021 I quit Searchmetrics and dedicated all of my time to Wiretrustee.

You could check my LinkedIn profile for further description of my career steps.

I like Java and Golang. I can also program JavaScript but it is not my favorite. In fact, the Wiretrustee dashboard application I programmed myself with React and Tailwind CSS.

What's an opinion you have that most people don't agree with?

I don't recall any outstanding examples. I’m usually in favor of a good and healthy argument and by the end of the discussion in 90% of cases people either agree with me or I agree with them.

Can you think of any arguments you've had in the past and what subjects they were about?

In my experience there were basically 2 dominant types of discussions.
Within the engineering team that I led, mostly programming and architecture related. And the ones with higher management - topics like budgets, plans, global migrations etc.

More you know about the product and how things are working the easier to "win" the discussions. Sounds simple, right? I was always curious about the details and how things are working even outside my team. This helped a lot :)

What is one app on your phone that you can’t live without that you think others should know about?

I can live without my smartphone, so that also applies to any app :).  Top ones that would make my life harder if removed are Chrome and my Banking app.

I would also recommend people checking Mysterium VPN.

If you could dictate that everyone in the world should read one book, what would it be?

Predictably Irrational, Revised: The Hidden Forces That Shape Our Decisions

If you could teach every 12 year old in the world one thing, what would it be and why?

Programming. You know why :), just joking.

First, there are many kids in need and I believe being able to program could give them a way to live a better life (I mean money-wise once they are able to work)

But probably, with that amount of specialists it won't be that profitable anymore, but it is a different story :)

Second, it develops logical thinking - programmers are mostly rational people :) it would be good to have more people like that.

If I gave you $10 million to invest in one thing right now, where would you put it?

My own business. My partner and I plan to offer Wiretrustee as a managed service. ZTNA (Zero Trust Network Access) is the future! 

What are you currently learning?

How to play tennis and I have made quite a bit of progress.

Also improving my golang skills. 

What have you been listening to lately?

Audible books about growth marketing. And since I live in Berlin - techno :)

Any favorite books so far?

1 Page Marketing Plan - this one is not actually about growth marketing but generally about marketing. It is pretty generic but explains the importance of a good marketing very well.
Another one is Growth Hacker Marketing - just started :)

Any techno recommendations?

Einmusik and Monolink.

How do you separate good project ideas from bad ones? 

It depends. But, usually just asking questions to understand if the idea really makes sense, how big is the problem that is being solved and how valuable the solution could be.

Asking questions to whom?

Let me rephrase the answer. If there is someone telling me about his/her idea there could be 2 instant outcomes - I feel the spark or not. If the idea sparks something in me then I do some research (e.g if there is something similar on the market or if it solves some problem) and ask more questions to the person who introduced the idea. Even if I don't have time (because of working on my own stuff) if the idea is cool I always spend some time researching it - just because of curiosity.

Pretty much the same with my ideas. And I also write down my conclusions and findings.

Why was Wiretrustee started?

There is a reddit post about it that I published in January 2021.

Shortly speaking - I wanted to host my personal data at home with easy access and without the hassle of configuring routers, setting up servers, firewalls etc. A plug n’ play device with an easy to use mobile app (similar to Google drive or next cloud).

There were two problems - software (network layer) and hardware.

While researching the topic I discovered Wireguard and learned a lot about NAT hole punching.

Apparently, Wireguard requires some configuration, lacks peer discovery and the hole punching mechanism. Well not that it lacks, but Wireguard itself is not supposed to do it.

So the software idea was born. 

At the same time Raspberry PI released compute module 4 with a PCIe slot which allows it to connect to drives. Before, it was only possible with USB to SATA magic which is not stable.

So the hardware was born - Wiretrustee CM4 SATA board.

Unfortunately, the current semiconductor market situation makes it complicated to produce.

Where did the name for Wiretrustee come from?

I was brainstorming and at some point it came to my head. 

Wire stands for a connection and trustee is basically a party that you can trust. So Wiretrustee creates a secure trusted connection between your peers :) of course with the help of Wireguard.

Are there any overarching goals of Wiretrustee that drive design or implementation?

We have a vision - secure private networks shouldn't be so complicated.

In fact, they are complicated now.  Look what happens in corporate networks - it is a maintenance hell :)

And our mission is to offer the easiest way of creating and maintaining secure private networks. This is the ultimate goal.

What is the most challenging problem that’s been solved in Wiretrustee, so far?

I think that the most challenging part is NAT hole punching. This technique makes it possible to discover connection candidates between peers (machines) and finally establish a peer to peer connection.

Luckily there are open source technologies like WebRTC pion/ice that cover most of our cases. We adopted it in our project. Big thank you to the pion team! 

By the way WebRTC is a common technology to establish peer to peer connection for a video call for instance.

What is your typical approach to debugging issues filed in the Wiretrustee repo?

Checking code :) We didn't have many bugs filed by the community so far but we released our first beta - let's see :)

What is the release process like for Wiretrustee?

We have CI/CD implemented with GitHub actions. 

Once a new tag has been pushed the release process is triggered automatically. E. g.

git tag v0.1.0-beta.1 

git push --tags 

Tests are going to be run (actually they run on every commit), code quality checks (go lint), then artifacts are created (Linux, Mac, Windows, docker images), then artifacts are pushed to repos (Deb, brew, docker hub, etc).

Is Wiretrustee intended to eventually be monetized if it isn’t monetized already?

If so, how?

Managed version. There are quite a few things you need to deploy and manage. 

If one doesn't want to do so we will be offering a managed version running in the cloud. The only thing you would need to do is to register and account and install an agent. You can create a secure private peer to peer network in under 5 minutes. See this video. I didn't edit it yet but it showcases Wiretrustee in under 5 min :)

How do you balance your work on open-source with your day job and other responsibilities?

I quit my day job in March 2021 and since then I'm focusing on Wiretrustee. Wiretrustee is my full time job now :)

How has that been so far?  Are you enjoying it?  Is it stressful?

Was great! I could fully focus on Wiretrustee and I enjoy it a lot.
I think that working as an employee was more stressful :)

What is the best way for a new developer to contribute to Wiretrustee?

Try it out at

https://beta.wiretrustee.com

and star it on GitHub :)

Contact us via dev@wiretrustee.com if you would like to contribute.

If you plan to continue developing Wiretrustee, where do you see the project heading next?

More features (e.g. ACLs, exit Nodes), mobile clients, and scaling.

What do you mean by scaling? Where are the bottlenecks?

Well, currently there is a Management Server that is used to distribute network changes, discover new peers, etc. Also Signal Server that helps peers to negotiate connections.
While these are lightweight and don't deal with traffic they still should be horizontally scalable to distribute load and make the system more reliable.
That way if they are down - the existing Peer-to-Peer connections are not disrupted.
We are working on a solution that will leverage distributed DBs like etcd or rqlite.

What motivates you to continue contributing to Wiretrustee?

My belief in our vision and mission and of course the community interest.

Are there any other projects besides Wiretrustee that you’re working on?

Wiretrustee Sata Board. It is related but not directly. The two projects can exist independently.

However, the semiconductor market crisis has made us pause the board project.

Do you have any other project ideas that you haven’t started?

Hundreds :)

Care to elaborate?  Maybe your top 5?

Heh, that was exaggerated :). I always generate ideas and I don't remember myself not having a project on a side while doing my full time job. There is one idea worth mentioning laying in the area of search market research with some sharing economy concepts. Providing the biggest search data database with a flexible and easy to use dashboard builder to research competition, analyse market trends, discover new products etc. Most of the existing platforms are complex, not flexible and expensive.

I also have an idea in a completely different area. I was thinking to create a device that would allow you to do some tasks with kids (e.g. homework) remotely. E.g. a drawing table with a camera. So that you could see each other and share the workspace. Drawing something, solving some math tasks etc. This is handy for parents who don't see kids often either because of traveling or other reasons. I thought something like that exists, but couldn't really find anything suitable.
Additionally you could offer some learning apps that you could use together.

What are some examples of the existing platforms in the search market research space you mentioned?

The market leaders basically: Searchmetrics, Semrush, Conductor, etc.
These are good platforms but I think that the data access should be easier and more flexible. Offering the freedom of "slicing and dicing" data with a sort of interactive builder instead of fixed dashboards.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

If you like a project, or using it, just write to contributors - they will find something for you :)

Console

Miller, Gluetun, and UnTab

Sponsorship

AppSumo

Get up to $10K 💰 to list your digital product on AppSumo between now and November 17th.

AppSumo is giving away $1 million from their Black Friday marketing budget to digital product creators.

  • $1,000 bonus - to the first 400 products to go live

  • $250 bonus - to the next 2,000 products to go live

  • 10 lucky entrepreneurs can win a $10K bonus!


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Why not spread the word by forwarding Console to the best engineer you know?


Projects

miller

Miller is like awk, sed, cut, join, and sort for data formats such as CSV, TSV, tabular JSON and positionally-indexed.

language: C, stars: 4410, watchers: 63, forks: 119, issues: 100

last commit: September 23, 2021, first commit: May 03, 2015

social: https://twitter.com/__jo_ker__

gluetun

VPN client in a thin Docker container for multiple VPN providers, written in Go, and using OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.

language: Go, stars: 733, watchers: 16, forks: 82, issues: 48

last commit: September 25, 2021, first commit: February 06, 2018

social: https://github.com/qdm12

untab

UnTab is a browser extension that gives you superpowers by making it possible to search for keywords in all open tabs with just a few keystrokes.

language: JavaScript, stars: 139, watchers: 1, forks: 18, issues: 14

last commit: May 23, 2021, first commit: September 12, 2020

social: https://twitter.com/blenderskool


Console is powered by donations. We use your donations to grow the newsletter readership via advertisement. If you’d like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation 😊

Donate To Console


An Interview With Quentin McGaw of Gluetun

Hey Quentin! Thanks for taking the time. Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like?

I'm from Europe around France. I did a master in electronic engineering at Imperial College London, where I first grasped software with languages like C++ and Python, and then a master in computer science at New York University, where I learnt more concepts. I started to work on open source during my time at New York University. My involvement in open source really took off when I began hosting my own NAS and needed some programs that didn't exist. A significant part of my programming knowledge comes from open source development and reading through posts, blogs and Reddit. I worked as a backend engineer in Go for the most part in the last few years, my last job being in the ad-tech sector. I recently dropped React and Typescript for Flutter for any frontend projects, and I love using Go for anything which is not frontend.

Who or what are your biggest influences as a developer?

The Go way of thinking which made me a better developer across all other languages. Rob Pike described a few fun Go proverbs illustrating this thinking, with some applying to other languages: go-proverbs.github.io

What's an opinion you have that most people don't agree with?

I think that transferring our minds to a virtual world would be amazing, and that simulating our body and feelings would fill the gap between our current life and this virtual world. We would be able to think, grow and build things potentially a lot faster; have a much smaller carbon footprint; not worry about our physical health. Most people out there are attached to how it is now, don’t want to change, and even prefer to die. To me, this concept is just a different ‘encoding’, the data carried wouldn’t really change.

I wonder how you feel about Facebook’s “move” into the metaverse with this answer.

I don't think Mark Zuckerberg is aiming at such technology, which is, in my view, at least 100 years away.  However, if Facebook (or similar) owns this technology, they should mathematically not be able to temper with your mind. I believe we will have new complementary technologies to secure it. We solved double spending starting with Bitcoin in 2008, I'm sure we can also have mechanisms and proofs that our virtual minds can't be tempered with by then. And maybe open source will play a role in there, if we still code things by then!

What’s your most controversial programming opinion?

It's fine to repeat code to be more explicit and improve readability. Go developers are a bit forced to do so due to the lack of generics for now, but I think this is better than using strange syntactic sugar to save a few lines of code and make it harder to read.

What is one app on your phone that you can’t live without that you think others should know about?

Mixplorer. It is the best file explorer ever. You can even run a local file server on your phone!

If you could teach every 12 year old in the world one thing, what would it be and why?

Financial and tax education. If a twelve-year old knows about triangles, he or she should know the basics of finance and of the taxation system, which both have quite an impact on one’s daily adult life.

If I gave you $10 million to invest in one thing right now, where would you put it?

I have a slightly complicated password manager idea in mind that I have been designing for a few years now, but I didn't code much of it yet. I would probably use the money to buy myself time to code and patent it, and then employ developers to help develop and maintain it.

If I gave you $100 million to invest in one thing right now, where would you put it?

This is rather a large sum to invest in one single thing! Provided I can keep a tiny percentage for myself to work on open source and live life nicely, I would aim at investing first to benefit everyone, not just myself. One such investment would be in companies developing the RISC-V CPU architecture, such as SiFive. I’m quite excited to replace amd64 and arm CPU architectures with Risc-v which should be more open and lead to better knowledge sharing in the long run.

What are you currently learning?

DNSSEC! I'm working on the v2 of qmcgaw/dns and the last complicated step is to implement DNSSEC validation for the DNS server I programmed. This should be used in Gluetun soon too!

What have you been listening to lately?

I listen to Changelog's Gotime every week to know about recent Go projects and software design.

How do you separate good project ideas from bad ones?

I don't separate them, but I order them by their usefulness and ease of development. Obviously, some will stay at the bottom of the list forever. I might also start a project - good or bad - because I feel like it, although this is rather rare.

Why was Gluetun started?

I bought myself a NAS with Docker back in 2018 and wanted an easy-to-use container to connect to my public VPN service provider Private Internet Access. I developed a simple shell script and a Dockerfile, and put it up on Github with the image on Docker Hub.

Where did the name for Gluetun come from?

It was initially named private-internet-access-docker, but was due a renaming since it supported more and more VPN providers. There was a discussion thread on Github about the renaming, and I ultimately decided that Gluetun would do. It comes from 'glue' and 'tun', which means glue things together to tunnel. Tun is also the kernel module name used for VPN tunneling. The only huge drawback I didn’t foresee is auto-correcting software changing it to gluten all. the. time. I even get gluten free products ads online now.

What was the biggest inspiration for Gluetun?

The inspiration came from the Github issues and support from the community. I did not have a need to improve it for quite a long time now from a personal standpoint. It's also the most complex piece of software I have touched so far and thus it is slightly addictive to develop and improve!

Are there any overarching goals of Gluetun that drive design or implementation?

One milestone is to make Gluetun a single static binary. It started as a bunch of subprocesses handled by a Shell entrypoint.  I then rewrote the entrypoint in Go back in 2020. Since then I have been slowly replacing third party subprograms with pure Go implementations which can be cross compiled statically. Examples are tinyproxy replaced by my own Go HTTP proxy or shadowsocks-libev replaced by my own Go Shadowsocks server implementation. Wireguard was also added with its official Go implementation more recently, and I’m about to replace Unbound with my own DNS over TLS Go implementation as well.

What's the motivation for making Gluetun a single static binary?  To make it easier for people to set up and use?

Essentially yes, with the aim to have a portable and cross platform Gluetun. And for the programming journey of course.

What trade-offs have been made in Gluetun as a consequence of the design goals?

Speed of development has been slowed down since I focus on Go-only implementations and cannot use existing programs from another language. For example, I will not add a SOCKS proxy binary to the Docker image and will instead code it myself, which takes more time.

What is the most challenging problem that’s been solved in Gluetun, so far?

One challenging aspect is to have all the VPN servers IP addresses bundled in the program. This was a design decision to prevent network traffic before connecting to the VPN server. Most VPN providers give out configuration files with hostnames, which unfortunately results in a DNS request sent from your real location, usually in plaintext. Any man in the middle such as your ISP, state or typical coffee shop hacker could block, modify and monitor the DNS request, resulting in potentially serious trouble. This is especially true since most users are still unaware of encrypted DNS. Multiple iterations were done to have this, such as manually coding them in Go source files. It’s now automated, all the data is written on a JSON file which is then embedded in the Go binary program, and the end user can update the IP addresses as well in case they become outdated.

Are there any competitors or projects similar to Gluetun?

There are several of them such as haugene/docker-transmission-openvpn or dperson/openvpn-client as well as some specific to certain VPN providers like thrnz/docker-wireguard-pia. Back in 2018, I wanted something simple to use for my provider, Private Internet Access (PIA) and I was having some fun toying with Docker and VPN. I also did not examine every existing possible solution that much. However, today, Gluetun is quite far ahead of every other project when it comes to tunneling through publicly known VPN servers with maximum security and privacy.

What was the most surprising thing you learned while working on Gluetun?

Go allows for a deep interaction with the Linux system without dependency. The TUN device is created by Go and syscalls, ip routes and rules are handled by Go code, there is even an official Wireguard implementation in Go. More recently I’m taking a peak at google/nftables to replace iptables and ip6tables calls to configure the firewall. I am glad I picked this language, although I did not know about all these advantages in early 2020 when I started rewriting the program.

What is your typical approach to debugging issues filed in the Gluetun repo?

From my perspective, there are three types of bug issues. The first one represents non-critical bug issues taking a long time to fix. These usually stay as such for multiple months. The second kind consists of critical issues which I can test and reproduce; I tend to fix these within a few hours. The last kind, which represents a good chunk of all the issues, are bug issues depending on the VPN provider servers. Unfortunately I cannot reproduce them since I am only subscribed to a single VPN provider. In this case, I usually need to ask the user to run commands and get their results back.

Maybe you could use the 10 million from the above question for multiple VPN subscriptions 😊

Well perhaps! Although I feel it's also nice to exchange with the user and get them involved in the resolution of the issue, especially since I'm usually fast at responding.

What is the release process like for Gluetun?

I make a release every 15 days approximately once the latest image has been proven stable for at least 3 days and without any issues recently created by users. This 3 days wait is actually quite nice for two reasons. Firstly, minor bug fix releases are rare ever since adopting this method. Secondly, I take this wait time as a 3 days vacation away from Gluetun, to work on other things, or nothing at all. 

How are you planning to monetize Gluetun?

I'm working on a mobile app to control Gluetun and I would sell it for a few dollars. I trust the community would enjoy using it as well as spend a few dollars to support the development of Gluetun (and friends).

How do you balance your work on open-source with your day job and other responsibilities?

I code for my open source projects about 30 minutes in the morning before starting my day job, and about one to two hours in the evening. I would code on open-source about half of my week-end.

What is the best way for a new developer to contribute to Gluetun?

One should run the development container I have configured to set up your development environment for Gluetun. This is especially needed since Gluetun can only be developed on Linux for now, due to the tight integration with it. The contributor can then pick one of the Github issues, or start a Github discussion if guidelines are needed.

Where do you see the project heading next?

I need to make the Gluetun HTTP server API more robust in order to support interacting with graphical user interfaces. Because it’s self hosted, the API server version will be out of my hands. I am therefore planning on introducing a breaking v4 release with all the interface foundations for such API communication.
If I could also magically solve all the Github issues constantly hovering at a count of 40, that would be great!

What motivates you to continue contributing to Gluetun?

The complexity and uniqueness of the project, the many improvements that can be made and the community trying their best to be helpful.

Are there any other projects besides Gluetun that you’re working on?

I’m working on a Go implementation of a custom DNS over TLS and DNS over HTTPs serverwhich should also act as a stable Go library. I have another popular project qmcgaw/ddns-updater. Similarly to Gluetun, it supports multiple DNS providers and dynamically updates your IP address for us poor souls without a fixed IP address. More recently, I developed deunhealth, a tiny Go program using the official Docker Go API to restart unhealthy containers automatically.

Do you have any other project ideas that you haven’t started?

A simple HTTP server to count the number of hits on a page and return the current counter value as an SVG badge. I would like to use this for pages of my repositories on Github as well as on my blog to measure traffic in an anonymous and simple way. Integrating such a server with Prometheus and Grafana would also yield fun plots.

Where do you see software development heading next?

Trained algorithms such as Github Copilot will become more and more present in our daily development life, although they can also produce bad code which is something new we have to worry about.
On the frontend side, I believe we are finally making progress to have a single language/framework to support desktop, mobile and web altogether. One such language is Flutter. This is changing relatively fast.
I also expect backend languages to transition from older languages like C++ or Java to newer well established ones like Go or Rust. However, this transition seems to be much slower than with frontend and will likely take several years.

Where do you see open-source heading next?

Companies will definitely open increasingly more of their code to the public.  This is especially true for security and/or infrastructure oriented projects, and this is beneficial to everyone.
Individual open source programmers will not grow in number much.
It’s rather unsustainable for individual programmers already. As an example, in my situation, I dedicated months of programming to Gluetun. This caused more and more users to use it and ask for new features, and I’m lagging behind more and more. Revenue wise, I appreciate every single donor out there, but the total donations amount to less than a week of professional work. The only way around is to develop code that can be used by enterprises which can support the project with more significant means. But even this is uncommon, where most companies will just use your code without supporting the project. Most developers I know do not work on open source since it costs time and stress and doesn’t generate revenue really. Maybe a reward system similar to Github Sponsor would change this dynamic, but I am not aware of any for now.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Accept code criticism with a learning mindset. I personally love code contributions but I also tend to nitpick on a lot of things.

Console

Noise, Diffsitter, and Scylla

Sponsorship

If you, or someone you know, is interested in sponsoring the newsletter, please reach out at console.substack@gmail.com


Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Why not spread the word by forwarding Console to your friends and colleagues?


Projects

scylla

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB.

language: C++, stars: 7107, watchers: 345, forks: 832, issues: 1772

last commit: August 29, 2021, first commit: August 9, 2014

https://twitter.com/ScyllaDB

noise

noise is an opinionated, easy-to-use P2P network stack for decentralized applications, and cryptographic protocols written in Go.

language: Go, stars: 1643, watchers: 74, forks: 188, issues: 22

last commit: March 30, 2020, first commit: June 13, 2018

https://twitter.com/PerlinNetwork

diffsitter

Diffsitter is a tree-sitter based AST difftool to get meaningful semantic diffs.

language: Rust, stars: 881, watchers: 9, forks: 15, issues: 12

last commit: September 13, 2021, first commit: June 26, 2020

https://afnan.io/


🎉 Shout out to Dragos for becoming a Gold Tier Patreon subscriber this week! Thank you for your support Dragos! 🎉

Console is powered by donations. We use your donations to grow the newsletter via advertisement. If you’d like to see the newsletter reach more people, or would just like to show your appreciation for the projects featured in the newsletter, please consider a donation 😊

Donate Today


An Interview With Botond Dénes of Scylla

Hey Botond! Thanks for joining us! Let’s start with your background. Where have you worked in the past, where are you from, how did you learn how to program, what languages or frameworks do you like?

I’m from a small village in Northern Transylvania, Romania. I was fascinated by computers as a child, although I mostly just played games on them. I was introduced to programming in high-school and liked it so much that I decided I want to become a programmer.

I started my career as a web-programmer but I switched to C++ as soon as I could. I stuck to it ever since (with a minor Python de-tour).

I like to work on Linux, do backend work, and to be exposed to a wide variety of challenges, so ScyllaDB is a perfect fit for me.

What is one app on your phone that you can’t live without that you think others should know about?

I mostly use my phone for phone stuff: communicating (calls, text, chat, email). Lately I started listening to YouTube videos while doing things. I find this a very useful way to turn something dull like mowing the lawn into an opportunity to learn something or just be entertained.

Have you looked into any podcasts for this?

Not yet, so far I have only subscribed to podcast versions of some YouTube channels that I already listen to. I have YouTube Premium, so I can listen to videos in the background, and hence I didn't have the motivation to look for podcasts yet.

If you could teach every 12 year old in the world one thing, what would it be and why?

Two things come to mind that I think are sorely missing from general education: how to handle money and the local equivalent of the highway code. Even if you never drive a car, knowing how the roads “work” makes traveling that much safer for you.

What are you currently learning?

C++. I feel like even after 10 years of using it there is still much to learn about it. Especially with the new standard release cadence, which we usually adopt as soon as mainstream compilers start supporting it.

What have you been listening to lately?

I have a broad range of interests: history, cars, economy, science, space, gaming, etc. I try to give everything a slice of my limited listening/free time.

Are there any overarching goals of Scylla that drive design or implementation?

Make efficient use of modern multi-core CPUs and fast SSDs. This is what dictated the thread-per-core architecture of Scylla as well as the exclusive use of asynchronous I/O and subsequently cooperative multitasking. All of this is provided by the open-source framework Scylla is built on: seastar.

Another important goal is self-tuning, to this end Scylla employs classic control theory to self-tune things like the scheduling and I/O priority of background tasks vs. foreground tasks.

What trade-offs have been made in Scylla as a consequence of these goals?

While I don’t feel we had to make any big trade-offs, for sure we have given up comfort when opting to write Scylla in C++ using seastar’s very awkward continuation API. C++ just gained native coroutine support, so things are getting considerably better on this front.

Are there any competitors or projects similar to Scylla? If so, what were they lacking that made you consider building something new?

Our main competitor is Apache Cassandra. Scylla was in fact started as a rewrite of Apache Cassandra in C++, with the goal to provide a vastly faster and more efficient version of it, which can also scale vertically to huge machines, as well as requiring little to no tuning cycles.

At what point did Scylla stop becoming a rewrite of Cassandra and start to become its own project? I'm guessing there was a point in time when a decision was made to start providing different behaviors than Cassandra supports, so I'm curious about that point in time and what that behavior was. Or, would you still consider Scylla a rewrite?

Although started as a rewrite, providing a faster and more efficient Apache Cassandra was gradually shifted out as a goal as we approached feature parity and we started aiming higher. We developed Alternator, a DynamoDB Compatible API to provide a more performant and lower cost alternative to DynamoDB users, we developed our own CDC API because we think the one Apache Cassandra has is cumbersome to use and we are working on making cluster operations strongly consistent by making use of RAFT, just to mention a few examples.

What was the most surprising thing you learned while working on Scylla?

Just how slow it can be to compile complex C++ code. I think when looking at compile times, Scylla is very competitive with notoriously slow to compile monsters like Chromium and Firefox. This is “thanks” to our heavy use of templates and inlining to maximise performance.

What is your typical approach to debugging issues filed in the Scylla repo?

Usually somebody first triages the issue, to make sure it is a legit bug indeed. Then it is picked up by the team-lead of the relevant team and it is assigned to an engineer. But the process is not written in stone. Sometimes engineers jump on issues that are found in the code they wrote on their own initiative.

How do you balance your work on open-source with your day job and other responsibilities?

Working on the Scylla open-source project is my day job, which is a dream come true for me. I occasionally work on proprietary projects as well (like Scylla-enterprise) but the vast majority of my time is spent on Scylla open-source.

What is the best way for a new developer to contribute to Scylla?

You can look for issues on the github issue tracker labeled with the label n00b. Be sure to check out CONTRIBUTING.md first and of course familiarizing yourself with the basics doesn’t hurt either.

What motivates you to continue contributing to Scylla?

Databases are in an ever greater demand thanks to the ever increasing amount and value of data. It is great to work on something that is such a critical piece of the infrastructure of almost every company. I feel like I’m doing something meaningful. Also contributing to ScyllaDB is simply a lot of fun for me, I love the kind of challenges I’m facing, I’m never bored.

Loading more posts…