This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Javalin

A simple and modern Java and Kotlin web framework.

language: Kotlin stars: 6999
repo: github.com/javalin/javalin
site: javalin.io

Toolong

A terminal application to view, tail, merge, and search log files (plus JSONL).

language: Python stars: 737
repo: github.com/Textualize/toolong
site: textualize.io

Popcorn Time

Popcorn Time is a multi-platform, free software BitTorrent client that includes an integrated media player.

language: JavaScript stars: 8490
repo: github.com/popcorn-official/popcorn-desktop
site: popcorntimeapp.netlify.app

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

ERPNext

Free and Open Source Enterprise Resource Planning (ERP)

language: Python stars: 16219
repo: github.com/frappe/erpnext
site: erpnext.com

aho

A Git implementation in AWK. This is a toy project to explore some of the internals of Git and newer features of GNU AWK.

language: Awk stars: 705
repo: github.com/djanderson/aho

Stride

Stride is an open-source C# game engine for realistic rendering and VR. The engine is highly modular and aims at giving game makers more flexibility in their development.

language: C# stars: 6007
repo: github.com/stride3d/stride
site: stride3d.net

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Stract

Stract is an open source web search engine targeted towards tinkerers and developers.

language: Rust stars: 180
repo: github.com/StractOrg/stract
site: stract.com

Rye

a Hassle-Free Python Experience

language: stars:
repo: github.com/mitsuhiko/rye
site: rye-up.com

Maybe

The OS for your personal finances

language: Rust stars: 15023
repo: github.com/maybe-finance/maybe
site: maybe.co

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

Pkl

A configuration as code language with rich validation and tooling.

language: Java stars: 1477
repo: github.com/apple/pkl
site: pkl-lang.org

Fabric

fabric is an open-source framework for augmenting humans using AI.

language: Python stars: 4403
repo: github.com/danielmiessler/fabric

WhisperKit

Swift native speech recognition on-device for iOS and macOS applications.

language: Swift stars: 773
repo: github.com/argmaxinc/WhisperKit

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Zed

Zed is a high-performance, multiplayer code editor from the creators of Atom and Tree-sitter.

language: Rust stars: 13668
repo: github.com/zed-industries/zed
site: zed.dev

OpenGFW

OpenGFW is a flexible, easy-to-use, open source implementation of GFW (Great Firewall) on Linux.

language: Go stars: 4601
repo: github.com/apernet/OpenGFW

Fish Shell

Fish is a smart and user-friendly command line shell for macOS, Linux, and the rest of the family. fish includes features like syntax highlighting, autosuggest-as-you-type, and fancy tab completions that just work, with no configuration required.

language: Rust stars: 23845
repo: github.com/fish-shell/fish-shell
site: fishshell.com

OSMnx

OSMnx is a Python package to easily download, model, analyze, and visualize street networks and other geospatial features from OpenStreetMap.

language: Python stars: 4526
repo: github.com/gboeing/osmnx
docs: osmnx.readthedocs.io/en/stable/

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

🎙️ Interview with Geoff of OSMnx - Python for Street Networks

Hey Geoff! Thanks for joining us! Let us start with your background.

I grew up in the Los Angeles suburbs. I am mostly self-taught with programming, but I took a couple Java classes in college. My PhD is in city and regional planning from UC Berkeley. While at Berkeley I taught myself Python, which I work with the most for spatial science computing. I’m currently a professor at USC, and I previously worked as a professor at Northeastern University and as a postdoc at Berkeley.

What is your favorite software tool?

No specific favorite, but most of my research tools revolve around the scientific Python stack, and particularly the geospatial science ecosystem. That includes tools like GeoPandas, Shapely, Rasterio, Cartopy, PySAL, and so forth.

Why was OSMnx started?

I started it as the toolkit underlying my doctoral dissertation (which analyzed US street networks’ geometric and topological properties).

How does OSMnx work?

OSMnx is a Python package to easily download, model, analyze, and visualize street networks and other geospatial features from OpenStreetMap. You can download and model walking, driving, or biking networks with a single line of code then analyze and visualize them. You can just as easily work with urban amenities/points of interest, building footprints, transit stops, elevation data, street orientations, speed/travel time, and routing.

Using OSMnx’s graph module, you can retrieve any OpenStreetMap spatial network data (such as streets, paths, rail, canals, etc) from the Overpass API and model them as NetworkX MultiDiGraphs (that’s a directed multigraph with possible self-loops and parallel edges). Using OSMnx’s features module, you can search for and download OpenStreetMap geospatial features (such as building footprints, grocery stores, schools, public parks, transit stops, etc) from Overpass API as a GeoPandas GeoDataFrame. This uses OpenStreetMap tags to search for matching elements. OSMnx can also geocode queries for you.

Once you’ve got a model, you can find nearest nodes/edges to coordinates, solve shortest paths, calculate network statistics, plot networks, and more.

Why Python?

1. Because the existing geospatial and network science ecosystems are rich to build upon

2. Because it’s a standard language for urban planners, engineers, and geographers to work with

3. Because it has a low barrier to entry for new non-technical audiences to get started using tools built in Python

Where did the name for OSMnx come from?

OpenStreetMap (OSM) + NetworkX (nx)

Who, or what was the biggest inspiration for OSMnx?

I wrote a paper about that! https://geoffboeing.com/publications/right-tools-for-job/

Are there any overarching goals of OSMnx that drive design or implementation? If so, what trade-offs have been made in OSMnx as a consequence of these goals?

Well the central goal is to be easy to use so that it “just works.” The original goal was to make something so dead simple that anyone could figure it out on Day 1 of learning Python. What that means in practice is you just need one line of pretty intuitive code to completely download and model the street network (or other infrastructure) of any city in the world. Then just 1 more line of code to calculate a large suite of statistics and indicators, or to visualize the network. The tradeoff has been building out a lot of under-the-hood code to handle a lot of different possibilities to make it all work relatively seamlessly.

What is the most challenging problem that’s been solved in OSMnx, so far? 

I talk about this in the paper I linked earlier (regarding the project’s inspiration): there’s lots of data on OpenStreetMap, but traditionally it was difficult or required a bunch of ad hoc code to turn it all into a graph-theoretic model for network analytics. OSMnx makes it automatic and theoretically-sound (from an urban/transport geography perspective, and from a network science perspective).

In particular, one thing OSMnx does automatically for you is “correct” the network topology. Its simplification module automatically processes the network’s topology from the original raw OpenStreetMap data, such that nodes represent intersections/dead-ends and edges represent the street segments that link them. This takes two primary forms: graph simplification and intersection consolidation.

Graph simplification cleans up the graph’s topology so that nodes represent intersections or dead-ends and edges represent street segments. This is important because in OpenStreetMap raw data, ways comprise sets of straight-line segments between nodes: that is, nodes are vertices for streets’ curving line geometries, not just intersections and dead-ends. OSMnx simplifies this topology by eliminating non-intersection/dead-end nodes while retaining the complete true edge geometry as an attribute on the simplified edge. This has the side benefit of making most network algorithms run much faster, because most scale with the node count.

Intersection consolidation is important because many real-world street networks feature complex intersections and traffic circles, resulting in a cluster of graph nodes where there is really just one true intersection as we would think of it in transportation engineering or urban design. Similarly, divided roads are often represented by separate centerline edges: the intersection of two divided roads thus creates 4 nodes, representing where each edge intersects a perpendicular edge, but these 4 nodes represent a single intersection in the real world. OSMnx can consolidate such complex intersections into a single node and optionally rebuild the graph’s edge topology accordingly.

What were the existing projects lacking that made you consider building something new?

What I needed just didn’t really exist. You can download raw OpenStreetMap data easily, but that didn’t help me build models with it. Tools like OSRM are great, but didn’t help me calculate the network statistics and indicators I was looking for. It was really a gap in the toolkit that needed filling.

Can you share some real-world use cases or applications where OSMnx has been particularly beneficial?

Pretty Maps has been a really cool downstream use of OSMnx. (Console Interview with Marcelo of Pretty Maps)

I used OSMnx for this project (https://geoffboeing.com/2019/09/urban-street-network-orientation/) studying street network orientation around the world, which got a lot of attention at the time.

I have also used it for visualizing street network “form” around the world to convey how urban planning and local geography shape urban patterns (for example, https://geoffboeing.com/2017/01/square-mile-street-network-visualization/)

I also liked this study from last year that built on some of my work: nature.com/articles/s41586-022-04486-7

Is OSMnx intended to eventually be monetized if it isn’t monetized already?

Nope, despite my wife’s protestations, this is my gift to the world.

What are you most proud of?

Honestly I’m just proud to have made something that other people find useful. I hate the tendency to endless reinvent the wheel in academic spatial science projects, and I think OSMnx has saved a lot of people a lot of time, while opening up new research avenues to lots of folks who needed a tool like this.

How do you balance your work on open-source with your day job and other responsibilities?

One of the nice things about being a professor is you get to choose how you spend your time (as long as you meet the job requirements of teaching X number of classes each semester and publishing Y number of papers to earn tenure). In many ways, supporting this open-source project *IS* my day job because it’s the toolkit that underlies most of my empirical and theoretical research.

Have you ever experienced burnout? How did you deal with it?

Constantly. As an academic, it’s just part of the landscape. I like to joke with my PhD students that being a professor means you have to work 70 hours a week… but you get to pick which 70 hours. Fortunately, I get winter break and summers off, so I can relax and go hiking in the San Gabriel Mountains and traveling for extended periods of time.

What is the best way for a new developer to contribute to OSMnx?

I keep the issue board pretty clean on GitHub, but there are always a few things open. I’m always looking for help with those. In particular right now, folks with experience in adding type annotations to existing codebases would be much appreciated.

Where do you see the project heading next?

I’m hoping to release v2 sometime in 2024. It will add type annotations and offer a new streamlined API that removes a few old deprecated ways of doing things in favor of leaner, more efficient ways.

Are there any other projects besides OSMnx that you’re working on?

I am also on the executive team of this project: healthysustainablecities.org that builds spatial software for calculating indicators of the built environment and its accessibility and walkability in collaboration with local partners around the world.

Like the interview? Receive more interviews like these of your favorite open source projects by subscribing to the free newsletter!

Subscribe now

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

redb

An embedded key-value database in pure Rust

language: Rust stars: 2618
repo: github.com/cberner/redb
site: redb.org

Spotube

🎧 Open source Spotify client that doesn't require Premium nor uses Electron! Available for both desktop & mobile!

language: Dart stars: 14259
repo: github.com/KRTirtho/spotube
site: spotube.krtirtho.dev

Street Complete

StreetComplete is an easy-to-use editor of OpenStreetMap data available for Android. It can be used without any OpenStreetMap-specific knowledge. It asks simple questions, with answers directly used to edit and improve OpenStreetMap data

language: Kotlin stars: 3492
repo: github.com/streetcomplete/StreetComplete
f-droid: f-droid.org/packages/de.westnordost.streetcomplete/

Planify

Task manager with Todoist support designed for GNU/Linux 🚀

language: Vala stars: 2570
repo: github.com/alainm23/planify

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

🎙️ Interview with Alain of Planify - Task manager with Todoist support designed for GNU/Linux

Hey Alain! Thanks for joining us! Let us start with your background.

Hi. I’m Alain. I'm a nerdy Peruvian guy living in Cusco. I love technology in every form and medium, learning new things every day, and constantly tackling new challenges. I started programming when I was 14 years old and currently I work as a frontend developer. I'm also an open source indie developer. I love music, spending time with my family and creating new things all the time.

Who or what are your biggest influences as a developer?

I don't think I have an influence at all, since I was a child I always liked to create and have things in order, and being a developer allows me to express it.

What’s your most controversial programming opinion?

I like to program and keep things simple, this makes my code look like it was written by a beginner, I like to use if else instead of switch, I avoid using do while, and I have a fascination for formatting my code without using Prettier.

What is your favorite software tool?

I currently use Fedora as operating system, VS Code as code editor, Gnome Calendar to have all my meetings synchronized and Planify and Todoist as task manager.

Why was Planify started?

Planify was created more than 5 years ago and was very much inspired by the design lines of the elementary OS project which I was a contributor to at some point. I created it as a necessity when trying to find a task manager for Linux that was up to the task, had a nice design and good functionalities. Planify was my first interaction with indie development and the use of Vala and Gtk.

How does Planify work?

Planify is a native desktop application, written in Vala, Gtk4 and libadwaita, uses a SQLite database to store the tasks and is synchronized with Todoist. so as not to have the app isolated on the computer.

Where did the name for Planify come from?

Planify was initially called Planner, and the name was kept until version 3, there was always discussion as there was a Gnome project with the same name (Gnome Planner). For version 4, it was decided to rewrite the application from scratch migrating to Gtk4 and libadwaita apply to Gnome Circle and change the name to Planify.

Who, or what was the biggest inspiration for Planify?

Planify is inspired by several popular task managers such as Things 3, Todoist, Planny, Tasks. From which we have taken a bit of each, both in design and functionality.

What is the most challenging problem that’s been solved in Planify, so far?

Planify has several design and development achievements that I was very proud to be able to implement.

1. Open a task detail: Initially the tasks are shown in one line, when you click on it, it expands and shows the full detail in a beautiful animation and superfluid. This requires a lot of coordination to reveal and hide elements, its implementation is simple but I think unique in a Linux app.

2. Calendar event display: This feature allows you to view calendar events in Today and Upcoming views. It syncs with the default calendar app in Gnome. 

3. Drag and drop and magic button: The drag and drop functionality feels natural and the magic button allows you to add tasks where you want and in the order you want. 

What was the most surprising thing you learned while working on Planify?

Planify was my first project in production. I started it when I was just starting my career as a developer. I think I learned a lot about design patterns, architecture, best practices, performance improvement, etc.

How did Planify get popular?

I consider Planify to be popular for its design and great attention to detail. Many of the UI elements are well thought out and friendly to what the user already knows and it syncs with Todoist which is the most popular task manager.

What is the release process like for Planify?

We don't have an official release schedule or development roadmap, mostly because I'm the only developer, but there is a long list of feature requests that I hope to land very soon.

Is Planify intended to eventually be monetized?

Planify is currently free to download and will probably remain so forever. The way to maintain the project is thanks to donations, there are not many, but knowing that the software is useful for someone and reading their thanks is the best way to motivate me.

What are you most proud of?

I liked the time when Planify exploded in popularity, I received messages of gratitude every day and there were several reviews of the app. As an anecdote the DTO of Doist (the company behind Todoist) contacted me and congratulated me for the work I did with their API to synchronize Todoist and gave me a premium account until 2030. 

How do you balance your work on open-source with your day job and other responsibilities?

Planify is not a full time project and not my only job, I take advantage of the moments I have free a couple of hours in the morning and weekends.

What is the best way for a new developer to contribute to Planify?

If you are a developer and have basic knowledge in Vala and Gtk you could help me with some bug fixes or improvements, there is a long list of feature requests to implement. If you are not a developer, don't worry, you could also help me with translations or documentation.

Where do you see the project heading next?

I would like Planify to be the best task manager for Linux, with support for multiple services and better features over time.

What motivates you to continue contributing to Planify?

It motivates me to know that many people use Planify.

Are there any other projects besides Planify that you’re working on?

Not at the moment, but I have another music player application called Byte that I would like to port to Rust and Gtk4. I think Rust is gaining a lot of ground in desktop development and its community is growing by leaps and bounds.

Where do you see software development heading next?

I like to know that many new indie developers are appearing to build many apps and create a new community, there is always something to learn and improve if it already exists.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

I started as a translator and then fixed small bugs that I found in the app I was using. I consider that independent development is a great opportunity to get involved and learn, most of these projects are maintained by only one person and the code is not so big and complex.

Loved the story of Planify? Subscribe to the free newsletter to receive interviews with more open source projects!

Subscribe now

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Shoelace

A collection of professionally designed, every day UI components built on Web standards. Works with all frameworks as well as regular HTML/CSS/JS.

language: TypeScript stars: 11081
repo: github.com/shoelace-style/shoelace
site: shoelace.style

GitUI

Blazing 💥 fast terminal-ui for git written in rust 🦀

language: Rust stars: 15984
repo: github.com/extrawurst/gitui

Celestia

A real-time space simulation that lets you experience our universe in 3D.

language: C++ stars: 1567
repo: github.com/CelestiaProject/Celestia
site: celestiaproject.space

Foliate

Read e-books in style. Foliate is an e-book reader for Linux.

language: JS stars: 4692
repo: github.com/johnfactotum/foliate
site: johnfactotum.github.io/foliate/

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

🎙️ Interview with John of Foliate, an e-book reader for Linux

Hey John! Thanks for joining us! Let us start with your background.

I became interested in building websites when I was quite young, but I never had any formal programming education and was mostly self-taught. Probably related to this, I seem to have an aversion towards complicated frameworks and prefer the more artisanal or what one might even call amateurish approach to development. I started out with PHP, but I’ve forgotten most of it at this point. Now I use JavaScript unless I have no choice, mostly simply out of familiarity, and also because of its ubiquity. But I also like how dynamic and expressive it is, particularly how easy it is to define anonymous, first-class functions.

What’s your most controversial programming opinion?

For JavaScript, I prefer four spaces for indentation and no semicolons. In my humble opinion, two space indentation is terrible. It’s too small. It makes everything unreadable. And adding semicolons requires extra keystrokes for very little benefit, and it also adds noise when reading the code. Unfortunately, I seem to be in the minority. For some reason, people who use four spaces seem to like semicolons (the classic Crockford style), and people who don’t use semicolons tend to prefer two spaces (Standard style). And then the most popular Google/Airbnb style uses two spaces and semicolons.

What is your favorite software tool?

GNOME Text Editor. Many people like to use fancy editors or IDEs, but personally I love the simplicity of working in a simple text editor.

Why was Foliate started?

I wanted to open EPUB files, but the existing projects were all lacking. I chanced upon Marta Milakovic’s blog, where she talked about the Google Summer of Code project where she was developing an e-book app for GNOME using WebKitGTK and Epub.js (as far as I know, that work was never really finished or released). So I decided to try building my own app with Epub.js in a WebView. And that’s how Foliate started.

How does Foliate work?

Foliate is a GTK app written in JavaScript, using GJS. It uses a WebKitGTK WebView for rendering the contents of the e-book. Previously, it used Epub.js, but now it has its own renderer called foliate-js, which is available as a separate library and included in Foliate as a git submodule. The renderer works similarly to Epub.js: it’s completely client-side and serves the contents in iframes with blob URLs, and it uses CSS multicolumn to handle pagination.

Why did you pick GTK?

I chose GTK mainly because I have personally been using GNOME and other desktop environments in the GTK ecosystem. GTK has bindings in many languages, and JavaScript was the one I was most familiar with.

Where did the name for Foliate come from?

The name was derived from “folio”, which means “leaf”, a term with various meanings connected to books and printing. The concept of a leaf, as illustrated in Foliate’s icon, is also related to the practice of drying and preserving leaves in books, where the leaf could also serve as a bookmark.

Are there any overarching goals of Foliate that drive design or implementation? If so, what trade-offs have been made in Foliate as a consequence of these goals?

I would say the biggest goal is simplicity, even if it means sacrificing efficiency or functionality sometimes. For example, unlike most other e-book apps, Foliate uses flat JSON files instead of a database. This makes everything dead simple, but without indexing, you can’t query the data efficiently.

What were the existing projects lacking that made you consider building something new?

I wanted something that (1) integrates well with the GNOME desktop, (2) is a “file viewer” that doesn’t require importing/copying/modifying files, (3) with a simple and distraction-free UI.

What was the most surprising thing you learned while working on Foliate?

I’m surprised at how often people (myself included) neglect to read specs carefully when implementing various standards. For example, EPUB has a spec called CFI (Canonical Fragment Identifiers). Recently, I did a quick review and discovered that most libraries and apps have incorrect CFI implementations. Another example is that I recently reimplemented OPDS support in Foliate, and I was surprised by the mistakes I made in the old version related to not reading the relevant specs. It’s easy to assume that things are correct if you’re just looking at a few examples, but sometimes even the tests could be incorrect or fail to cover some edge cases.

Is Foliate intended to be eventually monetized?

I started accepting donations recently. In the future I’m thinking that it could become pay-what-you-want similar to how elementary OS and elementary apps work today.

How did the project get popular?

It received some initial attention after it was made available on Flathub, and after that it was picked up by a couple of blogs and on Hacker News. The people behind Flathub and the GNOME ecosystem have done a terrific job enabling developers to distribute apps and reach users easily.

Have you ever experienced burnout? How did you deal with it?

Taking a break helps. But it’s easy to feel restless when you remember that there are people out there using and possibly depending on the app. So I try to remind myself that per the license, the app comes with absolutely no warranty, which certainly helps me sleep better at night.

If you plan to continue developing Foliate, where do you see the project heading next?

I would like to make a version of Foliate with only Web technologies, such as a progressive web app, a WebExtension, or possibly an Electron app. This will make it available on more platforms, especially on Windows where there doesn’t seem to be many e-book apps available, particularly since the demise of EPUB support in Edge (unlike say, macOS, where there’s Apple Books).

What motivates you to continue contributing to Foliate?

Mostly because I use it myself, but I’d also like to think of it as building something that helps the dissemination of knowledge, which makes me feel like being part of something greater than merely maintaining a simple utility app.

Where do you see software development heading next?

Most answers to this question today would probably include the word “AI”. The common adage is that code is read far more often than it is written. Right now there’s a lot of emphasis on AI writing the code but far less on reading the code. It would be interesting to see how AI could be used to grapple with the increasing complexity in software.

Where do you see open-source heading next?

I think the GNOME people have worked hard to build an ecosystem, and we’re starting to see a sort of Renaissance of indie apps on the platform.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

I would say: start small. It is a capital mistake to try to bring big changes without engaging with the maintainer first. Even more often, people are discouraged from contributing due to the sheer size or complexity of the codebase. In these cases, it helps to start small and move gradually from there.

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Harlequin

The SQL IDE for your terminal.

language: Python stars: 1240
repo: github.com/tconbeer/harlequin
site: harlequin.sh

Maestro

Maestro is a lightweight Unix-like kernel written in Rust. The goal is to provide a lightweight operating system able to use the safety features of the Rust language to be reliable.

language: Rust stars: 2002
repo: github.com/llenotre/maestro
site: blog.lenot.re

ACID Chess

ACID Chess is a chess computer written in Python, which can be used with any board. By filming the board, the contour of the board is recognized, and the positions of the individual pieces can be determined.

language: Python stars: 12
repo: github.com/ierror/acid-chess

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

🎙️ Interview With Bernhard of ACID Chess - Play Chess by filming the board

Hey Bernhard! Thanks for joining us! Let us start with your background.

I live in Munich and work for a company that manufactures computer-aided dental restorations. Previously I worked in a consultancy where we supported our customers in collecting and analyzing large amounts of data. 

My father has always had computers (even in the pre-internet era). He showed me a few things in Basic and gave me one of his books. That's how I got into programming.

My favorite programming languages are Python, sometimes Perl and JavaScript - I also like C# for GUI projects at work.

What’s your most controversial programming opinion?

The way we develop software in the corporate world today often leads to unnecessarily complex systems. Clean coding, object-oriented programming or other "sacred" patterns often lead us to focus more on ourselves and our software than on the actual problem. Don't get me wrong, these patterns are absolutely legitimate. I often wish that people would think about how to structure code in such a way that it is easy to read and doesn't simply follow a pattern.

What is your favorite software tool?

Is “Linux” a valid answer? :)

Why was ACID Chess started?

I've enjoyed playing a game of chess over the board from time to time for a few years now. When discussing the games and trying to understand why a game has taken such and such a turn, beginners are often overwhelmed. At some point, I came up with the idea of simply recording the game with the camera and recognising the moves using image recognition. The sequence of moves can then be fed into an engine for analysis. Using this as a basis to build a chess computer was somehow obvious.

How does ACID Chess work?

A camera films the chessboard, which is analyzed on the basis of individual frames.

In the first step, the board contour is determined, the board is cutted, transformed and cut into 64 squares. The squares are then classified to determine the game state.

1. Recognising the board contour: In order to prevent unnecessary dirt from interfering with the recognition of the individual board squares, the outline of the board is recognised first. To do this, I trained a neural network - the "Board Segmentation Model" - the border of the board is extracted from the outline of the board using openCV. The four corner points are then determined from this line. With the help of the 4 points, the board can be cut out and a "birds eye view" of the board can be created with a 4-point perspective transformation.

2. Cutting the board into 64 squares: From the "birds eye view" image of the board, OpenCV algorithms (medianBlur, auto_canny, HoughLinesP, ...) are then used to adaptively extract nine horizontal and nine vertical lines. Once these lines have been found, the image can be cut into 64 squares.

3. The classification of the squares: As I do not have enough training data available, I have decided not to recognise the individual figures, but to train only three classes for the "Square Classification Models":

1. is the square occupied by a black piece?

2. is the square occupied by a white piece?

3. is the square free?

Assuming that the board is in the starting position, all the necessary information is available to determine the orientation of the board and to recognise the moves.

Where did the name for ACID Chess come from?

While experimenting with OpenCV and the chessboard images, I created some crazy visualizations. A mate said "it looks like a chessboard on drugs" - that's how it got its name.

What is the most challenging problem that’s been solved in ACID Chess, so far

The recognition of the board contour and the recognition of the horizontal / vertical lines to intersect the square.

https://acid-chess.readthedocs.io/en/latest/dev/neural_networks.html

What was the most surprising thing you learned while working on ACID Chess?

Previously I only had theoretical knowledge of neural networks. The level that PyTorch and other frameworks have reached today makes it possible to extend existing networks and train your own network without having to study documentation for months.

What are you most proud of?

The whole chain, starting with the crawling of board images, data cleaning, classification, training of the models, and the GUI part of the application was written completely by myself.

How do you balance your work on side projects like these with your day job and other responsibilities?

With my day job, I occasionally have to work late in the evening or at the weekend, but generally I have very regular working hours. In summer, I cycle long distances a lot, but in winter I miss that. These side projects are a good balance at winter times.

Where do you see the project heading next?

The chess computer already works very well for me and a few friends. If you're not on Apple hardware, there are still a few things to improve. I also hope to receive better training data in the future, which I can use to significantly improve the stability of the board recognition and the square classification.

Are there any other projects besides ACID Chess that you’re working on?

I'm working with a friend on a software to manage bike tours. However, this has not yet reached a release-ready status 🙂 

Where do you see software development heading next?

AI will simplify and improve many things in software development, such as vulnerability detection, code reviews and so on. Whether this leads to better products, depends, as is so often the case, on how we use the tools.

Where do you see open-source heading next?

When I look at open source projects such as Linux, FreeBSD or the Django framework, for example, I am very confident that we will continue to see good open source software. Monetising open source software will continue to be difficult for "small developers". I have no idea how this could be improved, but I hope that better ways will be found in the future. 

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Read the contribution guidelines and simply start with it. Even if you have no idea about the big picture, you can often get involved in projects through smaller issues. In my experience, the other developers are usually very helpful and are always happy to get involved.

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Inure

An elegant and beautiful premium Android app manager for both rooted and non-rooted devices with a built-in terminal, analytics panel and an independent custom theme engine.

language: Kotlin stars: 729
repo: github.com/Hamza417/Inure

Plasmic

Plasma is a visual builder for React. Build apps, websites, content and integrate with your codebase.

language: TypeScript stars: 3527
repo: github.com/plasmicapp/plasmic
site: plasmic.app

Dockge

A fancy, easy-to-use and reactive self-hosted docker compose.yaml stack-oriented manager.

language: TypeScript stars: 4897
repo: github.com/louislam/dockge
site: dockge.kuma.pet

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

BlackHole

A Music Player App made with Flutter.

language: Dart stars: 8460
repo: github.com/Sangwan5688/BlackHole
F-droid: f-droid.org/packages/com.shadow.blackhole

Twenty

Building a modern alternative to Salesforce, powered by the community.

language: TypeScript stars: 6805
repo: github.com/twentyhq/twenty
site: twenty.com

Plate

The rich-text editor for React

language: TypeScript stars: 5750
repo: github.com/udecode/plate
site: platejs.org

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

qsv

qsv (Quicksilver) is a command line program for querying, indexing, slicing, analyzing, filtering, enriching, transforming, sorting, validating & joining CSV files.

language: Rust stars: 1085 last commit: today
repo: github.com/jqnatividad/qsv
site: qsv.dathere.com

OSMnx

OSMnx is a Python package to easily download, model, analyze, and visualize street networks and other geospatial features from OpenStreetMap.

language: Python stars: 4478 last commit: 2 weeks
repo: github.com/gboeing/osmnx
site: osmnx.readthedocs.io

Opal

Opal is a Ruby to JavaScript source-to-source compiler. It also has an implementation of the Ruby corelib and stdlib

language: Ruby stars: 4780 last commit: a month
repo: github.com/opal/opal
site: opalrb.com

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

🎙️ Interview with Elia of Opal - a Ruby to JavaScript source-to-source compiler

Hi Elia, Let’s start with your background! Where are you from, where have you worked in the past, how did you learn to program, and what languages or frameworks do you like?

Hi! I’m from Italy, I’ve been working as a software developer for about 15 years now, and I'm not one of those early childhood programmers that started with Basic and a Commodore 64, I had my initial encounter with programming in high school, whe I was really fascinated by DHTML and the ability to make things move on the screen, at the time I printed and read the online Netscape guide to JavaScript with really poor understanding of technical English and without understanding much of it.

What’s your most controversial programming opinion?

I don't know if it's controversial, but I was always attracted to making things beautiful for the user, or code reader, and that's what got me hooked into Ruby, it's really a beautiful language, with the same beauty of some mathematical formulas, and I think that's what makes it so appealing to many people. From this kind of love for the next person experiencing your app or code I think all sorts of good things come out, like the attention to details, the care for the user experience, the care for the code readability, but also the right amount of pragmatism to make things work and be useful. Having the individual person at the center is what matters, not a group or a "higher good" or a "greater good", but the single person, you can always sacrifice an individual person for the "greater good" but when you care for the next person you can't do that. I like that Ruby is released yearly on Christmas day, that's ultimately the message of Christmas, and the drive behind Ruby.

What is your favorite software tool?

I think I owe a lot to TextMate, and its main merits are two. The first one is that it's so well integrated with macOS that just by learning it you learn a lot about the OS, ultimately making it your IDE, fully espousing the consistency and "just works" approach that made Apple so successful. The second one is its extremely elegant way of solving extendability, going full on with the Unix philosophy of small commands that can interact with each other through piping and environment variables, I am still amazed that to this day the commenting in TextMate is handled by a Ruby script. The whole programming community owes it a lot, and modern VSCode is still full of references to TextMate throughout its documentation, as a testament to the profound impact it had.

What are you currently learning?

Programming wise and outside of my main open-source work I'm having some fun reimplementing TextMate features on VSCode, those are nice little puzzles to solve that have a practical application. I'm likely the only user to my extension for TextMate expats but it's ok, I kinda like being in such a niche.

Why was Opal started?

Can't say for sure as I wasn't the original author. But I can say how I bumped into it. It was circa 2011 and I was working on my first JavaScript-only project. At the time we still had to support IE7 and Coffescript was the new kid on the block, so new that we didn't dare use it on client's code. After falling for most of the now infamous JavaScript "WAT" moments I was looking for something better, and started researching projects compiling Ruby to JavaScript. I found a few, but Opal was the only one that was actually usable, and I started polishing it so I could use it in my projects. In 2012 I had the first bit of Ruby code running in production on the browser, and writing Ruby code for the browser was a real joy. I was hooked, and I started contributing more and more to Opal, eventually becoming the maintainer in 2015.

How does Opal work?

The way Opal works is very similar to most other languages compiling to JavaScript. It all starts with parsing code into what's called an abstract syntax tree, which then gets transformed into JavaScript code. The one thing that sets Opal apart from other attempts at porting Ruby to the browser is that it tries to fully leverage the underlying JavaScript engine by reusing its native objects and methods. This means that when you write a = [1, 2, 3] in Ruby, Opal will actually create a JavaScript array, and when you call a.map { |x| x * 2 } Opal will actually call the native JavaScript map method on the array passing an anonymous function that multiplies the value by 2. This approach has a few advantages, the first one is that it's really fast, the second one is that it's really small, and the third one is that it's really easy to debug. Just recently we were able to reimplement the Ruby Hash class using the native Map object, and this resulted in both speed and size improvements.

Where did the name for Opal come from?

The name was chosen in reference to the Ruby gemstone by the original author, Adam Beynon, referencing the famous gemstone that can be found in Australia. Of course the reference is I think it's a great name, Opal gemstones are really beautiful and I love Australia.

Who, or what was the biggest inspiration for Opal?

For me Coffeescript was really instrumental in opening my eyes to the possibility of writing Ruby code for the browser, I think Adam had a few other projects compiling Objective-C and other languages into JavaScript.

Are there any overarching goals of Opal that drive design or implementation? If so, what trade-offs have been made in Opal as a consequence of these goals?

This kind of venture is deemed to have lots of trade-offs, for example in early days one great Ruby feature like method_missing wasn't supported. Surprisingly though, we've seen contributions for many things that previously seemed impossible. Although we have great support for Node.js and other server-side environments, our main target platform is still the browser.

What is the most challenging problem that’s been solved in Opal, so far?

I think the most challenging problem was module inclusion (either via include or prepend) and the super keyword. The reason is that Ruby has a very complex inheritance model, and it's not easy to mirror that in JavaScript. We've been able to do it tho, using the prototype chain. That part is very challenging to reason about given its meta nature, but we need to give credit to JavaScript for being so flexible that a solution was eventually found.

For anyone interested in the details, I strongly encourage giving a read to the official Hacking guide and to the parts of the runtime.js file dedicated to core object and classes initialization.

Are there any competitors or projects similar to Opal? If so, what were they lacking that made you consider building something new?

If we're willing to define it so, we've recently got a newer "competitor" in the official Ruby support for WebAssembly, but I wouldn't treat it as a real competitor as our mission is to get Ruby on the browser and we can only root for an official viable solution. On the other hand it's still in its early days and, given its size, it's not suitable for generic web-application code except for a few good use cases.

What was the most surprising thing you learned while working on Opal?

I think the most surprising thing is how much you can do with so little code. The core of Opal is really small, and it's really surprising how much you can do with it. I think it's a testament to the power of Ruby, and the power of JavaScript as well. I also learned so much about both Ruby and JavaScript, that I think trying to implement additional features in Opal is a great way to improve both skills for any Ruby developer also working with JavaScript.

What is your typical approach to debugging issues filed in the Opal repo?

We have a weekly meeting where we discuss issues and PRs, and we try to give feedback as soon as possible. We do our best to stay abreast of everything but sometimes we miss things, and we're always happy to be pinged on issues and PRs. We also have a Slack channel where we're always happy to help.

What is the release process like for Opal?

We keep track of unreleased changelog entry in a separate file and we have a few scripts that will merge that into the main changelog file and bump the version number. We then tag the release and push it to Ruby gems using Bundler's gem helper tasks. We also have a few other scripts that will update the website and the documentation. The whole process is pretty smooth and we're able to release a new version in a matter of minutes.

Is Opal intended to eventually be monetized if it isn’t monetized already? If so, how? If it’s already monetized, what is your main source of revenue?

Opal is not monetized, and it's not intended to be, at least directly. Being a language implementation it won't have any monetizable application on its own. We maintain a funding page that we use to sustain infrastructure costs and possibly a website redesign, but that's about it. We're always open to sponsorships and donations and I think that has been the case for some fixes and recent features, I think that's a great model for open-source projects of this kind. I'm also personally thankful to my employer, Nebulab, for allowing me to spend some of my time on Opal.

What are you most proud of?

At some point I had a few ecommerce stores fully running their JavaScript code on Opal with a homegrown framework written in a hundred lines of Ruby. Those stores were very large fashion ecommerce stores serving tens of thousands of users a day.

How do you balance your work on open-source with your day job and other responsibilities?

I realized early on that I need to be in for the marathon and not the sprint, so I keep it best-effort and I try to keep it fun, having maintainers burn out it's a lose-lose scenario for any open-source project. Open source is a gift to the world, by the maintainers, I'm grateful for all the open-source I have the privilege to use, and I think this is a healthy attitude toward it.

What is the best way for a new developer to contribute to Opal?

I think the best way is to start using it, and report any issue or missing feature. If anyone wants to go deeper it's a treasure trove of puzzles to solve, thanks to Ruby's spec project providing a comprehensive test suite for the language. As mentioned above also have a Hacking guide that can help anyone get started.

If you plan to continue developing Opal, where do you see the project heading next?

There are some themes that we'd like to explore and some features we need to support better. We're looking into things like dead-code-elimination for reducing the bundle size even more for compiled projects and into improving the support for modules and importmaps. We're also exploring the possibility of supporting the new Ruby WebAssembly implementation featuring a compatible API so it's easier to write code that can run both. But there are so many ideas that it is hard to come up with a full list, things like using JavaScript symbols for Ruby symbols and method ids, automatic JavaScript libraries wrappers, and so on.

What motivates you to continue contributing to Opal?

After all these years I still love a good puzzle and I still love Ruby, it keeps me sharp on the language and on JavaScript, and although I regret not having any big production app running on Opal anymore, I still cherish providing such a great tool to the people using it everyday.

Are there any other projects besides Opal that you’re working on?

As part of my day Job I work every day on Solidus, the Rails e-commerce framework, lately I also had some involvement in View Component, and I wrote an ERB-formatter for VSCode to scratch my own itch that got some traction. As mentioned, I'm also maintaining a VSCode extension for stranded TextMate users that I use myself, but I jokingly gave it 5 stars on the VSCode marketplace because I really enjoy it .

Where do you see software development heading next?

Everyone's talking about AI, and I won't make an exception. As many I swing between interpreting my fears of it taking over the world in a Terminator-like fashion and the hope that it's a sort of luddite instinct that will eventually turn into developers just being more productive.

Where do you see open-source heading next?

I think open-source greatly increased efficiency and thus value-generation across the world, and at the same time it opened opportunities for many people that otherwise wouldn't have had them. I think it's a great thing and I hope it will continue to exist and thrive.

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

threestudio

threestudio is a unified framework for 3D content creation from text prompts, single images, and few-shot images, by lifting 2D text-to-image generation model

language: Python stars: 4133
repo: github.com/threestudio-project/threestudio

Kitchen Owl

KitchenOwl is a smart self-hosted grocery list and recipe manager. Easily add items to your shopping list before you go shopping. You can also create recipes and get suggestions on what you want to cook.

language: Python stars: 752
repo: github.com/TomBursch/kitchenowl
site: kitchenowl.org

Mullvad VPN

The Mullvad Browser is a privacy-focused web browser designed to minimize tracking and fingerprinting.

language: C++ stars: 909
repo: github.com/mullvad/mullvad-browser
site: mullvad.net/en/browser

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

Rapid

Rapid is a web-based modern editor for OpenStreetMap. Rapid integrates advanced mapping tools, authoritative geospatial open data, and cutting-edge technology to empower mappers to get started quickly.

language: JavaScript stars: 382
repo: github.com/facebook/Rapid
site: rapideditor.org

OpenTracks

OpenTracks is a sport tracking application that completely respects your privacy.

language: Java stars: 788
repo: github.com/OpenTracksApp/OpenTracks
site: opentracksapp.com

Lobe Chat

Outfit Anyone: Ultra-high quality virtual try-on for Any Clothing and Any Person

language: TypeScript stars: 11496
repo: github.com/lobehub/lobe-chat

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

RaspAP

RaspAP is feature-rich wireless router software that just works on many popular Debian-based devices, including the Raspberry Pi.

language: PHP stars: 4145
repo: github.com/RaspAP/raspap-webgui
site: raspap.com

Silex

Silex is a no-code tool for building websites. It also lets you code when needed. It can be used online, offline or in a JAMStack project.

language: TypeScript stars: 1847
repo: github.com/silexlabs/Silex
site: silex.me

AlaSQL

AlaSQL is an open source SQL database for JavaScript with a strong focus on query speed and data source flexibility for both relational data and schemaless data. It works in the web browser, Node.js, and mobile apps.

language: JavaScript stars: 6774
repo: github.com/AlaSQL/alasql
site: alasql.org

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

🎙️ Interview with Mathias Wulff of AlaSQL - a JS SQL Database for browser and Node.js

Hey Mathias! Thanks for joining us! Let us start with your background.

Born and raised in the heart of Denmark, my journey into the realm of programming was as much a product of my environment as it was of my curiosity. Denmark, with its robust educational system and a culture that encourages exploration and innovation, was the perfect incubator for my growing interest in technology. My early days weren't filled with the usual video game escapades; instead, I found myself fascinated by the inner workings of Windows 95, diving into regedit, eager to tweak and learn. This early exploration set the stage for a career driven by a desire to understand and manipulate the digital world.

Other than a few websites in the late 90s my programming journey began with a TI-83 calculator. Programming in TI-Basic, similar to Basic but with calculator-specific features, I learned the importance of efficient code structure, initially constrained by the belief that variables could only be single letters because of the examples in the manual. A misconception that forced me to plan ahead and think creatively within the limitations.

My professional path has been diverse. Starting as a fullstack developer, I ventured through various roles: a consultant, a technical product manager, and a CTO. Now, as the Head of Technology in a company offering online tutoring, I apply my accumulated knowledge to create impactful educational technologies. Each role offered unique insights, but the underlying theme has always been my desire to understand the human aspect of harnessing technology to solve complex problems successfuly. 

Who or what are your biggest influences as a developer?

My biggest influence has been _unwriter, an enigmatic figure who emerged during the tumultuous times of Bitcoin's second major fork. _unwriter's approach to solving problems wasn't just innovative; it was transformative. They tackled challenges at levels I hadn't even considered. The real eye-opener was not the volume of their output but the depth and breadth of their problem-solving approach. It wasn't just about writing code; it was about reimagining possibilities.

What’s your most controversial programming opinion?

I've always leaned towards Yoda Conditions, placing the constant before the variable in comparisons. It's a practice born from a blend of experience and a bit of suffering. But my broader, and perhaps more intriguing, opinion is about the nature of software itself. Software, in essence, is invisible. Yes, there are UI elements, but the system, the logic, and the architecture reside in the realm of the unseen. This invisibility makes software creation an art of aligning mental models across diverse groups. It's about creating a shared vision, a common language that transcends the barriers of different technical and non-technical perspectives.

What is your favorite software tool?

Surge.sh and Bun (Both CLI tools) hold a special place in my heart. Using them feels like being understood and anticipated by someone who cares deeply about your needs. They're like a thoughtful aunt who's known you since childhood, always ready with exactly what you need before you even realize it. These tools don't just aid in programming; they extend the programmer's mind, making the development process not just easier but more intuitive and enjoyable.

What are you currently learning?

I'm currently navigating the delicate balance between being hands-on and empowering others to take the lead. It's a dance of involvement and delegation, of knowing when to step in and when to step back. On a technical front, I'm exploring the capabilities and limitations of Large Language Models. Understanding their strengths and how to integrate them into workflows is a fascinating challenge. It's about harnessing their potential to inspire and augment human creativity and efficiency.

Why was AlaSQL started?

AlaSQL was the brainchild of Andrey Gershun, a programmer whose brilliance is unmatched in my experience. His need for a client-side SQL solution led to the creation of AlaSQL. I wasn't there at the inception, but having worked closely with Andrey, I can attest to his genius. His ability to see a need and create a tool that addresses it so elegantly is nothing short of remarkable.

How does AlaSQL work?

AlaSQL works by taking the SQL queries you write and converting them into equivalent JavaScript code. This conversion process allows users to manipulate data with the familiarity of SQL, coupled with the versatility of JavaScript. At its essence, AlaSQL recognises your need to filter, select, and manipulate data and simplifies the daunting task of implementing the code. 

Where did the name for AlaSQL come from?

The name AlaSQL is a nod to its nature – it's SQL, but with a twist. "A la" implies a likeness to SQL, but with unique adaptations and additions that make it more suited to the client-side development experience. It's a blend, a fusion of SQL's structure with JavaScript's flexibility and quirks.

Are there any overarching goals of AlaSQL that drive design or implementation? If so, what trade-offs have been made in AlaSQL as a consequence of these goals?

Our overarching goal with AlaSQL has always been to create a tool that 'just works'. Developer experience is at the forefront of our design philosophy. We aim for SQL-99 compatibility and reasonable speed, but these goals come with trade-offs. The SQL parser and lexical analysis, for example, are extensive but necessary for the robustness and versatility we offer. One of our long-term visions is to integrate AlaSQL as a build step in bundlers, allowing for a more streamlined, efficient deployment of code. This approach would be ideal for static SQL but presents challenges for dynamically generated code.

What is the most challenging problem that’s been solved in AlaSQL, so far?

One of the most challenging and rewarding aspects of working on AlaSQL has been bridging the gap between SQL's Abstract Syntax Tree (AST) and executable JavaScript code. The ability to translate complex SQL queries into efficient JavaScript operations is a feat that required deep understanding and creative problem-solving.

Are there any projects similar to AlaSQL? If so, what were they lacking that made you consider building something new?

Back in 2014, the tech landscape was different, especially in the realm of browser capabilities and JavaScript libraries. At that time, projects like jSQL and an emerging SQLite in the browser were notable. However, AlaSQL distinguished itself by focusing on being lightweight and more attuned to the needs of client-side development. It was a response to a gap in the market, providing features and ease of use that others lacked, tailored for the web developers of that era.

What was the most surprising thing you learned while working on AlaSQL?

One of the most enlightening aspects of working on AlaSQL has been the depth of Andrey Gershun's coding genius. Initially, it was a challenge to fully grasp the intricacies and innovations of his code. His approach to problem-solving and system design was ahead of its time, and understanding his methods provided valuable lessons in advanced programming and innovative thinking for me.

What is your typical approach to debugging issues filed in the AlaSQL repo?

When tackling bugs in AlaSQL, my approach is systematic. I start by recreating the issue in the command-line interface of AlaSQL. Then, I dive into the debugging process using 'bun --inspect-brk', which allows me to step through the code, trace the execution flow, and understand the root cause of the issue. 

What are you most proud of?

I'm immensely proud of the consistent contributions from our community. AlaSQL feels like a collective effort, a place where everyone comes together for a common cause – to make this tool better, not just for personal gain but for the sheer joy and satisfaction it brings. It's a testament to the spirit of open-source development.

How do you balance your work on open-source with your day job and other responsibilities?

I treat my work on AlaSQL as a form of mental relaxation, akin to solving puzzles. This approach allows me to engage with open-source projects in a way that is refreshing rather than taxing. It's a balancing act that involves knowing when to immerse myself in these projects and when to let it be.

Have you ever experienced burnout? How did you deal with it?

Burnout struck hard during the COVID-19 pandemic, a period when everything, including open-source participation, and staying on top of emails, seemed overwhelming. My response was to embrace this phase, allowing myself to step back and take time for rest and relaxation. This period of stepping away was crucial for my mental health and helped me return to my regular level of energy and perspective.

What is the best way for a new developer to contribute to AlaSQL?

For new developers looking to contribute to AlaSQL, I recommend starting with issues marked as 'good first issue'. Engage with the community, seek guidance, and dive into understanding the codebase. The most important aspect is learning to navigate and interpret the code, a skill that can be honed by using debugging tools like Bun for an enhanced and efficient development experience.

Where do you see the project heading next?

Looking ahead, AlaSQL is poised for significant evolution. Our focus is on modernizing the codebase, moving towards a modular structure that aligns with contemporary coding practices. We're also contemplating a lighter version of AlaSQL, catering to users who need less complexity but still desire the core functionalities. The idea of integrating AlaSQL into the build process remains a fascinating yet daunting prospect – one that promises to open new frontiers in efficiency and functionality.

What motivates you to continue contributing to AlaSQL?

The motivation for continuing with AlaSQL stems from the sheer joy of solving puzzles. It's about the challenge, the satisfaction of finding solutions, and the thrill of creating something that not only works but works beautifully.

How did AlaSQL get popular?

AlaSQL's rise to popularity is a testament to its utility and effectiveness. With 180k monthly downloads via npm and 3 million loads from websites, its widespread adoption speaks to its relevance and reliability. Its success is rooted in its ability to provide a powerful yet user-friendly SQL interface in the browser, meeting a critical need in the web development community.

Are there any other projects besides AlaSQL that you’re working on?

Rexreplace is definitely my darling pet-project. It gives you a smooth experience in performing search and replace operations from the command line. It's tailored to be part of one-liners and build scripts. The power lies in how the syntax instills confidence in what you're doing (in contrast to balancing awk, find, sed, and grep), and you also have the ability to use JavaScript to refine your actions further.

Where do you see software development heading next?

The landscape of software development is poised for a transformative shift, especially with the advent of large language models. These models are becoming akin to peer programmers, capable of handling complex coding tasks. The key skill in this evolving landscape will be the ability to precisely articulate what you want to achieve. This shift signifies a move from traditional coding, where the emphasis is on how to write the code, to a more intent-driven approach, where the focus is on what the end goal is. Developers will need to hone their skills in defining problems and objectives clearly, as language models take on the more routine or intricate coding tasks. By simplifying the coding process and reducing the barrier to entry, we open the door to a more diverse range of voices and perspectives, hopefully leading to a significant shift in the demographic landscape of the industry. This is particularly crucial in addressing the current gender imbalance in the field, which is heavily skewed towards men. By making software development more approachable and less daunting, we can encourage a more balanced representation of genders, leading to a more inclusive and varied tech community. This change isn't just about fairness; it's about enriching the field with different viewpoints and experiences, which is essential for true innovation and progress.

Where do you see open-source heading next?

I believe the mindset for supporting open-source projects through financial means is already present, but what has been missing is an infrastructure that truly supports nano-payments. This kind of system would allow for small financial contributions to be distributed efficiently across a project's (deep) dependency chain. For instance, a donation worth $1 could be split and shared among the various dependencies that make a project possible. This approach would transform how open-source projects are funded, ensuring a more sustainable and equitable distribution of resources. By overcoming the technical challenges of implementing such a system, we can leverage the existing goodwill in the community to create a more robust and self-sustaining open-source ecosystem.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

For those making their first steps into open-source, remember that every contribution, no matter its size, is deeply appreciated. When you're ready to start, focus on asking clear and concise questions. This helps not only in getting the guidance you need but also in making it easier for others in the community to assist you. It's important to understand the basics and communicate this understanding clearly. When presenting examples or issues, keeping them succinct and precise will facilitate better understanding and support. The open-source community thrives on collaboration, and every new voice adds value to the ongoing conversation. We welcome you to join in and contribute in any way you can!

What is one question you would like to ask another open-source developer that I didn’t ask you?

What was the driving force that prompted you to dive into your project? Understanding this can shed light on the diverse range of inspirations and passions that fuel open-source contributions. Each developer's journey is unique, and knowing what sparks that initial interest can provide valuable insights into the personal stories and motivations that enrich the open-source community.

Which open source maintainer should we interview next?

The creator of Bun would be an excellent choice for your next interview. Bun is a notable project in the open-source community, and the maintainer's insights and experiences would undoubtedly be valuable. Learning about their journey, challenges, and the vision behind creating Bun could offer a wealth of knowledge and inspiration to both current and aspiring open-source contributors.

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

PostHog is an open-source, all-in-one suite of product and data tools including product analytics, session replay, feature flags, A/B testing, and surveys.

From testing in product to analyzing user retention to running NPS surveys, PostHog supports it all. It is a single place to help you test, launch, and analyze new features to help you build a better product.

Get started for free

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Front Matter

Front Matter is a CMS running straight in Visual Studio Code. Can be used with static site generators like Hugo, Jekyll, Hexo, NextJs, Gatsby, and many more.

language: TypeScript stars: 1570
repo: github.com/estruyf/vscode-front-matter
site: frontmatter.codes

Fluvio

Fluvio is a high-performance distributed data streaming platform that's written in Rust, built to make it easy to run real-time applications.

language: Rust stars: 2165
repo: github.com/infinyon/fluvio
site: fluvio.io

Hedy

Hedy is a gradual programming language aimed at teaching programming and teaching Python. It teaches using different levels.

language: Python stars: 1072
repo: github.com/hedyorg/hedy
site: hedy.org

🎙️ Interview with Felienne of Hedy - a language for children to learn programming

Hey Felienne! Thanks for joining us! Let us start with your background.

I have the, maybe traditional, backstory of “teaching myself how to program” when I was a preteen. Playing with BASIC, making silly programs for myself, making games. After high school, I went on to major in computer science. I used to work mainly in C# but for a few years I mainly use Python! I really like the freedom that Python offers to sketch your thinking.

What's an opinion you have that most people don't agree with?

Investing in explicitly memorizing syntax you use often is totally worth the effort. Of course you do pick up some things just by using code, but if you want to get more effective, learning syntax is a great (and easy) way to get better.

What’s your most controversial programming opinion?

There can never be meaningful guidelines for “readable code”.

Apart from the basic coding style guidelines, like using Camel case or using verbs in function names, what is readable depends too much on your personal experience and knowledge. Is a list comprehension inherently more readable than a for loop doing the same? It only depends on whether you have seen and used comprehensions before. 

Readable code also depends on the task at hand. Is code duplication bad? It depends! If you are making a small change it is not helpful, but if you are trying to get a grasp of the whole code base, duplication helps because it is easier to find information. Are long methods bad? Yes if you are navigating the code base, no if you want to understand control flow.

What is your favorite software tool?

GitHub desktop! Having, I think, a reasonable understanding of how git works, I have tried and failed to deeply understand how to operate the git command line, but the basics just do not make sense to me. Many of the operations does do not align with my thinking about how stuff works. The UX, which I think we can certainly speak of for a command line too is just horrible, and made only for people who have already drunk liters of the kool aid. Like why is “checkout” making a new branch? There is no logic!

GitHub desktop places a usable UX over git, with revolutionary ideas like a button that is called “new branch” for making a new branch. I don’t get why people still prefer the command line for simple things, for I would say 90% of the tasks, you can use Desktop. (of course I sometimes use the command line for very complex merge conflicts but that is really limited to a few times a year).

What is your favorite book and why?

Let/s limit this to books about programming! Certainly “Working in Public” has made a huge impact on me recently. It gave me vocabulary for the things I was struggling with in managing Hedy. The book talks about how open source has changed from projects that were, in essence, clubs in which people knew each other, because the burden of entry was high (which of course in itself was very problematic too!) to projects that are more performances, in which a small group of people work “in public” and a large group of casual contributors dip in and out of the project, because with GitHub everyone can so easily open issues and pull requests, without really knowing the project.

If you had to suggest 1 person developers should follow, who would it be?

Grady Booch, most well known as the creator of UML, but nowadays both fighting AI nonsense on Twitter, and standing up for diversity and inclusion in the field. 

Why was Hedy started?

As with most projects I think, to solve my own problem! I was teaching middle schoolers programming (aged 12), and they found Scratch and other block-based languages to be too childish, they wanted a “grown up” language. That is of course very fitting for that age group; teenagers want to do stuff that adults do too. However, textual languages like Python or JavaScript are made to work with as a professional, they are not made to learn with. Just outputting something like “hello” already requires brackets and quotation marks, which for novices is very distracting. You can’t really learn about the concept of printing, without also learning a lot of syntax. And for more advanced concepts like conditionals or loops, the syntax is even weirder. Many of the kids in my class would eventually learn how to type “for i in range(4):” but they did not really grasp repetition as an idea. 

And, the error messages were so cryptic, especially for non-English speakers that I was teaching. What is a “Syntax error”, why is this “indent” “unexpected”. I had to run around all the time to put out small syntax fires, rather than talk about the cool things that kids could achieve with programming.

So I thought… What if I make a programming language that works in small steps? First we teach a concept, and then we slowly add the syntax. So first print hello, and then print “hello” and then print(“hello”). In the winter of 2019 I made the first prototype, just to be used in my class.

After a year, 100.000 Hedy programs had been created, and we had received several translations like Spanish and French, so I knew I was onto something! Fast forward to today when we have about 400.000 people on the site monthly, and are available in 47 languages, including non-latin languages like Simplified Chinese and Hindi and right to left languages like Arabic and Urdu. In terms of open source, we have over 350 contributors to date, and we merge about 75 pull requests a month. So it has been quite a ride!

How does Hedy work?

We have a really cool (hand drawn) overview of the architecture in our Wiki:

It summarizes what we do: The basic idea of Hedy is that we allow learners to write code using the Hedy syntax, in a web page. The code page, and other web pages, are rendered by Flask (a Python web framework), using templates in Jinja.

When running Hedy code from the webpage, the Hedy code is converted (transpiled) into Python on the server side, using the Lark parser framework. The transpiled Python code is sent to the front-end and run in the front-end using Skulpt, a Python interpreter written in JavaScript.

Why did you pick Python?

I chose Python because I knew Python, and because the Hedy learning trajectory leads to Python. This is also why we transpile to Python, because it is close to Hedy. In retrospect, this was maybe not the best move, because it means we have to rely on Skulpt to execute the Python in the browser. I really hope browsers will implement running Python so we can do that in a more direct way! (it is too late I feel for us to still switch to generation JavaScript directly).

Where did the name for Hedy come from?

Hedy is named after Hedy Lamarr! She is most known as a movie actress, but she was also an inventor, who helped develop the “frequency hopping” algorithm that you use everyday, since it is in Wi-Fi routers! 

Who, or what was the biggest inspiration for Hedy?

Kids in my classes that were struggling with error messages, especially the girls who would come in a bit nervous already into programming class. If they first thing you then see is “SyntaxError, unexpected EOL” it is easy to think you are not smart enough to be good at computers.

Are there any overarching goals of Hedy that drive design or implementation?

Yes, we have even documented our design goals for the language: https://github.com/hedyorg/hedy/wiki/Hedy-Language-Design! There is sometimes a trade off and to be honest we have sometimes sinned a bit against the design, because you want to have a language that is simple, but you also want kids to build cool programs that they are excited about. So some early levels add concepts (like sleep and clear screen) that are maybe a bit early in the learning trajectory but add depth. 

What is the most challenging problem that’s been solved in Hedy, so far? And what was the most surprising thing you learned?

Everything to do with localization, specifically for non-Latin languages. Did you know Arabic numerals (0, 1, 2, ..9) are not used in Arabic? I certainly did not know that before I started working on the Arabic version of Hedy. There were so many things to think about! The use of accents is another interesting issue, in French, repeat is “repété”, but what if a kid uses “repete”? Is that a syntax error or should it be allowed? 

English is a really simple language in terms of spelling, and therefor in PL design, a lot of concepts from other languages have not been considered. Like, we have the concept of “case sensitivity” but not of “diacritic sensitivity” to decide between “repété” and “repete” because English does not have accents. While localizing Hedy I ran into so many issues I ended up writing a paper about it: felienne.com/archives/7668.

If you could teach every 12 year old in the world one thing, what would it be and why?

What we are trying to do with Hedy is teaching all kids that programming can be learned. Many kids without prior programming experience, often girls and kids without a computer at home, feel programming is scary and not for them. Existing solutions don’t really address that well. Visual programming languages like Scratch don’t “scaffold”; they don’t tell you where to get started and how to approach things. Textual languages on the other hand, like Python or JavaScript start with throwing error messages around before kids have a chance to figure out if programming is for them, or even what programming *is*. 

We teach them that even though, like with all subjects, some kids go fast and some so slow, some reach far and others just take a steps, everyone can do a little bit of programming!

If I gave you $10 million to invest in one thing right now, where would you put it?

Hedy! We have so many things to explore still, improving error messages, helping teachers get better insights into what kids are doing and where they are struggling, using a combination of parsing and AI to translate programs from one natural language to the other so that all kids can learn from each other!

What are you currently learning?

Managing a large open source project. Hedy was never really meant to be a large project and in the beginning I really did not know how to manage it, in various ways! I did not really look at code in depth, when people submitted PR’s I mainly looked at whether it was working, and whether the code was not terrible, but I did not look at whether the style fit the project, or whether the 

What have you been listening to lately?

In terms of music? Bob Dylan, always. The thing about Dylan’s music is that there is a Dylan album for every mood, he has old school blues, but also stuff more leaning towards gospel, jazz, and pop. So whatever you are in the mood for, Dylan is the way to go.

In terms of (tech) podcasts, I love “tech won’t save us” and “the hard fork”.

How do you separate good project ideas from bad ones?

One answer to that is that I am a teacher, and I teach with Hedy myself, so I often have a feel for features that could help teachers. As a maintainer, I also have a lot of contact with other teachers that use Hedy, on our Discord channel with regulars, and in workshops we give to new teachers. But sometimes we still invest in “the wrong” features because they are interesting to build, or because we think something will be helpful while it is not.

What’s the funniest GitHub issue you’ve received?

I am not sure if this is really funny, but there was someone who made an issue (github.com/hedyorg/hedy/issues/1908) about quotation marks in my LICENSE file. I kindly asked him to bugger off after this issue/PR. It was interesting because I told him stuff like this was really overwhelming for maintainers, and after a while he replied saying that he has muted the repo since he was also overwhelmed with activity from it.

But looking back at this interaction, it is nice to observe that I have grown so much as a maintainer. Nowadays, I would not even accept such a silly PR and tell people a lot earlier that this type of behavior is harmful. We now also have strong community guidelines about unwanted suggestions, and a Discord server where we hang out with each other to joke about this kind of stuff.

What is your typical approach to debugging issues filed in the Hedy repo?

For errors about the Hedy language, I almost always start with writing a test and see if I can reproduce the error from a test. If I get there, I know fixing it will be doable. We are really big on TDD like development for the language part. See that it breaks, fix the test, see that no other tests are broken and then try to improve. We have a lot of manually written tests for the language, but we also have a test set of about 10.000 programs created by kids previously, which we also shouldn’t break with new changes (or at least not unintentionally).

What is the release process like for Hedy?

Every merge to main goes to our test environment, where we sometimes invite teachers to try out stuff, and where we also manually test some front-end features to see if all goes well and looks well. Usually we deploy the test environment to hedy.org every Friday late afternoon. This makes sense for us since then our traffic is lowest since schools are out :)  

Although every core contributor could do this, in practice I am always the one doing this (I am not even sure they know they could, haha!)

Is Hedy intended to eventually be monetized?

It is meant to be free and open source. I think it is quite wicked actually that schools spend so much money on software (and other digital materials!) while there are so many people that would build software and materials for free. I am happy that may tools for programming (code.org, Scratch, us) are showing that software can be both useful and user friendly and also free for schools. 

We are funded mainly by government subsidies from the Dutch government and occasional donations on GitHub (which are nice, of course) and a lot of volunteer effort!

What are you most proud of? 

Hedy definitely helped me in my career, because it got me the award for best CS researcher of the Netherlands (a small country, but still). But I think the best moment was when after schools were closed because of Covid, in the summer of 2021 I saw kids working with Hedy in real life, for the first time. Seeing the product come to live, to really see kids interact with it, was mind-blowing!

How do you balance your work on open-source with your day job and other responsibilities?

It is a bit hard to say because as an academic, it is not so well defined what my job is.  Certainly inventing new things is my job, so a big part of Hedy is just what I am supposed to do.

What is the best way for a new developer to contribute to Hedy?

Be excited about what Hedy is! Our docs explain that our most successful contributors fall in love with the idea, goals and philosophy of Hedy first, and look around to see where help is needed. Less successful contributors find small issues that bug them and submit pull requests without looking at issues that we created. 

If you start to love Hedy, join a bi-weekly contributors meeting. I think one of the best things I have done for my own enjoyment is start to have online meetings with regulars. It eases further (so often written) communication so much if you get to speak to each other, and we laugh a lot about silly ideas and plans and mistakes. It really created a team feeling and I am surprised I don’t see this more often in OS projects. It is so worth the hour I put in every two weeks!

What motivates you to continue contributing to Hedy?

The emails we get from people all around the world telling me their teaching has gotten so much better with Hedy, that their students are learning more and having more fun! It really shows we are on the right track. 

Are there any other projects besides Hedy that you’re working on?

None at the moment, because:

Do you have any other project ideas that you haven’t started?

I have been forever dreaming about a better refactoring tool (one to perform “cognitive refactorings” as I outline in my book) but I know I will never build that, it is too much work on top of everything. Throwing it out here so maybe someone else will haha!

Where do you see software development heading next?

GPT and Copilot and the like of course is an interesting development that has people reflect on their practice a lot. If done well they can really improve productivity, I recently used GPT to explain to me how to use a certain API, and it was impressively helpful, it really saved me a lot of search time! But it can also generate senseless stuff, or code with subtle bugs that are hard to spot. So I hope we will invest less in the hype and more in the “if done well” part. I certainly believe in GPT for code more than for text, because it can be executed, so many (not all!) errors are caught early on.

Where do you see open-source heading next?

I don’t know. “Working in Public” does paint a bit of a bleak picture in which the small number of maintainers will not be able to keep up with all contributors and users needing their attention. And I fear for a world in which that attention will also have to be split with “smart” bots that auto generated PR with an issue text as prompt! That is a dark side of GPT I see, sadly. 

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Find an issue you think you can do (maybe look for things labeled “good first issue”) and then reach out to the maintainers as documented in the docs. Some people want to get an email or a Discord message, others state that you can just go for it, but look at the preferences and follow these! That gives you the best chance of getting positive feedback.

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Mail-in-a-Box

Mail-in-a-Box helps individuals take back control of their email by defining a one-click, easy-to-deploy SMTP+everything else server: a mail server in a box.

language: Python stars: 12803 last commit: a month
repo: github.com/mail-in-a-box/mailinabox
site: mailinabox.email

Flipper

Beautiful, performant feature flags for Ruby. Flipper gives you control over who has access to features in your app. Enable or disable features for everyone, as you like.

language: Ruby stars: 3434 last commit: today
repo: github.com/flippercloud/flipper
site: flippercloud.io

Ansel

Ansel is an open-source photo-editing software for digital artists. A darktable fork minus the bloat, plus some design vision.

language: C stars: 422 last commit: 2 days
repo: github.com/aurelienpierreeng/ansel
site: ansel.photos

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

Minosoft

An open source Minecraft reimplementation written from scratch.

language: Kotlin stars: 638 last commit: today
repo: github.com/Bixilon/Minosoft

Moosync

Music player capable of playing local audio or from YouTube, Spotify and many more

language: JS stars: 648 last commit: 3 months
repo: github.com/Moosync/Moosync
site: moosync.app

Path to Senior Engineer

This repo is a handbook with all the resources you need to reach Senior Software Engineer! Contains courses, communities, etc.

stars: 1604 last commit: today
repo: github.com/jordan-cutler/path-to-senior-engineer-handbook

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Frigate

A complete and local NVR designed for Home Assistant with AI object detection. Uses OpenCV and Tensorflow to perform realtime object detection locally for IP cameras.

language: Python stars: 10876 last commit: today
repo: github.com/blakeblackshear/frigate
site: frigate.video

Tusky

An Android client for the microblogging server Mastodon

language: Kotlin stars: 2285 last commit: a week
repo: github.com/tuskyapp/Tusky
site: tusky.app

Grist

Grist is a modern relational spreadsheet. It combines the flexibility of a spreadsheet with the robustness of a database to organize your data and make you more productive.

language: JS stars: 5350 last commit: today
repo: github.com/gristlabs/grist-core
site: getgrist.com

🎙️ Interview with Paul of Grist - a modern relational spreadsheet with Python support

Hey Paul! Thanks for joining us! Let us start with your background.

I grew up on a small farm in rural Ireland, milking cows and herding goats. On the weekends, I learned to program on a ZX81, by sneaking up to it when my older siblings weren’t around, and randomly tweaking their programs to see what happened. While doing so, I noticed that I was indoors, warm, sipping tea, and not getting kicked or head-butted by farm animals. I wanted more of that sort of thing, so I went on to study computer engineering at the University of Limerick. After that, perhaps from nostalgia for the goats, I did a doctorate at MIT on robots that poke things to learn about them. If a robot ever pokes you to learn about you, my apologies (if you didn’t like it), or you’re welcome (if you did). This was the C/C++ phase of my life. Then I did some machine learning and computer vision at a small startup in NYC, for an early live video search engine that could detect people talking to their cats, streaming their coding sessions, or strumming guitars. This was my Python period. After that company was bought by Axon/Taser for use in law enforcement and surveillance, I moved on to another small startup in NYC that has so far had only the most benign of applications: Grist Labs. There’s still some Python in my life, but now Typescript dominates.

Who or what are your biggest influences as a developer?

I’ve a son in high school who is somewhat into tech, and I pay close attention to what he and his buddies get into. When they hacked together a Wordle variant by extracting some JSON from an academic paper on comparative linguistics, and getting ChatGPT to render it as a website, I knew it was time to pay attention. I also follow a lot of current and recent grad students, since that’s another good source of fresh ideas. For example Geoffrey Litt has an interesting vision for the future of software, and spent a lot of time wrestling with reactive databases (very relevant to Grist).

What’s your most controversial programming opinion?

At some point, I sat down to write a programming language called CosmicOS (cosmicos.github.io) for communicating with extraterrestrials. That’s not controversial exactly, but I do sometimes feel judged when I talk about it. For me, it is an interesting puzzle in communication, logic, simulation, and programming language design, and that’s all. Please don’t jump to the conclusion that I believe aliens have been coming to Earth to probe us! We have robots for that!

What is your favorite software tool?

Git. People complain, but those who lived through the before-times know that they were AWFUL. Also, it sounds like a breed of small goat.

Why was Grist started?

Grist is the brainchild of Dmitry Sagalovskiy. He was trying to do some organizing for the New York Math Circle nymathcircle.org, a math club. First they used a spreadsheet, which was easy but didn’t do everything needed. Then they used a custom site, with Django, MySQL, etc. That could do more, but needed a lot of nursing along. With Grist, stuff that almost works in a spreadsheet can actually work for real, with no need for app building. I joined the project because I’d had similar frustrations in some volunteer work for the Data Commons Co-op datacommons.coop, an organization with a kind of “bottom-up Google” vision. One of things we were trying to do at the Data Commons Co-op was to make a pooled directory of "alternative economy" organizations - cooperatives, community gardens, food co-ops, buying clubs, time banks, and so on. The Co-op's members are not individuals, but rather umbrella groups such as the US Federation of Worker Co-ops or the Solidarity Economy Network. Several of the Co-op's members had previously tried to run big splashy directories that proved expensive to maintain and failed once funding ran low. We wanted to reduce costs and raise quality by pooling effort. There's a lot of "gotchas" to avoid when doing that, I learned. One that struck me in particular was how important it was that groups were free to decide their own rules for who is "in" or "out" of the directory. Looking for agreement here is the path to madness, and not actually necessary. The technical challenge of designing a directory that could be efficiently added to and queried by each community from its own perspective was very interesting. It left me wanting to pitch in to make a really effective tool for data projects of all kinds. I felt bad that as programmers we have all these nice things for collaborating around code, but whenever I saw a group try to work on data together it looked so painful. It really doesn’t have to be that way.

How does Grist work?

Grist has three main parts. First, there’s a relational database (SQLite). The tables in your Grist spreadsheet are stored exactly as tables in SQLite, with all special spreadsheet fanciness kept in separate metadata tables. That separation does make for some extra work, for example when tables and columns are renamed, but it draws a reassuringly clear line between data and layout. The second part of Grist is a “data engine”, written in Python, whose purpose is to implement the best parts of spreadsheets, such as formulas and dependency-tracking between cells. Grist uses Python formulas, with Excel-like macros available via Python implementations. The third part of Grist is a node server to enable live collaboration and to enforce a very flexible security model. A cute trick in Grist is that you can use formulas to define who has access to data. So if there’s some very specific rule you have about who can see which rows, based on who the user is and what exactly is in the row, you can totally make that happen. This is a game-changer, since it means you can invite people to collaborate on a spreadsheet that in the past you’d have to keep at a distance, forcing you to do awful things like manually importing their emailed CSVs or (worse) randomly-formatted Excel files. Beyond that, Grist is extensible in a few ways. If you know how to write some HTML/JS/CSS to render a page from some JSON input (like my son and his friends did with ChatGPT), then you can immediately apply that to spreadsheet data in Grist. Grist will take care of sorting, filtering, and linking the data with the rest of the UI. So for example if we hadn’t created a map widget, you could do it yourself pretty easily, and others could use your widget without fuss.

Everything needed to view, edit, and collaborate on spreadsheets is open sourced. Some material used to operate our own SaaS, and some features developed for particular enterprise clients, are not. We’ve packaged Grist in a few forms: as a ready-to-go docker image for use as a server, as a desktop app (handy for offline use), and compiled to pure JavaScript for use on static websites without any special back-end.

Where did the name for Grist come from?

It is a portmanteau of Grid and List. I suppose we could have called it Lïd instead?

Who, or what was the biggest inspiration for Grist?

Frankly for me the biggest inspiration is sites like GitHub that have had a big impact on collaborative coding. I aspire to have something like that impact, this time for collaborative data. There’s a long road to go.

Are there any overarching goals of Grist that drive design or implementation? If so, what trade-offs have been made in Grist as a consequence of these goals?

The initial vision for Grist was a desktop spreadsheet app, with sharing via end-to-end encryption, and with none of your data stored or accessible outside your devices. We heard a lot of enthusiasm for the idea, but had a hard time getting people to actually install the app. So we switched to making a more conventional web app, where data is stored in a central server and there’s nothing to install locally. That really helped adoption, but was definitely a trade-off. There are hooks in Grist sitting and waiting for the day we can return our attention to this.

What is the most challenging problem that’s been solved in Grist, so far?

Granular access control was a doozy. I mentioned this feature earlier, but to recap: People often make copies of spreadsheets because there’s someone who should see just part of it, not the whole thing - but then the spreadsheets get out of sync quickly. We wanted to make this whole mess unnecessary, by allowing very flexible access control. We wanted to allow expressing access control as formulas, reusing experience that many spreadsheet users already have. So you could say, for example, that a user should be able to see rows for cities they manage, and be able to edit the ones assigned to them. Making the whole system efficient, and able to correctly propagate live changes to all users despite their differences in access rights was challenging. I think we only managed to do it because of a period of enforced tranquility during the pandemic lock-downs. We are currently working on a new vision for access control that we hope will make it not only flexible, but also easy, so even more users can take advantage of it. Our codename for this project is WYSIWYS (“What You See Is What You Share”). Under the hood, it will translate visual layouts into the rules needed to protect your data. If you want to dip into the granular access code, a lot of it lives here.

Are there any projects similar to Grist? If so, what were they lacking that made you consider building something new?

Airtable is often mentioned, as it is a spreadsheet/database hybrid. For me, I want a data container that can go anywhere the data is - in the cloud, on a desktop, on an airgapped network etc, and I want to be sure the container will work into the indefinite future. Airtable isn’t that. Baserow is sometimes mentioned also. I think it got started a few years after Grist. Something I value about Grist is the robustness of its data format. You can download a spreadsheet and upload it again in a different Grist installation without any fuss. You can read data from it with regular SQLite tooling. I feel I can trust it as an archive format. Baserow goes another route, using PostgreSQL for storage. That’s a great choice, and I hope Grist will support it too in future, but it does mean that individual spreadsheets are not as naturally self-contained as they were with, say, Excel.

What was the most surprising thing you learned while working on Grist?

That a small team building something general purpose, rather than solving a specific problem as all the sage advice urges, can actually succeed to a remarkable extent. It is a very delicate balancing act. Hats off to Dmitry for getting us this far.

How popular is Grist? How did it get popular?

We are big in France! Grist has seen heavy use at a very cool agency in the French government for coordinating local and regional projects, L’Agence Nationale de la Cohésion des Territoires (ANCT - here’s their description of Grist). The main reasons they like us are that we are open source, have good quality code, and plenty of features. They are particular fans of granular access control. They run probably the largest Grist service apart from ourselves, and have contributed many wonderful fixes and features. In the world at large, I’d say we’re just starting to get significant attention. We’ve had a few lucky posts on the selfhosted subreddit, and a couple of front-page Hacker News stories that have raised our profile in the tech community. Successful Appsumo and Product Hunt launches have helped also.

What is your typical approach to debugging issues filed in the Grist repo?

We have a very large and robust collection of tests, so most problems are caught long before they get to users. But if they do get through, there’s no real magic to figuring out the problem. One approach is to work with the user to find a way to replicate the issue, and then to strip things out until the problem goes away. Then you know that somewhere between the starting point and the ending point lies the problem, and you can just step between them to find it.

What is the release process like for Grist?

We aim for a monthly pulse, with coordination between the technical side of releasing a new stable version by our development team, and preparation of a lovely newsletter describing what has changed in the month by our marketing team. For our SaaS, we deploy changes on a weekly cycle, with disruptive changes behind temporary feature flags.

How is Grist monetized?

We have a SaaS, standing ready and willing to host your Grist spreadsheets so you don’t have to, with free and paid plans. We also have some enterprise clients. Because we were bullied into it, it is possible to sponsor us on GitHub, and a small number of very determined philanthropists do so. We are planning to release Grist on marketplaces such as the AWS Marketplace, so people can choose to pay for the convenience of delegating installation and configuration of self-managed Grist sites to us.

What are you most proud of?

When I hear developers from ANCT (the French government agency I mentioned) describe to others how and why they use Grist, and see the impact it is having, it gives me goosebumps. Recently we were co-presenting at a conference with them, and I got to see the full scope of their work. I realized that what they’ve built with Grist aligns closely with the vision of the Data Commons Co-op I volunteered at, but executed with way better funding. It makes me excited to close the circle and bring lessons from Grist users back to the grass-roots projects that originally motivated me to get into all this.

Have you ever experienced burnout? How did you deal with it?

Yes, I’ve come close. As Grist gets more widely known, we’ve also been scaling up our team, but there are some weeks when it feels like everything everywhere all at once is happening, and that’s too many things. I believe I deal with it by reflecting on my place in the universe and letting perspective soak back in. I have one of those small disks that says “The world was created for me” on one side, and “I am dust and ashes” on the other, and flipping that back and forth can help. I also become grumpy and reclusive, which can do wonders for naturally reducing my workload too.

What is the best way for a new developer to contribute to Grist?

If you are comfortable building and running tests, check out the “good first issue” label on the Grist repository. If you are more comfortable making web pages, you could read up on custom widgets and then think of a clever new way to visualize data in a table and add that. It can be as simple as a web page rendering some JSON you are handed by Grist. A pedigree chart for goats, perhaps, or an interpretation of some data as sheet music.

Where do you see the project heading next?

We’ve got a lot of fun features under development, some in collaboration with ANCT. I’d love to see two or three more ANCT-level contributors to the project, to get more done faster. Spreadsheets have a long tail of features that are really critical to someone, so there’s a lot to do. Any capable teams out there for whom Grist is great but would be better with just a few more key features, let’s talk!

Where do you see software development heading next?

It is hard to predict. LLMs and other models seem likely to change software development as much as the invention of the if statement. They are a new and very different building block, full of caprice and whimsy, which is not really a problem we’ve had to deal with in computer science before. I’m looking forward to it.

Where do you see open-source heading next?

I see open-source software as being the space within which humanity’s long term digital culture and treasure is accreting. Commercial software is important and impactful in the moment, but ultimately froth on a deeper tide. I was struck, when reading Vernor Vinge’s books, by his vision of software archaeologists in a distant future where, for example, buried deep in their infrastructure, the Unix epoch is still a thing. A glorious vision!?

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Before making a pull request, open an issue first to describe what you’d like to do, and why, and to check if the work would be welcome. Try to look at things from the perspective of the project maintainer. Everything you do will create work for them, from answering your questions, to reviewing your work, to suggesting improvements. So take a little extra time to make things easy for them any way you can. Don’t be discouraged by hostile or unresponsive people! Persist a bit, then move on - there are many happy, generous souls out there.

What is one question you would like to ask another open-source developer that I didn’t ask you?

What would you be doing with your life if we hadn’t somehow taught sand to think?

Back to you, what would you be doing with your life if we hadn’t somehow taught sand to think?

Trying to convince something even more unlikely to think? I have a whole theory of how to build logic gates with interwoven vines. Come to think of it, I used that as part of CosmicOS (e.g. here’s a D gate which compiles to this in the message).

Loved this interview? Want to get the future ones too? Subscribe to the free newsletter now!

Subscribe now

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Librum

Librum is an e-book reader with which you can manage your own online library and access it from any device anytime, anywhere. It has features like note-taking, bookmarking, and highlighting.

language: C++ stars: 1321 last commit: 5 days
repo: github.com/Librum-Reader/Librum
site: librumreader.com

Ladder

Ladder is a web proxy to help bypass paywalls. This is a self hosted version of 1ft.io and 12ft.io.

language: Go stars: 1995 last commit: today
repo: github.com/everywall/ladder

Livewire

Livewire is a full-stack framework for Laravel that allows you to build dynamic UI components without leaving PHP.

language: PHP stars: 21004 last commit: 1 day
repo: github.com/livewire/livewire
site: livewire.laravel.com

NewPipe

A libre lightweight streaming front-end for Android which supports YouTube, PeerTube, SoundCloud, and more

language: Java stars: 26036 last commit: 2 weeks
repo: github.com/TeamNewPipe/NewPipe
site: newpipe.net

🎙️ Interview with Stypox of New Pipe - A libre streaming front-end for Android

Hey! Thanks for joining us! Let us start with your background. 

I am Stypox, I am Italian and European, and I am currently studying computer science at University. I am fond of free software and have contributed to various projects since I learned how to code. I especially like building mobile applications because they can easily provide services to users right from their phone. I like modern technologies and I recently learned and used Rust (which I would strongly suggest even for e.g. backends), Kotlin (less verbose and less error-prone than Java) and Jetpack Compose (far better than normal Android development).
An interesting clue about how I started to code is the following: I was on a long trip by car and had nothing to do, so I decided to start to read a random C++ guide online. I got so excited about it that by the following long trip, I had already bought the book “The C++ Programming Language”.

What’s your most controversial programming opinion?

It seems like programming languages and frameworks are becoming more and more cluttered and slow, up to the point where the constant chip speedups do not keep up anymore. This is a serious problem for older devices that used to work well 5 years ago, but now are just too slow. I think we should stop putting layers of abstractions on top of one another, but rather reinvent frameworks and programming languages to achieve modernity and ease of use without sacrificing performance. Two great examples of this are Jetpack Compose and Rust, which provide modern and fast APIs.

Why was NewPipe started?

Back when NewPipe started, no serious YouTube frontend for Android existed. Therefore, NewPipe was created to provide a FOSS YouTube experience without trackers and with some additional features the official app doesn’t have.

How does NewPipe work?

NewPipe obtains the information to show on the screen in basically the same way a browser does it. Therefore, it makes requests to YouTube servers, pretending to actually be a browser, and it gets back data.

Where did the name for NewPipe come from?

One initial name proposal was GnuTube, however, considering such a name would have probably caused trademark issues, the “Gnu” was turned into “New”, and the “Tube” into “Pipe”.

Are there any overarching goals of NewPipe that drive design or implementation? If so, what trade-offs have been made in NewPipe as a consequence of these goals?

NewPipe does not only want to be a frontend for YouTube, but it also wants to promote streaming services that are more free. For this reason, multiservice support was introduced a few years ago, including support for PeerTube.

What is the most challenging problem that’s been solved in NewPipe, so far?

I would like to turn this question around: “what is the biggest problem that has NOT been solved yet?”. NewPipe’s player has been quite buggy from the beginning, and the code kept growing into spaghetti. Every time we fixed a part of the player, some other part would break, wasting a lot of user and developer time. We tried to do various refactors of the player, and although things have improved, we don’t think it’s possible to continue with the current structure and we would like to start afresh on that component.
Issues:  https://github.com/TeamNewPipe/NewPipe/issues/8616 https://github.com/TeamNewPipe/NewPipe/issues/7673
Refactor attempts: https://github.com/TeamNewPipe/NewPipe/pull/8170 https://github.com/TeamNewPipe/NewPipe/pull/5371 

Are there any projects similar to NewPipe? If so, what were they lacking that made you consider building something new?

NewPipe was one of the first apps of its kind, but later many more alternatives emerged, e.g. Invidious, SkyTube. Also, the fact that NewPipe’s extractor is separate from the app allowed some projects to use it directly without reinventing the wheel, e.g. LibreTube, Piped. Some community-maintained forks of NewPipe exist, too: e.g. NewPipe x SponsorBlock, NewPipe preunified, BraveNewPipe, …

How did the project get popular?

For various reasons:

• publishing the app on F-Droid has provided a lot of visibility among FOSS supporters

• there has been some media coverage

• NewPipe was trending on Hacker News a couple of times

• thanks to some NewPipe features that the official YouTube app doesn’t have, NewPipe could spread among non-techy people, too

What are you most proud of? 

When people have different ideas about how UIs should behave, it can be difficult to find a solution fitting everyone. This usually requires a deep understanding of the reasons behind the various standings in the community. A case when a common solution was successfully found was when we needed to add more buttons to the video details page: https://github.com/TeamNewPipe/NewPipe/pull/4534#issuecomment-741700417

Did NewPipe ever face any legal issues? 

This is how close we got to legal issues, however it ended well in a few days without us doing anything, so I don’t think it counts that much: https://newpipe.net/blog/pinned/announcement/newpipe-net-dmca-google-search/

How do you balance your work on open-source with your day job and other responsibilities?

That’s quite a difficult question. Working on open-source and on personal programming projects is often a source of happiness and satisfaction for me, so whenever I have time, I am more than happy to spend it on those things. However, considering that I am still studying, my timetable is quite messed up, and it happens quite often that I don’t have time to work on any open source or personal project for months. I always try to keep an eye on issues and IRC messages though, to find out about emergencies.

Have you ever experienced burnout? How did you deal with it?

Burnout for me happens when the next thing that needs to be done is quite big, but I’ve been out of the loop for a while, e.g. because of other duties. I tend to use some willpower anyway, and if that doesn’t work, I may set the project aside for some time and try to focus on something more exciting to get out of the burnout.

What is the best way for a new developer to contribute to NewPipe?

We would like to rewrite large chunks of the codebase soon, so contributors can chip in on https://github.com/TeamNewPipe/NewPipe/discussions/10118. We are not open for new pull requests at the moment, except for bug fixes.

Where do you see the project heading next?

We are planning a big rewrite to bring more stability and a modern look into NewPipe. Maintaining the current code has gotten a bit complicated, and we would like to switch to newer technologies. https://github.com/TeamNewPipe/NewPipe/discussions/10118 

Are there any other projects besides NewPipe that you’re working on?

I am building a FOSS voice assistant for Android named Dicio: https://github.com/Stypox/dicio-android. It already works, although it’s not perfect and has a limited number of skills available. I started this big project because I did not find any FOSS and on-device alternative to Google assistant.

Where do you see open-source heading next?

AI seems to be the next big thing, and there has been some open-source activity about it recently. It’s not impossible for open-source AI projects to keep up with big corporations, however there are still some things that need to be figured out, starting from the definition of when open AI really is. Which requirements do we need among these three, to define an AI as open: need open models, open training data and open source training code? And even if a project has all three, how can the normal user benefit from this, considering that AI models are generally not reproducible and require a lot of computational power and disk space to train and self-host?

Thanks for reading Console by CodeSee.io! Subscribe for free to receive the next interview.

Subscribe now

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Fleet

Open-source platform for IT and security teams with thousands of computers. Designed for APIs, GitOps, webhooks, YAML, and humans.

language: Go stars: 1674 last commit: today
repo: github.com/fleetdm/fleet
site: fleetdm.com

NewPipe

A libre lightweight streaming front-end for Android. Supports YouTube, SoundCloud, etc

language: Java stars: 25859 last commit: a week
repo: github.com/TeamNewPipe/NewPipe
site: newpipe.net

Spin

Spin is the open source developer tool for building and running serverless applications powered by WebAssembly.

language: Rust stars: 4252 last commit: today
repo: github.com/fermyon/spin
site: developer.fermyon.com/spin

Solidus

Spin is the open source developer tool for building and running serverless applications powered by WebAssembly.

language: Ruby stars: 4781 last commit: 2 days
repo: github.com/solidusio/solidus
site: solidus.io

Get high quality projects and interviews like these directly in your inbox by subscribing to the free newsletter!

Subscribe now

🎙️ Interview with Alberto of Solidus - an open source eCommerce framework

Hey Alberto! Thanks for joining us! Let us start with your background.

I’m Alberto from Chieti, a small town in the center of Italy. After I graduated in Computer Science in Rome, where I learned to code, I decided that the chaotic life wasn’t for me and I returned to my town with the goal of trying to live where a work-life balance is not a dream.

With my study-fellow Matteo Latini, living in another small city in Italy and a talented local designer Davide Di Stefano, we started Nebulab, a web development company that specializes in e-commerce and grow from 3 to some dozen of people all around the globe. 

Today, I’m the CTO of the company, and I'm still very connected with the code. My favorite language and framework are Ruby and Ruby on Rails, because they maximize the productivity of small teams, which is particularly important for us to help companies of any size.

Who or what are your biggest influences as a developer?

I’d say DHH (David Heinemeier Hansson). He is the creator of Ruby and Rails, and I respect its work and consistency over the years with the open source and its love for pragmatism. I read all the books he wrote with his partner Jason Fried, and the culture we are at Nebulab is also highly inspired by their thoughts. 

What’s your most controversial programming opinion?

Too many people push for using the latest trending technology before thinking about the real advantages to the health of the company they are working for. In particular, I’m generally against re-writing things instead of improving the existing ones. Most of the time, in my experience, it ends up costing more than fixing the actual problem with the current technology and, especially when it’s decided by some C-level or manager, it negatively impacts the team morale and long-term productivity. 

Why was Solidus started?

It’s a long, long story; let me try to summarize it as much as I can. Solidus is a fork of Spree, a popular open-source e-commerce platform, back in the days of the first DTCs explosion. When the company behind the project was acquired, it was clear that it was about to lose its open-source nature, so Solidus was born. A big chunk of the Spree community moved to Solidus and contributed until today to what it is. 

How does Solidus work?

Technically, Solidus is a Ruby on Rails Engine, which is a sort of plug-in that you can attach to any Rails application. Solidus enhances this application with the skeleton of functionalities needed to run an e-commerce store, leaving all the freedom to change it in any part as your business needs. It also provides a starter storefront, which is a ready-to-use template that any store can change with literally no limitation.

Where did the name for Solidus come from?

It comes from the ancient Roman coin (https://en.wikipedia.org/wiki/Solidus_(coin)) with a reference to the SOLID design principles (https://en.wikipedia.org/wiki/SOLID).

Is there a better name for an e-commerce platform whose target is developers? Probably, but I still love it.

What is the most challenging problem that’s been solved in Solidus, so far? 

One of the problems we solved has been making some of the most important piece of the business logic code pluggable. Instead of telling people to monkey-patch existing classes to change their behavior, we made the whole behavior swappable with a configuration class. If you need that class to do extra things, you can define your own, which inherits from the original, and add what you need. Here’s an example of the exposed configuration: https://github.com/solidusio/solidus/blob/main/core/lib/spree/app_configuration.rb#L414. With that, each store can quickly build its own logic to merge orders without patches here and there.

What was the most surprising thing you learned while working on Solidus? 

Well, it's the sheer diversity of needs in e-commerce. No matter how robust and flexible you build a system, there's always someone who'll throw a curveball at you with a unique requirement. It's like trying to create a one-size-fits-all glove in a world where everyone has different shaped hands. You've got your standard features, sure, but then comes along a merchant who needs something so specific, it's almost bespoke. This really drove home the point that in e-commerce, 'universal' is more a direction than a destination. It keeps you on your toes, constantly learning and innovating. And honestly, that's what makes this field so exciting.

How is Solidus currently monetized?

It’s not. Companies working with Solidus contribute to the project via GitHub. We are also collecting community donations via our Open Collective account. A group of stakeholders among community members chooses how to spend those funds, which is generally for events or support larger initiatives, like the Admin UX redesign we are currently working on. 

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Don’t be afraid to make mistakes. In my experience, a common aspect of the personality of open-source enthusiasts is the will to help people. This is what happened to me at the beginning as well: by just submitting a patch and showing genuine interest in solving problems, even without the proper skill set, I saw a lot of people trying to help out with what I was missing in order to understand the codebase better and complete my contribution. Being “mentored” while solving a specific problem I was facing kinda unlocked my growth for that specific area and made me a better developer; that’s precisely what I love about the open-source community. Of course, there are also people who are not willing to help and seem annoyed when contributions are not coming precisely as expected. To mitigate that, reading and sticking to the contributing guidelines of each project will help a lot, and when it’s not enough, well, you did your part! If your patch is rejected, you most likely have something to learn anyway. Move on to the next contribution!

Loved this post? Why not share it?

Share

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Geany

Geany is a small and lightweight integrated development environment.

language: C stars: 2840 last commit: 1 day
repo: github.com/geany/geany
site: geany.org

MarkText

MarkText is a simple and elegant markdown editor focused on speed and usability.

language: JS stars: 42026 last commit: Aug 2023
repo: github.com/marktext/marktext
site: marktext.cc

LocalSend

LocalSend is an open source cross-platform alternative to AirDrop.

language: Flutter stars: 21331 last commit: 2 days
repo: github.com/localsend/localsend
site: localsend.org

🎙️ Interview with Tien of LocalSend - An open source cross-platform alternative to AirDrop

Hey Tien! Thanks for joining us! Let us start with your background. 

Hi! I was born in Germany and my parents came from Vietnam.

Java was my first programming language. I found a really good German YouTube tutorial channel, “The Morpheus Tutorials,” and I watched every episode, programming some simple games with Java.

When I went to university, I learned many more languages. For a long time, Kotlin was my favorite, but eventually, I discovered Flutter. Since then, I’ve been eager to use Flutter in my first job.

Luckily, I found an open position at a software company that focuses on developing mobile apps with Flutter. I worked there for two years before deciding to become self-employed. Now, I develop products for clients and work on my own projects as well.

What’s your most controversial programming opinion?

To me, Python is overrated. I prefer programming languages that are statically typed. With Python, the code might look simpler, but you lose a lot of information if you don’t document your code properly. Of course, the same goes for JavaScript.

What is your favorite software tool?

Git and IntelliJ.

What are you currently learning?

I’m dabbling a bit in Rust.

Why was LocalSend started?

My sisters have different operating systems, and it was quite a challenge to share large numbers of photos and videos without compression from one device to another. We tried some open-source alternatives like Snapdrop, but unfortunately, the service was down at that time.

I wanted a solution that didn’t require any third-party service. I played around with shelf (a Flutter HTTP server package) a bit and realized that it works on all operating systems, including iOS and Android. After three weeks, I released the app on the App Store and Play Store with minimal features, like file selection and sending. The Windows version followed shortly after.

How does LocalSend work? 

LocalSend sets up a server on each device, and they discover each other by broadcasting a UDP message. Most importantly, file transmission is just a POST request — it’s essentially a file upload.

Why did you pick Flutter?

I like Flutter because it allows you to build cross-platform apps with very little effort. I don’t think I could have undertaken this project alone if I had chosen another framework.

Are there any projects similar to LocalSend? If so, what were they lacking that made you consider building something new?

Snapdrop is quite similar to LocalSend. The main issue with it is that it requires a third-party server. I wanted a solution that is completely independent of external networks.

What was the most surprising thing you learned while working on LocalSend?

Developing apps for Linux is really challenging. Without the contributions from the community, it would have been very difficult to create an app that works across all distributions. Even today, there are some issues that I can’t fix on my own.

How did the project get popular?

I think a Chinese website featured my app during the New Year. I can’t speak Mandarin, but I was really interested in how they discussed my app. In the following months, LocalSend gained users organically as more and more people wrote about it on the internet.

Is LocalSend intended to eventually be monetized if it isn’t monetized already? If so, how?

I don’t plan to monetize LocalSend. I’m happy to receive donations, but they’re just a nice extra.

What are you most proud of? 

I’m proud that LocalSend has garnered so much attention in such a short timeframe! I’ve even had clients from Germany compliment my GitHub profile because of LocalSend.

How do you balance your work on open-source with your day job and other responsibilities?

Usually, I divide my workload into days. For example, I might work on LocalSend for an entire day and then spend the next day working for my clients. Of course, if the workload becomes too much, I have to prioritize my actual job.

What is the best way for a new developer to contribute to LocalSend?

They can open issues on GitHub (especially with concrete reproduction steps), make minor UX improvements, or help translate the app into other languages!

Where do you see the project heading next?

I want LocalSend to support Bluetooth and Wi-Fi Direct file sharing, though both features require a deep understanding of low-level technologies.

Unfortunately, the Flutter ecosystem isn’t quite ready for that yet, so implementing these features would take a lot of time.

However, I also plan to provide a CLI version of LocalSend, so you can run a server without a GUI. This feature is more feasible, and I’ve already made some preparations for v1.12.0, so you can expect a CLI version soon.

What motivates you to continue contributing to LocalSend?

Like many developers, I enjoy seeing people use and appreciate my software. This project has also helped me build a reputation as a skilled developer.

Where do you see open-source heading next?

Open-source software reduces costs for consumers, as more and more open-source projects compete with commercial alternatives. In my opinion, open source offers significant marketing advantages to a project, especially within the IT community. People appreciate it, and so do I.

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

LinkWarden

LinkWarden is a self-hosted, open-source collaborative bookmark manager to collect, organize, and archive webpages.

language: TypeScript stars: 2835 last commit: 2 days
repo: github.com/linkwarden/linkwarden
site: linkwarden.app

Free Certifications

A curated list of free courses & certifications by the cloud study network.

stars: 10391 last commit: today
repo: github.com/cloudcommunity/Free-Certifications

LibrePhotos

A self-hosted open source photo management service.

language: Python stars: 5999 last commit: last week
repo: github.com/LibrePhotos/librephotos

🎙️ Interview With Niaz of LibrePhotos: self-hosted open source photo management service

Hey Niaz! Thanks for joining us! Let us start with your background.

Hello! Thank you for this interview. Let me tell you a bit about myself.

I was born in Germany, and I've primarily worked at the "Amt für Statistik Berlin-Brandenburg" (Office for Statistics Berlin-Brandenburg) throughout my career. We are a german government entity and create all the statistics about Berlin, and it’s surrounding areas. At the moment, I'm actively involved in maintaining their website, which you can find at statistik-berlin-brandenburg.de.

My journey into the world of programming began when I was quite young. As a kid, I was obsessed by the game "Age of Empires 2", which sparked in becoming a game developer.

Since then, I pretty much knew I wanted to learn to program, so I picked a high school which offered me the opportunity to do so.

I continued to learn programming by pursuing a formal education in computer science. I did a bachelor's degree in computer science at university, and am currently in the process of completing my master's thesis with the title ”Image Captioning in Personal Photo Management: A Comparative Study of Different Techniques” in the field of media and computer science.

Who or what are your biggest influences as a developer?

At this point, one of my most significant influences is Andreas Kling, the maintainer of Serenity OS. I find his approach to development truly inspiring. What stands out to me is his belief in the idea that "you can develop everything yourself." This perspective has had a profound impact on how I view technical complexity in software development.

For example, many people believe that it is impossible to write a browser from scratch. The Serenity OS project did it anyway, and the hand-made browser Ladybird is just very impressive and already supports a ton of websites.

Why was LibrePhotos started?

The inception of LibrePhotos is an interesting story that came about somewhat unexpectedly. It all began when I faced a challenging experience – a rejection from a job interview. To provide some background, I had applied for a TypeScript backend role after having already been a Java developer for two years. I had no prior experience in TypeScript at that point, but I had a referral at that company who encouraged me to apply anyway.

After successfully completing the technical interview, I then got rejected after I talked to the product manager. What made this rejection stand out was not the typical "not a good fit" reason, but rather that I was considered "too junior" for the role.

In response to this situation, I decided to channel my energy and anger into something constructive. I took a chance and decided to dive into working with Ownphotos, the precursor of LibrePhotos, with the intention of fixing a small bug to show to myself that I am by no means a developer who is too junior to dive into a complex codebase. 

As I made progress and successfully fixed the bug, I shared my achievement with the Ownphotos community on Discord. To my surprise, my contribution and enthusiasm sparked a wave of excitement within the Ownphotos community. People were genuinely excited about the work I was doing. This outpouring of support and enthusiasm from the community ultimately led to the birth of LibrePhotos. Since the fork, we added way more features than the original project and have built a community around it.

So, in a way, LibrePhotos was born out of a desire to turn a setback into an opportunity and a small bug fix into a community-driven project.

How does LibrePhotos work? 

Here's a behind-the-scenes technical overview:

The web frontend of LibrePhotos is written in React in combination with Redux Toolkit, and we leverage Mantine as a component library. 

There is a proof of concept app written in React Native and at some point we want to share the state management between the app and the web frontend.

Both frontends communicate over a REST API, with the backend.

The REST API also enable third party app developers to create their own apps. For example, UhuruPhotos, which is a native android application.

The core of LibrePhotos, the backend, is built using Python and leverages the Django web framework. We use it in combination with the Django Rest Framework (DRF) to create RESTful APIs.

PyTorch, a deep learning framework, plays a significant role in LibrePhotos for various machine learning tasks. 

For background job processing, LibrePhotos relies on Django Q2. This library provides a framework for handling asynchronous tasks.

How does LibrePhotos handle face recognition and other AI tasks? Does it do it locally?

It's all done locally, which means all the models and infrastructure are shipped too. It's a tricky thing to do as we have a wide variety of users. On the one hand, we have people on low-end devices with no computer science background, and on the other hand, there are Sys Admins running actual servers with Kubernetes. The one side complains about processing speed and the other side wants more accuracy and scaling options. We usually pick a default configuration, which works on most systems, while we hide more advanced options in the settings. As I will examine different image captioning models for my master thesis, I will implement a more modular architecture for shipping the models, which should help with that problem. The user can then choose the best model for their system, including quantized models for the very low end and state-of-the-art models for the high end.

Are there any overarching goals of LibrePhotos that drive design or implementation? If so, what trade-offs have been made in LibrePhotos as a consequence of these goals?

We are trying to solve to answer the w4 questions (what, where, when and who) for each image automatically or semiautomatically, as this is how people interact with a photo library.

For example, we have reverse geocoding, which turns coordinates into places you can search for, object detectors to find out what's in the image, semantic image search which helps you find phrases which match images, face clustering to find out who is in the image and a pretty complex engine to determine the correct timestamp. 

We have a clear preference to implement an automated system first, which leads to a not so great experience if you prefer manual editing of metadata.

Are there any projects similar to LibrePhotos? If so, what were they lacking that made you consider building something new? 

A lot of people have the idea that LibrePhotos is something new, but actually if you take into account that it is a fork of Ownphotos, we are the oldest one. Ownphotos was started in 2017 and Photoprism for example was started in 2018.

The biggest difference between us and Photoprism is, that our scope is bigger. We want to handle uploading / backing up images, and we want to ship apps in the future. 

Immich is also a great option, but it does run a pretty complicated architecture and  prefers importing into its own file structure over scanning existing galleries.

What was the most surprising thing you learned while working on LibrePhotos?

Before taking over as the maintainer, I only had development experience. Managing an open source project on the other hand needs a more diverse skill set. I am still learning how to do that.

It broadened my understanding why companies rely so much on telemetry these days. It was very surprising to me that power users who want new features are way more vocal than users who want less bugs. Also, users that donate money or pay money are usually in the later part, which makes estimating the impact of a feature vs a bug fix complicated.

What do you think about the state of self-hosting?

It is definitely growing a lot, but the biggest issue is still sustainability. While there is now a new model for open source called commercial open source software, it’s not yet clear how it will play out. 

VCs have figured out, that acquiring users by open sourcing the core product to compete with existing SaaS offerings can work out great. This leads to way more funding, which leads to more software, which is self-hostable. The goal at the moment is not revenue or profitability, which means in the short term we will have a lot of great software which is free and open source. 

After a couple of years, however, these solutions will get monetized to justify the investments, which will put features behind paywalls, closing the source code or changing the license.

How popular is LibrePhotos? 

We do not track anything, but Docker, Discord and GitHub do. We have 2.3 million docker pulls, ~1600 members on discord and ~6000 GitHub stars.

Is LibrePhotos intended to eventually be monetized? If so, how?

We will continue to only monetize by donations. I haven’t found a good way to monetize it in another way yet. SaaS does not make sense as this is pretty much the same business model as the big players and managed hosting does not seem to be that popular in the consumer space.

I applied for a grant from the Prototype fund in Germany, which would provide me with half a year of funding, but I did not get selected yet. Maybe research is another option, but I haven't found a concrete way to do this.

There is also not that much money in the space. The most monetized FOSS solution Photoprism and my estimation is that they make around 5k€ a month. Donations in FOSS also seem to be a winner takes all phenomenon, which means smaller project get exponentially less. 

What are you most proud of?

I am very proud that LibrePhotos made it into an Ars Technica article and that this project was in a popular Hacker News post. The only thing that is missing is a high ranking place on ProductHunt, and then I have all the essential nerd bragging rights haha

How do you balance your work on open-source with your day job and other responsibilities?

My job is pretty chill, so it’s easy to do some work in my free time. My studies however need more alignment. I usually pick courses where I can continue to work on LibrePhotos. Uploading and adding backing up images in the proof of concept was for example work I did for a course and my master thesis right now is about improving the automatic image captioning capabilities of LibrePhotos.

Have you ever experienced burnout? How did you deal with it?

Yes, for a while I had the ambition to do this full time after my studies. However, last year it became clear to me, that this is not an option based on the current trajectory.

I took a break for a bit, while I recalibrated how I align myself better with the project. Doing this as part of my studies was part of the solution, but I also switched to a more relaxed release schedule, gave myself more permission to not respond immediately to issues or pull request and scaled down the amount of public relation work I did.

Where do you see the project heading next?

There will be a new and novel way to do automatic image captioning in LibrePhotos. I will probably combine Blip with LLAMA 2 and see how many use cases it can cover and what will be possible with it. It's kind of experimental, but I am excited where this will lead.  This is all part of my master thesis, which means I will definitely do it, I even got a deadline when I have to hand it in and everything.

Otherwise, I will merge the pull requests contributors submit. If you want something added to LibrePhotos, you can anytime open one up. I am pretty open about new features and am not too strict about it as long as it improves the core product.

Where do you see software development heading next?

To me, it looks like the stack changes less often than it did in the past. We now have a lot of well established open source languages and frameworks. This means we will find less new ways of doing things and will focus more on more performant solutions and an ecosystem, where you can do everything in the language you like. Good examples of this are JavaScript and Python.

While LLMs seem to be helpful when coding, they still need a lot of human supervision. Programmers will get a bit more productive, but it will not be anything game-changing.

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

Product for Engineers is PostHog’s newsletter dedicated to helping engineers improve their product skills. Subscribe for free to get curated advice on building great products, lessons (and mistakes) from building PostHog, deep dives on top startups, and very cute hedgehog illustrations.

Subscribe for Free

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Leantime

Leantime is a strategic project management system for non-project managers.

language: PHP stars: 3147 last commit: today
repo: github.com/Leantime/leantime
site: leantime.io

Intermediate Graphics Library

A cross-platform library that commands the GPU. It encapsulates common GPU functionality with a low-level cross-platform interface. IGL is designed to support multiple backends implemented on top of graphics APIs like OpenGL and Vulkan with a common interface.

language: C++ stars: 2527 last commit: 2 days
repo: github.com/facebook/igl
site: facebook.github.io/igl

Valhalla

Valhalla is an open source routing engine and accompanying libraries for use with OpenStreetMap data. It also includes tools like time+distance matrix computation, isochrones, elevation sampling, map matching and tour optimization.

language: C++ stars: 3791 last commit: 2 days
repo: github.com/valhalla/valhalla
site: valhalla.github.io/valhalla

🎙️ Interview With Kevin and David of Valhalla - Open source routing engine for OSM

Hey Kevin and David! Thanks for joining us! Let us start with your backgrounds

Kevin: I’m from a tiny village called East Hanover in Pennsylvania. I got interested in programming and started learning it seriously in 2000 when I was in 11th grade thanks to a teacher who happened to also be a graduate professor at Penn State. Fresh out of a comp-sci undergrad program at Millersville, I started out in graphics programming building “video games for surgeons”. From there I went into a PhD program at the University of Delaware but dropped out of that to join Dave at MapQuest. Since then I’ve been working on mapping as a sub-genre of comp-sci ever since. Over the years I’ve worked at Mapzen (where we created Valhalla), Mapbox and now at Snapchat. C++ was my first language, has been my primary language throughout my (15 year) career and is still my favorite language by far though I like to use python and bash when the situation calls for it.

David: I am from Lancaster, Pennsylvania and now live near Port Deposit, Maryland. I’ve been in the software industry for over 40 years. I started at The Johns Hopkins University Applied Physics Laboratory doing mapping and computer graphics work while developing prototype naval command and control systems for nearly 18 years. I also taught Computer Graphics for the Johns Hopkins University Engineering for Professionals masters program for over 20 years. I spent nearly 15 years at MapQuest leading development of their driving directions software. I then moved to Mapzen, which is where we started Valhalla, and then to Mapbox (which uses Valhalla). I now work at Hammerhead Navigation (part of SRAM), developing mapping and navigation systems for bicycle head units - also using Valhalla.

I have worked with C++ for most of my career, and it is by far my favorite development language. Python is quite useful as well.

What happened to the video games for surgeons?

Kevin: It was a start-up called verify technologies. It was mildly successful in the US for a while, but eventually wasn't able to compete with other US companies that not only had the software but also had hardware patents on the bits you needed for simulating the instruments the surgical residents used to train with.

What’s your most controversial programming opinion?

Kevin: node.js is a psyop

David: I’m not sure it is controversial, but event driven software is difficult for me. No matter how hard I try, RxJava just baffles me. There must be better ways!

I also have never been a fan of XML and JSON as data transfer mechanisms. In Valhalla, we use protocol buffers, which are much more efficient.

What is your favorite software tool?

Kevin: I’ve been really liking CLion and really all the JetBrains tools lately regarding software development.

David: I’ve been doing Android programming lately and appreciate Android Studio. However, I’m old school though and still mostly use vi for C++ development, with occasional use of Visual C++. Maybe it is time to try CLion.

Why was Valhalla started? 

Kevin: Mapping solutions generally need to provide three separate but interconnected high-level bits of functionality. Those are: Search, Display and Routing. Valhalla was meant to solve the latter at an open source mapping API start-up called Mapzen which was run out of the Samsung Accelerator program (a start-up incubator). At the time the open source routing engine scene was still fairly young which allowed us (the team that conceived of Valhalla) the opportunity to build one from scratch by focusing on a few novel primary features not seen in competitors.

David:  When Valhalla was started, there were many routing solutions handling driving routes pretty well. However, other modes of transportation were second class citizens. Bicycles, pedestrian, motor scooters, and mulit-modal routing didn’t have the support and quality as automobile routing.

How does Valhalla work?

Kevin: This is quite a difficult question to answer simply because of the scope of the project but I’ll give it a try. Valhalla may likely be best understood by some of its design goals/philosophies coupled with a description of some of its components. Valhalla is a C++ library which provides a number of APIs centered around the key problem of routing, directions and wayfinding. The library is accessible via python bindings, an HTTP api or by calling the C++ library directly. The library is divided into several core bits of functionality:

Data-processing - to turn OSM data into a compacted and normalized graph

Routing - finding a path through the graph at run time

Directions - annotating that path with maneuvers and natural language descriptions of how to follow the path

The data processing is done offline and consists of pulling in OSM-formatted data, elevation, transit, historical traffic patterns or other data sources and building a graph that we can route on. Routing and directions are performed at request time.

Design-wise, Valhalla uses a tiled hierarchical data structure to store the graph. Each tile adheres to a custom binary format that requires (almost) no parsing. This allows Valhalla to memory-map the data and do a little bit of pointer arithmetic to facilitate graph searches. Tiles also lend themselves well to smaller footprint devices like cell phones.

Once a path is found it is annotated with maneuvers and text descriptions and the route itself is serialized to JSON (or protocol buffers) for the client/caller to consume in their application.

There are lots of different APIs apart from the typical A to B “shortest path” algorithm. Additionally, Valhalla supports real-time traffic conditions, adheres to time of day restrictions and allows for a large variety of customization of the “type” of route at request time.

David: I will add to what Kevin states above. While Valhalla does not directly support navigation along a prescribed path (generated from Valhalla’s routing methods), the information from Valhalla readily supports navigation and guidance. Valhalla’s maneuver generation (calling out where turns require guidance to the user) and directions are first class. Also, many commercial mapping data sources focus on driving. OSM (until recently) has had much better support for pedestrian and bicycling than commercial sources, and Valhalla enhances the ability to route and navigate for these travel modes.

How does Valhalla support real-time traffic conditions and adhere to time of day restrictions?

Kevin: The algorithms support tiled real-time data sources that, if present on the machine where the services are running, will be used to influence the computed route. We do this by creating a tar file which has real time tiles in it, initially blank. Then it is the system integrators job to write a process that updates the tar file on the fly. The routing process has this tar memory mapped, and the "sidecar" process also memory maps this same tar for writing updates.

We leave the process of getting those updates from whatever traffic data provider up to the person administrating the service. Since there are infinite ways the data could look or be delivered, it's best not to have too much of an opinionated stance there and let that up to individuals and their specific project needs.

Who, or what, was the biggest inspiration for Valhalla?

Kevin: I had been working at MapQuest with Dave and the rest of the Valhalla team for around 6 years. For the first 2 years I worked on the routing side of the maps API equation, but for the remaining 4 years I had been the lead developer on the display side creating visual map tiles that the end user would see. Unfortunately, MapQuest decided it wanted to relocate its primary development office out of Lancaster, Pennsylvania (where it was founded) and focus more on its office in Denver, Colorado. At that time Mapzen was looking to expand its offerings to include routing so we decided to join and become the team supporting that effort.

Divide and conquer is an old idea. If you can break a problem into many smaller easier to solve problems, and solve them individually, then getting the full solution is only a matter of scale. Not surprisingly, mapping, especially the display part of the problem, exploits this design paradigm. Specifically, we break the visual map into tiles and send those to the client/caller who wants to display them. In this way they can decide on how much of the map they want to see to some level of granularity, they can decide on the level of parallelism they want to use in fetching the tiles, and they can use the tiling (since its a uniform grid) as a cache key to avoid having to always fetch data if its data they had fetched before.

I suggested to Dave that when we join Mapzen we should consider tiling the graph; meaning we would chop the graph into a uniformly gridded (spatially) set of data blobs. He and I then briefly talked about how such a data structure could exist and maintain a proper topology. Mapzen was an open source company so we knew it would be open. Dave’s experience in routing made it clear that there had to be some degree of hierarchy in the graph structure (for performance reasons). This is when Dave said to me “We should call it T.H.O.R.: tiled hierarchical open-source routing”. To which I said, “there’s going to be a lot more components, we should call it Valhalla and each piece can be one of the Norse mythological figures”.

David: THOR was just one of the components of Valhalla - there are other code directories (not really libraries, rather groupings of code under the Valhalla umbrella) that we named with the Norse theme. Valhalla ties it together under a single repository. Kevin answered this quite thoroughly. Kevin’s idea of tiling the routing graph is a crucial part of Valhalla. It is a key differentiator and one that allows Valhalla to be successfully used in mobile environments.

Are there any overarching goals of Valhalla that drive design or implementation? If so, what trade-offs have been made in Valhalla as a consequence of these goals?

Kevin: There are several “requirements” that we adhere to as far as design/implementation goes:

• It should run on very modest hardware (Raspberry Pi, old android phones) with very modest resources

• The graph should contain general attribution so that at request time we can customize what makes the “shortest path” by weighting the graph attributes differently depending on request parameters

• The router should work with an incomplete graph (if you are missing tiles the router will still route)

• Data will remain backward and forward compatible within the same major version

The primary trade-offs we’ve hit so far are not being able to change around the graph data structure so that we don’t break data compatibility. We also trade run-time performance for the sake of request-time customizability.

David: The idea of tiling the routing graph has been key to allowing Valhalla’s use in mobile navigation applications. At Hammerhead, we have successfully used Valhalla for bicycle routing and navigation on custom Android hardware with modest processing power, memory, and storage.

What is the most challenging problem that’s been solved in Valhalla, so far? 

Kevin: It’s a very imprecise answer, but the final composition of all the parts that make the final user-facing API is the most significant problem that the software solves. There is incredible complexity in tying all the pieces together in a coherent and usable way I’ve already mentioned some, but there are a lot of discrete pieces:

• The offline ETLs (there are at least 5 different data sources we combine)

• Online ETLs for real-time traffic conditions and incidents

• The system for customizing the path finding to match the users request-time preferences

• Supporting like 8 modes of travel on a single dataset

• The spatial indexing system so we can find candidate edges in the graph given input coordinates

• The 8 different graph traversal algorithms that accomplish different tasks on the network

• The maneuver generation and system for localization the narrative that go with a given maneuver

• Different serializers depending on which of the 8 different APIs you called

• The operational design and componentry that ensures request isolation for maximal server stability

David: I’m not sure if it is truly solved, but developing a solution for isochrones was challenging and interesting. Isochrones allow a visual representation of where one can travel within specific time increments.

Are there any projects similar to Valhalla? If so, what were they lacking that made you consider building something new?

David: The closest I can think of are OSRM (Open Source Routing Machine) and GraphHopper.

OSRM is open source and very fast/efficient for solving many routing problems. However, it is not readily usable on low-end systems and mobile devices. Also, it is not very flexible - if you want to adjust costing (which is key to determining the lowest cost path) you have to regenerate the data. So it trades off customization and flexibility for high performance.

I’m not sure how much of GraphHopper is open source. It was somewhat mature when we started Valhalla. It is written in Java and has some customization as well as more performant options similar to OSRM. The main drawback compared to Valhalla is that it isn’t tailored to mobile environments. I think they supported an Android build at one time, but later they dropped support.

What was the most surprising thing you learned while working on Valhalla?

Kevin: Memory-mapping is one of the most useful tools that kernel designers ever thought of and I thank them every day for it. It allows you to do amazing things on very modest hardware. If you’ve not played with it, you should give it a try and spend a day learning about it.

David: I agree with Kevin on this one, though I learned about memory mapping a long time ago. I have successfully used memory mapping across many projects and organizations, starting with memory mapping with Unix workstations in the early 1990s. Being able to efficiently access data from files without copying to local structures or classes is such an advantage over other approaches.  In Valhalla, we structure the routing graph in such a way that when computing routes we can use pointers to memory mapped locations and cast them to structures (or classes) that we can immediately access in code. This is quite efficient!

Where is Valhalla currently being used?

Kevin:

• Tesla client/server-side navigation

• Mapbox (and its automotive customers like BMW)

• Mapillary (acquired by Facebook/Meta)

• Many many start-ups that want to focus on their core competency rather than build their own router

David: I will add that Valhalla has been used in several bicycle navigation computers (small devices mounted near the handlebars). The company I work for uses Valhalla. I also believe that Wahoo does as well (they did several years ago).

Does Valhalla get support from companies that are using it?

Kevin: Yep, a number of the contributors use Valhalla in their day jobs, which leads to contributions here and there, both large and small.

David: If you count the occasional code contribution back to Valhalla, then I would say yes.

Is Valhalla intended to eventually be monetized?

Kevin: No. Valhalla is technically part of the Linux Foundation. This allows us to seek funding for its development using Linux Foundation procedural guidance but thus far we haven’t needed to take advantage of that. Other private companies do monetize Valhalla as part of their own API offerings or as part of their primary product as a feature to paying customers, but the open source project is maintained by its founders, and we have never had plans to monetize it. I, for one, would oppose such an idea.

David: No.

What are you most proud of?

Kevin: I took great pride in working with the Tesla engineers to integrate Valhalla into their in-car navigation system, as well as with the BMW engineers on their integration as well. The automotive industry is an incredibly interesting and highly competitive space, and I’m proud that Valhalla could serve those use-cases.

David: Automobile navigation is an exciting (though highly competitive) industry, and I take pride seeing its use in this industry. However, as a long time cyclist, I really enjoy seeing my work with Valhalla used in this space.

How do you balance your work on open-source with your day job and other responsibilities?

Kevin: Over time, I’ve evolved primarily into a mentorship role. This means I do code review, design and Q and A on other folks who are cutting their teeth on the code-base. From time to time, I will jump in and implement something if the scope is manageable time-wise. I have a big family (6 kids) and so I’m primarily only able to contribute in the mornings, nights and weekends as most of the rest of my time is dominated by my day job and my family’s needs.

David: I leave most of the Valhalla open source work to Kevin and others, only occasionally responding to questions and issues. 

Have you ever experienced burnout? How did you deal with it?

Kevin: Yes. When you work for a company that primarily sells access to APIs and you are successful, you will end up inevitably courting many different companies with different needs all at once. The trick to navigating that is finding ways to kill 2 birds (requested features) with one stone (implementation/developer time) but often we are talking about killing 2 birds, a zebra, maybe an elk and a whale and it's just not possible to find a way to meet all their needs without dedicating separate efforts.

The only way to deal with it is to prioritize the various animals you’re hunting (is the metaphor still working?), assign developer resources starting at the top of the priority list and take some estimates. You’ll then need to negotiate on timelines with each animal individually and see what you need to juggle around. The problem is over constrained so at the end of the day you have to be ok with failing to bag some of your wild game.

David: I can’t think of any times I felt truly burnt out in my career. I moved to part-time employment this year after 41 years of full time work as a software engineer. I guess that is one way to alleviate burnout!

Where do you see the project heading next?

Kevin: Every day, a new person posts an issue asking about: can it do X thing? The answer can sometimes be “no and it likely will never do,” but more often than not the answer is “interesting, we could do it like this, is that what you are thinking, cool to make it happen I would suggest…”

David: I could see Valhalla adding support for navigation and guidance, so it can be more easily used in mobile solutions. For now, navigation support is left to the developers who integrate Valhalla into their project.

Are there any other projects besides Valhalla that you’re working on?

Kevin: There is a library called prime_server which Valhalla uses for its operational concerns. It's the thing that we wrap the library with to handle request-flow through the different pieces of the library in an operationally sound way. The API is very stable, but here and there we find issues, patch them, and upgrade Valhalla to take advantage of them.

David: No.

Where do you see software development heading next?

Kevin: Is this my cue to jump onto the AI bandwagon? Call me a skeptic or maybe a romantic but I truly hope that AI does not become a primary tool engineers make use of in the near future. At least not in the way that the evangelists would like us to believe, i.e. the AI writes your code for you after you rattle off a few natural language commands to it.

Don’t get me wrong, it could be great for a bit of task management or menial work. For example, in Star Trek the captain would say things like: “Computer, how many life forms are on that Borg Cube?” I could see a reasonable integration in an IDE where the AI does analytics on a code-base to help inform the engineer rather than to do the engineer's work for him. The engineer could say something like: “Find me instances of that object which never call their ::Foo method”.

I remember when people thought map-reduce was going to change the world, or when they thought machine learning was going to change the world. And to a degree they did; they became tools we use to solve problems. With AI, it could be another generic tool but the generative bits, I’m skeptical that they could be used for anything other than trivial tasks. The purpose of the AI is to take what it’s seen, identify patterns, and, in some form, regurgitate those that match the user’s intent or combination of intents. While combinations could be somewhat novel the components will not truly be. I don’t really want to think that engineering, which essentially requires a person to invent new things, will be boiled down to whatever things engineers of the past did as understood by an AI that was trained on them. I would prefer engineering to remain a discipline where the potential for truly novel ideas continues to exist.

David: I don’t think about this much. I remember in the 1990s when virtual reality was the next big thing. Then the hype died down and virtual reality found its uses (though not for everything the hype predicted). Same thing seems to happen every few years.

Liked this interview? Subscribe to the free newsletter to receive the next one in your inbox!

Subscribe now

Interested in sponsoring the newsletter, or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Clickvote

Add upvotes, likes, and reviews to any context.

language: TypeScript stars: 1281 last commit: today
repo: github.com/clickvote/clickvote
site: clickvote.dev

gpt-pilot

gpt-pilot is a dev tool that writes scalable apps from scratch while the developer oversees the implementation

language: Python stars: 5929 last commit: today
repo: github.com/Pythagora-io/gpt-pilot

Kestra

Kestra is a universal open-source orchestrator that makes both scheduled and event-driven workflows easy.

language: Java stars: 4359 last commit: 2 days
repo: github.com/kestra-io/kestra
site: kestra.io

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

daktilo

daktilo is a small command-line program that plays typewriter sounds every time you press a key. It also offers the flexibility to customize keypress sounds to your liking.

language: Rust stars: 586 last commit: today
repo: github.com/orhun/daktilo
site: daktilo.cli.rs

fail2ban

Daemon to ban hosts that cause multiple authentication errors

language: Python stars: 8843 last commit: 2 weeeks
repo: github.com/fail2ban/fail2ban
site: fail2ban.org

Wike

Wikipedia Reader for the GNOME Desktop

language: Python stars: 192 last commit: 3 days
repo: github.com/hugolabe/Wike
site: hugolabe.github.io/Wike

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!

This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io

🏗️ Projects

Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!

Vizro

Vizro is a toolkit for creating modular data visualization applications. Rapidly self-serve the assembly of customized dashboards in minutes - without the need for advanced coding or design experience.

language: Python stars: 997 last commit: 2 days
repo: github.com/mckinsey/vizro
site: vizro.readthedocs.io/en/stable

Bruno

Opensource IDE for exploring and testing APIs. Lightweight alternative to postman/insomnia.

language: JS stars: 1088 last commit: today
repo: github.com/usebruno/bruno
site: usebruno.com

PostgresML

PostgresML is a machine learning extension for PostgreSQL that enables you to perform training and inference on text and tabular data using SQL queries.

language: Rust stars: 4168 last commit: 1 day
repo: github.com/postgresml/postgresml
site: postgresml.org

Join thousands of other open-source enthusiasts and developers in the Open Source Hub Discord server to continue the discussion on the projects in this week's email!

Configu

Configu is a simple and generic standard for managing and collaborating software configurations. It is built to provide a flexible and extensible process for storing and orchestrating configuration data across different environments and systems.

language: TypeScript stars: 1209 last commit: 4 days
repo: github.com/configu/configu
site: configu.com

CubeFS

CubeFS is a cloud native unstructured data storage.

language: Go stars: 3934 last commit: 4 days
repo: github.com/cubefs/cubefs
site: cubefs.io

life-dashboard

E-Iink based dashboard for everyday life. Low power, heads up display running on a Kindle.

language: Go stars: 1229 last commit: 2 days
repo: github.com/davidhampgonsalves/life-dashboard

Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!

Leave a comment

Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!