Console 48
Sponsorships
Important, Not Important
In 1986, Richard Hammer, titan of American mathematics and pillar of Bell Labs, wandered over to the chemistry table in the lunch room and asked: “What are the most important problems in your field?”
The next week he asked, “What important problems are you working on?
And the next week he asked, “If what you are doing is not important, and if you don’t think it is going to lead to something important, why are you at Bell Labs working on it?”
There’s never been a better time to work on the world’s hardest problems, whether you’re an open-source software engineer, an epidemiologist, or a wind turbine technician.
And that’s where the Important, Not Important newsletter comes in.
We curate the most vital science news of the week, add contextual analysis, and give you data-driven Action Steps you can take to feel better, and drive transformational change.
From climate to clean energy, AI to antibiotics, food & water to whatever other fresh hell could either kill us all, or turn this place into Star Trek, we’ve got it covered.
Once a week. 10 minutes or less. Get it for free at importantnotnotimportant.com
GUN
GUN is an open source cybersecurity protocol for syncing decentralized graph data.
language: JavaScript, stars: 12750, watchers: 317, forks: 833, issues: 216
last commit: April 06, 2021, first commit: March 17, 2014
DVC
DVC is git for data and ML models.
language: Python, stars: 7684, watchers: 124, forks: 729, issues: 561
last commit: April 09, 2021, first commit: March 04, 2017
https://twitter.com/rkuprieiev
Tinyland
Tinyland is a very small version of the Bret Victor Dynamicland project.
language: Python, stars: 65, watchers: 7, forks: 8, issues: 5
last commit: October 15, 2019, first commit: September 27, 2019
https://twitter.com/emmazsmith
Wistalk
Wistalk allows you to analyze a Wikipedia user’s activity.
language: Python, stars: 7, watchers: 3, forks: 1, issues: 0
last commit: March 29, 2021, first commit: March 20, 2021
https://twitter.com/altilunium
An Interview With Mark Nadal of GUN
Hey Mark! Let’s start with your background. How did you learn how to program and what languages or frameworks do you like?
I’m a mathematician at heart, rebel in practice - so I dropped out of academics into what I thought was math’s cousin, programming. Lo and behold, there is rarely anything pure or logical about programming. But it was too late, code changes the world, not theories, so I keep trying to build the ideal with broken strings. My belief is to choose the most open and user facing technology tools, so while I do not like them, browser compatible javascript is the sword for my revolution.
Who or what are your biggest influences as a developer?
Claude Shannon and Feynman. They built certainty out of uncertainty, that’s marvelous. Don’t get too hung up on trends, build tools that will make being human move from tolerable to marvelous.
What's an opinion you have that most people don't agree with?
Most of my opinions. I’ll only mention one that will make people hate me rather than scorn me. “If your variable names are longer than a mathematical symbol, then your code will become bit rot.”
What’s your most controversial programming opinion?
I got a lot of flak for this one.
What is one app on your phone that you can’t live without that you think others should know about?
I hate apps, so I disagree with the question. Some of the few I do have though are SMS Backup+, SwiftKey, and Video Collage. A life saving tip tho is this hack to reduce robocall spam
If you could dictate that everyone in the world should read one book, what would it be?
I don’t read books. Write them.
If you could teach every 12 year old in the world one thing, what would it be and why?
First principle reasoning. We need more people who can independently invent without being seduced by head-nodding expertism.
If I gave you $10 million to invest in one thing right now, where would you put it?
Other than my own projects? Very few things in tech excite me, but Rik Arends’ Makepad, HARC’s Ohm, various AI projects, etc. are important for software to improve. For returns, obviously Elon Musk and Bitcoin.
What are you currently learning?
When I have time? Rust and writing my own AI trained physics engine.
What resources do you use to stay up to date on software engineering?
If you have to stay up to date on something, that’s an indication it will go out of fad as fast as it came in.
Do you listen to music while coding? If so, what have you been listening to lately?
How do you separate good project ideas from bad ones?
Whether they reduce complexity in the world or add to it.
Why was GUN started?
I had written a driver for MongoDB in 2010 but then later needed a graph database, so I started writing a driver for Neo4j in 2013 but was terrified to discover everything was serialized as a table with duplication. It would take weeks to implement, not support real-time updates, and would be locked into a bad master-slave ideology. So I had a choice, spend that time implementing yet another database driver, or use that time building my dream database? 6 years later, we did a quarter billion downloads in 2020 alone.
Does GUN stand for anything? If so, what?
Graph. Universe. Node.
Who, or what was the biggest inspiration for GUN?
Kinda like in the Silicon Valley show, my servers caught fire when my previous company was highlighted in the Wall Street Journal. Waking up to 3am pager calls is no fun, especially only to find that everything is working except the database. As a mathematician I had worked on plenty of applied distributed systems problems, so it was sad to see all the major databases at the time being designed around centralized solutions. It just didn’t make sense, master-slave systems are not as scalable or resilient, and a database is exactly the sort of thing you want to be fault tolerant. So I wanted to make the future be a little more egalitarian. And while many disagree with this analogy, historically the best way to prevent slavery has been to empower every man and woman with the ability to defend themselves - an alternative acronym for GUN is “Governed Under None”.
What was your previous company?
My previous company was a collaborative web design tool that used video game controls. I have a few videos and demos here.
Are there any overarching goals of GUN that drive design or implementation? If so, what tradeoffs have been made in GUN as a consequence of these goals?
GUN is what is considered a “highly available, strongly eventually consistent” database which means you should not use it for banking, according to the CAP Theorem. If you’d like to know more about this subject, check out our cartoon explainer at https://gun.eco/distributed/matters.html . The goal of GUN is to be maximally efficient with as minimal code or dependencies as possible, yet still be robust enough to survive nuclear apocalypses. A simple way to think about that first part is to imagine if billions of people were to use a Ready Player One like metaverse - I won’t stop improving GUN until it can handle that type of performance and scale. The implications of this from an engineering standpoint is that I’m trying to model and replicate the same algorithms that nature and reality itself uses for transmitting information. The second part is about creating a fault tolerant system that is not only easy for developers to build apps with, but also continue working indefinitely without any maintenance even in the face of external catastrophes. This is important from a human perspective, not an engineering one, because it allows people to have ownership over their own data and memories, in perpetuity, without being locked into somebody else’s platform. It is also a perfectly reasonable goal too, the pyramids of Egypt have stood for thousands of years built with materials that erode, yet the logic of Pythagoras and Euclid has not decayed by a single granule over millennia. So why shouldn’t apps, built with platonic algorithms, outlast the silicon sand they run on?
This seems to be a very similar design goal to Bitcoin. Were you at all influenced by those design goals?
No. But they're good design principles, it isn't hard to find a lot of p2p/decentralized people prior/after Bitcoin.
What is the most challenging problem that’s been solved in GUN, so far?
The entire protocol can be re-implemented in as short as 30 minutes, like I did on stage in this talk https://gun.eco/docs/Porting-GUN . All the database stuff turns out to be surprisingly easy. But the thing that 7 years later still gives me a hassle, headache, and nauseous pain? The hardest thing to implement is an easy, elegant API. Developers in the community swear by the API, gushing about how magically simple and powerful it is. But this is pure illusion by abstracting complexity, and this is what a good API does, it is deceptive. Solving timeless distributed systems problems is a piece of pancake compared to making those same solutions be appetizing to use.
What is your typical approach to debugging issues filed in the GUN repo?
First replicate it, then use this secret debugging technique the whole world should know about. Write a wrapper around your println/console.log that only logs a message if it matches a particular sequence number. Now put “1, 2, 3…” log statements into where you think the code flow must go to hit the bug. Even if the same function is called 20 times, the log will only print for the isolated pass relevant to the bug - or is not called at all (or with the data you expected), which incrementally indicates where your mental model of the code deviates from the actual call paths, thus probably the bug (or closer to it, ghosthunter!). It is hard to explain this process in text, if anybody wants to record a screencast of it with me, just ping me.
This isn't anywhere in the GUN code as a utility is it?
This shouldn't be in the source code, it will be deleted when I stop debugging haha and then `console.only(1, 'here')` ... `console.only(2, 'next?")` ... and so on. Somebody in my community also wrote a tracer for it too, that generates flow charts for a particular command. But this is all very hacky, by hand, not something easily/generally usable for others yet.
What an interesting technique! I feel this would be born of necessity while debugging distributed systems bugs since they're inherently more difficult to debug than non-distributed systems.
Actually no, this is just for single-thread/process code flow, we built panic-server for simulating distributed systems correctness & load testing.
What is the release process like for GUN?
We strictly adhere to the religious diet of “Do I feel like it? OK.” release schedule. I openly admit I’m wrong about this.
Is GUN intended to eventually be monetized?
I strongly disagree with “open-core” (crippleware) and other licenses, especially (A)GPL, etc. and am committed to MIT/Zlib/Apache2/BSD style licenses. Open source is about creating value for everyone. This means I will never be able to monetize GUN directly, and that is a good thing - it keeps it transparent, accountable, and free of adverse incentives or scammy schemes. If a tool is truly valuable, it will make building other services that can be monetized be exponentially easier to create and maintain. So someday I’ll charge (or more specifically be the payment processing layer) for a DBaaS service like Firebase, but federated, for anyone who is too lazy to 1-click deploy their own self-hosted solution.
Any idea when “someday” will come?
6 months in my dreams. So... 6 years by over-estimating engineering standards? :P My hard deadline is 2.5 years. Hopefully half that.
How do you balance your work on open-source with your day job and other responsibilities?
I’m blessed with the opportunity to work on this full time, thanks to donors and investors who believe in long term value creation and who are dedicated to fixing the mess that has taken over the internet. This also means, after a year like 2020, making sure you have the mental health to ground you for such challenges. At the end of the day, it means doing the right thing that will make you last and survive, not necessarily what is trendy or sexy.
Do you think any of your projects do more harm than good?
The technology I create is apolitical, though it, even its name, is deeply saturated in ideology. The name reminds good people to use something with care and caution - a database is a loaded gun when it contains something as sensitive as users’ data, don’t act or tread upon it lightly. Bad people will figure out how to destroy things, regardless of the tool or its warnings. It deeply saddens me that there is no doubt people who have used it for hateful things. My belief though, is that good will outgrow the bad, especially if technology and systems are designed to prevent monopoly control. And through and through, the essence of GUN is that it is code that enables techno-egalitarianism - no masters, no slaves, no monopolies, no dictators, no tyrants. It gives everyone the power, even if some are terrified that such power exists in the first place.
What is the best way for a new developer to contribute to GUN?
Jump into our chat room and start asking questions, complaining about bugs or problems, and share what ideas and projects you are passionate about! Being human is the best way to start. :) http://chat.gun.eco/
If you plan to continue developing GUN, where do you see the project heading next?
We crossed 40 million downloads in one month, and now I’m desperately trying to get the next version out and work to scale up to 100M monthly users.
What motivates you to continue contributing to GUN?
If I was passionate about something, that could change and the project would be dropped. But if I encounter a pain point, a problem so overwhelming that I cannot proceed or make progress, then I’m forced to keep working on it regardless of whether I’m passionate or not. GUN is not what I want to be doing, but it is the tunnel that must be bored through an insurmountable problem, that will let me achieve the dreams I actually seek.
Have you always been this intellectually fearless, or was there some kind of inflection point in your life that lead to this?
Why thank you. Both? It took me till last year to realize that most people don't understand anything I say because communication is primarily done through tribe signaling, not first principles explanations. So in one sense, I've really screwed it all up too, because I didn't know I was triggering. Oops.
Well, so, maybe you've fixed your communication style such that it's hitting more closely to home for people, but, I'm asking a more fundamental question. Not everyone wakes up and thinks "the world should be more egalitarian and I'm going to work on that". It isn't enough to do that technically, but you also need to change the zeitgeist of the population, which is non-trivial. So you have to be somewhat fearless to wake up and do that every day, when, as a skilled developer, you could go get a job at a mega-corp and enrich yourself from the moat they've created.
Good point. Thank you. It's cause and effect, I'm not a developer because I was looking for a job, the opposite, I'm a developer because I was looking for the highest impact but lowest friction effort. The best way to change minds is to create tools they use regardless of whether they ideologically agree with them or not. Humans feel the urge to fulfill the purpose of what is in their hands, so if you can put more hammers than swords in them, more paint brushes than guns, you may get a more creative society. This does not override your agency, it's just a matter of directing probabilities. Let economics work at scale, and you'll see specializations develop, because at the end of the day, we're human, we love to play, and having a sculpt in our hands begs us to poke and prod at the universe, to discover how we can mold it. Interaction is our way to dance with reality and communicate beyond the now, but across time and age. It shows animals we can face the void, and have created monuments. We were here for more than just our paychecks.
Like what you saw here? Why not share it!
Or, better yet, share Console!
Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week