Console #26
Krakens, Mario in Vim, and Infection Monkeys
Vim-Mario
vim-mario is…you guessed it…Mario running in Vim 😊.
last commit: Jan 28, 2020, first commit: Aug 3, 2014
distribyted
distribyted is a torrent client with on-demand file downloading as a filesystem.
last commit: 8 days ago, first commit: Dec 8, 2019
Kraken
Kraken is a P2P-powered Docker registry that focuses on scalability and availability open sourced by Uber.
last commit: 27 days ago, first commit: December 16, 2016
Infection Monkey
The Infection Monkey is an open source security tool for testing a data center’s resiliency to perimeter breaches and internal server infection. Infection Monkey uses various methods to self propagate across a data center and reports success to a centralized Monkey Island server.
last commit: September 30, 2020, first commit: August 30, 2015
https://twitter.com/Guardicore
Guardicore is hiring a remote team lead to for this project, if you’re interested you can apply here.
Cartography
Cartography is a Python tool that consolidates infrastructure assets and the relationships between them in an intuitive graph view powered by a Neo4j database open sourced by Lyft.
last commit: 9 hours ago, first commit: March 01, 2019
Maci
Maci stands for “Minimal Anti-Collusion Infrastructure”. It is a proof of concept for an anti-collusion voting mechanism using the Ethereum blockchain.
You can view a video presentation of the project here.
last commit: October 18, 2020, first commit: June 16, 2019
https://twitter.com/weijie_eth
Pigasus
Pigasus is an Intrusion Detection and Prevention System (IDS/IPS) that achieves 100Gbps using a single FPGA-equipped server.
last commit: 2 days ago, first commit: September 05, 2020
Gron
Make JSON greppable!
gron transforms JSON into discrete assignments to make it easier to grep for what you want and see the absolute 'path' to it. It eases the exploration of APIs that return large blobs of JSON but have terrible documentation.
last commit: today, first commit: September 08, 2012
An Interview With Antonio Navarro Perez of distribyted
There was a lot of discussion around whether the distribyted clients would seed their content, and if so, to what extent, when I posted about distribyted in the /r/opensource subreddit. Would you like to respond to these questions, or, clarify?
Yeah, sure. My intention with this project is to bring the torrent world closer to people. I understand the concerns of not being able to become a seeder, that’s why I’ll implement a way to keep a torrent file always on disk.
For the other hand, distribyted is intended to run on a server, making other tools or services compatible with the torrent ecosystem. This implies that distribyted will be seeding content always.
Also it uses a LRU persisted cache, allowing it to share as much space as you have. That might be game changing, per example sharing huge datasets (you can use the dataset at the same time you are sharing it).
What is your background?
I’m from south of Spain, I have 12 years of experience in several companies. I worked for really different ones, like a national TV channel, SAP, national banks, several startups, making tools to analyze source code… I also have a really deep knowledge about git internals, I contributed to a pure Go git client in the past: go-git. Also I like a lot databases. I worked on the past making gitbase. To make that work, we created go-mysql-server (the idea started on a company hackathon) a SQL parser and analyzer that allows us to implement anything as a table. Right now it is in heavy development in this fork from dolthub people. They use it as the dolt SQL engine.
I started to be interested in programming when I was 15 more or less. I got a quite old pc and some really old books and started to learn BASIC. My first program was a really simple conversational app. My father works with computers, so I have always been interested in them thanks to him. I started working for small companies, after that moving to Madrid to learn as much as I could.
I used a lot of languages and frameworks. Right now in my daily job I use Go and is my default option for pet projects or proof of concepts I have in mind. It is really easy to start with, easy to use, and the most important thing, it is fun! I have experience with Java, Scala, Go, Javascript. I also like C and C++ for microcontrollers. All the languages are good for something, all of them have good things and bad things. You need to choose the correct one for your purpose.
If I gave you $10 million to invest in one thing right now, where would you put it?
I’m not an investor, but I would say: keep $3-4 million to develop a personal idea, and invest the rest in companies that are helping with remote work right now.
Would that personal idea be distribyted, or something else?
I would do some validation before choosing a specific idea. So it depends.
Do you mean technical validation via a prototype, or something else?
I would make a working POC to validate it technically, and after that, test the idea to see how customers respond.
Anything in particular in the remote work space you would write a check for?
Salsify is one project that I find interesting in the remote work space.
What's an opinion you have that most people don't agree with?
Work with people is better than work alone.
What are you currently learning?
Right now I’m learning about distributed consensus algorithms, different and innovative use cases for Bitmaps. I’m also trying to spend time learning Rust.
Why was distribyted started?
The main idea is not new, there are a lot of tools allowing a user to mount a torrent as a folder. But, I wanted something more usable, allowing me to mount several folders and torrents, and intended to be used on a personal server.
Also, I liked the idea of using the torrent as it is with no extra tools to extract compressed files to be able to use the torrent content, that’s why distribyted is able to get the content from compressed files (ZIP right now) directly.
I’m always thinking about how the distributed web will be, and all of the unresolved problems we still have. I just wanted to make something I had in my head as easy as possible for the final user, giving to them a piece of this distributed web.
Are there any overarching goals of distribyted that drive design or implementation?
Yes, make it simple. It should be really easy for the end user to use it. Make it compatible with all major OSes. And make it open.
What trade offs have been made in distribyted as a consequence of these goals?
I think the bigger tradeoff is the use of CGO and special libraries to be compatible with Windows.
What is the most challenging problem that’s been solved in distribyted so far?
Maybe make everything lazy. No reader is called until it’s necessary. Also, make it work on Windows, a work still in progress.
How did you go about making every lazy loaded?
distribyted’s internal API does not load the entire filesystem tree if not needed from the beginning. To navigate through the virtual filesystem you only need file and dir names, and file sizes to make it work well. In the case of compressed files, you only need to read the header when requested, making directory listing possible.
What’s your typical approach to debugging issues filed in the distribyted repo?
I first check the described use case to reproduce the error to try to understand it better. When I’m able to reproduce the error manually, I try to reproduce it using unit testing. After that, I should have one or more new tests failing in my test suite. Then I can start fixing the error to make these tests pass.
Is distribyted intended to eventually be monetized if it isn’t monetized already?
Right now the project is not monetized. Maybe in the future I will do some research to make it monetizable. I would love to be able to live maintaining open source projects.
How do you balance your work on open source with your day job and other responsibilities?
Contributing to open source is like a game, a way to disconnect and express my ideas. I would love to dedicate more time to open source, but right now it is not possible. Normally, I dedicate some time on weekends or after work, or try to work with companies that encourage contributions to open source projects.
What is the best way for a new developer to contribute to distribyted?
You can test it, report issues, make PRs, propose new features, and enjoy it!
Where do you see distribyted heading next?
I still have a lot of ideas in my head that I want to implement. Everyday I discover new use cases that distribyted could cover. After stabilizing what is already done, I will start adding new functionalities and more documentation about how to integrate with different tools.
Where do you see open source heading next?
I would love to see a kind of Patreon but for programmers in the future.
Do you have any suggestions for someone trying to make their first contribution to an open source project?
Try the project you want to contribute to. Read its code, try to understand as much as you can. Start with small issues, to learn the contribution process for that project. You can also help others if they have questions.
Last Week’s Answers
If you were going to choose a search engine today, which would you choose?
This Week’s Question
If you aren't using distribyted, what is currently stopping you?
Like what you saw here? Why not share it?
Or, better yet, share Console!
Also, don’t forget to subscribe to get a list of new open-source projects curated by an Amazon software engineer directly in your email every week.