Console #130 -- Interview with Valerian of Sonic - a lightweight search backend
Console #130 -- Interview with Valerian of Sonic - a lightweight search backend
Featuring Blip, Conductor, and Sonic
š¤ Sponsor - PostHog
PostHog: Open-source product analytics you can self-host
With PostHog's open-source analytics platform, customer data never has to leave your infrastructure.
Featuring product analytics, session recording, feature flags, heatmaps, experimentation, and more - all available for free!
The best part is everything is seamlessly integrated, so you can spend less time configuring different tools and more time getting valuable insight into user behavior.
šļø Projects
Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!
š blip
Blip is a testing tool for seeing your Internet latency and packet loss. You can try it at gfblip.appspot.com
language: JS, stars: 1279, issues: 12, last commit: 6 June, 2017
repo: github.com/apenwarr/blip
site: gfblip.appspot.com
ā” Conductor
Conductor is a platform created by Netflix to orchestrate workflows that span across microservices.
language: Java, stars: 5615, issues: 70, last commit: 6 days
repo: github.com/Netflix/conductor
site: conductor.netflix.com
š¦ Sonic
Sonic is a fast, lightweight & schema-less search backend. It is an alternative to Elasticsearch that runs on a few MBs of RAM.
language: Rust, stars: 16,749, issues: 57, last commit: 17 days
repo: github.com/valeriansaliou/sonic
š¤ Interview With Valerian of Sonic
Hey Valerian! Thanks for joining us! Let us start with your background. Where are you from, where have you worked in the past, how did you learn to program, and what languages or frameworks do you like?
Hello! I am Valerian Saliou, co-founder and CTO of Crisp, at which we build a customer support software, where we provide features such as eg. a chatbox for websites, a chatbot and much more. I spend most of my time building software across a wide range of programming languages: Rust, Golang and JavaScript, from the backend to the frontend. I also do UI design on the side.
I was born in France, where I still live most of the time although I am also a part-time Digital Nomad. Before I founded Crisp with my co-founder, Baptiste Jamin, I have been working at Ericsson, and I also ran a popular open-source project called Jappix (also in the messaging sector).
Who or what are your biggest influences as a developer?
I learned to code shortly after I installed Linux on my first computer. I started with PHP and basic HTML/CSS, and later learned to code vanilla JavaScript.
I canāt really say a specific person or fellow developer influenced me at the time, Iād say the open-source community in general was a huge motivational factor as to why I learned programming doing open-source software. You get part of a fantastic community of passionate people, where the currency of exchange is measured in lines of code and contribution ā not dollars. This makes open-source software very accessible when you are 15 (as I was at the time).
However, as I grew up, I knew I wanted to start my own company; the people influencing me shifted from online open-source communities to Silicon Valley tech people. I then considered tech-engineers-entrepreneurs as models to follow.
Whatās your most controversial programming opinion?
I do not believe that tests, unit & functional, are necessary in most cases when building Web applications. I am aware that this goes against standard industry doxa, for which I often have to justify myself to other fellow developers.
What I typically notice amongst projects is that you can hide shitty code behind a lot of tests. However, the code is still shitty and will eventually become unmaintainable and unproductive to work with.
I believe that code quality and clarity is of uttermost importance. A well thought code design, with clear coding patterns, removes the need of tests. As a developer & entrepreneur, time is your most important asset. In doing the following, I manage maximize the utility / output of my work, versus spending half of my time writing tests for everything.
What is your favorite software tool?
Sublime Text. I do not like complex and heavy IDEs, I like to program in a simple and distraction-free environment, full screen mode. Iām easily distracted, therefore reducing the amount of information on screen helps me focus and be productive.
What is your favorite book and why?
I own many books that I enjoy. One of the books I have read that was particularly influential on me is The Sovereign Individual. It dates back from 1997, but predicted what we are seeing happening in the world right now: remote work and Digital Nomads roaming across countries while living and earning their money in a common global cyberspace. A world where you can basically move from one country to the other on the sole basis that you can operate your business on better terms. It is not an all-rosy book, as it also depicts the end of the industrial/post-industrial era system which brought so much wealth to everyone in the society. The big question is: whatās ahead?
If you could teach every 12 year old in the world one thing, what would it be and why?
Aside from programming, Iād teach them CAD and 3D printing, with a bit of electronics on the side. Thereās so much you can invent with this knowledge set, and so many issues you could help the world with, although thatās probably a lot to learn at once!
If I gave you $10 million to invest in one thing right now, where would you put it?
Most likely in the energy sector. Itās been under-invested for decades now, we need more research in nuclear energy and smaller scale nuclear reactor projects. Unsure though that $10 million would be enough to make any impact in this industryā¦!
For now, I am focusing my efforts in the renewables sector, by investing in photovoltaic power plants throughout France. I lend money to local companies that install PVs on agricultural sites roofs in the country-side. This brings extra income to farmers throughout the country, while making the local power grid a bit greener.
Note that, since itās already mostly green there in terms of CO2eq per kWh thanks to nuclear, I am now looking to proceed to similar investments in neighboring EU countries. I realized that I can have the most impact in terms of CO2eq reduction per kWh in gas or coal powered countries; certain countries in the EU emit 10-20x more CO2eq per kWh than the greener ones.
What are you currently learning?
Everything about coffee! After traveling to South America recently, I then noticed how terrible the coffee we drink in Europe and the US is.
Iāve upgraded my coffee maker. From now on, I mostly drink Espressos and buy freshly roasted grains. Also, I am reading the The World Atlas of Coffee to educate myself.
I realized that I was fond of sour coffee taste, which you can easily get when doing Espresso under high water temperature and high pressure, using coffee grains with an acidic characteristic. This occurs when you under-extract the coffee, meaning that the water flows too quickly through the grounds, therefore the ratio of acids you get out of the grounds is much higher compared to other flavors. If my description didnāt sound very tasteful, do not worry: a lot of people donāt like this coffee taste!
What have you been listening to lately?
Mostly progressive electronic music playlists. Thatās my fuel for coding when low on energy.
Why was Sonic started?
Crisp, which I founded, is growing fast, meaning that the number of messages that we store is also increasing dramatically. This is where we decided to build Sonic to make it possible for our users to search in messages, in a way that is cost efficient and minimizes sysadmin time. Sonic was also a great way to learn the internals of a search system, eg. the algorithms and data structures involved.
After analyzing all the open-source solutions available to us, we figured out that they did not fit our use case, that is, indexing billions of messages in a way that is the most RAM and disk-compact possible. Obviously, there are tradeoffs as nothing is free.
Where did the name for Sonic come from?
The Sonic name comes down from "Sonic the hedgehog", the popular video game where you control a super fast hedgehog. No wonder why I picked this name for a Rust-based search index!
Who, or what, was the biggest inspiration for Sonic?
Other open-source projects were resourceful for me to learn about the foundations of a search index.
What is the most challenging problem thatās been solved in Sonic, so far (code links encouraged)?
Mutex dead-locks post-release while running Sonic in production. Those were hard on me! A mutex is a lock you can acquire when acquiring a resource, eg. working on a variable in memory, so that other parts of your code that might be running at the same time do not overwrite what you were doing (code running in parallel eg. on another CPU core). If you forget to release the lock when you are done, then next time someone will acquire the lock the ālock.acquire()ā code will stay waiting forever. However, note that most dead-lock conditions are more complex and obscure than that, eg. they occur when you acquire multiple locks in a row, and some other code acquires the same locks in another order.
The consequence of a dead-lock condition occurring: our live Sonic instance was freezing at random times (sometimes multiple times a day), and I had to investigate iteratively over the span of multiple weeks to find the culprit. In the end, I needed external help from my co-founder, Baptiste, who doesnāt do Rust and didnāt work on Sonic ā he found the issue in a few hours!
Are there any competitors or projects similar to Sonic?
Meilisearch. However, I wouldnāt say competitors as they are friends.
Is Sonic intended to eventually be monetized if it isnāt monetized already?
I do not intend to monetize Sonic anytime soon. I run Crisp for a living, which as a company contributes to OSS projects and has a strong open-source culture.
How do you balance your work on open-source with your day job and other responsibilities?
Most of the time I work on open-source, there is a business reason on my agenda. Iād typically implement OSS features that serve Crisp. I see my OSS projects as a way to scratch my own itches and share my work so that other developers do not have to reinvent the wheel. Take Vigil, Bloom, Constellation, Sonic and Sales Tax: all four OSS projects were started because of the needs we had along the way building Crisp.
Have you ever experienced burnout? How did you deal with it?
I never experienced true full-fledged burnout, although I have sometimes been pushing so hard on myself while working on Crisp that I started to feel disconnected and disinterested from my work, which also happens to be a passion.
I usually manage to identify signs of burnout early when they occur, after which I take a few days off away from work. Take it easy, and come back later full of energy. Burnout is quite easy to identify when your work is also a passion, I assume itās trickier to catch if it isnāt.
What is the best way for a new developer to contribute to Sonic?
Iād say that you should take a look at the pending issues, and find something quick and simple to solve. Then, as you learn, gradually take on more demanding issues.
Where do you see the project heading next?
Sonic being a side-project of mine, I am not working on it full time. I however sometimes implement features, especially when they come handy for our main use-case, that is Crisp.
The features I will prioritize for future versions of Sonic are:
- High on the list of features to implement is clustering capabilities, so that Sonic instances can be mirrored (or even potentially sharded) to form a cluster in a High Availability scheme. That is, if one Sonic instance goes down, then clients can still read and write to the search index using other available instances.
- Then, come support for diacritics (ie. ignoring them), eg. "FranƧais" vs "Francais" should return the same search results.
- Also, holding a language-based dictionary of synonyms would help understand what the user wanted to search for and look for potential synonym matches if there are no exact nor fuzzy matches (eg. if the user looks for 'automobile', we may as well search for 'car').
- Last, but lower on the priority list, is to make Sonic embeddable as a library. Some developers have expressed the need of embedding Sonic into their projects, rather than using it as a server exposed over the network. You could imagine it being used in some hardware projects eg. automotive, where a Sonic instance would index data on an offline computer and allow for local searching, eg. like macOS's Spotlight Search.
Are there any other projects besides Sonic that youāre working on?
Yes! On the side, I am working on an open-source project called Prose. The goal is to build a high-quality suite of workspace messaging apps giving businesses and organizations more choice as to where their data goes (versus using Slack and Teams).
At home, I own a 3D printer. Whenever I have some time, I am also experimenting with IoT, more specifically designing HomeKit-compatible devices from scratch (eg. working in Fusion 360 for a casing design, building simple PCBs and programming in C++). Last project I made was a connected water level sensor for my plants water tanks.
Do you have any suggestions for someone trying to make their first contribution to an open-source project?
Contribute to a project you use often, which you deeply understand. Get familiar with how the code is organized, fork the projectās repository and try to follow the project developers coding style.
Do something simple at first, and document and explain it in a clear way to the maintainers. Popular projects receive LOTS of contributions and maintainer time is limited. Do not open a PR if you feel that the project maintainer will spend more time reviewing your work than coding the feature themselves.
Want to join the conversation about one of the projects featured this week? Drop a comment, or see what others are saying!
Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!