Console #184 - Interview with Paul of Grist - a modern relational spreadsheet with Python support
Console #184 - Interview with Paul of Grist - a modern relational spreadsheet with Python support
Featuring Frigate, Tusky, and Grist
🤝 Sponsor
This space is reserved for sponsors that support us to keep the newsletter going! Want to support Console? Send us a note at osh@codesee.io
🏗️ Projects
Browse through open source projects on OpenSourceHub.io, add your project to get more exposure and connect with other maintainers and contributors!
Frigate
A complete and local NVR designed for Home Assistant with AI object detection. Uses OpenCV and Tensorflow to perform realtime object detection locally for IP cameras.
language: Python stars: 10876 last commit: today
repo: github.com/blakeblackshear/frigate
site: frigate.video
Tusky
An Android client for the microblogging server Mastodon
language: Kotlin stars: 2285 last commit: a week
repo: github.com/tuskyapp/Tusky
site: tusky.app
Grist
Grist is a modern relational spreadsheet. It combines the flexibility of a spreadsheet with the robustness of a database to organize your data and make you more productive.
language: JS stars: 5350 last commit: today
repo: github.com/gristlabs/grist-core
site: getgrist.com
🎙️ Interview with Paul of Grist - a modern relational spreadsheet with Python support
Hey Paul! Thanks for joining us! Let us start with your background.
I grew up on a small farm in rural Ireland, milking cows and herding goats. On the weekends, I learned to program on a ZX81, by sneaking up to it when my older siblings weren’t around, and randomly tweaking their programs to see what happened. While doing so, I noticed that I was indoors, warm, sipping tea, and not getting kicked or head-butted by farm animals. I wanted more of that sort of thing, so I went on to study computer engineering at the University of Limerick. After that, perhaps from nostalgia for the goats, I did a doctorate at MIT on robots that poke things to learn about them. If a robot ever pokes you to learn about you, my apologies (if you didn’t like it), or you’re welcome (if you did). This was the C/C++ phase of my life. Then I did some machine learning and computer vision at a small startup in NYC, for an early live video search engine that could detect people talking to their cats, streaming their coding sessions, or strumming guitars. This was my Python period. After that company was bought by Axon/Taser for use in law enforcement and surveillance, I moved on to another small startup in NYC that has so far had only the most benign of applications: Grist Labs. There’s still some Python in my life, but now Typescript dominates.
Who or what are your biggest influences as a developer?
I’ve a son in high school who is somewhat into tech, and I pay close attention to what he and his buddies get into. When they hacked together a Wordle variant by extracting some JSON from an academic paper on comparative linguistics, and getting ChatGPT to render it as a website, I knew it was time to pay attention. I also follow a lot of current and recent grad students, since that’s another good source of fresh ideas. For example Geoffrey Litt has an interesting vision for the future of software, and spent a lot of time wrestling with reactive databases (very relevant to Grist).
What’s your most controversial programming opinion?
At some point, I sat down to write a programming language called CosmicOS (cosmicos.github.io) for communicating with extraterrestrials. That’s not controversial exactly, but I do sometimes feel judged when I talk about it. For me, it is an interesting puzzle in communication, logic, simulation, and programming language design, and that’s all. Please don’t jump to the conclusion that I believe aliens have been coming to Earth to probe us! We have robots for that!
What is your favorite software tool?
Git. People complain, but those who lived through the before-times know that they were AWFUL. Also, it sounds like a breed of small goat.
Why was Grist started?
Grist is the brainchild of Dmitry Sagalovskiy. He was trying to do some organizing for the New York Math Circle nymathcircle.org, a math club. First they used a spreadsheet, which was easy but didn’t do everything needed. Then they used a custom site, with Django, MySQL, etc. That could do more, but needed a lot of nursing along. With Grist, stuff that almost works in a spreadsheet can actually work for real, with no need for app building. I joined the project because I’d had similar frustrations in some volunteer work for the Data Commons Co-op datacommons.coop, an organization with a kind of “bottom-up Google” vision. One of things we were trying to do at the Data Commons Co-op was to make a pooled directory of "alternative economy" organizations - cooperatives, community gardens, food co-ops, buying clubs, time banks, and so on. The Co-op's members are not individuals, but rather umbrella groups such as the US Federation of Worker Co-ops or the Solidarity Economy Network. Several of the Co-op's members had previously tried to run big splashy directories that proved expensive to maintain and failed once funding ran low. We wanted to reduce costs and raise quality by pooling effort. There's a lot of "gotchas" to avoid when doing that, I learned. One that struck me in particular was how important it was that groups were free to decide their own rules for who is "in" or "out" of the directory. Looking for agreement here is the path to madness, and not actually necessary. The technical challenge of designing a directory that could be efficiently added to and queried by each community from its own perspective was very interesting. It left me wanting to pitch in to make a really effective tool for data projects of all kinds. I felt bad that as programmers we have all these nice things for collaborating around code, but whenever I saw a group try to work on data together it looked so painful. It really doesn’t have to be that way.
How does Grist work?
Grist has three main parts. First, there’s a relational database (SQLite). The tables in your Grist spreadsheet are stored exactly as tables in SQLite, with all special spreadsheet fanciness kept in separate metadata tables. That separation does make for some extra work, for example when tables and columns are renamed, but it draws a reassuringly clear line between data and layout. The second part of Grist is a “data engine”, written in Python, whose purpose is to implement the best parts of spreadsheets, such as formulas and dependency-tracking between cells. Grist uses Python formulas, with Excel-like macros available via Python implementations. The third part of Grist is a node server to enable live collaboration and to enforce a very flexible security model. A cute trick in Grist is that you can use formulas to define who has access to data. So if there’s some very specific rule you have about who can see which rows, based on who the user is and what exactly is in the row, you can totally make that happen. This is a game-changer, since it means you can invite people to collaborate on a spreadsheet that in the past you’d have to keep at a distance, forcing you to do awful things like manually importing their emailed CSVs or (worse) randomly-formatted Excel files. Beyond that, Grist is extensible in a few ways. If you know how to write some HTML/JS/CSS to render a page from some JSON input (like my son and his friends did with ChatGPT), then you can immediately apply that to spreadsheet data in Grist. Grist will take care of sorting, filtering, and linking the data with the rest of the UI. So for example if we hadn’t created a map widget, you could do it yourself pretty easily, and others could use your widget without fuss.
Everything needed to view, edit, and collaborate on spreadsheets is open sourced. Some material used to operate our own SaaS, and some features developed for particular enterprise clients, are not. We’ve packaged Grist in a few forms: as a ready-to-go docker image for use as a server, as a desktop app (handy for offline use), and compiled to pure JavaScript for use on static websites without any special back-end.
Where did the name for Grist come from?
It is a portmanteau of Grid and List. I suppose we could have called it Lïd instead?
Who, or what was the biggest inspiration for Grist?
Frankly for me the biggest inspiration is sites like GitHub that have had a big impact on collaborative coding. I aspire to have something like that impact, this time for collaborative data. There’s a long road to go.
Are there any overarching goals of Grist that drive design or implementation? If so, what trade-offs have been made in Grist as a consequence of these goals?
The initial vision for Grist was a desktop spreadsheet app, with sharing via end-to-end encryption, and with none of your data stored or accessible outside your devices. We heard a lot of enthusiasm for the idea, but had a hard time getting people to actually install the app. So we switched to making a more conventional web app, where data is stored in a central server and there’s nothing to install locally. That really helped adoption, but was definitely a trade-off. There are hooks in Grist sitting and waiting for the day we can return our attention to this.
What is the most challenging problem that’s been solved in Grist, so far?
Granular access control was a doozy. I mentioned this feature earlier, but to recap: People often make copies of spreadsheets because there’s someone who should see just part of it, not the whole thing - but then the spreadsheets get out of sync quickly. We wanted to make this whole mess unnecessary, by allowing very flexible access control. We wanted to allow expressing access control as formulas, reusing experience that many spreadsheet users already have. So you could say, for example, that a user should be able to see rows for cities they manage, and be able to edit the ones assigned to them. Making the whole system efficient, and able to correctly propagate live changes to all users despite their differences in access rights was challenging. I think we only managed to do it because of a period of enforced tranquility during the pandemic lock-downs. We are currently working on a new vision for access control that we hope will make it not only flexible, but also easy, so even more users can take advantage of it. Our codename for this project is WYSIWYS (“What You See Is What You Share”). Under the hood, it will translate visual layouts into the rules needed to protect your data. If you want to dip into the granular access code, a lot of it lives here.
Are there any projects similar to Grist? If so, what were they lacking that made you consider building something new?
Airtable is often mentioned, as it is a spreadsheet/database hybrid. For me, I want a data container that can go anywhere the data is - in the cloud, on a desktop, on an airgapped network etc, and I want to be sure the container will work into the indefinite future. Airtable isn’t that. Baserow is sometimes mentioned also. I think it got started a few years after Grist. Something I value about Grist is the robustness of its data format. You can download a spreadsheet and upload it again in a different Grist installation without any fuss. You can read data from it with regular SQLite tooling. I feel I can trust it as an archive format. Baserow goes another route, using PostgreSQL for storage. That’s a great choice, and I hope Grist will support it too in future, but it does mean that individual spreadsheets are not as naturally self-contained as they were with, say, Excel.
What was the most surprising thing you learned while working on Grist?
That a small team building something general purpose, rather than solving a specific problem as all the sage advice urges, can actually succeed to a remarkable extent. It is a very delicate balancing act. Hats off to Dmitry for getting us this far.
How popular is Grist? How did it get popular?
We are big in France! Grist has seen heavy use at a very cool agency in the French government for coordinating local and regional projects, L’Agence Nationale de la Cohésion des Territoires (ANCT - here’s their description of Grist). The main reasons they like us are that we are open source, have good quality code, and plenty of features. They are particular fans of granular access control. They run probably the largest Grist service apart from ourselves, and have contributed many wonderful fixes and features. In the world at large, I’d say we’re just starting to get significant attention. We’ve had a few lucky posts on the selfhosted subreddit, and a couple of front-page Hacker News stories that have raised our profile in the tech community. Successful Appsumo and Product Hunt launches have helped also.
What is your typical approach to debugging issues filed in the Grist repo?
We have a very large and robust collection of tests, so most problems are caught long before they get to users. But if they do get through, there’s no real magic to figuring out the problem. One approach is to work with the user to find a way to replicate the issue, and then to strip things out until the problem goes away. Then you know that somewhere between the starting point and the ending point lies the problem, and you can just step between them to find it.
What is the release process like for Grist?
We aim for a monthly pulse, with coordination between the technical side of releasing a new stable version by our development team, and preparation of a lovely newsletter describing what has changed in the month by our marketing team. For our SaaS, we deploy changes on a weekly cycle, with disruptive changes behind temporary feature flags.
How is Grist monetized?
We have a SaaS, standing ready and willing to host your Grist spreadsheets so you don’t have to, with free and paid plans. We also have some enterprise clients. Because we were bullied into it, it is possible to sponsor us on GitHub, and a small number of very determined philanthropists do so. We are planning to release Grist on marketplaces such as the AWS Marketplace, so people can choose to pay for the convenience of delegating installation and configuration of self-managed Grist sites to us.
What are you most proud of?
When I hear developers from ANCT (the French government agency I mentioned) describe to others how and why they use Grist, and see the impact it is having, it gives me goosebumps. Recently we were co-presenting at a conference with them, and I got to see the full scope of their work. I realized that what they’ve built with Grist aligns closely with the vision of the Data Commons Co-op I volunteered at, but executed with way better funding. It makes me excited to close the circle and bring lessons from Grist users back to the grass-roots projects that originally motivated me to get into all this.
Have you ever experienced burnout? How did you deal with it?
Yes, I’ve come close. As Grist gets more widely known, we’ve also been scaling up our team, but there are some weeks when it feels like everything everywhere all at once is happening, and that’s too many things. I believe I deal with it by reflecting on my place in the universe and letting perspective soak back in. I have one of those small disks that says “The world was created for me” on one side, and “I am dust and ashes” on the other, and flipping that back and forth can help. I also become grumpy and reclusive, which can do wonders for naturally reducing my workload too.
What is the best way for a new developer to contribute to Grist?
If you are comfortable building and running tests, check out the “good first issue” label on the Grist repository. If you are more comfortable making web pages, you could read up on custom widgets and then think of a clever new way to visualize data in a table and add that. It can be as simple as a web page rendering some JSON you are handed by Grist. A pedigree chart for goats, perhaps, or an interpretation of some data as sheet music.
Where do you see the project heading next?
We’ve got a lot of fun features under development, some in collaboration with ANCT. I’d love to see two or three more ANCT-level contributors to the project, to get more done faster. Spreadsheets have a long tail of features that are really critical to someone, so there’s a lot to do. Any capable teams out there for whom Grist is great but would be better with just a few more key features, let’s talk!
Where do you see software development heading next?
It is hard to predict. LLMs and other models seem likely to change software development as much as the invention of the if statement. They are a new and very different building block, full of caprice and whimsy, which is not really a problem we’ve had to deal with in computer science before. I’m looking forward to it.
Where do you see open-source heading next?
I see open-source software as being the space within which humanity’s long term digital culture and treasure is accreting. Commercial software is important and impactful in the moment, but ultimately froth on a deeper tide. I was struck, when reading Vernor Vinge’s books, by his vision of software archaeologists in a distant future where, for example, buried deep in their infrastructure, the Unix epoch is still a thing. A glorious vision!?
Do you have any suggestions for someone trying to make their first contribution to an open-source project?
Before making a pull request, open an issue first to describe what you’d like to do, and why, and to check if the work would be welcome. Try to look at things from the perspective of the project maintainer. Everything you do will create work for them, from answering your questions, to reviewing your work, to suggesting improvements. So take a little extra time to make things easy for them any way you can. Don’t be discouraged by hostile or unresponsive people! Persist a bit, then move on - there are many happy, generous souls out there.
What is one question you would like to ask another open-source developer that I didn’t ask you?
What would you be doing with your life if we hadn’t somehow taught sand to think?
Back to you, what would you be doing with your life if we hadn’t somehow taught sand to think?
Trying to convince something even more unlikely to think? I have a whole theory of how to build logic gates with interwoven vines. Come to think of it, I used that as part of CosmicOS (e.g. here’s a D gate which compiles to this in the message).
Loved this interview? Want to get the future ones too? Subscribe to the free newsletter now!
Interested in sponsoring the newsletter or know of any cool projects or interesting developers you want us to interview? Reach out at osh@codesee.io or mention us @ConsoleWeekly!