28 - Serverless Data Query Systems
Welcome to another edition! I'm Daniel Imfeld, and here I share things I've read recently, updates on what I've been working on, and occasionally nascent blog post drafts.
Book Progress
Progress has been… slow, but useful. I’m rethinking things a bit more, this time to start out introducing MapLibre instead of D3 at the very beginning of the book. MapLibre handles a lot of things that you need to do yourself in D3, so I think it will make it a bit easier for readers to jump in. I’ll still cover D3 later in the book with a focus on static visualizations. Up to 17,000 words now and climbing…
I am feeling like my progress and motivation on this book have plateaued, but I’m pushing through. Getting a little bit done each day until I break through the the motivation comes back.
Serverless Data Query Systems
I spent some time over the weekend reading about Honeycomb’s Retriever data query system, which is based almost entirely on S3 and Lambda. I may be building a similar system later this year, though not in the observability space, and having experienced how well Honeycomb’s system so wanted to see how their world-class querying worked under the hood.
There’s not a lot of information about it, but what is out there was useful. I wrote up my notes here. Notably, they chunk everything by timestamp, and there are no secondary indexes in the system. This is both because the nature of their system precludes favoring any particular column over another, and also to allow a high rate of ingestion with data becoming available almost immediately.
If I do end up developing my own system inspired by Retriever, rest assured I’ll be going into great detail about it here.
Links and Reading
Where to set the standard by Sarah Guo is an exhortation to move quickly. The biggest advantage startups have when trying to break into existing markets is the agility that larger, more established companies can not match. This short essay urges you to embrace that advantage, and not worry about high standards scaring people away. High performers embrace high standards.
Excuse me, is there a problem? by Jason Cohen is a great overview of analyzing the market for your potential startup. But the big contribution from this article is a quantitative framework for rating your business idea and classifying it as appropriate for a VC-funded scale up, a medium-growth bootstrapped business, or just something not worth pursuing as more than a side project.
Rich Harris, the creator of Svelte gave a talk about his spicy opinions on web development. Rich is well-spoken and makes good points, with a desire not just to convince but to spark discussion with those who disagree.