EVC #9: Heterogeneous computing and pick-your-own-hardware architectures

Founder Bolt with Paul Butler, founder of Drifting in Space

In the last edition of the newsletter I talked about WebAssembly, and how helpful it can be to package code and make it work the same way, regardless of what platform it’s executed on. When it comes to web applications, cloud providers have already abstracted away all hardware-specific decisions; engineers know the high-level specs of their hardware (GBs of RAM, number of vCPUs, etc) but rarely know what they are working on.

There are two reasons that can push a a company to go down to the hardware level:

• Gross margins / cost: Dropbox is a great example; they started on cloud, but eventually built their own data centers to fit their specific need for storage. This helped them improve gross margins from 33% to 67% in the span of two years. Sarah Wang and Martin Casado wrote a great post on The Cost of Cloud and how it affects companies’ gross margins.

• Performance / revenue: hedge funds go all the way down to the fiber cables to get millisecond speed improvements. Funds like Jane Street even have hardware teams designing FPGAs to achieve the best possible performance for their specific trading use case.

Even leaving hardware changes aside, as engineering moved from on-prem to cloud, the architectural design of the backends we build hasn’t drastically changed. We still mostly treat the cloud as a shared on-prem for all of our customers. Having to support all user loads on the same backend leads to code that is highly over-engineered for the average use case, but necessary for the top percentile of use cases.

One of the companies I invested in, Drifting in Space, built Jamsocket, a framework that helps engineers support “session-lived backends”. Instead of having a shared pool of infrastructure, every time a user starts a new instance of your app in the browser, Jamsocket spins up a new backend specifically for them. This type of architecture can make it much easier to size infrastructure based on single-user use load, rather than one-size-fits-all. Similarly, Deepnote lets users select the exact GPU they want to power their notebook and bills users accordingly, rather than just having fixed “tiers”.

As more applications move to into the browser, this type of architecture will become more and more common. I’m excited to see what new use cases this type of architecture can unlock, and what other tools can be built to make it easy for engineers to support them. As usual, let me know if you are working (or thinking!) about anything in this space :)

Fun Reads

• ‎Signals and Threads: Programmable hardware with Andy Ray: a podcast by the Jane Street team on their work on FPGA, what it takes to design one, the economics of it, and more.

• How Figma’s multiplayer technology works (2019): Evan Wallace, CTO of Figma, gives a technical background of how their product handles multiplayer capabilities.

• Slack Incident on 2-22-22: Interesting article by Laura Nolan from Slack on their February outage. Spoiler alert: it was partially caused by a cache control plane named after a McDonald’s sandwich.

Founder Bolt ⚡️

Today’s Founder Bolt is with Paul Butler, co-founder of Drifting in Space, the company creating infrastructure for session-lived backends that I mentioned above. They are the maintainers of Spawner, their open source offering, and are working on Jamsocket, their hosted version. They also have other cool open source libraries like Aper, which helps with synchronizing data mutations across a network.

Paul and Taylor (his co-founder) are hiring for a Sr Rust Dev and a Typescript role if these problems sound interesting to you!

What’s one productivity hack that you really love?

I've recently become a Kanban fanatic. Any process that I can turn into a Kanban board, I do. We've used post-it note Kanbans to manage customer outreach and investments throughout our seed round, and we use Kanbans in Linear for our recruiting process and back-office tasks. I think of the Kanban as a state machine, with each column being a possible state; from this it is very clear what action needs to be taken (and by whom) in order to advance a task through that set of states.

What’s the latest product your engineering team has adopted at work?

We've been introducing NATS into our system as the connective tissue between nodes and our main platform. It provides a really nice, natural set of primitives to build on, and then gets out of your way.

What technology/architecture are you the most excited about trying? What about getting rid of?

I'm excited about WebTransport, a new way to do client/server communication from within the browser that uses UDP as the underlying transport layer (via QUIC). It differs from WebSocket in a number of ways, but the most notable is that it supports unreliable transport. This opens up possibilities for low-latency browser-based applications, as well as low-latency pixel-streaming via WebCodecs.

I'm looking forward to the industry diversifying beyond Kubernetes. It's a great piece of software, but its complexity scares a lot of developers away. While we will continue to use Kubernetes, we're moving towards using it just as our internal mode of deployment, and removing it as a dependency of our core platform.

What’s one side project you started but never finished?

I've made a couple of attempts at writing a 3D renderer that outputs just a vector wireframe. This is a surprisingly hard problem compared to rendering a raster wireframe, since with rasters you can just go back to front and "paint over" any obscured lines, but with vectors you have to calculate the intersections and slice up the lines. I was able to get some nice results, but I fell down a rabbit hole of trying to implement a fast data structure for a sweep line algorithm and never produced a finished project. It's basically how I learned Rust, though.