Why I'm Building a Database Engine in C#
15 points by hongminhee
15 points by hongminhee
Nobody builds high-performance database engines in .NET.
See also RavenDB (in C#) or any number of databases in Java (Cassandra, Neo4j, QuestDB, Elasticsearch, Kafka, etc.).
To me the justifications against rust read as a skillfamiliarity issue (the developer is more familiar with C# than Rust) rather than inherent problems.
In C or Rust, you’d build much of this yourself or stitch together crates/libraries with varying quality. In .NET, this is production-grade and free: ILogger and OpenTelemetry for observability, BenchmarkDotNet for rigorous micro-benchmarks, NUnit for testing, IConfiguration for settings. All well-documented, all interoperable, all maintained by Microsoft or battle-tested OSS communities.
Rust’s ecosystem for the surrounding infrastructure (logging, DI, configuration, testing) is also less mature than .NET’s, and as a solo developer, my velocity matters. I chose the language where I ship faster.
Rust has tracing for logging, OpenTelemetry for observability, criterion for benchmarks, testing built in, serde + toml for settings (as well as confy / figment if you want that approach). DI is an OOP concept for the most part and easy to get the good bits in rust without a framework / blessed approach.
This feels like post hoc rationalization a bit. And that's ok.
This says a lot about how it can be done. I don't be believe we ever got around to why we would write another database engine.
Actually, it does say an awful lot about why a new database engine would be useful, specifically for high performance games using the entity component system (ECS) pattern.
Interestingly, the article doesn't mention Unity even once. It turns out that C# is used in a number of games as a side effect of the Unity game engine / libraries / toolset.
Slop.
Edit: both the article and the code.
Correct me if I'm wrong, and please do because I can hardly claim to have fully comprehended the whole article, but aren't you basically just showing that you can write C code inside a C# context? All of the solutions you mention force you to write C-like code and turn off all the extra stuff that C# provides. How is this different than writing a DB engine in Python, but all the important parts are just calling embedded C functions?
Conceptually, writing low-level C# might be similar to writing C, but it’s still C#, so you can compile it once into an architecture-independent binary format. You also get to use the same tools for authoring, debugging, and packaging.
Extending Python with C requires bringing in an entire separate ecosystem and dealing with architecture-dependent artifacts (cf. “wheels”), which is a big hassle, IMHO. Integrating build systems is a pain that I don’t think has ever been solved in a convenient, cross-platform way.
There's a world of a difference, because plain C# without all the low-level shenanigans is plenty fast in itself so you need to optimize far less code, Python will be 100 times slower. And there is no impedance mismatch at all between your optimized parts and the higher level parts, they are in the same language and need no marshaling at all, no handling in the build system and they can cohabitate in the same files.