Hypershell: A Type-Level DSL for Shell-Scripting in Rust powered by Context-Generic Programming
26 points by soareschen
26 points by soareschen
I think this focuses a lot on the how, but what I was more interested in as someone new to CGP is the why. I read most of the post and do not feel like I understand why I’d want CGP or Hypershell.
For instance, the native HTTP streaming example doesn’t seem better to me than just writing it out with async Rust:
async fn main() {
let body = reqwest::get("https://www.rust-lang.org")
.await
.expect("HTTP get")
.bytes_stream();
let hash = hash_stream(body).await.expect("HTTP body sha256");
println!("{hash}");
}
async fn hash_stream<E: Error>(
bytes: impl futures::Stream<Item = Result<Bytes, E>>,
) -> Result<u64, E> {
todo!()
}
(full code, doesn’t compile on the playground because of a missing reqwest feature, but works locally)
What are the concrete benefits you see for CGP compared to code like this?
I think having the docs first show the limitations in the normal code, and then how CGP solves them would make the value proposition clear. It could be more involved code than the specific example I chose if the issues only arrise in larger programs.
Also, the post is very verbose. When you’re showing code I don’t think there’s value in explaining every single detail. I’m personally more interested in the new concepts the example introduces, and if I want the nitty gritty I’ll dig into the crate’s docs, or even build the example locally.
Hi ThinkChaos, thanks for the thoughtful feedback! You’re absolutely right that for the specific case of computing a checksum from an HTTP request, it can be done quite straightforwardly using a standard HTTP client like reqwest.
The example in the post is intentionally simplified to be accessible to readers from different backgrounds. The goal was to illustrate how to extend the DSL with new syntax, rather than to claim a practical advantage for that exact use case. That said, I do have a slightly more involved example here that might better show the value proposition — streaming from the Bluesky firehose via a native Websocket syntax.
The post is primarily aimed at developers interested in building DSLs in Rust, so it’s more a showcase of what CGP enables than a pitch to use Hypershell in production. But to your point, I can clarify what sets it apart from plain async Rust.
First, Hypershell is designed to make streaming pipelines across diverse domains — like HTTP, CLI tools, native Rust functions, and WebSockets — composable with a consistent, shell-like syntax. This removes the burden of manually stitching together different libraries and managing their interoperability.
Second, a core focus of Hypershell is to decouple program definition from execution. While the example uses reqwest, it could easily be swapped for another backend like isahc. This separation also enables additional features like mocking, caching, or proxying without changing the user’s code.
Finally, since the post is geared toward DSL design, I’d encourage you to look beyond shell scripting. For instance, there’s a brief mention of HTML rendering that hints at broader use cases.
I appreciate your perspective — and even if CGP and Hypershell don’t seem compelling right now, I hope the ideas around CGP will prove useful when you encounter new CGP-based DSLs or frameworks built by others that solve problems you care about.
Thanks for replying, here’s what I still don’t understand, hopefully it’s useful :)
That said, I do have a slightly more involved example here that might better show the value proposition
To me, this also seems easier to write using standard Rust abstractions. Maybe that’s because I don’t fully grasp CGP, but that’s the audience here! Hence why I think a comparison with CGP benefits being pointed out would help.
This removes the burden of manually stitching together different libraries and managing their interoperability.
The stitching is still done somewhere right? I.E. if I want to use a library you haven’t already wrapped I’ll need to do it myself?
While the example uses reqwest, it could easily be swapped for another backend like isahc. This separation also enables additional features like mocking, caching, or proxying without changing the user’s code.
It does but I can also do that with traits and generics. This is my core misunderstanding: why should I use CGP instead of those? At what point do the extra cons become worth it?
To me, this also seems easier to write using standard Rust abstractions.
CGP actually consists of a spectrum of programming patterns that can be used either as a combo or on their own. The simplest CGP patterns are very close to standard Rust abstractions, and you might not even need the cgp library to use them. The more advanced abstractions simply arise from the simpler abstractions once we follow along the design principles of CGP.
It is also worth noting that the benefits of abstractions would only become apparent if the same pattern is repeatedly used. But when given as examples, any abstraction introduced would only be used once, thus making them seem not as useful. If you are looking for more complex use cases, there are a few more examples available at the repository, including native Websocket, parallel comparison, and if statement.
In case if you are interested, I am also writing a book that covers the basic CGP patterns from the ground up.
The stitching is still done somewhere right?
Yes, you still have to first implement the interface for a library. But the main advantage CGP provides is that once you define the interface for one library, it can interop much easier with all other libraries without needing a “central” authority to manage that interop layer.
For instance, you can implement a Websocket provider for Hypershell. But once you do that, your code will seamlessly interop with the HTTP and CLI implementations with minimal overhead.
It does but I can also do that with traits and generics. This is my core misunderstanding: why should I use CGP instead of those?
CGP is the design pattern that arises from Rust traits to lift the coherence restrictions. The consumer traits can be any trait you would have implemented with vanilla Rust traits already. The main power that CGP provides are the provider traits, which can be generated by adding a #[cgp_component] to any vanilla Rust trait to turn them into a CGP trait.
So to answer your question, a good start would be just to define vanilla Rust traits, and then add #[cgp_component] to them once you hit a roadblock on needing to define overlapping or orphan instances. On the other hand, once you have that newfound freedom, you would realize that the coherence restrictions have been unconsciously shaping us on how we design our Rust traits. So if our mindset opens up, we would find a lot more trait designs out there that can significantly benefit from becoming a CGP trait.
At what point do the extra cons become worth it?
The compile time for CGP-based applications is actually pretty fast, if not faster, for non-DSL use cases, especially if we limit to only simple CGP patterns. On the other hand, steep learning curve and error messages can be a major challenge, especially for developers who are unfamiliar with functional programming.
Currently, the target audience for CGP is for developers who want to build reusable components or DSLs for other people to use. If you are building applications for your own use, there is an unfortunate dilemma that it is less obvious of the benefits CGP could bring, until the application grows and becomes too large and too complex.
The main advantage CGP provides is that once you define the interface for one library, it can interop much easier with all other libraries without needing a “central” authority to manage that interop layer. […] Currently, the target audience for CGP is for developers who want to build reusable components or DSLs for other people to use. If you are building applications for your own use, there is an unfortunate dilemma that it is less obvious of the benefits CGP could bring, until the application grows and becomes too large and too complex.
This seems like great info to put front and center in the intro docs! I didn’t see any explanation of this in the little browsing I did.
The main thing I had read was the CGP overview, and that’s pretty abstract, so not obvious how it would make my life better in practice.
And re the first: point my initial reaction is CGP is that central authority, so you might want to address that in the docs too.
Thanks for your feedback. I’m still working on how to present CGP in the clearest and most effective way, so it’s really helpful to hear what resonated and what was missing.
And re the first: point my initial reaction is CGP is that central authority, so you might want to address that in the docs too.
That’s a fair point. CGP is a central authority in the limited sense that it defines a few core traits, such as DelegateComponent and HasField, which are required for interoperation. However, these traits are intentionally minimal and designed to avoid the kind of feature creep often seen in other centralized systems.
Beyond those core traits, the CGP macros live in separate crates and mostly provide syntactic convenience for writing CGP-compatible code. These macros contain much of the implementation complexity, but they do not create a central bottleneck. Users are free to adopt different macro systems or surface syntaxes, and those will still interoperate with code written using the standard CGP macros.
One alternative way I’ve thought about presenting CGP is as a meta-framework for building domain-specific frameworks. But “framework” can feel too heavy or off-putting to some developers, especially in functional programming circles. That’s why I’m currently experimenting with the idea of framing CGP as a tool for building DSLs instead. Hopefully this makes the purpose and benefits clearer for more people.
Estimated reading time: 1~2 hours (≈16 500 words).
At that point, this isn’t a blog post, it’s a documentation site with all the pages cated together!
I think it would be useful to split some of these sections into separate pages (one large page is typically difficult to read or reference, even with lots of anchors throughout). Then, it would be great to see a high-level explanation for what Hypershell is, probably written in such a way that it also explains CGP for those of us who haven’t heard of that either. A good structure for that would be:
Hello everyone!
I’m excited to share Hypershell, a modular, type-level DSL for writing shell-script-like programs directly in Rust! It’s powered by Context-Generic Programming (CGP), which enables unprecedented modularity and extensibility, allowing you to easily extend or modify the language syntax and semantics.
I created Hypershell as a response to previous community feedback on CGP, seeking more practical application examples. Hypershell serves as a proof of concept, demonstrating how CGP can build highly modular DSLs — not just for shell-scripting, but also for areas like HTML or parsing. While it’s an experimental project, shell-scripting was chosen as a fun and approachable example for all programmers. The blog post covers features like variable parameters, streaming I/O, native HTTP requests, and JSON encoding/decoding within your shell pipelines
A core innovation behind this is CGP’s ability to bypass Rust’s trait coherence restrictions, allowing for powerful dependency injection and eliminating tight coupling between implementations and concrete types. This means you can customize and extend Hypershell in ways that are typically very challenging in Rust.
Please feel free to share your thoughts and feedback on the project. I’m here to answer any questions!
Echoing ThinkChaos’s comment I scrolled through quite a bit of it but still had no idea why I would use it.
In particular, my experience with heavily templated C++ libraries makes me nervous to use something that looks like it’s tossing around a lot of angle brackets – type level code usually takes longer to compile, has worse error messages, and is less ergonomic to write. It would help if you compared to how somebody would naively write the same script in bash or in non-Hypershell Rust and explain the advantages of the Hypershell version.
Looks interesting – I added a section for Rust here, with xshell and Hypershell, and a section for OCaml:
https://github.com/oils-for-unix/oils/wiki/Internal-DSLs-for-Shell
Feel free to edit the page!